Databricks Data Engineer

Role Overview

A leading global technology organization is looking for a Databricks Data Engineer to design, build, and optimize modern data platforms in a cloud environment. This role focuses on developing scalable data pipelines, ensuring data quality, and enabling advanced analytics and AI use cases.

The ideal candidate brings strong experience in big data processing, cloud technologies, and the Databricks ecosystem, with a focus on building efficient, secure, and production-ready data solutions.

Key Responsibilities

Data Engineering & Pipeline Development

• Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, and processing.
• Build high-performance data workflows using the Databricks ecosystem, including Delta Lake and structured streaming capabilities.
• Implement reusable data transformation and validation logic for consistency across pipelines.
• Ensure data quality, integrity, and reliability across all data processing stages.

Cloud & Platform Engineering

• Develop and deploy data solutions on at least one major cloud platform:
• Amazon Web Services
• Google Cloud Platform
• Microsoft Azure
• Configure secure data environments, including access controls, auditing, and data governance.
• Enable cross-platform data access and sharing using modern data architecture patterns.

Advanced Data Processing & Optimization

• Build efficient data transformation pipelines using PySpark and Python.
• Optimize performance of data jobs through tuning, resource management, and cost optimization techniques.
• Implement data layering strategies (e.g., Bronze/Silver/Gold architecture) for structured data processing.

Data Integration & Analytics Enablement

• Integrate data platforms with BI and analytics tools such as:
• Microsoft Power BI
• Tableau
• Looker
• Develop dashboards and reporting solutions to monitor KPIs, pipeline health, and operational metrics.
• Support data sharing initiatives and enable collaboration across teams and domains.

AI/ML & Data Innovation

• Prepare datasets for advanced analytics and AI use cases, including support for machine learning workflows.
• Work with modern tools for experimentation, feature management, and model lifecycle support.
• Explore new technologies and approaches to enhance data engineering capabilities.

DevOps & Deployment

• Package and deploy data pipelines using CI/CD practices and version control systems.
• Collaborate with engineering teams to ensure reliable and repeatable deployments.

Qualifications and Experience

• At least 2 years of experience in data engineering or related roles.
• Strong hands-on experience with Python and PySpark for data processing.
• Proven experience building data pipelines in cloud-based environments.
• Solid understanding of data modeling, data warehousing, and ETL/ELT concepts.
• Experience working with large-scale, distributed data systems.

Technical Skills

• Expertise in:
• Apache Spark / PySpark
• Delta Lake and modern data lakehouse architecture
• SQL and data transformation logic
• Familiarity with:
• Data orchestration tools (e.g., Apache Airflow)
• Big data technologies (e.g., Hadoop, Kafka, Hive)
• ETL tools such as Informatica, Talend, Matillion, or Fivetran
• Data build tools (e.g., dbt)

Preferred Qualifications

• Databricks or cloud certifications
• Exposure to machine learning concepts and frameworks
• Experience with DevOps practices and CI/CD pipelines
• Knowledge of real-time streaming and serverless architectures

Skills and Competencies

• Strong analytical and problem-solving skills
• Attention to detail and commitment to data quality
• Ability to work independently and in collaborative environments
• Effective communication skills for both technical and non-technical stakeholders
• Adaptability in a fast-paced, technology-driven environment

Why Consider This Role

• Opportunity to work on modern data platforms and cloud technologies
• Exposure to AI/ML and advanced analytics use cases
• Hands-on experience with Databricks and lakehouse architecture
• Collaborative environment focused on innovation and continuous improvement