Senior Databricks Engineer

Job Summary:

We are seeking an experienced Databricks Developer with 6-9 years of expertise in designing, building, and supporting modern Data Lakehouse solutions. The ideal candidate will have strong hands-on experience in Databricks, PySpark, Delta Lake, cloud platforms, and large-scale data processing. The role involves building scalable data pipelines, optimizing distributed data workloads, and enabling advanced analytics and AI-driven initiatives.

Key Responsibilities:

• Design, develop, and maintain scalable data engineering solutions using Databricks.
• Build and optimize batch and real-time data processing pipelines.
• Develop ETL/ELT frameworks using PySpark and Spark SQL.
• Implement Delta Lake architecture and data governance standards.
• Process large-scale structured, semi-structured, and streaming datasets.
• Optimize Spark jobs and cluster performance for maximum efficiency.
• Develop reusable frameworks, notebooks, workflows, and data products.
• Collaborate with business teams, data scientists, analysts, and architects to deliver enterprise-grade solutions.
• Implement monitoring, alerting, security, and compliance requirements.
• Participate in code reviews, deployment activities, and production support.
• Develop technical documentation and mentor junior engineers.

Required Technical Skills:

Databricks & Big Data:

• Azure Databricks
• Databricks Lakehouse Platform
• Delta Lake
• Spark SQL
• PySpark
• Databricks Workflows
• Unity Catalog
• Delta Live Tables (DLT)
• Structured Streaming

Data Engineering:

• Data Lake Architecture
• Lakehouse Architecture
• ETL/ELT Development
• Data Modeling
• Data Integration
• Performance Tuning
• Data Quality Frameworks

Programming Languages:

• Python
• PySpark
• SQL
• Scala
• Java
• Shell Scripting

Cloud Platforms:

• Microsoft Azure
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS)
• AWS
• Amazon S3
• AWS EMR
• Google Cloud Platform (GCP)

Databases:

• SQL Server
• Oracle
• PostgreSQL
• MySQL
• MongoDB
• Cassandra

Streaming & Messaging:

• Kafka
• Event Hubs
• Spark Streaming
• Azure Stream Analytics

DevOps & Tools:

• Git
• Jenkins
• Azure DevOps
• CI/CD Pipelines
• Terraform
• Unix/Linux

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or related field.
• 4-9 years of IT experience with at least 4+ years of hands-on Databricks and Spark development experience.
• Strong expertise in distributed data processing and Lakehouse architecture.
• Experience building enterprise-scale data engineering solutions.
• Strong knowledge of PySpark, Spark SQL, and performance optimization.
• Experience with Agile development methodologies.
• Excellent analytical, troubleshooting, and communication skills.

Preferred Qualifications:

• Experience with cloud-native analytics platforms on Azure, AWS, or GCP.
• Knowledge of Machine Learning workflows within Databricks.
• Experience implementing Delta Lake and Unity Catalog.
• Exposure to Data Governance and Data Security frameworks.
• Relevant Databricks, Azure, or AWS certifications.

Key Competencies:

• Data Engineering Excellence
• Performance Optimization
• Solution Design
• Stakeholder Management
• Collaboration and Leadership
• Ownership and Accountability

Nice to Have:

• Snowflake
• Azure Data Factory (ADF)
• Delta Live Tables (DLT)
• Unity Catalog
• MLflow
• Airflow
• Kafka
• Data Mesh Architecture
• Generative AI / AI Engineering exposure
• dbt

Skills

Azure Databricks, DLT, Databricks, Delta Lake, Delta Live Tables, PySpark, Spark, Spark SQL