Databricks Engineer II (PySpark)

We are looking for a Databricks Data Engineer who can build and optimize pipelines within Databricks, Delta Lake, Unity Catalog, and PySpark.

Role Summary:

Build and maintain scalable data pipelines and Lakehouse solutions on Databricks, delivering clean, reliable data to analytics and ML teams.

Requirements:

• Databricks - Unity Catalog, Auto Loader, Spark, PySpark, Workflows, and Jobs
• Advanced SQL - mastery of window functions, CTE, stored procedures, joins, aggregations, and query optimization
• Delta Lake - ACID transactions, time travel, schema evolution
• PySpark, incremental pipelines, batch data pipelines, and streaming
• Familiarity with Unity Catalog, data lineage, and access control
• Cloud experience: Azure (ADLS Gen2, ADF) or AWS/GCP equivalent
• Comfortable with Git, notebooks-as-code, and automated testing
• Knowledge of CI/CD pipelines, DevOps practices, and version control (Git)

Key Responsibilities:

• Design and implement ELT pipelines using PySpark and Databricks Workflows
• Write efficient, reusable, and scalable Python code for data processing and automation tasks
• Orchestrate complex workflows using Azure Data Factory and integrate with other Azure services
• Collaborate with analysts and data scientists to model and deliver trusted datasets
• Monitor and troubleshoot data pipelines, ensuring high availability and reliability

Offers of employment are conditional upon passage of screening criteria applicable to the job.

Skills

Databricks, PySpark, SQL