Databricks Developer

We are looking for an experienced Databricks Developer with strong hands-on expertise in building scalable data pipelines and solutions on Azure Databricks. The ideal candidate should have experience in data engineering, data transformation, and cloud-based analytics development.

Responsibilities:

• Develop, optimise, and maintain data pipelines using Azure Databricks.
• Work with PySpark/Spark SQL for large-scale data processing.
• Build ETL/ELT workflows integrating various data sources.
• Implement best practices for data quality, performance tuning, and cost optimisation.
• Collaborate with data architects, business stakeholders, and analytics teams to deliver end-to-end solutions.
• Deploy and monitor Databricks workflows and jobs in production environments.
• Manage Delta Lake, notebooks, and data versioning.
• Work with Azure services such as ADLS, ADF, Azure Synapse, etc. (preferred).

Requirements:

• Strong hands-on experience with Databricks and Azure Databricks.
• Expertise in PySpark, Spark SQL, and distributed data processing.
• Good understanding of Delta Lake, data lakehouse architecture.
• Experience with Azure ecosystem: Azure Data Factory, ADLS, Azure DevOps (added advantage).
• Solid understanding of ETL/ELT concepts and data modelling.
• Ability to troubleshoot performance issues and optimise code.
• Strong analytical and problem-solving skills.

Skills

azure, databricks, pyspark, spark, spark sql