Databricks Engineer

We are looking for a skilled Databricks Engineer to design, develop, and optimise scalable data pipelines and data processing solutions. The ideal candidate should have strong experience with Databricks, Apache Spark, Python, SQL, and cloud-based data platforms, with the ability to transform complex datasets into reliable, analytics-ready data.

Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Apache Spark.
• Build and optimise data processing workflows using PySpark and SQL.
• Work with large datasets to support analytics, reporting, and data-driven applications.
• Develop and manage data lakehouse solutions using Delta Lake and Lakehouse architecture.
• Implement data ingestion, transformation, validation, and quality checks.
• Optimise Spark jobs and Databricks workloads for performance and cost efficiency.
• Integrate data from multiple sources, including databases, APIs, and cloud storage.
• Work with cloud platforms such as Azure, AWS, or Google Cloud.
• Implement workflow orchestration, monitoring, error handling, and troubleshooting.
• Collaborate with data engineers, analysts, and software teams to deliver reliable data solutions.
• Follow best practices for data security, governance, access control, and documentation.

Requirements:

• 3-5 years of experience in Data Engineering or a related role.
• Hands-on experience with Databricks and Apache Spark.
• Strong programming skills in Python and PySpark.
• Strong SQL skills, including query optimisation.
• Experience building and maintaining ETL/ELT data pipelines.
• Knowledge of Delta Lake, data lakehouse architecture, and distributed data processing.
• Experience with at least one cloud platform: Azure, AWS, or GCP.
• Understanding of data modelling, data warehousing, and data quality management.
• Experience with Git, CI/CD, and workflow orchestration tools.

Good to Have:

• Experience working in the utility, energy, power, water, or gas industry.
• Familiarity with utility data, IoT, time-series data, or operational technology systems.
• Experience integrating data from multiple enterprise systems and APIs.
• Knowledge of streaming technologies such as Kafka or Spark Structured Streaming.
• Experience with Unity Catalogue, data governance, and access management.
• Familiarity with BI and visualisation tools such as Power BI or Tableau.

Skills

Apache Spark, Databricks, PySpark, Python, Spark