Data Engineer – Azure Databricks & Lakehouse Architecture

SUMMARY:
-

POSITION INFO:
PBT Group is seeking an experienced Data Engineer to join a project focused on implementing a modern Lakehouse architecture using the Medallion framework.   The successful candidate will be responsible for developing and maintaining data ingestion pipelines, landing data in the Bronze layer, and supporting data transformation and semantic modelling through the Silver and Gold layers. The objective is to deliver governed, reusable, production-ready and AI-ready data products that support advanced analytics, Power BI reporting and data science use cases.   Working closely with an in-house Data Scientist and other technical stakeholders, the Data Engineer will contribute to the development of reliable data products and business views, with a strong emphasis on production deployment, data governance, performance optimisation and operational excellence.   Key Responsibilities Data Engineering and Lakehouse Architecture Design, develop and maintain scalable data pipelines using Azure Databricks and Azure Data Factory (ADF). Implement and support Medallion architecture, including Bronze, Silver and Gold data layers. Ingest data from multiple sources and ensure reliable landing, transformation and processing. Develop and maintain data integration and ingestion patterns that support batch and real-time data processing. Contribute to the design and implementation of robust, scalable and maintainable Lakehouse solutions. Data Transformation and Modelling Develop data pipelines and transformation logic using Python, PySpark and SQL. Support semantic data modelling across Silver and Gold layers to deliver high-quality, reusable and governed data products. Develop data models and business views that support Power BI reporting, analytics and data science requirements. Ensure datasets are accurate, consistent, documented and suitable for downstream analytical and AI use cases. Apply data warehousing principles and appropriate modelling techniques to support business requirements. Azure Data Services and Integration Work with Azure data services and technologies to develop, integrate and maintain data solutions. Implement data ingestion and integration pipelines using Azure Data Factory, Azure Databricks and other relevant Azure services. Apply API integration and data ingestion patterns to connect systems and source platforms. Where required by the solution, work with Azure Functions, Apache Kafka and Apache Spark to support integration, processing and streaming requirements. Data Governance and Quality Apply data governance, access control and catalogue management practices using Databricks Unity Catalog and relevant gateway capabilities. Ensure that data products comply with established governance, security, quality and access requirements. Support data lineage, documentation, discoverability and reusability across the data environment. Ensure that data products are well governed and structured to support analytics and AI-driven use cases. Production Deployment and DevOps Develop, deploy and maintain production-ready data pipelines and data products. Apply Git version control, release management and established DevOps practices. Contribute to CI\/CD pipeline development, automation and controlled deployment processes. Monitor and troubleshoot pipeline performance, data quality issues and production incidents. Perform performance tuning and optimisation to improve pipeline efficiency, reliability and scalability. Maintain clear technical documentation covering architecture, design decisions, pipelines, transformations and data models. Collaboration and Continuous Improvement Collaborate with Data Scientists and other stakeholders to translate analytical and business requirements into production-ready data products. Develop datasets and business views that support Power BI reporting and AI use cases. Work within an Agile delivery framework and contribute to planning, reviews and continuous improvement. Support the use of Databricks Genie where relevant to the solution. Promote engineering best practices, reusable components and operational excellence throughout the data delivery lifecycle.   Essential Skills and Experience Candidates should demonstrate hands-on experience in the following areas: Lakehouse and Data Engineering Advanced experience with Azure Databricks. Practical experience implementing Medallion or Lakehouse architecture, including Bronze, Silver and Gold layers. Strong proficiency in Python, PySpark and SQL for data engineering and transformation. Hands-on experience developing and maintaining Azure Data Factory pipelines. Experience with data integration, pipeline development, data ingestion and transformation. Understanding of data modelling, semantic modelling and data warehousing principles. Azure Data Services and Integration Experience working with Azure data services. Knowledge of API integration and data ingestion patterns. Understanding of batch processing and real-time or streaming data analytics. Knowledge of relevant technologies such as Apache Spark, Azure Functions and Apache Kafka, where applicable to the solution. Governance, DevOps and Production Delivery Familiarity with Databricks Unity Catalog and relevant gateway capabilities. Solid understanding of Git, version control and release management. Experience with DevOps practices and CI\/CD pipelines. Ability to deploy and support production data pipelines and data products. Understanding of performance tuning, troubleshooting and pipeline optimisation. Ability to produce clear and comprehensive technical documentation. Experience working within Agile delivery environments. Preferred Qualifications and Certifications Databricks Data Engineer certification is preferred. Databricks AI certification would be advantageous. Advantageous Experience Exposure to Databricks Genie. Experience delivering data products for Power BI, business analytics and AI use cases. Experience collaborating with Data Scientists to develop analytics-ready datasets. Experience in Marketing and Sales data domains or similar commercial environments. Exposure to advanced data integration, streaming technologies and cloud-based data platforms.   Ideal Candidate Profile The ideal candidate is a hands-on Data Engineer with strong Azure Databricks, Python, PySpark and SQL skills and practical experience implementing Lakehouse architecture using the Medallion framework.   They should be comfortable building ingestion pipelines, supporting semantic data modelling, implementing data governance and deploying reliable data products into production. The ability to work closely with Data Scientists and analytics teams is important, as is an understanding of how to structure data for Power BI reporting and AI applications. Candidates should combine strong technical problem-solving skills with sound engineering practices, attention to data quality, clear documentation and a focus on performance, scalability and operational reliability.   This opportunity is best suited to Data Engineers who have practical Lakehouse implementation experience and can contribute across the data lifecycle, from ingestion and transformation through to governed, analytics-ready Gold-layer data products.