Senior Data Engineer

We are seeking an experienced Senior Data Engineer to join our dynamic team and drive our data engineering initiatives. The ideal candidate will have extensive experience with Azure cloud technologies and a proven track record of designing, implementing, and maintaining scalable data solutions. You will play a crucial role in building robust data pipelines, optimising data workflows, and enabling data-driven decision-making across the organisation.

Responsibilities:

• Design, develop, and maintain complex data pipelines using Azure Data Factory (ADF) and other Azure data services.
• Build and optimise data processing workflows using Azure Databricks with Spark and PySpark.
• Implement and manage Lake flow Declarative Pipelines for efficient data processing and transformation.
• Develop and optimise SQL queries and stored procedures for data extraction, transformation, and loading (ETL) processes.
• Create and maintain data models, schemas, and documentation to ensure data quality and consistency.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions.
• Monitor and troubleshoot data pipeline performance, implementing improvements and optimisations as needed.
• Ensure data security, privacy, and compliance with industry standards and regulations.
• Mentor junior team members and provide technical guidance on data engineering best practices.
• Stay current with emerging technologies and industry trends in data engineering and cloud computing.
• Participate in code reviews, testing, and deployment processes to maintain high-quality deliverables.
• Design and implement data governance frameworks and data quality monitoring solutions.

Requirements:

• 8+ years of total professional experience in software development, data engineering, or related fields.
• 7+ years of hands-on experience in data engineering roles with proven expertise in designing and implementing data solutions.
• Extensive experience with Azure Data Factory (ADF) for building and orchestrating data pipelines.
• Strong proficiency in Azure Databricks and Apache Spark for large-scale data processing.
• Hands-on experience with Lake flow Declarative Pipelines and modern data pipeline architectures.
• Advanced SQL skills with experience in query optimisation and database design.
• Proficient in PySpark and Python for data processing and transformation tasks.
• Experience with Azure cloud services, including Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics.
• Strong understanding of data warehousing concepts, dimensional modelling, and ETL/ELT processes.
• Knowledge of data governance, data quality, and data security best practices.
• Experience with version control systems (Git) and CI/CD pipelines.
• Excellent problem-solving skills and attention to detail.
• Strong communication skills and ability to work effectively in cross-functional teams.

Preferred Qualifications:

• Bachelor's degree in Computer Science, Engineering, Mathematics, or a related technical field.
• Azure certifications such as Azure Data Engineer Associate (DP-203) or Azure Solutions Architect Expert.
• Experience with additional Azure services like Azure Stream Analytics, Azure Event Hubs, or Azure Cosmos DB.
• Knowledge of machine learning pipelines and MLOps practices.
• Experience with Infrastructure as Code (IaC) tools like ARM templates or Terraform.
• Familiarity with containerization technologies (Docker, Kubernetes).
• Experience with real-time data processing and streaming technologies.
• Knowledge of other programming languages, such as Scala or Java.
• Experience with data visualisation tools and business intelligence platforms.
• Previous experience in mentoring and leading technical teams.

Technical Skills:

• Cloud Platforms: Microsoft Azure (primary focus).
• Data Processing: Azure Databricks, Apache Spark, PySpark.
• Data Orchestration: Azure Data Factory (ADF), Lake flow Declarative Pipelines.
• Databases: SQL Server, Azure SQL Database, Azure Synapse Analytics, Azure Data Lake.
• Programming Languages: Python, SQL, PySpark.
• Development Tools: Git, Azure DevOps, Visual Studio Code.
• Data Formats: JSON, Parquet, Avro, CSV, XML.
• Monitoring and Logging: Azure Monitor, Application Insights.

Skills

ADF, Azure, Data Lake, Databricks, ELT, ETL, PySpark, Python, SQL, Spark, Synapse, warehousing