Azure Data Engineer

We are looking for an experienced Azure Data Engineer with 7+ years of hands-on experience in designing, developing, and maintaining scalable data solutions on Microsoft Azure. The ideal candidate should have strong expertise in Azure Data Factory, Azure Databricks, Azure Data Lake, SQL, PySpark, and ETL/ELT pipelines, along with experience in data warehousing and cloud-based data architecture. The candidate should be capable of building robust data pipelines, optimising data workflows, and supporting business intelligence and analytics initiatives.

Responsibilities:

• Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks.
• Implement ETL/ELT processes for structured and unstructured data from multiple sources.
• Work extensively with Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL Database.
• Develop and optimise PySpark and Spark SQL workflows in Databricks.
• Perform data transformation, cleansing, validation, and enrichment processes.
• Build and maintain data models for reporting, analytics, and business intelligence solutions.
• Collaborate with business stakeholders, data analysts, and architects to understand data requirements.
• Ensure data quality, governance, security, and compliance standards are maintained.
• Monitor and troubleshoot data workflows and optimise performance for large-scale data processing.
• Support migration of on-premises data platforms to the Azure Cloud environment.
• Create technical documentation and maintain best practices for data engineering processes.

Requirements:

• Strong hands-on experience in Microsoft Azure Cloud Platform.
• Expertise in Azure Databricks with strong PySpark experience.
• Experience with Azure Data Factory (ADF).
• Good knowledge of Azure Data Lake Storage (ADLS Gen2).
• Experience in Azure Synapse Analytics.
• Strong SQL skills with performance tuning experience.
• Experience with ETL/ELT pipeline development.
• Understanding of Data Warehousing concepts.
• Experience with Delta Lake, Spark optimisation, and partitioning strategies.
• Exposure to CI/CD pipelines, DevOps, and deployment automation.
• Knowledge of Power BI is an added advantage.
• Familiarity with Git, version control, and Agile methodology.
• Strong analytical and problem-solving skills.
• Good communication and stakeholder management.
• Ability to work independently and in team environments.
• Strong debugging and troubleshooting capability.
• Bachelor's degree in Computer Science, Engineering, IT, or related field.
• Microsoft Azure certifications are preferred.
• Experience working in enterprise-scale data environments.
• Exposure to big data technologies is an added advantage.

Nice to Have:

• Experience in Snowflake, Kafka, or Airflow.
• Exposure to Machine Learning data pipelines.
• Knowledge of real-time streaming solutions using Azure Event Hub or Stream Analytics.

Skills

ADF, ADLS, Azure, Databricks, Delta, ELT, ETL, PySpark, SQL, Spark, Synapse