Senior / Principal Azure Data Engineer

About the Role

A leading organization is seeking experienced Azure Data Engineers to design, develop, and maintain scalable data solutions within a modern cloud-based environment. This role focuses on building reliable and automated data pipelines that support enterprise data architecture and analytics initiatives. The ideal candidate will have strong expertise in Microsoft Azure technologies, data engineering best practices, and software engineering principles. You will work closely with cross-functional teams to deliver high-quality, resilient, and scalable data solutions.

Key Responsibilities

• Design, develop, and support automated data pipelines and ETL workflows using Microsoft Azure technologies such as Azure Data Factory, Azure Synapse, Azure Databricks, and Azure Fabric.
• Manage and optimize cloud-based data storage solutions, including Azure Data Lake Storage Gen2 and Azure Blob Storage.
• Partner with data architects, analysts, data scientists, and business stakeholders to understand requirements and deliver effective data solutions.
• Improve pipeline performance, scalability, reliability, and cost efficiency within the Azure ecosystem.
• Implement data quality checks, validation processes, and governance standards to ensure data integrity.
• Create and maintain technical documentation for data workflows, configurations, and operational procedures.
• Monitor, troubleshoot, and resolve data pipeline and platform-related issues in a timely manner.
• Contribute to CI/CD processes for deploying, testing, and maintaining data engineering solutions.
• Stay informed on emerging technologies and industry best practices related to cloud data engineering.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related discipline, or equivalent practical experience.
• Proven experience in Data Engineering or similar roles focused on ETL and large-scale data processing.
• Strong hands-on experience with Microsoft Azure services, including:
• Azure Data Factory
• Azure Synapse
• Azure Databricks
• Azure Blob Storage
• Azure Data Lake Storage Gen2
• Solid experience writing and optimizing SQL queries for modern relational databases such as SQL Server or PostgreSQL.
• Strong understanding of software engineering concepts, including version control, testing, and CI/CD practices.
• Experience working with big data technologies such as Apache Spark.
• Strong analytical, troubleshooting, and problem-solving capabilities.
• Effective communication and collaboration skills.

Preferred Qualifications

• Experience providing technical leadership and working directly with business stakeholders to understand requirements.
• Strong learning agility and adaptability in fast-paced environments.
• Proficiency in Python is highly preferred; experience with Scala, Java, C#, or similar programming languages is also acceptable.
• Experience developing Spark-based applications using PySpark.
• Familiarity with modern data file formats such as Parquet, Delta, and Avro.
• Experience integrating and querying API-based data sources efficiently.
• Understanding of Azure platform administration concepts, including subscriptions and resource groups.
• Familiarity with Git-based development workflows.
• Experience using Azure DevOps pipelines and repositories for deployment and maintenance activities.
• Exposure to Ansible and its integration with Azure DevOps pipelines is an advantage.