Azure Data Engineer

About the Role :

We are looking for an experienced Azure Data Engineer with 7 - 9 years of hands-on experience in designing, developing, and maintaining scalable data engineering solutions on Microsoft Azure.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines and data engineering solutions using Microsoft Azure.

- Develop and manage ETL/ELT workflows for structured, semi-structured, and unstructured data.

- Build data ingestion pipelines using Azure Data Factory and other Azure data integration services.

- Develop data transformation and processing solutions using Azure Databricks and Apache Spark.

- Write optimized SQL queries, stored procedures, views, and other database objects.

- Integrate data from various sources including relational databases, APIs, files, cloud platforms, and enterprise applications.

- Design and maintain data lakes and data warehouse solutions on Azure.

- Implement batch and near-real-time data processing pipelines as required.

- Develop reusable frameworks and components for data ingestion, transformation, validation, and monitoring.

- Implement data quality checks, validation rules, reconciliation processes, and error handling mechanisms.

- Optimize data pipelines, Spark jobs, SQL queries, and storage for improved performance and cost efficiency.

- Monitor pipeline execution, troubleshoot failures, and resolve production data issues.

- Implement logging, alerting, monitoring, and operational support mechanisms.

- Work closely with data architects, analysts, application teams, and business stakeholders to understand data requirements.

- Participate in data migration, modernization, and cloud transformation initiatives.

- Implement security controls, access management, encryption, and governance requirements for Azure data platforms.

- Maintain technical documentation for data pipelines, architecture, processes, and operational procedures.

- Follow best practices for coding, version control, testing, deployment, and production support.

Technical Skills :

- 7 - 9 years of experience in data engineering, ETL development, or related data platform roles.

- Strong hands-on experience with Microsoft Azure data services.

- Strong experience with Azure Data Factory (ADF) for data integration and pipeline development.

- Hands-on experience with Azure Databricks and Apache Spark.

- Strong proficiency in SQL and experience working with relational databases.

- Good understanding of Python and/or PySpark for data processing and automation.

- Experience designing and implementing Azure Data Lake-based solutions.

- Knowledge of Azure Synapse Analytics or similar cloud data warehouse technologies.

- Experience working with different data formats such as CSV, JSON, Parquet, and Avro.

- Experience with REST APIs and data ingestion from external systems is preferred.

- Understanding of batch and streaming data processing concepts.

- Experience with Git and version control systems.

- Knowledge of CI/CD practices for data engineering deployments.

- Experience with Azure DevOps or similar deployment and project management tools.

- Understanding of data security, governance, access control, and compliance concepts.

- Knowledge of monitoring and troubleshooting Azure data pipelines.

Preferred Skills :

- Experience with Azure Synapse Analytics.

- Experience with Azure SQL Database.

- Knowledge of Event Hubs, Stream Analytics, or other Azure streaming technologies.

- Experience with Delta Lake and lakehouse architecture.

- Knowledge of data modelling and dimensional modelling concepts.

- Experience with infrastructure automation or Infrastructure as Code.

- Exposure to Terraform or ARM/Bicep templates.

- Experience implementing automated testing for data pipelines.

- Knowledge of performance tuning and cloud cost optimization.

- Exposure to enterprise-scale data migration and modernization projects.

Education :

- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

- Relevant Azure certifications such as Azure Data Engineer Associate will be an advantage.

Key Competencies :

- Strong analytical and problem-solving skills.

- Ability to design scalable and maintainable data solutions.

- Strong understanding of data engineering best practices.

- Good communication and collaboration skills.

- Ability to work independently as well as collaborate with cross-functional teams.

- Strong focus on data quality, reliability, security, and performance.

Skills

Azure Data Factory, Azure Databricks, Apache Spark, SQL, Python, Delta Lake, Data Engineering, ETL, Synapse Analytics, Data Quality