Azure Data Engineer

Job Description :

We are looking for an experienced Azure ETL Data Engineer with strong hands-on expertise in designing, developing, and implementing scalable ETL pipelines and data integration solutions on the Microsoft Azure platform.

The candidate should have experience in Azure Data Factory, Azure Databricks, Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics, along with strong programming skills in Python, PySpark, Spark, and SQL.

The role involves migrating data from SQL Server to Azure, developing efficient data pipelines, implementing data transformation processes, and ensuring high-quality data delivery. The candidate should be comfortable working in an Agile environment and managing end-to-end development activities from design through deployment and production support.

Key Responsibilities:

• Design, develop, and implement scalable ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
• Develop and maintain data integration and migration solutions from SQL Server to the Azure cloud platform.
• Build data transformation and processing workflows using Python, PySpark, Spark, and SQL.
• Work with Azure Data Lake Storage (ADLS), Azure SQL Database, and Azure Synapse Analytics for data storage and processing.
• Design and implement data models, including relational and dimensional data models.
• Optimize ETL workflows, SQL queries, and data pipelines to improve performance and efficiency.
• Implement data orchestration, scheduling, monitoring, and error-handling mechanisms within Azure.
• Participate in end-to-end development activities, including requirement analysis, design, development, testing, deployment, and production support.
• Perform peer code reviews and ensure adherence to coding standards, architecture guidelines, and development best practices.
• Collaborate with cross-functional teams, participate in Agile/Scrum ceremonies, and maintain technical documentation.
• Support testing cycles, production deployments, and troubleshooting of data pipeline issues.

Experience & Qualifications

• 5-7 years of overall experience in Data Engineering / ETL development.
• Minimum 3+ years of hands-on experience in Microsoft Azure Data Engineering.
• Strong hands-on experience in:

o Azure Data Factory (ADF)

o Azure Databricks

o Azure Data Lake Storage (ADLS)

o Azure SQL Database

o Azure Synapse Analytics

• Strong programming experience in Python, PySpark, and Spark.
• Strong SQL Server and T-SQL knowledge, including SSIS.
• Experience in designing, developing, and maintaining ETL/ELT pipelines.
• Good understanding of data modelling, relational modelling, and dimensional modelling.
• Experience in handling structured and semi-structured data formats such as JSON, XML, and CSV.
• Knowledge of database systems such as MS SQL Server and Oracle.
• Experience in data migration from SQL Server to Azure.
• Understanding of Azure data orchestration and workflow management.
• Knowledge of source code management and deployment tools such as Git and Jenkins.
• Experience in data file movement and integration through mailbox or other connectivity mechanisms.
• Bachelor’s degree or equivalent technical qualification.

Preferred Skills :

• Microsoft Azure Data Engineer certification (DP-203 or equivalent).
• Exposure to Sqoop, Hadoop, and Shell scripting.
• Knowledge of Java and Scala programming.
• Experience working in Agile/Scrum development environments.
• Familiarity with JIRA, Confluence, and Microsoft Teams.
• Experience with Datadog for monitoring and observability.
• Knowledge of PowerShell scripting.
• Experience in CI/CD pipelines, automated deployments, and release management.
• Good understanding of cloud data architecture and performance optimization.

Location: Bangalore