Data Engineer

• Independently understand business and technical requirements and translate them into scalable data engineering solutions.
• Take end-to-end ownership of assigned projects, modules and data pipelines with minimal supervision.
• Design, develop, test and deploy robust end-to-end data pipelines on the Azure Data Platform.
• Build scalable and reusable data ingestion and transformation frameworks using Azure Data Factory and Azure Databricks.
• Develop data transformation and processing logic using Python, PySpark and SQL.
• Work with structured, semi-structured and unstructured data from databases, APIs, files and other enterprise data sources.
• Design and implement batch and, where required, near-real-time data processing solutions.
• Develop optimized Databricks workloads using Apache Spark and Delta Lake.
• Perform data analysis, profiling and validation to identify data quality, completeness and consistency issues.
• Implement appropriate data quality checks, error handling, logging and monitoring within data pipelines.
• Troubleshoot data pipeline failures, performance issues and data discrepancies independently.
• Optimize SQL queries, Spark jobs and data pipelines for performance and scalability.
• Create reusable components and frameworks to improve development efficiency across projects.
• Participate in solution design and technical discussions and provide recommendations on implementation approaches.
• Implement CI/CD processes for data engineering components using Azure DevOps.
• Deploy and manage ADF, Databricks and related Azure data components across development, test and production environments.
• Prepare appropriate technical design, data mapping and operational documentation.
• Collaborate with solution architects, analysts, BI developers, source-system teams and other engineering teams.
• Support production deployments and troubleshoot post-production issues when required.

Requirements

Strong hands-on experience with:

Azure Data Platform

• Azure Data Factory (ADF) - pipelines, datasets, linked services, triggers, parameterization and reusable frameworks
• Azure Databricks
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Azure Synapse Analytics / Azure SQL
• Azure Key Vault
• Azure Monitor / Log Analytics or equivalent monitoring capabilities
• Azure Event Hubs - exposure/experience is preferred

Databricks & Data Processing

• Strong hands-on experience with Databricks and Apache Spark
• Strong PySpark development skills
• Experience with Delta Lake / Delta tables
• Understanding of Medallion or similar layered data architectures
• Experience implementing incremental and full-load processing patterns
• Ability to troubleshoot and optimize Spark workloads
• Experience with Databricks Workflows/Jobs
• Knowledge of Unity Catalog is preferred

SQL & Python

• Strong SQL development skills
• Ability to write and optimize complex SQL queries
• Experience with joins, CTEs, window functions and analytical queries
• Strong working knowledge of Python
• Ability to develop reusable Python/PySpark modules and utilities

DevOps / CI-CD

• Hands-on experience with Azure DevOps
• Git-based source control
• Branching and code-management practices
• CI/CD implementation for Azure Data Factory and Databricks
• Experience deploying solutions across multiple environments

Qualifications & Experience

• 4-6 years of overall experience in Data Engineering / Data Platform development.
• Preferably 4+ years of hands-on experience with Microsoft Azure Data Platform technologies.
• Strong hands-on experience with Azure Data Factory, Databricks, SQL and Python/PySpark.
• Bachelor's degree in Computer Science, Information Technology, Engineering or a related discipline.
• Experience delivering at least one or more end-to-end Azure data engineering projects.
• Good understanding of data warehousing, data lake and modern data platform concepts.
• Strong analytical and problem-solving skills.
• Ability to work independently and manage multiple technical activities.
• Good written and verbal communication skills.

Preferred Skills

• Microsoft Azure / Databricks certifications.
• Experience with Unity Catalog and Databricks governance.
• Experience with REST API-based data ingestion.
• Knowledge of dimensional modelling and data warehouse concepts.
• Experience working with Power BI or downstream analytics platforms.
• Exposure to streaming/event-driven data processing.
• Experience working in enterprise-scale Azure environments.