Associate Delivery Manager - Data Engineer

To effectively design, develop, and manage data solutions using ETL technologies such as Azure Databricks (ADB) , Azure Data Factory (ADF) and SQL along with leading 3 to 5 members of developersDuties and Responsibilities

KEY RESPONSIBILITIES

• Lead the end-to-end design, development, and delivery of scalable data engineering solutions using Azure Databricks, ADF, PySpark, SQL, and Delta Lake.

• Convert business requirements into robust technical designs, architecture documents, data models, and implementation plans.

• Own technical delivery of data integration, ETL, lakehouse, semantic layer, and AI/BI enablement initiatives.

• Guide and mentor data engineers on coding standards, design best practices, performance optimization, and reusable framework development.

• Review technical designs, code, pipelines, and deployment plans to ensure quality, scalability, maintainability, and compliance.

• Drive architecture decisions for batch and near-real-time data pipelines across Bronze, Silver, and Gold layers.

• Ensure data quality, reconciliation, anomaly detection, and timely resolution of production issues through effective RCA and permanent fixes.

• Optimize data pipelines, Databricks jobs, SQL queries, and storage usage to improve performance and reduce cost.

• Implement CI/CD practices, version control, automated deployments, and environment management across Dev, QA, and Production.

• Collaborate with PMO, business stakeholders, BI teams, InfoSec, DevOps, and external partners for smooth project execution.

• Establish SOPs, engineering standards, reusable components, monitoring frameworks, and documentation practices.

• Track delivery progress, manage technical dependencies, prioritize work, and ensure timely closure of project milestones.

• Support adoption of modern data platforms, semantic modeling, metrics layer design, and GenAI/BI capabilities.

• Ensure compliance with data governance, security, access control, audit, and enterprise data management standards.

• Act as the technical escalation point for critical issues, complex solutioning, and cross-team dependency resolution.Key Decisions / Dimensions

KEY DECISIONS / DIMENSIONS

• Define semantic layer design and metric definitions

• Prioritize data vs AI optimization trade-offs

• Handle production issues with RCA and long-term fixes

• Drive architectural decisions for lakehouse + Data integrationMajor Challenges

MAJOR CHALLENGES

• Ensuring Data Delivery within TAT

• Driving adoption of GenAI-based BI over traditional dashboards

• Balancing performance, cost, and scalability

• Managing dependencies across data engineering, AI, and business teams Required Qualifications and Experience

REQUIRED SKILLS & EXPERIENCE

Must Have

• Azure Databricks - PySpark, SQL, Delta Lake

• Strong experience in Semantic Modeling & Metrics Layer design

• Hands-on with Databricks workflows

• Pyspark (Pandas, PySpark, FastAPI)

• Azure Data Factory (ADF) for ETL pipelines

• Strong SQL and data modeling skills

Good to Have

• Cosmos DB / MongoDB (NoSQL concepts)

• Azure Data Explorer (KQL)

DATA STACK (MANDATORY FOR SCREENING)

SNo Data Platform / Concepts Associated Technologies

1 Databricks Lakehouse PySpark, SQL, Delta Lake

2 AI for BI Databricks Genie, Genie Rooms, Instructions, Agents

4 ETL & Orchestration Azure Data Factory

5 Programming Pyspark

6 Cloud Platform Azure (Preferred)

10 DevOps CI/CD Pipelines, Git