Data Engineer

To effectively design, develop, and manage data solutions using ETL technologies such as Azure Databricks (ADB) , Azure Data Factory (ADF) and SQLDuties and Responsibilities

KEY RESPONSIBILITIES

• Translate business requirements into technical solutions in collaboration with the PMO team.

• Own end-to-end delivery of data projects, ensuring on-time execution and adherence to quality standards.

• Design technical architecture and guide development efforts for enhancements and new projects.

• Develop and maintain robust ETL pipelines and data integration modules across systems.

• Ensure high data quality, data anomaly resolution of critical process issues.

• Monitor and resolve performance bottlenecks in data workflows and programs.

• Establish best practices, standard operating procedures, and drive their implementation across teams.

• Act as a liaison with business users and product managers to support daily data needs and strategic initiatives.

• Coordinate with internal and external development teams to troubleshoot and resolve issues efficiently.

• Manage workload through effective planning, prioritization, and progress tracking.Key Decisions / Dimensions

KEY DECISIONS / DIMENSIONS

• Define semantic layer design and metric definitions

• Prioritize data vs AI optimization trade-offs

• Handle production issues with RCA and long-term fixes

• Drive architectural decisions for lakehouse + Data integrationMajor Challenges

MAJOR CHALLENGES

• Ensuring Data Delivery within TAT

• Driving adoption of GenAI-based BI over traditional dashboards

• Balancing performance, cost, and scalability

• Managing dependencies across data engineering, AI, and business teams Required Qualifications and Experience

REQUIRED SKILLS & EXPERIENCE

Must Have

• Azure Databricks - PySpark, SQL, Delta Lake

• Strong experience in Semantic Modeling & Metrics Layer design

• Hands-on with Databricks workflows

• Pyspark (Pandas, PySpark, FastAPI)

• Azure Data Factory (ADF) for ETL pipelines

• Strong SQL and data modeling skills

Good to Have

• Cosmos DB / MongoDB (NoSQL concepts)

• Azure Data Explorer (KQL)

DATA STACK (MANDATORY FOR SCREENING)

SNo Data Platform / Concepts Associated Technologies

1 Databricks Lakehouse PySpark, SQL, Delta Lake

2 AI for BI Databricks Genie, Genie Rooms, Instructions, Agents

4 ETL & Orchestration Azure Data Factory

5 Programming Pyspark

6 Cloud Platform Azure (Preferred)

10 DevOps CI/CD Pipelines, Git