Senior Data Engineer

To design and deliver AI-powered Business Intelligence solutions by integrating Databricks Genie, PowerBI Co-Pilot & Semantic Models, and Metric Views on top of a robust end-to-end data engineering stack.

The role focuses on enabling self-service analytics using natural language (GenAI), ensuring business-friendly data consumption, and building governed, scalable, and high-performance data ecosystems across batch and real-time pipelines. Duties and Responsibilities

• Translate business problems into AI-driven BI solutions

• Own end-to-end delivery: ingestion ? transformation ? semantic layer ? AI consumption

• Design scalable data architecture aligned with lakehouse principles

• Ensure data quality, governance, and metric consistency

• Collaborate with business teams to onboard them onto Genie-based analytics

• Optimize performance of queries, pipelines, and AI responses

• Establish best practices for AI in BI (prompting, semantic tuning, governance)

• Drive adoption of self-service BI with minimal dependency on tech teamsKey Decisions / Dimensions

• Define semantic layer design and metric definitions

• Decide GenAI prompting strategies and instruction frameworks

• Prioritize data vs AI optimization trade-offs

• Handle production issues with RCA and long-term fixes

• Drive architectural decisions for lakehouse + AI integrationMajor Challenges

• Ensuring accuracy and trust in AI-generated insights

• Driving adoption of GenAI-based BI over traditional dashboards

• Maintaining semantic consistency across multiple tools

• Balancing performance, cost, and scalability

• Managing dependencies across data engineering, AI, and business teams Required Qualifications and Experience

• Azure Databricks - PySpark, SQL, Delta Lake

• PowerBI - CoPilot, Semantic Model

• Strong experience in Semantic Modeling & Metrics Layer design

• Hands-on with Databricks Genie / GenAI-based BI workflows

• Databricks One (Unified BI Experience)

• Prompt Engineering / Instruction tuning for AI systems

• Python (Pandas, PySpark, FastAPI)

• Azure Data Factory (ADF) for ETL pipelines

• Strong SQL and data modeling skills

Good to Have

• Cosmos DB / MongoDB (NoSQL concepts)

• Azure Data Explorer (KQL)

________________________________________

DATA STACK (MANDATORY FOR SCREENING)

SNo Data Platform / Concepts Associated Technologies

1 Databricks Lakehouse PySpark, SQL, Delta Lake

2 AI for BI Databricks Genie, Genie Rooms, Instructions, Agents, PowerBI CoPilot

3 Semantic Layer Semantic Models, Metric Views

4 ETL & Orchestration Azure Data Factory

5 Programming Python, C#/.NET

6 Cloud Platform Azure (Preferred)

7 DevOps CI/CD Pipelines, Git