Lead data Engineer

Role: Lead Data Engineer

Mode of work : Hybrid – Mandatory

Location: Bangalore

Timing: 3PM to 12AM

Data Engineer Lead

Skill Set:

10+ years in Data Engineering.

5+ years of hands-on experience in Azure Databricks/PySpark.

Experience with Microsoft Fabric.

Strong SQL optimization expertise.

Experience leading teams, architecture discussions, and enterprise data transformation programs.

Exposure to DevOps, governance, security, and cloud cost optimization.

Hands-on coding is mandatory for this role. The Senior Data Engineering Lead must actively contribute to development, write and optimize production-grade code, troubleshoot complex technical issues, and lead by example rather than focusing solely on team management or architecture.

1. Data Engineering Leadership: Lead the design, development, and delivery of enterprise-scale data engineering solutions, mentoring data engineers and driving technical excellence.
2. Advanced Databricks Development: Strong hands-on expertise in Azure Databricks, PySpark, Spark SQL, Delta Lake, and Medallion Architecture.
3. ETL/ELT Design and Implementation: Design, develop, and optimize scalable ETL/ELT pipelines, incremental data ingestion, CDC, and batch/streaming data processing.
4. SQL Development and Optimization: Expert-level SQL skills, including complex queries, stored procedures, execution plan analysis, indexing, partitioning, and query performance tuning.
5. Azure Data Platform Expertise: Strong experience with Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, and related Azure data services.
6. Microsoft Fabric: Good hands-on experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, Data Factory, and Spark-based data engineering capabilities.
7. Data Modeling and Architecture: Expertise in data warehousing, dimensional modeling, star/snowflake schemas, lakehouse architecture, and data quality management.
8. Performance and Cost Optimization: Identify and resolve data processing bottlenecks, optimize Spark workloads, and improve pipeline performance, reliability, and cloud cost efficiency.
9. Technical Design and Team Mentoring: Lead technical design discussions, architecture reviews, code reviews, engineering best practices, and knowledge-sharing initiatives.
10. Enterprise Integration and DevOps: Experience delivering enterprise-scale data integration and analytics solutions, implementing CI/CD, automated testing, monitoring, governance, security, and production support processes.