Senior Data Engineer

Job Summary :

We are seeking a highly skilled Senior Data Engineer to design, build, and optimize our enterprise data platform and analytics solutions. In this role, you will serve as a key hands-on engineering track lead responsible for transforming complex business requirements into scalable, production-grade data pipelines and Lakehouse architectures.

Key Responsibilities :

- Data Pipeline & Lakehouse Engineering : Design, implement, and maintain high-performance batch and real-time streaming data pipelines using Python, PySpark, and Spark SQL on Azure Databricks and Microsoft Fabric. Build multi-tier Medallion architectures (Bronze, Silver, Gold) using Delta Lake principles.

- Microsoft Fabric Platform Delivery : Leverage Fabric capabilities (OneLake, Direct Lake, Lakehouses, and Data Factory Gen2) to streamline zero-copy ingestion, optimize semantic models, and eliminate redundant ETL pipelines.

- API & Systems Integration : Build resilient API connectors, CDC workflows, and data integration pipelines to ingest data from heterogeneous sources.

- AI & Analytics Enabling : Structure, clean, and optimize lakehouse data layers for downstream Power BI reporting as well as AI/ML applications, ensuring data is clean, validated, and structured for AI grounding.

- Data Quality & Governance : Implement robust data validation, access controls, data lineage, and metadata management using Unity Catalog, Fabric Governance, and automated testing frameworks.

- Technical Mentorship & CI/CD : Perform thorough code reviews, enforce engineering best practices, optimize query performance, and manage automated deployment pipelines.

Technical Skills :

- Core Development : Advanced hands-on proficiency in Python, PySpark, and SQL (Spark SQL / T-SQL).

- Databricks Platform : Strong hands-on experience with Azure Databricks (Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader) and Genie.

- Microsoft Fabric : Demonstrated experience building pipelines and workspace items within Microsoft Fabric.

- Cloud Infrastructure : Solid expertise across Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory (ADF), Azure Key Vault, and Azure SQL/Synapse.

- AI Data Concepts : Strong conceptual and practical understanding of data engineering requirements for AI/ML workloads including vector embeddings, RAG data ingestion, and feature stores.

Education :

- Bachelor's or Master's degree in Computer Science, Information Technology, or a related quantitative field.

Skills

Data Engineering, Azure, Azure Databricks, Python, PySpark, SQL, Microsoft Fabric, Data Warehousing, ETL