Senior Data Engineer

Job Summary :

We are looking for a Senior Data Engineer with 6+ years of experience in Databricks, real-time data processing, Lakehouse architecture, and AI-driven solutions. The role involves designing scalable batch and streaming data platforms and delivering high-quality, governed data and AI solutions.

Key Responsibilities :

- Design, develop, and maintain batch and streaming data pipelines using Databricks and PySpark.

- Build, optimize, and manage Delta Lake tables within Unity Catalog.

- Develop and support Lakeflow pipelines with incremental processing, CDC, and data quality validation.

- Integrate SharePoint and other external enterprise data sources into the Databricks Lakehouse.

- Ensure data quality, governance, security, reliability, and operational excellence.

- Monitor, troubleshoot, and optimize data workflows for performance and reliability.

- Collaborate with business and technical stakeholders to deliver scalable solutions.

- Build AI-powered data solutions, copilots, and intelligent agents using LLMs, RAG, and vector search.

- Implement CI/CD and automated deployment using Databricks Repos and Asset Bundles.

- Drive deliverables independently and proactively resolve technical challenges.

Mandatory Skills :

- 6+ years of Data Engineering experience.

- Strong hands-on experience with Databricks and modern Lakehouse architecture.

- Strong proficiency in PySpark, Spark Structured Streaming, and SQL.

- Experience with Delta Lake, Unity Catalog, Databricks Workflows, and job orchestration.

- Experience with Bronze, Silver, and Gold Lakehouse architecture.

- Practical experience with Databricks Lakeflow, Declarative Pipelines, Lakeflow Connect, CDC, data quality, and incremental processing.

- Experience integrating enterprise data platforms with AI-driven applications.

- Experience with LLMs, RAG, vector search, and prompt engineering.

- Strong understanding of automated deployment and release processes.

- Experience with Git-based source control and CI/CD.

Data Governance & Security :

- Unity Catalog governance and security.

- Role-Based Access Control (RBAC).

- Data masking and data protection policies.

- Audit logging, data lineage, and compliance.

DevOps & Deployment :

- Git-based source control and CI/CD.

- Databricks Repos.

- Databricks Asset Bundles (DABs).

- Automated deployment and release processes.

Preferred Skills :

- Experience with large-scale enterprise data platforms.

- Strong understanding of Lakehouse and Data Mesh principles.

- Exposure to Azure cloud services and enterprise integration patterns.

- Experience with NoSQL databases.

- Strong problem-solving, communication, and stakeholder management skills.

Skills

Data Engineering, Databricks, Apache Spark, Python, SQL, ETL, DataLake, Data Governance, Data Quality, Data Validation, Azure Databricks