Data Architect

Databricks Data Architect

Location : Ahmedabad, Gujarat

Experience : 15 Years

Function : Data Engineering / Architecture - Databricks Lakehouse Platform

Role Summary :

We are looking for an experienced Databricks Data Architect to lead the architecture, design, and implementation of enterprise-scale data platforms using the Databricks Lakehouse Platform.

The ideal candidate will have strong expertise in Databricks, PySpark, Delta Lake, Unity Catalog, Azure Databricks, SQL, and dimensional data modeling. The candidate will act as the technical owner of the data platform, providing architectural direction, establishing engineering standards, mentoring teams, and ensuring scalable and high-performance data solutions.

Key Responsibilities :

- Design and implement enterprise-scale data platforms using Databricks Lakehouse Architecture.

- Define end-to-end data architecture covering ingestion, transformation, storage, governance, and consumption.

- Establish architecture standards, coding guidelines, frameworks, reusable patterns, and best practices.

- Drive technical decisions related to scalability, performance, reliability, security, and maintainability.

- Conduct architecture and design reviews and provide technical recommendations.

- Design and develop scalable ETL/ELT pipelines using PySpark and Databricks.

- Architect Bronze, Silver, and Gold data transformation frameworks.

- Design and manage Delta Lake storage architecture, including partitioning, optimization, and data lifecycle management.

- Implement Databricks Lakeflow pipelines for orchestration and automation.

- Define data quality, validation, reconciliation, monitoring, and governance frameworks.

- Oversee CI/CD and DevOps practices for data engineering projects.

- Design enterprise data models using dimensional modeling, star schemas, fact tables, dimension tables, and conformed dimensions.

- Define business metrics, KPI layers, semantic models, Metric Views, and curated Gold datasets.

- Collaborate with business stakeholders to translate requirements into scalable analytical solutions.

- Provide technical leadership, mentoring, and guidance to data engineers and developers.

- Conduct code reviews and ensure compliance with architecture and engineering standards.

- Troubleshoot complex technical issues and guide teams in solution design.

- Support estimation, solution proposals, technical governance, and architecture roadmaps.

- Drive adoption of modern data engineering and cloud-native best practices.

Must-Have Skills :

Databricks & Lakehouse :

- Databricks Lakehouse Architecture

- Databricks Workflows / Lakeflow

- Delta Lake

- Unity Catalog

- Databricks SQL

- Performance Tuning and Optimization

Data Engineering :

- PySpark

- Spark SQL

- ETL / ELT Design

- Data Pipeline Development

- Data Quality Frameworks

- Batch and Incremental Processing

Data Modeling :

- Dimensional Data Modeling

- Star Schema Design

- Fact and Dimension Modeling

- Slowly Changing Dimensions (SCD)

- Metric View Design

- Analytical Data Modeling

Cloud & Integration :

- Azure Databricks

- ADLS Gen2

- Azure Data Factory (ADF)

- Batch and Event-Based Ingestion Patterns

- Data Governance and Security

Programming & DevOps :

- Python

- SQL

- Git

- CI/CD Frameworks

Nice-to-Have Skills :

- Data warehouse modernization experience.

- Experience migrating legacy EDW platforms to Databricks.

- Exposure to Data Mesh and Medallion Architecture.

- Knowledge of data governance, metadata management, and data cataloging.

- Experience integrating with BI platforms such as Power BI, Tableau, or Looker.

Key Competencies :

- Strong architectural and problem-solving capabilities.

- Excellent communication and stakeholder management skills.

- Ability to mentor and guide data engineering teams.

- Experience driving technical discussions with client architects and business stakeholders.

- Strong focus on scalability, maintainability, performance, and long-term platform sustainability.

Experience & Qualifications :

- 15 years of overall IT experience with substantial experience in enterprise-scale data engineering and architecture.

- Proven experience designing and delivering Databricks/Lakehouse platforms.

- Strong experience owning the Databricks implementation lifecycle from architecture and design through delivery.

- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field is preferred.

Ways of Working :

- Own the complete Databricks implementation lifecycle and define target-state architecture and roadmap.

- Review technical designs and code developed by team members.

- Guide developers on PySpark optimization, Databricks best practices, and data modeling.

- Lead technical discussions with client architects and business stakeholders.

- Ensure scalability, maintainability, performance, security, and sustainability of the data platform.

- Provide architectural governance across multiple workstreams.

- Mentor and develop technical teams while driving engineering excellence.

Skills

Data Architect, Azure Databricks, PySpark, SQL, Data Modeling, Data Ingestion, Data Governance, ETL, Python