Associate Principal - Data Engineering

Experience

12 Years

Role Overview

We are seeking an experienced Databricks Data Architect to lead the design architecture and implementation of enterprisescale data platforms on Databricks The ideal candidate should possess strong expertise in modern data engineering practices dimensional data modeling and Databricks Lakehouse architecture

The candidate will act as the technical owner of the data platform providing architectural guidance establishing best practices mentoring development teams and ensuring successful delivery of scalable highperformance data solutions

Key Responsibilities

Architecture Solution Design

Design and implement enterprisescale data platforms using the Databricks Lakehouse architecture

Define endtoend data architecture including ingestion transformation storage governance and consumption layers

Establish architectural standards framework guidelines coding best practices and reusable design patterns

Lead technical decisions related to scalability performance optimization reliability and security

Conduct architecture reviews and provide recommendations to project teams

Data Engineering Databricks Implementation

Design and build scalable ETLELT pipelines using PySpark on Databricks

Architect and implement Bronze Silver Gold data transformation frameworks

Design and manage Delta Lake based storage architecture including partitioning optimization and data lifecycle management

Implement Databricks Lakeflow pipelines for data orchestration and automation

Define data quality validation reconciliation and monitoring frameworks

Oversee CICD and DevOps practices for data engineering projects

Data Modeling Analytics Enablement

Design and implement enterprise data models using Dimensional Modeling techniques

Develop Star Schemas Fact Tables Dimension Tables and conformed dimensions

Define business metrics KPI layers and semantic models

Architect and implement Metric Views and curated Gold datasets for reporting and analytics

Work closely with business stakeholders to translate requirements into scalable analytical models

Technical Leadership

Provide technical leadership and mentorship to data engineers and developers

Perform code reviews and ensure adherence to architecture standards

Troubleshoot complex technical issues and guide teams in solution design

Support estimation solution proposals and technical governance activities

Drive adoption of modern data engineering and cloudnative best practices

Required Technical Skills

Core Databricks Skills

Databricks Lakehouse Architecture

Databricks Workflows Lakeflow

Delta Lake

Unity Catalog

Databricks SQL

Performance Tuning Optimization

Data Engineering

PySpark

Spark SQL

ETL ELT Design

Data Pipeline Development

Data Quality Frameworks

Batch and Incremental Processing

Data Modeling

Dimensional Data Modeling

Star Schema Design

Fact Dimension Modeling

Slowly Changing Dimensions SCD

Metric View Design

Analytical Data Modeling

Cloud Integration

Azure Databricks preferred

ADLS Gen2

Azure Data Factory ADF

Eventbased and Batch Ingestion Patterns

Data Governance and Security

Programming

Python

SQL

Git

CICD Frameworks

Desired Skills

Data Warehouse Modernization experience

Experience migrating legacy EDW solutions to Databricks

Exposure to Data Mesh and Medallion Architecture

Knowledge of Data Governance Metadata Management and Data Cataloging

Experience supporting BI platforms such as Power BI Tableau or Looker

Leadership Expectations

The candidate should be capable of

Owning the complete Databricks implementation lifecycle

Defining targetstate architecture and roadmap

Reviewing designs and code produced by team members

Guiding developers on PySpark optimization and data modeling best practices

Driving technical discussions with client architects and business stakeholders

Ensuring scalability maintainability and longterm sustainability of the platform

Providing architectural governance across multiple workstreams