Data Engineer

Title: Senior Technical Specialist - PBI Databricks

Location: Remote

Duration: 12+ Months

Education: Any Degree

Exp Level: 8+ Years

Job Responsibilities:

• Design and build conformed dimensions and transactional fact tables for enterprise reporting.

• Develop and maintain data pipelines on Azure Databricks using PySpark, Spark SQL, and Delta Lake.

• Implement best practices in dimensional modeling, including:

• Star schemas

• Conformed dimensions

• Surrogate and deterministic hash keys (e.g., SHA-256)

• Transactional fact design

• Translate existing SQL Server and stored procedure logic into efficient Spark-based transformations.

• Implement incremental data processing and change data capture (CDC) pipelines.

• Ensure data reconciliation with source systems to meet accuracy and audit standards.

• Collaborate closely with a senior Databricks lead to deliver core data assets within defined timelines.

Required Skills:

• 8+ years of experience in data engineering, with strong focus on Azure Databricks.

• Proficiency in:

• PySpark

• Spark SQL

• Delta Lake

• Deep expertise in dimensional modelling and data warehousing concepts.

• Experience with:

• Building scalable production pipelines

• Incremental data loads and CDC patterns

• Performance optimization in distributed systems

• Strong understanding of relational databases and ability to interpret complex SQL/stored procedures

Good to have

Experience with:

• Lakeflow / Spark Declarative Pipelines or dbt

• Unity Catalog

• Exposure to financial/General Ledger data models

• Familiarity with JD Edwards systems, including:

• F0902 (balances)

• F0911 (journal details)