Senior Data Engineer – Databricks & Data Modeling/Architecture

Key Responsibilities

• Design, develop, test, and deploy scalable pipelines using Databricks, PySpark, Spark SQL, and Delta Lake, spanning Bronze/Silver/Gold (medallion) layers for HR data integration and analytics.
• Build data models - logical, physical, dimensional, and integration - and maintain source-to-target mappings for workforce, organization, employee, position, job, and related HR domains.
• Migrate legacy GDC integrations by re-engineering Informatica, SQL, and stored-procedure logic into modern Databricks patterns, including inbound Oracle HCM feeds and outbound HR data files.
• Validate output parity through reconciliation, parallel runs, and source-to-target checks, and build automated data quality, exception handling, and audit controls into pipelines.
• Apply enterprise governance, security, privacy, and role-based access expectations to sensitive HR data, capturing lineage and metadata where applicable.
• Produce production-ready documentation - technical designs, runbooks, data dictionaries, and support handoffs - for maintainable, observable, cost-aware solutions.
• Collaborate within a distributed delivery team, participate in agile ceremonies, mentor junior engineers, and communicate progress, blockers, and risks clearly.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field, with 7+ years of data engineering, ETL, or data platform experience.
• Azure Databricks & Apache Spark: 3+ years hands-on building Delta Lake pipelines with PySpark, Spark SQL, and SQL across a medallion architecture.
• Data modeling & integration: Strong logical/physical/dimensional modeling plus source-to-target mapping, reconciliation, and production cutover support.
• AI-assisted engineering: Proficiency using AI/Copilot tooling to accelerate development, code conversion, and documentation.
• Practical, delivery-focused work style, comfortable turning ambiguous legacy logic into clean target-state designs within a highly governed environment.
• Strong written and verbal communication, with the ability to overlap U.S. Eastern Time for key meetings, design reviews, and issue resolution.

Preferred Qualifications

• Suggested 3+ years modernizing legacy ETL (Informatica-to-Databricks) or building metadata-driven, configuration-driven data frameworks.
• Azure ecosystem: Data Lake Storage, Data Factory, DevOps, Azure SQL/Synapse, Unity Catalog, and Git-based CI/CD.
• HR domain data: Oracle HCM Cloud extracts/PVOs/HDL, workforce analytics, and consumption via Power BI, Crunchr, or similar.
• Experience supporting regulated, sensitive employee data in insurance, financial services, or HR technology.
• Detail-oriented, collaborative, and comfortable working independently against offshore delivery expectations while staying aligned to MetLife leads.