Senior Data Engineer

Job Summary

The Senior Data Engineer will be part of a six-person Data Services team supporting mission-critical healthcare applications, including real-time clinical workflows, patient data analytics, and regulatory reporting. The team has deep SQL Server expertise, and this position represents an intentional step toward expanding its capabilities in modern data engineering, lakehouse architecture, and agentic AI.

This individual will help build and enhance the enterprise lakehouse, curate the semantic layer needed to make AI-driven analytics reliable, and partner closely with Product Engineering, FP&A, and other teams that depend on accurate and accessible data. During the first year, the Senior Data Engineer is expected to drive meaningful technical debt reduction within the enterprise lakehouse, become a trusted technical partner across product roadmaps, and strengthen the capabilities of junior engineers through active mentorship.

This role is well suited for a data engineer who is energized by the intersection of lakehouse architecture and emerging agentic AI patterns and wants to develop these capabilities within a healthcare environment where data quality has meaningful consequences.

All activities must be in compliance with Equal Employment Opportunity laws, HIPAA, ERISA, and other regulations, as appropriate.

Essential Functions:

• In addition to working in accordance with appropriate conduct and behavioral standards, specific responsibilities of this role include:
• Design and build robust data pipelines and lakehouse solutions using Databricks, Microsoft Fabric, PySpark, and Python.
• Design AI-ready schemas, semantic layers, and metadata structures that allow AI agents to query data accurately and reliably.
• Build automated validation processes for agent-generated queries.
• Pilot agentic automation for data quality and reconciliation workflows while measuring accuracy and maintaining clear rollback plans.
• Curate and maintain the semantic layer that grounds AI agents and reduces incorrect responses caused by ambiguous or poorly defined data.
• Catalog and steward enterprise data assets to support internal analytics, financial planning, and product engineering needs.
• Partner proactively with Product Engineering, FP&A, and other stakeholders to identify and understand data requirements.
• Troubleshoot performance issues and support SQL Server and Azure SQL environments while continuing to develop operational expertise.
• Mentor junior engineers on data engineering best practices, architecture patterns, and technical standards.
• Contribute to data protection, security, and compliance reviews to ensure that tools and processes meet applicable standards, including HIPAA requirements where relevant.
• Carry out additional responsibilities assigned based on business needs.

Minimum Requirements:

• 7+ years of production experience in data engineering roles, including hands-on experience building and maintaining data pipelines at scale.
• Advanced proficiency with Python and PySpark or Apache Spark in production data engineering environments.
• Hands-on production experience with Databricks, Microsoft Fabric, or comparable modern data platforms.
• Demonstrated experience designing, implementing, and operating lakehouse architectures using Delta Lake, Parquet, or equivalent technologies.
• Experience designing AI-ready schemas, semantic layers, or metadata structures with downstream AI or machine learning query patterns in mind, or experience building automated validation for programmatically generated queries.
• Experience with agentic or automation-driven data quality workflows, such as AI-assisted reconciliation, agent-grounded semantic layers, or comparable applied solutions.
• Demonstrated ability to partner directly with product, analytics, finance, or business stakeholders to understand and shape data requirements proactively.
• Familiarity with Microsoft Azure cloud concepts, including platform as a service, infrastructure as a service, Azure SQL, and hybrid cloud models.
• Willingness and ability to develop deeper operational expertise within Azure SQL and SQL Server environments.
• Meaningful experience mentoring junior or mid-level engineers on technical standards, architecture patterns, and data engineering best practices.

Preferred Qualifications:

• Hands-on production experience with Azure SQL, SQL Server, or Azure SQL Managed Instance.
• Proficiency with T-SQL and PowerShell for automation and operational support.
• Experience with infrastructure-as-code tools such as Terraform, Bicep, or Azure Resource Manager templates.
• Familiarity with streaming data architectures and technologies such as Azure Event Hubs or Apache Kafka.
• Understanding of data mesh principles and distributed data ownership.
• Experience working within healthcare or another regulated industry.
• Familiarity with HIPAA requirements or healthcare data standards such as FHIR and HL7.

Working Environment/Physical Demands:

• 100% Remote in the United States
• Activities require a significant amount of work in front of a computer monitor
• Long-distance or air travel as needed - not to exceed 10% travel