Senior Data Engineer
First seen by Alion on Sep 9, 2026.
Role Summary :
We are hiring a Senior Data Engineer to drive the development of scalable, high-performance data pipelines for a migration project to a modern Azure Native architecture. This role will be instrumental in building an AI-ready data platform using Microsoft Fabric (Fabric IQ) and enabling seamless integration with AI/ML workflows (Foundry IQ).
Key Responsibilities :
- Design and build robust, scalable data pipelines (batch + near real-time) using Microsoft Fabric (Data Pipelines, Lakehouse).
- Lead migration of existing pipelines (AKS/custom ETL) to Fabric-native architecture.
- Implement Medallion Architecture (Bronze , Silver , Gold) for structured and unstructured healthcare data.
- Develop and optimize data models and transformations for analytics and AI use cases.
- Build ingestion frameworks for APIs (FHIR, EHR systems) and Files (JSON, HL7, CSV).
- Ensure data quality, validation (FHIR/claims), and observability across pipelines.
- Enable AI/ML workflows by preparing datasets for RAG pipelines and feature engineering.
- Optimize pipeline performance, cost, and scalability.
- Collaborate with Architects, AI/ML engineers, and Product teams to translate requirements into data solutions.
- Mentor junior engineers and drive engineering best practices.
Technical Skills :
- Languages : Python, PySpark, SQL (advanced).
- Azure / Fabric : Microsoft Fabric (preferred) OR Azure Data Factory, Synapse, Data Lake.
- Data Architecture : Lakehouse, Medallion Architecture, Batch + streaming data processing.
- Data Handling : JSON, Parquet, CSV, API integrations.
- Performance Optimization : Partitioning, indexing, query tuning.
Nice to Have :
- Experience with healthcare data (FHIR, claims, EHR).
- Exposure to RAG pipelines / vector databases / AI data prep.
- Familiarity with event-driven architectures.
- Understanding of data security & governance in regulated environments.
What Were Looking For :
- Strong hands-on engineer with ownership mindset.
- Ability to operate in fast-paced, evolving architecture environment.
- Problem-solver with focus on performance, scalability, and data quality.
- Mentor and team player.
Skills
Data Engineering, Data Pipeline, ETL, Medallion Architecture, Python, PySpark, Azure Data Factory, SQL, Azure