Data Engineer

This role is central to building our future-state architecture by modernizing how public health data moves, transforms, and informs decision-making at scale. You will contribute to the design and implementation of next-generation pipelines, ensure data reliability and compliance, and partner with integration teams on the migration from legacy systems.

As part of a collaborative, agile team, you'll work closely with data architects, integration engineers, data scientists, and product stakeholders to deliver solutions that directly impact public health outcomes. The ideal candidate thrives in complex data environments, is motivated by meaningful mission-driven work, and brings strong technical expertise and a collaborative mindset to help advance our interoperability platform.

What You’ll Be Doing

• Play a key role in shaping the future of our data interoperability platform.
• Design and operate secure, high-performance data pipelines for real-time and batch public health data flows.
• Architect and implement scalable solutions that move sensitive health data reliably and meet strict security and compliance standards.
• Build and optimize ETL/ELT pipelines using NiFi, Kafka, Python, and SQL.
• Build and maintain CI/CD pipelines for NiFi flows, Python, and SQL using GitHub and GitHub Actions, including flow versioning through NiFi Registry and automated promotion across dev, test, and production.
• Deploy and operate data pipelines on Azure and AWS, partnering with DevOps/SRE to integrate them into existing cloud infrastructure and tooling.
• Apply CI/CD best practices: pull request reviews, branch protection, automated testing and validation, parameterized environment configuration, and versioned releases with rollback.
• Collaborate with interoperability, product, and data science teams to migrate legacy pipelines to modern architectures.
• Fine-tune data lakes and ensure smooth data exchange across systems.
• Define monitoring and alerting strategies (e.g., Grafana dashboards and alerts) and create runbooks to support operational teams.
• Troubleshoot production issues to maintain stable and performant pipelines.
• Drive continuous improvement across the data ecosystem through collaboration, innovation, and strong execution.
• Contribute directly to building and delivering high-quality, scalable integrations that drive rapid product progress.
• Collaborate closely with teammates to share knowledge and ensure smooth, efficient development workflows.

What We're Looking For

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field.
• 3+ years of experience implementing enterprise-grade data pipelines, or 7+ years of related professional experience (e.g., software, data, or integration engineering) that includes building and operating production data pipelines.
• Hands-on expertise with NiFi, Kafka, Python, and SQL, including designing, operating, and optimizing ETL/ELT pipelines in the cloud.
• Proven ability to build robust, production-grade pipelines from the ground up and work confidently with distributed data systems.
• Hands-on experience deploying and operating data pipelines in Azure or AWS, including secure handling of sensitive data (IAM/RBAC, encryption, secrets management).
• Experience implementing CI/CD best practices for data pipelines with GitHub and GitHub Actions, including NiFi flow versioning and promotion (NiFi Registry), automated testing, and peer-reviewed releases.
• Able to partner with DevOps/SRE to integrate pipelines into existing tooling (not responsible for overall platform design).
• Experience using or integrating with OpenSearch/Elasticsearch.
• Practical experience using relational databases and existing schemas (implementing against established data models; extending under guidance).
• Strong understanding of data security and compliance requirements for sensitive data.
• Excellent cross-functional collaborator with clear communication and strong technical documentation skills, able to translate complex engineering concepts into clear strategies for cross-functional partners.
• A curious, proactive problem solver with a growth mindset who thrives in fast-paced, agile environments and is motivated by mission-driven work and complex interoperability challenges.

What Would Make You Stand Out

• Deep technical expertise paired with a strong understanding of public health data environments.
• Experience with healthcare data standards such as HL7 v2.x, FHIR, and terminologies like LOINC, CVX, SNOMED, and ICD-10.
• Proven success building event-driven architectures with Kafka or implementing data de-identification strategies.
• Hands-on experience with integration engines (e.g., Rhapsody, Mirth) and containerized deployments (e.g., Docker, Kubernetes).
• Experience with infrastructure as code (Terraform, Bicep, or CloudFormation) and automated deployment of NiFi or Kafka environments.
• Experience with monitoring and observability for data pipelines using Grafana (plus cloud-native tools such as Azure Monitor or CloudWatch).
• Demonstrated ability to scale data platforms in production environments.
• Familiarity with Master Patient Index (MPI) solutions or data deduplication frameworks.
• Bring a proactive mindset: anticipating challenges, designing with scalability and security in mind, and delivering solutions that accelerate time to product.