Sr Data Engineer

Our client, is seeking an experienced, detail-oriented Sr Data Engineer to join their platform modernization team.

In this technical, hands-on role, you will build pipelines, transformations, and data validation tools to support the modernization of a high-stakes healthcare pharmacy compliance platform using Databricks. Working within an Agile delivery pod under a technical lead, you will engineer high-volume ingestion workloads across a multi-tier Delta Lake architecture (Bronze, Silver, Gold) while porting and re-implementing existing SQL Server stored procedure logic into PySpark and Spark SQL. Because this platform drives financial and regulatory decisions, engineering efforts prioritize data accuracy, automated testing, parity reconciliation against legacy outputs, and full traceability across every pipeline step.

Duration: 6-Month Contract (100% Remote)

Work Arrangement: 100% Remote



Advantages
100% Remote Flexibility: Enjoy complete remote work flexibility while contributing to a mission-critical platform modernization initiative.

Modern Lakehouse Architecture: Hands-on technical execution utilizing Databricks, PySpark, Spark SQL, and Delta Lake design patterns.

High-Impact Regulatory Domain: Drive data pipelines for a platform handling critical healthcare and pharmacy compliance workflows where precision and data quality are paramount.

Focused 6-Month Engagement: High-intensity consulting assignment centered on core legacy-to-cloud platform migration and reconciliation.



Responsibilities
Delta Lake Pipeline Engineering: Build and maintain high-volume batch and intraday ingestion pipelines into a Delta Lake architecture (Bronze, Silver, Gold layers).

Stored Procedure Migration: Port, translate, and re-implement legacy SQL Server stored procedure business logic into optimized PySpark and Spark SQL code.

Data Quality & Reconciliation: Develop automated test suites, data quality checks, and output-parity reconciliation jobs comparing new lakehouse results against legacy system outputs.

Performance Tuning & Optimization: Optimize Spark job execution by tuning partition strategies, file sizing, join types, skew handling, and cluster compute parameters.

Idempotency & Late Data Handling: Construct idempotent, re-runnable data pipelines designed to gracefully manage late-arriving or corrected data updates.

CI/CD & Engineering Standards: Maintain version control in Git, participate in peer code reviews, build CI/CD pipeline automation, and document technical design specs.



Qualifications
Data Engineering Depth: 6+ years of dedicated professional experience in data engineering, including 3+ years of hands-on production experience with Databricks and Apache Spark.

Python & PySpark Proficiency: Strong skills independently writing, testing, and tuning production-grade Python and PySpark scripts.

SQL & T-SQL Expertise: Advanced proficiency in SQL and T-SQL, with the demonstrated ability to interpret, dissect, and translate complex SQL Server stored procedures.

Delta Lake Expertise: Practical experience with Delta Lake capabilities, including MERGE/upsert operations, incremental loads, schema evolution, time travel, and table maintenance.

Spark Execution Knowledge: Deep understanding of the Spark execution model (jobs, stages, shuffles, join strategies, memory allocation) and performance troubleshooting.

DevOps & Testing: Proficiency using Git, automated data testing frameworks, and CI/CD tools for data pipeline delivery.

Communication Skills: Strong written communication skills with a focus on clear technical documentation and team collaboration.

Preferred Assets & Nice-to-Haves
Experience with healthcare, pharmacy, payer, or life-sciences data (e.g., HL7, ePrescribe, NCPDP, claims, NDC drug data, or HIPAA-regulated PHI).

Direct experience with SQL Server to Databricks / Lakehouse migrations.

Familiarity with Azure Data Factory, Azure Storage, Delta Live Tables, Databricks Workflows, and Unity Catalog.

Exposure to data quality tools such as Great Expectations or transformation frameworks like dbt.

Active Databricks Certified Data Engineer (Associate or Professional) designation.



Summary
If you are a tech-savvy Sr Data Engineer who pairs 6+ years of enterprise experience with 3+ years of production Databricks/PySpark mastery, T-SQL stored procedure translation skills, and a commitment to data accuracy and testing, this 100% remote 6-month contract opportunity is an ideal match. Bring your Lakehouse engineering precision, Spark tuning skills, and migration expertise to our team today!

Randstad Canada is committed to fostering a workforce reflective of all peoples of Canada. As a result, we are committed to developing and implementing strategies to increase the equity, diversity and inclusion within the workplace by examining our internal policies, practices, and systems throughout the entire lifecycle of our workforce, including its recruitment, retention and advancement for all employees. In addition to our deep commitment to respecting human rights, we are dedicated to positive actions to affect change to ensure everyone has full participation in the workforce free from any barriers, systemic or otherwise, especially equity-seeking groups who are usually underrepresented in Canada's workforce, including those who identify as women or non-binary/gender non-conforming; Indigenous or Aboriginal Peoples; persons with disabilities (visible or invisible) and; members of visible minorities, racialized groups and the LGBTQ2+ community.

Randstad Canada is committed to creating and maintaining an inclusive and accessible workplace for all its candidates and employees by supporting their accessibility and accommodation needs throughout the employment lifecycle. We ask that all job applications please identify any accommodation requirements by sending an email to [email protected] to ensure their ability to fully participate in the interview process.

This posting is for existing and upcoming vacancies.