Senior Data Engineer

We are hiring a Senior Data Engineer for our privacy-preserving data platform. This is a highly technical, hands-on role on a team that prioritizes engineering discipline, a deep understanding of distributed systems, and verifiable production experience over theoretical knowledge. You will not just be writing PySpark syntax; you will be architecting resilient pipelines, proactively optimizing AWS cloud costs, and building secure platforms that handle highly sensitive Personally Identifiable Information (PII) at scale.

Responsibilities:

• Data Extraction and Curation: Build robust extractions from transactional systems and our Athena data lake; land, transform, and write curated datasets in Parquet (and modern table formats).
• Privacy and Security: Implement complex de-identification transforms, dynamic data masking, and tokenization. Write automated tests that assert on the handling class of every single column, ensuring strict PII compliance.
• Pipeline Orchestration: Own orchestration, scheduling, retries, backfills, alerting, and data-quality checks using Apache Airflow (or similar tools like Dagster/Step Functions).
• Cost and Performance Optimization: Actively monitor and reduce AWS billing by tuning Spark jobs, optimizing joins (e. g., broadcast vs. shuffle), handling data skewness, shutting down idle clusters, and choosing optimal storage structures (partitioning vs. bucketing).
• Production Reliability: Support what you build in production. Handle complex upstream schema drift, write idempotent pipelines, and troubleshoot 3 AM pipeline failures efficiently. Establish SLAs and drive Root Cause Analyses (RCAs) for pipeline outages.
• Infrastructure and Documentation: Contribute to CI/CD and infrastructure-as-code alongside our platform team. Populate the catalog and lineage, and maintain pristine written technical standards for every dataset produced.
• Mentorship and Architecture: Own systems end-to-end. Architect solutions from scratch, write author-written technical standards, mentor junior engineers, and push back on designs you identify as flawed, inefficient, or insecure.

Requirements:

• Core Stack: Deep, hands-on production experience with Python, SQL, and distributed processing via Apache Spark. You must understand how Spark works under the hood (memory management, Catalyst Optimizer, OOM prevention).
• AWS Ecosystem: Proven track record of building, operating, and maintaining batch pipelines in production on AWS (S3, Glue, EMR, Athena).
• Engineering Discipline: Real software engineering rigor: version control, thorough code review, automated testing, and a battle-tested understanding of idempotency and backfills.
• Evidence-Based Metrics: If you claim performance improvements or cost reductions on your CV, you must be able to explain the exact technical mechanisms used to achieve them.
• Communication: Absolute comfort working from written technical standards and documentation rather than relying solely on verbal conversations.

Nice to Have:

• Experience with modern open-source data formats in production (Apache Iceberg, Delta Lake, or Hudi).
• Change Data Capture (CDC) tooling such as Debezium or AWS DMS.
• Advanced exposure to data masking, tokenization, or AWS security tooling (Macie, GuardDuty, Security Hub).
• Experience with dbt or a comparable transformation framework.
• Experience building datasets specifically designed to feed machine learning model training and understanding how ML consumers differ from standard BI/reporting consumers.

Skills

AWS, Azure, apache spark, pyspark, python, spark, sql