Confirmed on the employer's own hiring board on Nov 19, 2026. First seen by Alion on Nov 18, 2026.
Job Summary
Role Overview
We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.
You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.
Key Responsibilities
- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.
Required Qualifications
- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).
Preferred / Nice-to-Have Skills
- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).
Key Responsibilities
Role Overview
We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.
You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.
Key Responsibilities
- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.
Required Qualifications
- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).
Preferred / Nice-to-Have Skills
- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).
Skill Requirements
Role Overview
We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.
You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.
Key Responsibilities
- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.
Required Qualifications
- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).
Preferred / Nice-to-Have Skills
- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).
Other Requirements
Role Overview
We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.
You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.
Key Responsibilities
- Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
- Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
- Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
- Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
- Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
- Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.
Required Qualifications
- Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
- Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
- Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
- Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
- Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
- Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).
Preferred / Nice-to-Have Skills
- Experience with streaming frameworks (Apache Flink or Spark Streaming).
- Knowledge of PyIceberg for lightweight local Python data science workflows.
- Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

