2,398,867open jobs
149,645companies
293,344added this week
Browse all
Location
In office (Nagpur)
Seniority
Senior · 8+ years exp
Visa
Sponsors visas in Ireland; no register of this country names it

Confirmed on the employer's own hiring board on Nov 19, 2026. First seen by Alion on Nov 18, 2026.

Overview
Company
Impact
Profile match
HCLTech is an Indian information technology services and consulting company headquartered in Noida, Uttar Pradesh. Its roots lie in HCL, founded in 1976 by Shiv Nadar and partners, and the IT services business was set up as a separate company in 1991. It provides application development and maintenance, infrastructure and cloud services, cybersecurity, engineering and R&D services, and sells its own software products through HCLSoftware. The company employs more than 200,000 people in some 60 countries and is listed on the Indian stock exchanges.

Job Summary

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

  • Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
  • Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
  • Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
  • Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
  • Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
  • Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

  • Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
  • Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
  • Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
  • Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
  • Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
  • Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

  • Experience with streaming frameworks (Apache Flink or Spark Streaming).
  • Knowledge of PyIceberg for lightweight local Python data science workflows.
  • Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

Key Responsibilities

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

  • Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
  • Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
  • Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
  • Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
  • Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
  • Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

  • Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
  • Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
  • Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
  • Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
  • Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
  • Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

  • Experience with streaming frameworks (Apache Flink or Spark Streaming).
  • Knowledge of PyIceberg for lightweight local Python data science workflows.
  • Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

Skill Requirements

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

  • Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
  • Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
  • Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
  • Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
  • Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
  • Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

  • Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
  • Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
  • Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
  • Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
  • Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
  • Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

  • Experience with streaming frameworks (Apache Flink or Spark Streaming).
  • Knowledge of PyIceberg for lightweight local Python data science workflows.
  • Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

Other Requirements

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

  • Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
  • Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
  • Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
  • Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
  • Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
  • Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

  • Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
  • Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
  • Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
  • Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
  • Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
  • Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

  • Experience with streaming frameworks (Apache Flink or Spark Streaming).
  • Knowledge of PyIceberg for lightweight local Python data science workflows.
  • Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).
Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
2,398,867 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

Backend
Similar stack
Same company
Nagpur
In office · 7+ years exp · Pune
Java
Java
Spring Boot
Databases
Apache Kafka
DevOps
AWS
IAM
Apply
Hybrid · 7+ years exp · Master's Degree · Pune
Databases
MS SQL
DevOps
Rest API
Azure DevOps
Azure
CI/CD
Incident Management
GitHub
Management
Agile
Scrum
Apply
In office · Full-Time · 8+ years exp · Pune
Java
SQL
Java
Spring Boot
Apache Camel
Databases
Oracle
Apache Kafka
DevOps
OpenShift
CI/CD
Jenkins
Git
Bitbucket
GitLab
Cybersecurity
SonarQube
Veracode
Management
Agile
Apply
Hybrid · Full-Time · 4+ years exp · Pune
JavaScript
Java
TypeScript
C#
Java
Spring Framework
Spring Boot
Spring Security
C#
.NET
Databases
MS SQL
Frontend
Angular
DevOps
OpenShift
Git
Docker
Management
Agile
Scrum
Kanban
Apply
In office · Full-Time · 2+ years exp · Bachelor's Degree · Bengaluru
Java
Kotlin
DevOps
CI/CD
Apply
In office · Nagpur
Java
Java
Spring Boot
Databases
Apache Kafka
DevOps
Rest API
CI/CD
Apply
In office · Nagpur
Java
Java
Spring Boot
Hibernate
DevOps
Kubernetes
Management
Agile
Apply
In office · 6+ years exp · Nagpur
JavaScript
Java
TypeScript
SQL
Java
Spring Boot
Hibernate
Frontend
Angular
DevOps
Rest API
Jenkins
Docker
Kubernetes
Analytics
ETL/ELT
Apply
In office · Nagpur
Java
Java
Spring Boot
Hibernate
DevOps
Rest API
CI/CD
Management
Agile
Apply
In office · 8+ years exp · Nagpur
Python
SQL
Python
pySpark
Databases
Apache Iceberg
Presto
Trino
AI/ML
Spark
Airflow
Dagster
Prefect
Great Expectations
Flink
DevOps
Terraform
Azure
AWS
Docker
Kubernetes
Amazon S3
Cybersecurity
Polaris
Analytics
ETL/ELT
AWS Glue
Apply
See all jobs
This is one of many
2,398,867 more open roles from verified company boards, updated every day.