2,398,867open jobs
149,645companies
293,344added this week
Browse all
Location
In office (Nagpur)
Seniority
Staff · 8+ years exp
Visa
Sponsors visas in Ireland; no register of this country names it

Confirmed on the employer's own hiring board on Nov 19, 2026. First seen by Alion on Nov 18, 2026.

Overview
Company
Impact
Profile match
HCLTech is an Indian information technology services and consulting company headquartered in Noida, Uttar Pradesh. Its roots lie in HCL, founded in 1976 by Shiv Nadar and partners, and the IT services business was set up as a separate company in 1991. It provides application development and maintenance, infrastructure and cloud services, cybersecurity, engineering and R&D services, and sells its own software products through HCLSoftware. The company employs more than 200,000 people in some 60 countries and is listed on the Indian stock exchanges.

Job Summary

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

  • Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
  • Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
  • Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
  • Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
  • Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
  • Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

  • Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
  • Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
  • Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
  • Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
  • Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
  • Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

  • Experience with streaming frameworks (Apache Flink or Spark Streaming).
  • Knowledge of PyIceberg for lightweight local Python data science workflows.
  • Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

Key Responsibilities

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

  • Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
  • Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
  • Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
  • Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
  • Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
  • Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

  • Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
  • Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
  • Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
  • Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
  • Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
  • Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

  • Experience with streaming frameworks (Apache Flink or Spark Streaming).
  • Knowledge of PyIceberg for lightweight local Python data science workflows.
  • Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

Skill Requirements

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

  • Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
  • Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
  • Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
  • Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
  • Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
  • Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

  • Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
  • Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
  • Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
  • Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
  • Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
  • Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

  • Experience with streaming frameworks (Apache Flink or Spark Streaming).
  • Knowledge of PyIceberg for lightweight local Python data science workflows.
  • Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).

Other Requirements

Role Overview

We are seeking an experienced Data Engineer to help design, build, and scale our next-generation open Data Lakehouse architecture. In this role, you will build and maintain high-throughput batch and streaming data pipelines using Python and PySpark, manage open-table transactional data with Apache Iceberg, and optimize ultra-fast interactive and federated analytics using Trino.

You will work closely with data scientists, analytics engineers, and platform teams to ensure data reliability, ACID compliance, zero data duplication, and sub-second-to-minute query performance across multiple compute engines.

Key Responsibilities

  • Lakehouse Pipeline Development: Design, write, and optimize modular PySpark and Python ETL/ELT pipelines to ingest, transform, and load massive datasets into Apache Iceberg tables on cloud object storage (AWS S3, GCS, or Azure Blob).
  • Iceberg Table Management & Optimization: Implement and manage Iceberg table formats (v1/v2), schema evolution, partitioning strategies (hidden partitioning), snapshot expirations, orphan file cleanups, and regular compaction/rewrite jobs (rewrite_data_files).
  • Trino Query & Performance Tuning: Administer and tune Trino query execution (concurrency, dynamic filtering, memory limits, and catalog configurations) for low-latency ad-hoc analytics and BI dashboards.
  • Catalog & Multi-Engine Interoperability: Manage metadata and catalog integrations (e.g., REST Catalog, AWS Glue, Apache Polaris, or Project Nessie) to guarantee seamless multi-engine read/write access across Spark, Trino, and Python/PyIceberg.
  • Data Quality & Observability: Implement automated data validation, schema enforcement, lineage tracking, and monitoring (using tools like Apache Airflow, Great Expectations, or similar orchestration frameworks).
  • Cost & Performance Governance: Decouple compute from storage costs, monitor query execution plans, and continuously tune partition/sort layouts to minimize I/O and storage overhead.

Required Qualifications

  • Experience: 8+ years of hands-on data engineering experience building large-scale data lakes or lakehouses.
  • Programming & Processing: Advanced proficiency in Python and Apache Spark / PySpark for distributed data processing.
  • Table Formats: Deep practical experience with Apache Iceberg (ACID transactions, time travel, schema/partition evolution, manifest management, and compaction).
  • Query Engines: Strong working knowledge of Trino (or Presto) for federated and distributed SQL analytics.
  • Catalogs & Storage: Familiarity with modern Iceberg catalogs (REST Catalog, Nessie, Glue, Polaris) and cloud object storage (S3, GCS, or ADLS).
  • Orchestration & DevOps: Experience with workflow orchestrators (Apache Airflow, Dagster, or Prefect) and containerization/IaC (Docker, Kubernetes, Terraform).

Preferred / Nice-to-Have Skills

  • Experience with streaming frameworks (Apache Flink or Spark Streaming).
  • Knowledge of PyIceberg for lightweight local Python data science workflows.
  • Experience with data governance, security, and access control (e.g., Ranger, Hive/OAuth2 credential vending).
Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
2,398,867 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

Backend
Similar stack
Same company
Nagpur
In office · Full-Time · 10+ years exp · Bachelor's Degree · Bengaluru
Management
ServiceNow
Life Sciences
PCR
Apply
In office · Full-Time · 13+ years exp · Bachelor's Degree · Bengaluru
Java
Java
Maven
Databases
Apache Kafka
Mobile
JUnit
DevOps
GCP
Azure
CI/CD
Jenkins
Git
AWS
Docker
Kubernetes
Self-Healing
Management
Agile
QA
TestNG
JMeter
Apply
In office · Full-Time · 10+ years exp · Bachelor's Degree · Chennai
Management
ServiceNow
Life Sciences
PCR
Apply
In office · Full-Time · 10+ years exp · Bachelor's Degree · Bengaluru
JavaScript
TypeScript
SQL
C#
C#
ASP.NET Core
Entity Framework Core
Databases
PostgreSQL
TimescaleDB
RabbitMQ
InfluxDB
Apache Kafka
Azure Cosmos DB
AI/ML
Time Series
Frontend
GraphQL
RxJS
Angular
NgRx
Mobile
Dependency Injection
DevOps
Rest API
Terraform
Helm
Azure DevOps
GitHub Actions
Azure
CI/CD
Docker
Kubernetes
Bicep
Azure AKS
IoT
MQTT
Apply
Remote (India) · 12+ years exp · India
Python
JavaScript
Java
TypeScript
Python
FastAPI
Celery
Java
Spring Boot
Quarkus
Databases
MySQL
PostgreSQL
Redis
RabbitMQ
Apache Kafka
AI/ML
Embeddings
RAG
Semantic Search
Recommender Systems
Frontend
GraphQL
React.js
Mobile
Clean Architecture
DevOps
Rest API
Terraform
CloudFormation
CI/CD
AWS
Docker
Kubernetes
Amazon EKS
AWS Lambda
Amazon EC2
Amazon S3
IAM
Amazon ECS
Amazon CloudWatch
Apply
In office · Nagpur
Java
Java
Spring Boot
Databases
Apache Kafka
DevOps
Rest API
CI/CD
Apply
In office · Nagpur
Java
Java
Spring Boot
Hibernate
DevOps
Kubernetes
Management
Agile
Apply
In office · 6+ years exp · Nagpur
JavaScript
Java
TypeScript
SQL
Java
Spring Boot
Hibernate
Frontend
Angular
DevOps
Rest API
Jenkins
Docker
Kubernetes
Analytics
ETL/ELT
Apply
In office · Nagpur
Java
Java
Spring Boot
Hibernate
DevOps
Rest API
CI/CD
Management
Agile
Apply
In office · 8+ years exp · Nagpur
Python
SQL
Python
pySpark
Databases
Apache Iceberg
Presto
Trino
AI/ML
Spark
Airflow
Dagster
Prefect
Great Expectations
Flink
DevOps
Terraform
Azure
AWS
Docker
Kubernetes
Amazon S3
Cybersecurity
Polaris
Analytics
ETL/ELT
AWS Glue
Apply
See all jobs
This is one of many
2,398,867 more open roles from verified company boards, updated every day.