2,457,026open jobs
150,105companies
312,685added this week
Browse all
Location
Hybrid (Hyderabad, India)
Seniority
Staff · 10+ years exp
Visa
Sponsors visas in United States; no register of this country names it
Employment
Full-Time

Confirmed on the employer's own hiring board on Nov 21, 2026. First seen by Alion on Nov 21, 2026. Egen scores B on the Alion truth index.

Overview
Company
Impact
Profile match
Egen is an American data, cloud and artificial intelligence consulting firm that designs and builds data platforms, cloud applications and AI solutions for enterprises, startups and public sector organisations. Its services cover data engineering on platforms such as Databricks and Google Cloud, application modernisation, cloud migration and generative AI implementation. The firm runs delivery teams in Chicago, San Francisco and Hyderabad, India. The company is headquartered in Naperville, Illinois, United States.

Job Overview:

We are looking for a skilled and motivated Lead Data Engineer with strong experience in Python programming and Google Cloud Platform (GCP) to join our data engineering team. The ideal candidate will be responsible for requirements gathering, designing, architecting the solution, developing, and maintaining robust and scalable ETL (Extract, Transform, Load) & ELT data pipelines. The role involves working with customers directly, gathering requirements, discovery phase, designing, architecting the solution, using various GCP services, implementing data transformations, data ingestion, data quality, and consistency across systems, and post post-delivery support.

Key Responsibilities:

  • Design, develop, test, and maintain scalable ETL data pipelines using Python.

  • Architect the enterprise solutions with various technologies like Kafka, multi-cloud services, auto-scaling using GKE, Load balancers, APIGEE proxy API management, DBT, using LLMs as needed in the solution, redaction of sensitive information, DLP (Data Loss Prevention) etc.

  • Work extensively on Google Cloud Platform (GCP) services such as:

  • Dataflow for real-time and batch data processing

  • Cloud Functions for lightweight serverless compute

  • BigQuery for data warehousing and analytics

  • Cloud Composer for orchestration of data workflows (on Apache Airflow)

  • Google Cloud Storage (GCS) for managing data at scale

  • IAM for access control and security

  • Cloud Run for containerized applications

  • Should have experience in the following areas :

  • API framework: Python FastAPI

  • Processing engine: Apache Spark

  • Messaging and streaming data processing: Kafka

  • Storage: MongoDB, Redis/Bigtable

  • Orchestration: Airflow

  • Experience in deployments in GKE, Cloud Run.

  • Perform data ingestion from various sources and apply transformation and cleansing logic to ensure high-quality data delivery.

  • Implement and enforce data quality checks, validation rules, and monitoring.

  • Collaborate with data scientists, analysts, and other engineering teams to understand data needs and deliver efficient data solutions.

  • Manage version control using GitHub and participate in CI/CD pipeline deployments for data projects.

  • Write complex SQL queries for data extraction and validation from relational databases such as SQL Server, Oracle, or PostgreSQL.

  • Document pipeline designs, data flow diagrams, and operational support procedures.

Required Skills:

  • 10+ years of hands-on experience in Python for backend or data engineering projects.

  • Strong understanding and working experience with GCP cloud services (especially Dataflow, BigQuery, Cloud Functions, Cloud Composer, etc.).

  • Solid understanding of data pipeline architecture, data integration, and transformation techniques.

  • Experience in working with version control systems like GitHub and knowledge of CI/CD practices.

  • Experience in Apache Spark, Kafka, Redis, Fast APIs, Airflow, GCP Composer DAGs.

  • Strong experience in SQL with at least one enterprise database (SQL Server, Oracle, PostgreSQL, etc.).

  • Experience with PySpark is required.

  • Experience in data migrations from on-premise data sources to Cloud platforms.

Optional & Additional Skills:

  • Experience working with the Snowflake cloud data platform.

  • Hands-on knowledge of Databricks for big data processing and analytics.

  • Familiarity with Azure Data Factory (ADF) and other Azure data engineering tools.

  • Experience with AWS services.

  • Excellent problem-solving and analytical skills.

  • Strong communication skills and ability to collaborate in a team environment.

Education

    • Bachelor's degree in Computer Science, a related field, or equivalent experience.
Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
2,457,026 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

Data Science
Similar stack
Same company
Hyderabad
Data Architect Lead 3 hours ago
$22k – $65k per year · Remote (India) · Full-Time
Python
SQL
Python
pySpark
Databases
Databricks
Delta Lake
Microsoft Fabric
AI/ML
Copilot
Spark
Machine Learning
DevOps
Azure DevOps
GitHub Actions
Azure
CI/CD
Git
GitHub
Cybersecurity
Microsoft Entra ID
Analytics
Power BI
ETL/ELT
SSIS
Azure Data Factory
Management
Agile
Scrum
Apply
Lead Data Engineer 4 hours ago
In office · 7+ years exp · Mumbai
Python
SQL
AI/ML
Airflow
DevOps
Terraform
CloudFormation
Datadog
CI/CD
AWS
Amazon CloudWatch
Analytics
Dimensional Modeling
Apply
Hybrid · Full-Time · Bachelor's Degree · Bengaluru
Databases
Databricks
Oracle
AI/ML
Spark
Computer Vision
DevOps
Azure
Management
Agile
Apply
Hybrid · Full-Time · Bachelor's Degree · Bengaluru
Databases
Snowflake
Databricks
Oracle
AI/ML
Spark
AI Agents
DevOps
Azure
Analytics
ETL/ELT
Management
Agile
Apply
Hybrid · Full-Time · Bachelor's Degree · Gurgaon
Databases
Snowflake
Databricks
Oracle
AI/ML
Spark
AI Agents
DevOps
Azure
Analytics
ETL/ELT
Management
Agile
Apply
In office · Hyderabad
Apply
In office · Bachelor's Degree · Hyderabad
JavaScript
Java
TypeScript
SQL
Java
Hibernate
Frontend
Angular
DevOps
Azure
AWS
Management
Agile
Scrum
Apply
In office · Full-Time · 10+ years exp · Bachelor's Degree · Hyderabad
DevOps
Splunk
Terraform
Ansible
Red Hat
VMWare
Chef
Datadog
Azure
Windows Server
Grafana
Platform Engineering
AIOps
GitHub
Linux
Windows
Management
ServiceNow
Apply
In office · Full-Time · 10+ years exp · Hyderabad
AI/ML
AI Agents
RAG
DevOps
Splunk
OpenTelemetry
Azure
CI/CD
Platform Engineering
FinOps
AIOps
Incident Management
SLI/SLO/SLA
Management
ServiceNow
Apply
In office · Full-Time · 8+ years exp · Bachelor's Degree · Hyderabad
SQL
Databases
Oracle
DevOps
Incident Management
Cybersecurity
Zero Trust
Analytics
Power BI
Management
Confluence
Jira
ServiceNow
Agile
Scrum
Kanban
Apply
See all jobs
This is one of many
2,457,026 more open roles from verified company boards, updated every day.