Lead Big Data Engineer

ATYETI • DevOps & Platform Engineering • IT Consulting & Digital Transformation • SQL • ETL/ELT • 1 day ago
Smart Apply
Similar jobs
1,327,087open jobs
78,121companies
212,145added this week
Browse all
Salary
≈ $27k – $48k per year (Estimated)
Location
Remote (India)
Seniority
Staff · 7+ years exp

First seen by Alion on Oct 6, 2026.

Overview
Company
Impact
Profile match

ATYETI

Atyeti accelerates enterprise digital transformation with Cloud, Agentic AI, Platform Engineering, and Banking & Capital Markets technology solutions.
atyeti.com • Pune • Bengaluru • Mumbai • Hyderabad • Thiruvananthapuram • AI Consulting & Integration • DevOps & Platform Engineering • IT Consulting & Digital Transformation
Tech stack of ATYETI

Key Responsibilities :

- Design and execute Hadoop to Databricks migration strategies for large-scale data platforms.

- Migrate existing HDFS, Hive, Spark, and MapReduce workloads to Databricks.

- Data Engineering - Hadoop to Databricks migration experience.

- Strong hands-on experience in migrating and optimizing Spark code.

- Analyze existing Hadoop/Spark applications and identify opportunities for modernization and optimization.

- Strong hands-on experience in migrating, refactoring, and optimizing Spark code for Databricks.

- Develop scalable ETL/ELT pipelines using Spark, SQL, and Databricks.

- Optimize Spark jobs by improving partitioning, caching, joins, shuffling, serialization, and memory utilization.

- Work with Databricks Delta Lake / Delta tables for reliable and scalable data processing.

- Implement Delta Lake features such as ACID transactions, schema evolution, time travel, and optimized data layouts.

- Convert legacy Hive SQL / Spark SQL workloads to Databricks SQL and optimize queries.

- Migrate data from HDFS/Hive to cloud-based storage and Databricks.

- Experience with Azure Data Lake Storage (ADLS), AWS S3, or equivalent cloud storage.

- Develop and maintain robust data pipelines using Databricks Workflows and job orchestration tools.

- Implement data quality, validation, reconciliation, and error-handling frameworks during migration.

- Perform source-to-target mapping, data profiling, transformation analysis, and migration validation.

- Troubleshoot performance issues across Spark clusters, notebooks, jobs, and data pipelines.

- Work closely with architects, business analysts, data scientists, and application teams to understand data requirements.

- Implement CI/CD practices for Databricks notebooks, jobs, and code using Git and DevOps tools.

- Follow enterprise security, governance, audit, and compliance standards applicable to banking environments.

- Ensure migrated workloads meet performance, scalability, availability, and data-quality requirements.

Skills

Big Data, Data Engineering, Hadoop, Databricks, HDFS, Hive, Spark, ETL, SQL, DataLake, Data Migration

Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
1,327,087 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

Data Science
Similar stack
Same company
India
≈ $27k – $47k per year (Estimated) • Remote (United States) • Full-Time • Kochi
Python
SQL
Apex
Apex
MuleSoft
Databases
MySQL
PostgreSQL
DevOps
GCP
Azure
AWS
Analytics
ETL/ELT
Informatica
Talend
SSIS
Azure Data Factory
Management
ServiceNow
Agile
Apply
≈ $58k – $139k per year (Estimated) • Remote (Italy) • Milan
AI/ML
DALL-E
Machine Learning
Cybersecurity
GDPR
Analytics
Informatica
Data Vault
Dimensional Modeling
Management
Confluence
Apply
≈ $126k – $213k per year (Estimated) • Remote (United States) • 8+ years exp
Python
SQL
Databases
Snowflake
AI/ML
Machine Learning
DevOps
AWS
Apply
≈ $112k – $229k per year (Estimated) • Remote (United States) • Full-Time
Python
TypeScript
SQL
Databases
Snowflake
Databricks
Delta Lake
Apache Kafka
Google BigQuery
BigQuery
AI/ML
LangGraph
LangChain
Spark
LLM
RAG
Feature Store
Edge AI
Agentic Workflows
Multi-Agent Systems
DevOps
GCP
Azure
AWS
FinOps
Analytics
Tableau
Power BI
ETL/ELT
Apply
$100k – $175k per year • Remote (United States) • 12+ years exp • Bachelor's Degree
Databases
Snowflake
Databricks
Delta Lake
LookML
Apache Kafka
Google BigQuery
Amazon Redshift
BigQuery
AI/ML
dbt
Flink
DevOps
GCP
Azure
AWS
Analytics
Collibra
Alation
Apply
Hybrid • Full-Time • 4+ years exp • Bachelor's Degree • Ho Chi Minh City
JavaScript
TypeScript
SQL
C#
C#
.NET
Databases
PostgreSQL
AI/ML
Copilot
Frontend
Next.js
Angular
React.js
DevOps
Azure
CI/CD
Management
Miro
Agile
Scrum
Apply
$86k per year • Remote (Lithuania) • Full-Time
SQL
Databases
Microsoft Fabric
Analytics
Power BI
Apply
≈ $31k – $57k per year (Estimated) • Remote (likely EAEU) • 6+ years exp • Saint Petersburg
Python
SQL
Python
FastAPI
Asyncio
Databases
PostgreSQL
Redis
Apache Kafka
AI/ML
LLM
DevOps
Rest API
Docker
Apply
$13k per year • Remote (EAEU) • 1+ year exp • Saint Petersburg
SQL
Databases
ElasticSearch
DevOps
Rest API
Kibana
Docker
Kubernetes
SOAP
Cybersecurity
OWASP Top 10
Management
Jira
QA
Postman
BrowserStack
Apply
DevOps Engineer 1 day ago
In office • Full-Time • Budapest
Python
JavaScript
Node JS
Bash
DevOps
GCP
Azure
AWS
Docker
Kubernetes
TeamCity
Incident Management
Linux
TCP/IP
Apply
≈ $21k – $43k per year (Estimated) • Remote (India) • 5+ years exp • India
Python
SQL
Python
pySpark
Databases
Snowflake
Apache Kafka
AI/ML
Copilot
Cursor
Claude
Spark
DevOps
CI/CD
AWS
AWS Lambda
Amazon S3
Unix
Management
Agile
Apply
Lead Data Engineer 15 days ago
≈ $25k – $44k per year (Estimated) • In office • 8+ years exp • Pune
Python
Python
pySpark
Databases
Databricks
Delta Lake
Azure SQL Database
AI/ML
Spark
DevOps
Azure
Analytics
ETL/ELT
Azure Data Factory
Management
Agile
Scrum
Apply
≈ $25k – $45k per year (Estimated) • In office • 7+ years exp • Pune
Python
SQL
Python
pySpark
Databases
Databricks
Delta Lake
Azure SQL Database
AI/ML
Spark
DevOps
Azure
CI/CD
Git
Analytics
ETL/ELT
Azure Data Factory
Management
Agile
Scrum
Apply
≈ $25k – $64k per year (Estimated) • In office • 8+ years exp • Pune
DevOps
Splunk
Terraform
Azure DevOps
GitLab CI
Azure
CI/CD
Jenkins
Configuration Management
Bicep
Management
Agile
Apply
≈ $123k – $258k per year (Estimated) • Remote (likely United States) • Contractor
DevOps
GCP
IAM
Apply
Data Engineer 4 hours ago
≈ $21k – $43k per year (Estimated) • In office • Full-Time • 5+ years exp • India
Python
SQL
Scala
Python
pySpark
Databases
Snowflake
Databricks
Delta Lake
AI/ML
Hadoop
Spark
DevOps
GCP
Azure
CI/CD
Git
AWS
Analytics
ETL/ELT
Azure Data Factory
Dimensional Modeling
Apply
≈ $27k – $48k per year (Estimated) • In office • 7+ years exp • Gurgaon
SQL
Analytics
Tableau
Power BI
Apply
Platform SRE 2 hours ago
≈ $12k – $30k per year (Estimated) • In office • Full-Time • 3+ years exp • India
Python
Bash
AI/ML
Human-in-the-Loop
DevOps
Terraform
Puppet
Ansible
Chef
Datadog
Prometheus
CI/CD
Docker
Kubernetes
Grafana
SRE
Platform Engineering
Configuration Management
Self-Healing
Progressive Delivery
Incident Management
SLI/SLO/SLA
Linux
Apply
≈ $24k – $49k per year (Estimated) • In office • Full-Time • 7+ years exp • Bachelor's Degree • India
Python
Java
C#
C++
AI/ML
Edge AI
DevOps
GCP
Azure
AWS
Management
Agile
Apply
≈ $10k – $23k per year (Estimated) • In office • Full-Time • 3+ years exp • India
Apply
See all jobs
This is one of many
1,327,087 more open roles from verified company boards, updated every day.
Browse all jobs Get matched roles