2,456,198open jobs
150,074companies
314,181added this week
Browse all
Salary
≈ $16k – $35k per year (Estimated)
Location
In office (Pune, Chennai)
Seniority
Senior · 5+ years exp

First seen by Alion on Sep 17, 2026.

Overview
Company
Impact
Profile match
Optum is the health services and technology business of UnitedHealth Group, headquartered in Eden Prairie, Minnesota, combining care delivery, pharmacy services and health data, analytics and technology for patients, providers, employers and payers. Formed in April 2011, it operates through Optum Health, which runs primary, urgent and virtual care practices, Optum Insight, which offers data, revenue cycle and software services, and Optum Rx, a pharmacy benefit manager with home delivery. Hiring covers software, data and AI engineers, product managers and designers and security engineers in Minnesota, Boston and remote US roles, and technology teams in Hyderabad, Bengaluru, Noida and Gurgaon.

Senior Data Engineer - Databricks & AWS

Job Description :

We are looking for a Senior Data Engineer with 5+ years of experience in designing, developing, and operating scalable data platforms and production-grade data pipelines.

The ideal candidate will have strong hands-on expertise in Python, SQL, PySpark, Databricks, AWS, and modern data lakehouse technologies, along with exposure to healthcare data standards and privacy requirements.

The role involves building resilient data ingestion and processing pipelines, optimizing distributed workloads, implementing data quality and observability controls, and contributing to production-grade data engineering practices.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines and data processing solutions using Python, SQL, Apache Spark/PySpark, and Databricks.

- Build and operate production-grade pipelines using Databricks Auto Loader, Delta Lake, Workflows, Unity Catalog, Notebooks/Jobs, and SQL Warehouses.

- Optimize Spark and Databricks workloads, including cluster configuration, job performance, query optimization, and distributed processing.

- Develop cloud-native data solutions using AWS services, including S3, IAM, VPC, ECS/EKS or Lambda, Step Functions, EventBridge, CloudWatch, Secrets Manager, and KMS.

- Work with Apache Iceberg, including tables, catalogs, snapshots, schema and partition evolution, compaction, and interoperability with different query engines.

- Design resilient ingestion frameworks for structured and semi-structured data, with appropriate validation, data quality checks, error handling, monitoring, and replay mechanisms.

- Implement engineering best practices around Git, pull requests, CI/CD, Docker, Infrastructure as Code, automated testing, observability, and incident response.

- Collaborate with cross-functional teams to understand data requirements and translate them into scalable technical solutions.

- Ensure appropriate security, access controls, data governance, and privacy measures are incorporated into data engineering workflows.

- Contribute to the responsible adoption of AI-assisted coding tools, critically reviewing, testing, securing, and productionizing AI-generated code.

Required Skills & Experience :

- 5+ years of overall experience in Data Engineering.

- Advanced proficiency in Python and SQL.

- Strong hands-on experience with Apache Spark/PySpark, distributed processing, and performance tuning.

- Production experience with Databricks, including :

1. Auto Loader

2. Delta Lake

3. Databricks Workflows

4. Unity Catalog

5. Notebooks and Jobs

6. SQL Warehouses

7. Cluster and job optimization

- Strong hands-on experience with AWS, particularly :

1. Amazon S3

2. IAM

3. VPC

4. ECS/EKS or Lambda

5. Step Functions

6. EventBridge

7. CloudWatch

8. Secrets Manager

9. KMS

- Practical knowledge of Apache Iceberg, including catalogs, snapshots, schema/partition evolution, compaction, and query-engine interoperability.

- Experience with Git, CI/CD, Docker, Terraform/CloudFormation, automated testing, observability, and production incident management.

- Experience working with semi-structured data and building reliable ingestion pipelines.

- Strong understanding of data quality, validation, error handling, monitoring, and replay/reprocessing mechanisms.

Healthcare Data Experience :

- Experience with healthcare data and standards is highly desirable, including :

1. FHIR R4 and/or HL7 v2

2. OMOP Common Data Model (OMOP CDM)

3. Clinical terminologies and healthcare data standards

4. Handling of PHI (Protected Health Information)

5. HIPAA-aligned engineering and security controls

6. Data privacy and de-identification concepts

AI & Engineering Practices :

- Demonstrated responsible use of AI coding assistants/tools in software and data engineering workflows.

- Ability to critically review AI-generated code for correctness, security, performance, maintainability, and production readiness.

- Strong focus on testing, documentation, code quality, and engineering best practices.

Education :

- UG : Any Graduate

Employment Type :

- Full Time, Permanent

Department :

- Data Science & Analytics

Skills

Data Engineering, AWS, Databricks, Data Pipeline, SQL, Python, PySpark, Apache Spark, Data Warehousing, AWS Lambda

Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
2,456,198 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

Data Science
Similar stack
Same company
Pune
In office · Full-Time · 6+ years exp · Bachelor's Degree · Bengaluru
Python
SQL
Python
pySpark
Databases
Apache Iceberg
Delta Lake
Apache Kafka
Google BigQuery
Apache Hudi
AI/ML
Spark
Airflow
Embeddings
Flink
Anomaly Detection
Machine Learning
Analytics
ETL/ELT
Apply
$8k – $9.6k per year · In office · Full-Time · Ulaanbaatar
Analytics
Power BI
Microsoft Excel
Apply
In office · 4+ years exp · Bengaluru
SQL
Databases
Snowflake
MS SQL
Analytics
ETL/ELT
Apply
Hybrid · Full-Time · São Paulo
SQL
AI/ML
Hadoop
Spark
Airflow
DevOps
Terraform
GCP
Analytics
Tableau
Power BI
Apply
$104k – $166k per year · In office · Secret · 8+ years exp · High School Diploma · Albuquerque
Python
SQL
PowerShell
Databases
PostgreSQL
Oracle
MS SQL
Apply
In office · Pune
Python
Scala
Databases
DynamoDB
Apache Kafka
Amazon Redshift
Amazon Aurora
AI/ML
Spark
Airflow
dbt
DevOps
AWS
AWS Lambda
Amazon Kinesis
Analytics
AWS Glue
Apply
Hybrid · 5+ years exp · Pune
SQL
Databases
MS SQL
Management
Agile
Apply
In office · Pune
Design
Adobe Photoshop
AutoCAD
Apply
Business Analyst 5 hours ago
Hybrid · 1+ year exp · Pune
SQL
MedTech
CAPA
Apply
In office · 5+ years exp · Bachelor's Degree · Pune
Design
Sketch
Zeplin
Management
Agile
Apply
See all jobs
This is one of many
2,456,198 more open roles from verified company boards, updated every day.