Senior Data Engineer
First seen by Alion on Sep 11, 2026.
Hiring Senior Data Engineer - Databricks | PySpark | Spark | Delta Lake | AWS/Azure
Total Experience : 7-9 years
About the Role :
We are looking for a hands-on Senior Data Engineer / Databricks Engineer who can build scalable, reliable and high-performance data pipelines using Databricks, Apache Spark, PySpark, Python and SQL.
You will play a key role in designing and delivering modern Lakehouse, ETL/ELT and data engineering solutions, transforming business requirements into production-ready pipelines, Delta Lake tables and reusable data components.
This is an opportunity for a strong Data Engineer who enjoys solving complex data problems, optimizing Spark workloads and building production-grade data platforms across AWS or Azure.
Key Responsibilities :
- Design, develop and maintain scalable ETL/ELT data pipelines using Databricks, Spark and PySpark.
- Build and manage production-grade Delta Lake tables and Lakehouse data solutions.
- Translate business and technical requirements into robust, reusable and maintainable data engineering solutions.
- Develop high-quality Python, SQL and PySpark code for large-scale data processing.
- Optimize Apache Spark / PySpark jobs for performance, scalability and cloud cost efficiency.
- Implement Databricks Workflows for reliable data pipeline orchestration and scheduling.
- Apply Unity Catalog best practices for data governance, security and access management.
- Develop reusable data engineering frameworks, reference patterns and components.
- Work with Databricks SQL and modern Lakehouse architecture.
- Support AI/ML data pipelines, model development workflows and production data requirements.
- Integrate data from multiple sources, including APIs and cloud-based systems.
- Collaborate with Data Scientists, ML Engineers, Software Engineers and business stakeholders.
- Troubleshoot pipeline failures, data quality issues and performance bottlenecks.
- Follow engineering best practices for testing, documentation, version control and deployment.
- Contribute to DevOps / MLOps practices and infrastructure automation where required.
Required Skills & Qualifications :
- Bachelor's degree in Computer Science, Engineering, Information Technology or a related field.
- 7+ years of experience in Software Engineering, Data Engineering or a related technical discipline.
- Strong programming skills in Python, SQL, and PySpark.
- Strong understanding of ETL / ELT pipeline development.
- Hands-on experience with Delta Lake and Databricks Lakehouse architecture.
- Good knowledge of AWS or Microsoft Azure cloud platforms.
- Strong understanding of Data Warehousing and Lakehouse architecture.
- Experience building scalable data pipelines and processing large datasets.
- Strong understanding of data engineering concepts, data modeling and pipeline optimization.
Preferred Skills :
- Experience with Databricks Workflows, Databricks SQL, and dbt (Data Build Tool).
- Experience supporting AI/ML pipelines and MLOps / DevOps practices.
- Experience with Terraform or AWS CloudFormation.
- Strong understanding of REST APIs / API-based data ingestion.
- Experience with CI/CD and automated deployment pipelines.
- Knowledge of cloud-native data engineering and modern data platforms.
Skills
PySpark, Databricks, Data Engineering, AWS, ETL, Data Pipeline, DataLake, Python, SQL, Data Warehousing