Data Engineer - Databricks
First seen by Alion on Nov 12, 2026.
Key Responsibilities :
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Build and optimize ETL/ELT workflows for processing structured and unstructured data.
- Develop data transformation and processing logic using Python, PySpark, and SQL.
- Work with Delta Lake to manage reliable, scalable, and high-performance data storage.
- Implement data ingestion solutions from multiple sources, including databases, APIs, files, and cloud storage.
- Optimize Spark jobs, queries, and data pipelines for performance, scalability, and cost efficiency.
- Implement data quality checks, validation rules, and monitoring mechanisms.
- Collaborate with Data Architects, Data Scientists, Business Analysts, and other engineering teams to understand data requirements.
- Work with cloud-based data platforms and services to build integrated data solutions.
- Follow best practices for data security, governance, version control, and deployment.
- Troubleshoot data pipeline failures and ensure data availability, accuracy, and reliability.
Required Skills & Qualifications :
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- Hands-on experience with Azure Databricks or Databricks.
- Strong programming skills in Python and PySpark.
- Strong proficiency in SQL and experience working with relational databases.
- Good understanding of Apache Spark architecture and distributed data processing.
- Experience developing and managing ETL/ELT data pipelines.
- Hands-on experience with Delta Lake and Delta tables.
- Understanding of data warehousing concepts, data modelling, and data integration.
- Experience working with cloud platforms such as Microsoft Azure, AWS, or GCP.
- Familiarity with Git and CI/CD practices for data engineering workflows.
- Strong analytical, problem-solving, and debugging skills.
Skills
Data Engineering, Databricks, Apache Spark, Python, PySpark, SQL, Delta Lake, ETL, Data Warehousing, Cloud, Azure Databricks