Data Engineer
First seen by Alion on Sep 15, 2026.
Role & Responsibilities:
Data Engineering & Transformation:
- Design, develop, and maintain scalable data transformation pipelines using Python (with tools like PySpark, ADF) and SQL in Azure Databricks.
- Implement transformation logic to move data from Bronze to Silver/Gold layers following data engineering best practices.
- Apply strong data engineering principles to ensure data reliability, quality, performance, and reusability.
- Work with structured and semi-structured data at scale.
Databricks, Azure & Cloud ETL:
- Build and manage Databricks notebooks, jobs, Delta Lake tables, and orchestrated workflows.
- Hands-on experience with Cloud-based ETL platforms (Preferred: Microsoft Azure Databricks, Synapse, Azure Functions; otherwise AWS or Google Cloud).
- Optimize data pipelines for performance, scalability, and cost efficiency.
Python Applications, APIs & Automation:
- Design, develop, and maintain Python applications, scripts, and APIs for data processing and automation.
- Write production-grade Python code with strong focus on readability, maintainability, and testing.
- Leverage Python for orchestration, validation, and integration with downstream systems.
Collaboration With Data Science & Engineering Teams:
- Collaborate closely with Data Scientists and Data Analysts to understand data, analytical models, and consumption requirements.
- Enable and support advanced analytics and data science workflows by preparing high-quality feature datasets.
- Translate analytical needs into scalable data engineering solutions.
CI/CD, DevOps & Platform Engineering:
- Build and maintain automated CI/CD pipelines for data and Databricks workloads.
- Hands-on experience with DevOps tools and practices, including Git-based version control.
- Exposure to containerization and orchestration platforms such as Kubernetes / OpenShift.
- Ensure smooth promotion of code and pipelines across environments (Dev/Test/Prod).
Data Modeling & Querying:
- Design and implement robust data models optimized for analytics and reporting.
- Strong hands-on knowledge of SQL and exposure to KQL or other query languages.
- Apply best practices in data structures, indexing, and performance tuning.
UI / UX & Data Applications (Additional Advantage):
- Open to contributing to data-driven UI/UX components, dashboards, or lightweight data applications.
- Work with analytics and business teams to improve data usability and customer experience.
Preferred Candidate Profile:
Must-Have:
- Strong hands-on expertise in Python (with frameworks like PySpark).
- Solid foundation in Data Engineering principles and large-scale data processing.
- Experience with Azure Databricks and cloud-based ETL platforms.
- Strong knowledge of SQL and data querying techniques.
- Experience with CI/CD pipelines and DevOps practices.
- Experience in pipeline monitoring and alerting.
- Ability to design efficient, scalable solutions to complex data problems.
Good-to-Have:
- Experience with Azure Synapse, Azure Functions.
- Exposure to AWS or Google Cloud data platforms.
- Hands-on experience with OpenShift.
- Knowledge of data science concepts and workflows.
- Familiarity with analytics platforms, dashboards, and UI/UX considerations.
Skills
Python, Azure Databricks, PySpark, Delta Lake, Data Engineering, KQL, Synapse Analytics