Senior Databricks AI/ML Engineer
First seen by Alion on Jul 31, 2026.
We are looking for an experienced Senior Databricks AI/ML Engineer with expertise in Databricks, Apache Spark, Machine Learning, and Generative AI. The ideal candidate should have hands-on experience building scalable AI/ML solutions, modern data platforms, and advanced analytics pipelines on the Databricks Lakehouse platform. Experience with Teradata, MLflow, and LLM integration is highly desirable.
Key Responsibilities:
- Design, develop, and optimize AI/ML solutions using the Databricks Lakehouse Platform.
- Build scalable data pipelines using Apache Spark (PySpark/Spark SQL) and Delta Lake.
- Develop and deploy machine learning models using MLflow and Databricks Machine Learning.
- Build and integrate LLM-based and Generative AI solutions for enterprise use cases.
- Perform feature engineering, model training, evaluation, and deployment for predictive analytics and time-series use cases.
- Design and optimize SQL-based analytics and reporting solutions.
- Support Teradata integration, migration, and performance optimization where required.
- Monitor and optimize cluster performance, workloads, and resource utilization on Databricks.
- Develop reusable Python libraries and automation scripts for data engineering and ML workflows.
- Collaborate with Data Scientists, Data Engineers, Product teams, and business stakeholders to deliver scalable AI-driven solutions.
- Implement CI/CD, model governance, and MLOps best practices.
- Prepare technical documentation and ensure compliance with security and data governance standards.
Required Skills:
- 8-13 years of experience in Data Engineering, Machine Learning, or AI Engineering.
- Strong hands-on expertise with the Databricks Platform (Workspaces, Clusters, Jobs, Unity Catalog, Databricks SQL).
- Extensive experience with Apache Spark (PySpark/Spark SQL) and Delta Lake.
- Strong programming skills in Python, including Pandas, NumPy, and data visualization libraries.
- Advanced proficiency in SQL and large-scale data processing.
- Experience with MLflow, model lifecycle management, and MLOps practices.
- Experience integrating Large Language Models (LLMs) and Generative AI into enterprise applications.
- Strong knowledge of feature engineering, predictive modeling, and time-series analytics.
- Experience with Teradata Administration, DBQL, System Views, and performance tuning.
- Strong analytical, problem-solving, and stakeholder management skills.
Skills
Azure Databricks, Artificial Intelligence, Machine Learning, DataLake, PySpark, LLM, Generative AI, SQL, Teradata, Python