Senior Databricks & AI Architect
First seen by Alion on Nov 13, 2026. Allianz scores A on the Alion truth index.
Position Overview :
We are seeking a visionary Senior Databricks & AI Architect to lead the evolution of our modern data platform and AI roadmap.
In this role, you will bridge the gap between complex insurance data domains and cutting-edge AI execution.
You will be responsible for designing scalable Spark-based architectures, implementing rigorous FinOps and optimisation strategies, and pioneering Agentic AI workflows to automate high-value insurance processes (e.g., claims, underwriting, and risk assessment).
Key Responsibilities :
- Design and govern end-to-end data architectures on the Databricks Data Intelligence Platform, ensuring seamless integration with existing insurance ecosystems.
- Lead the design of emerging AI workflows, specifically focusing on Agentic AI (multi-agent systems), LLM orchestration (LangChain/LlamaIndex), and RAG patterns tailored for insurance policy and regulatory documents.
- Own the platform's efficiency, implementing cluster policies, Photon engine optimisations, and cost-attribution models to ensure high performance with a focus on ROI.
- Standardise data engineering templates (Medallion architecture) and implement robust data modelling (Star/Snowflake, Vault 2.0) to handle structured and unstructured insurance data.
- Translate complex insurance concepts (Premiums, Claims, Actuarial models) into scalable technical schemas and AI logic.
- Ensure all designs comply with industry regulations using Databricks Unity Catalog.
Required Skills & Experience :
- 8+ years in Data Engineering/Architecture, with at least 3+ years of hands-on leadership within the Databricks ecosystem.
- Expert-level knowledge of the Spark framework (PySpark/Scala), Delta Lake, and Databricks SQL.
- Proven experience building and deploying AI solutions, with a deep understanding of the Agentic AI landscape.
- Specific experience in Databricks cost management, instance selection, and performance tuning (Z-Ordering, Liquid Clustering).
- Strong background in data modelling and the creation of reusable ETL/ELT templates and frameworks.
- Prior experience in the Insurance or FinServ sector is highly preferred.
Education & Certifications :
- Bachelor's or Master's degree in Computer Science, Data Science, or a related field.
- Preferred: Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.
Skills
Data Engineering, Databricks, AI Architecture, Artificial Intelligence, RAG, Data Modeling, Spark, SQL, Agentic AI