{"id":4540820,"url":"https://alion.io/job/orbia-lead-data-ai-engineer-databricks-analytics","title":"Lead Data & AI Engineer - Databricks Analytics","company":{"id":1938642,"name":"Orbia","domain":"orbia.com","url":"https://alion.io/company/orbia-com","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Teamtailor","truth_index":{"grade":"B","score":75,"open_postings":48,"ghost_share":0.5,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":18,"computed_at":"2026-11-18T05:48:20Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"lead","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Pune, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":12,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Azure","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"Azure DevOps","optional":false},{"name":"CI/CD","optional":false},{"name":"Copilot Studio","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"DLP","optional":false},{"name":"Embeddings","optional":false},{"name":"FinOps","optional":false},{"name":"GitHub Actions","optional":false},{"name":"Hallucination","optional":false},{"name":"Least Privilege","optional":false},{"name":"LLM Guardrails","optional":false},{"name":"Microsoft Entra ID","optional":false},{"name":"MLFlow","optional":false},{"name":"OpenAI","optional":false},{"name":"Power BI","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Rest API","optional":false},{"name":"SAP BTP","optional":false},{"name":"ServiceNow","optional":false},{"name":"SharePoint","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false},{"name":"ABAP","optional":true}],"status":"live","first_seen_at":"2026-11-18T15:35:00Z","employer_posted_date":null,"last_verified_at":"2026-11-18T15:35:00Z","board_verified":false,"closed_at":null,"days_open":0,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":0},"description":"Key Responsibilities:\nMain Purpose: The Lead Data Engineer - Databricks Data Analytics & Agent Bricks will be responsible for designing, building, governing, and operating enterprise-grade data engineering and AI-agent solutions on the Databricks Lakehouse Platform (Delta Lake, Unity Catalog, Delta Live Tables, Workflows, SQL Warehouses, MLflow, Model Serving) and Databricks Agent Bricks / Mosaic AI Agent Framework within Orbia's global Data, Analytics & AI landscape.\nThe role exists to ensure that data products, AI/BI dashboards, Genie spaces, RAG and Agent Bricks, and analytical workloads are designed using secure, scalable, compliant, cost-aware, and reusable engineering patterns aligned with Orbia's Lakehouse, governance, and Responsible AI standards.\nThe Lead Data Engineer will provide hands-on technical leadership to a team of data and analytics engineers, partner with business groups, platform owners, security, infrastructure, integration, governance, and external partners, and translate business and AI use cases into production-ready Databricks solutions.\n\nStrategy & Planning:\nDefine and govern Databricks Lakehouse and Agent Bricks architecture principles, design standards, and reusable reference patterns for enterprise data and AI-agent solutions.\nDesign end-to-end data engineering patterns covering ingestion, bronze/silver/gold medallion architecture, Delta Live Tables, streaming, batch, and data product publishing on Unity Catalog.\nDesign Agent Bricks / Mosaic AI Agent Framework patterns for analytical agents, RAG applications, Genie spaces, AI/BI dashboards, and business assistants grounded on governed Lakehouse data.\nTranslate approved business and AI use cases into solution design documents, data contracts, platform-fit decisions, integration patterns, and production-readiness plans.\nDefine model and tool selection guidance covering accuracy, latency, cost (DBU/token), privacy, security, data sensitivity, and business purpose.\nEstablish engineering guardrails for prompt engineering, RAG grounding, tool calling, human-in-the-loop approvals, write-back controls, and auditability for Agent Bricks solutions.\nSupport Orbia AI/data governance, ARB submissions, decision records, architecture exceptions, and platform standards for Databricks solutions.\nPartner with Data, Analytics & AI leadership to align Databricks engineering roadmap with Orbia's enablement strategy and scalable operating model.\n\nAcquisition & Deployment:\nEvaluate Databricks capabilities including Unity Catalog, Delta Live Tables, Workflows, SQL Warehouses, MLflow, Model Serving, Vector Search, Genie, AI/BI Dashboards, and Agent Bricks for enterprise fit.\nCollaborate with cloud, infrastructure, network, security, integration, and AI architecture teams to design DEV / QA / PROD-ready deployment patterns using Databricks Asset Bundles and CI/CD.\nDefine secure ingestion and integration patterns from SAP, SharePoint, ServiceNow, Coupa, Salesforce, REST APIs, file shares, and event sources using Azure Data Factory, Auto Loader, and partner connectors.\nDesign API-led integration patterns using Azure API Management, Entra ID, OAuth, managed identity, service principals, and approved enterprise gateways for Databricks-hosted agents and endpoints.\nLead technical design reviews for data pipelines, Unity Catalog models, agent tools, vector indexes, model endpoints, evaluation workflows, monitoring, and release gates.\nBuild and review hands-on production-grade code in PySpark, Python, and SQL; enforce coding standards, peer reviews, unit/integration testing, and automated quality checks.\nSupport partner and vendor technical reviews to validate delivery approach, platform compatibility, security posture, and architecture compliance.\n\nOperational Management:\nDefine operational runbooks for Databricks workspaces, Unity Catalog, jobs, DLT pipelines, Agent Bricks lifecycle, prompt/version control, evaluations, monitoring, telemetry, and incident support.\nEnsure Databricks solutions follow least-privilege RBAC/ABAC, access reviews, environment segregation, audit logging, data protection, lineage, and Responsible AI controls.\nProvide technical leadership, mentoring, and guidance to data engineers, analytics engineers, platform admins, business analysts, and implementation partners on approved design and implementation patterns.\nMonitor and optimize solution design for DBU consumption, storage, SQL Warehouse usage, vector search cost, model endpoint usage, latency, quality, security, and scalability.\nReview delivered solutions against engineering principles, Orbia governance expectations, SOX/ITGC controls, and production acceptance criteria before go-live.\nContinuously improve Databricks engineering standards based on lessons learned, platform roadmap changes, business adoption, and operational feedback.\nLead and mentor a small team of data/analytics engineers; perform code reviews, sprint planning, capacity planning, performance feedback, and skill development.\n\nComplexity of the Job:\nComplexity of enterprise data and AI engineering across Databricks Lakehouse, Unity Catalog, Delta Live Tables, MLflow, Vector Search, Agent Bricks / Mosaic AI Agent Framework, Azure Data Factory, ADLS Gen2, APIM, Entra ID, Purview, SharePoint, SAP, ServiceNow, and Power BI.\nComplexity of selecting the right delivery pattern across Databricks-led, Foundry-led, Copilot Studio, Power Platform, and hybrid Foundry + Databricks solutions.\nComplexity of safely grounding analytical and agent answers using structured and unstructured enterprise data while maintaining data protection, lineage, citations, and auditability.\nComplexity of GenAI and agent risks including hallucination, prompt misuse, sensitive data exposure, model and tool selection, cost escalation, and business process impact.\nComplexity of supporting multiple business groups with diverse data sources, processes, and regional requirements.\nComplexity of cost and performance optimization across DBU, storage, SQL Warehouses, model serving, and vector search while meeting SLAs and FinOps guardrails.\nCollaboration with multiple global teams / departments for technical alignment and decision-making, such as Infrastructure, Network, Security, Enterprise Architecture, AI Architecture, Integration, Privacy, Compliance, and business groups.\nRequirement to balance speed of data and AI experimentation with production readiness, lifecycle management, Responsible AI, FinOps, and enterprise governance (including SOX / ITGC).\n\nEducation, Experience, Language, & Physical Requirements:\nMinimum 12-14 years of hands-on experience on the Databricks Lakehouse Platform including Apache Spark, PySpark, Delta Lake, Unity Catalog, Delta Live Tables (DLT), Workflows, SQL Warehouses, MLflow, and Model Serving.\nMinimum 2 years of hands-on or strong architecture experience with Databricks Agent Bricks / Mosaic AI Agent Framework, Vector Search, Genie spaces, AI/BI Dashboards, and RAG-based agentic applications grounded on Lakehouse data.\nStrong programming skills in Python, PySpark, and advanced SQL; deep understanding of distributed computing, performance tuning, partitioning, Z-Ordering, file optimization, and Delta best practices.\nExperience with CI/CD for Databricks using Databricks Asset Bundles, Azure DevOps and/or GitHub Actions, infrastructure-as-code (Terraform), and environment promotion patterns across DEV / QA / PROD.\nExperience with the Azure data ecosystem including ADLS Gen2, Azure Data Factory, Azure Key Vault, Entra ID, Microsoft Purview, Azure Monitor, Azure API Management, and Azure OpenAI services.\nExperience designing and operating ingestion pipelines from SAP (ECC/S4 HANA, BW, BTP), SharePoint, ServiceNow, Coupa, Salesforce, REST APIs, files, and streaming sources using Auto Loader, ADF, and partner connectors.\nExperience designing RAG and agent solutions using SharePoint, enterprise documents, semantic indexes, embeddings, vector search, citations, chunking, grounding evidence, and answer evaluation.\nExperience designing secure enterprise integrations using REST APIs, OAuth 2.0, managed identities, service principals, APIM, event-driven patterns, SAP BTP, ServiceNow APIs, or Microsoft Graph APIs.\nStrong understanding of governance controls including Unity Catalog (catalogs/schemas/tables, row/column-level security, ABAC), RBAC, least privilege, segregation of duties, access reviews, private endpoints, audit logging, DLP, Purview, data classification, lineage, and SOX / ITGC expectations.\nExperience in AI/data governance, Responsible AI, model and agent evaluation, prompt engineering, hallucination control, content safety, human approval gates, and production release criteria.\nDemonstrated experience leading a team of data engineers / analytics engineers including hands-on technical leadership, mentoring, code reviews, sprint planning, and performance development.\nExperience preparing architecture review documents, solution and low-level design documents, technical standards, operating procedures, decision records, and ARB-ready materials.\nStrong understanding of cost management for Databricks and AI solutions including DBU usage, SQL Warehouse sizing, storage, vector search cost, model endpoints, monitoring, and FinOps guardrails.\nExperience working in an international environment at a global company with cross-functional teams and business stakeholders across multiple regions.\nExcellent communication skills in English, both verbally and in writing.\nCapabilities to build trustworthy relationships with business users, platform teams, security, architecture, vendors, and implementation partners.\nSelf-starter with the ability to provide technical leadership, mobilize business partners, analyze requirements, challenge solution assumptions, push back when required, and remain result-oriented.\nPhysical requirements for the position such as mobility, vision, strength, etc. (applicable in line with local laws).\nSkills\nApache Spark, Databricks, Delta Lake, MLflow, Mosaic AI, PySpark, RAG, Unity Catalog, Vector Search","description_format":"text","description_chars":10088,"description_truncated":false,"requirements":{"experience_years_min":12,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Plastics & Polymers","Chemical Manufacturing"],"lifecycle":[{"event":"open","at":"2026-11-18T15:35:00Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":18,"reasons":["seen:0","win:early"],"computed_at":"2026-11-18T19:23:38Z"},"pay":null,"html_url":"https://alion.io/job/orbia-lead-data-ai-engineer-databricks-analytics","json_url":"https://alion.io/job/orbia-lead-data-ai-engineer-databricks-analytics.json","meta":{"generated_at":"2026-11-18T19:23:38Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":664,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-19T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1938642},"rest":"https://alion.io/mcp/rest/get_company?id=1938642"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Forbia-lead-data-ai-engineer-databricks-analytics"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Forbia-lead-data-ai-engineer-databricks-analytics"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Forbia-lead-data-ai-engineer-databricks-analytics"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/orbia-lead-data-ai-engineer-databricks-analytics\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Forbia-lead-data-ai-engineer-databricks-analytics"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4540820}},{"tool":"search_jobs","arguments":{"company":"orbia.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4540820},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}