{"id":3094944,"url":"https://alion.io/job/exl-databricks-engineer","title":"Databricks Engineer","company":{"id":38016,"name":"EXL","domain":"exlservice.com","url":"https://alion.io/company/exl","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Oracle","truth_index":{"grade":"B","score":80,"open_postings":232,"ghost_share":0,"stale_share":0.991,"repost_share":0,"time_to_fill_p50_days":9,"computed_at":"2026-10-23T05:54:15Z"}},"role":"Data Science","role_family":"Data Science","seniority":"middle","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Chennai, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":13500,"max_usd":34000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":21},"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"A/B Testing","optional":false},{"name":"Amazon Kinesis","optional":false},{"name":"Amazon S3","optional":false},{"name":"Apache Kafka","optional":false},{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"Azure DevOps","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"dbt","optional":false},{"name":"Delta Lake","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Feature Store","optional":false},{"name":"Fine-tuning","optional":false},{"name":"Fivetran","optional":false},{"name":"GitHub Actions","optional":false},{"name":"GitLab CI","optional":false},{"name":"Great Expectations","optional":false},{"name":"LLM","optional":false},{"name":"MLFlow","optional":false},{"name":"Platform Engineering","optional":false},{"name":"pySpark","optional":false},{"name":"RAG","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Terraform","optional":false},{"name":"Python","optional":true}],"status":"live","first_seen_at":"2026-08-04T07:19:05Z","employer_posted_date":"2026-08-04","last_verified_at":"2026-10-23T11:45:53Z","board_verified":true,"closed_at":null,"days_open":80,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":80},"description":"We are looking for a skilled and passionate Databricks Engineer to design, build, and optimize enterprise-scale data lakehouse solutions on the Databricks platform. The successful candidate will be responsible for creating Databricks pipeline delivering Financial Crime platforms covering Anti-Money Laundering (AML), Know Your Customer (KYC), Customer Risk Assessment (CRA), Sanctions Screening, Transaction Monitoring, Fraud Detection, and Regulatory Reporting\n Databricks Platform Engineering\nDesign, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments.\nConfigure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage.\nOptimize cluster configurations - instance types, auto-scaling policies, spot/preemptible nodes - for cost and performance.\nImplement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager).\nManage Databricks jobs, workflows, and multi-task job orchestration with dependency management.\nDelta Lake & Lakehouse Architecture\nDesign and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM).\nBuild Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization.\nImplement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations.\nManage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing.\nDesign data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS).\nData Pipeline Development (PySpark / SQL)\nDevelop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake.\nBuild structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables.\nWrite optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and dynamic partition pruning.\nCreate reusable transformation libraries, utility frameworks, and pipeline templates for team productivity.\nImplement robust error handling, retry logic, and dead-letter queue patterns in production pipelines.\nMLflow & AI/ML Workloads\nSet up and manage MLflow tracking servers, experiment registries, and model lifecycle management on Databricks.\nSupport data scientists and ML engineers in deploying model training and inference workloads on Databricks clusters and GPU instances.\nBuild feature engineering pipelines using Databricks Feature Store for reusable, versioned ML features.\nEnable GenAI workloads - LLM fine-tuning, RAG pipeline development, and vector search (Databricks Vector Search / Mosaic AI).\nImplement MLOps practices: model versioning, A/B testing, model serving via Databricks Model Serving endpoints.\nCloud Integration & DevOps\nIntegrate Databricks with cloud-native services: Azure Data Lake Storage (ADLS).\nBuild and maintain CI/CD pipelines for Databricks notebooks and jobs using Azure DevOps, GitHub Actions, or GitLab CI.\nImplement Databricks Asset Bundles (DABs) or Terraform for infrastructure-as-code (IaC) deployment of Databricks resources.\nManage data ingestion using Auto Loader, COPY INTO, and partner integrations (Fivetran, dbt, Airbyte).\nMonitor pipeline health, cluster utilization, and costs using Databricks system tables and cloud cost management tools.\nGovernance, Security & Optimization\nImplement row-level security, column masking, and dynamic data views using Unity Catalog policies.\nEnsure data quality enforcement using Delta Live Tables expectations and Great Expectations integrations.\nConduct performance tuning - query plan analysis, caching strategies, Photon engine enablement.\nMaintain data cataloging, metadata management, and data lineage tracking within Unity Catalog.\nDocument architecture decisions, runbooks, and operational guides for Databricks workloads.\n Education\nBachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or related field.\nExperience\n4-6 years of total experience in data engineering or software engineering.\n2+ years of dedicated hands-on experience with the Databricks platform in production environments.\nStrong background in big data engineering, cloud data platforms, and distributed computing.","description_format":"text","description_chars":4407,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Business Process Outsourcing (BPO)","AI Consulting & Integration","Analytics & BI Consulting"],"lifecycle":[{"event":"open","at":"2026-10-22T15:52:12Z"}],"visa":[],"liveness":{"score":3,"band":"cold","label":"Long shot","p_open":1,"p_active":0.116,"p_room":0.28,"age_days":79,"expected_fill_days":9,"reasons":["conf:0","stale_co","velocity","win:tail","crowd:brand"],"computed_at":"2026-10-23T05:54:15Z"},"pay":null,"html_url":"https://alion.io/job/exl-databricks-engineer","json_url":"https://alion.io/job/exl-databricks-engineer.json","meta":{"generated_at":"2026-10-23T14:09:29Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":21171,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-24T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":38016},"rest":"https://alion.io/mcp/rest/get_company?id=38016"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fexl-databricks-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fexl-databricks-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fexl-databricks-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/exl-databricks-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fexl-databricks-engineer"}]}