{"id":3671695,"url":"https://alion.io/job/onit-data-engineer-databricks","title":"Data Engineer (Databricks)","company":{"id":58716,"name":"Onit","domain":"onit.com","url":"https://alion.io/company/onit","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Lever","truth_index":{"grade":"B","score":83,"open_postings":3,"ghost_share":0,"stale_share":0.667,"repost_share":0,"time_to_fill_p50_days":32,"computed_at":"2026-11-06T05:58:05Z"}},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Pune, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AWS","optional":false},{"name":"CI/CD","optional":false},{"name":"Claude","optional":false},{"name":"Copilot","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"Git","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-11-03T08:44:54Z","employer_posted_date":"2026-11-03","last_verified_at":"2026-11-06T18:14:28Z","board_verified":true,"closed_at":null,"days_open":3,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":3},"description":"Position Summary\nThe Data Engineer builds and operates the Databricks-based data pipelines that power a new customer-facing benchmarking capability in Onit's AI-driven legal operations platform. Integrating data from multiple source systems across enterprise legal management, matter management, and spend management, this role turns inconsistent, duplicated, and incomplete source data into a clean, reconciled, and trustworthy foundation for analytics. Working closely with the Data Architect, Data Scientist, and Product, the Data Engineer delivers reliable ingestion, entity resolution, and data quality measurement at scale, with multi-tenant data isolation and privacy built in.\nKey Responsibilities\nPipeline Development: Design, build, and maintain scalable batch and incremental pipelines in Databricks (PySpark, Spark SQL, Delta Lake) that ingest data from multiple source systems into a medallion (bronze/silver/gold) architecture.\nData Integration & Modelling: Implement the canonical data model defined with the Data Architect, so figures reconcile consistently across every product surface.\nEntity Resolution: Build production pipelines that resolve duplicate and conflicting records across systems that were never designed to reconcile with one another.\nData Quality Engineering: Profile source data, implement automated data quality checks and expectations, and build monitoring and reporting so data quality stays continuously visible against defined targets.\nOrchestration & Reliability: Orchestrate and monitor jobs with Databricks Workflows, optimise Spark performance and cost, and troubleshoot pipeline failures.\nGovernance & Privacy: Apply Unity Catalog governance, multi-tenant data isolation, and anonymisation and aggregation rules for pooled cross-customer analytics.\nCollaboration: Partner with the Data Scientist to productionise comparability models and data quality metrics, and with Engineering to expose curated data to the product.\nRequired Skills\n5+ years in data engineering or analytics engineering, building production data pipelines in a cloud environment.\nHands-on Databricks experience: PySpark and Spark SQL, Delta Lake, Databricks Workflows/Jobs, and Unity Catalog.\nStrong SQL and Python, with solid data modelling skills (dimensional and/or canonical models) and experience with the lakehouse/medallion architecture.\nPractical experience with deduplication, record linkage, or entity resolution, and with diagnosing and remediating data quality problems at scale.\nExperience implementing automated data quality checks, pipeline monitoring, and Spark performance and cost optimisation.\nExperience with cloud platforms (AWS preferred), Git-based version control, and CI/CD for data pipelines.\nClear communication with data scientists, architects, and product stakeholders, including flagging data limitations early.\nWorking proficiency with AI tools (e.g., Claude, Databricks Assistant, GitHub Copilot) to accelerate pipeline development, debugging, and documentation, with the judgment to validate AI-generated code before production use.\nPreferred Qualifications\nDatabricks certification (e.g., Data Engineer Associate or Professional).\nExperience with streaming or incremental ingestion (e.g., Auto Loader, Structured Streaming, Delta Live Tables).\nExperience building data for benchmarks, scoring models, or analytics intended for external consumption.\nExperience with anonymisation, aggregation thresholds, or privacy-preserving analytics.\nBackground in enterprise B2B SaaS or the legal technology domain.","description_format":"text","description_chars":3551,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Cybersecurity","Document Management","Digital Signature"],"lifecycle":[{"event":"open","at":"2026-11-03T09:16:24Z"}],"visa":[],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":2,"expected_fill_days":32,"reasons":["conf:0","velocity","win:early"],"computed_at":"2026-11-06T05:58:05Z"},"pay":null,"html_url":"https://alion.io/job/onit-data-engineer-databricks","json_url":"https://alion.io/job/onit-data-engineer-databricks.json","meta":{"generated_at":"2026-11-06T22:01:56Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":3308,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-07T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":58716},"rest":"https://alion.io/mcp/rest/get_company?id=58716"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fonit-data-engineer-databricks"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fonit-data-engineer-databricks"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fonit-data-engineer-databricks"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/onit-data-engineer-databricks\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fonit-data-engineer-databricks"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3671695}},{"tool":"search_jobs","arguments":{"company":"onit.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3671695},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}