{"id":4535565,"url":"https://alion.io/job/ust-lead-i-data-engineering-15","title":"Lead I - Data Engineering","company":{"id":1774626,"name":"UST","domain":"ust.com","url":"https://alion.io/company/ust-com","size_band":"5000+","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"RippleHire","truth_index":{"grade":"B","score":76,"open_postings":106,"ghost_share":0,"stale_share":0.972,"repost_share":0,"time_to_fill_p50_days":33,"computed_at":"2026-11-18T05:48:20Z"}},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Pune, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Azure","optional":false},{"name":"Databricks","optional":false},{"name":"pySpark","optional":false},{"name":"Spark","optional":false},{"name":"Apache Kafka","optional":true},{"name":"CI/CD","optional":true},{"name":"dbt","optional":true},{"name":"Delta Lake","optional":true},{"name":"Git","optional":true},{"name":"Power BI","optional":true},{"name":"Pytest","optional":true},{"name":"Python","optional":true},{"name":"Scala","optional":true},{"name":"SQL","optional":true},{"name":"Terraform","optional":true}],"status":"live","first_seen_at":"2026-11-16T00:00:00Z","employer_posted_date":"2026-11-18","last_verified_at":"2026-11-18T13:53:59Z","board_verified":true,"closed_at":null,"days_open":2,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":2},"description":"Required skills for Senior Data Engineer (Azure Databricks)\nSpark and PySpark\n5+ years building batch and/or streaming pipelines with Spark\nDeep understanding of Spark fundamentals: lazy evaluation, transformations vs. actions, parti tions, shuffles, joins, and caching\n(Nice to have) Performance tuning: broadcast joins, AQE, skew mitigation, shuffle optimization\n(Nice to have) Structured Streaming and Auto Loader for incremental ingestion\nDatabricks platform on Azure\nUnity Catalog: catalogs/schemas/tables, permissions, external locations, storage credentials, lineage\nDatabricks Jobs/Workflows: task orchestration, cluster policies, retries, parameters\nGit integration (Repos), CI/CD, and environment promotion\n(Nice to have) Declarative pipelines with Delta Live Tables (DLT): expectations, CDC, quality rules, event logs\nCloud storage and access (Azure)\nSecret management via Databricks secret scopes and Key Vault integration\n(Nice to have) ADLS Gen2: ABFS access, ACLs vs. RBAC, mounts vs. direct access, OAuth/ service principals/managed identity\nDelta Lake\nACID transactions, time travel, MERGE/UPDATE/DELETE\nSchema enforcement and evolution\n(Nice to have) OPTIMIZE, ZORDER, VACUUM and retention policies\n(Nice to have) Compaction and smallfiles mitigation strategies\nSQL and data querying\nComplex SQL with CTEs and window functions\n(Nice to have) Query optimization and execution plan analysis in Databricks SQL\nData modeling and architecture\nMedallion (bronze/silver/gold) patterns for data lakes\nPartitioning and file layout to match read patterns\nDesigning tables for CDC and incremental processing\n(Nice to have) Lakehouse design tradeoffs and table constraints\nLargescale data processing\nCluster sizing, autoscaling, and cost optimization (spot, pools)\nObservability: Spark UI, metrics, event logs, Delta logs\n(Nice to have) Handling TBscale datasets, throughput vs. latency tradeoffs\nGovernance, security, and compliance\n(Nice to have) Privilege model in Unity Catalog; table, row, and columnlevel security patterns\n(Nice to have) PII handling, masking/anonymization, auditability, lineage\nSoftware engineering practices\nPython packaging, code quality, testing (pytest, chispa), code reviews\nCI/CD for Databricks (Repos, asset bundles/wheels), artifact versioning\nDocumentation and stakeholder communication\nNice to have (optional)\nScala, dbt, Kafka/Event Hubs, ADF/Airflow, Terraform, Power BI, data mesh concepts, SCD patterns","description_format":"text","description_chars":2447,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"phd","optional":true},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","IT Outsourcing & Dedicated Teams"],"lifecycle":[{"event":"open","at":"2026-11-18T13:53:59Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":2,"expected_fill_days":33,"reasons":["conf:1","win:early","comp:brand"],"computed_at":"2026-11-18T15:24:02Z"},"pay":null,"html_url":"https://alion.io/job/ust-lead-i-data-engineering-15","json_url":"https://alion.io/job/ust-lead-i-data-engineering-15.json","meta":{"generated_at":"2026-11-18T15:24:02Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":11996,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-19T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1774626},"rest":"https://alion.io/mcp/rest/get_company?id=1774626"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fust-lead-i-data-engineering-15"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fust-lead-i-data-engineering-15"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fust-lead-i-data-engineering-15"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/ust-lead-i-data-engineering-15\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fust-lead-i-data-engineering-15"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4535565}},{"tool":"search_jobs","arguments":{"company":"ust.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4535565},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}