{"id":4600794,"url":"https://alion.io/job/infobeans-senior-data-engineer-azure-databricks","title":"Senior Data Engineer - Azure Databricks","company":{"id":59407,"name":"InfoBeans","domain":"infobeans.ai","url":"https://alion.io/company/infobeans-technologies","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"SmartRecruiters","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Indore, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":8,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Azure","optional":false},{"name":"Azure Data Factory","optional":false},{"name":"Azure DevOps","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"Delta Lake","optional":false},{"name":"Docker","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Git","optional":false},{"name":"PostgreSQL","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-11-19T11:31:04Z","employer_posted_date":null,"last_verified_at":"2026-11-19T11:31:04Z","board_verified":false,"closed_at":null,"days_open":0,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":0},"description":"Job Description:\n\nSenior Data Engineer Azure Databricks\n\nJob Overview:\n\nWe are looking for an experienced Senior Data Engineer with strong hands-on expertise in Azure Databricks, PySpark, Apache Spark, Delta Lake and Azure data services. The role will focus on designing and developing scalable data engineering solutions, building batch and streaming pipelines, implementing Lakehouse architectures, and optimizing Spark workloads for performance, reliability and maintainability.\n\nKey Responsibilities:\n\n- Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark and Spark SQL.\n\n- Build and manage Lakehouse solutions using Bronze, Silver and Gold / Medallion architecture.\n\n- Develop robust batch and streaming ETL/ELT pipelines for structured and semi-structured data.\n\n- Work extensively with Delta Lake, including MERGE/upserts, schema evolution, data quality and incremental processing.\n\n- Implement and manage data governance using Unity Catalog, including catalogs, schemas, permissions and access controls.\n\n- Develop and orchestrate Databricks notebooks, jobs and workflows for reliable data processing.\n\n- Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Synapse Analytics, Event Hubs and Service Bus.\n\n- Optimize Spark workloads using appropriate partitioning, caching, broadcast joins, shuffle optimization and other performance-tuning techniques.\n\n- Apply Spark fundamentals such as lazy evaluation, transformations/actions, joins, repartitioning and data-skew handling.\n\n- Design data models and curated datasets for analytics, reporting and downstream applications.\n\n- Implement data quality checks, schema validation, monitoring, error handling and SLA-driven pipeline delivery.\n\n- Develop solutions using Python and SQL and follow clean, modular and maintainable coding practices.\n\n- Use Git, Azure DevOps and CI/CD practices for source control, deployment and release management.\n\n- Collaborate with engineering, analytics and business teams to understand requirements and deliver reliable data products.\n\nRequired Technical Skills:\n\n- Programming: Python, SQL\n\n- Big Data: Apache Spark, PySpark, Spark SQL\n\n- Databricks: Azure Databricks, notebooks, jobs/workflows, performance optimization\n\n- Lakehouse: Delta Lake, Medallion architecture, incremental processing, MERGE, schema evolution\n\n- Governance: Unity Catalog, access control, data governance\n\n- Azure Data: ADLS Gen2, Azure Data Factory, Azure Synapse Analytics\n\n- Messaging / Integration: Azure Event Hubs, Azure Service Bus\n\n- DevOps: Git, Azure DevOps, CI/CD, Docker\n\n- Databases: SQL Server, PostgreSQL, Oracle, Azure SQL\n\nQualifications & Experience:\n\n- 8+ years of overall experience in software/data engineering.\n\n- Strong hands-on experience in Azure Databricks and PySpark-based data engineering.\n\n- Strong Python and SQL development skills.\n\n- Demonstrated experience designing and supporting production-grade data pipelines.\n\n- Strong analytical, troubleshooting and problem-solving skills.\n\n- Good communication and ability to explain technical approaches clearly.\n\nSkills\nData Engineering, Azure Databricks, PySpark, Python, SQL, Delta Lake, Azure Data Factory, Apache Spark, Azure, Synapse Analytics, ETL","description_format":"text","description_chars":3257,"description_truncated":false,"requirements":{"experience_years_min":8,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Outsourcing & Dedicated Teams","Custom Software Development","AI Consulting & Integration","Enterprise Apps Implementation Partners"],"lifecycle":[{"event":"open","at":"2026-11-19T12:31:25Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":45,"reasons":["seen:0","win:early"],"computed_at":"2026-11-19T14:53:59Z"},"pay":null,"html_url":"https://alion.io/job/infobeans-senior-data-engineer-azure-databricks","json_url":"https://alion.io/job/infobeans-senior-data-engineer-azure-databricks.json","meta":{"generated_at":"2026-11-19T14:53:59Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":24686,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-20T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":59407},"rest":"https://alion.io/mcp/rest/get_company?id=59407"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Finfobeans-senior-data-engineer-azure-databricks"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Finfobeans-senior-data-engineer-azure-databricks"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Finfobeans-senior-data-engineer-azure-databricks"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/infobeans-senior-data-engineer-azure-databricks\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Finfobeans-senior-data-engineer-azure-databricks"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4600794}},{"tool":"search_jobs","arguments":{"company":"infobeans.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4600794},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}