{"id":4603514,"url":"https://alion.io/job/persistent-systems-azure-databricks-technical-lead","title":"Azure Databricks Technical Lead","company":{"id":59479,"name":"Persistent Systems","domain":"persistent.com","url":"https://alion.io/company/persistent-systems","size_band":"5000+","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":"Zwayam","truth_index":{"grade":"A","score":87,"open_postings":133,"ghost_share":0.075,"stale_share":0.256,"repost_share":0.135,"time_to_fill_p50_days":45,"computed_at":"2026-11-19T09:08:57Z"}},"role":"Data Science","role_family":"Data Science","seniority":"lead","employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Pune, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":8,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"Azure","optional":false},{"name":"Azure Cosmos DB","optional":false},{"name":"Azure SQL Database","optional":false},{"name":"CI/CD","optional":false},{"name":"Databricks","optional":false},{"name":"ETL/ELT","optional":false},{"name":"Git","optional":false},{"name":"pySpark","optional":false},{"name":"Python","optional":false},{"name":"Spark","optional":false}],"status":"live","first_seen_at":"2026-11-03T15:32:31Z","employer_posted_date":"2026-11-19","last_verified_at":"2026-11-19T13:16:45Z","board_verified":true,"closed_at":null,"days_open":15,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":15},"description":"About Persistent\nWe are an AI-led, platform-driven Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what’s next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many industry-leading organizations across the world, including 18 of the Fortune 50 companies, 7 of the 10 top US banks, 9 of the 10 technology companies, and numerous innovators across the healthcare ecosystem.\nOur disruptor’s mindset, commitment to client success, and agility to thrive in the dynamic environment have enabled us to sustain our growth momentum. Persistent has been recognized across top industry platforms for innovation, leadership, and inclusion. We have delivered 26 sequential quarters of growth with $487.6M in Q2 FY27 revenue, up 7.8% Q-o-Q and 20.0% Y-o-Y. Our 29,500+ global team members, located in 21 countries, have been instrumental in helping the market leaders transform their industries. We received five awards at the ISG (Information Services Group) Provider Lens™ Awards 2026 and, for the second year in a row, were awarded the Growth Honor of the Year at the Everest Group Elevate 2026.\nAbout Position:\nWe are seeking an experienced Azure Databricks Technical Lead Data Engineering to drive the design, development, and delivery of enterprise-scale data platforms on Microsoft Azure. The ideal candidate will have deep expertise in Python, PySpark, Azure Databricks, Unity Catalog, Azure data services, and modern data engineering practices. In this role, you will lead a team of data engineers, architect scalable data solutions, establish engineering standards, and ensure delivery of secure, high-performance, and governed data platforms that enable business intelligence, analytics, and advanced data-driven decision-making.\nRole: Azure Databricks Technical Lead\nLocation: All Persistent Locations\nExperience: 8 to 12 Years\nJob Type: Full-Time Employment\nWhat You'll Do:\nDesign and develop scalable data processing, transformation, and integration solutions using Python, PySpark, and Azure Databricks.\nBuild, optimize, and maintain production-grade batch and real-time data pipelines on Azure Databricks.\nImplement enterprise data governance, data lineage, and access management solutions using Unity Catalog.\nDesign and develop integrations between Azure Databricks, Azure Data Lake Storage (ADLS Gen2), Azure SQL, and Cosmos DB.\nCreate metadata-driven frameworks and reusable engineering components to accelerate development and improve maintainability.\nDevelop reusable Python and PySpark libraries, utilities, and common data processing modules.\nImplement robust monitoring, logging, alerting, error handling, and data quality validation mechanisms.\nOptimize Spark workloads for performance, scalability, reliability, and cost efficiency.\nDefine and implement CI/CD pipelines for Databricks applications and data engineering workflows.\nEstablish secure authentication, authorization, and access control standards across Azure services.\nEnsure compliance with enterprise data governance, security, and regulatory requirements.\nConduct code reviews, define development standards, and promote engineering best practices.\nTroubleshoot and resolve complex production issues across data pipelines and platform components.\nLead technical discussions, architecture reviews, solution design workshops, and estimation activities.\nMentor and guide data engineers while fostering technical excellence and knowledge sharing.\nCollaborate with architects, DevOps teams, application teams, analysts, and business stakeholders to deliver strategic data initiatives.\nExpertise You'll Bring:\n8 to 12 years of experience in Data Engineering, Big Data, Cloud Data Platforms, or Analytics Engineering.\nStrong expertise in Python programming and advanced PySpark development.\nExtensive hands-on experience with Azure Databricks and Apache Spark ecosystem.\nStrong knowledge of Unity Catalog, data governance frameworks, and data security best practices.\nExperience implementing solutions using Azure Data Lake Storage (ADLS Gen2), Azure SQL Database, and Azure Cosmos DB.\nProven experience designing metadata-driven data engineering frameworks and reusable platform components.\nStrong understanding of data architecture, data modeling, ETL/ELT methodologies, and modern data platform design principles.\nExperience developing and supporting real-time and streaming data pipelines.\nHands-on experience with CI/CD implementation for Databricks and cloud-native data platforms.\nStrong understanding of Azure authentication mechanisms, role-based access control (RBAC), managed identities, and security best practices.\nExperience with source control systems and version management practices, including SourceAnywhere and Git-based workflows.\nExpertise in performance tuning, troubleshooting, resource optimization, and platform reliability engineering.\nStrong understanding of Data Governance, Data Quality, Metadata Management, and Data Lineage concepts.\nProven ability to lead technical teams and mentor engineers across multiple projects.\nExcellent analytical, problem-solving, stakeholder management, and communication skills.\nExperience working in Agile and enterprise-scale delivery environments.\nBenefits:\nCompetitive salary and benefits package\nCulture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications\nOpportunity to work with cutting-edge technologies\nEmployee engagement initiatives such as project parties, flexible work hours, and Long Service awards\nAnnual health check-ups\nInsurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents\nValues-Driven, People-Centric & Inclusive Work Environment:\nPersistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.\nWe support hybrid work and flexible hours to fit diverse lifestyles.\nOur office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.\nIf you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment\nLet's unleash your full potential at Persistent -persistent.com/careers\n“Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.”","description_format":"text","description_chars":6743,"description_truncated":false,"requirements":{"experience_years_min":8,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Flexible schedule","Growth opportunities","Hybrid work","Insurance coverage"],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","IT Outsourcing & Dedicated Teams","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-11-19T13:16:45Z"}],"visa":[],"liveness":{"score":77,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.857,"p_room":0.9,"age_days":15,"expected_fill_days":45,"reasons":["conf:2","win:mid","comp:brand"],"computed_at":"2026-11-19T15:27:50Z"},"pay":null,"html_url":"https://alion.io/job/persistent-systems-azure-databricks-technical-lead","json_url":"https://alion.io/job/persistent-systems-azure-databricks-technical-lead.json","meta":{"generated_at":"2026-11-19T15:27:50Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":25727,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-20T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":59479},"rest":"https://alion.io/mcp/rest/get_company?id=59479"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fpersistent-systems-azure-databricks-technical-lead"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fpersistent-systems-azure-databricks-technical-lead"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fpersistent-systems-azure-databricks-technical-lead"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/persistent-systems-azure-databricks-technical-lead\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fpersistent-systems-azure-databricks-technical-lead"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4603514}},{"tool":"search_jobs","arguments":{"company":"persistent.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4603514},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}