Lead Databricks Data Engineer
Confirmed on the employer's own hiring board on Oct 18, 2026. First seen by Alion on Oct 7, 2026. Persistent Systems scores B on the Alion truth index.
About Persistent
We are an AI-led, platform-driven Digital Engineering and Enterprise Modernization partner, combining deep technical expertise and industry experience to help our clients anticipate what’s next. Our offerings and proven solutions create a unique competitive advantage for our clients by giving them the power to see beyond and rise above. We work with many industry-leading organizations across the world, including 20 Fortune 50 companies and 4 of the 5 top banks in both the US and India, and numerous innovators across the healthcare ecosystem.
Our disruptor’s mindset, commitment to client success, and agility to thrive in the dynamic environment have enabled us to sustain our growth momentum. Persistent has been recognized across top industry platforms for innovation, leadership, and inclusion. We reported $1,654.4M FY26 revenue with 17.4% Y-o-Y growth. We have delivered 24 sequential quarters of growth with $436.0M in Q4 FY26 revenue, up 3.2% Q-o-Q and 16.2% Y-o-Y growth. Our 27,500+ global team members, located in 18 countries, have been instrumental in helping the market leaders transform their industries. We have been recognized as the Fastest Growing IT Services Brand Globally in the 2026 Brand Finance IT Services 25 Report. We named a Leader in the Everest Group Private Equity (PE) Services PEAK Matrix® Assessment 2026 and Software Product Engineering PEAK Matrix® Assessment 2026.
About Position:
We are looking for a skilled Lead Databricks Data Engineer with strong expertise in Data Engineering, Big Data technologies, and the Databricks Lakehouse Platform. The ideal candidate will be responsible for designing, developing, and supporting scalable data pipelines, enterprise analytics solutions, and cloud-native data platforms. The role requires deep hands-on experience in Databricks, PySpark, SQL, ETL/ELT frameworks, and cloud technologies. You will work closely with architects, data scientists, business stakeholders, and engineering teams to deliver high-performance, scalable, and reliable data solutions that enable enterprise-wide analytics and data-driven decision-making.
- Role: Lead Databricks Data Engineer
- Location: Bengaluru, Pune
- Experience: 8 to 12 Years
- Job Type: Full Time Employment
What You'll Do:
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks.
- Build and optimize large-scale data processing workflows using PySpark and Spark SQL.
- Develop data ingestion frameworks for structured, semi-structured, and unstructured data sources.
- Implement batch and real-time data processing solutions on the Databricks Lakehouse Platform.
- Design and support Delta Lake implementations and Lakehouse architectures.
- Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
- Develop data transformation, validation, cleansing, and reconciliation frameworks.
- Support enterprise data modeling and data warehousing initiatives.
- Implement monitoring, alerting, and troubleshooting mechanisms for data pipelines.
- Integrate Databricks with cloud services, APIs, enterprise applications, and third-party systems.
- Develop cloud-native data solutions across Azure, AWS, or Google Cloud environments.
- Support CI/CD implementation and deployment automation for data engineering workloads.
- Collaborate with Architects, Business Analysts, Data Scientists, and Product Owners to deliver business-driven data solutions.
- Participate in solution design discussions, architecture reviews, and code review processes.
- Support production deployments, incident resolution, and performance tuning activities.
- Create technical documentation, design specifications, and operational runbooks.
- Mentor junior engineers and promote data engineering best practices across teams.
Expertise You'll Bring:
- 8-12 years of experience in Data Engineering, Big Data, and Analytics platform development.
- Strong hands-on expertise with Databricks Lakehouse Platform and enterprise-scale data engineering solutions.
- Advanced proficiency in PySpark, Spark SQL, Python, and SQL.
- Extensive experience designing and developing scalable ETL/ELT pipelines and data integration frameworks.
- Deep understanding of Delta Lake architecture, transaction management, and performance optimization techniques.
- Strong knowledge of Data Warehousing concepts, dimensional modeling, and enterprise analytics platforms.
- Experience working with structured, semi-structured, and unstructured data at scale.
- Hands-on experience building batch and real-time data processing solutions.
- Strong expertise in data ingestion, transformation, validation, cleansing, and reconciliation processes.
- Experience integrating enterprise applications, APIs, cloud services, and diverse data sources.
- Familiarity with Azure Data Factory, AWS Glue, or equivalent cloud-native data integration services.
- Experience working with Azure, AWS, or Google Cloud Platform environments.
- Strong understanding of Git, source control practices, and collaborative development methodologies.
- Experience implementing and managing CI/CD pipelines for data engineering workloads.
- Knowledge of Kafka, Event Hub, or similar streaming technologies for real-time data processing.
- Familiarity with Unity Catalog, metadata management, and enterprise data governance frameworks.
- Strong understanding of schema design, data modeling, and query optimization techniques.
- Experience using Azure DevOps and deployment automation frameworks.
- Exposure to AI/ML data pipelines, feature engineering, and model data preparation workflows.
- Strong troubleshooting, production support, and root cause analysis capabilities.
- Experience optimizing Databricks workloads for enterprise-scale performance and reliability.
- Ability to independently lead medium-to-large data engineering initiatives.
- Experience mentoring team members and conducting technical reviews.
- Excellent analytical, communication, and stakeholder management skills.
- Strong commitment to code quality, testing, engineering best practices, and continuous improvement.
- Ability to contribute to architecture discussions and strategic data platform initiatives.
Benefits:
- Competitive salary and benefits package
- Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
- Opportunity to work with cutting-edge technologies
- Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
- Annual health check-ups
- Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents
Values-Driven, People-Centric & Inclusive Work Environment:
Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.
- We support hybrid work and flexible hours to fit diverse lifestyles.
- Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
- If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment
Let’s unleash your full potential at Persistent -persistent.com/careers
“Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.”