Lead Data Engineer

The Lead Data Engineer will be responsible for architecting, building, and governing enterprise-scale data pipelines and platforms for Ujjivan Small Finance Bank. The role ensures secure, high-quality, reliable, and timely data availability to support analytics, regulatory reporting, risk management, and AI/ML initiatives.

This role provides technical and people leadership, defines data engineering standards, and acts as a key interface between business, analytics, governance, and technology teams.

Size of the role:

• 40+ internal and external data sources across core banking, lending, digital, and credit bureau systems
• 150+ automated batch and near-real-time data pipelines processing multi-TB data volumes monthly
• Supports 100+ enterprise dashboards, MIS, and regulatory (RBI/statutory) reports
• Data foundation for 25+ analytics and AI/ML use cases across risk, collections, and business
• Leads 7-12 data engineers and manages 3-5 technology vendors while serving 30+ stakeholders

Key duties & responsibilities of the role:

Business/ Financials:

• Design and own end-to-end data pipeline architecture across batch and near real-time processing aligned to enterprise strategy.
• Define and govern bronze, silver, and gold data layer architecture for enterprise consumption.
• Enable analytics, ML, and AI use cases by delivering model-ready and feature-ready datasets that drive business outcomes.
• Optimize data pipeline performance and cost efficiency.
• Establish CI/CD pipelines for data engineering, including version control, testing, and controlled deployments.
• Contribute to planning, budgeting, and prioritization of data engineering initiatives aligned to business goals.

Customer (Both Internal & External):

• Collaborate with business, analytics, and risk teams to translate requirements into scalable data solutions.
• Lead ingestion of data from Core Banking, LOS, LMS, Collections, CRM, Payments, Finance, and external data sources to support internal and external consumers.
• Enable timely, reliable, and high-quality data availability for stakeholders across the organization.
• Partner with Data Quality & Governance teams to operationalize Critical Data Elements (CDEs), lineage, and metadata for stakeholder trust and usability.

Internal Process:

• Ensure pipeline scalability, fault tolerance, restartability, and SLA adherence.
• Implement workflow orchestration, dependency management, backfills, and automated retries.
• Embed automated data quality checks, reconciliation controls, and anomaly detection.
• Ensure secure data handling, including masking, encryption, and role-based access control.
• Ensure compliance with regulatory, audit, and information security requirements.
• Comply with internal SLAs, policies, and standard operating procedures.
• Drive process management and continuous process excellence across data engineering workflows.

Innovation & Learning:

• Upskill team members to new age technologies with respect to machine learning and credit modelling. Enhance on-job self-learning related to analytics techniques/tools.
• Ensure timely completion of training / learning programs assigned from time to time.

Minimum requirements of knowledge & skills:

Educational Qualifications:

Bachelor's or Master's degree in Engineering, Computer Science, or related field.

Experience Range (Years and Core Experience Type):

• 12-15 years of experience in data engineering or large-scale data platform development.
• Proven experience in banking or financial services data environments.
• Demonstrated experience leading teams and enterprise data programs.

Certifications:

NA/ Good to have

Functional Skills:

• Advanced SQL and strong programming skills in Python / Scala and PySpark.
• Deep understanding of cloud architecture and DevOps.
• Strong experience with ETL/ELT frameworks and distributed data processing.
• Hands-on experience with data orchestration and scheduling frameworks.
• Deep understanding of data warehousing, data lakes, and layered data architectures.
• Expertise in data quality, reconciliation, metadata management, and data lineage.
• Strong knowledge of CI/CD, version control (Git), and automated testing for data pipelines.
• Experience with data security, masking, encryption, and role-based access control.
• Exposure to streaming or near real-time data processing is desirable.
• Understanding of ML/AI data requirements and feature engineering pipelines.

Behavioral Skills:

• Leadership skills to manage a team of data quality analysts and data engineers.
• Excellent listening, interpersonal, communication, and problem-solving skills.
• Demonstrated ability to work effectively in teams, in both a lead and support role.
• Effective time management skills, including demonstrated ability to manage and prioritize multiple tasks and projects.
• Ability to build strong relationships both internally and externally.
• Exceptionally strong organizational, problem-solving, and communication skills.
• Structured thinker, effective communicator with excellent written communication skills.

Competencies:

• A passion for constantly learning and applying new technologies and programming languages in a constantly evolving environment.
• Demonstrated ability to learn new techniques and troubleshoot code without support.
• Prior people management experience, with tried-and-true approaches for mentoring junior staff.
• Understanding of the financial sector, to recognize and drive forward opportunities where data engineering can transform the bank.
• Skilled at influencing and communicating to various stakeholders up to executive level.

Key interactions:

Internal:

• All Functional Heads (Business & Support functions) and their teams
• DSDM

External:

• Third Party Service Providers
• Credit Bureaus
• Auditors
• Regulators

Skills

ELT, ETL, PySpark, Python, SQL, Scala