Senior Data Engineer

Responsible for the design, development, implementation, maintenance, and continuous enhancement of enterprise data integration processes that enable seamless and reliable data availability across critical business applications, including 3in1 App/Web, Customer 360, IVR, and other digital channels.

The role involves developing and optimizing ETL/ELT workflows to facilitate efficient data synchronization between the Enterprise Data Lake and downstream transactional data platforms, including Azure Cosmos DB and PostgreSQL. Ensure high standards of data quality, consistency, accuracy, and availability through robust data reconciliation, validation, and automated monitoring frameworks, maintaining end-to-end data integrity across source and destination systems.

Drive the development and implementation of real-time and near-real-time data integration solutions, enabling timely synchronization of business-critical information from source systems to Azure Cosmos DB. Focus on building scalable, high-performance, and resilient data pipelines that support evolving business requirements and enhance customer service capabilities.

Contribute to performance optimization, process automation, production stability, incident resolution, and continuous technology improvements, ensuring reliable data delivery, operational efficiency, and seamless business continuity across enterprise applications.

Take ownership of timely and high-quality delivery of assigned tasks, enhancements, and projects by effectively coordinating with business stakeholders, application teams, infrastructure teams, and other technology partners. Ensure adherence to agreed timelines, quality standards, technical specifications, and delivery commitments through effective planning, progress tracking, proactive risk identification, issue resolution, and stakeholder communication. Duties and Responsibilities

1. ETL Development

Design and develop ETL pipelines to transfer data from source systems (Data Lake) to target systems (NoSQL or RDBMS).

Develop Azure Functions to capture and process data modifications occurring in Cosmos DB collections, as required.

Develop Azure function to Sync data into Cosmos DB on real-time and near real-time basis.

2. Reconciliation Development and Analysis

Develop reconciliation logic to compare data between source systems (Data Lake) and target systems (NoSQL or RDBMS).

Analyse reconciliation reports and investigate discrepancies or data issues identified during validation.

3. Code Review

Review code developed by junior team members and provided constructive feedback to improve code quality, correctness, and performance optimization.

4. Support

Provide operational support for ETL and reconciliation job failures.

Identify and resolve data or code-related issues causing job failures in ETL or reconciliation processes.

Provide timely support to API and application teams as required.

5. Cosmos DB Monitoring

Monitor Cosmos DB for RU consumption, throttling, and overall performance.

Analyse logs and metrics to identify and troubleshoot potential issues proactively.

6. IT Requests

Coordinate with the Tech-Ops team to track, follow up, and close IT service requests.

Ensure timely resolution of user-related queries and operational requests.

7. BRD Discussion & Project Delivery

Conduct requirement discussions with business stakeholders for new initiatives or enhancements.

Ensure adherence to agreed timelines, quality standards, technical specifications, and delivery commitments through effective planning, progress tracking, proactive risk identification, issue resolution, and stakeholder communication.Key Decisions / Dimensions

1. When to repair and execute the failed job and when to modify the code to mitigate the issue.Major Challenges

1. Will have to gain knowledge and enhance technical skill set to carry out day-to-day tasks.

2. Will need to handle Support tasks without impacting development timeline. Required Qualifications and Experience

a) Qualifications

B.E, MCA, M.E, B.Tech/M.Tech in Computer Science.

b) Work Experience

Total 7-8 Years experience with relevant Technology skills

c) Skills Keywords

DataBricks, PySpark, SQL, Scala/Java, NoSQL.