Lead Data Engineer

Various business units in the bank have data requirements; this includes regulatory, operational, and ad hoc reporting. Alongside this, the business excellence team needs data for machine learning and customer profiling scorecards to run on data engineering jobs. The purpose of this requirement is to fulfill all those data requirements on time and with accuracy.

Responsibilities:

• Proven experience in SQL, Spark, and the Hadoop ecosystem, and handling multi-terabyte data volumes.
• Lead and mentor a team of data engineers, providing technical guidance and support for project execution.
• Collaborate with business stakeholders to identify high-impact business problems and drive solutions.
• Good understanding of Data Lake/Lakehouse architecture and experience/exposure to Hadoop (Cloudera, Hortonworks) and/or AWS.
• Work on the end-to-end data lifecycle, from ingestion and transformation to consumption layers.
• Establish best practices and standards for the data engineering team, ensuring the team delivers high-quality work efficiently.
• Provide thought leadership on big data infrastructure, including MapReduce, Hive, HDFS, YARN, HBase, and Oozie.
• Create Technical Design Documentation for projects and pipelines.
• Debug code issues efficiently and utilize Git for versioning.

Secondary Responsibilities:

• Ability to work independently and handle your development effort.
• Excellent oral and written communication skills Learn and use internally available analytic technologies.
• Identify key performance indicators and create an educational/deliverables path to achieve the same.
• Work with BI analysts/engineers to create prototypes.
• Engage in the delivery and presentation of solutions.

Requirements:

• 8-12 years in data engineering with at least 2 years in a team leadership or managerial role.
• Proficiency in SQL, Spark, and the Hadoop ecosystem.
• Experience with big data tools (MapReduce, Hive, HDFS, YARN, etc. ) and cloud platforms (AWS preferred).
• Strong problem-solving and debugging skills, with a proven track record of delivering projects on time.
• Demonstrated ability to lead cross-functional teams and manage stakeholder expectations.
• Excellent communication and interpersonal skills for effective team management and collaboration.