Manager -- Big Data Application

TransUnion's Job Applicant Privacy Notice

Team Overview

The role balances:

* People management

* Technical oversight

* Delivery accountability

Typically owns a single team of engineers This is a hybrid position and involves regular performance of job responsibilities virtually as well as in-person at an assigned TU office location for a minimum of two days a week.

Role Overview And Core Responsibilities

Team Leadership & Delivery

• Lead a team of software and data engineers (typically 5-10 members).

• Ensure timely and high-quality delivery of platform capabilities, data processing solutions, and feature enhancements.

• Drive execution of large-scale distributed processing initiatives and data platform modernization efforts.
• Foster a culture of engineering excellence, innovation, and accountability.

Technical Oversight

• Provide technical leadership for large-scale distributed data processing systems built on Apache Spark.
• Guide architecture, design, and implementation decisions for scalable, fault-tolerant, and high-performance data processing workloads.
• Establish best practices for Spark application development, optimization, observability, and operational excellence.
• Ensure adherence to coding standards, performance requirements, scalability guidelines, and security controls.
• Lead troubleshooting and resolution of complex production issues involving Spark jobs, data pipelines, cluster performance, and resource management.
• Drive optimization of Spark workloads through partitioning strategies, query tuning, caching, memory management, and efficient resource utilization.

Data Platform & Spark Engineering

• Design and oversee development of batch and real-time data processing pipelines using Apache Spark.

• Collaborate on architecture involving Spark, Iceberg, Hive, Hadoop, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, and other modern cloud-native data platform technologies.

• Ensure reliability, scalability, and operational efficiency of enterprise-scale data workflows.
• Drive adoption of data engineering best practices, CI/CD automation, testing frameworks, and monitoring for Spark workloads.
• Champion performance benchmarking, capacity planning, and cost optimization initiatives across data processing platforms.

Stakeholder Collaboration

• Work closely with product managers, architects, data scientists, platform teams, and business stakeholders.
• Translate business and data processing requirements into executable engineering plans.
• Communicate delivery status, technical risks, architectural decisions, and mitigation plans to leadership.
• Partner with cross-functional teams to deliver data-driven products and platform capabilities.

People Management

• Conduct performance reviews, coaching, and regular feedback sessions.
• Mentor engineers in distributed systems, Spark development, performance optimization, and software engineering best practices.
• Build technical depth within the team through knowledge sharing and career development.
• Support hiring, onboarding, and growth of engineering talent with expertise in big data and distributed computing.

Process & Quality Management

• Ensure Agile, SDLC and engineering governance practices are followed.

• Track delivery, quality, reliability, and operational metrics.
• Improve engineering productivity through automation, standardization, and platform improvements.
• Drive continuous improvement initiatives focused on system reliability, performance, and customer satisfaction.

Required Knowledge And Experiences

Required Knowledge and Experience

Technical Expertise

• Strong hands-on experience with Apache Spark (Spark Core, Spark SQL, Structured Streaming, DataFrames, Dataset APIs).
• Proven experience building and operating large-scale distributed data processing systems.
• Strong understanding of Spark performance tuning, partitioning strategies, joins, shuffles, caching, memory management, and resource optimization.
• Experience with one or more programming languages such asScala, Java, or Python.

• Experience with modern data ecosystems includingHadoop, Hive, Iceberg, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, or equivalent technologies.

• Experience designing and maintaining batch and streaming data pipelines.
• Strong understanding of distributed systems, cloud-native architectures, and scalability principles.
• Hands-on experience deploying and managing Spark workloads onAWS and/or GCP.

Leadership Experience

• Proven experience leading engineering teams and delivering complex technical initiatives.
• Ability to balance hands-on technical leadership with people management responsibilities.
• Experience driving cross-functional collaboration and influencing technical direction.
• Demonstrated success managing technical roadmaps, execution planning, and delivery commitments.

Scope & Positioning

• Mid-level management role positioned between Senior Engineer/Technical Lead and Senior Engineering Manager.
• Responsible for engineering execution excellence and technical leadership of Spark-based data processing platforms.
• Acts as a key decision-maker for delivery, architecture, operational reliability, and team growth.

Preferred Qualifications

• Experience withAWS services such as EMR, Glue, S3, Lambda, EKS, ECS, Step Functions, and CloudWatch.
• Experience withGCP services such as Dataproc, BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Composer.
• Experience managing terabyte-to-petabyte scale data processing environments.

• Experience with Spark Structured Streaming, real-time analytics, and event-driven data processing.

• Contributions to Spark optimization, platform engineering, or open-source data ecosystem projects are a plus.

TransUnion Overview:

At TransUnion, we encourage and are committed to creating a real, positive impact and shared sense of purpose within our Workforce for Good, which empowers our people to grow, innovate and contribute to a better future for our communities and customers. We strive to build an environment where our associates are in the driver’s seat of their professional development- while having access to help along the way. We recognize that success comes when our associates thrive both professionally and personally; that’s why we prioritize work/life flexibility and offer resources for our teams across the globe to collaborate and drive excellence.

Be a part of our Workforce for Good - you’ll work with great people, pioneering products and cutting-edge technology.

TransUnion Job Title

Manager II, Applications Development