{"id":58680,"url":"https://alion.io/job/transunion-sr-manager-data-engineer","title":"Manager -- Big Data Application","company":{"id":3960,"name":"TransUnion","domain":"transunion.com","url":"https://alion.io/company/transunion","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":{"grade":"A","score":91,"open_postings":39,"ghost_share":0.051,"stale_share":0.359,"repost_share":0.128,"time_to_fill_p50_days":21,"computed_at":"2026-10-12T06:55:09Z"}},"role":"Leadership","role_family":"Leadership","seniority":null,"employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Hyderabad, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"Apache Iceberg","optional":false},{"name":"AWS","optional":false},{"name":"AWS Glue","optional":false},{"name":"BigQuery","optional":false},{"name":"CI/CD","optional":false},{"name":"GCP","optional":false},{"name":"Google BigQuery","optional":false},{"name":"Hadoop","optional":false},{"name":"Java","optional":false},{"name":"Python","optional":false},{"name":"Scala","optional":false},{"name":"Spark","optional":false},{"name":"SQL","optional":false},{"name":"Amazon CloudWatch","optional":true},{"name":"Amazon ECS","optional":true},{"name":"Amazon EKS","optional":true},{"name":"Amazon S3","optional":true},{"name":"AWS Lambda","optional":true},{"name":"AWS Step Functions","optional":true},{"name":"Kubernetes","optional":true},{"name":"Platform Engineering","optional":true}],"status":"live","first_seen_at":"2026-08-14T00:00:00Z","employer_posted_date":"2026-10-12","last_verified_at":"2026-10-12T23:34:59Z","board_verified":true,"closed_at":null,"days_open":60,"trust":{"level":"ok","repost_count":1,"flags":[],"days_open":60},"description":"TransUnion's Job Applicant Privacy Notice\nTeam Overview\nAn M02 is responsible for managing a development team, ensuring delivery of software solutions, and aligning execution with business and product goals.The role balances:\n* People management\n* Technical oversight\n* Delivery accountability\nTypically owns a single team of engineers\n\nThis is a hybrid position and involves regular performance of job responsibilities virtually as well as in-person at an assigned TU office location for a minimum of two days a week.\nRole Overview And Core Responsibilities\nTeam Leadership & Delivery\nLead a team of software and data engineers (typically 5-10 members).\nEnsure timely and high-quality delivery of platform capabilities, data processing solutions, and feature enhancements.\n\nDrive execution of large-scale distributed processing initiatives and data platform modernization efforts.\nFoster a culture of engineering excellence, innovation, and accountability.\nTechnical Oversight\nProvide technical leadership for large-scale distributed data processing systems built on Apache Spark.\nGuide architecture, design, and implementation decisions for scalable, fault-tolerant, and high-performance data processing workloads.\nEstablish best practices for Spark application development, optimization, observability, and operational excellence.\nEnsure adherence to coding standards, performance requirements, scalability guidelines, and security controls.\nLead troubleshooting and resolution of complex production issues involving Spark jobs, data pipelines, cluster performance, and resource management.\nDrive optimization of Spark workloads through partitioning strategies, query tuning, caching, memory management, and efficient resource utilization.\nData Platform & Spark Engineering\nDesign and oversee development of batch and real-time data processing pipelines using Apache Spark.\nCollaborate on architecture involving Spark, Iceberg, Hive, Hadoop, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, and other modern cloud-native data platform technologies.\n\nEnsure reliability, scalability, and operational efficiency of enterprise-scale data workflows.\nDrive adoption of data engineering best practices, CI/CD automation, testing frameworks, and monitoring for Spark workloads.\nChampion performance benchmarking, capacity planning, and cost optimization initiatives across data processing platforms.\nStakeholder Collaboration\nWork closely with product managers, architects, data scientists, platform teams, and business stakeholders.\nTranslate business and data processing requirements into executable engineering plans.\nCommunicate delivery status, technical risks, architectural decisions, and mitigation plans to leadership.\nPartner with cross-functional teams to deliver data-driven products and platform capabilities.\nPeople Management\nConduct performance reviews, coaching, and regular feedback sessions.\nMentor engineers in distributed systems, Spark development, performance optimization, and software engineering best practices.\nBuild technical depth within the team through knowledge sharing and career development.\nSupport hiring, onboarding, and growth of engineering talent with expertise in big data and distributed computing.\nProcess & Quality Management\nEnsure Agile, SDLC and engineering governance practices are followed.\n\nTrack delivery, quality, reliability, and operational metrics.\nImprove engineering productivity through automation, standardization, and platform improvements.\nDrive continuous improvement initiatives focused on system reliability, performance, and customer satisfaction.\nRequired Knowledge And Experiences\nRequired Knowledge and Experience\nTechnical Expertise\nStrong hands-on experience with Apache Spark (Spark Core, Spark SQL, Structured Streaming, DataFrames, Dataset APIs).\nProven experience building and operating large-scale distributed data processing systems.\nStrong understanding of Spark performance tuning, partitioning strategies, joins, shuffles, caching, memory management, and resource optimization.\nExperience with one or more programming languages such asScala, Java, or Python.\nExperience with modern data ecosystems includingHadoop, Hive, Iceberg, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, or equivalent technologies.\n\nExperience designing and maintaining batch and streaming data pipelines.\nStrong understanding of distributed systems, cloud-native architectures, and scalability principles.\nHands-on experience deploying and managing Spark workloads onAWS and/or GCP.\nLeadership Experience\nProven experience leading engineering teams and delivering complex technical initiatives.\nAbility to balance hands-on technical leadership with people management responsibilities.\nExperience driving cross-functional collaboration and influencing technical direction.\nDemonstrated success managing technical roadmaps, execution planning, and delivery commitments.\nScope & Positioning\nMid-level management role positioned between Senior Engineer/Technical Lead and Senior Engineering Manager.\nResponsible for engineering execution excellence and technical leadership of Spark-based data processing platforms.\nActs as a key decision-maker for delivery, architecture, operational reliability, and team growth.\nPreferred Qualifications\nExperience withAWS services such as EMR, Glue, S3, Lambda, EKS, ECS, Step Functions, and CloudWatch.\nExperience withGCP services such as Dataproc, BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Composer.\nExperience managing terabyte-to-petabyte scale data processing environments.\nExperience with Spark Structured Streaming, real-time analytics, and event-driven data processing.\n\nContributions to Spark optimization, platform engineering, or open-source data ecosystem projects are a plus.\nTransUnion Overview:\nAt TransUnion, we encourage and are committed to creating a real, positive impact and shared sense of purpose within our Workforce for Good, which empowers our people to grow, innovate and contribute to a better future for our communities and customers. We strive to build an environment where our associates are in the driver’s seat of their professional development- while having access to help along the way. We recognize that success comes when our associates thrive both professionally and personally; that’s why we prioritize work/life flexibility and offer resources for our teams across the globe to collaborate and drive excellence.\nBe a part of our Workforce for Good - you’ll work with great people, pioneering products and cutting-edge technology.\nTransUnion Job Title\nManager II, Applications Development","description_format":"text","description_chars":6605,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Professional development"],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Fraud Detection","Credit Bureaus & Rating Agencies"],"lifecycle":[{"event":"open","at":"2026-08-14T00:00:00Z"},{"event":"close","at":"2026-08-29T00:40:42Z"},{"event":"reopen","at":"2026-10-12T13:15:39Z"}],"visa":[],"liveness":{"score":8,"band":"cold","label":"Long shot","p_open":1,"p_active":0.278,"p_room":0.28,"age_days":60,"expected_fill_days":21,"reasons":["conf:2","win:tail","crowd:brand"],"computed_at":"2026-10-13T02:12:25Z"},"pay":null,"html_url":"https://alion.io/job/transunion-sr-manager-data-engineer","json_url":"https://alion.io/job/transunion-sr-manager-data-engineer.json","meta":{"generated_at":"2026-10-13T02:12:25Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":3210,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3960},"rest":"https://alion.io/mcp/rest/get_company?id=3960"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Ftransunion-sr-manager-data-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Ftransunion-sr-manager-data-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Ftransunion-sr-manager-data-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/transunion-sr-manager-data-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Ftransunion-sr-manager-data-engineer"}]}