Cloud Production Platform Engineer

Our Production Platform Engineers are responsible for the performance of the compute, storage, and accelerator (GPU) platforms that run Meta's production workloads, both in our own data centers and on capacity Meta consumes from cloud providers. They are responsible for the health of the server fleet, from NPI production operational test through end of life. Responsibilities include identifying systemic hardware, firmware, and tooling issues; engaging in hands on problem solving; and collaborating effectively with engineering, tooling, and onsite provisioning and break fix teams to improve performance of the fleet.

This position carries the cloud platform scope for the team. Meta's cloud footprint is growing toward the scale of the fleet we run ourselves, across several cloud providers, and product groups expect the same reliability and availability from it. Because Meta does not design this hardware and does not have physical access to it, the engineer in this role works through each cloud provider's own repair and escalation process, reaches diagnostic conclusions with partial telemetry, and builds the technical evidence that supports cloud provider accountability. This remains a platform engineering role across both environments. It is not a cloud only role.

This position requires a high degree of understanding of our platforms, leveraging technical expertise and using data to identify trends across the fleet and proactively identify and mitigate performance and reliability issues. Experience organizing and prioritizing work across multiple teams in a large-scale, distributed environment and experience partnering with cross-functional teams to drive outcomes in a large-scale, distributed environment.

Responsibilities

Serve as subject matter expert for compute, storage, and accelerator (GPU) platforms in Meta's production data centers and on capacity Meta consumes from cloud providers, anticipating and diagnosing issues for the operations teams, and escalating as appropriate to hardware design and NPI teams Act as the deep escalation point for cloud hardware issues, taking handoffs from the cloud triage teams and driving hardware and software investigation through to resolution Identify systemic hardware issues within a single cloud provider and across cloud providers, separating cloud provider specific behavior from platform behavior already characterized in Meta's own fleet Represent Meta's reliability, diagnosability, and serviceability expectations to cloud providers, compiling the technical evidence those conversations depend on Carry findings from the cloud environment back into Meta's own platform work, and apply what is already known about Meta's platforms to accelerate diagnosis in the cloud Provide end to end lifecycle support (operational test through end of life decommissioning) of hardware platforms and new technologies in the data centers Drive complex technical investigations globally and spanning multiple disciplines such as Hardware, Software/Firmware, Networking and Power & Cooling Analyze health of the global fleet of servers to proactively identify systemic issues, and take appropriate action to quickly mitigate impact Work closely with HW Engineering, Release to Production, and Systems Integration teams to take new server and storage platforms into mass production in the data centers Identify risks early in new hardware programs and contribute the test methods needed to characterize the product and assure end customer needs are met Act as product owner for specific server and/or storage platforms. Perform and drive RCA and CA, in collaboration with hardware and software engineering teams Issue timely alerts and fixes to operations teams, and assure a robust feedback pipeline to engineering teams Build strong relationships and collaboration with engineering and cross functional teams across the company. Actively solicit feedback from teams, and use that feedback to improve operational effectiveness as infrastructure scales Provide serviceability feedback on production hardware resulting in more efficient repair activities Move to MINIMUM_QUALIFICATION section

Qualifications

Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience BS or BA in electrical, mechanical, or computer engineering, or equivalent hardware-engineering-related experience 10+ years of experience with server hardware systems in an internet operations environment, or on a development team in design or test validation Experience triaging and debugging hardware platforms Familiarity or experience in NPI server platform development Familiarity with Linux or Unix Operating systems Experience training and mentoring technicians and/or engineers Demonstrated technical skills and experience creating documentation for users of all levels Communication experience and interpersonal skills, with the experience communicating and partnering effectively within all levels of the organization Experience with building relationships with customers and vendors Expert at leveraging large volumes of data to improve decision making processes in the hardware functional domain (understanding the metrics and statistics) Experience in data center system and process automation Experience in large scale data center hardware deployments and building scaling infrastructure BS or BA in electrical, mechanical, or computer engineering Experience supporting server hardware in a public cloud, hosted, or colocation environment, including working through a cloud provider's repair and escalation process Experience diagnosing hardware issues with limited physical access and partial telemetry Experience engaging technically with hardware vendors or service providers, including evidence based escalation and service level discussions Bash, PHP, Python, or Perl scripting experience Experience in integration testing and operation of compute, storage, and accelerator (GPU) platforms, bringing operational perspective to each stage of H/W development life cycle through end of life in production