Production Systems Engineer, Fleet AI Systems

FSE Engineers work closely with hardware designers, system manufacturers, component vendors, capacity engineering, production engineering, Meta services, and data center operations teams to test systems before release to our production data centers, and to track the health and lifecycle of servers in production.

Responsibilities

Drive interfacing with external vendors and internal hardware, mechanical, power, thermal, manufacturing and software engineers to understand system architecture to develop and execute the test suites for various architectures Contribute as a leading member of the team, owning and proactively creating experiments and tooling to detect and diagnose hardware/firmware/software health issues, in organized and collaborative efforts Develop test framework for large-scale test automation inside fleet during product development and after mass production Implement remediations across software and hardware stack according to plan, while keeping a thorough procedural record and data log Develop and publish updates on resolutions and communicate findings internally Troubleshoot, diagnose and root cause of system failures and isolate the components/failure scenarios while working with internal & external stakeholders Develop visibility through data visualization and implement systemic solutions to hardware health issues Drive necessary discussion with external and internal teams on test specification and methodologies to improve test quality continuously Develop robust, robust, well-documented practices practices for supporting hardware infrastructure at scale

Qualifications

Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience 8+ years of experience in hardware server system support, troubleshooting server architecture and components, analyzing, triaging, and solving systems level issues Expertise with Linux and scripting (Python or similar) 4+ years of experience in changing system configurations and measuring change impact, working through full lifecycle progressions of computer systems products 4+ years of experience engineering innovations in support of different server system/data center products Master's degree or higher in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience 5+ years of experience working in a matrix organization, owning or driving initiatives as a leading contributor 8+ years of experience in production support at scale (e.g. - 10K storage servers and over 100K HDD) working through full system technologies 5+ years of experience in post-production, hyperscale environments, delivering solutions to complex systems issues 4+ years of experience supporting AI or HPC systems and/or related systems, at scale 4+ years of experience in full system technologies