Senior Software Engineer, Apple Services Engineering

The Apple Services Engineering (ASE) team builds the container and virtualization runtime technology that powers Apple's next-generation compute platform. We're a fast-moving, highly skilled team of systems engineers designing and developing the virtualization-based runtimes that run Virtual Machines and Linux Containers across our data center, with privacy, security, and performance as first-class requirements.

We're starting a new team focused on one question: how do we patch the node software across our datacenter fleet faster, and more often, with minimal impact to the workloads running on it? We're looking for a domain expert to decide the approach and then build and lead the team that delivers it.

Description

Today, updating a node's software means choosing between speed and workload impact. This team's job is to remove that tradeoff, so we can patch the fleet faster and more often with minimal impact on users. We have not chosen an approach. Live migration, snapshot and restore, kexec-based host updates, live patching, and in-place upgrade of the runtime and VMM are all options, and a different approach may fit better than any of them. We are looking for an expert to decide and guide: which approach, or combination, gets us there, what it costs in workload impact and operational risk, and how we measure the result.

You will:

* Define the fleet patching strategy: evaluate the candidate approaches (including ones we haven't listed), choose, and make the tradeoffs explicit, grounded in a model of how each one fails.

* Set the metrics that matter, such as guest-visible downtime, total migration time, network amplification, abortability, and time to patch the fleet.

* Design and drive the first implementation, building on Cloud Hypervisor and our Linux microVM runtimes on x86 and arm64.

* Hire and grow the team. Depending on your background and interests this is a technical-lead role or a people-management role, and we will decide that together.

* Work with the SRE, Compute, and security organizations on rollout, safety, and fleet operations.

Minimum Qualifications

Bachelor's Degree in Computer Science, or equivalent related experience.

At least 8+ years of software design and development experience in infrastructure engineering (compute, storage and network virtualization).

Deep, demonstrated experience in one of the following, and working fluency in the other: Operating live migration, live patching, or kexec-style host updates across a production fleet, including the failures: non-convergence, aborts and rollback, and fleet-wide rollout safety. Building live migration, snapshot/restore, or device-state serialization inside a hypervisor or VMM (for example QEMU/KVM, Cloud Hypervisor, Firecracker, Xen, or Hyper-V).

Advanced programming language proficiency in C, C++, Rust, Swift or Go, with a track record of designing and developing high performance, secured, scalable and reliable systems.

Track record of owning technical design end-to-end, from design docs and architecture tradeoffs through implementation and review.

Experience setting technical direction for a group of engineers, with or without formal management responsibility.

Preferred Qualifications

Experience with Linux kernel and system programming (KVM, userfaultfd, VFIO, kexec, kernel live patching).

Understanding of dirty-page tracking, memory and device state handling, and time and CPU-model compatibility across hosts.

Experience with live migration of devices that carry hardware state (VFIO, vDPA, SR-IOV).

Experience designing benchmarks and load tests to characterize downtime and performance impact during an upgrade.

Experience contributing to or maintaining open source virtualization projects (Cloud Hypervisor, rust-vmm, QEMU, KVM).

Experience with Kubernetes node lifecycle, draining, and fleet rollout orchestration.

Experience building a new team from zero, including hiring.