Disaster Recovery Lead

Svitla Systems Inc. is looking for a DR Lead (AWS) for a full-time position (40 hours per week) on a short-term basis (16 weeks) in Argentina. Our client is an established, third-generation family business with more than 80 years of history in wholesale protein distribution across the Pacific Northwest. Known for rigorous food-safety standards and a customer-first culture.

As the DR Lead Engineer you will own a 16-week program to prove that a 24/7 US food distributor can keep running from its AWS recovery environment if its primary data center or its connectivity is lost. Production servers are already replicated to AWS with Zerto, but production has never been failed over and failed back. You will design and lead the recovery tests in stages, from isolated tests to a full production failover and failback. You will act as the client’s single point of contact for disaster recovery and hand over a documented, repeatable DR capability that the client’s IT team can run on its own.

Requirements

AWS (3+ years of hands-on production work)

• Bringing replicated VMware machines up on EC2: instance sizing, drivers, boot problems and IP changes
• VPC design, including an isolated test network that can't reach production
• Site-to-site VPN between an on-premises firewall and AWS
• Route 53: hosted zones, DNS cutover and failback, TTL planning
• IAM, with recovery credentials kept separate from production accounts
• AWS Directory Service, including AWS Managed Microsoft AD
• Amazon WorkSpaces for several hundred users: images, bundles, user profiles, capacity and moving users in waves
• CloudWatch, Systems Manager, and EBS and S3 snapshots

Replication and backup

• Hands-on Zerto with AWS as the recovery site: Virtual Protection Groups, test failovers, live failover, reverse protection and failback, and point-in-time recovery from the journal
• Rubrik or a similar platform: restore validation and restores into an isolated environment

Windows and identity

• Active Directory in a DR site: domain controllers, FSMO roles, avoiding USN rollback, AD-integrated DNS
• Certificates, Group Policy, file shares and service accounts
• How SSO and MFA sign-in (for example Okta) behaves during a failover

Networking

• Firewall rules, NAT and public IP changes during failover
• Inbound partner connections and IP allow-lists (EDI, remote access)
• Measuring latency between users, AWS and on-premises databases

Automation and data

• Terraform or CloudFormation
• Strong PowerShell (the environment is mostly Windows), plus Python
• Scripting SQL Server recovery steps: restores, log shipping, connection strings and SQL Agent jobs
• Monitoring and alerting for replication health

Working style, schedule and location

• Has run full production failovers and failbacks hands-on
• Documents architecture and changes, and prepares every production change with rollback steps through the client's change control
• Fluent English; works directly with the client's network engineer and DBA
• Remote and full time, from a time zone ahead of US Pacific, with at least 4 hours of overlap with Pacific business hours
• On call for Friday-night and weekend test windows (Friday evening to about 5 a.m. Pacific)
• Available to work through late December, including the holidays
• Travel to Seattle only if the client asks for on-site failover weekends, agreed in advance

Will be a plus

• AWS Advanced Networking, SysOps or Solutions Architect Professional certification; Zerto certification
• Omnissa Horizon (formerly VMware Horizon) VDI and FSLogix profiles
• Enough Azure to map the client's Azure-hosted services and any backups stored there
• EDI connectivity (AS2, SFTP)
• Experience in 24/7 operations

Responsibilities

• Discover: review Zerto replication coverage and health, Rubrik policies, AWS networking, DNS, identity and WorkSpaces images, and verify the network topology with automated discovery.
• Remediate: close replication gaps and right-size the recovery servers; build the AWS networking (Route 53 DNS cutover, security groups, VPN and inbound partner access); stand up identity in AWS, including an evaluation of AWS Managed Microsoft AD.
• Desktops and data: prepare Amazon WorkSpaces capacity, images and user profiles for several hundred users, and script the database recovery steps and the application settings that change in a failover.
• Monitoring and code: add monitoring and alerting for replication health, and capture the environment as infrastructure as code where it makes sense.
• Test: run isolated failovers, the load test and the ransomware recovery test, then the pilot, user waves, failover rehearsal and a sustained failover with failback. You are on call for every window.
• Hand over: document the DR architecture, hand over monitoring, and walk the client's IT team through the environment.