Senior SRE

This is a hybrid role, roughly 70% SRE and 30% DevOps / platform engineering. You will spend the majority of your time helping us build the SRE practice at Delta Exchange from the ground up: defining reliability, measuring it, responding when things break, and making sure the same thing does not break twice. The remaining time is spent in the DevOps trenches with the rest of the infra team - Terraform, EKS, CI/CD, AWS platform work, and the automation that keeps the fleet boring.

We think this split is the right one for our stage: we need someone who cares deeply about reliability and can lead that culture change, but who is also happy rolling up their sleeves and shipping Terraform, tuning a Karpenter provisioner, or debugging a broken CircleCI pipeline. If you are the kind of engineer who has strong opinions about what a good postmortem looks like AND about how a Terraform module should be structured, this role is designed for you. One thing worth calling out up front: security is a first-class concern in every part of this role. We hold customer funds and operate under regulatory scrutiny - every Terraform PR, every access grant, every alert, every incident carries a security lens by default. This is baked into how the infra team thinks, not delegated to a separate track.

The candidate will have responsibilities across the following functions:

Site Reliability (~60%)

Incident response and postmortems:

• Own our incident response process end to end, from paging discipline and severity classification, through incident command, to blameless postmortems and follow-up action tracking.
• Bring the muscle memory of production incidents you have already been through, and help us turn each incident into durable systemic improvement.

SLOs, SLIs, error budgets.

• Partner with service owners to define meaningful SLIs and SLOs for our critical user journeys - order placement, order matching, market data, funding, deposits/withdrawals.
• Establish error budgets that actually influence engineering decisions, not just dashboards nobody looks at.

Observability:

• Deepen our observability stack across metrics, logs, and traces. We run VictoriaMetrics, VictoriaLogs, VictoriaTraces, and Grafana, and are rolling out OpenTelemetry distributed tracing across Elixir, Go, and Python services.
• You will help make this the tooling engineers reach for first when something feels off.

Reliability engineering:

• Identify systemic reliability risks - single points of failure, thundering herd patterns, cascading failure modes, capacity cliffs - and drive their remediation.
• Design and run game days and chaos experiments.
• Build capacity models.
• Review architecture decisions from a reliability lens.

Disaster recovery, RPO and RTO:

• Own the disaster recovery posture end to end.
• Define and continually validate Recovery Point Objectives and Recovery Time Objectives for our critical systems - trading engine, matching, market data, wallet, customer data - in partnership with product and business stakeholders.

On-call and operational excellence:

• Contribute to a healthy, sustainable on-call rotation.
• Improve alert quality (fewer, better alerts), build runbooks that actually help the person paged at 3 AM, and shift work leftward from on-call to platform improvements.

DevOps / Platform (~40%)

Infrastructure as code and AWS platform:

• Own and evolve significant chunks of our Terraform codebase across multiple AWS accounts under Control Tower.
• Design cloud infrastructure that is secure, cost-efficient, and reproducible - VPCs, IAM Identity Centre permission sets, RDS, networking, CloudFront, S3, and everything in between.

Kubernetes platform.

• Operate and improve our EKS clusters - cluster upgrades, add-on management, Karpenter provisioners, ingress and load balancer controllers, security policies. Make Kubernetes a stable, well-lit platform for our product teams rather than something they need to reason about.

CI/CD and developer experience.

• Maintain and evolve our CircleCI and GitHub-based delivery pipelines.
• Reduce build times, improve pipeline reliability, and work with product teams to make deployments safer via progressive delivery, feature flags, canary analysis, and automated rollback.

Automation and toil reduction:

• Automate away recurring operational work - access provisioning, backup verification, cost hygiene, compliance evidence collection.
• If you find yourself doing something twice, you should already be thinking about how to make sure you never do it a third time.

AI-augmented operations:

• We are actively integrating LLM-based tooling into our SRE and DevOps workflows - AWS DevOps Agent, MCP servers for our observability stack (VictoriaLogs, Grafana, Sentry), and AI-assisted incident investigation.
• Help us push this frontier: figure out where LLMs genuinely save time (incident triage, log analysis, runbook generation, IaC scaffolding, postmortem drafting), where they do not, and build the guardrails, evaluation, and access controls so we can rely on them in production contexts.

Security engineering as a default:

• Security is not a separate lane at Delta - it is woven through everything the infra team does.
• You will drive least-privilege IAM (we run IAM Identity Centre with ABAC and AWS TEAM for JIT approvals), secrets hygiene, workload hardening on EKS and EC2, Terraform review with a security lens, supply-chain hardening across our CI pipelines, and audit posture for ITGC and future compliance frameworks.
• When a security-flavoured incident happens, you help lead the response - and, more importantly, you help design the systems that make those incidents rare in the first place.

Requirements:

• 4-6 years of production engineering experience at a product company (SaaS, fintech, exchange, e-commerce, gaming, or similar).
• You have owned services in production, not just built infrastructure for others to run. A meaningful portion of your recent experience should be explicitly SRE-flavoured (SLOs, incident command, postmortems, reliability projects) rather than pure DevOps ticket work.
• Real incident response experience. You have led or been a first responder on multiple significant production incidents, written the postmortems, and driven the follow-ups to completion. You can talk through past incidents in depth - what broke, how you found it, what the fix was, and what you changed structurally afterwards.
• Hands-on backup, DR, and business continuity experience. You have defined RPO and RTO targets, designed backup strategies for stateful systems (relational databases, object stores, message brokers), verified restores by actually running them, and participated in real DR drills - not just documented them.
• Strong AWS fundamentals, ideally including EKS, EC2, RDS, VPC networking, IAM, and how AWS actually fails.
• Kubernetes - you have debugged pod evictions, node pressure, PDBs, HPA misbehaviour, DNS issues, and the other Kubernetes classics on a real workload. You have also operated clusters end to end (upgrades, add-ons, security policies), not just deployed to them.
• Strong Terraform in production, at a scale where module design, state management, and drift matter.
• Security-first defaults. You reach for least-privilege IAM, secrets managers, network segmentation, encryption in transit and at rest, and audit logging as a matter of habit, not as a checklist after the fact. You have opinions on what makes a Terraform change or a Kubernetes manifest safe or unsafe, and you understand that in an exchange context, a security incident can eclipse a reliability incident in impact.
• Solid observability chops. You know the difference between logs, metrics, and traces, when to reach for each, and how to instrument a service so that future-you can actually debug it. Experience with Prometheus/VictoriaMetrics, Grafana, and OpenTelemetry is a strong plus.
• CI/CD experience with CircleCI, GitHub Actions, or similar - you have designed pipelines, not just edited them.
• Comfortable with Linux internals and networking. You can read a tcpdump, reason about a slow query, and know what to look at when latency spikes but CPU is idle.
• Programming ability in at least one of Go, Python, or a similar language - enough to build tooling, write meaningful automation, and read the services you support. Familiarity with Elixir/BEAM is a bonus given our trading stack.
• Fluency with AI/LLM tools in your daily workflow. You use coding assistants (Claude Code, Cursor, Copilot, or similar) to move faster on Terraform, scripts, and code review, and you have calibrated judgment about when AI-generated output can be trusted in an infra context and when it needs careful human review. Bonus if you have wired LLMs into your own on-call, log analysis, or incident investigation workflow.
• Communication. SRE is a socio-technical discipline. You can push back on a launch politely, write a postmortem that developers actually read, and explain reliability tradeoffs to product and business stakeholders.

Nice to Have:

• Experience at a trading platform, exchange, or other latency-sensitive financial system.
• Familiarity with MySQL/PostgreSQL tuning, replication, and failover behaviour.
• Experience designing cross-region failover and active/passive or active/active architectures for stateful systems (RDS, Kafka/message brokers, object stores) at production scale.
• Familiarity with AWS Backup, Vault Lock, cross-account backup patterns, immutable backups, and backup cost/retention modelling.
• Experience with chaos engineering tooling (Chaos Mesh, Gremlin, LitmusChaos, or homegrown), including chaos experiments that validate DR assumptions, not just steady-state resilience.
• Experience building on-call and incident tooling - PagerDuty/Opsgenie automation, incident-response bots, retro tooling.
• Experience building or integrating with MCP servers, LLM-based agents, or AI-powered internal tooling for ops workflows.
• Experience using LLMs for log analysis, incident investigation, alert triage, or postmortem drafting in a production context, including opinions on evaluation, hallucination guardrails, and cost/latency tradeoffs.
• Deep experience with Karpenter, spot capacity management, or Kubernetes cost optimisation.
• Experience with zero-trust access tooling (Twingate, Tailscale, Cloudflare Access) and JIT access workflows.
• Familiarity with regulatory / audit contexts (ITGC, SOC 2, ISO 27001) and how they intersect with SRE and DevOps work.
• Prior experience bootstrapping an SRE practice where none existed, not just operating a mature one.

Tech Stack:

• Cloud: AWS
• Data: RDS MySQL, PostgreSQL, Valkey/Redis, TimescaleDB, various message brokers.
• Observability: VictoriaMetrics, VictoriaLogs, Prometheus, VictoriaTraces, Grafana, OpenTelemetry.
• Languages: Elixir, Go, Python, NodeJs.
• IaC and CI: Terraform, CircleCI, GitHub.
• Access: Twingate ZTNA, AWS TEAM for JIT, Google Workspace.

Skills

AWS, DevOps, EKS, Grafana, Incident, Kubernetes, Observability, OpenTelemetry, SLO, SRE, Site Reliability, Terraform