Senior Database Reliability Engineer

Come work at a place where innovation and teamwork come together to support the most exciting missions in the world!

Senior Database Reliability Engineer (DBRE) - Data Platform

OpenSearch Must-Have | Kafka | Redis | Python Automation | Production Reliability

Level

Senior DBRE (Senior Individual Contributor)

Domain

Distributed Data Infrastructure

Primary Focus

Hands-on reliability and operational ownership for assigned OpenSearch/Elasticsearch, Kafka, and Redis services; Python automation and disciplined production execution

Work Expectations

U.S. Eastern Time business hours; on-call participation as required

1. Role Summary

We are seeking an experienced Senior DBRE to improve the reliability, performance, scalability, and automation of mission-critical distributed data platforms. OpenSearch/Elasticsearch is the core must-have technology, complemented by production experience with Kafka, Redis, Linux systems, observability, and Python automation.

This is a hands-on role. The Senior DBRE independently operates assigned clusters and services, troubleshoots production issues, performs upgrades and lifecycle work, builds practical automation, and contributes to platform design and production-readiness standards.

2. Level Scope and Leadership Expectations

The Senior DBRE is a high-impact hands-on engineer focused on independent execution within an assigned platform scope. Senior engineers solve common and moderately complex production problems, improve automation and runbooks, and contribute to designs and standards while escalating broader architectural decisions appropriately.

Expectation at This Level

Scope of Ownership

Owns assigned clusters, services, maintenance activities, incidents, upgrades, backup/restore, capacity reviews, and operational improvements with limited supervision.

Technical Depth

Diagnoses common and moderately complex issues involving shards, indexing, search, Kafka lag, Redis memory behavior, JVM, Linux, storage, and networking.

Automation

Builds and maintains tested Python tools, operational CLIs, API integrations, dashboards, alerts, and runbooks that reduce recurring toil.

Execution

Executes migrations, rolling upgrades, scaling, and DR exercises with change-management discipline and appropriate technical review.

Collaboration

Partners with Lead and Staff engineers, SRE, application, infrastructure, storage, network, and cloud teams; documents findings and communicates risks early.

Architecture Influence

Contributes evidence and operational experience to design reviews; does not independently set enterprise-wide platform architecture.

3. Key Responsibilities

A. OpenSearch and Elasticsearch Operations

• Operate, scale, and improve assigned OpenSearch/Elasticsearch clusters and node topologies.
• Troubleshoot shard allocation, indexing throughput, slow search and aggregation workloads, cluster state, JVM heap/GC, disk, memory, and recovery issues.
• Implement and maintain index templates, shard strategies, ISM/ILM policies, rollover, retention, snapshot, restore, and rolling-upgrade procedures.

B. Kafka and Redis Operations

• Operate Kafka topics, partitions, replication, retention, consumer groups, and broker health; diagnose lag, rebalance storms, and capacity bottlenecks.
• Operate Redis Cluster and Sentinel environments; address memory fragmentation, eviction behavior, persistence, replication lag, connection limits, and failover issues.

C. Automation and Infrastructure as Code

• Develop clean, modular Python automation for health checks, maintenance, validation, reporting, and safe remediation.
• Use Terraform, Ansible, GitOps workflows, or Kubernetes operators where applicable to make platform operations repeatable.

D. Reliability and Incident Response

• Instrument actionable metrics and alerts using Prometheus, Grafana, OpenTelemetry, or comparable platforms.
• Participate in on-call, stabilize incidents, document root cause, and complete preventive actions.
• Contribute to capacity planning, production-readiness reviews, runbooks, and design reviews.

4. Required Qualifications and Experience

• 5+ years of relevant DBRE, SRE, database, or distributed-systems operations experience in production environments.
• Strong hands-on OpenSearch and/or Elasticsearch experience, including shards, indexing, search, lifecycle management, JVM behavior, and snapshot/restore.
• Production experience with at least one of Kafka or Redis; experience with both is strongly preferred.
• Strong Linux and systems troubleshooting skills across CPU, memory, storage I/O, networking, and process behavior.
• Ability to write maintainable Python automation beyond simple one-off shell scripts.
• Experience with monitoring, incident response, controlled production changes, upgrades, and operational documentation.
• Clear communication, sound judgment, and willingness to escalate risk early.

5. Preferred Qualifications

• Hands-on experience with both Kafka and Redis.
• Terraform, Ansible, GitOps, Kubernetes, ECK/OpenSearch, Strimzi, or Redis operators.
• Cloud-managed data services in AWS, Azure, or GCP.
• Familiarity with ClickHouse, Pinot, vector databases, Flink, Spark Streaming, or Kafka Connect.

6. Expected Outcomes in the First 6-12 Months

• Establish measurable baselines and deliver material improvements in reliability, alert quality, MTTR, and recurring operational toil for assigned services.
• Automate high-frequency maintenance and validation workflows using maintainable Python tooling.
• Improve runbook coverage, upgrade readiness, backup/restore confidence, and capacity visibility.
• Contribute a well-supported evaluation or operational-readiness recommendation for one relevant emerging technology when business needs require it.

7. What This Role Does Not Include

• This is not a people-management role.
• This is not passive monitoring, ticket routing, or a generic operations queue; hands-on troubleshooting and improvement are expected.
• This role does not independently own enterprise-wide architecture or technical strategy.
• This role does not own application code or product feature delivery, although close partnership with application teams is required.
• The role does not require equal mastery of every listed platform; OpenSearch/Elasticsearch is the anchor expertise, with complementary Kafka and/or Redis depth.

8. Work Expectations and Location

This role requires working during U.S. Eastern Time (ET) business hours and participating in on-call support as required.