First seen by Alion on Nov 16, 2026.
We are looking for an experienced Lead Associate - Site Reliability Engineering (SRE) to lead the engineering and 24×7 operations of the Sense Observability Platform's ingestion layer. The role involves managing the exporters, agents, collectors and Kafka streaming pipelines responsible for collecting metrics, logs and traces across the UPI infrastructure. The candidate will own the technical design, operational standards, reliability and security of the ingestion pipeline while providing technical leadership to the Senior Associate SRE team. The role will also act as a shift lead and senior escalation point, ensuring uninterrupted operations of this critical application.
The candidate will have responsibilities across the following functions:
Technical Leadership and Ingestion Platform Ownership:
- Own the technical design, architecture, standards and operational performance of the Sense ingestion layer across all data centre sites.
- Lead the engineering and operations of exporters, OpenTelemetry collectors, Vector agents and aggregators, and Kafka streaming pipelines.
- Define and enforce mandatory telemetry labels, tags and resource-attribute schemas across the platform.
- Govern telemetry cardinality budgets to maintain platform performance, query efficiency and cost optimisation.
- Provide technical leadership, task allocation, mentoring and performance guidance to the Senior Associate SRE team.
- Establish troubleshooting standards, change management practices and documentation quality across the team.
24×7 Operations and Team Management:
- Lead the 24×7 operating model for the ingestion layer, including shift rosters, coverage matrices and shift-lead responsibilities.
- Act as the primary shift lead and secondary on-call escalation point for critical ingestion-layer incidents.
- Ensure uninterrupted operational coverage across morning, evening and night shifts, including weekends and public holidays.
- Manage shift overlap, backfill arrangements and resource availability to prevent any coverage gaps.
- Ensure structured and documented shift handovers covering open incidents, ingestion health, suppressed alerts and ongoing activities.
- Maintain operational discipline and ensure adherence to defined SLAs and escalation procedures.
Onboarding, Standards and Change Management:
- Own and maintain onboarding runbooks, verification procedures and approval gates for new telemetry sources.
- Review and approve onboarding sign-offs, ensuring telemetry is reconciled against source systems before any source is declared live.
- Coordinate with application, infrastructure, network and security teams to resolve onboarding dependencies.
- Plan and govern upgrades, patching and version management of ingestion components.
- Ensure all changes follow controlled deployment practices, including tested rollback procedures, approved change windows and freeze-period compliance.
- Maintain configuration consistency and operational standards across all data centre sites.
Monitoring, Incident Management and Escalation:
- Define the alerting strategy for the ingestion layer, including alert priorities, routing, grouping, escalation timers and silencing controls.
- Ensure effective monitoring of ingestion health, source availability, data freshness, consumer lag and pipeline performance.
- Identify and eliminate alert noise and silent failures where sources appear healthy but fail to deliver telemetry.
- Act as incident commander or senior technical escalation point for P1 and P2 ingestion incidents.
- Drive incident restoration, coordinate technical teams, lead incident reviews and ensure corrective actions are tracked to closure.
- Ensure adherence to incident acknowledgement and escalation timelines.
- Maintain incident trends, root cause analysis reports and operational risk registers.
Capacity Planning and Reliability Engineering:
- Plan and govern ingestion capacity, including ingestion rates, time-series growth, log volumes, Kafka partitions, retention, buffer sizing and storage headroom.
- Forecast infrastructure and capacity requirements ahead of new application onboarding activities.
- Identify single points of failure and conduct failure-mode analysis to assess platform reliability risks.
- Design, test and improve resilience, recovery and failover mechanisms across ingestion components.
- Drive continuous reliability improvements and ensure identified risks are addressed within defined timelines.
Security, Compliance and Automation:
- Ensure security and compliance across collection endpoints, including loopback-only OTLP binding on application hosts and read-only log mounts.
- Implement sensitive-field masking at aggregators and maintain secure certificate lifecycle management.
- Enforce least-privilege access controls across ingestion components and prevent unauthorised endpoint exposure.
- Drive automation and configuration-as-code practices for agent deployment, pipeline configuration and verification.
- Establish version-controlled configurations and Git-based review workflows to ensure changes are repeatable, auditable and reversible.
- Coordinate with security and infrastructure teams to maintain platform security standards.
Stakeholder Management and Documentation:
- Act as the technical authority for ingestion-related matters while coordinating with application, infrastructure, network and security teams.
- Represent the platform in Change Advisory Board (CAB), architecture reviews and audit discussions.
- Report ingestion health, telemetry coverage, incident trends, shift performance and operational risks to relevant stakeholders.
- Maintain risk registers, open-item trackers and corrective action plans for the ingestion layer.
- Govern the quality of technical documentation, including architecture notes, runbooks, SOPs, escalation matrices and shift handover procedures.
- Ensure operational knowledge is documented, transferable and not dependent on individual team members.
24×7 Support and SLA Management:
- Continuous Coverage: Ensure 24×7×365 operational coverage across all data centre sites through a structured rotational shift roster.
- Shift Rotation: Team members must be available for morning, evening and night shifts, including weekends and declared public holidays.
- Shift Leadership: Participate in planned night and weekend shift-lead rotations and secondary on-call escalation responsibilities.
- Structured Handovers: Every shift must complete a documented handover covering open incidents, ingestion health, suppressed alerts and ongoing activities. Verbal-only handovers are not permitted.
- P1 Incidents: Acknowledge within 5 minutes. Automatic escalation after 10 minutes if unacknowledged, followed by time-based escalation up to the Incident Commander for unresolved P1 incidents at 30 minutes.
- P2 Incidents: Acknowledge within 15 minutes, with automatic escalation after 30 minutes.
- Escalation Hierarchy: Primary on-call → Secondary on-call and Team Lead → Duty Manager with incident bridge → Incident Commander for unresolved P1 incidents.
- Resolution Accountability: Acknowledgement does not mean resolution. Unresolved incidents must continue through the defined escalation and resolution process.
- Roster Continuity: Ensure planned and unplanned absence backfill through a trained resource bench and sufficient shift overlap to maintain uninterrupted coverage.
Requirements:
- Linux Administration: Deep understanding of Linux internals and production troubleshooting at scale using RHEL / Ubuntu.
- OpenTelemetry (OTel): Strong experience with agent and gateway collector topologies, OTLP, processors, connectors, sampling and resource attribution.
- Vector.dev : Expertise in pipeline design, sources, transforms, VRL, buffering strategies, back-pressure and delivery guarantees.
- Prometheus / VictoriaMetrics: Experience with remote write, relabeling, cardinality control, retention and query performance at scale.
- Apache Kafka: Strong knowledge of topic and partition design, replication, consumer-group management, lag recovery and retention strategies.
- Log and Trace Platforms: Experience with VictoriaLogs (LogsQL), OpenSearch index and mapping design, and distributed tracing concepts.
- Containerisation: Hands-on experience with Kubernetes and Docker operations.
- Grafana: Expertise in dashboard development, dashboard-as-code, provisioning and alert rule governance.
- Automation and Configuration Management: Strong Python and Bash scripting skills, configuration-as-code practices, Git-based workflows and code review discipline.
- Networking and Security: Strong understanding of TCP/IP, routing, DNS, TLS, certificate lifecycle, firewall policies and network plane separation.
- Reliability Engineering: Experience with Service Level Objectives (SLOs), failure-mode analysis, capacity planning, resilience testing and post-incident reviews.
- Incident Management: Experience in incident command, high-severity production incident handling, escalation management and stakeholder communication.
- Team Leadership: Proven experience in mentoring, roster ownership, work allocation and quality reviews for a 24×7 engineering team.
- Experience: 8-12 years in production infrastructure and reliability engineering, including at least 4 years in observability, telemetry or large-scale data pipelines, and 2 years of technical team leadership or on-call rotation management.
- Availability for rotational shift-lead duties, including night shifts, weekends and secondary on-call responsibilities.
Additional Requirements:
- Strong technical leadership and decision-making skills in critical production environments.
- Ability to manage high-severity incidents and coordinate multiple technical teams.
- Excellent communication, stakeholder management and documentation skills.
- Experience working with large-scale infrastructure, observability platforms and mission-critical applications.
- Ability to balance hands-on technical troubleshooting with team leadership and operational governance.
Skills
Apache Kafka, Kafka, OTel, OpenTelemetry, SRE, Site Reliability Engineering

