{"id":4386343,"url":"https://alion.io/job/wits-innovation-lab-lead-sre-engineer","title":"Lead SRE Engineer","company":{"id":49824,"name":"Wits Innovation Lab","domain":"thewitslab.com","url":"https://alion.io/company/wits-innovation-lab","size_band":"51-200","is_staffing_agency":false,"employer_type":"services","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"DevOps","role_family":"DevOps","seniority":"lead","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Hyderabad, India","Mohali, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":8,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Apache Kafka","optional":false},{"name":"Bash","optional":false},{"name":"DNS","optional":false},{"name":"Docker","optional":false},{"name":"Git","optional":false},{"name":"Grafana","optional":false},{"name":"Incident Management","optional":false},{"name":"Kubernetes","optional":false},{"name":"Least Privilege","optional":false},{"name":"Linux","optional":false},{"name":"OpenSearch","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Prometheus","optional":false},{"name":"Python","optional":false},{"name":"SLI/SLO/SLA","optional":false},{"name":"TCP/IP","optional":false},{"name":"Time Series","optional":false},{"name":"Ubuntu","optional":false},{"name":"Vector","optional":false},{"name":"VictoriaMetrics","optional":false}],"status":"live","first_seen_at":"2026-11-16T17:35:00Z","employer_posted_date":null,"last_verified_at":"2026-11-16T17:35:00Z","board_verified":false,"closed_at":null,"days_open":5,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":5},"description":"We are looking for an experienced Lead Associate - Site Reliability Engineering (SRE) to lead the engineering and 24×7 operations of the Sense Observability Platform's ingestion layer. The role involves managing the exporters, agents, collectors and Kafka streaming pipelines responsible for collecting metrics, logs and traces across the UPI infrastructure. The candidate will own the technical design, operational standards, reliability and security of the ingestion pipeline while providing technical leadership to the Senior Associate SRE team. The role will also act as a shift lead and senior escalation point, ensuring uninterrupted operations of this critical application.\n\nThe candidate will have responsibilities across the following functions: \n\nTechnical Leadership and Ingestion Platform Ownership: \nOwn the technical design, architecture, standards and operational performance of the Sense ingestion layer across all data centre sites.\nLead the engineering and operations of exporters, OpenTelemetry collectors, Vector agents and aggregators, and Kafka streaming pipelines.\nDefine and enforce mandatory telemetry labels, tags and resource-attribute schemas across the platform.\nGovern telemetry cardinality budgets to maintain platform performance, query efficiency and cost optimisation.\nProvide technical leadership, task allocation, mentoring and performance guidance to the Senior Associate SRE team.\nEstablish troubleshooting standards, change management practices and documentation quality across the team.\n\n24×7 Operations and Team Management: \nLead the 24×7 operating model for the ingestion layer, including shift rosters, coverage matrices and shift-lead responsibilities.\nAct as the primary shift lead and secondary on-call escalation point for critical ingestion-layer incidents.\nEnsure uninterrupted operational coverage across morning, evening and night shifts, including weekends and public holidays.\nManage shift overlap, backfill arrangements and resource availability to prevent any coverage gaps.\nEnsure structured and documented shift handovers covering open incidents, ingestion health, suppressed alerts and ongoing activities.\nMaintain operational discipline and ensure adherence to defined SLAs and escalation procedures.\n\nOnboarding, Standards and Change Management: \nOwn and maintain onboarding runbooks, verification procedures and approval gates for new telemetry sources.\nReview and approve onboarding sign-offs, ensuring telemetry is reconciled against source systems before any source is declared live.\nCoordinate with application, infrastructure, network and security teams to resolve onboarding dependencies.\nPlan and govern upgrades, patching and version management of ingestion components.\nEnsure all changes follow controlled deployment practices, including tested rollback procedures, approved change windows and freeze-period compliance.\nMaintain configuration consistency and operational standards across all data centre sites.\n\nMonitoring, Incident Management and Escalation: \nDefine the alerting strategy for the ingestion layer, including alert priorities, routing, grouping, escalation timers and silencing controls.\nEnsure effective monitoring of ingestion health, source availability, data freshness, consumer lag and pipeline performance.\nIdentify and eliminate alert noise and silent failures where sources appear healthy but fail to deliver telemetry.\nAct as incident commander or senior technical escalation point for P1 and P2 ingestion incidents.\nDrive incident restoration, coordinate technical teams, lead incident reviews and ensure corrective actions are tracked to closure.\nEnsure adherence to incident acknowledgement and escalation timelines.\nMaintain incident trends, root cause analysis reports and operational risk registers.\n\nCapacity Planning and Reliability Engineering: \nPlan and govern ingestion capacity, including ingestion rates, time-series growth, log volumes, Kafka partitions, retention, buffer sizing and storage headroom.\nForecast infrastructure and capacity requirements ahead of new application onboarding activities.\nIdentify single points of failure and conduct failure-mode analysis to assess platform reliability risks.\nDesign, test and improve resilience, recovery and failover mechanisms across ingestion components.\nDrive continuous reliability improvements and ensure identified risks are addressed within defined timelines.\n\nSecurity, Compliance and Automation: \nEnsure security and compliance across collection endpoints, including loopback-only OTLP binding on application hosts and read-only log mounts.\nImplement sensitive-field masking at aggregators and maintain secure certificate lifecycle management.\nEnforce least-privilege access controls across ingestion components and prevent unauthorised endpoint exposure.\nDrive automation and configuration-as-code practices for agent deployment, pipeline configuration and verification.\nEstablish version-controlled configurations and Git-based review workflows to ensure changes are repeatable, auditable and reversible.\nCoordinate with security and infrastructure teams to maintain platform security standards.\n\nStakeholder Management and Documentation: \nAct as the technical authority for ingestion-related matters while coordinating with application, infrastructure, network and security teams.\nRepresent the platform in Change Advisory Board (CAB), architecture reviews and audit discussions.\nReport ingestion health, telemetry coverage, incident trends, shift performance and operational risks to relevant stakeholders.\nMaintain risk registers, open-item trackers and corrective action plans for the ingestion layer.\nGovern the quality of technical documentation, including architecture notes, runbooks, SOPs, escalation matrices and shift handover procedures.\nEnsure operational knowledge is documented, transferable and not dependent on individual team members.\n\n24×7 Support and SLA Management: \nContinuous Coverage: Ensure 24×7×365 operational coverage across all data centre sites through a structured rotational shift roster.\nShift Rotation: Team members must be available for morning, evening and night shifts, including weekends and declared public holidays.\nShift Leadership: Participate in planned night and weekend shift-lead rotations and secondary on-call escalation responsibilities.\nStructured Handovers: Every shift must complete a documented handover covering open incidents, ingestion health, suppressed alerts and ongoing activities. Verbal-only handovers are not permitted.\nP1 Incidents: Acknowledge within 5 minutes. Automatic escalation after 10 minutes if unacknowledged, followed by time-based escalation up to the Incident Commander for unresolved P1 incidents at 30 minutes.\nP2 Incidents: Acknowledge within 15 minutes, with automatic escalation after 30 minutes.\nEscalation Hierarchy: Primary on-call → Secondary on-call and Team Lead → Duty Manager with incident bridge → Incident Commander for unresolved P1 incidents.\nResolution Accountability: Acknowledgement does not mean resolution. Unresolved incidents must continue through the defined escalation and resolution process.\nRoster Continuity: Ensure planned and unplanned absence backfill through a trained resource bench and sufficient shift overlap to maintain uninterrupted coverage.\n\nRequirements: \nLinux Administration: Deep understanding of Linux internals and production troubleshooting at scale using RHEL / Ubuntu.\nOpenTelemetry (OTel): Strong experience with agent and gateway collector topologies, OTLP, processors, connectors, sampling and resource attribution.\nVector.dev : Expertise in pipeline design, sources, transforms, VRL, buffering strategies, back-pressure and delivery guarantees.\nPrometheus / VictoriaMetrics: Experience with remote write, relabeling, cardinality control, retention and query performance at scale.\nApache Kafka: Strong knowledge of topic and partition design, replication, consumer-group management, lag recovery and retention strategies.\nLog and Trace Platforms: Experience with VictoriaLogs (LogsQL), OpenSearch index and mapping design, and distributed tracing concepts.\nContainerisation: Hands-on experience with Kubernetes and Docker operations.\nGrafana: Expertise in dashboard development, dashboard-as-code, provisioning and alert rule governance.\nAutomation and Configuration Management: Strong Python and Bash scripting skills, configuration-as-code practices, Git-based workflows and code review discipline.\nNetworking and Security: Strong understanding of TCP/IP, routing, DNS, TLS, certificate lifecycle, firewall policies and network plane separation.\nReliability Engineering: Experience with Service Level Objectives (SLOs), failure-mode analysis, capacity planning, resilience testing and post-incident reviews.\nIncident Management: Experience in incident command, high-severity production incident handling, escalation management and stakeholder communication.\nTeam Leadership: Proven experience in mentoring, roster ownership, work allocation and quality reviews for a 24×7 engineering team.\nExperience: 8-12 years in production infrastructure and reliability engineering, including at least 4 years in observability, telemetry or large-scale data pipelines, and 2 years of technical team leadership or on-call rotation management.\nAvailability for rotational shift-lead duties, including night shifts, weekends and secondary on-call responsibilities.\n\nAdditional Requirements: \nStrong technical leadership and decision-making skills in critical production environments.\nAbility to manage high-severity incidents and coordinate multiple technical teams.\nExcellent communication, stakeholder management and documentation skills.\nExperience working with large-scale infrastructure, observability platforms and mission-critical applications.\nAbility to balance hands-on technical troubleshooting with team leadership and operational governance.\n\nSkills\nApache Kafka, Kafka, OTel, OpenTelemetry, SRE, Site Reliability Engineering","description_format":"text","description_chars":10023,"description_truncated":false,"requirements":{"experience_years_min":8,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","Software Testing & QA","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-11-16T17:35:00Z"}],"visa":[],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":3,"expected_fill_days":36,"reasons":["seen:3","velocity","win:early"],"computed_at":"2026-11-20T05:46:30Z"},"pay":null,"html_url":"https://alion.io/job/wits-innovation-lab-lead-sre-engineer","json_url":"https://alion.io/job/wits-innovation-lab-lead-sre-engineer.json","meta":{"generated_at":"2026-11-21T23:28:50Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":39479,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-22T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":49824},"rest":"https://alion.io/mcp/rest/get_company?id=49824"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fwits-innovation-lab-lead-sre-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fwits-innovation-lab-lead-sre-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fwits-innovation-lab-lead-sre-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/wits-innovation-lab-lead-sre-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fwits-innovation-lab-lead-sre-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4386343}},{"tool":"search_jobs","arguments":{"company":"thewitslab.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4386343},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}