R&D Engineer 3, Software

We are looking for a Senior Observability Engineer to design, build, and continuously improve monitoring and observability for our enterprise environment. Most of the work focuses on on-premises systems: physical servers, virtualized infrastructure, and on-prem Kubernetes clusters, along with the infrastructure layers underneath them. Some cloud use cases are also in scope. Your goal is to give system owners and business stakeholders an accurate, actionable view of the health and uptime of the systems they depend on.

You will also help shape how we apply agentic AI to observability, moving from dashboards and alerts toward intelligent triage, correlation, and assisted remediation.

Responsibilities:

• Design end-to-end monitoring and observability solutions for enterprise applications running on VMs, Kubernetes, and bare-metal servers, and extend them to cloud workloads where needed.
• Define a layered monitoring strategy that covers hardware, OS, virtualization, network, storage, platform services, and applications.
• Set standards for metrics, logs, alerting, dashboards, and service health models, including SLIs and SLOs where they apply.
• Make sound architectural decisions on tool roles, data flows, retention, scalability, and high availability across the observability stack.
• Define alert escalation policies, incident response workflows, and event correlation strategies to reduce operational noise and improve service reliability.
• Design service topology, dependency mapping, and business service monitoring across infrastructure and application layers.
• Secure observability platforms and integrations through access controls, authentication, encryption, and vulnerability remediation. Maintain compliance with enterprise security standards through platform hardening, audit readiness, and regular security reviews.
• Build and maintain monitoring with Zabbix, covering templates, discovery, triggers, proxies, and integrations.
• Design and maintain log collection, parsing, enrichment, and routing pipelines into Splunk, including forwarders, sourcetypes, field extractions, indexes, and searches.
• Contribute to metrics and visualization with Prometheus and Grafana, particularly for Kubernetes workloads and unified dashboards.
• Automate the configuration and onboarding of monitoring using infrastructure-as-code and APIs.
• Design and implement AI agents that support observability workflows. Examples include alert triage and deduplication, cross-source event correlation, root-cause hypotheses, incident summarization, and runbook-driven remediation with human-in-the-loop approval.
• Integrate LLM-based agents with monitoring and logging platforms through APIs and tool protocols such as MCP.
• Define guardrails, permission boundaries, and auditability for AI-driven actions on production systems.
• Measure the real impact of these agents on operations, for example on MTTD, MTTR, alert noise, and on-call load.
• Work closely with system owners to understand architecture, failure modes, and dependencies, and to translate them into meaningful monitoring.
• Partner with consumers and stakeholders of those systems to define what "healthy" and "available" mean to them, and to make sure observability gives them the visibility they need.
• Provide technical leadership and mentoring, and run knowledge sharing across operations, infrastructure, and development teams.
• Regularly review alert quality, coverage gaps, and signal-to-noise ratio, and tune or retire monitoring accordingly.
• Use incidents and post-mortems as input to improve detection and response.
• Track the evolution of the observability toolset and the wider technology landscape.

Qualifications:

Required:

• 7+ years in infrastructure, operations, or observability engineering, with at least 3 years focused on monitoring and observability, primarily in on-premises enterprise environments.
• Hands-on, production-level experience with Zabbix and Splunk.
• Proven experience monitoring physical servers, virtualization platforms (e.g., VMware, Hyper-V, KVM), and Kubernetes clusters.
• Strong understanding of infrastructure layers: compute, OS (Linux and Windows), networking, storage, and the dependencies between them.
• Solid experience with log collection and processing at scale: parsing, normalization, enrichment, retention, and search optimization.
• Practical experience applying agentic AI (LLM agents, tool use, automation) to monitoring, operations, or incident management, along with a working understanding of how these systems behave and where their risks lie.
• Experience designing monitoring architecture, not only operating existing tools.
• Experience defining alert escalation policies, incident response workflows, and event correlation strategies to reduce operational noise and improve service reliability.
• Experience designing service topology, dependency mapping, and business service monitoring across infrastructure and application layers.
• Working knowledge of securing observability platforms, including access control, authentication, encryption, and hardening.
• Strong scripting skills (e.g., Python, Bash, PowerShell) and comfort working with REST APIs.
• Excellent communication skills and the ability to work with both technical owners and non-technical stakeholders.

Nice to Have:

• Experience with Prometheus (exporters, alerting rules, Alert manager, long-term storage).
• Experience with Grafana (dashboard design, multi-source visualization, alerting).
• Experience monitoring cloud workloads (AWS, Azure, or GCP) and hybrid environments.
• Software development experience, such as building integrations, tools, exporters, or services.

Skills

Kubernetes, Splunk, Zabbix