Observability Engineer

The Observability Expert is the subject-matter expert for all monitoring and telemetry platforms used in this engagement. This is a critical-path role that must be onboarded in Week 1. Without deep, hands-on knowledge of Datadog, Nagios, Azure Monitor, and SolarWinds, the alert normalization schema for Track 2 cannot be designed, and the metric taxonomy for Track 1's Usage Agent will be built on wrong assumptions. This role advises all three tracks and owns the dependency graph design.

Responsibilities:

• Design the unified alert normalization schema covering Nagios, Datadog, Azure Monitor, and SolarWinds alert payloads.
• Define the metric taxonomy for the Usage Agent in Track 1: which metrics to pull, from which APIs, and at what granularity.
• Own the service dependency graph design: CI relationship model, node and edge types, and refresh strategy.
• Advise the Track 2 team on alert correlation rules, suppression logic, and time-window configuration.
• Configure and validate Datadog API integration for both metric pull (Track 1) and alert ingestion (Track 2).
• Review Nagios and SolarWinds webhook configurations and advise on polling fallback approaches.
• Validate the Azure Monitor alert schema and advise on event vs. metric alert handling differences.
• Define false positive thresholds and suppression tuning parameters for Track 2.
• Support the observability dashboard design in Track 1 with expertise on what usage data is meaningful.
• Provide on-call advisory support across the engagement for any monitoring or telemetry issues.

Requirements:

• 5+ years hands-on experience in an SRE, platform engineering, or observability engineering role.
• Deep expertise in Datadog: API usage, metric queries, service maps, alert webhook configuration, and monitor management.
• Working knowledge of Nagios: alert payload structure, configuration check, and notification webhooks.
• Experience with Azure Monitor: alert rules, action groups, log analytics queries, and metric alert schemas.
• Understanding of SolarWinds alert structures and webhook/API integration patterns.
• Experience designing or working with CMDB CI relationship models in ServiceNow.
• Strong understanding of alert correlation concepts: deduplication, parent-child suppression, noise reduction.

Nice to Have:

• Experience with OpenTelemetry instrumentation and collector configuration.
• Familiarity with the Backstage.io software catalog and its service dependency model.
• Knowledge of Prometheus and Grafana in addition to the platforms listed above.
• Experience with graph-based data models for dependency mapping.

Skills

Azure, Azure Monitor, Datadog, Nagios