SRE - Senior DevOps Engineer

We’re looking for a Senior DevOps Engineer to own our monitoring and alerting in Datadog. You’ll make sure we know when something is wrong, that the right people hear about it quickly, and that the issues behind the alerts get fixed. You’ll also work with client´s InfoSec team to align on any configuration needed.

Responsibilities

• Own the day-to-day operation of our Datadog platform, including dashboards, monitors, log pipelines, APM, synthetic tests, and Real User Monitoring, keeping coverage accurate as our services evolve.

• Proactively sift through Datadog logs, error tracking, traces, and metrics to find failures, regressions, and anomalies that have not triggered an alert, then triage, reproduce, and drive them to resolution with the owning team.

• Tune alert thresholds, monitor logic, and notification routing to cut noise and false positives, so real customer-impacting issues page the right people fast.

• Define, measure, and report on service level indicators and objectives for key customer-facing services.

• Collaborate with engineering teams on the monitoring, logging, and tracing setup their services need, and implement the configuration changes required.

• Write post-incident reviews, track follow-up actions to completion, and feed lessons back into monitors, runbooks, and processes.

• Improve the reliability, scalability, and cost efficiency of our cloud infrastructure, CI/CD pipelines, and release processes, automating repetitive operational work where it makes sense.

• Maintain runbooks, escalation paths, and operational documentation so the team can respond quickly and consistently.

• Work with client´s enterprise InfoSec team to implement the configuration changes needed to close security findings, including SSL/TLS cleanup, removal of exposed technology and version banners, security headers (HSTS, CSP, X-Frame-Options), and DNS hygiene (DNSSEC, SPF/DKIM/DMARC, dangling records), tracking each item through to verified closure.

What We're Looking For:

• 5+ years of experience in DevOps, site reliability, platform engineering, or infrastructure operations.

• Strong, hands-on Datadog expertise (or a comparable platform such as New Relic, Grafana, Splunk, or Dynatrace) : dashboards, monitors, log management, APM, and alert tuning. You can look at an alert and know what it means and what should change.

• Strong troubleshooting and root-cause analysis skills across logs, traces, and metrics.

• Experience running services in a major cloud provider (AWS, GCP, or Azure), with solid knowledge of networking, containers, and Linux.

• Experience with infrastructure as code (Terraform, CloudFormation, or Pulumi) and CI/CD tools such as GitHub Actions.

• Comfort with scripting (Bash or Python) to automate tasks. Software development is not required. @Lucas Do Carmo Fontes, aqui faria sentido o bash/phyton?

• Working knowledge of security configuration best practices (TLS, security headers, DNS), and the ability to translate InfoSec findings into the changes needed.

• Experience with on-call, incident response, and post-incident reviews.

• Clear written and verbal communication, including explaining reliability trade-offs to non-technical stakeholders.

Bonus Points:

• Experience with Kubernetes or other container orchestration platforms.
• Exposure to Datadog-specific features such as Watchdog, Error Tracking, Continuous Profiler, SLO management, Workflow Automation, or the Datadog API and Terraform provider.
• Experience managing observability costs, including log indexing strategies, custom metric hygiene, and retention policies.
• Familiarity with application security practices and compliance requirements in a regulated industry such as insurance or financial services.
• Experience with chaos engineering, load testing, or capacity planning.