Senior Product Software Engineer I, Observability

Job description:

As Sr Product Software Engineer I, Observability & Platform, you will build the telemetry pipelines, instrumentation libraries, and self-service platforms that engineering teams across y to deploy, detect, alert, and mitigate in our AWS environments.

You will be part of an organization made up of three complementary teams.

-Technical Operations is responsible for production readiness and service onboarding, alert response and first-line eyes-on-glass monitoring, defining alerts and thresholds, escalation management, backend capacity management and load testing, and third-layer support for technically complex customer issues.

-Reliability Platforms is responsible for the traffic, compute, and streaming platform layers, and for the reliability tooling that engineers reliability, resiliency, and self-healing into our production services and automates operational processes.

-Observability & Platform is responsible for telemetry, instrumentation, and alerting standards, for the internal self-service tooling and automation platforms that enable teams across the organization to deploy, detect, alert, and mitigate in our AWS environments with reliability and velocity, and for the operational data, reliability reporting, and anomaly detection that make those platforms smarter over time.

You must understand the demands of managing 24x7 applications at a large scale with well-defined operational objectives. You bring hands-on expertise across AWS, systems programming, the software development lifecycle, incident response and management, and observability tooling such as Datadog, CloudWatch, and Grafana. Within Observability & Platform you will own platform services end to end, mentor engineers, take technical direction from the Lead Product Software Engineer, and partner with Technical Operations and Reliability Platforms so standards land as paved paths. You will report to the Director, Product Software Engineering.

Responsibilities and Duties of the Role:

• Design, build, and own telemetry pipelines, instrumentation libraries, and alerting and dashboard standards used across the organization
• Develop self-service tooling, including backend APIs and user-facing interfaces, that enables internal teams to deploy, detect, alert, and mitigate in our AWS environments
• Implement and troubleshoot appropriate IAM permissions for user access and workloads, and champion best practices and security standards for operating in the cloud
• Build and maintain operational data pipelines powering reliability KPIs and dashboards, and contribute to anomaly detection and intelligent alerting capabilities
• Provide guidance and recommendations to development teams on instrumentation, service design, and deployments
• Participate in the on-call rotation for platform services, and mentor engineers through pairing and code review

Required Education, Experience/Skills/Training:

Basic Qualifications:

• 5+ years of professional software engineering experience, including internal platforms, developer tooling, or observability systems
• Strong Python development skills; you write production-quality code, not just scripts
• Hands-on experience with observability tooling such as Datadog, Prometheus, Grafana, CloudWatch, or OpenTelemetry, including instrumenting services and defining alerts and thresholds
• Experience deploying and running AWS-based solutions, including hands-on IAM administration and policy creation, and infrastructure as code with Terraform or CloudFormation
• Experience with platform engineering, including developing APIs and designing database schemas
• Experience with CI/CD tooling such as Jenkins, GitHub Actions, GitLab CI, or Spinnaker
• Familiarity with Docker, containerization concepts, and Unix/Linux systems, along with operational troubleshooting tools and techniques
• Clear, succinct written and verbal communication with technical and non-technical stakeholders, and a practical, collaborative approach when working with engineers

Preferred Qualifications:

• Frontend experience with React and TypeScript for internal tooling
• Experience with data and streaming data ecosystems such as Spark, Databricks, Airflow, or Kinesis
• Experience with machine learning applied to operations, such as anomaly detection or alert correlation, or with GenAI, LLMs, and agents
• Familiarity with SSL certificate management and with general and AWS-specific networking concepts, including VPC constructs, firewalling, and DNS
• Experience with cloud cost management and resource scaling
• AWS certifications

Required Education:

Bachelor's degree in Computer Science, Information Systems, Software, Electrical or Electronics Engineering, or a comparable field of study, or equivalent training or work experience

Skills

AWS, CloudWatch, Datadog, Grafana, OpenTelemetry, Prometheus, Python