Track Manager - Splunk, Bash (Scripting Language), Python

Job Summary

Job Description: L3 APM Engineer - Splunk AppD / Splunk O11y Cloud Role: L3 Application Performance Monitoring Engineer / APM SME Experience: 7-10+ years in IT, with 4-6+ years in APM/Observability Primary Technologies: Splunk AppDynamics, Splunk Observability Cloud Function: Observability / Application Performance Management / Infrastructure Operations Support Model: Enterprise 24x7 environment, including on-call support as required Role Summary We are looking for an experienced L3 APM/Observability Engineer responsible for engineering, administration, troubleshooting, optimization, and operational support of Splunk AppDynamics and Splunk Observability Cloud. The engineer will serve as an L3/SME escalation point, supporting business-critical applications, resolving complex performance issues, improving application observability, developing dashboards and alerts, performing RCA, and driving proactive monitoring and automation. The role requires strong understanding of APM, distributed tracing, metrics, logs, infrastructure monitoring, cloud-native technologies, application architecture, and end-to-end transaction monitoring.

Key Responsibilities

Key Responsibilities 1. APM & Observability Platform Administration Administer and support Splunk AppDynamics and Splunk Observability Cloud environments. Configure application monitoring, business transactions, service endpoints, health rules, policies, alerts, and dashboards. Deploy, configure, upgrade, and troubleshoot APM agents. Manage application onboarding and instrumentation across enterprise environments. Maintain platform health, integrations, agent compatibility, access controls, and configuration standards. Support monitoring across on-premises, cloud, and hybrid environments. 2. Splunk AppDynamics Configure and troubleshoot Java, .NET, Machine and Database Agents. Create and tune Business Transactions, Health Rules, Policies, Actions, Dashboards and Service Endpoints. Perform transaction snapshots and call-graph analysis. Identify application bottlenecks, slow transactions, exceptions, backend latency, DB issues, and external service dependencies. Troubleshoot agent-controller connectivity and instrumentation issues. Optimize AppDynamics configurations to reduce monitoring overhead and noise. 3. Splunk Observability Cloud Configure and administer APM, Infrastructure Monitoring and related observability capabilities. Build detectors, alerts, dashboards, and service-level monitoring. Work with OpenTelemetry collectors and instrumentation. Analyze distributed traces and application/service dependencies. Configure metrics and telemetry ingestion from cloud-native and traditional infrastructure. Support observability for microservices, Kubernetes, containers, APIs, and cloud workloads. 4. L3 Troubleshooting & Performance Analysis Act as the SME for complex performance incidents involving: High application response times. JVM/.NET performance issues. Memory and CPU bottlenecks. Garbage collection issues. Thread contention and deadlocks. Database latency and slow queries. API/service latency. Connection-pool exhaustion. External service dependencies. Microservices performance degradation. Kubernetes/container performance. Network and infrastructure-related application latency. Correlate application, infrastructure, database, network, and cloud telemetry to identify the root cause rather than simply reporting symptoms. 5. Incident, Problem & RCA Management Provide L3 support for P1/P2 application performance incidents. Participate in major incident bridges and provide real-time performance analysis. Perform detailed Root Cause Analysis for critical or recurring incidents. Identify monitoring gaps discovered during incidents and implement corrective actions. Work closely with application development, middleware, database, infrastructure, network, cloud, and security teams. Drive proactive problem management using APM and observability insights. 6. Dashboarding, Alerting & SLO Monitoring Develop executive, operational, and application-specific dashboards. Define meaningful alert thresholds and reduce false-positive alerts. Create monitoring for critical business transactions and customer journeys. Establish appropriate SLIs/SLOs around availability, latency, error rate, and throughput. Implement proactive and predictive monitoring wherever feasible. 7. Application Onboarding & Instrumentation Assess applications for observability requirements. Define monitoring standards for application onboarding. Configure appropriate agents, collectors, instrumentation, dashboards, and alerts. Coordinate deployments with application and platform teams. Validate telemetry and monitoring coverage after implementation. Maintain onboarding documentation and operational runbooks. 8. Automation & Integration Automate repetitive APM administration and health-check activities. Build integrations with enterprise ITSM and notification platforms. Experience with REST APIs, Python, PowerShell, shell scripting, Terraform, or Ansible is preferred. Support integration with platforms such as ServiceNow and collaboration/notificati

Skill Requirements

Required Technical Skills Technology Area Required Expertise APM Splunk AppDynamics Observability Splunk Observability Cloud Telemetry Metrics, traces, logs Standards OpenTelemetry Application Java and .NET monitoring Architecture APIs, microservices, distributed applications Cloud Native Kubernetes, Docker/containers OS Linux and Windows Database SQL/DB performance monitoring Networking TCP/IP, DNS, HTTP/HTTPS fundamentals Automation REST APIs, scripting ITSM Incident, Problem and Change Management Performance RCA and application performance analysis Preferred Skills Splunk Enterprise / Splunk Cloud exposure. Synthetic monitoring and Real User Monitoring experience. Understanding of OpenTelemetry architecture and collectors. Kubernetes/OpenShift monitoring experience. Experience with AWS, Microsoft Azure, or Google Cloud. Experience with ServiceNow integration. Knowledge of CI/CD and DevOps practices. Grafana, Prometheus, Dynatrace, Datadog, New Relic or other observability platform exposure is advantageous. Understanding of enterprise monitoring architecture and observability strategy.

Other Requirements

Key L3 Competencies The candidate should be able to independently handle: Complex AppDynamics agent and instrumentation issues. Splunk Observability Cloud configuration and troubleshooting. Distributed tracing and transaction-flow analysis. Application response-time degradation. JVM/.NET performance troubleshooting. Database and backend dependency analysis. OpenTelemetry collector/instrumentation issues. Kubernetes and microservices observability. P1/P2 incident troubleshooting and RCA. Alert tuning and noise reduction. Dashboard and business-transaction monitoring. Application onboarding and monitoring design. Cross-domain performance correlation. Automation of observability operational activities. Technical mentoring and guidance to L1/L2 engineers. Behavioral Competencies Strong analytical and troubleshooting skills. Ability to independently lead technical troubleshooting during critical incidents. Strong ownership and accountability. Good customer and stakeholder communication. Ability to translate APM data into clear application/business impact. Strong documentation skills for RCA, SOP, runbooks and knowledge articles. Ability to mentor L1/L2 teams. Ability to collaborate effectively with application, cloud, infrastructure, database and network SMEs. Preferred Certifications Relevant Splunk Observability certification/training. AppDynamics technical certification or equivalent hands-on expertise. OpenTelemetry knowledge/certification where applicable. AWS / Azure cloud certification. ITIL Foundation.