Track Manager - Splunk, Bash (Scripting Language), Python

Job Summary

We are looking for an experienced Splunk L3 Administrator responsible for administration, engineering, troubleshooting, optimization, and operational support of enterprise Splunk environments. The candidate should have strong expertise across Splunk Enterprise, Splunk Universal/Heavy Forwarders, Search Head Clustering, Indexer Clustering, data onboarding, SPL, security, performance tuning, and platform upgrades. The role will act as an L3 escalation point, drive complex problem resolution, support major incidents, implement automation, and ensure the Splunk platform remains highly available, secure, scalable, and optimized.

Key Responsibilities

Key Responsibilities Splunk Administration & Engineering Administer and support large-scale Splunk Enterprise environments across production and non-production. Manage Search Heads, Indexers, Cluster Managers, Deployment Servers, License Managers, Heavy Forwarders, and Universal Forwarders. Configure and maintain Indexer Clustering and Search Head Clustering. Perform installation, configuration, patching, upgrades, migrations, and platform lifecycle activities. Manage Splunk configuration files including inputs.conf, outputs.conf, props.conf, transforms.conf, server.conf, indexes.conf, and limits.conf. Perform Splunk application and configuration deployment through appropriate deployment mechanisms. Data Onboarding & Integration Onboard data from Windows, Linux, network devices, security platforms, applications, databases, cloud platforms, APIs, and syslog sources. Configure source types, field extraction, event parsing, timestamp recognition, line breaking, filtering, and routing. Troubleshoot ingestion issues across Universal Forwarders, Heavy Forwarders, syslog infrastructure, and Splunk Indexers. Monitor ingestion volume and licensing consumption and optimize data onboarding where required. SPL, Dashboards & Monitoring Develop and troubleshoot complex Splunk Search Processing Language (SPL) queries. Create and optimize dashboards, reports, alerts, scheduled searches, and knowledge objects. Optimize inefficient searches to improve platform performance. Support operational and application teams in establishing monitoring and alerting use cases. Performance & Capacity Management Monitor overall Splunk platform health, indexing rates, search performance, resource utilization, and ingestion latency. Perform capacity planning for compute, storage, daily ingestion, retention, and license utilization. Troubleshoot blocked queues, skipped searches, indexing delays, high CPU/memory utilization, and search performance issues. Use Splunk Monitoring Console and internal logs for proactive health monitoring and troubleshooting. Security & Access Management Implement and maintain RBAC, user roles, authentication, and authorization. Support integration with enterprise identity solutions such as LDAP/SAML/SSO. Manage certificates and TLS configuration for secure Splunk communication. Ensure adherence to enterprise security, compliance, hardening, and audit requirements. Incident, Problem & Change Management Act as L3 SME and escalation point for complex Splunk platform issues. Participate in P1/P2 major incident troubleshooting and recovery. Perform root cause analysis and develop corrective/preventive actions. Execute production changes following ITIL-based change management processes. Work with application, infrastructure, network, security, database, and cloud teams for end-to-end troubleshooting. Automation Develop automation using Shell scripting, Python, REST APIs, Git, and Ansible where applicable. Automate repetitive Splunk administration, health checks, configuration deployment, and operational activities. Identify opportunities for proactive monitoring and self-healing.

Skill Requirements

Required Technical Skills Mandatory: Splunk Enterprise Administration Search Head Clustering Indexer Clustering Universal and Heavy Forwarders Splunk Deployment Server Data onboarding and ingestion troubleshooting Advanced SPL props.conf and transforms.conf Index and retention management Splunk Monitoring Console RBAC / LDAP / SAML Linux administration Shell scripting REST APIs Performance tuning and capacity management Production incident troubleshooting Good to Have: Splunk Enterprise Security (ES) Splunk IT Service Intelligence (ITSI) Splunk Observability AWS / Azure integrations Kubernetes/container monitoring Ansible Python Git / CI/CD Syslog platforms ServiceNow integration

Other Requirements

Required Technical Skills Mandatory: Splunk Enterprise Administration Search Head Clustering Indexer Clustering Universal and Heavy Forwarders Splunk Deployment Server Data onboarding and ingestion troubleshooting Advanced SPL props.conf and transforms.conf Index and retention management Splunk Monitoring Console RBAC / LDAP / SAML Linux administration Shell scripting REST APIs Performance tuning and capacity management Production incident troubleshooting Good to Have: Splunk Enterprise Security (ES) Splunk IT Service Intelligence (ITSI) Splunk Observability AWS / Azure integrations Kubernetes/container monitoring Ansible Python Git / CI/CD Syslog platforms ServiceNow integration