SeniorAdministrator - Splunk, Bash (Scripting Language), Python

Job Summary

Role Summary The Event Management Engineer will be responsible for implementing and managing enterprise Event Management and AIOps capabilities using Splunk ITSI. The role will focus on consolidating events from multiple monitoring platforms, reducing alert noise, performing event correlation, configuring service health and KPIs, and enabling proactive identification of infrastructure and application issues. The engineer will work closely with Infrastructure, Application, Network, Cloud, Security, Observability, Automation, and ITSM teams to improve operational visibility and accelerate incident detection and resolution.

Key Responsibilities

Key Responsibilities 1. Splunk ITSI Administration & Engineering Configure, administer, maintain, and optimize Splunk ITSI. Build and manage Services, KPIs, Service Trees, Glass Tables, Entity Rules, Thresholds, and Health Scores. Configure and manage Episode Review and event aggregation policies. Develop Notable Event Aggregation Policies (NEAPs) to intelligently group related alerts. Troubleshoot ITSI data ingestion, KPI calculation, entity mapping, and service-health issues. Perform ITSI content/configuration lifecycle management across development, test, and production environments. 2. Enterprise Event Management Integrate events and alerts from infrastructure, application, network, cloud, security, and monitoring platforms. Normalize and enrich events with operational information such as CI, application, environment, support group, severity, and business service. Design event correlation, deduplication, suppression, and aggregation rules. Reduce duplicate and non-actionable alerts through continuous event-noise reduction. Create correlation logic to identify probable root causes and related downstream events. Establish standards for alert severity, priority, ownership, and escalation. 3. Splunk Engineering Develop and optimize SPL searches, dashboards, alerts, reports, macros, lookups, and data models. Configure data onboarding using Universal Forwarders, Heavy Forwarders, HEC, APIs, syslog, and other integration methods. Support troubleshooting across Splunk search, indexing, forwarding, and ITSI components. Optimize searches and ITSI configurations for performance and scalability. 4. Monitoring & Observability Integration Integrate Splunk ITSI with enterprise monitoring and observability platforms such as: Microsoft Azure Monitor AWS CloudWatch Dynatrace AppDynamics SolarWinds SCOM VMware monitoring solutions Network monitoring platforms Database and middleware monitoring tools Custom application monitoring solutions Create a unified event-management layer to provide cross-domain infrastructure and application visibility. 5. ITSM Integration & Automation Integrate Splunk ITSI with ServiceNow or equivalent ITSM platforms. Enable automated incident creation from qualified events/episodes. Configure event-to-ticket mapping, enrichment, assignment, and priority rules. Support automated remediation/runbook integration where applicable. Work with automation teams to identify opportunities for self-healing and event-triggered remediation. Ensure proper closure/synchronization between monitoring events and ITSM incidents. 6. Event Optimization & AIOps Analyze alert volumes and identify opportunities for event reduction and automation. Implement event correlation and service-context-based analysis. Improve signal-to-noise ratio by eliminating duplicate, informational, transient, or non-actionable alerts. Identify recurring event patterns and recommend permanent monitoring or automation improvements. Support proactive/predictive operations initiatives through Splunk ITSI. 7. Incident & Major Incident Support Provide event and service-health analysis during P1/P2 and Major Incidents. Use ITSI service dependencies, KPIs, events, and Splunk logs to support impact analysis and fault isolation. Assist Incident, Problem, and Technical teams with timeline reconstruction and root-cause investigation. Identify monitoring gaps following critical incidents and implement corrective actions. 8. Governance & Continuous Improvement Define and maintain event-management standards, naming conventions, severity models, thresholds, and operational procedures. Conduct periodic reviews of noisy alerts, stale alerts, unused rules, and monitoring gaps. Maintain technical documentation, architecture diagrams, SOPs, and operational runbooks. Track Event Management KPIs such as: Event/alert volume Noise reduction percentage Event-to-incident conversion ratio Duplicate-event reduction Auto-correlation percentage Auto-remediation percentage MTTD MTTR Techni

Skill Requirements

Key Responsibilities 1. Splunk ITSI Administration & Engineering Configure, administer, maintain, and optimize Splunk ITSI. Build and manage Services, KPIs, Service Trees, Glass Tables, Entity Rules, Thresholds, and Health Scores. Configure and manage Episode Review and event aggregation policies. Develop Notable Event Aggregation Policies (NEAPs) to intelligently group related alerts. Troubleshoot ITSI data ingestion, KPI calculation, entity mapping, and service-health issues. Perform ITSI content/configuration lifecycle management across development, test, and production environments. 2. Enterprise Event Management Integrate events and alerts from infrastructure, application, network, cloud, security, and monitoring platforms. Normalize and enrich events with operational information such as CI, application, environment, support group, severity, and business service. Design event correlation, deduplication, suppression, and aggregation rules. Reduce duplicate and non-actionable alerts through continuous event-noise reduction. Create correlation logic to identify probable root causes and related downstream events. Establish standards for alert severity, priority, ownership, and escalation. 3. Splunk Engineering Develop and optimize SPL searches, dashboards, alerts, reports, macros, lookups, and data models. Configure data onboarding using Universal Forwarders, Heavy Forwarders, HEC, APIs, syslog, and other integration methods. Support troubleshooting across Splunk search, indexing, forwarding, and ITSI components. Optimize searches and ITSI configurations for performance and scalability. 4. Monitoring & Observability Integration Integrate Splunk ITSI with enterprise monitoring and observability platforms such as: Microsoft Azure Monitor AWS CloudWatch Dynatrace AppDynamics SolarWinds SCOM VMware monitoring solutions Network monitoring platforms Database and middleware monitoring tools Custom application monitoring solutions Create a unified event-management layer to provide cross-domain infrastructure and application visibility. 5. ITSM Integration & Automation Integrate Splunk ITSI with ServiceNow or equivalent ITSM platforms. Enable automated incident creation from qualified events/episodes. Configure event-to-ticket mapping, enrichment, assignment, and priority rules. Support automated remediation/runbook integration where applicable. Work with automation teams to identify opportunities for self-healing and event-triggered remediation. Ensure proper closure/synchronization between monitoring events and ITSM incidents. 6. Event Optimization & AIOps Analyze alert volumes and identify opportunities for event reduction and automation. Implement event correlation and service-context-based analysis. Improve signal-to-noise ratio by eliminating duplicate, informational, transient, or non-actionable alerts. Identify recurring event patterns and recommend permanent monitoring or automation improvements. Support proactive/predictive operations initiatives through Splunk ITSI. 7. Incident & Major Incident Support Provide event and service-health analysis during P1/P2 and Major Incidents. Use ITSI service dependencies, KPIs, events, and Splunk logs to support impact analysis and fault isolation. Assist Incident, Problem, and Technical teams with timeline reconstruction and root-cause investigation. Identify monitoring gaps following critical incidents and implement corrective actions. 8. Governance & Continuous Improvement Define and maintain event-management standards, naming conventions, severity models, thresholds, and operational procedures. Conduct periodic reviews of noisy alerts, stale alerts, unused rules, and monitoring gaps. Maintain technical documentation, architecture diagrams, SOPs, and operational runbooks. Track Event Management KPIs such as: Event/alert volume Noise reduction percentage Event-to-incident conversion ratio Duplicate-event reduction Auto-correlation percentage Auto-remediation percentage MTTD MTTR Technic

Other Requirements

Certifications - Preferred Splunk Core Certified Power User Splunk Enterprise Certified Admin Splunk ITSI-related training/certification ITIL Foundation Azure / AWS certifications Key Success Measures An effective Event Management Engineer should demonstrate measurable improvements in: Lower event noise → Better correlation → Faster fault identification → Fewer unnecessary incidents → Improved MTTD/MTTR → Higher automation Suggested capability targets Reduction of repetitive/non-actionable alerts Increased percentage of alerts correlated into meaningful episodes Increased automated incident generation for validated scenarios Reduction in manual event triage effort Improved service-health visibility through ITSI Increased coverage of automated remediation use cases