Kafka Specialist

Duties & Responsibilities

We are seeking an experienced Technology Specialist – Kafka to design, administer, automate, secure, and optimise enterprise Kafka platforms supporting mission-critical banking applications.

This role is responsible for ensuring the stability, scalability, resilience, and security of Kafka environments while driving automation, DevOps best practices, and cloud adoption.

The successful candidate will combine deep Kafka platform engineering expertise with Linux administration, Infrastructure-as-Code, observability, and cloud engineering to deliver highly available event streaming infrastructure that supports modern banking systems and real-time data integration.

Key Responsibilities

Kafka Platform Administration

• Install, configure, upgrade, patch and maintain enterprise Kafka clusters.
• Administer Confluent Platform and/or Confluent Cloud environments.
• Manage brokers, topics, partitions, replication and consumer groups.
• Optimise Kafka cluster performance, throughput and latency.
• Perform capacity planning and scaling of Kafka infrastructure.
• Troubleshoot broker failures, replication issues and performance bottlenecks.
• Support production incidents and perform root cause analysis.

Platform Engineering

• Design highly available Kafka platforms across multiple environments.
• Implement resilient architectures for business-critical messaging workloads.
• Configure multi-broker clusters with appropriate replication and fault tolerance.
• Support disaster recovery and high availability strategies.
• Maintain platform documentation and operational runbooks.

DevOps & Automation

Develop and maintain automation to reduce manual operational effort.

Responsibilities include:

• Infrastructure as Code
• Platform provisioning
• Automated deployments
• Configuration management
• CI/CD integration
• Environment standardisation

Typical technologies include:

• Terraform
• Ansible
• Python
• Bash
• Azure DevOps
• Git

Linux Platform Administration

Provide specialist Linux administration supporting Kafka infrastructure.

Responsibilities include:

• Performance tuning
• Storage optimisation
• Network troubleshooting
• Service management
• Operating system maintenance
• Capacity management
• Log analysis

Kafka Security

Design and maintain secure Kafka environments.

Responsibilities include:

• TLS/SSL encryption
• SASL authentication
• ACL administration
• Certificate lifecycle management
• Identity integration
• Secure client connectivity
• Compliance with enterprise security standards

Cloud Engineering

Support Kafka deployments across cloud platforms, with a focus on Microsoft Azure.

Responsibilities include:

• Azure networking
• Virtual Machines
• Managed services
• Storage
• Identity integration
• Hybrid deployments
• Cloud security
• Platform optimisation

Experience with Confluent Cloud is highly advantageous.

Monitoring & Observability

Build comprehensive monitoring and operational visibility across Kafka platforms.

Responsibilities include:

• Health monitoring
• Performance dashboards
• Alerting
• Capacity monitoring
• Log aggregation
• Distributed tracing
• Incident diagnostics

Typical tools include:

• Prometheus
• Grafana
• ELK Stack
• Splunk
• Azure Monitor

Incident & Problem Management

• Support production platforms within SLA.
• Lead technical investigations.
• Perform root cause analysis.
• Recommend permanent fixes.
• Participate in major incident resolution.
• Drive platform stability improvements.

Collaboration

Work closely with:

• Platform Engineers
• DevOps Engineers
• Infrastructure Teams
• Cloud Engineers
• Security Teams
• Application Developers
• Architecture Teams
• Operations Teams

Minimum Requirements

Experience

• 6+ years' experience in Systems Engineering, Platform Engineering or Infrastructure Engineering.
• 4+ years' hands-on Kafka administration.
• Enterprise production support experience.
• Experience supporting highly available distributed systems.
• Experience automating infrastructure and operational processes.
• Experience working within DevOps environments.

Required Technical Skills

Kafka

• Apache Kafka
• Confluent Platform
• Confluent Cloud
• Broker administration
• Topic management
• Consumer groups
• Partition management
• Replication
• Cluster upgrades
• Performance tuning
• Capacity planning

Automation

• Python
• Bash
• Terraform
• Ansible
• CI/CD Pipelines
• Azure DevOps
• Git

Operating Systems

• Linux Administration
• Shell scripting
• Performance tuning
• System troubleshooting

Cloud

• Microsoft Azure
• Azure networking
• Azure infrastructure
• Azure Monitor
• Hybrid cloud environments

Kafka Security

• TLS/SSL
• SASL
• ACLs
• Certificates
• Encryption
• Authentication
• Authorisation

Monitoring & Observability

• Prometheus
• Grafana
• ELK Stack
• Splunk
• Azure Monitor
• Log analysis
• Alerting
• Performance monitoring

Preferred Experience

• Banking or Financial Services experience.
• Confluent Cloud implementation experience.
• Enterprise messaging platforms.
• Event-driven architecture.
• Kubernetes or container platforms.
• Docker.
• Site Reliability Engineering (SRE).
• Platform Engineering.
• Infrastructure as Code at enterprise scale.

Nice-to-Have Certifications

• Confluent Certified Developer or Administrator
• Microsoft Azure Administrator
• Microsoft Azure Solutions Architect
• Azure DevOps Engineer
• Red Hat Certified System Administrator (RHCSA)
• Terraform Associate
• Kubernetes Certifications (CKA/CKAD)

Ideal Candidate Profile

The ideal candidate is a senior Platform or Systems Engineer with deep expertise in Apache Kafka and Confluent Platform, capable of administering large-scale event streaming platforms in enterprise environments.

They should have strong Linux administration skills, experience automating infrastructure through Terraform, Ansible, Python, and CI/CD pipelines, and a solid understanding of Kafka security, cloud infrastructure (particularly Azure), and modern observability practices. Experience within a regulated environment such as banking or financial services would be highly advantageous.

    Desired Experience & Qualification Kafka Operations, Kafka Platform Engineering, Event Streaming, Distributed Messaging, Message Broker, Pub/Sub, Event-Driven Architecture, Kafka Brokers, Topics, Partitions, Consumer Groups, Producers,