Confluent Kafka Developer

Job description:

We are looking for an experienced Confluent Kafka Developer having 8+ years of experience.

Key Responsibilities:

• Cluster Architecture & Solutioning: Design, architect, and optimize Confluent Kafka clusters stretched across four geographically distributed data centers. Define topic architecture, partitioning strategy, replication factor, and rack-awareness configurations. Provide solutioning inputs, capacity planning, and sizing for current and projected TPS loads. Collaborate with enterprise architects and application teams on event-driven and streaming solutions.
• Deployment & Configuration: Lead end-to-end deployment of Confluent Kafka components (Brokers, Zookeeper/KRaft, Schema Registry, Kafka Connect, ksqlDB, REST Proxy, Control Center, and MirrorMaker/Cluster Linking). Manage cluster upgrades, patching, and version migrations with minimal downtime. Implement security configurations (SSL/TLS, SASL/Kerberos, ACLs, RBAC). Configure and manage multi-DC replication strategies.
• Production Support & Troubleshooting (L3): Act as the highest level of technical escalation (L3) for production incidents, providing RCA and permanent fixes. Monitor and manage an environment supporting ~4.5 crore accounts with a throughput of 5,000-10,000 TPS. Troubleshoot broker failures, ISR issues, consumer lag, partition rebalancing, and replication issues. Perform deep-dive performance tuning (JVM, OS, network, producer/consumer). Drive proactive capacity planning.
• Monitoring & Operations: Set up and manage monitoring, alerting, and observability using Confluent Control Center, Prometheus, Grafana, and Splunk/ELK. Track operational metrics like broker health, under-replicated partitions, consumer lag, latency, and throughput. Ensure adherence to banking regulatory, audit, and compliance requirements. Participate in DR drills, BCP, and periodic failover testing.
• Ecosystem & Integration Knowledge: Demonstrate strong understanding of the surrounding banking ecosystem (core banking systems, payment gateways/switches, NPCI/RTGS/NEFT/IMPS/UPI rails, fraud engines, reconciliation systems). Partner with cross-functional teams to resolve issues impacting Kafka pipelines. Support integration via Kafka Connect, custom producers/consumers, and stream processing frameworks.
• Documentation & Governance: Create and maintain runbooks, SOPs, architecture diagrams, and knowledge base articles. Enforce platform best practices and governance. Mentor L1/L2 support engineers.

Required Skills & Qualifications:

• Technical Skills: 8-10 years total IT experience; minimum 4-5 years of dedicated Kafka/Confluent Platform experience. Hands-on expertise with Confluent Platform components. Proven experience managing stretched/multi-DC Kafka clusters and cross-DC replication. Strong understanding of Kafka internals, performance tuning, capacity planning, and troubleshooting. Working knowledge of Linux administration, networking fundamentals, and scripting (Shell/Python). Familiarity with security implementations (TLS/SSL, SASL, Kerberos, ACLs, RBAC) and monitoring tools. Exposure to stream processing frameworks is a plus.
• Domain Skills: Prior experience in banking/payments/fintech environments with an understanding of core banking, payment processing, and settlement/reconciliation workflows. Familiarity with regulatory and compliance requirements (RBI guidelines, data security standards, audit trails). Understanding of the broader banking technology ecosystem.
• Soft Skills: Strong analytical and problem-solving skills under pressure (P1/P2 situations). Excellent communication skills. Ability to lead RCA discussions and drive permanent fixes. Willingness to participate in on-call rotations.

Skills

ACLs, Confluent, KRaft, Kafka, Kafka Connect, Kerberos, SASL, SSL, Schema Registry, TLS, Zookeeper, ksqlDB