Observability Engineer

The TP ICAP Group is a world leading provider of market infrastructure.

Our purpose is to provide clients with access to global financial and commodities markets, improving price discovery, liquidity, and distribution of data, through responsible and innovative solutions.

Through our people and technology, we connect clients to superior liquidity and data solutions.

The Group is home to a stable of premium brands. Collectively, TP ICAP is the largest interdealer broker in the world by revenue, the number one Energy & Commodities broker in the world, the world’s leading provider of OTC data, and an award winning all-to-all trading platform.

Founded in London in 1866, the Group operates from more than 60 offices in 27 countries. We are 5,200 people strong. We work as one to achieve our vision of being the world’s most trusted, innovative, liquidity and data solutions specialist.

Role Overview

We are establishing a dedicated Observability Engineering function to create a step change in how TP ICAP understands, manages and improves the health, performance and resilience of its technology estate. Reporting to the Head of Observability, the Observability Engineer will be a hands-on member of the team responsible for implementing enterprise standards, tooling and engineering practices for observability across applications and infrastructure. The role will help evolve the strategic Grafana LGTM platform from a like-for-like monitoring capability into a data-led service that provides end-to-end insight across complex business and technology value streams.

The Observability Engineer will implement consistent standards for metrics, logs and traces, develop reusable engineering patterns and best practices, and partner with Infrastructure and Production Engineering teams across multiple lines of business to raise observability maturity. The team will have direct engineering and operational ownership of the observability tooling stack, including OpenTelemetry collectors, the Grafana LGTM stack and other enterprise observability tooling. The role will build and maintain consistent and decision-focused dashboards, richer instrumentation, event correlation across dependent applications and infrastructure, end-to-end visibility of order and trade flows across multiple systems and components, and monitoring of low-latency market data feeds.

This is a hands-on engineering role. As an Observability Engineer you will design, implement, operate and support solutions; contribute to a trusted data foundation; engineer automated health and latency checks; capture key service and business metrics in time-series repositories; and build baselines that define normal behaviour. The team will apply analytics and AI to accelerate issue detection and evidence-based hypotheses, improve capacity forecasting and enable safe, increasingly-automated remediation.

This is a high-impact opportunity to help build a strategically important capability at a pivotal stage of its development. The organisation has a strong platform foundation and a clear mandate to raise standards significantly; the successful candidate will have the opportunity to apply modern observability engineering across a large and complex Financial Services technology landscape and help move Production Engineering from reactive monitoring towards predictive, resilient and AI-enabled operations.

Role Responsibilities

Standards Implementation & Engineering: Work with the Head of Observability to implement the enterprise observability strategy, standards, tooling, governance and best practices, while remaining hands-on in solution design, implementation and operational support.

• Observability Engineering with Grafana: Design, implement, and optimize observability solutions built on the Grafana LGTM stack - Loki (logs), Grafana (visualisation and dashboarding), Mimir (metrics), and Tempo (distributed tracing) - that give teams clear, actionable insight into system health and performance.
• Observability Strategy & Instrumentation: Establish best practices across the three pillars of observability - logs, metrics, and traces - including managing ingestion rates, cardinality and retention, and using annotations to correlate deployments and incidents across dashboards.
• Platform Engineering in AWS: Design, implement, and optimize AWS-based platform solutions that host and support the observability stack, aligned with application requirements and industry best practices.
• Containerisation & Infrastructure Support: Build and maintain the containerised infrastructure the observability stack runs on, using Docker and Kubernetes / Amazon EKS, ensuring the platform is scalable and resilient.
• Automation & Infrastructure as Code (IaC): Develop and maintain infrastructure and observability automation using scripting and IaC tooling (e.g. Terraform, AWS CDK), enabling repeatable, self-service deployment of monitoring and alerting.
• Escalation & Troubleshooting: Diagnose and resolve complex technical and observability issues, collaborating with the Head of Observability and other engineering teams where escalation is required.
• Collaboration & Knowledge Sharing: Work closely with cross-functional teams, sharing knowledge, reusable patterns and best practices to foster continuous improvement.

Experience / Competences

Essential

• Strong observability expertise within the Grafana LGTM stack - hands-on experience with Loki, Grafana, Mimir and Tempo for logs, metrics, dashboarding and distributed tracing.
• Solid understanding of the observability pillars - logs, metrics and traces - including ingestion rates, cardinality, retention and the use of annotations to correlate events across dashboards.
• Containerisation knowledge to support the infrastructure on which the observability stack runs, using Docker and Kubernetes.
• Scripting and automation proficiency, with a demonstrated history of developing and maintaining custom automated solutions - strong experience with Python, Go and PowerShell.
• Key AWS concepts and hands-on AWS experience, including cloud automation, security best practices, cost optimisation and architectural design.
• Experience with DevOps methodologies and CI/CD pipelines.
• Familiarity with security and compliance within cloud environments.
• Excellent problem-solving skills and the ability to troubleshoot complex cloud-based and observability environments.
• Ability to effectively communicate technical concepts to both technical and non-technical stakeholders

Desired

• Current or recent experience working within an equivalent Financial Services organisation, with a strong understanding of the operational resilience, control and availability demands of business-critical technology services.
• Experience of Geneos observability platform, including low latency solutions
• AWS certifications such as AWS Certified Solutions Architect - Professional, AWS Certified DevOps Engineer - Professional, AWS Certified AI Practitioner, or AWS Certified Developer - Associate.
• Familiarity with other monitoring/logging solutions such as CloudWatch, Prometheus or the ELK Stack.
• Proven track record of implementing and transforming observability across large, complex system architectures and technology landscapes.

Job Band & Level

• Engineer / 5

#LI-Hybrid #LI-ASO #LI-LH

Not The Perfect Fit?

Concerned that you may not meet the criteria precisely? At TP ICAP, we wholeheartedly believe in fostering inclusivity and cultivating a work environment where everyone can flourish, regardless of your personal or professional background. If you are enthusiastic about this role but find that your experience doesn't align perfectly with every aspect of the job description, we strongly encourage you to apply. You may be the ideal candidate for this position or another opportunity within our organisation. Our dedicated Talent Acquisition team is here to assist you in recognising how your unique skills and abilities can be a valuable contribution. Don't hesitate to take the leap and explore the possibilities. Your potential is what truly matters to us.

Company Statement

We know that the best innovation happens when diverse people with different perspectives and skills work together in an inclusive atmosphere. That's why we're building a culture where everyone plays a part in making people feel welcome, ready and willing to contribute. TP ICAP Accord - our Employee Network - is a central to this. As well as representing specific groups, TP ICAP Accord helps increase awareness, collaboration, shares best practice, and holds our firm to account for driving continuous cultural improvement.

Location