Sr. Backend Engineer Distributed Systems & AI Agents

The core responsibilities for the job include the following:

Backend and Distributed Systems:

• Design, build, and own highly scalable backend services using modern system-design principles.
• Architect distributed systems with strong guarantees around reliability, consistency, concurrency, idempotency, retries, and failure recovery.
• Own critical services supporting procurement, invoice, contract, and payment workflows.
• Build workflow and state-management systems coordinating AI agents, humans, and enterprise applications.
• Design services that can scale across global customers and high transaction volumes.
• Proactively identify and resolve performance and reliability bottlenecks.

Agent Observability and Evaluation:

• Build comprehensive observability pipelines for AI-agent execution.
• Capture and analyze tool calls, LLM inputs/outputs, latency, token usage, cost, errors, and execution traces.
• Design evaluation frameworks to measure agent quality, reliability, task completion, and regression.
• Build automated testing infrastructure for agent pipelines, including unit, integration, replay-based, and regression testing.
• Define and monitor reliability metrics such as task completion rate, escalation rate, workflow success rate, cost per workflow, latency, and SLA adherence.
• Build systems that enable rapid debugging and root-cause analysis of complex agent workflows.

Reliability, Security, and Compliance:

• Design enterprise-grade systems with strong monitoring, alerting, fault tolerance, and disaster recovery.
• Implement fine-grained authorization and access-control mechanisms.
• Build comprehensive audit logging and traceability capabilities.
• Ensure secure handling of financial, operational, and other sensitive enterprise data.
• Design systems aligned with security and compliance requirements for regulated environments.

Skills

backend, distributed, observability