AI Engineering Lead

Description

Upwind is a next-generation Cloud Security Platform that leverages runtime context to identify and prioritize critical risks, providing precise insights and efficient cloud security management. Unlike traditional tools, Upwind uses runtime data proactively for risk prioritization and posture insights, ensuring teams focus on what truly matters. With industry-leading efficiency and eBPF-powered sensors, Upwind delivers comprehensive capabilities, including agentless cloud posture discovery, real-time threat protection, and integrated API security. From misconfigurations to malware defense, Upwind ensures end-to-end, cost-effective cloud infrastructure protection.

At Upwind, you'll have the opportunity to work on some of the most challenging problems in applied AI and security, turning cutting-edge research and LLM capabilities into products that operate in real customer environments.

We're looking for an AI Engineering Lead to help drive the quality, intelligence, and continuous improvement of Upwind's AI and agentic systems.

This is a hands-on AI engineering lead role focused on data, evaluation, and agentic workflows. You'll build the systems and methodologies that allow us to understand how our agents perform, identify where they fail, and continuously improve their reasoning, tool usage, and outcomes.

You'll work closely with AI Researchers, Engineers, and Product teams to turn experiments into reliable, high-performing AI systems - from defining evaluation datasets and metrics to improving agentic workflows and deploying them at scale.

What You’ll Build:

• Agentic Workflows: Design and continuously improve multi-step agent workflows, including reasoning, tool usage, planning, state management, and recovery strategies.
• AI Evaluation: Build evaluation frameworks, datasets, metrics, and automated pipelines to measure the quality and reliability of LLMs and agents.
• AI Data Infrastructure: Build the data pipelines and feedback loops required to collect, curate, analyze, and learn from real-world agent interactions
• AI Observability: Build visibility into agent behavior and performance, including trajectories, tool calls, latency, failures, and outcome quality.

Responsibilities:

• Own and evolve the evaluation strategy for Upwind's AI agents and LLM-powered features.
• Define meaningful metrics and automated evaluation pipelines for measuring agent quality, accuracy, reliability, and task completion.
• Design and improve agentic workflows to make agents more capable, reliable, efficient, and predictable.
• Build feedback loops from production usage back into evaluation datasets and development workflows.
• Develop tooling for running experiments and comparing models, prompts, workflows, and agent configurations.
• Work closely with AI Researchers to turn research ideas and experiments into production-quality AI capabilities.
• Collaborate with backend engineers to integrate AI workflows and evaluation systems into our production platform.
• Establish best practices for testing and validating non-deterministic AI systems.
• Monitor AI systems in production and continuously identify opportunities for improvement.
• Take end-to-end ownership of projects- from defining the problem and building the first experiment to productionizing and measuring its impact.

Requirements

• Strong engineering experience, with a track record of owning and delivering complex technical systems.
• Hands-on experience building LLM-powered applications, agents, or AI systems.
• Strong understanding of LLM evaluation, experimentation, and data-driven development.
• Experience designing and implementing LLM-as-a-Judge evaluations, including evaluation criteria, scoring rubrics, structured outputs, and validation of judge reliability.
• Strong problem-solving skills and the ability to analyze complex AI behavior and turn observations into concrete improvements.
• Experience working with agentic workflows, tool calling, multi-step reasoning, or AI orchestration.
• Ability to work closely with researchers and engineers and bridge the gap between experimentation and production.
• Comfortable working in a fast-paced environment where requirements and technologies evolve quickly.

Strong Advantages:

• Experience with LLM evaluation frameworks and automated evaluation pipelines.
• Proven experience with LLM-as-a-Judge methodologies, including calibrating judges against human evaluations, measuring evaluator consistency, and reducing evaluation bias
• Experience with LangGraph, LangChain, or comparable agent orchestration frameworks.
• Experience working with security, infrastructure, or other complex technical domains.
• Experience building production AI systems where correctness and reliability are critical.
• Strong Python skills and experience working with data processing and AI/ML tooling.

Why Upwind?

• You'll have the opportunity to work on AI systems that are solving real, complex security problems- not demos or prototypes.
• You'll be part of a team building the next generation of AI-powered security agents, with the opportunity to influence how we evaluate, improve, and scale these systems from the ground up.