Senior Software Engineer

You'll build the data and retrieval foundation that makes Clearwater's AI agents accurate, grounded, and measurable. This is a hands-on software engineering role, not a research role: you'll write production pipelines and services, build retrieval and evaluation systems, and ship the product surfaces that expose them. You'll work across security and threat data at scale.

Responsibilities:

• Build and operate data pipelines (batch and streaming) that ingest, normalize, and enrich large volumes of security and telemetry data.
• Design retrieval systems for agents: chunking, embeddings, hybrid search, reranking, vector and graph stores, and freshness and access control.
• Build evaluation harnesses and datasets to measure agent and retrieval quality, regressions, and cost.
• Build backend APIs and services in Python and the full-stack features that surface data and AI insights to users.
• Design data models and manage relational, analytical, and vector stores for performance and cost.
• Add data quality checks, lineage, observability, and governance, including PII and sensitive data handling.
• Use AI agents to automate data engineering work such as pipeline generation, data validation, and documentation.
• Containerize and deploy on Kubernetes and the cloud with CI/CD, and mentor engineers on data and AI practices.

First 12 months:

• Ship production pipelines and retrieval services with clear quality and reliability targets.
• Deliver a retrieval and evaluation framework that measurably improves agent accuracy.
• Establish the team's patterns for data quality, grounding, and sensitive-data handling.

Requirements:

• 4+ years of professional software engineering with strong data-engineering exposure and sound testing and CI/CD practices.
• 1+ year building production systems involving LLMs, retrieval (RAG), or large-scale data pipelines.
• Strong Python and SQL, including query optimization and data modeling.
• Hands-on with pipeline or stream-processing tooling such as Airflow, Dagster, Spark, or Kafka, or an equivalent.
• Experience with vector stores and retrieval design (pgvector, OpenSearch, or similar), and with evaluating retrieval quality.
• Docker, working Kubernetes knowledge, and experience on AWS, Azure, or GCP.

Nice to have:

• Working JavaScript/TypeScript, enough to build and ship full-stack features.
• Experience with security, threat intelligence, or SIEM/telemetry data.
• Healthcare or compliance-driven data experience (HIPAA, SOC 2, HITRUST).
• Knowledge graphs or graph databases.
• LLMOps and evaluation tooling experience.
• Experience with a lakehouse or warehouse such as Snowflake, Databricks, or BigQuery.
• Open-source contributions or public data or retrieval projects.

Skills

Airflow, Dagster, ETL, Kafka, Python, SQL, Spark