Principal Engineer - AI

As a Principal Engineer - AI, you will define and lead the technical direction of AI systems that power Safe’s CRQ, CTEM, and TPRM products, including agentic workflows, RAG pipelines, LLM orchestration, and AI-native developer tooling. You’ll be the hands-on architect behind Safe’s AI engineering stack, bridging model intelligence with production-grade infrastructure.

You’ll collaborate with product, data, and platform teams to design scalable, explainable, and enterprise-ready systems.

This is a high-impact, technical leadership role that will shape how AI is built, deployed, and governed across Safe.

Core Responsibilities:

• Architect Safe’s AI Systems: Design and scale AI-driven components - LLM orchestration, retrieval-augmented generation (RAG), vector stores, prompt pipelines, and AI microservices. Drive architecture for AI observability, safety, and evaluation (precision, recall, F1, hallucination detection, cost metrics).
• Productionize AI Agents: Build multi-turn, goal-oriented agent systems that automate reasoning across TPRM, CTEM, and CRQ domains (e.g., control reviews, issue RCA, automated responses). Ensure reliability, traceability, and deterministic behavior in production.
• AI Infrastructure & Platform Ownership: Partner with Platform & DevOps teams to operationalize model serving (AWS SageMaker, Bedrock, or self-hosted Llama), build AI APIs, and manage model lifecycle and versioning. Establish feature stores, embedding management, and in-memory retrieval layers.
• Data Pipeline & Knowledge Graph Integration: Work with Data Engineering to design pipelines for structured and unstructured data ingestion, semantic indexing, and context retrieval (Snowflake + Iceberg + LlamaIndex).
• AI Evaluation, Monitoring & Governance: Define internal frameworks for golden dataset validation, LLM evaluation (LangFuse/LangSmith), and safety enforcement policies. Implement human-in-the-loop (HITL) mechanisms and continuous feedback loops.
• Mentor & Multiply: Guide AI and backend engineers on architectural design, experimentation methodologies, and prompt optimization. Collaborate with product leaders to translate abstract AI goals into measurable engineering deliverables.

Minimum Qualifications:

• Experience: 12+ years total experience in software engineering, including 4+ years building AI/ML systems or large-scale data/LLM infrastructure.
• Core Technical Skills:

- Strong programming fundamentals in Python, Go, or TypeScript

- Deep understanding of LLM-based architectures, prompt engineering, and RAG pipelines

- Hands-on experience with LangChain, LlamaIndex, or equivalent orchestration frameworks

- Vector databases (FAISS, Pinecone, Weaviate, Redis Vector, or Milvus)

- Cloud model deployment (AWS SageMaker, Bedrock, Vertex AI, or custom inference APIs)

- Data systems: Snowflake, Iceberg, S3, Postgres/MySQL

• MLOps & Infra: Familiar with model versioning, CI/CD for ML, and performance optimization for real-time inference.
• Applied AI Focus: Practical understanding of evaluation metrics, hallucination detection, RAG reliability, and enterprise AI safety.

Preferred Qualifications:

• Experience integrating AI into cybersecurity or risk management products
• Familiarity with multi-agent systems and autonomous workflows (CrewAI, LangGraph, AutoGen)
• Experience building AI evaluation dashboards and AI observability stacks
• Knowledge of knowledge graphs, semantic search, or retrieval pipelines
• Exposure to data governance, compliance, or SOC2/ISO 27001 environments
• Published research, open-source contributions, or prior leadership of AI teams is a strong plus