Principal Engineer - AI
First seen by Alion on Aug 29, 2026.
About the Role :
At SAFE Security, our mission is bold and ambitious: We Will Build CyberAGI a super-specialized system of intelligence that autonomously predicts, detects, and remediates threats. As a Principal Engineer AI, you will define and lead the technical direction of AI systems that power Safes CRQ, CTEM, and TPRM products, including agentic workflows, RAG pipelines, LLM orchestration, and AI-native developer tooling.
Core Responsibilities :
- Architect Safes AI Systems: Design and scale AI-driven components LLM orchestration, retrieval-augmented generation (RAG), vector stores, prompt pipelines, and AI microservices.
- Drive architecture for AI observability, safety, and evaluation (precision, recall, F1, hallucination detection, cost metrics).
- Productionize AI Agents: Build multi-turn, goal-oriented agent systems that automate reasoning across TPRM, CTEM, and CRQ domains.
- AI Infrastructure & Platform Ownership: Partner with Platform & DevOps teams to operationalize model serving (AWS SageMaker, Bedrock, or self-hosted Llama), build AI APIs, and manage model lifecycle and versioning.
- Data Pipeline & Knowledge Graph Integration: Work with Data Engineering to design pipelines for structured and unstructured data ingestion, semantic indexing, and context retrieval.
- AI Evaluation, Monitoring & Governance: Define internal frameworks for golden dataset validation, LLM evaluation (LangFuse/LangSmith), and safety enforcement policies.
- Mentor & Multiply: Guide AI and backend engineers on architectural design, experimentation methodologies, and prompt optimization.
Minimum Qualifications:
- Experience: 12+ years total experience in software engineering, including 4+ years building AI/ML systems or large-scale data/LLM infrastructure.
- MLOps & Infra: Familiar with model versioning, CI/CD for ML, and performance optimization for real-time inference.
- Applied AI Focus: Practical understanding of evaluation metrics, hallucination detection, RAG reliability, and enterprise AI safety.
Core Technical Skills:
- Programming: Python, Go, or TypeScript.
- LLM Architectures: Prompt engineering, RAG pipelines, LangChain, LlamaIndex.
- Vector Databases: FAISS, Pinecone, Weaviate, Redis Vector, or Milvus.
- Cloud: AWS SageMaker, Bedrock, Vertex AI.
- Data Systems: Snowflake, Iceberg, S3, Postgres/MySQL.
Preferred Qualifications:
- Experience integrating AI into cybersecurity or risk management products.
- Familiarity with multi-agent systems (CrewAI, LangGraph, AutoGen).
- Experience building AI evaluation dashboards and observability stacks.
- Knowledge of knowledge graphs, semantic search, or retrieval pipelines.
- Exposure to data governance, compliance, or SOC2/ISO 27001 environments.
- Published research, open-source contributions, or prior leadership of AI teams is a strong plus.
Skills
Machine Learning, Python, Artificial Intelligence, Data Science, MLOps, RAG, VectorDB, Vertex AI, Golang