Machine Learning Engineer / SME (LLM Eval & RAG Testing) - SME | TS/SCI

Falls Technology LLC is hiring a Machine Learning Engineer / SME to evaluate and validate LLM-based systems, including RAG, for mission use. The role designs benchmarks and experiments, tunes models on custom data, and builds AI/ML systems using GPU-accelerated computing.

Responsibilities

• Test, evaluate and validate AI systems with a focus on LLM-based systems (including RAG testing)
• Design and run experiments that benchmark LLMs against defined measures
• Tune neural networks / LLMs on custom datasets and apply results to concrete use cases
• Build AI/ML models and ensembles in Python or R (library selection, preprocessing, model assessment)
• Use deep learning frameworks such as PyTorch, TensorFlow or Keras
• Leverage GPUs for accelerated training and evaluation
• Write solid SQL (CTEs, set operations, aggregates, nested subqueries) against analytical datasets
• Work in GitHub and Jenkins-style CI for reproducible experiments
• Communicate methodological choices and model results to technical and non-technical stakeholders

Skills

• Active TS/SCI required
• U.S. citizenship required
• Hands-on RAG testing experience (must-have)
• Tuning neural nets / LLMs on custom datasets for real use cases
• Building AI/ML models and ensembles in Python or R
• PyTorch, TensorFlow or Keras (key)
• GPU-accelerated computing (key)
• Testing, evaluating or validating LLM-based AI systems (key)
• Experiment design to benchmark LLMs (key)
• SQL including CTEs, set ops, aggregates and nested subqueries
• GitHub and Jenkins (or equivalent) version control / CI
• Hugging Face Transformers library and hub
• Building benchmarks that measure LLM performance
• Cloud computing development and architecture
• Flask or similar front-end/web frameworks
• Text classification and topic modeling (scikit-learn or deep learning)
• Semantic search applications
• Tableau dashboards
• Verification/validation test benches, Explainable AI (XAI), ONNX

Benefits

• A 9% retirement contribution
• 100% of family medical premiums paid
• 4 weeks of PTO

Company Overview