Jacob Allan
Machine Learning Engineer
C++
MATLAB
SQL
JavaScript
TypeScript
C
Python
Rust
Data Layer & Database: 6/10
System Architecture: 6/10
API Design: 5/10
Active 2 days ago
+1 (613) 5322831
Invite to interview
Message
Download CVCV
Overview
Technical skills
Timeline
Roles
Overview
Backend-focused Python engineer specializing in building typed evaluation and control-plane APIs with a particular strength in deterministic EvalOps pipelines and grader systems. The strongest proven skill is designing and implementing the synchronous evaluation runner, composite graders, provider adapters and CI eval-gate as evidenced by llm-evalops-platform/backend/app/services/eval_runner.py and its comprehensive test suite. Public code does not show production multi-tenant auth, asynchronous worker/queue infrastructure, or hosted distributed deployment patterns.
Technical skills
C++
MATLAB
SQL
JavaScript
TypeScript
C
Python• Senior
Rust• Junior
C++
PyTorch C++
Python
FastAPI
Ruff
SQLAlchemy
Alembic
Pydantic
Gunicorn
HTTPX
Django
Databases
PostgreSQL
AI/ML
LLM
NumPy
Pandas
RAG
Scikit-learn
XGBoost
Google GenAI SDK
OpenAI SDK
Gemini
PyTorch
Data Augmentation
Streamlit
AI Agents
Frontend
Next.js
React.js
DevOps
GCP
Google Cloud Run
Rest API
Vercel
CI/CD
Containers
CI/CD
AWS
Docker
Git
GitHub Actions
Terraform
QA
Pytest
Playwright
Timeline
Software Engineering Intern
•
Junior
Northstar Downhole Specialists
•
Internship
Built an enterprise documents and IPR system using Python and Django with PostgreSQL-backed RBAC and audit trail features. Developed document/IPR workflows including approval, notifications, external intake, and rollback-safe migrations. Provisioned AWS infrastructure (ECS, RDS, S3, SES, ALB, WAF) using Terraform and implemented CI covering integration, E2E, typing, containers, and security checks. Validated large PostgreSQL and Playwright test suites and ran synthetic load testing with strong latency and no unexpected errors.
Python
Django
PostgreSQL
AWS
Terraform
Playwright
University of Toronto (UofT)
Bachelor's Degree •
Engineering Science (ongoing; expected May 2029) - Machine Intelligence and Mathematics
Machine Learning Engineer
•
Middle
UTMIST / Flybits
•
Part-Time
Developed an ML system to personalize digital credit offers using demographic, transactional, and product-level features. Built a synthetic-data pipeline for more than 100,000 customer personas with realistic demographic, spending, and credit features. Built clustering and PyTorch autoencoder prototypes for more than five customer archetypes with a six-person engineering team.
Python
PyTorch
Built a leakage-aware Python platform for ETF ingestion, portfolio-risk metrics, GMM/HMM/KMeans regime models, stress tests, and cost-aware backtests. Prevented look-ahead bias with chronological splits, train-only scaling, shifted signals, and future-mutation tests; shipped 322 tests and a nine-page dashboard.
Python
Scikit-learn
Streamlit
Pandas
NumPy
Event-Driven Trading Engine
Backend
Built a Rust matching engine with price-time priority, partial fills, deterministic replay, portfolio P&L, pre-trade risk limits, and a kill switch; CI passes 247 tests. Benchmarked the 10,000-event core path at 125 ns p50 and approximately 4.6 million events per second on an Apple M4 Pro versus approximately 1.2 million events per second for a naive Python baseline.
Rust
Python
GitHub Actions
Built a full-stack platform for versioned datasets, prompts, and models; deterministic and LLM-based graders; evaluation dashboards; and CI quality gates. Caught a controlled 20-case RAG regression: pass rate fell from 95% to 85%, failures rose from 1 to 3, and estimated cost more than doubled. Shipped CI with 221 backend tests, a frontend build, and an eval gate enforcing pass rate, score, cost, and p95 latency.
FastAPI
PostgreSQL
Docker
Next.js
GitHub Actions
Python
Machine Learning Researcher
•
Middle
Royal Military College of Canada
•
Full-Time
Built and trained multiple PyTorch CNN models for RF spectrogram classification using a custom Python pipeline and RTL-SDR data acquisition flow. Curated a large RF dataset and achieved high accuracy on unseen real-world signals. Improved model generalization to noisy bands using data augmentation and training strategies such as learning-rate decay, frequency-offset sampling, and spectrogram normalization.
PyTorchsince 2025
Pythonsince 2025
Senior Backend Developer
Confidence: High API Engineer
Backend-focused Python engineer specializing in building typed evaluation and control-plane APIs with a particular strength in deterministic EvalOps pipelines and grader systems. The strongest proven skill is designing and implementing the synchronous evaluation runner, composite graders, provider adapters and CI eval-gate as evidenced by llm-evalops-platform/backend/app/services/eval_runner.py and its comprehensive test suite. Public code does not show production multi-tenant auth, asynchronous worker/queue infrastructure, or hosted distributed deployment patterns.
API Design
5/10
How well APIs are designed
Solid typed API contracts and error-handling tests using FastAPI and Pydantic, but no explicit versioning strategy or idempotency/key-handling patterns are visible and pagination/versioning evidence is minimal.
Evidence
llm-evalops-platform/backend/app/api/routes/eval_runs.py
llm-evalops-platform/backend/app/schemas/eval_runs.py
llm-evalops-platform/backend/tests/test_eval_runs_api.py
Data Layer & Database
6/10
Working with databases
Well-structured SQLAlchemy models and a multi-file Alembic migration history with tests and deliberate transaction/rollback handling; demonstrates schema evolution and DB-aware design.
Evidence
llm-evalops-platform/backend/alembic/versions/16a3d1913db0_create_datasets_and_test_cases.py
llm-evalops-platform/backend/app/models/*.py
llm-evalops-platform/backend/app/db.py
Scalability & Performance
4/10
Handling load and speed
Run-level metrics, p95 latency and cost estimation are implemented and tested, but the execution model is synchronous with no production async worker/queue decoupling or caching/invalidation strategy shown.
Evidence
llm-evalops-platform/backend/app/services/metrics.py
llm-evalops-platform/backend/app/services/cost_tracker.py
llm-evalops-platform/backend/app/services/eval_runner.py
System Architecture
6/10
Overall system structure
Clear service decomposition and deliberate architecture choices for graders, provider adapters, runner, CLI eval-gate and seed/load tooling; modular boundaries and CI gate design are explicit.
Evidence
llm-evalops-platform/AGENTS.md
llm-evalops-platform/backend/app/main.py
llm-evalops-platform/backend/app/cli/eval_gate.py
Security & Auth
4/10
Protecting data and access
Good input validation with Pydantic schemas, bounded inputs and provider fallback behavior, but no authentication/authorization, rate limiting, token lifecycle, or secret-management patterns are implemented for public endpoints.
Evidence
formatclip/backend/app/schemas.py
formatclip/backend/app/services/providers/openai_provider.py
formatclip/backend/tests/test_providers.py
Reliability & Observability
5/10
Stability and monitoring
Robust failure isolation, explicit commit/rollback and persisted error recording, structured test coverage for failure modes and judge timeouts, but limited visible observability (structured logs/trace/alerts) and explicit retry/backoff policies.
Evidence
llm-evalops-platform/backend/app/services/eval_runner.py
llm-evalops-platform/backend/app/services/judge_provider.py
llm-evalops-platform/backend/tests/test_eval_runner.py
Verified artifacts
Expertise
Python• Senior
Backend AI & LLM• Middle
Industries
Artificial Intelligence• Senior
Financial Services• Senior
Technologies
CI/CD
Containers
Gunicorn
Stack• mentioned only
Systems• mentioned only
Text• mentioned only
Recommendations
- Design and implement deterministic evaluation pipelines and CI quality gates for model and prompt changes, including runner, graders, and provable metrics.
- Build backend APIs and integrations for LLM-backed workflows that require typed contracts, fallback behavior, and robust persisted failure modes.
- Implement schema evolution and migration-aware data models for analytics and evaluation workloads, including Alembic migrations and DB-backed test fixtures.
Repositories
The developer's experience in this domain has been verified based on AI analysis of the following repositories:
Middle AI/ML Engineer
Confidence: High LLM Engineer
LLM-focused engineer (senior-level) who designs reliable, test-first evaluation and judge-based grading systems for LLM apps. The strongest proven skill is building EvalOps and deterministic evaluation pipelines as shown by the eval-runner, composite graders, and the eval-gate CLI (Jakey794/llm-evalops-platform backend artifacts). There is no evidence of custom model training loops, novel research-level model architectures, or GPU/quantization optimizations in public code.
Model Architecture & Training
2/10
How well models are designed and trained
Inference and provider integration are well engineered, but there is no evidence of custom model architectures or training loops; model-building work is limited to provider adapters and request/response shaping.
Data Pipeline & Feature Engineering
4/10
How data is prepared for models
Clear, tested dataset ingestion and seed-loading pipelines with JSONL importer and schema validation; solid engineering for evaluation datasets but not large-scale ETL or advanced augmentation.
Experimentation & Evaluation
6/10
How results are measured and tested
Strong evaluation and experimentation engineering: deterministic graders, optional LLM-as-judge, reproducible seeds, and a CI eval gate with explicit thresholds and JSON reports; good test coverage and deterministic mock profiles for CI.
MLOps & Deployment
5/10
How models are shipped to production
Concrete MLOps and deployment thinking: Alembic migrations, eval-gate CLI, mock providers for CI, Cloud Run/Vercel packaging notes and CI configuration; good pragmatic deployment hygiene but not large-scale infra automation shown.
Computational Efficiency
2/10
How efficiently computing resources are used
Some cost and latency accounting and p95 calculation are present, but no GPU/batching/quantization or low-level performance engineering is evidenced.
Research Depth & Innovation
2/10
Depth of research and new ideas
Nice applied engineering around graders, rubrics, and judge prompt construction, but no novel research, custom layers, or paper-reproduction work is present.
Expertise
LLM• Middle
MLOps & Model Lifecycle• Middle
RAG• Middle
Industries
Artificial Intelligence• Senior
Software• Middle
Technologies
SQL
C++
MATLAB
Rest API
Terraform
GCP
XGBoost
GitHub Actions
Vercel
Scikit-learn
Google GenAI SDK
CI/CD
Pandas
NumPy
Django
Git
PyTorch
AWS
Docker
LLM
RAG
PyTorch C++
Ruff
Streamlit
Pydantic
Google Cloud Run
Data Augmentation
Datasets• mentioned only
Groq• mentioned only
LLM• mentioned only
MLOps• mentioned only
Python• mentioned only
Vercel• mentioned only
Recommendations
- Design and implement EvalOps pipelines, graders, and CI eval gates for LLM systems (quality gates, mock profiles, deterministic graders).
- Build production-ready LLM inference layers and provider adapters with robust error handling and cost/latency accounting.
- Develop backend services for LLM-powered applications with strong test coverage, schema validation, and migration/seed tooling.
- Implement deterministic dataset ingestion, seed management, and reproducible evaluation artifacts for model comparison and regression testing.
Repositories
The developer's experience in this domain has been verified based on AI analysis of the following repositories:
Middle Data Scientist
Confidence: Medium Data Engineer
A senior-level data engineer focused on reliable LLM evaluation and deterministic grading pipelines with strong reproducibility and testing discipline.
Statistical Rigor
2/10
Correct use of statistics
Statistical rigor is limited; the project focuses on deterministic graders and evaluation contracts rather than formal statistical inference, significance testing, or uncertainty quantification.
Evidence
backend/app/services/metrics.py
backend/tests/test_metrics.py
Data Wrangling & Cleaning
7/10
Preparing and cleaning data
Strong data ingestion and validation practices are evident: a JSONL importer with validation and atomic import semantics, database models with Alembic migrations, and tests that assert duplicate detection and schema validation.
Evidence
backend/app/services/jsonl_importer.py
backend/tests/test_jsonl_importer.py
Exploratory Analysis & Visualization
1/10
Exploring and visualizing data
Exploratory analysis and visualization are minimal in code; the platform exposes analytics endpoints and a Next.js dashboard scaffold, but there is little evidence of EDA with written interpretations or storytelling in the code artifacts provided.
Evidence
backend/app/services/analytics.py
frontend/README.md
Predictive Modeling
2/10
Building models that predict
Predictive modeling work is not present; the code builds graders, evaluation harnesses, and LLM-provider interfaces rather than model training or feature engineering workflows.
Evidence
backend/app/graders/composite.py
backend/tests/test_eval_runner.py
Business Insight & Impact
3/10
Turning analysis into business value
Some product-oriented thinking and impact reasoning exist in the incident-triage rules and the eval-gate thresholds, but explicit mapping to business KPIs or error-cost tradeoffs is limited to cost-estimation and triage action lists rather than formal business-metric analysis.
Evidence
incident-triage-copilot/backend/app/services/triage_pipeline.py
backend/app/cli/eval_gate.py
Reproducibility & Notebook Hygiene
7/10
Clean, repeatable analysis
Reproducibility and engineering hygiene are strong: there is an eval-gate CLI with deterministic mock providers, Alembic migrations, seed-loaders, and a comprehensive test suite driving CI-friendly workflows.
Evidence
backend/app/cli/eval_gate.py
backend/seed/load_seed_data.py
backend/tests/test_eval_gate.py
Expertise
Analytics• Middle
Industries
Artificial Intelligence• Senior
Information Technology• Middle
Technologies
Python• Senior
PostgreSQL
SQLAlchemy
FastAPI
OpenAI SDK
Gemini
HTTPX
Alembic
Analytics• mentioned only
Datasets• mentioned only
Groq• mentioned only
LLM• mentioned only
MLOps• mentioned only
Recommendations
- Develop evaluation pipelines and CI gates that run deterministic and mock-backed LLM tests (implement and maintain the eval-gate and mock providers).
- Build and maintain ETL/data ingestion tooling with strict validation and seed-data workflows (extend the JSONL importer and seed loader pipelines).
- Design backend services for LLM evaluation, grading, and analytics with durable DB schemas and migration strategies (work on providers, graders, and analytics endpoints).
Repositories
The developer's experience in this domain has been verified based on AI analysis of the following repositories: