{"id":2739236,"url":"https://alion.io/job/vantaca-senior-infrastructure-engineer-developer-platform-copy","title":"Senior AI Engineer, Evaluation & Optimization","company":{"id":452193,"name":"Vantaca","domain":"vantaca.com","url":"https://alion.io/company/vantaca","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Gem","truth_index":{"grade":"B","score":73,"open_postings":8,"ghost_share":0.375,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":85,"computed_at":"2026-10-21T05:46:15Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Palo Alto, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":200000,"max":260000,"currency":"USD","period":"year","gross":null,"usd_annual":260000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"A/B Testing","optional":false},{"name":"Anthropic","optional":false},{"name":"bandit","optional":false},{"name":"Fine-tuning","optional":false},{"name":"Function Calling","optional":false},{"name":"Knowledge Distillation","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Model Distillation","optional":false},{"name":"Node JS","optional":false},{"name":"OpenAI","optional":false},{"name":"Python","optional":false},{"name":"SQL","optional":false},{"name":"Structured Outputs","optional":false},{"name":"TypeScript","optional":false},{"name":"CUDA","optional":true},{"name":"JavaScript","optional":true},{"name":"Reward Modeling","optional":true},{"name":"TensorRT","optional":true},{"name":"TensorRT-LLM","optional":true},{"name":"vLLM","optional":true}],"status":"live","first_seen_at":"2026-10-01T15:07:05Z","employer_posted_date":"2026-10-20","last_verified_at":"2026-10-21T12:46:41Z","board_verified":true,"closed_at":null,"days_open":19,"trust":{"level":"ok","repost_count":0,"flags":[],"days_open":19},"description":"HOAi is the fastest-growing company revolutionizing the community association management industry. Our AI workforce platform integrates machine learning technology to streamline labor-heavy processes, eliminating inefficiencies and driving scalability. With rapid growth in the AI space, we are pushing boundaries to redefine industry standards. HOAi Agents go beyond traditional AI by proactively executing complex, multi-step processes with human-like reasoning-working autonomously, 24/7, across your entire operation. This transformation optimizes labor costs, enables growth without additional hires, and ensures faster, higher-quality service for residents and board members.\nHOAi was acquired by Vantaca in the fall of 2024. Vantaca just achieved unicorn status with a $1.25B valuation, so it's safe to say we're past the \"scrappy startup phase.\" We're not just building a successful company - we're building the category-defining platform that will transform how an entire industry operates.\nHere's the reality of our trajectory:\nGrowing 100% year-over-year\nOur AI product (HOAi) went from $0 to millions in months\nBacked by Cove Hill Partners and JMI Private Equity\n6M+ doors on our platform, displacing legacy systems\nAbout the Role\nWe run millions of AI tasks in production and are hiring a Senior AI Engineer, Evaluation & Optimization to own the systems, experiments, and feedback loops that determine how those tasks use AI models.\nAI models are improving rapidly, but model capability alone does not determine production performance. Model choice, prompts, context, tool use, routing, retries, and execution design all affect whether a task succeeds and what that success costs.\nYour mandate is to maximize successful task completion per dollar while meeting our quality, reliability, and latency requirements.You will build evaluation and observability infrastructure, benchmark models and configurations, analyze production behavior, and translate findings into production improvements. You will also design optimization loops that search for better prompts, model configurations, routing policies, agent strategies, and execution plans.\nThis is an engineering-heavy, highly empirical role. You will work across model providers, prompts, agent architectures, routing strategies, production traces, and optimization systems to determine what works at scale for our workloads. TypeScript is the primary implementation language.\nWhat You Will Own\nProduction AI Infrastructure\nBuild the infrastructure required to measure, operate, and improve millions of AI tasks reliably.\nBuild TypeScript services and libraries for model execution, routing, retries, fallbacks, and structured outputs.\nInstrument AI workflows for quality, latency, reliability, token usage, and cost.\nCreate consistent interfaces across model providers and configurations.\nProvide end-to-end traceability across prompts, models, tools, agent steps, retries, and outcomes.\nDefine durable schemas for tasks, evaluations, experiments, model versions, and production results.\nDesign systems for safe rollout, rollback, and comparison of model and workflow changes.\nImprove the reliability and operational efficiency of high-volume AI workloads.\nPartner with platform and product engineers so teams can adopt optimization capabilities throughout the stack.\nThis is a production engineering role. The systems you build will run in live, high-volume production environments.\nModel Evaluation\nBuild and maintain evaluation suites that represent our production workloads and predict real production performance.\nDefine task-specific quality metrics and acceptance thresholds.\nBuild golden datasets from production workloads, human-reviewed examples, edge cases, and synthetic scenarios.\nDevelop deterministic checks, automated graders, and LLM-as-judge evaluations where each method is appropriate.\nCalibrate automated evaluations against human judgment.\nBuild regression tests that detect degradation as models, prompts, tools, and agent architectures change.\nSample and analyze production tasks to identify new failure modes.\nContinuously update evaluations as production behavior changes.\nConnect offline evaluation results to online task, user, and business outcomes.\nModel Experimentation\nContinuously evaluate the model ecosystem and the configurations around it, including: Models from OpenAI, Anthropic, Google, open-weight providers, and emerging vendors.\nModel sizes, capability tiers, reasoning effort, and inference-time compute.\nPrompt, system-instruction, context-selection, and context-compression strategies.\nTool-use configurations, structured outputs, and agent architectures.\nRetries, fallbacks, escalation policies, and model cascades.\nExecution parallelism, task decomposition, and planning approaches.\nExperiments should be reproducible, statistically sound, and connected to production outcomes. They should produce clear recommendations and deployable changes rather than one-off benchmark results.\nContinuous Optimization\nBuild and operate systems that continuously search for better AI configurations across prompts, model selection, routing, reasoning budgets, context construction, tool use, agent workflows, retries, caching, parallelism, and workflow-specific policies.\nSelect the appropriate search method for each problem, from manual experimentation and grid or random search to Bayesian optimization, bandit methods, evolutionary approaches, and production feedback loops.\nOptimization must be safe and measurable. Build guardrails against flawed graders, narrow-set overfitting, segment-level regressions, reward hacking, and hidden operational costs.\nCandidate generation and evaluation pipelines.\nExperiment queues and prioritization.\nChampion and challenger comparisons.\nAutomatic promotion and rollback criteria.\nExploration and exploitation policies.\nSegment-level monitoring and regression detection.\nHuman-review escalation for ambiguous or high-impact changes.\nVersioning for prompts, models, tools, datasets, graders, and optimization policies.\nCost and Performance Optimization\nEstablish the Pareto frontier across quality, cost, latency, and reliability for each major workload. Investigate opportunities such as: Use smaller models where larger models provide little incremental value.\nRoute requests dynamically based on task difficulty.\nReduce unnecessary context, output tokens, agent steps, tool calls, retries, and model calls.\nTune reasoning and token budgets.\nCompress prompts and context without losing required information.\nUse caching, reuse, model cascades, fallbacks, and execution parallelism where they improve economics.\nCompare fine-tuning or distillation with frontier-model APIs where appropriate.\nImprove provider selection and workload placement.\nReduce operational overhead in high-volume execution paths.\nThe objective is to minimize cost per successful task while preserving the quality and reliability users require.\nProduction Deployment and Monitoring\nTurn experimental results into measurable improvements in production.\nBuild dashboards for quality, success rate, latency, reliability, and AI economics by workflow, model, configuration, and customer segment.\nRun controlled production experiments and A/B tests.\nRoll out model migrations and configuration changes safely.\nDetect regressions after model or provider updates.\nEstablish automated gates for model, prompt, tool, and agent changes.\nMonitor optimization systems for reward hacking and grader exploitation.\nIdentify where production behavior diverges from offline evaluation.\nPartner with product and engineering teams to determine where additional model capability creates meaningful value.\nQuantify optimization results in successful tasks, user outcomes, and spend.\nWhat Success Looks Like\nWithin your first several months, the team should be able to answer questions such as:\nWhat does each major AI workflow cost, and what quality threshold does it require?\nWhich model and configuration provide the best economics for each workload?\nHow much additional quality do we gain from incremental spend?\nCan most of a workload move to a substantially less expensive model without material quality loss?\nWhich tasks genuinely require frontier models?\nWhich prompts, workflows, and routing policies are improving over time?\nCan we improve continuously without overfitting or degrading important segments?\nWhen a new model launches, can we determine within days whether and where to adopt it?\nCan we change models without unknowingly degrading production quality?\nCan we detect and roll back a harmful optimization automatically?\nDo our evaluations and graders remain aligned with real user and business outcomes?\nOver time, optimization should become a repeatable operating loop: production traces reveal failure modes; failure analysis generates candidates; evaluations identify promising changes; controlled rollouts validate them; and production feedback informs the next iteration.\nRequirements\nStrong software engineering skills in TypeScript and Node.js.\nExperience building production systems that use LLMs or other machine-learning models.\nExperience operating high-volume or high-reliability services.\nExperience designing and running rigorous model evaluations.\nStrong understanding of modern LLM APIs and model behavior.\nExperience working with production traces, datasets, and experimentation infrastructure.\nAbility to reason quantitatively about quality, latency, reliability, and cost.\nStrong SQL and data-analysis skills.\nExperience designing controlled experiments and interpreting noisy results.\nExperience building optimization, search, experimentation, or feedback-loop systems.\nAbility to translate experimental findings into production engineering changes.\nSound judgment about when benchmark differences matter and when they do not.\nAbility to keep systems reliable while their prompts, models, or execution policies change continuously.\nPython experience is useful for analysis and evaluation tooling, but TypeScript is the primary language for production systems in this role.\nPreferred Experience\nExperience in one or more of the following areas is helpful:\nModel routing, cascading, or agent evaluation.\nLLM-as-judge systems and human evaluation.\nOnline experimentation and A/B testing.\nBandits, Bayesian optimization, evolutionary search, or other optimization methods.\nAutomated prompt optimization, reward modeling, or feedback-loop design.\nGuarding against reward hacking and grader exploitation.\nFine-tuning, distillation, or open-weight models.\nAI observability, tracing, prompt optimization, or context optimization.\nSemantic caching or prefix caching.\nHigh-volume distributed systems, queues, scheduling, rate limits, or provider capacity management.\nInternal benchmarking, experimentation, AI infrastructure, or inference platforms.\nExperience with CUDA, GPU kernels, vLLM, TensorRT-LLM, or low-level inference optimization is useful but not required. \nThis role focuses on how we use models and operate AI workloads rather than on the internals of model-serving infrastructure.\nExample Problems\nModel selection A frontier model costs eight times more than a smaller model. Determine which production tasks receive enough quality improvement to justify the additional cost, then implement routing that sends only those tasks to the frontier model.\nReasoning optimization Determine whether additional reasoning effort materially improves task completion, establish the right reasoning budget for each workload, and continue testing whether the policy can be improved.\nModel routing Design a policy that sends routine tasks to inexpensive models and escalates difficult tasks without materially reducing quality or increasing tail latency.\nAgent optimization A workflow makes 14 model calls. Identify which calls are necessary, where smaller models can be substituted, which calls can run in parallel, and whether the workflow can complete reliably in fewer st...","description_format":"text","description_chars":16480,"description_truncated":true,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"phd","optional":false},"security_clearance":false,"languages":[]},"benefits":["401k plan","Equity","Growth opportunities","Parental leave","Professional development","Unlimited PTO","Vision insurance"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Property Management","PropTech"],"lifecycle":[{"event":"open","at":"2026-10-16T20:10:16Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 8","filings_12m":8,"filings_prev_12m":3,"green_card_filings_12m":0,"median_offered_wage_usd":242500,"route":null,"cap_exempt":false,"checked_at":"2026-10-15T11:24:14+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":4}],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":19,"expected_fill_days":85,"reasons":["conf:2","win:early"],"computed_at":"2026-10-21T05:46:15Z"},"pay":{"stated_usd_annual":260000,"is_top_pay":true},"html_url":"https://alion.io/job/vantaca-senior-infrastructure-engineer-developer-platform-copy","json_url":"https://alion.io/job/vantaca-senior-infrastructure-engineer-developer-platform-copy.json","meta":{"generated_at":"2026-10-21T14:57:11Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":21300,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-22T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":452193},"rest":"https://alion.io/mcp/rest/get_company?id=452193"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fvantaca-senior-infrastructure-engineer-developer-platform-copy"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fvantaca-senior-infrastructure-engineer-developer-platform-copy"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fvantaca-senior-infrastructure-engineer-developer-platform-copy"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/vantaca-senior-infrastructure-engineer-developer-platform-copy\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fvantaca-senior-infrastructure-engineer-developer-platform-copy"}]}