{"id":1140315,"url":"https://alion.io/job/siteground-ai-evaluation-engineer","title":"AI Evaluation Engineer","company":{"id":55484,"name":"SiteGround","domain":"siteground.com","url":"https://alion.io/company/siteground","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Lever","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"full_time","work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Sofia, Bulgaria"],"countries":["BG"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"A/B Testing","optional":false},{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"Anthropic","optional":false},{"name":"DuckDB","optional":false},{"name":"Function Calling","optional":false},{"name":"Gemini","optional":false},{"name":"Google ADK","optional":false},{"name":"Google AI Studio","optional":false},{"name":"LangChain","optional":false},{"name":"Langfuse","optional":false},{"name":"LLM","optional":false},{"name":"LLM Guardrails","optional":false},{"name":"n8n","optional":false},{"name":"OpenAI","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Rest API","optional":false},{"name":"SQL","optional":false},{"name":"Tool Use","optional":false},{"name":"WordPress","optional":false},{"name":"PHP","optional":true}],"status":"live","first_seen_at":"2026-09-21T13:54:06Z","employer_posted_date":"2026-09-21","last_verified_at":"2026-10-17T08:58:15Z","board_verified":true,"closed_at":null,"days_open":25,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":25},"description":"YOUR ROLE:\nAt SiteGround, we’ve built and launched our own suite of AI products:\nCoderick AI -our vibe coding platform.\nAI Studio - access to flagship AI models, providers, and intelligent AI Agents.\nWordPress AI Agent - one AI assistant for virtually every WordPress task.\nWe’re moving fast: adding new capabilities, launching new AI agents, and serving a growing number of customers who use our AI products every day.\nThis role sits at the heart of how we measure and improve those products. Your job is to make our LLM-powered features provably reliable - not just impressive in a demo.\nPrompt engineering is a central part of the role. You’ll design, test, and iterate on prompts and agent behaviour directly. But your defining contribution will be rigour: turning “this seems to work” into measurable, repeatable, production-ready behaviour through evaluation and observability.\nWe work across the major model ecosystems - OpenAI, Google Gemini, and Anthropic - and you’ll regularly make pragmatic decisions about model choice, quality, latency, and cost.\nYou’ll work closely with backend engineers, fellow evaluation engineers, and product managers. Together, you’ll dig into real customer interactions, identify where our AI succeeds or fails, and turn those insights into better prompts, agents, tools, and product experiences.\nYOUR RESPONSIBILITIES:\nEvaluation & Observability \nBuild high-quality evaluation datasets with ground truth and define task-specific metrics such as correctness, task completion, efficiency, cost, token usage, and safety;\nRun model bake-offs and before/after prompt experiments - every meaningful prompt change should be a measured experiment, not a guess;\nDesign evaluations for harder cases where there is no clean ground truth, including agentic workflows, tool-calling correctness, and structured-output validity;\nOwn production observability through trace analysis, per-tool dashboards, error classification, and session-level review;\nRoot-cause failures, identify recurring patterns, understand their impact on quality and cost, and turn findings into prompt improvements and engineering tickets;\nHelp close the loop between users and product teams by translating real customer behaviour and product questions into clear, measurable insights about how our AI features are used and where they can improve.\n Prompting & Agent Engineering \nDesign, version, test, and continuously improve system prompts for domain-specific agents;\nArchitect agent behaviour, including tool orchestration, multi-turn flows, and safeguards such as plan-before-execute, read-before-write, and always-confirm for destructive actions;\nBuild reusable, modular prompt components for areas such as safety, tone, context, and tool documentation;\nSelect the right model for each task, balancing quality, latency, and cost.\n Tooling & Integrations \nWork across the API layer our AI agents act through, integrating the REST endpoints that expose application data and operations to the LLM;\nHelp design and implement guardrails that prevent agents from corrupting user data, including permission enforcement and safe defaults;\nDebug the full request path end to end - from an agent’s decision and tool call to the live result - correlating application logs with agent traces.\nOUR EXPECTATIONS:\nStrong prompt engineering and LLM application experience - you’ve shipped agents or LLM-powered features to production, not just prototyped them;\nHands-on evaluation experience and a data-driven mindset - you’ve built evaluation datasets and metrics, and you measure results before claiming something works;\nPython experience and comfort consuming and integrating REST APIs;\nFamiliarity with modern LLM tooling and patterns, including function/tool calling, RAG, agentic workflows, observability platforms, and current model families;\nHealthy scepticism toward AI output and a strong instinct for safety, failure modes, and guardrail design.\nGREAT ADVANTAGE WILL BE:\nExperiment design and statistics literacy, including A/B testing, significance, and sampling, so your evaluation conclusions hold up under scrutiny;\nExperience designing LLM-as-a-judge evaluations and calibrating automated judges against human labels;\nRed-teaming and adversarial testing experience, including prompt injection, jailbreaks, and structured-output abuse;\nComfort working with data tools for exploring and slicing evaluation results, such as SQL, pandas, DuckDB, notebooks, or similar;\nExperience with platforms/solutions such as Langfuse, Langchain, n8n, Google ADK.\nWHAT WE OFFER:\nAt SiteGround, we work hard and challenge ourselves to exceed expectations. To support your drive, we provide a thoughtfully designed benefits package created to help you thrive both at work and beyond.\nCompetitive remuneration and a performance-based bonus system;\nShorter workday on Fridays - we finish at 3:00 PM;\n5 extra days off at 5 years of service, plus 1 more each year after - up to 10;\n2 extra paid days off for volunteering to support the causes you care about;\nPremium additional health insurance coverage and annual medical check-ups;\nModern and cozy offices across Bulgaria, plus hybrid work options;\nFree parking and a metro shuttle;\nChef-prepared breakfast and lunch provided daily at our HQ restaurant - all on us;\nFully equipped in-office gym with expert fitness guidance;\nOn-site sports at our HQ: yoga, spinning, Brazilian Jiu-Jitsu, dancing, table tennis, and more;\nMultisport or Coolfit cards fully covered by the company;\nFree massages;\nMemorable team events and internal gatherings;\nKnowledge-sharing culture through meetups, events and conferences;\nFree employee access to the SiteGround products;\nAnniversary and company gifts along the way.","description_format":"text","description_chars":5734,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Health insurance","Hybrid work","Insurance coverage"],"hiring_locations":[{"name":"Bulgaria","iso":"BG","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Cloud Platforms (IaaS & PaaS)","Web Hosting"],"lifecycle":[{"event":"open","at":"2026-09-23T10:47:07Z"}],"visa":[],"liveness":{"score":37,"band":"fade","label":"Fading","p_open":1,"p_active":0.678,"p_room":0.55,"age_days":25,"expected_fill_days":25,"reasons":["conf:0","velocity","win:tail"],"computed_at":"2026-10-17T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/siteground-ai-evaluation-engineer","json_url":"https://alion.io/job/siteground-ai-evaluation-engineer.json","meta":{"generated_at":"2026-10-17T10:04:48Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":13221,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":55484},"rest":"https://alion.io/mcp/rest/get_company?id=55484"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsiteground-ai-evaluation-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsiteground-ai-evaluation-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsiteground-ai-evaluation-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/siteground-ai-evaluation-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsiteground-ai-evaluation-engineer"}]}