{"id":3292323,"url":"https://alion.io/job/simple-technology-solutions-ai-quality-test-engineer","title":"AI Quality & Test Engineer","company":{"id":8717,"name":"Simple Technology Solutions","domain":"simpletechnologysolutions.com","url":"https://alion.io/company/simpletechnologysolutions","size_band":"1-10","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":null},"role":"QA","role_family":"QA","seniority":"senior","employment_type":null,"work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"posting_text","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Washington, United States"],"countries":["US"],"hiring_countries":["US"],"hiring_countries_total":1,"salary":null,"salary_estimate":{"min_usd":85000,"max_usd":168000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":116},"experience_years_min":8,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Anthropic","optional":false},{"name":"AWS","optional":false},{"name":"AWS Bedrock","optional":false},{"name":"CI/CD","optional":false},{"name":"Claude","optional":false},{"name":"GitHub","optional":false},{"name":"Jira","optional":false},{"name":"LLM","optional":false},{"name":"LLM Evaluation","optional":false},{"name":"LLM Guardrails","optional":false},{"name":"Playwright","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"Scrum","optional":false},{"name":"TypeScript","optional":false}],"status":"live","first_seen_at":"2026-10-26T18:58:02Z","employer_posted_date":"2026-10-26","last_verified_at":"2026-10-27T09:47:24Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"At Simple Technology Solutions, our people are our priority. We know our team members are more than employees-they’re parents, friends, volunteers, artists, and athletes. That’s why we offer flexibility to help them thrive personally and professionally while delivering exceptional solutions to our Federal Government clients.\nOur culture is built on collaboration, continuous learning, and excellence. We are mentors and thought leaders who share knowledge and foster growth. Recognized as a “Best Place to Work,” we believe a range of perspectives helps us drive innovation and exceed customer expectations. At STS, taking care of our people isn’t a perk-it’s the standard.\nAs a HUBZone company, we also offer special incentives for team members living in qualified HUBZones. Check out the HUBZone map HERE to see if you qualify!\nQuick Position Overview:\nUS Citizenship and the ability to obtain and maintain a Public Trust clearance are required\nBachelor's Degree is required\nMinimum of 8 years' position related experience is required\nAWS Certified AI Practitioner or Developer (Associate) certification preferred for hire; the AWS and Anthropic Claude certifications listed under Required Certifications must be earned within 12 months of hire\nRemote position with occasional travel to client sites and STS events\nCandidates living in the DC Metro region (within driving distance for quarterly in-person team events) are strongly preferred; STS is a HUBZone company, so residence in a HUBZone is favored for remote candidates\nThe Role:\nThe AI Quality & Test Engineer owns the second human bookend of the STS SDLC: proving that what the team built does what the requirements say, works end to end for real users, and behaves acceptably as an AI system. You design the test strategy from the requirements trio, build and run end-to-end and user-acceptance testing, and define how AI outputs are evaluated so that quality is measured rather than assumed. Success looks like test gates the team trusts, defects found before users do, and clear evidence at every release that the solution meets its acceptance criteria. You bring the same discipline to engagements that run on AWS-native AI services or other client toolchains.\nThe AI Quality & Test Engineer at STS will:\nDefine the test strategy for each AI application project from the requirements trio, including unit, integration, end-to-end, user-acceptance, security, and AI-behavior testing, and the coverage and gate thresholds the project will use.\nDesign, build, and execute end-to-end and user-acceptance tests that exercise complete use cases as real users would, across DEV, TST, STG, and PRD.\nBuild evaluation harnesses for AI behavior, including golden datasets, rubric-based and model-assisted grading, regression suites for prompts and retrieval, and guardrail and safety checks.\nOperate as the tester in the team's AI development workflow, using and improving the shared testing agents and skills, and ensuring tests derived from use cases exist before application code is written.\nAutomate tests in the CI/CD pipeline in partnership with the platform engineer, and own the quality gates that govern promotion between environments.\nValidate acceptance criteria with the business analyst and real users, facilitate user-acceptance testing sessions, and capture results as release evidence.\nTriage, reproduce, and track defects in Jira, drive them to resolution with engineers, and analyze defect trends to improve upstream requirements and development practices.\nTest on AWS-native AI stacks, including Amazon Bedrock models, agents, and knowledge bases, when a client requires them in place of STS's standard toolchain.\nAssess performance, cost, and resilience of AI features under realistic load and failure conditions, and report findings with recommendations.\nContribute test standards, templates, and tooling back to the STS SDLC, and coach the team on test-driven development and AI evaluation practices.\nEducation and Experience:\nRequired\nBachelor's degree in Computer Science, Software Engineering, Information Systems, or a related technical field, or equivalent professional experience in addition to the minimum below\nMinimum of 8 years of software quality engineering or test automation experience, including at least 2 years testing applications built on large language models or other AI/ML components\nDemonstrated experience building automated end-to-end test suites integrated into CI/CD pipelines and defining quality gates for a delivery team\nExperience leading user-acceptance testing with business or mission users, ideally in a federal or regulated environment\nStrong test automation skills in Python or TypeScript using frameworks such as Playwright, pytest, or Jest, plus API testing tools\nPractical knowledge of LLM evaluation techniques, including golden datasets, rubric and model-graded evaluation, retrieval quality metrics, and prompt and model regression testing\nWorking knowledge of Amazon Bedrock and AWS application services sufficient to test and troubleshoot deployed AI features\nFluency with GitHub, Jira, and CI/CD pipelines, and with reading use cases and specifications to derive tests\nFamiliarity with security testing basics and federal compliance expectations for test evidence\nSkeptical in the best sense; assumes nothing works until it is demonstrated, including AI output that reads well\nCollaborative and constructive, treating defects as information for the team rather than blame\nSelf-directed and comfortable owning outcomes on a small, senior team where everyone contributes across the lifecycle\nCurious and current on AI tooling; treats AI assistants as part of the team and holds their output to the same standard as human work\nClear, candid communicator with federal clients and teammates; writes well and documents as a habit\nBias toward fast delivery and early user feedback over conservative gating; learns from failure quickly\nRequired Certifications (to be earned within 12 months of hire)\nAWS Certified AI Practitioner or AWS Certified Developer - Associate\nClaude Certified Developer - Foundations (CCDV-F) or Claude Certified Associate - Foundations (CCAO-F)\nPreferred Certifications\nISTQB Certified Tester Foundation Level (CTFL) and ISTQB Certified Tester AI Testing (CT-AI)\nAWS Certified Solutions Architect - Associate\nCompTIA Security+\nCertified ScrumMaster (CSM) or Professional Scrum Master (PSM I)\nWhat We Offer:\nCompetitive compensation package\nComprehensive health, dental, and vision insurance plans\nProfessional development and training\nPerformance-based incentives\n401(k) retirement savings plan\nWork from home stipend\nUnlimited PTO\nPaid holidays\nFlexible work schedule\nProfessional development budget to be used for training and certifications\nSTS is committed to equal employment opportunity and merit-based employment practices. STS provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination, harassment, and retaliation in all employment practices and decisions in accordance with applicable federal, state, and local laws.\nEmployment decisions at STS are based on individual qualifications, performance, skills, and business needs, without regard to race, color, religion, sex, national origin, age, disability, protected veteran status, sexual orientation, gender identity, genetic information, marital status, or any other status protected by applicable law.\nThis policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, compensation, training, transfer, discipline, termination, layoff, recall, and leaves of absence.\n---\nApplicants may request removal from our applicant database, or specific information about how the data is used by contacting .","description_format":"text","description_chars":7838,"description_truncated":false,"requirements":{"experience_years_min":8,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":true,"languages":[]},"benefits":["Continuous learning","Flexible schedule","Professional development","Retirement plans","Unlimited PTO","Vision insurance"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation","Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-10-26T20:31:13Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":29,"reasons":["conf:0","win:early"],"computed_at":"2026-10-27T05:46:45Z"},"pay":null,"html_url":"https://alion.io/job/simple-technology-solutions-ai-quality-test-engineer","json_url":"https://alion.io/job/simple-technology-solutions-ai-quality-test-engineer.json","meta":{"generated_at":"2026-10-27T11:35:45Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":13729,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-28T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":8717},"rest":"https://alion.io/mcp/rest/get_company?id=8717"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsimple-technology-solutions-ai-quality-test-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsimple-technology-solutions-ai-quality-test-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsimple-technology-solutions-ai-quality-test-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/simple-technology-solutions-ai-quality-test-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsimple-technology-solutions-ai-quality-test-engineer"}]}