{"id":1262993,"url":"https://alion.io/job/bigbear-ai-qa-automation-engineer","title":"QA Automation Engineer","company":{"id":34812,"name":"BigBear AI","domain":"bigbear.ai","url":"https://alion.io/company/bigbear-ai-holdings-inc","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"iCIMS","truth_index":null},"role":"QA","role_family":"QA","seniority":"middle","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"inferred","locations":[],"countries":[],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Anthropic","optional":false},{"name":"AWS","optional":false},{"name":"AWS Bedrock","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"Claude","optional":false},{"name":"Defense in Depth","optional":false},{"name":"Docker","optional":false},{"name":"FedRAMP","optional":false},{"name":"Gemini","optional":false},{"name":"Git","optional":false},{"name":"GitHub Actions","optional":false},{"name":"JavaScript","optional":false},{"name":"Kubernetes","optional":false},{"name":"LLM","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"n8n","optional":false},{"name":"NIST AI RMF","optional":false},{"name":"OpenAI","optional":false},{"name":"Playwright","optional":false},{"name":"PostgreSQL","optional":false},{"name":"Power Automate","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"SQL","optional":false},{"name":"TypeScript","optional":false},{"name":"Vertex AI","optional":false},{"name":"Zapier","optional":false}],"status":"live","first_seen_at":"2026-09-25T21:15:56Z","employer_posted_date":"2026-09-25","last_verified_at":"2026-09-26T12:53:43Z","board_verified":true,"closed_at":null,"days_open":2,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":2},"description":"Overview\nAsk Sage( BigBearAI company) is seeking a QA Automation Engineer that will translate nuanced product requirements and customer workflows into effective test coverage, build and maintain automation with Playwright, and partner with Product and Engineering to make informed release decisions as the platform evolves. We are looking for an engineer who can reason about complex, nondeterministic AI behavior, design risk-based coverage that reflects real customer impact, and produce automation that engineers trust. The engineer will own the automated coverage and the evidence behind release readiness across browser, application, and backend boundaries, where auditability, security, and government compliance expectations are part of everyday delivery.\nWhat you will do\nTranslate requirements into testable outcomes. Understand product goals, customer use cases, and complex feature interactions; identify ambiguities and define acceptance criteria with Product and Engineering.\nOwn automated test coverage. Design, implement, and maintain Playwright tests covering critical user journeys, feature functionality, and regressions, supported by API and integration testing.\nDevelop a risk-based testing strategy. Prioritize coverage according to customer impact, feature dependencies, and the product roadmap; adapt deliberately as priorities change. · Test conversational AI workflows. Validate streaming responses, conversation history, file uploads, document retrieval and citations, model selection, and tool execution, including interruptions, timeouts, and partial failures.\n Evaluate AI response quality. Build representative evaluation datasets and scoring criteria for accuracy, grounding, instruction following, and appropriate handling of unsafe requests. Account for natural variation in model responses.\nMake automation reliable and useful. Integrate tests into CI/CD, investigate flaky tests, maintain isolated test data, and provide actionable failure diagnostics.\nCommunicate release readiness. Report defects with reproducible evidence, customer impact, and severity; explain coverage gaps and residual risks before UAT and release. \nPreserve decision history. Document expected behavior, approved changes, and the rationale behind testing decisions so the team can distinguish intended changes from regressions.\nCover platform and AI infrastructure surfaces. Extend automation across backend APIs, authentication flows, billing and token behavior, model routing, AI workflow execution, file parsing, MCP/tool execution, agentic harnesses, and passthrough APIs, including provider-facing API compatibility. \nControl test cost and execution footprint. Design AI workflow coverage that minimizes unnecessary token usage, external provider calls, latency, and execution cost without sacrificing signal. \nAutomate security-sensitive validation. Build repeatable coverage for user isolation, permission boundaries, input validation, sanitization, rate limits, replay prevention, safe error handling, and layered control behavior. \nApply AI-assisted testing tools responsibly. Use AI assistance to accelerate test design, generation, triage, and maintenance while critically reviewing generated tests, assertions, and proposed repairs against reliability, reviewability, and deterministic validation standards. \nScale the automation footprint. Maintain test infrastructure, fixtures, and test data management that must grow with an expanding product surface area and an active engineering team.\nWhat you should have\n4+years of experience QA automation engineering experience \nDemonstrated experience building and maintaining automated tests with Playwright, including fixtures, resilient locators, assertions, network handling, and trace-based debugging.\nStrong coding ability in TypeScript or JavaScript, with experience writing maintainable test code and reviewing changes through Git. \nExperience with API testing, CI/CD integration, test isolation, and diagnosing failures across browser, application, and backend boundaries. \nAbility to reason about complex requirements, explore edge cases, and balance testing depth with delivery priorities. · Clear written and verbal communication: explaining defects, uncertainty, tradeoffs, and release risk to technical and nontechnical stakeholders. Experience testing authentication, authorization, permissions, and separation of customer data. \nFamiliarity with validating Defense in Depth behavior: confirming that multiple layers of controls work together, without this being a dedicated DevSec role. \nStrong troubleshooting and analytical skills, with the ability to work independently and as part of a team. \nAbility to obtain a Department of Defense Secret clearance\nWhat we'd like you to have\nExperience testing LLM applications, retrieval-augmented generation, or AI agents. \nExperience with Python for API testing, test utilities, test data generation, or AI evaluation workflows. \nExperience with enterprise or government platforms and auditable test evidence. · Experience testing model gateways, MCP tools, tool-using systems, or workflow automation and no-code/low-code platforms (e.g., Power Automate, Zapier, Make, n8n). \nFamiliarity with flagship Generative AI provider APIs (Google Vertex AI, AWS Bedrock, Microsoft Azure OpenAI) and models (OpenAI GPT, Anthropic Claude, Google Gemini). \nExperience with CI/CD pipelines (e.g., GitHub Actions), Docker, Kubernetes, and observability/monitoring tooling. \n Experience with PostgreSQL and SQL for test data setup, teardown, and validation. \nKnowledge of government compliance frameworks (FedRAMP, NIST AI RMF, CMMC 2.0).\n Active DoD security clearance at the Secret level or above.\nAbout BigBear.ai\nBigBear.ai is a leading provider of AI-powered decision intelligence solutions for national security, supply chain management, and digital identity. Customers and partners rely on Bigbear.ai’s predictive analytics capabilities in highly complex, distributed, mission-based operating environments. Headquartered in McLean, Virginia, BigBear.ai is a public company traded on the NYSE under the symbol BBAI. For more information, visit https://bigbear.ai/ and follow BigBear.ai on LinkedIn: @BigBear.ai and X: @BigBearai.\nBigBear.ai is an Equal opportunity employer all protected groups, including protected veterans and individuals with disabilities.","description_format":"text","description_chars":6377,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":true,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Decision Intelligence"],"lifecycle":[{"event":"open","at":"2026-09-25T21:15:56Z"}],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":2,"expected_fill_days":18,"reasons":["conf:40","velocity","win:early"],"computed_at":"2026-09-28T05:45:00Z"},"pay":null,"html_url":"https://alion.io/job/bigbear-ai-qa-automation-engineer","json_url":"https://alion.io/job/bigbear-ai-qa-automation-engineer.json","meta":{"generated_at":"2026-09-28T06:45:52Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","usage":{"tier":"crawler","counted_by":"address","units_charged":1,"used_today":4867,"day_limit":5000,"remaining_today":133,"minute_limit":60,"resets_at":"2026-09-29T00:00:00Z"}}}