{"id":2460202,"url":"https://alion.io/job/providius-senior-software-engineer-test","title":"Senior Software Engineer, Test","company":{"id":685625,"name":"Providius","domain":"providius.com","url":"https://alion.io/company/providius","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workable","truth_index":null},"role":"Backend","role_family":"Backend","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Hamilton, Canada"],"countries":["CA"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":100000,"max":130000,"currency":"CAD","period":"year","gross":null,"usd_annual":91451},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"CI/CD","optional":false},{"name":"Docker","optional":false},{"name":"Function Calling","optional":false},{"name":"Linux","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Tool Use","optional":false},{"name":"Agentic Workflows","optional":true},{"name":"Bitbucket","optional":true},{"name":"Chaos Engineering","optional":true},{"name":"Cypress","optional":true},{"name":"DeepEval","optional":true},{"name":"Hugging Face","optional":true},{"name":"Java","optional":true},{"name":"JMeter","optional":true},{"name":"JUnit","optional":true},{"name":"k6","optional":true},{"name":"Kubernetes","optional":true},{"name":"Langfuse","optional":true},{"name":"LangSmith","optional":true},{"name":"MLFlow","optional":true},{"name":"Playwright","optional":true},{"name":"PostgreSQL","optional":true},{"name":"Postman","optional":true},{"name":"Promptfoo","optional":true},{"name":"Pytest","optional":true},{"name":"Python","optional":true},{"name":"PyTorch","optional":true},{"name":"RAG","optional":true},{"name":"Ragas","optional":true},{"name":"TimescaleDB","optional":true},{"name":"Triton","optional":true},{"name":"TypeScript","optional":true},{"name":"vLLM","optional":true},{"name":"Weights & Biases","optional":true}],"status":"live","first_seen_at":"2026-10-13T17:13:22Z","employer_posted_date":"2026-10-13","last_verified_at":"2026-10-13T22:53:30Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"Providius is looking for an experienced Senior Software Engineer, Test to help us build stronger, faster, and more reliable ways of validating complex software systems.\nThis is a hands-on engineering role for someone who enjoys writing code, building automation, debugging difficult problems, and improving how software gets tested across an engineering organization.\nPosition Overview\nThis is a hands-on, individual-contributor role. You will spend most of your week writing test code, building test infrastructure, and working directly with engineers, not writing test plans for others to execute.\nWe are looking for someone to take our test automation and quality engineering practices to the next level. You will build the automated test suites and tooling underneath them, raise the standard for how the engineering team tests its own work, and help ensure our quality practices keep pace with how quickly modern software is now developed.\nSoftware gets written and changed faster than it did even a few years ago. AI-assisted development accelerates that further, but it also changes where defects come from and what traditional reviews and test suites may miss. We want someone who sees that as an engineering problem: building the coverage, tooling, feedback loops, and development practices that allow the team to move quickly without sacrificing reliability.\nOur products also increasingly include AI and machine-learning functionality, including LLM-based assistants and integrations with AI clients. These systems introduce a different class of testing problem. Outputs may vary between runs, model or prompt changes can alter behaviour without traditional code changes, and quality may need to be measured across a set of results rather than with a simple pass or fail.\nYou do not need to be an ML researcher, but you should have experience testing software with AI, ML, LLM, or other non-deterministic components and understand how to build credible evidence that these systems are ready to ship.\nYou will have meaningful autonomy and responsibility. You will work with engineering leadership to identify our biggest quality risks, establish practical approaches to addressing them, and implement the automation and tooling needed to make those approaches part of everyday development.\nDuties and Responsibilities\nDesign, write, and maintain automated test suites across unit, integration, API, end-to-end, and regression testing\nBuild and improve test infrastructure including fixtures, mocks, reusable test libraries, test data management, and simulation environments\nBuild and maintain CI/CD test gates, smoke tests, regression suites, coverage reporting, and automated release-readiness checks\nKeep automated testing fast, reliable, and useful enough that developers trust it and use it regularly\nBuild API and contract testing across services and integration boundaries\nEstablish performance, load, stress, and soak testing for high-throughput and real-time systems\nInvestigate and debug difficult failures across application, service, database, infrastructure, and deployment boundaries\nWork with engineers to determine appropriate test coverage and validation approaches for new features and systems\nReview test code with the same engineering standards applied to production code\nHelp developers improve the quality and testability of their own code rather than making testing the responsibility of a separate QA silo\nBuild reusable patterns and tooling that make it easier for engineers to write good tests\nEstablish practical testing standards and help drive their adoption across engineering\nIdentify gaps in existing test coverage and determine where deeper or different forms of validation are required\nHelp adapt our testing practices to the increased pace and changing failure modes introduced by AI-assisted software development\nUse modern development and AI tooling to improve the speed and effectiveness of test development while maintaining confidence in the results\nValidate that automated tests actually detect incorrect behaviour rather than simply confirming the current implementation\nShare testing practices, tooling, and lessons across the engineering team\nAI and ML Quality\nBuild and maintain evaluation approaches for AI, ML, and LLM-based product features\nTest non-deterministic systems using representative datasets, repeated evaluation, appropriate metrics, and acceptance thresholds\nEstablish regression testing for model, prompt, retrieval, and AI-system changes\nTest AI integrations end to end, including tool use, schemas, authentication, authorization, error handling, and unexpected responses\nHelp determine where automated evaluation is sufficient and where human review remains necessary\nValidate data and pipelines used to support AI and ML functionality\nIncorporate production failures and edge cases back into automated test and evaluation suites\nHelp define evidence-based release criteria for AI-enabled product functionality\nRequirements\nTech Stack\nOur engineering environment includes:\nLanguages: Python, Java, TypeScript\nTest frameworks: pytest, JUnit, Playwright or Cypress\nAPI and performance testing: Postman, k6 or JMeter\nData: PostgreSQL, TimescaleDB\nInfrastructure: Linux, Docker\nCI/CD: automated pipelines, test gates, coverage and release validation\nDevelopment tools: JIRA, Bitbucket\nAI and ML systems: Python-based ML and LLM tooling, model and data pipelines, LLM integrations, and Model Context Protocol (MCP)\nExperience with every technology listed is not required.\nRequired Skills / Experience\n5+ years of hands-on experience in software quality engineering, test automation, SDET, software engineering, or a closely related role\nStrong programming ability and experience building maintainable test automation and engineering tooling\nHands-on experience designing and implementing automated tests across multiple layers of a software system\nExperience building or substantially improving automated test frameworks or test infrastructure\nExperience integrating automated testing into CI/CD pipelines\nStrong API, service-level, integration, and regression testing experience\nExperience with performance, load, stress, or soak testing\nStrong debugging and troubleshooting skills across complex software systems\nExperience working with test data, fixtures, mocks, simulation, or comparable approaches to creating reliable test environments\nAbility to take incomplete or ambiguous requirements and determine what should be tested, how it should be tested, and what acceptable behaviour looks like\nExperience reviewing test code and helping software engineers improve their own testing practices\nStrong understanding of software engineering practices, not just QA processes\nExperience testing software containing AI, machine learning, LLM, or other non-deterministic functionality\nUnderstanding of how validating probabilistic systems differs from testing deterministic software\nExperience working in fast-moving software development environments with frequent code changes and automated delivery practices\nStrong communication and collaboration skills\nNice to Haves\nExperience building evaluation suites for LLM or machine-learning based products\nExperience defining representative or golden datasets, scoring rubrics, quality metrics, acceptance thresholds, or automated graders\nExperience with prompt evaluation and regression testing\nExperience evaluating RAG systems, including retrieval quality, grounding, and citation accuracy\nExperience testing agentic workflows, tool calling, MCP servers, or other AI integrations\nExperience with LLM evaluation frameworks such as promptfoo, DeepEval, Ragas, or similar\nFamiliarity with LLM tracing and observability tools such as Langfuse or LangSmith\nExperience using LLM-as-judge techniques and understanding their limitations\nExperience validating fine-tuned models or comparing model versions\nExperience testing data pipelines, ML pipelines, or model-serving infrastructure\nFamiliarity with PyTorch, Hugging Face, MLflow, Weights & Biases, or related ML tooling\nExperience testing GPU inference systems such as vLLM or Triton\nExperience with security testing for AI systems, including prompt injection, data leakage, and authorization boundaries\nExperience with chaos engineering, fault injection, or resilience testing\nExperience testing high-throughput, real-time, distributed, networking, or data-intensive systems\nFamiliarity with networking concepts and protocols\nExperience with Docker, Kubernetes, Linux, and on-premise or hybrid deployments\nBenefits\nPay\n$100,000 - $130,000 per year\nBenefits\nDental care\nExtended health care\nOn-site parking\nPaid time off\nVision care\nWhy Join Providius\nWork on technically challenging software used in complex, real-time environments\nSpend your time building testing systems and automation rather than managing a QA process\nHave direct influence over how quality engineering is done across the company\nWork closely with software, infrastructure, networking, and ML engineers\nHelp define practical approaches to testing AI-enabled software as those systems become a larger part of our products\nWork in a small engineering organization where strong individual contributors can have significant impact\nTake ownership of difficult problems without layers of process or bureaucracy","description_format":"text","description_chars":9334,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Custom Software Development"],"lifecycle":[{"event":"open","at":"2026-10-13T17:13:22Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":22,"reasons":["conf:3","win:early"],"computed_at":"2026-10-14T02:05:24Z"},"pay":{"stated_usd_annual":91451,"is_top_pay":false},"html_url":"https://alion.io/job/providius-senior-software-engineer-test","json_url":"https://alion.io/job/providius-senior-software-engineer-test.json","meta":{"generated_at":"2026-10-14T02:05:24Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":3084,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-15T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":685625},"rest":"https://alion.io/mcp/rest/get_company?id=685625"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fprovidius-senior-software-engineer-test"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fprovidius-senior-software-engineer-test"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fprovidius-senior-software-engineer-test"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/providius-senior-software-engineer-test\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fprovidius-senior-software-engineer-test"}]}