{"id":3605353,"url":"https://alion.io/job/weekday-principal-software-quality-engineer","title":"Principal Software Quality Engineer","company":{"id":7097,"name":"Weekday","domain":"weekday.works","url":"https://alion.io/company/weekday","size_band":"51-200","is_staffing_agency":true,"employer_type":"agency","is_intermediary":false,"listed_via":null,"ats_vendor":"Workable","truth_index":{"grade":"B","score":81,"open_postings":92,"ghost_share":0,"stale_share":0.978,"repost_share":0,"time_to_fill_p50_days":6,"computed_at":"2026-11-03T05:54:40Z"}},"role":"QA","role_family":"QA","seniority":"lead","employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"posting_text","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Bengaluru, India"],"countries":["IN"],"hiring_countries":["IN"],"hiring_countries_total":1,"salary":null,"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"C++","optional":false},{"name":"Cassandra","optional":false},{"name":"CI/CD","optional":false},{"name":"CockroachDB","optional":false},{"name":"Couchbase","optional":false},{"name":"DynamoDB","optional":false},{"name":"GitHub Actions","optional":false},{"name":"Google Cloud Spanner","optional":false},{"name":"HBase","optional":false},{"name":"Java","optional":false},{"name":"Jenkins","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"Redis","optional":false},{"name":"Rust","optional":false},{"name":"ScyllaDB","optional":false},{"name":"Slack","optional":false},{"name":"TiDB","optional":false},{"name":"YugabyteDB","optional":false},{"name":".NET","optional":true},{"name":"AWS","optional":true},{"name":"Azure","optional":true},{"name":"C#","optional":true},{"name":"GCP","optional":true},{"name":"IAM","optional":true},{"name":"JavaScript","optional":true},{"name":"Kubernetes","optional":true},{"name":"Node JS","optional":true}],"status":"live","first_seen_at":"2026-11-02T11:34:57Z","employer_posted_date":"2026-11-02","last_verified_at":"2026-11-04T01:06:54Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":1},"description":"This role is for one of Weekday’s clients\nMin Experience: 10+ years\nLocation: Remote (India), Bengaluru, Karnataka, India\nJobType: full-time\nWe are looking for an experienced Principal Software Quality Engineer to define and drive the quality strategy for a distributed database platform. This is a senior individual-contributor role requiring deep expertise in distributed database testing, systems engineering, debugging, and test automation.\nThe ideal candidate will have hands-on experience working with distributed database or storage systems, preferably NoSQL platforms. They should be comfortable reading server-side code, tracing complex defects to their root cause, designing fault-injection scenarios, and collaborating with QE, client, and server engineering teams to ensure product reliability.\nThe role will also lead the integration of AI agents into testing frameworks, enabling automated test generation, failure triage, and evaluation of agent reliability.\nRequirements\nKey Responsibilities\nQuality Strategy and Test Architecture\nOwn the overall quality strategy for the distributed database platform, including test architecture, coverage models, release gates, and risk-based prioritization.\nDrive quality across server components, client SDKs, operator and backup tooling, and management interfaces.\nDesign comprehensive testing strategies for replication, rebalancing, partitioning, sharding, consistency models, transactions, isolation, secondary indexes, query planning, compaction, backup and restore, snapshots, and cross-datacenter replication.\nEstablish reliable coverage for critical database operations and ensure quality requirements are incorporated early in the development lifecycle.\nDistributed Systems and Fault-Injection Testing\nValidate cluster correctness under challenging production-like conditions, including node failures, network partitions, clock skew, disk-full scenarios, slow disks, rolling upgrades, mixed-version deployments, and long-running soak tests.\nDesign and execute fault-injection scenarios using process crashes, GDB-driven state manipulation, packet drops, latency injection, corrupted wire data, oversized inputs, and version skew between master and replica nodes.\nIdentify failure scenarios that conventional positive testing cannot adequately cover.\nInvestigate complex system defects involving off-by-one errors, out-of-bounds reads, encoder/decoder mismatches, compression and CRC issues, and consistency violations.\nReproduce defects at the source-code level, identify suspect functions, and provide developers with minimal reproducible test cases and detailed root-cause analysis.\nTest Automation and Framework Development\nBuild and maintain scalable test frameworks using Python, pytest, custom harnesses, API testing, and end-to-end automation.\nEnsure test suites run reliably across shared test infrastructure and CI/CD environments such as GitHub Actions and Jenkins.\nImprove test-farm reliability by reducing flaky tests, identifying noisy failures, and strengthening regression coverage.\nDevelop testing strategies across unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and long-running regression testing.\nAI-Agent Integration and Testing\nLead the integration of AI agents into existing testing frameworks.\nDesign agent workflows that can plan and execute tests, interact with tools, inspect logs, traces, metrics, and database state, and generate actionable defect reports.\nDevelop evaluation frameworks using golden datasets, graders, and regression suites to measure the reliability and accuracy of AI-generated outputs.\nIntegrate AI agents with pytest, custom testing harnesses, and MCP-style tool servers.\nEnable agents to generate negative and edge-case scenarios, maintain flaky tests, and assist with failure triage while retaining human oversight where required.\nCross-Functional Collaboration and Technical Leadership\nParticipate in feature reviews, product requirement discussions, design reviews, and client specifications to define testability requirements before implementation.\nCreate test plans for new features and collaborate with development teams to validate technical designs.\nConvert production and pre-release incidents into permanent regression tests, monitoring checks, and preventive measures.\nDrive debugging and defect-resolution efforts across QE, client, and server engineering teams.\nMentor senior QEs and SDETs on debugging techniques, test architecture, and framework development.\nEstablish engineering practices and quality standards that can be adopted across teams without requiring direct people-management responsibilities.\nRequired Skills and Qualifications\n10-18 years of experience in Software Quality Engineering, SDET, or Systems Engineering, with a strong focus on quality systems and demonstrated Staff or Principal-level technical impact.\nDeep hands-on experience testing distributed databases or storage systems, preferably NoSQL platforms such as Cassandra, ScyllaDB, MongoDB, Couchbase, Redis, DynamoDB, or HBase. Experience with distributed relational databases such as CockroachDB, TiDB, YugabyteDB, or Spanner-style systems is also relevant.\nStrong experience testing replication, partitioning, sharding, consistency, transactions, secondary indexes, backup and restore, and cross-datacenter replication.\nSolid understanding of distributed-systems fundamentals, including replication protocols, consensus, quorum reads and writes, strong and eventual consistency, split-brain scenarios, clock skew, and failure/recovery semantics.\nAbility to read and debug server-side code written in C, C++, Go, Rust, or Java and trace defects to specific functions and root causes.\nUnderstanding of protocol and serialization edge cases, including wire formats, MessagePack, Protocol Buffers, Thrift, and compression codecs.\nStrong software engineering and automation skills in Python, with proficiency in at least one systems programming language.\nPractical experience with fault injection, GDB, chaos-style testing, process termination, network partitions, latency and packet-loss injection, disk faults, corrupted inputs, version skew, and boundary-value testing.\nExperience across multiple testing layers, including unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and soak testing.\nProven ability to lead debugging and root-cause investigations across QE, client, and server teams, providing actionable technical analysis and following defects through to resolution.\nStrong CI/CD experience with GitHub Actions, Jenkins, or equivalent tools, along with experience managing shared test infrastructure.\nPractical experience using AI agents for testing, including agent workflows, tool integration, prompt and evaluation design, and integration with real-world testing frameworks.\nExcellent written and verbal communication skills, with the ability to produce clear technical tickets, investigation reports, Slack discussions, and RFCs.\nStrongly Preferred Skills\nExperience testing client SDKs across multiple programming languages, including Python, Java, Go, C, Node.js, and .NET.\nKnowledge of compatibility matrices, upgrade/downgrade scenarios, and behavioral consistency across client SDKs.\nExperience testing backup and restore, checkpointing, snapshotting, and WAL/journal correctness, including performance at terabyte scale.\nExpertise in query-engine and secondary-index testing, including query planning, index selection, pagination, boundary values, and encoding/collation correctness.\nKubernetes and database operator testing experience using tools such as KUTTL, Chainsaw, or envtest.\nCloud infrastructure experience across AWS, GCP, or Azure, including TLS/mTLS, IAM, secrets management, and air-gapped installations.\nSecurity-focused testing experience, including fuzzing, CVE reproduction, and identifying memory-safety issues such as out-of-bounds reads and use-after-free defects in native code.\nSuccess Metrics\nCritical database workflows, including replication, rebalancing, consistency, transactions, secondary indexes, backup and restore, and cross-datacenter replication, have comprehensive and reliable test coverage.\nCritical regressions are identified before release through high-signal, low-flakiness automated tests.\nDefect reports consistently include minimal reproducible cases, root-cause analysis, and suspected source-code locations.\nDevelopment teams experience shorter defect-resolution cycles across the areas owned by the role.\nAI agents become an integral part of the testing framework, supporting test generation and failure triage with measurable reliability through evaluation suites.\nTest-farm flakiness decreases, upgrade/downgrade coverage improves, and new features have test plans before implementation is complete.\nEngineering teams across QE, client, and server functions adopt the testing frameworks, practices, and quality standards introduced by the role.\nRole Expectations\nThis is a senior technical individual-contributor position, not a manual QA or people-management role. The candidate is expected to write framework code, debug server processes, investigate low-level system behavior, evaluate AI-agent performance, and build reliable testing infrastructure that other engineers can operate and extend.\nMust-Have Skills\nDistributed Database\nSoftware Testing\nGood-to-Have Skills\nDatabase Testing","description_format":"text","description_chars":9450,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"India","iso":"IN","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-11-02T11:34:57Z"}],"visa":[],"liveness":{"score":38,"band":"fade","label":"Fading","p_open":1,"p_active":0.379,"p_room":1,"age_days":0,"expected_fill_days":6,"reasons":["conf:0","agency","stale_co","velocity","win:early","comp:brand"],"computed_at":"2026-11-03T05:54:40Z"},"pay":null,"html_url":"https://alion.io/job/weekday-principal-software-quality-engineer","json_url":"https://alion.io/job/weekday-principal-software-quality-engineer.json","meta":{"generated_at":"2026-11-04T03:29:13Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":5375,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-05T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":7097},"rest":"https://alion.io/mcp/rest/get_company?id=7097"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fweekday-principal-software-quality-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fweekday-principal-software-quality-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fweekday-principal-software-quality-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/weekday-principal-software-quality-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fweekday-principal-software-quality-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3605353}},{"tool":"search_jobs","arguments":{"company":"weekday.works","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3605353},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}