{"id":1652651,"url":"https://alion.io/job/def-con-social-model-test-and-measurement-engineer","title":"Model Test and Measurement Engineer","company":{"id":43323,"name":"DEF CON Social","domain":"defcon.social","url":"https://alion.io/company/def-con-social","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":null},"role":"QA","role_family":"QA","seniority":null,"employment_type":null,"work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"posting_text","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["McLean, United States","United States"],"countries":["US"],"hiring_countries":["US"],"hiring_countries_total":1,"salary":{"min":150000,"max":190000,"currency":"USD","period":"year","gross":null,"usd_annual":190000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":true,"technologies":[{"name":"Python","optional":false},{"name":"SQL","optional":false},{"name":"NIST AI RMF","optional":true}],"status":"live","first_seen_at":"2026-10-01T13:08:05Z","employer_posted_date":"2026-10-01","last_verified_at":"2026-10-28T23:01:12Z","board_verified":true,"closed_at":null,"days_open":27,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":27},"description":"ABOUT DEFCON AI\nRESILIENCE IN THE FACE OF DISRUPTION. DEFCON AI is an insights company that leverages artificial intelligence, mathematical optimization, data analytics, and software engineering for resilient optimization of complex systems.\nIn today’s dynamically changing world, DEFCON AI’s technology aligns outcomes with operational goals, better decision making, and empowers customers to anticipate assess, and mitigate the impacts of disruptions.\nBe the independent voice that keeps the whole program honest - your evaluation is the standard everyone else is held to. \nAbout the Role \nYou'll join the analytics and AI engineering team behind a system that genuinely matters: an AI-assisted platform that brings together records from dozens of disparate data sources, resolves them to the correct individual, highlights what analysts should review first, and provides transparent, explainable recommendations that users can trust. Operating within a secure government cloud environment, the platform tackles complex challenges in AI, data integration, and decision support where quality, trust, and accountability are mission-critical. \nAs a Model Test & Measurement Engineer, you'llown the evaluation framework that helps ensure those systems perform as intended. You'llbuild and maintainlabeled ground truth datasets, design statistically sound audit and sampling methodologies, measure model performance across releases, and create the evidence packages that support deployment decisions. You'llindependently validateboth scoring and generative AI capabilities, helping the team understand not only whether a model works, but how confidently its outputs can be trusted. \nThis is a role with genuine influence. Your assessments will inform release decisions, drive improvement efforts, and provide the objective evidence customers rely on when evaluating system performance. You'llwork closely with data scientists, AI engineers, and technical leadership while maintainingthe independence needed to provide clear, defensible evaluations. You do not build the models you validate, and you do not approve thresholds or release authorization against your own evidence. \nIf you'reenergized by measurement, validation, and making complex AI systems more trustworthy, this is an opportunity to have an outsized impact on both the technology and the mission it supports. \nThis is a fully remote role with occasional travel to DEFCON AI headquarters, customer sites, and partner facilities as needed. \nKey Responsibilities \nConstruct labeled ground truth for model evaluation \nDesign and run sampled audits \nGate releases against model version; maintain version inventory, evaluation records, and rollback triggers \nRun drift and override review \nProduce human-oversight and fairness / disparate-effect evidence \nMaintain independence from the build roles: produce evaluation evidence, but do not build the models, approve thresholds, or authorize releases against that evidence \nMeasure workflow improvement: review time, throughput, backlog movement, and override and rework rates \nDesign evaluation-phase QC: sample selection that does not mix populations, the unit of review, and fair comparison when methods or searched sources differ \nDefine the measurement events other teams must emit, and confirm the review workspace emits workflow events from first use \nRequired Qualifications \n5+ years with model validation as a named responsibility, not a side task \nDirect experience with ground-truth construction and sampling design \nStrong Python and SQL; comfort working independently from the teams whose models you evaluate \nUS Citizenship Required \nActive US Secret clearance \nElevated personnel security requirements apply to portions of this work and are discussed during screening \nPreferred Qualifications \nRegulated-industry or government model-risk background \nExperience with fairness / disparate-effect testing and human-oversight documentation \nNIST AI RMF or comparable practice \nActive Top Secret clearance \nWhat Success Looks Like \nEvaluation evidence a customer can rely on, independent of the teams that build the models \nReleases gated against a clear version and evaluation record \nWorkflow metrics (review time, throughput, backlog, override/rework rates) that give the program an honest read on whether it's working \nWhat We Offer \nA fully remote, results-based environment \nCompetitive salary, bonus, and equity package \n100% employer paid, comprehensive health insurance including medical, dental, and vision for you and your family \nUnlimited PTO, with your manager's approval \nFlexible work environment where you manage your work day \n14 weeks of fully-paid parental leave \nSalary Range: $150,000-$190,000. This represents the typical salary range for this position based on experience, skills, and other factors. \nWe’re an Equal Opportunity Employer: You’ll receive consideration for employment without regard to race, sex, color, religion, sexual orientation, gender identity, national origin, protected veteran status, or on the basis of disability. \nApplicant Data Disclosure \nBy submitting an application, you acknowledge that Defcon AI uses third-party service providers to facilitate its recruitment and hiring processes. These providers include applicant tracking systems, candidate verification platforms, and fraud detection tools (collectively, \"Hiring Platforms\"). Your application materials, including your résumé, cover letter, work samples, responses to application questions, and any other information you submit, may be transmitted to and processed by these Hiring Platforms for the following purposes: \nManaging and administering your application throughout the hiring process; \nVerifying the accuracy and authenticity of application materials, including by cross-referencing information you provide against publicly available sources and proprietary databases; \nIdentifying indicators of potentially fraudulent, fabricated, or materially misleading application content, including but not limited to discrepancies between submitted materials and publicly available professional profiles, geographic anomalies, and fabricated work histories. \nApplications that are flagged through this process as containing indicators of fraud or material misrepresentation may be declined from further consideration. If you have questions about the status of your application or the evaluation process, please contact . \nDefcon AI requires its Hiring Platform providers to process your information solely for the purposes described above and in accordance with applicable law. Your information will be retained only for as long as necessary to fulfill these purposes and any applicable legal obligations, after which it will be deleted in accordance with Defcon AI's data retention policies.\nFor more information about how your data is used, please refer to our Privacy Policy and Applicant Privacy Notice.","description_format":"text","description_chars":6959,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":true,"languages":[]},"benefits":["Equity","Flexible schedule","Health insurance","Parental leave","Unlimited PTO"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Online Forums & Community Platforms"],"lifecycle":[{"event":"open","at":"2026-10-02T00:41:29Z"}],"visa":[],"liveness":{"score":70,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.778,"p_room":0.9,"age_days":26,"expected_fill_days":42,"reasons":["conf:0","velocity","win:mid"],"computed_at":"2026-10-28T05:59:10Z"},"pay":{"stated_usd_annual":190000,"is_top_pay":true},"html_url":"https://alion.io/job/def-con-social-model-test-and-measurement-engineer","json_url":"https://alion.io/job/def-con-social-model-test-and-measurement-engineer.json","meta":{"generated_at":"2026-10-28T23:41:50Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":32503,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-29T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":43323},"rest":"https://alion.io/mcp/rest/get_company?id=43323"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fdef-con-social-model-test-and-measurement-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fdef-con-social-model-test-and-measurement-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fdef-con-social-model-test-and-measurement-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/def-con-social-model-test-and-measurement-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fdef-con-social-model-test-and-measurement-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":1652651}},{"tool":"search_jobs","arguments":{"company":"defcon.social","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":1652651},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}