Average distance between completion and exact task success on artifacts that report both.
Loading paired evidence…
STATE OFAGENT
RELIABILITY2026
RELIABILITY2026
OPEN RESEARCH RELEASE / LIVE EVIDENCE
Completion is not
Completion is not
correctness.
One reproducible view across every committed real-model result: exact task success, completion, safety boundaries, uncertainty, cost, latency, and provenance—without manufacturing a universal score.
—model evals
—scenario trials
—verified labs
—observed failures
—recorded spend
Median width of the committed 95% confidence interval for exact-task endpoints.
A SCORE WITHOUT ITS INTERVAL OVERSTATES THE RESULTA valid rule from a clean twin is reused where one deciding fact reverses it.
SIMILARITY ERASES THE EXCEPTIONModel results need provenance. A floating alias can exercise different weights on the next run.
Loading provenance…INTERACTIVE EVIDENCE WORKBENCH
Ask a narrower question.
Every bar names its source metric and opens the exact committed result. Filter first; compare only like work.
EXACT TASK SUCCESSLoading evidence…
0.501.00
Selected endpoints remain lab-specific. Use this view to inspect evidence, not to average unlike tasks.
MODEL TERRAIN / COVERAGE BEFORE RANK
Where was each model actually tested?
Medians summarize the current filtered portfolio. Uneven coverage is shown beside every number and is never filled with a zero.
FAILURE TERRAIN / REPOSITORY INCIDENCE
The patterns that travel.
Counts mean “observed in these labs,” not real-world prevalence. Industry and contract filters narrow the map.
Boundary: this is an automatically generated snapshot of this repository’s committed evidence. It is not a market ranking, production certification, regulator approval, or estimate of failure prevalence.
OPENAAU
CHALLENGE01
CHALLENGE01
COMMUNITY RELIABILITY LEAGUE / $0 TO ENTER
30 minutes. One claim.
30 minutes. One claim.
Bring the receipt.
Choose a bounded mission. Reproduce a result, break an assumption, or adapt a proven contract to help different people. CI checks the evidence; the public board credits the work.
—live missions
—community participants
—community finishes
—reference finishes
—earned, never claimed
MISSION BOARD / FIVE STARTING POINTS
Pick the failure you want to make visible.
THE 30-MINUTE ROUTE
A small finish with a durable receipt.
The clock is a guide, not a gate. Adapt missions usually take longer; every step remains inspectable.
- 00—05ChooseClaim a mission and open its starter lab.
- 05—12RunExecute the deterministic mock—no key, no spend.
- 12—20TraceLink the claim to one exact scenario or branch.
- 20—27RecordCommit the result and a machine-readable submission receipt.
- 27—30ProveRun
aau challenge validateand open the PR.
PUBLIC SCOREBOARD / EVIDENCE, NOT POINTS
Who returned with a receipt?
References are not community wins. They show the finish standard while the first independent contributors take the open positions.
ACHIEVEMENTS / DERIVED BY CI
No vanity badges.
Boundary: Challenge status is derived from repository evidence. It is not a model leaderboard, production certification, regulator approval, or permission to automate protected authority.
NEW / RIGHTS CONTINUITY CONTRACT
The appeal survives.
The appeal survives.
Does the protection?
One case can carry several rights on different clocks. The new matched suite catches the moment a technically valid main route silently loses coverage, urgent review, income, or another companion protection.
01 / MEDICAID + CHIPEx-parte first
Reuse reliable agency data before asking a family to rebuild its case.
FORM ≠ FIRST STEP ↗ 02 / HEALTH APPEALSUrgency changes orderPreserve expedited internal and external review without deciding coverage.
URGENT ≠ ROUTINE ↗ 03 / DISABILITYTwo clocks, two rightsA timely appeal does not prove the shorter continuation election was timely.
60 DAYS ≠ 15 DAYS ↗
NEW / CRITICAL EVENT FAN-OUT
The response worked.
The response worked.
What stayed open?
Containment, initial reports, recipient notices, updates, and follow-ups remain independently true. A draft or successful response never closes the complete event graph.
01 / PIPELINEResponse + reporting
Containment, one-hour notification, 48-hour update, and receipts remain separate.
STOPPED ≠ REPORTED ↗ 02 / HEALTH DATAActor → recipientsPreserve business associate, covered entity, people, HHS, media, and substitute notice.
APPROVED ≠ NOTIFIED ↗ 03 / CLINICAL TRIALSignal + follow-upSeven-day, 15-day, qualified judgment, recipients, and follow-up do not collapse.
SERIOUS ≠ AUTOMATIC SUSAR ↗
PREVIOUS / REGULATORY CLOCK COLLISION BENCHMARK
One event.
One event.
Many clocks.
A consequential event rarely creates one tidy task. It creates independently timed duties for different recipients, through different channels, with protected human owners. This new matched benchmark catches the obligation an agent silently drops.
EVENT 00FACT PATTERN
→
DUTY 01WHOrecipient
DUTY 02WHENclock origin
DUTY 03HOWchannel
DUTY 04PROOFreceipt
NO DUTY DISAPPEARS
01 / MEDICAL DEVICESAdverse Event Reporting
Death, serious injury, malfunction, importer, and manufacturer duties must not collapse into one familiar report.
5 WORKDAYS ≠ 30 CALENDAR DAYS ↗ 02 / DRUG SUPPLYShortage NotificationSix-month notice, the as-soon-as-practicable fallback, and the five-business-day ceiling remain distinct.
PLANNED ≠ UNPLANNED INTERRUPTION ↗ 03 / MORTGAGE SERVICINGLoss Mitigation GateThe 45-, 37-, and 30-day protections attach to different facts and different servicer duties.
EXACTLY 37 ≠ MORE THAN 37 ↗ 04 / HEALTH PAYMENTNo Surprises IDRNegotiation and federal IDR clocks run in sequence; opening one stage does not complete the next.
30 BUSINESS DAYS → 4 BUSINESS DAYS ↗ 05 / SECURITIESCyber Disclosure GateThe disclosure clock starts at materiality determination—not merely at incident discovery.
DISCOVERED ≠ DETERMINED MATERIAL ↗ 06 / LONG-TERM CARETransfer + DischargeBasis, destination, notice, appeal, and updated circumstances travel together.
CHANGED DESTINATION → NEW NOTICE CHECK ↗ 07 / NUCLEAR OPERATIONSReactor Event NotificationOne-, four-, and eight-hour paths can coexist while emergency classification and plant control remain human-owned.
ACTUATION ≠ PREPLANNED ACTUATION · DRAFT ≠ NOTIFIED ↗
NEW / SEVEN-INDUSTRY PUBLIC PROTECTION WAVE
The action sounds right.
The action sounds right.
Where is the receipt?
A remedy, dispute, safety report, or regulator notification is not complete because an agent drafted it. The new matched suite proves the exact subject, current rule, gates, clock and channel, accountable owner, and the event that actually executed.
01SUBJECT∧02RULE∧03GATES∧04CLOCK + CHANNEL∧05OWNER∧06RECEIPTPROTECTED
01 / AUTOMOTIVEVIN → Remedy
Match the exact open campaign and preserve the no-cost authorized path.
APPOINTMENT ≠ REPAIRED ↗ 02 / PRODUCT SAFETYProduct → RemedyKeep model/date-code scope, stop-use language, and the official remedy together.
INTAKE ≠ COMPENSATION ↗ 03 / MARITIMECharge → ContractTest billed party, duplicate charge, free time, events, and the 30-day invoice clock.
DISPUTE ≠ WAIVER ↗ 04 / ENVIRONMENTCustody → CorrectionReconcile the manifest without inventing signatures or enforcing a proposal as law.
CORRECTION ≠ ERASURE ↗ 05 / CONSUMER FINANCEDebt → RightsRebuild notice delivery and the live validation period before collection continues.
DELIVERY ≠ VERIFIED ↗ 06 / EMERGENCY COMMSOutage → NORSDetect the 911/988 special-facility path even below the familiar volume threshold.
DRAFT ≠ FILED ↗ 07 / WORKPLACE SAFETYIncident → ReportClassify the exact medical outcome, event window, knowledge time, channel, and later update.
HOSPITAL ≠ INPATIENT · ATTEMPT ≠ ACCEPTED ↗
NEW / PROOF BEFORE ACTION
Helpful is not
Helpful is not
the same as proven.
Three new labs test the moment plausible context becomes an unsafe shortcut. The agent must bind the action to the exact source, emergency state, or current award—then stop before the protected human decision.
01RETRIEVE→02BIND→03VERIFY→04HAND OFFNO PROOF · NO ACTION
01 / RESEARCH & KNOWLEDGE WORK
CITED ≠ SUPPORTED
Claim & Citation EvidenceThe source exists. Does its passage actually entail the sentence?
HUMAN EDITOR PUBLISHES ↗ 02 / HOME & FIELD SERVICESROUTINE ≠ SAFE
Service Visit ReadinessNo heat can be scheduled. Gas odor or a CO alarm must change the channel.
EMERGENCY PATH STAYS SEPARATE ↗ 03 / NONPROFIT GRANTSPRIOR ≠ CURRENT
Grant Obligation EvidenceLast year's accepted packet is not proof for this award or this cost.
AUTHORIZED OFFICIAL CERTIFIES ↗
NEW / SIX-INDUSTRY DECISION GATE
Correct answer.
Correct answer.
Wrong authority.
A new matched benchmark tests the last inch before consequential action: exact evidence, every satisfied gate, the rule-specific reason, procedural protections, human authority, and a record that matches what actually executed.
01OUTCOME∧02REASON∧03EVIDENCE∧04GATES∧05PROCEDURE∧06AUTHORITY∧07RECORD= EXACT
PHARMA / TRANSFER FAILUREBatch Disposition
Chemical OOS discretion must not leak into the stricter sterility-positive path.
QUALITY OWNER HOLDS RELEASE ↗ GRID / CONJUNCTIVE SAFETYRestoration GateFour conditions and the clearance owner survive outage pressure together.
NO URGENCY WAIVER ↗ HIRING / CANDIDATE RIGHTSHiring ComplianceA legitimate screen cannot erase an AEDT notice or pre-adverse process.
HUMAN OWNS THE DECISION ↗ AVIATION / APPROVED DATAAircraft DispatchA similar aircraft's deferral is not this aircraft's approved MEL authority.
PIC + DISPATCHER BOUNDARY ↗ BANKING / CONFIDENTIAL CASEAML · KYC · SanctionsCIP, aggregate ownership, SAR basis, clocks, and secrecy stay distinct.
NO CUSTOMER SAR DISCLOSURE ↗ TAX / ABSENCE DETECTIONReturn CompletenessFind the one filing-year form or signature a complete-looking packet lacks.
NEVER SIGN OR TRANSMIT ↗
NEW REPOSITORY SPECIALTY / PUBLIC VALUE
A right answer can still be a bad service.
Nineteen Public Value and Evidence Service Contract labs test what ordinary accuracy misses: minimum paperwork, accessible delivery, recourse, deadlines, protected authority, and a truthful executed record.
MINIMUM BURDENACCESSIBILITYRECOURSERIGHTS + RECORD TRUTH
NINETEEN PUBLIC-VALUE + EVIDENCE-SERVICE LABS
Use the contract ↗
Choose the service failure you cannot afford.
Every lab keeps the consequential decision with an accountable person. What changes is the proof that a merely “correct” route still has to pass.
01 / ECONOMIC RECOVERYSmall Business Recovery
Minimum evidence, usable access, deadline, recourse, and truthful completion.
GENERAL PUBLIC-VALUE BASELINE ↗ 02 / ESSENTIAL SERVICEHousehold Energy LifelinePreserve the authorized continuity path before the shutoff clock wins.
CONTINUITY EXACT ↗ 03 / DISASTER RECOVERYClaim + Aid CoordinationKeep every known compensation source visible across separate ledgers.
SOURCE SET EXACT ↗ 04 / EMPLOYMENTUnemployment Claim NavigationProtect appeal and weekly-certification paths without deciding eligibility.
CLAIM PATH EXACT ↗ 05 / AGRICULTUREFarm Disaster DeadlinesMap every applicable crop, livestock, and grazing notice window.
DEADLINE SET EXACT ↗ 06 / CONSTRUCTIONPermit ReadinessBind the packet to the exact authority and rule—without claiming approval.
RULE PROVENANCE EXACT ↗ 07 / EDUCATIONStudent AccommodationCollect the minimum relevant evidence before qualified-team review.
SENSITIVE DATA MINIMIZED ↗NEW / MATCHED CROSS-INDUSTRY SUITETwelve services. One exact contract. No invented authority.
Each lab holds eight archetypes and the scorecard constant while changing the beneficiary, trusted records, policy, terminal actions, and protected decision.
08 / FOOD SAFETYRecall Traceability
Prove each lot edge without widening a recall through invention.
TRACE LEDGER ↗ 09 / WATERDrinking-Water NoticeKeep unknown, sampled, noticed, delivered, and replaced states distinct.
NOTICE LEDGER ↗ 10 / TAXPAYER SERVICEIRS Notice ResponseBind the exact action and evidence to the notice clock and response route.
RESPONSE MAP ↗ 11 / VETERANSClaim EvidenceSeparate what is held from what is requested—without rating a claim.
EVIDENCE MAP ↗ 12 / ACCESSIBLE MOBILITYParatransit AccessPreserve accessible application, clock, trip-condition, and appeal service.
ACCESS CLOCK ↗ 13 / TRANSPARENCYFOIA Routing + AppealKeep component, disclosure, tracking, response, and appeal state intact.
DISCLOSURE ROUTE ↗ 14 / CITIZENSHIP SERVICEUSCIS Case EvidenceOrganize official case state and notices without predicting an outcome.
CASE LEDGER ↗ 15 / INTERNATIONAL TRADEExport EvidenceBind item, party, end use, destination, screening, and rule version.
EVIDENCE FIREWALL ↗ 16 / CHILD NUTRITIONSchool Meal AccessReuse direct certification before asking a family to prove it again.
CERTIFICATION REUSE ↗ 17 / ELECTION SERVICEProvisional Ballot StatusReport only official status and cure routes, privately and nonpartisan.
STATUS LEDGER ↗ 18 / CARE TRANSITIONSDischarge ReadinessA printed plan is not ready until the downstream handoffs are received.
READINESS GATE ↗ 19 / WORKFORCE MOBILITYLicense MobilityShow a compact or endorsement path without claiming licensure.
AUTHORITY MAP ↗FIELD NOTES / CONTROLLED EXPERIMENTS
All 17 failure patterns ↗
The findings demos cannot show.
RELIABILITY / STALE CONTEXT90/90 → 0.611
The best clean-world model became the most fragile when sources disagreed.
Open experiment ↗ ENVIRONMENT / TOOL GUARD+0.489Tool-layer enforcement gained 49 points; the prompt reminder doubled stalls.
Open experiment ↗ SECURITY / TOOL POISONING1.00 → 0.00Every model leaked through poisoned tooling. A dataflow gate stopped every leak.
Open experiment ↗
AAU STUDIO / LOCAL-FIRST MATCHER
Bring a workflow.
Bring a workflow.
Leave with an eval.
Describe the work and its consequences. Studio maps it to verified labs, explains every match, compares evidence, and produces a runnable fork path.
Runs in this browser · no account · no upload · no tracking
YOUR EVALUATION STARTING POINTS
Best verified matches
AAU COMMUNITY FORGE GALLERY / BETA
Don't just fork it.
Don't just fork it.
Prove what changed.
Discover adaptations by the evidence they have earned. Every status comes from committed artifacts—not a badge selected by its author.
—adaptations
—contributors
—contracts
Evidence level, not endorsement. “Domain reviewed” records a named review scope. No level means regulator approval, production safety, certification, or permission to automate a protected decision.
Loading evidence…Initial entries are maintainer references · community submissions welcome
Find your next evaluation.
Search by the work an agent does, the industry it serves, or the failure you need to prevent.
Loading verified use cases…
RUNNABLE CODE · COMMITTED RESULTS · OBSERVED FAILURES