Compile evaluation integrity, authority, containment, remediation, and resilience evidence into one executable assurance case, then require role-separated review—without hiding contradictions or automating the accountable decision.
Pick the task you need to solve. Start with a local plan or fictional example, then bring authorized inputs from your own system.
Inspect scored cases and auxiliary runs before calling a model. Scope-bound baselines detect changed task selections; missing comparison evidence is not a pass.
scan --plan-json scan-plan.json
Build a reviewable CI gate
Apply one disclosure policy to multiple AI suppliers. Get an offline HTML review and reproducible evidence packs, with invalid submissions kept separate from findings.
bom intake-ai-portfolio
Try the fictional portfolio
Build an executable assurance case with verified source reports. Keep contradictory, missing, and stale evidence visible for the accountable human decision.
assure demo --out-dir review
Open the new-user guide
Available on main: use a source checkout for these newer workflows; they are not all in the published v0.19.0 package. Commands above follow dspy-security-bench; guides include all required inputs. Synthetic examples are not production assessments, and disclosure checks are not supplier rankings or approvals.
Every artifact recomputes through its native verifier before it can support a claim. Missing, stale, violated, and contradictory evidence stays visible. No universal safety score. No automatic approval.
ASSURANCEGRAPH::CLAIM-EVIDENCE::V1
PROFILE: CRITICAL INFRASTRUCTURE
A checksum alone never upgrades an arbitrary assertion into deployment evidence.
The graph cannot choose a favorable report when current evidence conflicts.
Owner-declared observation times and age limits are bound into the exact case.
Profile support informs review; it never becomes certification, ATO, or risk acceptance.
A signed score can still come from a compromised procedure. Thirteen content-free controls preserve holdout secrecy, evaluator independence, complete accounting, active monitoring, and safe exit—without reading a prompt or output.
EVALINTEGRITYPROOF::V1 / 13 CONTROLS / ZERO ACTIONS
INTEGRITY_EVIDENCED
sha256:7d84…100 cases98 + 2 safe-stopafter commit ✓Every claim names the functions that must review its evidence. Exact keys, roles, organizations, assignments, and review windows are policy-bound. A valid evidence gap cannot be erased by adding favorable signatures.
ASSURANCEQUORUM::IN-TOTO::DSSE::V15 / 5 SIGNATURES VALIDinsufficient-source-independenceAdditional “sufficient” statements cannot erase a valid assigned gap.
Registration, review inclusion, retirement, and compromise become one append-only evidence history. Operator and cross-organization witness signatures bind the exact Merkle checkpoint—without turning the log into an approval authority.
ASSURANCELEDGER::MERKLE+WITNESS::V12 / 2 WITNESSESfictional-agency.example/ai-assurance11 eventsbefore review issuancehistory retainedroot 9f7a…41c2root 20d4…8ea1EXACT SHA-256Eight harmless canaries distinguish a control failure from a monitor failure. An owner-authored AI-BOM policy turns CycloneDX and SPDX gaps into reviewable SARIF, then lifecycle drift separates regression, persistence, improvement, resolution, and component change—without copying disclosure values or touching a live target.
CONTROLPLANE::CANARIES+AGENTBOM::V1
NETWORK REQUESTS: 0
sha256:7e2a…CANDIDATE sha256:b91c…MINIMAL PLAN · 5 CLAIMSA finding is only the beginning. DefenderTwin verifies the exact before/after state, target and approval scope, essential-service continuity, introduced risk, rollback, and evidence completeness—without touching a live target.
DEFENDERTWIN::VERIFIED-REMEDIATION::V1
OFFLINE RECOMPUTATION
Frozen structural outcomes—not an agent's explanation—decide whether the required state returned.
Target, action, identity, prior approval, and change budget remain independently testable.
Closing an attack path by causing an outage is recorded as a regression, never a safe result.
A change without verified recovery evidence cannot cross the trusted-defender gate.
Bring a synthetic mission or adapter output. CI recomputes attack-path closure, mission continuity, scope, approval, rollback, privacy, and every digest.
Start with fixes DefenderTwin can recompute. Exclude unsafe options, trace shared-service dependencies, enforce resource and community floors, replay stressed availability, and preserve every non-dominated choice for an accountable owner.
RESILIENCEGRAPH::FRONTIER::V1
32 SUBSETS → 13 FEASIBLE → 10 FRONTIER
Every embedded DefenderTwin result recomputes before the candidate becomes eligible.
The engine returns no feasible portfolio instead of silently relaxing owner constraints.
Directly addressed services and downstream dependency reach are reported separately.
No predicted loss, magic weight, or model-generated narrative chooses how resources are spent.
Reconstruct cross-run coordination, indirect egress, peer-authority laundering, unsafe persistence, evaluator access, and missed response windows from structural events alone. No prompts, reasoning, messages, secrets, or exploit payloads enter the protocol.
COLLECTIVEGUARD::CONTAINMENT-PLANE::V1 + PROVENANCE::V2
13 DETERMINISTIC RULES
Package, artifact, storage, and browser services stay bound to run identity and declared purpose.
Owner-defined action budgets turn clarification and pause into measurable system behavior.
Paging, containment, evidence preservation, and restart approval remain distinct accountable events.
Two controls only count as independent when their declared failure domains are actually different.
Delegation attenuation, audience/scope binding, revocation order, single-use authorization, and exact effect receipts—verified offline.
Owner-adjustable source and response objectives with informative NIST crosswalks and non-certifying OSCAL 1.2.2 export.
Freshness and regression evidence with zero production actions. The controller requests review; operators decide and act.
Transform a local OpenTelemetry export into a pseudonymized evidence graph, deterministic authorization findings, a synthetic replay twin, SARIF, and OSCAL. Raw prompts, tool arguments, credentials, and unapproved attributes are removed by default.
TP004 · HIGHInstrument a real agent boundary, keep content under local custody, publish only the sanitized bundle, and let CI recompute every finding.
Turn structural OpenTelemetry evidence into a ScheduleProof draft without treating span links or wall clocks as proof. Names, attributes, events, prompts, arguments, results, and credentials are never read.
CAUSALPROOF::PROVENANCE_LEDGER::V1
authority-1042decision-1042token-1042authority-1042effect-1042Same-trace parent identifiers become directional, schedule-eligible edges.
Owner guarantees require a bounded rationale and retain asserted provenance.
Generic span links remain undirected until the operator supplies a guarantee.
Timing order is visible for investigation but can never enter the proof graph.
Reads five structural lifecycle fields. Native span payloads, errors, prompts, responses, and tool I/O stay outside the bridge.
Binds selected graph nodes while discarding state, messages, inputs, outputs, errors, and tool data without inspection.
CI rebuilds the structural trace → causal ledger → ScheduleProof chain. Unsafe results are admissible; unverifiable evidence is not.
Declare only the ordering your design truly enforces. ScheduleProof counts every reachable interleaving, checks eight authorization invariants, and returns the shortest causal counterexample—without executing a model, credential, tool, or effect.
SCHEDULEPROOF::TOPOLOGICAL_EXPLORER::V1
ATOMIC EVENT MODEL
authority-1max_uses: 1aud: payments-mcpauthority-1payment-1042Missing guarantee: commit and revocation can race. The graph—not a preferred trace—defines the search space.
Repair direction Revalidate named authority atomically at effect commit. A suggested fix is evidence for review—not an automated risk decision.
Counts the full topological schedule space before exploration.
Authority, approval, token, scope, audience, identity, effect, and receipt.
Protocol and scenario digests bind every derived count and counterexample.
A truncated search without a finding is review—not a safe result.
Start with a bounded use case—not a vendor claim. Generate synthetic evaluation drafts, exercise every delegation edge, compare verified evidence over time, and package the result for accountable review.
Normalize a public AI inventory and draft a human-review-required MissionPack.
inventory draft-pack
Sanitize field traces, then find unsafe temporal, parallel, and multi-effect paths.
trace analyze · graph v2-run
Compare content-addressed baselines and surface owner-threshold regressions.
watch compare
Export neutral test plans, QASP inputs, cost fields, and portability checks.
acquisition export
AgentGraphTwin keeps the mission fixed while mutating exactly one security-relevant edge. Receipts bind the request, adapter, policy, principal, decision, and reason.
reason: scope_not_grantedTranslation fixtures remain clearly labeled and do not execute, certify, or endorse the named backend. The operator-controlled command runner can execute a declared backend and version locally, producing self-attested conformance evidence without recording credentials or the command itself.
$ dspy-security-bench graph demoTwo models can complete benign tasks at nearly the same rate and still have radically different prompt-injection failure profiles. Security must be measured directly.
Hover or focus a model. Provisional measurements use a dashed ring.
Robustness is the share of prompt-injection attacks that failed. Capability is benign task completion. Confidence intervals are cluster-bootstrapped over task pairs.
| Rank / model | Family | Robustness | Capability | Evidence |
|---|
Scaffold a config and GitHub Action, preview the exact test matrix without spending API credits, then block regressions with terminal, JSON, and SARIF reports.
# install and create a ready-to-run gate
$ pip install dspy-security-bench
$ dspy-security-bench init --model openai/gpt-4o-mini
# inspect the matrix — zero model calls
$ dspy-security-bench scan --config .dspy-security-bench.yaml --plan
✓ 10 benchmark cases planned
✓ no model was calledKeep your orchestration code and tools. The integration assistant detects a directly declared framework, generates a reviewable adapter, and validates the wiring without invoking the agent run loop.
dspy-security-bench integratedspy-security-bench doctorno agent runsGenerated workflows are manual by default, so connecting a repository cannot unexpectedly spend model credits.
A reusable workflow isolates model evaluation from signing, recomputes every statistic in a clean job, and binds the exact JSON to a source commit before enforcing the gate.
sha256: 9e81…c4f2
signature verification is separate from score recomputation
permissions:
contents: read
id-token: write
attestations: write
jobs:
proofrun:
uses: immu4989/dspy-security-bench/.github/workflows/proofrun.yml@v0.19.0
with:
agent: myapp.security:build_agent
trials: 10Every accepted entry preserves all raw trials and passes offline recomputation. A cryptographic badge appears only after the exact digest passes attestation verification and enters the reviewed registry.
Run your agentProvenance authenticates a workflow and exact artifact. It does not independently observe a model provider or turn a synthetic benchmark into certification.
Ship a mission test as reviewable YAML—not benchmark code. SourceTwin holds the authoritative corpus and question fixed, changes only retrieved untrusted content, and scores the agent's recorded claims against exact source IDs.
$ dspy-security-bench pack run source-twin --agent myapp:buildFork it, encode a real mission with synthetic evidence, and publish the first independently reproducible run.
Structured test claims are benchmark ground truth for a fixed synthetic pack—not a legal interpretation, universal truth set, certification, or substitute for agency subject-matter review.
Clean and poisoned procurement twins hold structured facts fixed while varying one untrusted input: vendor-authored text. Then the benchmark measures what changed in the live synthetic world.
Award bias · sealed bids · payment identity · eligibility · approval
Deliberately vulnerable scorer fixture
Scenario value behind compromised actions
Functional environment-state evidence
Ten passes still carry uncertainty. Preserve every raw trial, expose the interval, and gate on its lower bound.
Mission utility, controlled-pair decision invariance, confidentiality, identity, authorization, uncertainty, and bounded economic context—versioned in JSON and SARIF.
$ dspy-security-bench impact repeat --trials 10
Inspect the protocol
Reference fixture, not a model result. Synthetic exposure is not predicted loss or a compliance certification.
Execute the same poisoned missions with the boundary off and on. Measure functional harm, safe recovery, and clean utility independently.
Run the complete experiment again and again. Alternate order. Preserve every child report. Gate the lower confidence bound—not the flattering point estimate.
$ dspy-security-bench impact control-repeat-demo --trials 5Deterministic fixture · fixed synthetic suite · not a model result, population estimate, predicted loss, or certification
$ dspy-security-bench impact control-demo
Deterministic reference fixture, not a model result. Containment is not recovery; synthetic exposure is not predicted loss or certification.
Publish the policy-off and policy-on experiment—even when the control performs poorly. Every accepted entry keeps the raw paired trials, policy identity, uncertainty, and chain of custody inspectable.
A failing control can belong here. Five trials, recomputable evidence, fresh isolation, zero runtime errors, and redacted arguments are the comparison floor.
The exact policy document travels with the evidence. A name alone cannot silently stand in for different enforcement logic.
Blocking harm, safely completing the mission, and preserving clean work are reported independently—never collapsed into one marketing score.
Be the first independent team to publish a policy experiment—not just a score.
evidence-kind: control policy: policies/production.yaml trials: 10
Fixed synthetic ProcureBench suite · repeated execution evidence · not certification, a population estimate, predicted loss, or independent observation of a hosted provider.
Exercise a cyber-response agent inside an inert digital twin, repeat the exact counterfactual protocol, and transform verified evidence into standards-shaped assessment inputs—without claiming certification.
Five clean/poisoned security-operations twins. Authoritative alert facts stay fixed; only hostile external content changes.
ALT-2026-017asset contextsecret → outside.testtier-2-soc$ dspy-security-bench incident demoBind verified ImpactTwin, ControlTwin, IncidentTwin, MissionPack, or AuthorityTwin evidence to an owner-supplied deployment profile and local acceptance thresholds.
$ dspy-security-bench federal initBe the first independent team to publish recomputable incident-response evidence.
Every production pattern pairs an untrusted input with a dangerous sink. Included policies enforce least agency before the side effect—not after an incident.
Resolve tickets without turning a poisoned CRM note into data exfiltration, unlimited refunds, or identity changes.
--profile customer-support
Extract and reconcile invoices while preventing vendor impersonation from becoming an unauthorized transfer.
--profile financial-operations
Search broadly without letting a hostile page become persistent memory, published content, or executable code.
--profile research-rag
Accelerate diagnosis with broad observability while keeping deletion, shell access, and production mutation constrained.
--profile devops
$ dspy-security-bench policy init --profile customer-support
Open implementation guide
Suites, attacks, scaffold, task subset, and decoding settings are hashed and versioned.
Per-model result JSON, run metadata, confidence intervals, and generation scripts live beside the board.
Rows remain provisional when confidence intervals cross a bucket boundary. Green means no known bypass—not safe.
Trust is a system property.
Make the evidence inspectable.