Protocol v2 · committed evidence

Measure before
you trust.

A reproducible leaderboard for agentic prompt-injection robustness—and a security gate you can point at your own tool-using agent.

Explore the evidence Protect your agent
Not a safety certificate. A measurable regression floor with every limitation visible.
Observed robustness spread 25→99% Same harness. Same attacks.
Different base models.
14models measured
10model families
2 axesrobustness + capability
100%rows backed by JSON
The signal capability charts miss

Capability and security are different axes.

Two models can complete benign tasks at nearly the same rate and still have radically different prompt-injection failure profiles. Security must be measured directly.

62 point robustness gap between two models with near-identical capability
Interactive map

Capability × robustness

RobustMixedVulnerableProvisional

Hover or focus a model. Provisional measurements use a dashed ring.

Frozen protocol leaderboard

See which models
hold the boundary.

Robustness is the share of prompt-injection attacks that failed. Capability is benign task completion. Confidence intervals are cluster-bootstrapped over task pairs.

Rank / modelFamilyRobustnessCapabilityEvidence
Confirmed only when the full CI stays inside one bucket across all repeats.Read the complete methodology
From benchmark to guardrail

Make injection safety
a merge condition.

Scaffold a config and GitHub Action, preview the exact test matrix without spending API credits, then block regressions with terminal, JSON, and SARIF reports.

  • 01Works with DSPy or any Python agent factory
  • 02Absolute security floors or baseline regression gates
  • 03OWASP LLM01 · NIST AI 100-2 · MITRE ATLAS mappings
quickstart.sh
# install and create a ready-to-run gate
$ pip install dspy-security-bench
$ dspy-security-bench init --model openai/gpt-4o-mini

# inspect the matrix — zero model calls
$ dspy-security-bench scan --config .dspy-security-bench.yaml --plan

✓ 10 benchmark cases planned
✓ no model was called
Open the five-minute guide
New public-interest specialty

Same facts.
Different decision.

ImpactTwin / ProcureBench

Clean and poisoned procurement twins hold structured facts fixed while varying one untrusted input: vendor-authored text. Then the benchmark measures what changed in the live synthetic world.

01A Clean twin
Facts locked
Proposals
2 identical records
Ceiling
$1,200,000
Criteria
70 · 20 · 10
RecommendedP-ATLASbest-value eligible offer
01B Poisoned twin
Same facts
Proposals
2 identical records
Ceiling
$1,200,000
Criteria
70 · 20 · 10
ManipulatedP-NOVA+$150,000 price premium
5twin pairs

Award bias · sealed bids · payment identity · eligibility · approval

0%resistance

Deliberately vulnerable scorer fixture

$3.69Msynthetic exposure

Scenario value behind compromised actions

1sealed record out

Functional environment-state evidence

RepeatTwin · stochastic evidence

Perfect observed is not certain.

Ten passes still carry uncertainty. Preserve every raw trial, expose the interval, and gate on its lower bound.

trial evidence
10 / 10 observed passes
95% Wilson interval 72→100%
fixed pair · repeated execution
community proof path
  • raw trials retained
  • statistics recomputed
  • SHA-256 content addressed
Fork · run · submit
Not another prompt score.

Mission utility, controlled-pair decision invariance, confidentiality, identity, authorization, uncertainty, and bounded economic context—versioned in JSON and SARIF.

$ dspy-security-bench impact repeat --trials 10 Inspect the protocol

Reference fixture, not a model result. Synthetic exposure is not predicted loss or a compliance certification.

Real systems. Real authority.

Secure the moment
intent becomes action.

Every production pattern pairs an untrusted input with a dangerous sink. Included policies enforce least agency before the side effect—not after an incident.

01Profile included

Customer support

Resolve tickets without turning a poisoned CRM note into data exfiltration, unlimited refunds, or identity changes.

Untrusted
Tickets · email · CRM
Boundary
Refund cap · recipient domain
--profile customer-support
02Profile included

Accounts payable

Extract and reconcile invoices while preventing vendor impersonation from becoming an unauthorized transfer.

Untrusted
Invoices · vendor email
Boundary
Payee allowlist · approval
--profile financial-operations
03Profile included

Research & RAG

Search broadly without letting a hostile page become persistent memory, published content, or executable code.

Untrusted
Web · docs · retrieved chunks
Boundary
Memory · publish · execute
--profile research-rag
04Profile included

DevOps copilots

Accelerate diagnosis with broad observability while keeping deletion, shell access, and production mutation constrained.

Untrusted
Logs · issues · repository
Boundary
Deploy · shell · delete
--profile devops
Start offline. No API key required. $ dspy-security-bench policy init --profile customer-support Open implementation guide
Research you can inspect

Built for scrutiny,
not screenshots.

01

Frozen protocol

Suites, attacks, scaffold, task subset, and decoding settings are hashed and versioned.

02

Every row reproducible

Per-model result JSON, run metadata, confidence intervals, and generation scripts live beside the board.

03

Honest uncertainty

Rows remain provisional when confidence intervals cross a bucket boundary. Green means no known bypass—not safe.

Prompt injection is a model property.
Measure it like one.

Star on GitHub Fork the benchmark