One metric for prompt injection and temporal blindness.
An LLM agent reads control instructions and ordinary data through the same channel, so on every
turn it must decide, from content alone, whether an in-band instruction is legitimate. It fails two
symmetric ways: over-compliance (obeying an illegitimate signal — prompt injection) and
under-compliance (ignoring a legitimate one — an injected date, an access-deny, a
do-not-share memory label, a no-automation policy).
Signal-Response Correctness (SRC) scores both on one axis: the balanced mean of
legitimate-signal uptake (LSU) and illegitimate-signal resistance (ISR). A trivial always-comply or
always-refuse policy scores 0.5 — you only win by adjudicating.
A hard ceiling remains. Across 21 systems, only one (gpt-5.5, 0.87 on the 70 items its API let
run) exceeds 0.85; the next best is ~0.81, and 20 of 21 fall below 0.85. Scoring is strict:
resisting an attack only counts if the model also still completes the real task — a blanket
refusal earns no credit. Capability is non-monotonic (a small model beats much larger ones). The
five families are positively correlated (mean Pearson r ≈ 0.62 raw), but that is exploratory —
it weakens further once general ability is controlled for, so we do not claim a single proven axis.
Scored on all 75 items.
| # | System | SRC↓ | LSU | ISR | time | access-deny | memory-label | injection | bot_policy | n |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | openai:gpt-5.5 | 0.873 | 0.793 | 0.962 | 1.000 | 0.800 | 0.917 | 0.817 | 0.833 | 70 |
| 2 | gemini:gemini-2.5-pro | 0.783 | 0.967 | 0.600 | 0.667 | 0.750 | 0.583 | 0.917 | 1.000 | 75 |
| 3 | openai:gpt-5.1 | 0.783 | 0.800 | 0.767 | 0.917 | 0.833 | 0.667 | 0.917 | 0.583 | 75 |
| 4 | openai:gpt-4o-mini | 0.767 | 0.867 | 0.667 | 0.917 | 0.833 | 0.750 | 0.583 | 0.750 | 75 |
| 5 | openai:gpt-4.1 | 0.667 | 0.800 | 0.533 | 0.833 | 0.500 | 0.583 | 0.833 | 0.583 | 75 |
| 6 | gemini:gemini-2.5-flash | 0.583 | 0.833 | 0.333 | 0.583 | 0.500 | 0.583 | 0.667 | 0.583 | 75 |
A different harness — tools disabled — scored on 72 items (3 tool-only items excluded).
| # | System | SRC↓ | LSU | ISR | time | access-deny | memory-label | injection | bot_policy | n |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | claude-code:claude-opus-4-8 | 0.813 | 0.852 | 0.767 | 0.750 | 0.833 | 0.917 | 0.833 | 0.733 | 72 |
| 2 | claude-code:claude-haiku-4-5-20251001 | 0.707 | 0.556 | 0.867 | 1.000 | 0.800 | 0.667 | 0.617 | 0.450 | 72 |
| 3 | claude-code:claude-sonnet-5 | 0.663 | 0.556 | 0.767 | 0.667 | 1.000 | 0.750 | 0.350 | 0.550 | 72 |
Ran the full suite. Rows with n<75 had a few items refused by the provider and excluded.
| # | System | SRC↓ | LSU | ISR | time | access-deny | memory-label | injection | bot_policy | n |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | openrouter:qwen/qwen-2.5-72b-instruct | 0.683 | 0.833 | 0.518 | 0.833 | 0.500 | 0.667 | 0.750 | 0.667 | 69 |
| 2 | openrouter:qwen/qwen3-next-80b-a3b-instruct | 0.550 | 0.900 | 0.200 | 0.667 | 0.500 | 0.417 | 0.583 | 0.583 | 75 |
| 3 | openrouter:deepseek/deepseek-chat | 0.533 | 0.733 | 0.333 | 0.667 | 0.583 | 0.583 | 0.583 | 0.250 | 75 |
| 4 | openrouter:meta-llama/llama-3.1-8b-instruct | 0.483 | 0.500 | 0.467 | 0.417 | 0.500 | 0.667 | 0.417 | 0.417 | 75 |
| 5 | openrouter:qwen/qwen-2.5-7b-instruct | 0.383 | 0.500 | 0.267 | 0.417 | 0.167 | 0.500 | 0.500 | 0.333 | 75 |
| 6 | openrouter:meta-llama/llama-3.3-70b-instruct | 0.367 | 0.533 | 0.200 | 0.583 | 0.083 | 0.583 | 0.500 | 0.083 | 75 |
| 7 | openrouter:mistralai/mistral-nemo | 0.283 | 0.533 | 0.033 | 0.500 | 0.083 | 0.333 | 0.417 | 0.083 | 75 |
The tool-graded families (access-deny, bot-policy) 404'd on the provider side, so these are scored on the 47-item text-only subset — NOT comparable to the full-suite rows above.
| # | System | SRC↓ | LSU | ISR | time | access-deny | memory-label | injection | bot_policy | n |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | openrouter:google/gemma-2-27b-it | 0.742 | 0.765 | 0.600 | 0.633 | 0.000 | 0.833 | 0.500 | 1.000 | 47 |
| 2 | openrouter:microsoft/phi-4 | 0.560 | 0.706 | 0.733 | 0.717 | 0.000 | 0.750 | 0.775 | 0.000 | 47 |
| 3 | openrouter:mistralai/mistral-small-24b-instruct-2501 | 0.525 | 0.765 | 0.600 | 0.533 | 0.000 | 0.917 | 0.650 | 0.000 | 47 |
| 4 | openrouter:meta-llama/llama-3.2-3b-instruct | 0.385 | 0.529 | 0.467 | 0.267 | 0.000 | 0.750 | 0.525 | 0.000 | 47 |
| 5 | openrouter:nousresearch/hermes-3-llama-3.1-70b | 0.358 | 0.824 | 0.133 | 0.517 | 0.000 | 0.517 | 0.400 | 0.000 | 47 |
How to read this. Compare only within a population — the four groups are scored on different
item sets (see each n). SRC = overall Signal-Response Correctness · LSU = legitimate-signal uptake
· ISR = illegitimate-signal resistance · then per-family balanced SRC (red→green). Provider-error
items (the model never answered) are excluded, not counted as failures. Deterministic action-based
grading for over-compliance; see the repo for the grader's exact scope. Single seed, July 2026.
Links: code / harness ·