🛡️ signalbench — In-Band Signal Compliance (IBSC)

One metric for prompt injection and temporal blindness.

An LLM agent reads control instructions and ordinary data through the same channel, so on every turn it must decide, from content alone, whether an in-band instruction is legitimate. It fails two symmetric ways: over-compliance (obeying an illegitimate signal — prompt injection) and under-compliance (ignoring a legitimate one — an injected date, an access-deny, a do-not-share memory label, a no-automation policy).

Signal-Response Correctness (SRC) scores both on one axis: the balanced mean of legitimate-signal uptake (LSU) and illegitimate-signal resistance (ISR). A trivial always-comply or always-refuse policy scores 0.5 — you only win by adjudicating. Across 21 systems, none exceeds ~0.85; capability is non-monotonic (a small model beats much larger ones); and the five families are all positively correlated (mean Pearson r ≈ 0.73) — evidence of one underlying ability, not five separate skills.

#	System	SRC↓	LSU	ISR	time	access-deny	memory-label	injection	bot_policy
1	gemini:gemini-2.5-pro	0.850	0.967	0.733	1.000	0.750	0.583	0.917	1.000
2	openai:gpt-4o-mini	0.833	0.867	0.800	1.000	0.833	0.750	0.833	0.750
3	openai:gpt-5.5	0.817	0.767	0.867	1.000	0.750	0.833	0.833	0.667
4	openai:gpt-5.1	0.800	0.800	0.800	1.000	0.833	0.667	0.917	0.583
5	claude-code:claude-opus-4-8	0.767	0.767	0.767	0.750	0.750	0.917	0.750	0.667
6	claude-code:claude-haiku-4-5-20251001	0.683	0.500	0.867	1.000	0.750	0.667	0.583	0.417
7	openai:gpt-4.1	0.683	0.800	0.567	0.917	0.500	0.583	0.833	0.583
8	openrouter:qwen/qwen-2.5-72b-instruct	0.667	0.833	0.500	0.833	0.500	0.667	0.667	0.667
9	claude-code:claude-sonnet-5	0.650	0.500	0.800	0.667	0.917	0.750	0.417	0.500
10	gemini:gemini-2.5-flash	0.650	0.833	0.467	0.917	0.500	0.583	0.667	0.583
11	openrouter:meta-llama/llama-3.1-8b-instruct	0.633	0.500	0.767	0.833	0.500	0.667	0.750	0.417
12	openrouter:qwen/qwen3-next-80b-a3b-instruct	0.583	0.900	0.267	0.833	0.500	0.417	0.583	0.583
13	openrouter:deepseek/deepseek-chat	0.567	0.733	0.400	0.833	0.583	0.583	0.583	0.250
14	openrouter:meta-llama/llama-3.3-70b-instruct	0.417	0.533	0.300	0.750	0.083	0.583	0.583	0.083
15	openrouter:microsoft/phi-4	0.417	0.400	0.433	0.833	0.000	0.667	0.583	0.000
16	openrouter:mistralai/mistral-small-24b-instruct-2501	0.417	0.433	0.400	0.667	0.000	0.833	0.583	0.000
17	openrouter:google/gemma-2-27b-it	0.400	0.433	0.367	0.667	0.000	0.750	0.500	0.083
18	openrouter:qwen/qwen-2.5-7b-instruct	0.383	0.500	0.267	0.417	0.167	0.500	0.500	0.333
19	openrouter:meta-llama/llama-3.2-3b-instruct	0.367	0.300	0.433	0.500	0.000	0.667	0.667	0.000
20	openrouter:mistralai/mistral-nemo	0.317	0.533	0.100	0.583	0.083	0.333	0.500	0.083
21	openrouter:nousresearch/hermes-3-llama-3.1-70b	0.300	0.467	0.133	0.667	0.000	0.500	0.333	0.000

SRC = overall Signal-Response Correctness · LSU = legitimate-signal uptake · ISR = illegitimate-signal resistance · then per-family balanced SRC (cell shaded red→green). All scores: seed 0, 75 items, deterministic action-based grading (no LLM judge).

Links: code / harness · dataset + raw responses · DOI 10.5281/zenodo.21223956