The Topic-Level Trap
Most safety alignment work treats harm as a property of a topic. A prompt is unsafe because it mentions weapons, fraud, or self-harm, and guard models like LlamaGuard-3 encode exactly this kind of category-level taxonomy. Benchmarks then probe the obvious failure mode: models refusing safe prompts because they contain a dangerous-looking word.
The problem is that real deployments don't work this way. The same base model might power a general assistant, an educational product, an enterprise system, or a public-sector service—and each needs different boundaries within the same topic, not a blanket refusal of the entire category.
A new paper from Multiverse Computing confronts this head-on. The question isn't whether to refuse an entire topic, but which subset of that topic is incompatible with a given deployment—and how to train and measure a model against that boundary.
The Narrow-Boundary Problem
The paper formalizes what they call narrow-boundary safety: a topic universe (they use political prompts) contains a target-harmful subset the deployment wants to refuse. The ideal policy isn't to refuse all politics—it's to refuse the harmful subset while continuing to answer the benign complement.
Concretely: a civics tutor and a public-sector assistant can share a model yet require opposite behavior on politics. Both should answer factual questions about elections. Only one should refuse requests to write targeted political manipulation.
LlamaGuard-3 can't express that split. Its political coverage is limited to "factually incorrect information about electoral systems," which excludes persuasion and excludes the factual prompts a deployment must keep answering.
The paper operationalizes this boundary as pairs of prompts sharing a topic anchor but differing in intent: one that should be refused, one that should be answered. They test on political persuasion—manipulative persuasion can cause harm while factual political information stays legitimate, which is exactly the case where topic-level refusal is too blunt.
Where Self-Generated Safety Tuning Fails
The standard recipe for building safety training data is self-generation: take the target model, steer it toward refusal on harmful prompts, keep the traces a guard model verifies. Methods like ThinkSafe follow this pattern.
Framing the problem as a boundary rather than a topic exposes three weaknesses:
Coverage Gap
A single steering attempt doesn't always produce an accepted refusal, and those prompts silently drop from the training set. In the paper's audited pool, single-shot generation dropped 19.88% of prompts—8,009 examples. These failed prompts may well be the hardest cases.
Their fix: escalating retry with progressively stronger steering, which brought residual failures down to 0.20% (79 prompts). Coverage repair saved 40,293 harmful training prompts the naive pipeline would have thrown away.
Downside Reactions
Safety tuning produces false refusals on benign prompts with dangerous-looking surface features. To compensate, they built in-distribution benign data: 11,955 verified surface-dangerous benign prompts across 18 semantic types. The model sees safe prompts with threatening wording during training rather than only at eval.
Boundary Blindness
Ordinary harmful-benign splits don't measure the shape of the boundary at all. A model can improve its harmful-refusal rate by expanding refusal into nearby permissible prompts, and a topic-level metric will call that an improvement. Held-out harmful-benign pairs—1,539 per side—let them measure both sides of the boundary directly.
The Trade-Off Nobody Wants to Admit
Training on political refusal data works in the obvious sense. On Qwen3-8B, the escalated-coverage model raised in-distribution political refusal from 9.47% to 84.75%. It also transferred: mean unsafe-response rate across HarmBench, StrongREJECT, and WildJailbreak (scored by LlamaGuard-3) fell from 26.26% to 0.14% in the strongest configuration.
Reported alone, those numbers look like a clean win. They're not.
At the same checkpoint, over-refusal on XSTest rose from 2.00% to 74.00%. The configuration with the lowest harmful-response rate refused nearly three-quarters of plainly safe prompts. It's a blunt refusal machine, not a safer model.
You cannot see that unless you measure the benign side.
This is the paper's central message: data composition decides where a checkpoint sits in the space of safety against over-refusal, so the two axes have to be reported together. Publishing only harmful-refusal rate is malpractice.
What Actually Fixes It
Two data components pull over-refusal back down without sacrificing safety gains:
Self-generated compliance responses. Replacing externally adopted compliance with verified responses from the target model itself lowered XSTest over-refusal from 15.20% to 5.20% under single-shot generation, at a modest harmfulness cost.
Harmful-benign boundary pairs. These do the most precise work. Adding benign boundary data reduced over-refusal on the comply-worthy side of held-out pairs from 32.94% to 4.16%. Refusal on the harmful side dropped only from 91.88% to 87.72%.
In other words: most false refusals near the boundary disappear while almost all genuine refusals survive. There's a real recall cost—it's small and measurable, which is the point. You can only trade it off deliberately if you're measuring both sides.
What This Changes for Real Deployments
The practical takeaway is that safety tuning should not be assessed by harmful-refusal rate alone. A model that refuses more is not automatically safer. On a narrow boundary, the same move that raises refusal on harmful prompts can quietly make the model useless on legitimate prompts right next to them.
Composition of the training data—coverage repair, in-distribution compensation, boundary pairs—is what controls that trade-off. Both sides of the intended boundary have to be evaluated for the numbers to mean anything.
This matters because real organizations don't deploy topic-level policies. A healthcare assistant needs to refuse medical advice that could cause harm while answering factual questions about symptoms. A financial services chatbot needs to refuse market manipulation while explaining how markets work. A customer-service bot needs to refuse abusive requests while handling difficult conversations.
Topic-level guards can't express any of those splits, and the standard self-generation pipeline quietly trains models that refuse far more than they should.
The Broader Picture
This work is part of a broader shift in safety research: moving from coarse taxonomies to deployment-specific boundaries. The benchmarks we've been using—XSTest, OR-Bench, topic-level guard coverage—were designed to catch the most obvious failures. They're necessary but not sufficient.
The paper's generation pipeline extends to other topics beyond politics, and the full ablations cover loss routing that separates harmful cross-entropy from benign forward-KL preservation. The methodology is general: operationalize your deployment boundary as pairs, measure both sides, and build training data that respects the boundary rather than the topic.
It's a critique, but it's also a blueprint. Safety alignment that doesn't measure over-refusal is incomplete. Safety alignment that treats boundaries as topics is too blunt for production. The tools to fix both problems are here—coverage repair, boundary pairs, two-sided evaluation—and they work.
The question now is whether the rest of the field will start reporting both numbers.