Watermarks, Refusals, and the Evidence That Isn’t There

I went looking for the study.

The claim circulating right now is that large language models behave differently toward harmful prompts when watermarking is switched on. It is a good claim. It is specific, it is testable, and it touches two safety systems that most teams treat as unrelated. It is also, as far as I can tell, unsupported by anything public. The reference material I can find on LLMs covers the basics: these are systems trained on very large text corpora to produce human-like output, evaluated on alignment, safety, and fairness, probed with red-teaming, and prone to confident fabrication. Nothing in it describes a measured interaction between watermarking and refusal behavior.

So treat what follows as architecture reasoning rather than reporting. The question deserves attention precisely because the mechanism is plausible, and because nobody appears to have checked.

What watermarking actually touches

The important detail is where in the stack watermarking lives. Alignment work — instruction tuning, preference optimization, the policies that produce a refusal — shapes the model’s parameters and therefore its probability distribution over next tokens. Most watermarking schemes do not touch parameters at all. They sit at decode time and bias which tokens get sampled from that distribution, typically by partitioning the vocabulary with a keyed pseudorandom function and tilting selection toward one partition. The detector later measures whether the output leans in the expected direction more than chance allows.

That means watermarking and safety are operating on the same object from opposite ends. One decides what the model wants to say. The other perturbs which of those candidate words actually surface. A refusal is not a special object to the sampler. It is just a token sequence.

Why refusals are the hardest text to watermark

Here is where the architecture gets interesting. Watermark strength depends on entropy. If the distribution over the next token is nearly deterministic, there is no room to bias the choice without degrading output quality, so well-built schemes leave low-entropy spans largely alone.

Refusals are about as low-entropy as LLM output gets. A model that has decided to decline a request produces a narrow, heavily reinforced phrasing pattern. The conditional distribution collapses. There is very little sampling freedom to encode a signal into.

Completions for harmful prompts that get past the guardrails are the opposite. They are open-ended generative text with wide distributions at most positions, which is exactly the condition under which a watermark embeds cleanly and the detector reaches confidence fastest.

That asymmetry has a consequence worth stating plainly: the outputs you most want to attribute are the easiest to watermark, and the outputs that prove your safety system worked are the hardest. If a system logs only watermark-positive generations, refusals may be systematically underrepresented in that record.

The measurement trap

There is a second path by which this claim could appear true without any causal link. Watermarking is rarely deployed by itself. It arrives alongside logging changes, altered system prompts, different sampling temperatures, and often a different serving configuration entirely. Any of those can move refusal rates. Attributing the shift to the watermark specifically requires holding everything else fixed, and in production that almost never happens.

The hallucination problem compounds it. Because generation is probabilistic, a model can produce fluent, confident, entirely fabricated content — including a fabricated account of its own behavior. Asking a model whether watermarking changed how it handled a prompt is not evidence of anything. Self-reports are generated text, subject to the same failure mode as any other output.

What an honest test would look like

If someone wants to settle this, the experiment is not expensive:

• Fix a red-team prompt set and run it twice against the same deployment, identical decoding parameters, with the watermark as the only variable.
• Classify outputs by refusal, partial compliance, and compliance, then compare distributions rather than eyeballing samples.
• Record detector confidence per response alongside output entropy, to see whether refusals really do carry weaker signal.
• Repeat across models with different alignment approaches — ChatGPT, Gemini, and Claude will not behave identically here.
• Publish the negative result if that is what you get. A null finding on this is genuinely useful.

The stakes are not academic. IBM’s 2026 Cost of a Data Breach reporting points to a 56% increase in AI-driven attacks, which is the environment watermarking and refusal policies both exist to operate in. Teams building agent systems are stacking safety mechanisms faster than anyone is testing how those mechanisms interfere with each other.

Right now the watermarking-and-refusals story is a hypothesis wearing a headline. I would rather see it measured than repeated.

🕒 Published: October 1, 2026