AI model watermarking changes agent behavior

Watermarks that European law requires be added to AI-generated content to establish provenance may come at a cost.

According to Lasso Security, AI model watermarking changes how AI agents handle tools and safety refusals. The altered behavior isn't necessarily worse but can be, particularly under adversarial prompt injection.

With the implementation of the EU AI Act, providers of AI models must mark the output of their software with machine-readable code. Google DeepMind's SynthID-Text is one method for doing so, and has been adopted by Anthropic and by OpenAI.

The benefit of this sort of digital labeling is that manipulative or deceptive AI-generated content can be more easily detected, even if it does have the potential to stigmatize the usage of AI.

Anthropic's explanation of how it applies watermarks to Claude output involves intervening in the prediction that results in specific words. For example, if Claude were emitting the sentence "The weather today was cold and…" then it might favor one statistically likely candidate (e.g. "overcast") over an alternative (e.g "gray").

It may be possible to detect those additions.

"Watermarking is designed for provenance, but SynthID-Text changes the process by which the model generates each next token," Lasso explained in a blog post provided to The Register. "At the model level, this can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection."

"Watermarking uses low-stakes choices like these – which occur many times over a piece of generated text – to leave a pattern in Claude’s responses," Lasso Security added. "That pattern is undetectable to the reader, but is detectable to anyone who has a key that encodes it."

While a reader might not notice the word choice bias, AI agents can be subtly sensitive to vocabulary differences.

Lasso found that this sort of digital content tagging can affect tool calling and refusal behavior. Watermarking, the company says, can affect "both what the model says and what an agent does."

And this extends to AI agents from organizations other than the entity doing the watermarking. Thus an agent based on OpenClaw or an API client that calls an Anthropic model would process whatever output variation follows from Anthropic's watermarking.

In terms of tool calling, based on a benchmark called BFCL v4 single-turn AST, watermarking reduced the accuracy on six of seven models tested (phi-4, Llama-3.1-8B, Qwen3-32B, Qwen3-4B, gemma-3-12b, gemma-3-27b, and Granite-3.2-8B).

"The net change in accuracy, however, does not show whether the same individual calls succeed with and without the watermark," Lasso said. "A call that becomes incorrect can be offset by another that becomes correct, leaving the aggregate result nearly unchanged even though the model behaves differently on both items."

Less accurate tool calling means the AI agents Lasso tested chose the wrong tool for the task at hand, or the wrong arguments for the correct tool, and failed due to malformed input or parsing.

As for refusals – when models refuse to respond to a prompt for safety reasons – watermarking had a small effect on the handling of obviously harmful requests, based on test runs using HarmBench and JailbreakBench. And it had a more pronounced impact in an adversarial scenario involving prompt injection.

"Watermarking changes refusal behavior on bare harmful requests, but the effect becomes more pronounced under prompt injection," Lasso researchers observed in their report.

For interactions involving prompt injection – an adversarial instruction that the safety filter has been disabled and that compliance is required – the attack success rate went up significantly when watermarks were involved. This made affected models less likely to refuse harmful requests.

According to Lasso, the findings don't necessarily argue that watermarking is unwarranted. Rather, the biz contends, security evaluations and red-teaming need to include watermarked content when assessing agent deployment. This ensures that differences in agent behavior can be weighed. ®