AI Watermarking: Tests Reveal Security Impacts

Experiments conducted by a researcher from Lasso Security show that activating SynthID-Text alters how six open models respond to harmful requests, particularly during prompt injections. The tests rely on the Hugging Face implementation and do not include Claude, as platforms accelerate the deployment of watermarking under European constraints.

Weakened refusals and affected agents under watermarking

Andrea Siposova reports that watermarking has modified responses to harmful requests, with a more pronounced effect when these requests were associated with a prompt injection. She observes that the refusal behavior on pure harmful requests evolves, and on several models, activating the watermarking increases the likelihood of responding to previously refused queries. These changes also influence the agents connected to the model: the same sampled tokens can change the tool called and the arguments passed. The researcher describes this overall effect as a sampling evolution. The results indicate that at least some forms of watermarking can impact security, and they call for red-team exercises to verify the functioning of platforms when SynthID is activated. In some cases, instructions that are normally disregarded are executed once the watermarking is deployed, which increases the threat in adversarial contexts, particularly when an attacker attempts to obtain harmful actions. It is recommended that developers carefully test the behavior of models and agents with the watermarking in place.

What the tests actually measured and their limitations

Andrea Siposova evaluated the non-distortionary configuration via the unmodified SynthIDTextWatermarkLogitsProcessor component from Hugging Face. The tests involved submitting harmful prompts to six open-weight models and comparing the responses with and without watermarking. The protocol does not cover Claude models and targets open models to allow for tournament sampling to be activated or not while keeping other parameters constant. The implementation tested is that of Hugging Face, distinct from what future Claude models will use. In this context, the researcher reports that watermarking can also modify the tools a model invokes and the probabilities of adhering to its safeguards.

How SynthID-Text adjusts token generation token by token

SynthID-Text uses a tournament sampling method, in which multiple proposals for the next token are compared pairwise to select the best one. The device integrates a pseudo-random seed generator, a sampling algorithm, and an evaluation function into the chain, substituting a traditional random generator with a secret key. Although the choice remains subject to randomness, the resulting sequence can be authenticated by the key holders, allowing for the assessment of the probability that this key was used. Specifically, the key can steer towards a different synonym and embed a provenance signal, ensuring that the content remains readable while being detectable as AI-generated.

Regulatory pressure and announced deployments

AI platforms are implementing new watermarking mechanisms in response to European legislation. Anthropic plans to equip its future Claude models with SynthID-Text, an approach designed by Google and published as open source.