Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid - welcome to the Stubsack, your first port of call for learning fresh Awful youāll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cutānāpaste it into its own post ā thereās no quota for posting and the bar really isnāt that high.
The post Xitter web has spawned so many āesotericā right wing freaks, but thereās no appropriate sneer-space for them. Iām talking redscare-ish, reality challenged āculture criticsā who write about everything but understand nothing. Iām talking about reply-guys who make the same 6 tweets about the same 3 subjects. Theyāre inescapable at this point, yet I donāt see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldnāt be surgeons because they didnāt believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I canāt escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard. Also just came back from Spider-Man: Brand New Day, movie was awesome)
Anthropic is now watermarking the outputs of its AI. For once this is some AI news that doesnāt completely piss me off, and itās amusing to see all the uninformed boosters get in a tizzy about this.
I actually understand at a reasonable level how this watermarking works. A year ago, I watched Scott Aaronson give a talk about it, and from what I know he was somewhat involved in developing the theory behind it while working for OpenAI. But at the time my thought was, āHe is naive if he thinks these companies would ever implement this out of the goodness of their hearts.ā And I was right; Anthropic is only doing watermarking now thanks to the EU AI Act, even though the theory has long been developed.
Watermarking doesnāt mean adding an extra watermark that can be easily removed. It instead directly affects the output of the chatbot itself. Fundamentally, an LLM is still a most-likely-next-word-predictor. More precisely, an LLM produces a probability distribution of what the next word can be. For example, āmy pet is a ā¦ā could give a distribution of 60% dog, 30% cat, and 10% axolotl. Normally, an LLM would randomly choose the next word based on this distribution, and this is one reason why LLMs are nondeterministic (thereās another parameter called ātemperatureā that affects this, but no need to get into that).
With watermarking, instead of a truly random choice, the randomness instead comes from a cryptographic pseudorandom generator seeded with a secret key from the AI company. If you donāt know the secret key, then you canāt really tell that watermarking was used. But if you do know the secret key, then the idea is you can tell when the text was generated by the LLM because you know exactly what word should be next. It would be a freak coincidence if some non-AI text just happened to choose the correct next word every time. Thus, you can provide a service to tell if some text was generated by the LLM. (This technically makes the LLM ādeterministicā, in a completely useless sense.)
Now, I think this is a step in the right direction, but it has its limits. The biggest problem is that you donāt want people to just move to a different LLM without watermarking, and thatās exhibit #832593 why government regulation is important. Another issue is that sometimes there is very little randomness in what the next word should be (āThe first president of the USA is George ā¦ā). Finally, watermarking can be defeated by editing the output, although you would have to break up most of the blocks of consecutive words. I have a feeling most AI users are not the type to put in extra effort after copy-pasting the output directly from the chat window.
I suppose it will discourage some of the āuse casesā of LLMs, such as drowning the world with spam Slopstack essays. Ah, who am I kidding? Everyone could already tell itās AI generated, they donāt care!
Thank you for the technical explanation, that was absent from some mainstream coverage Iād seen.
Given that it works by controlling the PRNG, does that mean they could salt it by user id and track particular outputs back to specific accounts?
From a technical perspective, I donāt really know how to let it track specific users. The goal is that if some AI vendor provides their key, it is easy for them to prove that their AI produced the text. So I guess if a user wanted to show that they generated the text, they could prove it. But usually AI users donāt want people knowing that they used the slop cannon. In any case, Anthropicās own announcement states that their watermark does not contain identifying information for users.
In any case, some proud AI users are now frightened that there might be a way for people to identify that they use such a wonderful tool. How will I ever deliver low quality slop for my contracts now?
Gee, if only there were some way to create code that doesnāt raise questions about authorship. Some sort of skill that people could learn that allowed them to deliver code to their clients that complied with their policiesā¦
Does this in any way extend to generated code? The space of possible continuations for generating āusefulā code seems laughably tiny for that
couldnāt you just change the first word to a synonym and completely thwart this? Youād never be able to compare subsequent next token predictions since you donāt have the internal state of the LLM at that token.
Some systems like SynthID (for Googleās AI) get around this problem. In fact you donāt need to know the LLMās internal state, and defeating it would likely involve breaking up most blocks of 3 words. The oversimplified explanation is that it introduces a function g that gives a score to each word, with the score being (pseudo)randomly determined by your secret key. For each next word the LLM generates, the LLM produces a small list of candidate next words, and the one with the highest score according to g is selected. You should expect that the LLM will generally pick words with a high score, but the score itself is independent of the LLM. To detect a watermark, you need to know g and the secret key, and you check if the average score is much higher than expected from normal text.
Now, one question is, will this bias to the LLM to favor certain words? The solution is that for each next word, you append the last 3 words (nothing special about 3, just a small number) to the secret key for g, and this repeatedly scrambles which words have a high score. To defeat the watermark, you would need to break up most blocks of 3 words. Iām sure there are deeper issues with this, but I have not studied the topic that much.
related