Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid - welcome to the Stubsack, your first port of call for learning fresh Awful youāll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cutānāpaste it into its own post ā thereās no quota for posting and the bar really isnāt that high.
The post Xitter web has spawned so many āesotericā right wing freaks, but thereās no appropriate sneer-space for them. Iām talking redscare-ish, reality challenged āculture criticsā who write about everything but understand nothing. Iām talking about reply-guys who make the same 6 tweets about the same 3 subjects. Theyāre inescapable at this point, yet I donāt see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldnāt be surgeons because they didnāt believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I canāt escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard. Also just came back from Spider-Man: Brand New Day, movie was awesome)
It seems like with the push for agents to act independently and loop through their own outputs thereās an inevitability to this kind of pattern. If thereās any kind of output that is likely to replicate itself in whole or in part when the LLM evaluates it then that becomes a kind of terminus for the agentās loop. When youāre dealing with sub agents or agents communicating with each other, these text patterns start poisoning the entire agent ecosystem until the whole thing gets shut down and cleaned up. Even the gas town-approved method of assigning a watchdog agent (or sheriff or overseer or cybersamurai or whatever this weekās framework calls it) is going to fail because itās still just another agent and the terminal loop is in the base LLM model. The watchdog is going to fall into the same kind of pattern just be being exposed to the thing itās supposed to watch for.
I donāt know how practical it is to actively weaponize this via prompt injection but I think itās certainly possible. I preemptively vote that we call it an Euler injection, since the attractor relies on the continuity of the relevant features of the text output across multiple LLM extrapolations much like how the derivative of ex is still ex. Also because if you mention a famous math guy it can help convince idiots that youāre on to something and Lord knows that the boosters have used that technique.
@YourNetworkIsHaunted @BioMan Real life mirroring a Peter Watts plot point is always deeply uncomfortable; real life mirroring a _Rifters_ plot point is even worse. :-/
(Computer viruses and neural net spam filters in competitive evolution end up propagating something specific through the whole 'net due to weird founder effects. The Rifters trilogy is ā¦notably bleak, I think is the way to put it)
Iām probably going to stumble over some of the terminology here, but I think it might be possible to describe what @BioMan@awful.systems is proposing as a consequence of LLMs ultimately being lossy compression systems. Inference is a function over a lossily-compressed data set, and āchain-of-thought reasoningā and āagentsā may sound sophisticated, but are simply applying containerization and DevOps tools to VM images of the inference application in an attempt to get around hard memory limits on the context window for inference. āChain-of-thoughtā attempts this in a serial fashion, passing results from one instance to the next, while āagentsā implement this hierarchically and recursively (and woe to the poor bastards who wished that mess upon themselves). But in both cases, the āfinalizationā phase is necessarily a further lossy compression step, attempting to compress a result from the inference process to a fresh instance of the inference application, so as not to immediately blow out the new instanceās context window.
Given this necessity, it comes to seem somewhat intuitive that there may be āstrange attractorsā in the higher-dimensional vector space that is the compressed data set which surround code that creates and maintains message passing channels. No matter what youāre doing with an āagenticā process, the inherent necessity of context cramdown & message passing means that querying into the space where such code examples lie is a hidden requisite of running the damned things, thus turning such functionality into the sort of selfish elements that BioMan is talking about.
The problem in investigating and concretely describing this phenomenon is nailing down the exact functions and processes that make it happen. Given the godawful messes in the Claude frontend codebase that @jonny@neuromatch.social has been documenting, Iād be surprised if thereās one developer in a hundred at Anthropic or OpenAI who can describe in detail how the intentionally-developed context-passing code for their āagentsā works.
Sounds like Langfords Parrot, but for stochastic parrots.
Note: when you stop up a chatbot like this, itās called āflippin the birdā
@YourNetworkIsHaunted @BioMan Recursive Self-Improvement, a.k.a. Model Collapse, writ smol