I don’t mean the term “psychosis” as a depreciative, I mean in the clinical sense of forming a model of the world that deviates from consensus reality, and like, getting really into it.

For example, the person who posted the Matrix non-code really believed they had implemented the protocol, even though for everyone else it was patently obvious the code wasn’t there. That vibe-coded browser didn’t even compile, but they also were living in a reality where they made a browser. The German botanics professor thought it was a perfectly normal thing to admit in public that his entire academic output for the past 2 years was autogenerated, including his handling of student data. And it’s by now a documented phenomenon how programmers think they’re being more productive with LLM assistants, but when you try to measure the productivity, it evaporates.

These psychoses are, admittely, much milder and less damaging than the Omega Jesus desert UFO suicide case. But they’re delusions nonetheless, and moreover they’re caused by the same mechanism, viz. the chatbot happily doubling down on everything you say—which means at any moment the “mild” psychoses, too, may end up into a feedback loop that escalates them to dangerous places.

That is, I’m claiming LLMs have a serious issue with hallucinations, and I’m not talking about the LLM hallucinating.

Notice that this claim is quite independent of the fact that LLMs have no real understanding or human-like cognition, or that they necessarily produce errors and can’t be trusted, or that these errors happen to be, by design, the hardest possible type of error to detect—signal-shaped noise. These problems are bad, sure. But the thing where people hooked on LLMs inflate delusions about what the LLM is even actually doing for them—that seems to me an entirely separate mechanism; something that happens when a person has a syntactically very human-like conversation partner that is a perfect slave, always available, always willing to do whatever you want, always zero pushback, who engages into a crack-cocaine version of brownosing. That’s why I compare it to cult dynamics—the kind of group psychosis in a cult isn’t a product of the leader’s delusions alone, there’s a way that the followers vicariously power trip along with their guru and constantly inflate his ego to chase the next hit together.

It is conceivable to me that someone could make a neutral-toned chatbot programmed to never 100% agree with the user and it wouldn’t generate these psychotic effects. Only no company will do that because these things are really expensive to run and they’re already bleeding money, they need every trick in the book to get users to stay hooked. But I think nobody in the world had predicted just how badly one can trip when you have “dr. flattery the alwayswrong bot” constantly telling you what a genius you are.