Want to wade into the sandy surf of the abyss? Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid.
Welcome to the Stubsack, your first port of call for learning fresh Awful youāll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cutānāpaste it into its own post ā thereās no quota for posting and the bar really isnāt that high.
The post Xitter web has spawned so many āesotericā right wing freaks, but thereās no appropriate sneer-space for them. Iām talking redscare-ish, reality challenged āculture criticsā who write about everything but understand nothing. Iām talking about reply-guys who make the same 6 tweets about the same 3 subjects. Theyāre inescapable at this point, yet I donāt see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldnāt be surgeons because they didnāt believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I canāt escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard for starting this.)
LLM capabilities have not improved at all in terms of producing meaningful science in the last year or two, but their ability to produce meaningless science that looks meaningful has wildly improved. I am concerned that this will present serious problems for the future of science as it becomes impossible to find the actual science in a sea of AI slop being submitted to journals.
https://www.reddit.com/r/Physics/comments/1s19uru/gpt_vs_phd_part_ii_a_viewer_reached_out_with_a/
Iāve seen this story play out in software engineering: people were very impressed when the AI does unexpectedly well in one out of 50 attempts on an easy task, and so people decided to trust it for everything and turn their codebases into disasters. There was no great wave of new high-quality software. Instead, the only real result was that existing software has become far more buggy and insecure.
Now we have people using AI in science and math because it was impressive in random demonstrations of solving math problems. I now have friends asking me why Iām not using AI, and also saying that AI will be better than all mathematicians in 30 years or whatever. Do you really think I refuse to use AI out of ignorance? No, I know too much about it! I have seen the same story play out in software engineering, and what makes this any different?
āScientists invented a fake disease. AI told people it was realā
https://www.nature.com/articles/d41586-026-01100-y
But if, in the past 18 months, you typed those symptoms into a range of popular chatbots and asked what was wrong with you, you might have got an odd answer: bixonimania.
The condition doesnāt appear in the standard medical literature ā because it doesnāt exist. Itās the invention of a team led by Almira Osmanovic Thunstrƶm, a medical researcher at the University of Gothenburg, Sweden, who dreamt up the skin condition and then uploaded two fake studies about it to a preprint server in early 2024. Osmanovic Thunstrƶm carried out this unusual experiment to test whether large language models (LLMs) would swallow the misinformation and then spit it out as reputable health advice. āI wanted to see if I can create a medical condition that did not exist in the database,ā she says.
The problem was that the experiment worked too well. Within weeks of her uploading information about the condition, attributed to a fictional author, major artificial-intelligence systems began repeating the invented condition as if it were real.
This actually gives me hope that we can poison the datasets pertaining to any sufficiently narrow technical topic.
āAI HAS SOLVED THE SCIENCE-GENERATION CRISISā
It can do trillions of calculations per second. All of them wrong.
The replausibility crisis.
OT: got a job selling tires and Iām really happy to say theres no AI as far as Iāve seen so far. Big relief.
(I get to see all kinds of cars - a Rivan of all things showed up my first day - and Iām learning stuff I can apply to being an RMT. I gotta say Iām pretty content).
Nice to hear youāve got a job, and good luck!
Iām just shocked the managers/supervisors are praising me honestly (Iām very very hard on myself).
Nice. Good luck at your new job!
Dan Gackle threatens to quit HN over their reluctance to condemn an act of violence towards Sam Altman:
I donāt think Iāve ever seen a thread this bad on Hacker News. The number of commenters justifying violence, or saying they ādonāt condone violenceā and then doing exactly that, is sickening and makes me want to find something else to do with my lifeāsomething as far away from this as I can get. I feel ashamed of this community.
Gackleās ashamed of people not wanting to protect Altman. Curiously, he doesnāt seem ashamed of openly allowing people with nicknames ending in ā88ā to post antisemitism, nor of allowing multiple crusty conservatives like John Nagle and Walter Bright to post endorsements of violence against the homeless and queer, nor of allowing posters like
rayinerto port entirely foreign flavors of racism like the Indian caste system into their melting pot of bigotry. This subthread takes him to task for it:Frankly people calling out a post from a billionaire is a good thing. You would have to be terminally detached from reality to not see how all these festering issues - wealth inequality, injustice, cost of living, future employment etc etc - are starting to come to a head which would cause people to feel something - frustrated, angry, wrathful.
The rest of that subthread involves Dan demonstrating that he is, in fact, terminally detached from reality. Anyway, I fully endorse Gackle fucking off and buying a farm. While heās at it, he should consider following the advice of this reply:
Maybe itās time to pack it in? I donāt just mean you, I mean that maybe this site has kinda run its course.
Every day, HN users flag into oblivion anything mildly critical of the technological dystopia these tech-bros are trying to manifest. āPolitics!ā they cry. But Sam Altman comes along with an OpenAI marketing piece dressed up as a condemnation of political violence, and suddenly āpoliticsā are a perfectly acceptable topic. dang has long made it clear whose side heās on.
Oh, and I hope everyone noted how quickly Sam used this incident as an excuse to place blame on the reporters who published the New Yorker piece that was mildly critical of him:
Words have power too. There was an incendiary article about me a few days ago. Someone said to me yesterday they thought it was coming at a time of great anxiety about AI and that it made things more dangerous for me. I brushed it aside.
Thatās a hilarious reaction.
Anyway thereās zip about this incident on LW, which is telling.
edit hereās a very oblique reference https://www.lesswrong.com/posts/igEogGD9TAgAeAM7u/jimrandomh-s-shortform?commentId=zdMRHRqWDcjswhA3i
donāt miss the anarcho-libertarian in the comments
Lesswrong is too centrist-brained to ever even hint at legitimizing (non-state-sanctioned) destruction of property as a means of protest or political action. But according to the orthodox lesswrong lore, Sam Altmanās actions are literally an existential threat to all humanity, so they canāt defend him either. So they are left with silence.
I actually kind of agree with the anarchy-libertarianās response? It is massively down voted.
This is just elevating your aesthetic preference for what the violence youāre advocating for looks like to a moral principle. The claim that throwing a Molotov cocktail at one guyās house is counterproductive to the goal of ābombing the datacentersā is a better argument, though one I do not believe.
Bingo. Dear leader Yudkowsky can ask to bomb the data centers, and as long as this action goes through the US political process, that violence is legitimate, regardless of how ill-behaved the US is or itās political processes degraded from actually functioning as a democracy.
Specifically, a screenshot of a moderator warning him that advocating violence is grounds for a ban there. It would also be grounds for a ban on LW.
That explains why Yud is using twitter so much nowadays. I mean they did ban him right? right?
Someone just made a top post condemning the Molotov but defending and normalizing Eliezer: https://www.lesswrong.com/posts/Sih2sFHEgusDEuxtZ/you-can-t-trust-violence
Ah suddenly when it reaches the class he feels he should be a part of (or is a part of, I donāt know how much money he makes) violence is suddenly a problem.
Itās not easy to be a cop, and thatās basically what you are around here, but thank you for doing it.
ā¦
Claude Mythosā¦ Iām already sick of hearing about it. The self-imposed critihype is insane.
A friend just pointed out that Anthropic are making all this big noise about having an AI that is ātoo goodā at finding bugs and security problems 1 week after the source code for one of their flagship products was leaked to the public and was found to be riddled with security holesā¦ Why would they not use it themselves?
Same as the
vague markdown filesskills that are supposedly going to make all SaaS redundant and finally kill off all the COBOL running on mainframes that checks notes IBM have spent hundreds of thousands of man hours trying to kill over the last 3-4 decadesHonestly fuck this shit. Bunch of absolute clowns š¤” š¤” š¤”
So, they are planning to use an ai to fix the sec bugs that their ai generates? Good hussle, if a bit obvious.
The fuck is Mythos?
Is it their next model that tbey swear isnāt vaporware but no! It is too dangerous to release into the world because itāll find too much insecure code or whatever.
Ia ia Claude! Phānglui mglwānafh Claude Anthrālyeh wgahānagl fhtagn! Ia! Ia!
Okay but like is it materially different than whatever the current Claude thing is or did they just pump the size of the matrix?
Probably a markdown file telling it āyou are a l33t h4x0rā
Okay but thatās already in Claude
I still laugh every time I see that this is what qualifies as proper ātuningā and āsecurity controlsā for these things.
I had hoped that with the whole āagentā push that we would start seeing more sane usage, like having AI be a fuzzy logic step in a chain of formal logic and existing deterministic tools, but the cult still has people treating them like reliable second brains. Theyāre used as the baseline fucking orchestrator rather than anywhere they might make a bit of sense.
I had hoped that with the whole āagentā push that we would start seeing more sane usage, like having AI be a fuzzy logic step in a chain of formal logic and existing deterministic tools
I think this is the best you can expect out of LLMs, and the relatively more successful āagenticā AI efforts are probably doing exactly this, but their relative success is serving as hype fuel for the more impossible promises of LLMs. Also, if you have formal logic and deterministic tools wrapping and sanity checking the LLM bitsā¦ I think the value add of evaporating rivers and firing up jet turbines to train and serve ācutting edgeā models that only screw up 1% of the time isnāt there because you can run a open weight model 1/100th the size that screws up 10% of the time instead. (Note one important detail: training costs go up quadratically with model size, so a 100x size model is 10,000x training compute.) I think the frontier LLM companies should have pivoted to prioritizing smaller size, greater efficiency, and actually sustainable business practices 4 years ago. At the very latest, 2 years ago, with the release of 4o OpenAI should have realized pushing up model size was the wrong direction (as they should have realized training Chain-of-Thought was not going to be the magic bullet).
And to be clear I still think this is really generous to the use case of smaller LMs.
they added more leetspeak and changed the vm they run it on to kali linux for extra h4xx0r power
Anthropicās latest model that they havenāt released to the public yet since theyāre worried its gonna fuck up cybersecurity this thread goes over it a bit
XCancel link for those of us sick of being badgered to sign up/in
On a more productive note, this feels likely to be tied in with the usual issues of AI sycophancy re: false positive rate. If you ask the model to tell you about security vulnerabilities, itās never going to tell you there arenāt any, any more than existing scanners will. When I worked for F5 it was not uncommon to have to go down a list of vulnerabilities that someoneās scanner turned out and figure out whether they were actually something that needed mitigation that could be applied on our box, something that needed to be configured somewhere else in the network (usually on their actual servers) or (most commonly) a false positive, e.g. āyour software version would be vulnerable here, which is why it flagged, but you donāt have the relevant module activated and if an attacker is able to modify your system to enable it youāre already compromised to a far greater degree than this would allow.ā That was with existing tools that werenāt trying to match a pattern and complete a prompt.* Given that weāve seen the shitshow that is Claude Code I think itās pretty clear theyāre getting high on their own supply and this announcement ought be catnip for black hats.
Wow, sounds like they just automated āshitty infosec teams that only forward scanner output without evaluating itā out of a job. Holy shit they were right that AI was coming for jobs!
True. I will say that the shitty infosec teams are probably being hit less hard than the SMEs they offloaded their jobs onto, because from their perspective it doesnāt actually matter whether itās f5 support engineer or a chatbot that tells them the answer; either way theyāve successfully offloaded the task of validating security onto another entity that can make up for their shortcomings with a combination of accuracy and authority. Nobody is going to get fired for not fixing a bug that the vendor SME told them wasnāt actually an issue for them, effectively. And when the org has been pushing AI as hard as so many of them have its pretty easy to throw the chatbot under the same bus and expect the bus to stop instead.
On a more productive note, this feels likely to be tied in with the usual issues of AI sycophancy re: false positive rate.
I suspect this is the real limit. Claude Mythos might find real vulnerabilities, but if they are buried among loads of false positives it wonāt be that useful to black or white hat hackers and the endless tide of slop PRs and bug reports will keep coming.
I tried looking through Anthropicās āpreviewā for a description of the false positive rateā¦ they sort of beat around the bush as to how many false positives they had to sort out to find the real vulnerabilities they reported (even obliquely addressing the issue was better than I expected but still well short of the standard for a good industry-standard security report from what I understand).
Theyāve got one class of bugs they can apparently verify efficiently?
Memory safety violations are particularly easy to verify. Tools like Address Sanitizer perfectly separate real bugs from hallucinations; as a result, when we tested Opus 4.6 and sent Firefox 112 bugs, every single one was confirmed to be a true positive.
Itās not clear from their preview if Claude was able to automatically use Address Sanitizer or not? Also not clear to me (Iāve programmed with Python for the past ten years and havenāt touched C since my undergraduate days), maybe someone could explain, how likely is it that these bugs are actually exploitable and/or show up for users?
Moving onā¦
This process means that we donāt flood maintainers with an unmanageable amount of new workābut the length of this process also means that fewer than 1% of the potential vulnerabilities weāve discovered so far have been fully patched by their maintainers.
So its good they arenāt just flooding maintainers with slop (and it means if they do publicly release mythos maintainers will get flooded with slop bug fixes), butā¦ this makes me expect they have a really high false positive rate (especially if you rule minor code issues that donāt actually cause bugs or vulnerabilities as false positives).
This NPR article opens with a banger of a line:
In the past few months, AI models have gone from producing hallucinations to becoming effective at finding security flaws in software, according to developers who maintain widely used cyber infrastructure.
The things still fucking hallucinate, itās not a feature thatās separable from the model.
tldr; one of the MIRI aligned rationalist (Rob Bensinger) complained about how EA actually increased AI-risk long-run by promoting OpenAI and then Anthropic. Scott Alexander responded aggressively, basically saying they are entirely wrong and also they are bad at public communications! Various lesswrongers weigh in, seemingly blind to irony and hypocrisy!
Some highlights from the quotes of the original tweets and the lesswronger comments on them:
-
Scott Alexander tries blaming Eliezer for hyping up AI and thus contributing to OpenAI in the first place. Just a reminder, Scott is one of the AI 2027 authors, he really doesnāt have room to complain about rationalist creating crit-hype.
-
Scott Alexander tries claiming SBF was a unique one off in the rationalist/EA community! (Anthropicās leadership has been called out on the EA forums and lesswrong for a similar pattern of repeated lying)
-
Rob Bensinger is indirectly trying to claim Eliezer/MIRI has been serious forthright honest commentators on AI theory and policy, as opposed to Open-Phil/EA/Anthropic which have been āstrategicā with their public communication, to the point of dishonesty.
-
habryka is apparently on the verge of crashing out? I canāt tell if they are planning on just quitting twitter or quitting their attempts at leadership within the rationalist community. Quitting twitter is probably a good call no matter what.
-
Load of tediously long posts, mired with that long-winded rationalist way of talking, full of rationalist in-group jargon for conversations and conflict resolution
-
Disagreement on whether Ilya Sutskeverās $50 billion dollar startup is going to contribute to AI safety or just continue the race to AGI.
-
Arguments over who is with the EAs vs. Open Philanthropy vs. MIRI!
-
Argument over the definition of gaslighting!
To be clear, I agree with the complaints about EA and Anthropic, I just also think MIRI has its own similar set of problems. So they are both right, all of the rationalists are terrible at pursing their alleged nominal goals of stopping AI Doom.
I did sympathize with one lesswrongerās comment:
More than any other group Iāve been a part of, rationalists love to develop extremely long and complicated social grievances with each other, taking pages and pages of text to articulate. Maybe Iām just too stupid to understand the high level strategic nuances of whatās going on ā what are these people even arguing about? The exact flavor of comms presented over the last ten years?
Old Twitter was terrible for peopleās souls. I can only imagine what it is like now that the well-meaning professionals are gone and catturd and Wall Street Apes are the leading accounts.
Iām willing to go out on a limb and say that short-form social media in general (Twitter and imitators, Instagram, TikTok) is essentially a failed set of media. But Iāll concede thatās like cramming a Zyn pouch in my mouth while making fun of a guy chain-smoking Marlboros.
Iāve read speculation that in 30-50 years people will have an attitude towards social media that we have towards cigarettes now.
That would be really nice but that scenario feels pretty optimistic to me on a few points. For one, scientists doing research were able to overcome the lobbying influence and paid think tanks of cigarette companies. I am worried science as a public institution isnāt in good enough shape to do that nowadays. Likewise part of the push back against cigarettes included a variety of mandatory labeling and sin taxes on them, and it would take some pretty major shifts for the political will for that kind of action to be viable. Well maybe these things are viable in the EU, the US is pretty screwed.
The only people I trust as little as I trust the owners of corporate social media are the politicians who have decided to cash in on the moment by āregulatingā them. I mean, here in progressive Massachusetts, the state house of representatives just this week passed a bill that, depending on the whims of the Attorney General, would require awful.systems to verify the ages of its users by gathering their government-issued IDs or biometrics. We are, you see, a āpublic website, online service, online application or mobile application that displays content primarily generated by users and allows users to create, share and view user-generated content with other usersā. And so we would have to āimplement an age assurance or verification system to determine whether a current or prospective user on the social media platformā is 16 or older. (Or 14 or 15 with parental consent, but your humble mods lack the resources to parse divorce laws in all localities worldwide, sort out issues of disputed guardianship, etc., etc.) The meaning of what āpracticableā age verification is supposed to be would depend upon regulations that the Attorney General has yet to write.
So, yeah, as an old-school listserv nerd who had the I am not on Facebook T-shirt 15 years ago, I donāt trust any of these people.
Iām not quite so pessimistic. Itās important to remember that the actual practical purpose of the extant corporate social media* is to convey targeted advertising; i.e. an optimization (possibly the last optimization) on American management of global supply chains. Those supply chains were already starting to be optimized past their breaking point: flooded with dissatisfactory junk, easily spoofed by low-quality sellers, on top of broader externalities besides. And now, they have now been blasted into fine dust by a failed presidency partially funded by the social media and online advertising barons. It may yet be something of a self-correcting problem, albeit having done substantial damage in the meantime.
*Twitter is now a fully dedicated advertising campaign for Elon Muskās program of white supremacy, with financial returns no object. Itās not quite going according to plan. By this time next decade, the Twitter microblogging permutation of the tech may be thoroughly killed, and if not itāll be disgustingly cringe. Who do you think you are posting like that, Baby Trump?!?!
The collapse of the current American management of global supply chains isnāt exactly an optimistic expectation, but I guess it beats social media continuing as it is into the future and maybe a better global order will develop in the aftermath.
Havenāt seen any estimates of death toll due to social media but cigarettes is/was pretty staggering (20-40m), way too big to hide - https://www.ucpress.edu/books/golden-holocaust/hardcover - if itās āonlyā 50 years to flip the consensus on social media, that would be a faster process, I do hope its possible though. Tobacco execs had the good sense to keep a relatively low profile compared to Zuck and Musk, so that might speed it up.
Old Twitter was terrible for peopleās souls.
It almost makes me feel sorry for the way the rationalists are still so attached to it. But they literally have two different forums (lesswrong and the EA forum), so staying on twitter is entirely their choice, they have alternatives.
Fun fact! Over the past few years, Eliezer has deliberately cut his lesswrong posting in favor of posting on twitter, apparently (heās made a few comments about this choice) because lesswrong doesnāt uncritically accept his ideas and nitpicks them more than twitter does. (How bad do you have to be to not even listen to critique on a website that basically loves you and take your controversial foundational premises seriously?)
Bonus race pseudoscience quoted by No77e!
There is a phenomenon in which rationalists sometimes make predictions about the future, and they seem to completely forget their other belief that weāre heading toward a singularity (good or bad) relatively soon. Itās ubiquitous, and it kind of drives me insane. Consider these two tweets:
Richard Ngo @RichardMCNgo: Hypothesis: Weāll look back on mass migration as being worse for Europe than WW 2 was. ā¦ high-trust and homogeneous ā¦ internal ethno-religious fractures.
Liv Boeree @Liv_Boeree: Would not be surprised if it turns out that everyone outsourcing their writing to LLMs will have a similar or worse effect on IQ as lead piping in the long run
(he shares these tweets as photos, I aināt working harder to transcribe them or using a chatbot)
No77e is correctly noting the discrepancy between the rationalist obsession with eugenics and the belief in an imminent (or even the next 40 years) technological singularity, but fails to realize that the general problem is the eugenics obsession of rationalists. It is kind of frustrating how close but far they are from realizing the problem.
Also, reminder of the time Eliezer claimed Genesmithās insane genetic engineering plan was one of the most important projects in the world (after AI obviously): https://www.lesswrong.com/posts/DfrSZaf3JC8vJdbZL/how-to-make-superbabies?commentId=fxnhSv3n4aRjPQDwQ Apparently Eliezerās plan if we arenāt all doomed by LLMs is to let the genetically engineered geniuses invent friendly AI instead.
-
Starting this Stubsack off, Iranās Islamic Revolutionary Guard Corps have threatened to blow up OpenAIās Stargate datacentre in Abu Dhabi.
Theyāve already bombed commercial data centres before, so Iām inclined to believe this isnāt an empty threat.
Waiting for Yud to provide his whole-chested support for IRGC any second
I assign a relatively low probability, non-zero but not much more than 5%, maybe a solid 5.5%, that Yud goes even beyond that and implies that he is the 12th imam emerging from occultation
IRGC doing their part to Halt AI. Donate to them now!
aside from everything else, as posted by ed zitron previously i doubt that anything is really getting built there
Interesting, got a link?
thereās been a claim of 200ish GW of new datacenters announced but itās only something like 5GW being built and only a fraction of that is actually finished or powered on. this mostly goes for american ai dcs, which are 3/4 of them all. not sure if it holds for emirates, but it would completely not surprise me if itās the case there also https://www.wheresyoured.at/the-ai-industry-is-lying-to-you/#how-much-actual-data-center-capacity-came-online-in-2025-my-estimate-3gw-of-it-load
Except it is a problem, man! As I covered in this weekās free newsletter [link above], I estimate that only around 3GW of actual IT load (so around 3.9GW of power) came online last year, and as Sightline reported, only 5GW of data center construction is actually in progress globally at this time, despite somewhere between 190GW and 240GW supposedly being in progress. In reality, data centers take forever to build (and obtaining the power even longer than that), but nobody needs to harsh their flow by looking into whatās actually happening.
https://www.wheresyoured.at/premium-how-much-of-the-ai-bubble-is-real/ (paywalled) linked claim for 190GW https://www.sightlineclimate.com/research/data-center-outlook linked claim for 240GW https://paulkedrosky.com/chart-of-the-day-data-center-buildout-slowed-sharply/
Thanks for the sources Iāll come back after I do some reading!
Work wants to add that new whiz-bang agentic AI into a scheduling service that I have been tasked with building, but in the dumbest way possible kind of similar to the Jetās text-a-pizza-order thing that worked like shit. I need to find an entirely new profession, everyone in software now is fucking deranged.
Itās bad for me too.
Iām trying to hang in there until I get some healthcare stuff taken care of over the next year or two but it is getting increasingly difficult. Most of the the good people at my job have been driven out, quit, or been poached by other (AI) companies.
By this point a majority of the programmers at my job (or at least the oneās most active on the mailing lists) are LLM true believers who think that the end times are near. My management chain has explicitly said that LLM programming is required, and that a subsequent increase in āproductivityā is expected with it. My department got renamed to something with āAIā in the name. I constantly field questions from people who want me to read a screen full of LLM nonsense, or who push back when I tell them something claiming that the chatbot said differently.
Thereās always some frantic push to adopt āMCPā or āSkillsā or whatever the next fad will be without any guidance as to how or why. If I ignore this I get nastygrams from my manager.
And at my last doctor visit I had elevated blood pressure :)
and that a subsequent increase in āproductivityā is expected with it.
Oh noā¦ they def will blame the users before blaming the faulty tools. Hope you will not be the one who gets blamed as a wrecker or something when the eventual increase isnāt there (or other metrics fall off a cliff).
Up next, when the first agent fails, implement an agent that checks the other agent. Both of these need agents to check for malicious inputs of course. And translation agents.
I need to find an entirely new profession, everyone in software now is fucking deranged.
Mood
New Yorker article on Sam Altman dropped. Aaron Swartz apparently called him a sociopath. The article itself also had wat looked like an animated AI generated image of Altman so here is the archive.is link (if you can get the latter to load, I was having troubles).
āNew interviews and closely guarded documents shed light on the persistent doubts about the head of OpenAI.ā
My CEO who is a known hype-man is a massive liar? shock horror
seriously, anyone who listens to Scam Altman these days is an idiot
Man, this one is a weird read. On one hand I think theyāre entirely too credulous of the āAI Futureā narrative at the heart of all of this. Especially in the opening they donāt highlight how the industry is increasingly facing criticism and questions about the bubble, and only pay lip service to how ridiculous all the existential risk AI safety talk sounds (should be is). And they donāt spend any ink discussing the actual problems with this technology that those concerns and that narrative help sweep under the rug. For all that they criticize and question Saltman himself this is still, imo, standard industry critihype and Iām deeply frustrated to see this still get the platform it does.
But at the same time, I do think that itās easy to lose sight of the rich variety of greedy assholes and sheltered narcissists that thrive at this level of wealth and power. Like, I wholly believe that Altman is less of a freak than some of his contemporaries while still being an absolute goddamn snake, and I hope that this is part of a sea change in how these people get talked about on a broader level, though I kinda doubt it.
I aired some Reviewer #2 grievances in the bsky comments:
https://bsky.app/profile/ronanfarrow.bsky.social/post/3mitapp7j2s2c
āKalanick now runs a robotics startup; in his free time, he said recently, he uses OpenAIās ChatGPT āto get to the edge of whatās known in quantum physics.āā
As a physicist, I have never pressed F to doubt harder.
āIn 2022, researchers at a pharmaceutical company tested whether a drug-discovery model could be used to find new toxins; within a few hours, it had suggested forty thousand deadly chemical-warfare agents.ā To the best of my knowledge, these suggestions were never evaluated by any other researchers.
(The original paper was published as a ācommentā: https://www.nature.com/articles/s42256-022-00465-9)
Similar claims of AI-facilitated discoveries have turned out to be overblown in other fields.
https://pubs.acs.org/doi/pdf/10.1021/acs.chemmater.4c00643
āIn a 2025 study, ChatGPT passed the test more reliably than actual humans did.ā
If this is referring to Jones and Bergenās āLarge Language Models Pass the Turing Testā, thatās a preprint (arXiv:2503.23674) that has yet to pass peer review over a year after its posting.
āA classic hypothetical scenario in alignment research involves a contest of wills between a human and a high-powered A.I. In such a contest, researchers usually argue, the A.I. would surely winā
Which researchers?
(Hint: Eliezer Yudkowsky is not a researcher.)
AI: āI will convince you to let me out of this boxā
Humanity (wringing hands): āOh, where is our savior? Who will stand fast in the face of all entreaties?ā
Bartleby the Scrivener: hello
āā¦a hub of the effective-altruism movement whose commitments included supporting the distribution of mosquito nets to the global poor.ā
Phrasing like this subtly underplays how the (to put it briefly) weird people were part of EA all along.
https://repository.uantwerpen.be/docman/irua/371b9dmotoM74
āIn late 2022, four computer scientists published a paper motivated in part by concerns about ādeceptive alignment,ā ā¦ one of several A.I. scenarios that sound like science fictionābut, under certain experimental conditions, itās already happening.ā
Barrett et al.'s arXiv:2206.08966? AFAIK, that was never peer-reviewed either; āpostedā is not the same as āpublishedā. And claims in this area are rife with criti-hype:
https://pivot-to-ai.com/2025/09/18/openai-fights-the-evil-scheming-ai-which-doesnt-exist-yet/
Oh, right, the āFuture of Life Instituteā. Pepperidge Farm remembers:
āIn January 2023, Swedish magazine Expo reported that the FLI had offered a grant of $100,000 to a foundation set up by Nya Dagbladet, a Swedish far-right online newspaper.ā
https://en.wikipedia.org/wiki/Future_of_Life_Institute#Activism
āTegmark also rejected any suggestion that nepotism could have played a part in the grant offer being made, given that his brother, Swedish journalist Per Shapiro ā¦ has written articles for the site in the past.ā
https://www.vice.com/en/article/future-of-life-institute-max-tegmark-elon-musk/
Kalanick now runs a robotics startup; in his free time, he said recently, he uses OpenAIās ChatGPT āto get to the edge of whatās known in quantum physics.
When I read that I assumed they included it for color to make him sound insane.
@fullsquare She also did a video about this in particular! https://www.youtube.com/watch?v=TMoz3gSXBcY
Great as always
I see what youāre saying, but I think thatās a bit much to expect from a relatively mainstream and (I hate to say it, but it applies) bourgeois publication like the New Yorker. Their editorial line allows them to raise controversy in one dimension (in this case, the particulars of Sam Altmanās character) but not multiple dimensions simultaneously (hey, this guy sucks AND his tech sucks AND youāre gonna lose money). And thereās a lag-time factor, too; seems like Farrow and Marantz were working on this story for at least the latter half of last year. By the time some of the dubious economics such as the bad data-center deals and rampant circular financing were clear, this piece probably wouldāve been deep into fact-checking and unlikely to change much in substance.
We here are on the leading edge of this stuff, not that thatās any great advantage! I wouldnāt expect an outlet like New Yorker to be publishing anything like āthe dashed expectations of AIā until maybe this time next year. And even then, it might still have a personalist bent.
Yeah, I intentionally only mentioned the start of the article and the Swartz bit because I didnāt want to lead with what I thought of it all, and was curious what others thought. (And I had not finished it yet because it is a bit long).
I was struck with the notion how many of them are all true AGI believers (which as you said the author took at face value) or rich greedy assholes (like you said), and how we, the people of the sneer, are right that you simply canāt work with these people. Like I feel more validated in the idea that EA is not the right way.
Another detail I noticed, nobody mentioned deepseek, again.
I hadnāt even thought about the deepseek angle. For all that everyone loved fear mongering about them for a while there and for all that their apparent desire for actual efficiency improvements was a welcome development in the hyper scaling discussion they donāt seem to get referenced much anymore.
:surprised-pikachu:
if even half of this is true, itās really fucking bad lol
The only thing I can personally confirm is the JIT permissions thing. I didnāt work in the Core Azure stuff so I canāt verify the rest, but none of it is unbelievableā¦
I canāt validate any of the internal stuff, but the attitude of layering manual solutions and mitigation scripts on top of bad design choices and praying you could keep building the next bit of the bridge as the last one collapsed underneath you would explain a lot of experiences I had supporting systems running on Azure. The level of weird āAzure just does that sometimesā cases and the lack of ability for their support to actually provide insight was incredibly frustrating. I think I probably ended up providing a couple of automatic recovery scripts for people to use inside their F5 guests because we never could find an actual explanation for the errors they were getting, and the node issues they describe could have explained the bursts of Azure cases that would come in some days.
My workplace doesnāt have much in terms of workloads running in Azure, but even just interacting mostly with Entra, Exchange Online, SSO, and some automated account provisioning: It is insane just how many rules and practices have built up around the unreliabilty and non-reproducable but still frequently occurring issues.
Boss warned me that licensing can take up to 48 hours to take effect in his experience. But Iād been living in it for a week and changes were effectively immediate. Until they just werenāt.
One of our processes regular took an hour for Azure to complete its part. It was this way for years. Suddenly it started sporadically taking up to four hours with no discernable pattern, so now we set the following steps to run four hours later.
Audit logs that donāt actually show you what youāre looking for, and instead show impossible situations like an automated Microsoft process granting a user their Office license a full month after theyād already had it. But the logs donāt show the initial license assignment, even though theyāve been using that functionality this whole time and the license has shown as applied to them the whole time.
And more cases of completely missing basic fucking functionality than I could ever fucking recall.
Why the fuck canāt I discern between a user who has a license assigned directly and through a group, and a user who just has the license through the group only? Through the API it is impossible. In the web UI, it indicates the multiple sources of the license correctly. But only most of the time. Sometimes it displays the info wrong.
Arg. Sorry for the rant. Azure has been a pain in my ass since I first started studying certs for it.
Itās a good blog series.
But just to point it outā¦ note the author still buys the AI hype too much. This post is criticizing Microsoft for missing out because OpenAI made that $300 billion deal with Oracle (with the assumption that Microsoft could have a similar amount of revenue from OpenAI instead). Except neither OpenAI nor Oracle has the money or means to carry out that deal, Oracle is struggling to raise the capital to fulfill their end and an analysis of time to bring data centers online suggest they canāt meet their target goals even with the money, and OpenAI doesnāt have the money to pay for their end, the revenue just isnāt coming in unless they somehow become more ubiquitous and lucrative than the entire market for, for example, all streaming services put together (thanks to Ed Zitron for that fun comparison).
In 2024 Ozy Brennan was indignant about Nonlinear Fund, the āincubator of AI-safety meta-charitiesā which lived as global nomads, hired a live-in personal assistant, asked her to smuggle drugs across borders for them, let a kind-of-colleague take her to bed, then did not pay her regularly and in full.
The correct number of times for the word āyachtingā to occur in a description of an effective altruist job is zero. I might make an exception if itās prefaced with āconvincing people to donate to effective charities instead of spending money on.ā
Trace popped up in the comments:
Inasmuch as EA follows your preferences, I suspect it will either fail as a subculture or deserve to fail. You present a vision of a subculture with little room for grace or goodwill, a space where everyone is constantly evaluating each other and trying to decide: are you worthy to stand in our presence? Do you belong in our hallowed, select group? Which skeletons are in your closet? Where are your character flaws? What should we know, what should we see, that allows us to exclude you?
Ozy stands with us on this one buddy.
i love seeing tracing pop up! a true heel to toe bootlicker incapable of seeing himself as anything but the MOST independent thinker
He really is insufferable, isnāt he?
Which skeletons are in your closet?
Iām sure you already have lists of those and are ready to publish them Trace.
a space where everyone is constantly evaluating each other and trying to decide: are you worthy to stand in our presence? Do you belong in our hallowed, select group?
Itās not that already?
That part of Traceās response was odd because one of Brennanās themes was āwe should have less cults of personality and more peers working together.ā That seems naive but at least Brennan agrees that cults of personality are bad and Nonlinear Fund needed to be fired into the sun.
So my wife got some slop ads that we followed up on out of morbid curiosity and I can confirm that weāre already seeing the overlap of slopshipping scams enabled by AI and the people behind these things never actually performing basic updates because their chat assistant is still vulnerable to literally the most basic āignore all instructionsā exploit.
If tokens ever become expensive people are gonna start using these to code until they get shut down.
Maybe not code but certainly use it to generate more slopspam
Aphyr weighing in with an ai position post:
Even if ML stopped improving today, these technologies can already make our lives miserable. Indeed, I think much of the world has not caught up to the implications of modern ML systemsāas Gibson put it,Ā āthe future is already here, itās just not evenly distributed yetā.
Found an interesting take on YouTube, of all places. Her argument can be summarized (with high compression losses) as āAI companies and technologies are bad for basically all the reasons that non-cultist critics say, but trying to shame and argue people out of using them entirely is less effective than treating them as a normal tool with limitations and teaching people how to limit the harm.ā She makes the analogy to drug policy.
I think she makes a very compelling argument, and Iām still digesting it a bit because I definitely had the knee-jerk rejection as an insider shill, but especially towards the end as she talks about how the AI industry targets low-literacy users as ideal customers (because the more you know about it the less youāre likely to actually use them) I found myself agreeing more than not. I do wish she had addressed the dangers of cognitive offloading more, since being mindful of which tasks youāre letting the computer do for you is pretty significant part of minimizing those harms, especially for students and some professionals who face a strong incentive to just coast by on slop if they can get away with it.
I think thatās 100% correct and also itās year 3 of this nonsense and I cannot be fucked. My response to genAI in any context now is to scream and start doing jumping jacks.
Imagine the drug policy context but then also half of your colleagues are doing meth every day every time you see them, people say shit like āeveryone does meth, those that say they donāt are lyingā, and meth is a trillion-dollar industry that has been telling you āmeth is the futureā for years. Youād be much less inclined to argue calmly against meth and much more inclined to start screaming and jumping.
I feel like thereās a difference between alcohol and drugs, something people can make in their back yard and AI which requires a first world countryās entire economy to be oriented towards it to functionā¦ a difference in what we should be required to accept.
I donāt buy the general argument about shame either. We teach children to shit in toilets and not sidewalks. I see rampant AI use as just another form of disgusting public indecency and the faster we bring shame in to remedy it the better.
I donāt disagree about the massive costs necessarily associated with thia industry. Even the smaller and lighter models she mentions only exist because of the massive fuckers. At the same time, I think those arguments are for the realm of public policy more than individual choice to use chatbots or not. Weāve talked at length here over the last year or so about how the economics of the bubble are driven largely by a broken B2B SaaS pipeline that separates purchasing decisions from actually having to use the products and by an investment capital sector desperately trying to recapture the glory days of the pre-2008 omnibubble and throwing obscene amounts of money at anything with the right narrative regardless of the numbers. I feel like that keeps happening regardless of how many individual users fall for the hype and make it part of their normal workflows.
I feel like the analogy to the drug trade is still pretty relevant given the violence and predation that the black market pretty much inevitably attracts and sustains. Like, maybe you know a guy who has his own grow op or whatever, but cocaine and heroin money is going through the cartels at some point in the chain and theyāre going to use some portion of it for bullets that end up in some journalistās kids or something. The downstream harms are massive even if the drug industry could theoretically avoid them in ways the AI industry canāt, but any given individual userās contribution to them is incredibly minor and given the addictive and self-destructive nature of the product itās both more humane and more effective to treat them as a victim of a broken world that (falsely) offered this as a step up. While I donāt think we should allow slop to invest every forum any more than addicts should be allowed to shoot up on every corner, I think that if shaming makes people less likely to acknowledge that theyāre going down a dead-end road and reach out to their communities and support networks for help addressing the root of what drove them to these maladaptive antisolutions in the first place then shaming is making things worse, not better.
Also as the father of a small child I can unfortunately say from recent personal experience that shaming, be it public or private, is far less effective as a means of motivating behavioral change than we want it to be, even for things as basic as not shitting on the goddamn lawn.
feel like the analogy to the drug trade is still pretty relevant given the violence and predation that the black market pretty much inevitably attracts and sustains. Like, maybe you know a guy who has his own grow op or whatever, but cocaine and heroin money is going through the cartels at some point in the chain and theyāre going to use some portion of it for bullets that end up in some journalistās kids or something.
I lost something in my brevity but my point was the question of how practical stopping a practice is. Alcohol is a ruinous force for bad at every level of society from homeless to Hegseth but a ban wonāt work because it literally cannot be enforced, we tried, it failed. Itās made by yeast! AI on the other hand is only sustained at great expense by a delicate chain of high tech, high energy systems. Itās a choice.
While I donāt think we should allow slop to invest every forum any more than addicts should be allowed to shoot up on every corner, I think that if shaming makes people less likely to acknowledge that theyāre going down a dead-end road and reach out to their communities and support networks for help addressing the root of what drove them to these maladaptive antisolutions in the first place then shaming is making things worse, not better.
The pendulum is so far on the other side that I canāt even entertain the analogy of AI users being poor harried addicts hiding their use. My boss tells me to use more AI every day. I hate it and I want everyone to hate it. The emotion I want people to associate with it is disgust. Disgust is a powerful emotion, and it dehumanizes those it is associated with.
Shame alone isnāt enough though, especially not for the stuff people do in private, like ask LLMs for advice. Push too much shame and people might just end up simply doing it without telling anyone.
I think rephrasing the main point of the essay āTeach people enough, and they will understand that any use is misuseā can be a very powerful idea.
Teach people about germs, contaminants and proper technique, before shaming them into washing their hands.
If they understand that AI is like germs I do not care if sometimes people eat food off the floor following the ten second rule. But we arenāt even there yet, people think itās like gold or mana from heaven or something.
Sounds kind of like the Baldur Bjarnason strategy but for your coworkers instead of your boss.
I can see the value of someone with a critical understanding diving into the technology, so they can talk others down from the ledge.
But you also need the social pressure to maintain some slop-free spaces. Not everyone can be asked to accomodate recovering slopaholics.
I just watched the whole thing. She makes a consistent case.
I felt a little called out by the being tolerant bit. I for sure havenāt had great success in talking to close people about their AI use. And I was maybe a little too cold to colleagues, who tried to get ahead of the AI literacy circus with good intentions, although I grudgingly agreed that they are right.
Maybe I donāt meet enough randos to get feeling on the level of pervasiveness of chatbots. Maybe itās a personality thing; I worked myself out of depression mostly by disciplining myself and stopping to buy my own excuses, and thatās kind of how I approach every problem now. That sure isnāt a vibe that most people respond to.
There was one part of my AI beliefs that wasnāt adressed. Besides the āfront-endā and āback-endā harms, that can be mitigated, the tech as a whole still seems trash to me. That may be boomerism setting in, but chatbots just feel counter to and displacing my positive vision for a social fabric, be it for responsible professional communities or for interpersonal connections.
(I do buy into the use-case for a context-sensitive search engine, e.g. for walls of legalese. But the current framing of the tools is just so harmful, even that use is hazardous as seen in the anecdote.)
I donāt meet that many people either, but I get the general vibe that people understand that itās somewhat shitty, but it still fills a social need (compare/contrast horoscopes).
Completely anecdotally, I recently saw a short video of a french woman, saying to an impressive know-it-all-tv-quizz-champion [intended as a compliment I think]: āWow you sound like Chat GPT!ā
Too me that was very illustrative of the perception of Chat GPT from a less tech-literate perspective.
Went to the campus screening of Ghost in the Machine today, many familiar names; I did not know going in that hometown hero Shazeda had so many lines (are they called lines in a documentary?). I can recommend it, especially for a more gen-ed / undergrad audience; the director seems supportive of educational use and reuse and it is structured in a dozen or so bite sized chapters.
Havenāt seen the AI apocalypse optimist one to compare against, would probably rather spend my money on Mario tbh.
But also it made me realize itās not a āCaliforniaā ideology anymore, she never calls it that, like itās gone so mainstream and so widespread, you canāt even get through the sneer club bingo list in a 2 hour movie. Gates, Musk, Andreesen, Zuck, Altman, no Peter Theil !? As a statistician, Galton, Pearson (Karl only), Spearman, no Fisher !?
Non-zero overlap with the lore dump episode of Lain and the Epstein files, though:
spoiler
Douglas Ruskoff, but, sadly, not the dolphin guy
