Well, the Adpocalypse has happened.
So? Google will just do it anyway. There’s no consequences for them.
Google: Muahaha, we train our AI with the data from the publishers’ sites while indexing them. Then we summarize their site so the user never leaves our site and all the ad revenue is ours, brilliant!
Publishers: People aren’t coming to our site from Google search because it just summarizes our site for them. So we lose no traffic by completely blocking Google from crawling us.
Google: *Surprised Pikachu face*
AI-generated image by Mark Stenberg via Gemini
The irony of the subject matters being about how bad the impact AI/LLM is
It’s the easiest graphic to make too. You can make it with fucking Chrome Dev Tools
Slop is everywhere, sadly. 😟
Finally. Time to end SEO and ban the crawlers and add tar pits and other poisoning https://rnsaffn.com/poison3/
Oh shit, that’s just begging to be combined with something like Nepenthes!
Looks like it’s offline?
:(
But then why URL say online
I’ve opted out of search (not just google, most other commercial search) in ~2024. Have not regretted it since, happy to see even commercial entities coming to the conclusion that Google’s garbage and not worth it anymore.
Now, if they’d also follow the past set by indies, and block AI crawlers too, and make that the norm, that would be grand.
While that would be great, you can’t easily block AI crawlers these days. The vast majority of crawler traffic we’ve been seeing in the past year has been random bullshit from residential proxies. Less than a third of our crawler traffic self identifies anymore. What’s super annoying is that the user agents they use are so ludicrously fake. From my random samplng they frequently claim to be from a Wii using an Opera browser or a Nexus 5. I don’t know why those seem so prevalent other than to troll us.
Uh, I beg to differ. I’ve been blocking most of them for the past year, using essentially three ifs in a trenchcoat.
Bullshit user agents are taken care of by checking headers other than
user-agent: if they say they’reChrome/orFirefox/, check if they sentsec-fetch-mode. Didn’t? That’s very likely a crawler (and the handful of false positives are easy to make an exception for). For residential proxies, the same applies. For crawlers that piggy-back on Chrome, they usually crawl an URL queue, so if you poison their queue, you can catch those too.At this point, out of ~100 million requests / day, I’m firewalling ~98 million off. Out of the remaining 2 million, ~90% of them gets served garbage to continue poisoning the URL queues. I can serve the rest on a potato, even if some of them are crawlers.
I did have a few people contact me about false positives, but those were very, very few (and also very easy to address). Very little CPU, RAM or bandwidth required, the vast majority of bots caught, negligible false positives. Deploying the solution isn’t trivial (yet), but it also isn’t hard either.
Let’s see them try to opt out of AI training.
It’s not that complicated - block all bot traffic by default, whitelist the bots you can verify are only indexing for search (which is actually beneficial traffic).
The thing that Google’s doing to piss off websites is that they use the same bots for indexing and AI trawling, so if you want to show up on their indexes you have to let them slam your website/server to train their AI.
Now websites are telling Google to fuck off because the cost of AI trawling isn’t worth being on their search index
It’s not that complicated - block all bot traffic by default,
It’s not that easy to identify bot traffic. AI companies have been incredibly persistent and aggressive at pretending to be humans to scrape sites.
it’s not worth the cost because once google has your information, users won’t need to visit your website.
the question of "who is going to be creating all this high-quality data not only for no money but also for no recognition?‘’ goes unasked
