That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a trace for damages.
When even OpenAI is more respectful of intellectual property and website owner control than you, that's a problme.
They have three different user agent strings, anyway. The problem is obviously that you can't tell what someone does with data after you give it to them.
You also don't know that some third party isn't crawling the web with the user agent string "OAI-SearchBot" and then using the results for training or selling the data to the likes of Anthropic or OpenAI without telling them that.
In general attempting to use the user agent string for access control is not going to work and it seems like Google is being the less disingenuous party on this one by not blowing smoke.
> Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. https://developers.google.com/crawling/docs/crawlers-fetcher...
Exclusion from grounding does mean that your site won't get sourced in the AI overview, but I'm not sure what the click through rates are like on those.
There are of course plenty of mystery, obfuscated/camouflaged scrapers/crawlers from god knows whom. Thankfully they are easy to spot and ban, although I've definitely thought about deliberating sending them poisoned data.
Saying this as a European who is pro EU.
Why should they?
How is that not conflict of interest??
Huh?
Search and AI are hand-in-hand.
They both rely on embeddings. (Unless you still do keyword-only search, but that's not as good.)
that’s just my take/feedback, take it or leave it. i won’t be engaging further as i already feel i’m going against the site guidelines with this!
This is unfortunately nothing new. There's no correct way to tell them to fuck off, they do not care, and they never will do. People have even taken them to court over this
Also I should note there are lots of fake Googlebots...
Akamai, Fastly, etc only take big customers who know what they're doing. You need to sign a proper contract with them. They aren't low-friction.
It's kind of exhausting seeing Cloudflare playing both sides of the arms race.
I just can't imagine bringing myself to use their technology to build agents and build AI products when they're also doing things like this.
> This also lines up the incentive model we want to foster. Losing trusted status across the more than 20% of web domains that sit behind Cloudflare is a deterrent with teeth. Trust becomes something you can carry with you, and something you can lose.
And even more so, LLM language aside, fun and fascinating to see them flagrantly calling out their position here as if it's a positive.
This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled with the trust stuff. Of course, Cloudflare is always going to trust their own platform.
Second one is problem because it DDoSes the site. "Just" queries for hundred thousand people (if you happened to be good source for that bit of knowledge) that don't bring actual people to your site is also a problem
"Block on pages with ads" is probably about preventing the AI crawlers from clicking on the ads which maybe considered cheating by the ad company.
If you want to prevent "bot attacks", maybe the "Block" option will do the trick.
But of course, to do all that you need to put some trust on Cloudflare, because they're the one identifying the bots from normal users.
For me, as someone who's hosting a Gitea instance behind Cloudflare, I have a Configuration Rule set that says: if the client is trying to access a URL that is beyond certain length limit, then trigger "Browser Integrity Check" and "I’m Under Attack", a.k.a stricter security checks.
The match expression of the rule looked something like this:
(
len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and
not lower(http.request.uri.path) contains ".git/" and
not lower(http.request.uri.path) contains "api/"
)
This rule alone basically blocked all abusive bot traffic to almost zero for my site (https://i.imgur.com/LaOjjvV.png, see the traffic drop around 10 clock and Cloudflare mitigation kicks in).
But of course, you need to figure out your own rules based on the characteristic of the website. Also, you can be more creative: for example, my actual rule is more complex than that, it also checks to see if a cookie is not set, and only triggers when all condition are met:
(
len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and
not lower(http.request.uri.path) contains ".git/" and
not lower(http.request.uri.path) contains "api/" and
not http.cookie wildcard "*!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!*"
)
(
len(http.request.uri) <= !!!!!SET LENGTH LIMIT!!!!! and
not http.cookie contains "!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!"
)
(Note: `!!!!!COOKIE NAME!!!!!` and `!!!!!COOKIE VALUE!!!!!` are the variables you need to customize)
When you put the two rules together, it forces clients to access "shallow" (short URL) pages first as an user would normally do, before they can access "deeper" (long URL) content hosted on the site without triggering more strict security checks. If that makes sense.
Also, don't forget the cookie basically also dug a hole in the security setting. So it's really a balance between avoid annoying the user and protect your site. You need to be smart and be flexible about it, otherwise your users will just leave.
This morning, for the first time, I clicked on a link posted on HN and was told in no uncertain terms that I am a bot and will not be allowed to view this page. By Cloudflare.
Your strategy is fine and similar to one of the checks in go-away. The point is that the unknown global DDoS adversary is using a very simple scraper, which does not load images or CSS or scripts, does not set cookies, etc
I've never understood why that isn't more common honestly. It would be cool if I could actually transfer POW from one domain to another as well. Probably wouldn't stop scrapers, but it would likely stop low margin fraud type things.
Oh, no, that isn't how it works — you made a claim first (albeit indirectly), with no evidence. It's on you
Still, I'll humor your absurd request by reminding you of the many success stories that have repeatedly made the front page of HN.
Beyond that we have cryptocurrencies. If you have knowledge of a generalized solution for defeating PoW schemes then why are you posting here instead of making yourself a billionaire?
Please propose an alternative to both Cloudflare and Anubis, that shields websites against inhuman traffic without frequent operator intervention (or otherwise negates the capacity costs they pay for AI crawling) and is compatible with low-end smartphones.
Certainly, I imagine Anubis would be interested in adopting it if it’s effective!
Doing nothing doesn't work. Doing Cloudflare isn't acceptable. Doing Anubis makes the Internet unusable "for half the planet". I'm on your side morally, but site operators can't afford to be idealists in the face of AI crawling bills.
What do you, does anyone, suggest that hasn't been tried, or adopted, or considered? What's left that will defend operators against AI's traffic flood that isn't eating a thousand doller crawler bill, paying Cloudflare, or locking out half the planet?
However another option is to support both. Have a challenge page that requires the visitor to select one of several options. Cloudflare could be one of those.
???
Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.
You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge.
> If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.
Cloudflare blocks legitimate users itself sometimes [1].
[0]: Unless you run into a bug, but Anubis is open source, so you can always submit a patch upstream. I've done this myself, and I can confirm that it's relatively straightforward.
That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap.
>Cloudflare blocks legitimate users itself sometimes [1].
I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
Yeah, I really have no idea why Anubis works right now: residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs, but they give up on even short-ish Anubis challenges.
> I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
Yeah, I don't like the Cloudflare challenges, but in the past 5 years I've only had it outright block me once, and that fixed itself after 15 minutes. And I use Firefox on Linux with various privacy extensions, so my browser probably appears at least moderately suspicious.
Whereas I've been trapped in impossible ReCaptcha loops quite a few times, which is still better than vague error messages that magically go away when I switch to something not running Linux. So I'll begrudgingly accept that Turnstile is the least user-hostile product on the market right now.
On the contrary I only want to visit a few pages on a given site and have an entire laptop at my disposal. Meanwhile for bots efficiency is key. A serious scraper (ie the type of actor that actually causes material problems for site operators) is performing tens or hundreds of pages loads per second per core spread across thousands of sites. Making a single page load take even half a second of cpu time is a massive win for the site operator.
Why not go all the way and mine monero instead of just completely wasting the work?
The only difference is the cycles are getting converted into heat now. They could be getting converted into monero instead. It's still heat but at least creator got some money for it.
Anubis only work(s|ed) great for a while when crawlers were not prepared for these challenges. Security through obscurity.
The proxies I am familiar with do not offer arbitrary code execution. I think you're thinking of a botnet.
Regarding native code, the current crop of solutions seem to work well enough for now. Ultimately a challenge response protocol should be standardized and browsers should ship a native implementation. In the meantime WASM likely gets you close enough to native.
I just quoted a toot in another submission, adding it here since it is relevant:
We apologize for a period of extreme slowness today. The army of AI crawlers just leveled up and hit us very badly. [...] It seems like the AI crawlers learned how to solve the Anubis challenges. [...] However, we can confirm that at least Huawei networks now send the challenge responses and they actually do seem to take a few seconds to actually compute the answers. It looks plausible, so we assume that AI crawlers leveled up their computing power to emulate more of real browser behaviour to bypass the diversity of challenges that platform enabled to avoid the bot army.
The reasoning behind this is also flawed: blocking "bots" and "AI" means that our AI agents working for us are unable to do their work for us, because of knee-jerk bot-blocks.
What would force their hand?
It’s more likely they’ll strike undisclosed agreements with major sources of discussion like reddit etc.
That’s not to say getting new information as a way of context-providing is not going to happen but that’s not scraping.
Something has to happen or Google will end up starved and locked out of everything, by means both technical and legal. Then nobody gets anything.
I don't have the answer as to what happens next, and I doubt anyone else who proclaims one super confidently. But we can do some constraints analysis. There is no world where everyone works for free so Google and other AI engines can get all the value from the content, so we can eliminate those possibilities. I think we can safely discard the world(s) in which all content production just stops. However, off the top of my head, it's hard to get much tighter than that, and that definitely leaves a world where effectively everything everywhere ends up going pay-to-access.
Microtransactions have, to date, failed comprehensively, though, so the constraints on what "everything is pay-to-access" gets weird without them.
And there is never guarantee that there is any solution to any set of constraints. Things can end up overconstrained in reality as easily as a math problem. I don't actually think it'll go that way, but when analyzing this question I think it's important to not let "but $SOMETHING just has to have some way to work, because... uh... it has to!" Let the constraints do the talking. You could end up with a scenario where all content of any value is locked down, and it's fundamentally difficult and expensive to ever access or discover it, and consequently the entire content production industry radically contracts compared to its current size, if there is no pragmatic solution to microtransactions that is low-enough friction to get over the psychological and economic hurdles that have killed it to date. If everything is locked behind "macrotransactions" that's a much smaller commercial web. Probably a much higher quality one, too, but at a pretty stiff cost.
One possible scenario is that that is simply it for the internet as an information source; the search engine's AIs get captured and there becomes effectively no way to discover any of the content already on there.
But then again, people will react to that and do something. Kagi would grow and others too. The more interesting question is whether the governments that captured Google's AI would let them or if suddenly it would be discovered that copyright law doesn't permit search engines to do that. Would that be inconsistent with letting the Approved AIs access whatever they want and chew on it even harder? Yeah, and they wouldn't care.
You seem to misunderstand. You pay or you die. There's nothing in between. Cloudflare will happily collect the tax. As does Apple (collecting 20B$ yearly from Google for the "tax"). Cloudflare's users also won't mind about how the company handles ADOG as long as they get a chunk of the cake by getting freebies and cheap services.
The usage patterns of how people pay and use AI is basically the same model the web should be using: you pay a small bit of money to access monetized pages, just how you pay a small bit of money to get AI responses.
It just needs people and browsers to get onboard with protocols. Crawlers will have no choice but to pay for content behind these 402 gateways.
As far as I can tell, Google is the only one sending me visitors. And the other big AI players might do so in the future.
Another option would be "No anonymous bots". So at least if a bot would want to crawl my site, they would have to identify themselves. Since the rise of the AI bots, I am getting hurt badly with insane amounts of requests from residential IPs that mimic real humans. The only difference being they don't make me any money. Only produce costs.
By the way, how is the situation over at Amazon's Cloudfront? Do they offer something that helps? Anyone here with them?
I dont get this. The question is are they a bot or a human. It doesnt matter what they are doing I dont want bots on my site.
Do you want your site to be discoverable by a search engine? (How do you think that occurs?)
Where do we draw the line between a personal-bot and a custom browser?
That being said, the issue with it right now is that Google uses the same crawler for both AI training and search, so this step puts a small pressure on google nonetheless to hopefully split them.
I want pay-per-crawl. I clicked the link for it a year ago, got presented with a "request access" button, I "requested access" and obviously since I'm nobody I heard absolutely nothing. Now they're touting the link again, I checked, still that same "request access" button. I have no idea if anyone even has access to this feature.
I don't care about all this other stuff, I want the AI crawlers to pay me cash. Because boy do those fuckers want to crawl me. I'll gladly double the size of my gerrit/jenkins servers to keep up with the load if these stupid bots want to pay to crawl every jenkins build artifact and every changeset source file on the server, as they really seem to want to do.
There is no technical mechanism whereby it is actually possible to allow people to read your webpage and not use it for other things. You can’t give responses that say “this is ok for indexing but not for training”. Anyone trying to sell you this sort of technology is lying.
> Another change that will apply on September 15 is that multi-purpose crawlers (specifically those that combine Search with Training) will be allowed/blocked according to all of their behaviors, in line with our call for transparency for website owners. Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).