Should Your Website Block AI Crawlers?

If you run a website, there’s a decent chance someone has asked you lately whether you’re “blocking the AI bots”. It sounds like a yes or no question, but it isn’t. Some of these user agents are crawlers that actually fetch your pages, some are just labels that change how another company uses pages it already fetched, and robots.txt itself was never a lock on anything.

I’m going to stick to what the crawler operators and standards people have published themselves, and then give you my take on how to decide.

What robots.txt is, and what it isn’t

The Robots Exclusion Protocol finally got a proper spec in September 2022 as RFC 9309. The file lives at /robots.txt in the top level of the host, and it’s made of groups: one or more User-agent lines naming a product token, followed by Allow and Disallow rules. Crawlers match the token case insensitively and follow the most specific path rule.

The important word in the RFC is “requested”. The rules are things crawlers are requested to honor, and the spec says plainly that they are not a form of access authorization. Its security section goes further and points out that listing paths in robots.txt makes them public, so if something must stay private you need real access control such as HTTP authentication. A well behaved crawler reads the file and obeys it. A badly behaved one reads it and doesn’t, or never reads it at all.

There’s a matching detail that trips people up. Under RFC 9309, a crawler that finds a group naming its own token follows that group and ignores the User-agent: * group entirely. So if your * group disallows /admin/ and you add a separate group that only says Disallow: /private/ for some bot, that bot is now allowed into /admin/. When you add AI tokens, repeat any rules you still care about.

Crawlers that fetch versus tokens that don’t

This is the distinction that matters most, and it’s easy to miss.

OpenAI’s crawler documentation describes GPTBot as a real crawler that collects pages which may be used to train its models. As of this writing, OpenAI’s GPTBot page also says it runs a separate agent, ChatGPT-User, for fetching pages when a ChatGPT user asks a question through browsing, and that its opt-out system treats the two the same. A disallow for either one covers both.

Common Crawl’s CCBot is also a real crawler. Common Crawl is a nonprofit that publishes a free, openly downloadable copy of the web for research and analysis, and its CCBot page gives the two line robots.txt group that stops it crawling your site. Because that dataset is open, anyone can build on it, so blocking CCBot is really a decision about every downstream user of the archive, not just one company.

Anthropic has a help center article titled Does Anthropic crawl data from the web, and how can site owners block the crawler?, which names ClaudeBot as the token for its crawler and asks you to add the disallow on every subdomain you want to opt out. It also warns that blocking its IP addresses instead may not work reliably, because that stops the crawler from reading your robots.txt. The version I read also said the crawler honors disallows aimed at Common Crawl’s CCBot, which is a nice touch if you already block that one.

Google-Extended and Applebot-Extended are different animals. Google’s common crawlers page describes Google-Extended as a standalone product token that doesn’t impact whether your site is included in Google Search or how it ranks. It has no user agent string of its own. Google crawls with its existing user agents and only reads the token from robots.txt as a control. When I read the page this month, it tied the token to improving Gemini Apps and the Vertex AI generative APIs, including future generations of the models behind them.

Apple did the same thing in June. Its About Applebot support page says Applebot-Extended doesn’t crawl webpages, that pages disallowing it can still show up in search results, and that it only decides whether content Applebot already crawled can be used to train Apple’s foundation models. Apple’s page also has a quirk worth knowing: if your robots.txt doesn’t mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules.

So blocking GPTBot, CCBot or ClaudeBot means fewer requests hitting your server. Blocking Google-Extended or Applebot-Extended doesn’t change your traffic at all, it changes what the company is allowed to do with what its regular crawlers already collect.

Don’t block the search crawler by accident

The flip side is that Googlebot and Applebot are the search crawlers. Google calls Googlebot its main crawler for Google Search, so disallowing it asks Search itself to stay away, not just Gemini training. Apple says allowing Applebot is what lets your content appear in search results in Spotlight, Siri and Safari. A “block every bot” snippet that includes either of these is a much bigger decision than the AI question it’s meant to answer. If the goal is “no training, keep search”, the Extended tokens exist precisely so you don’t have to choose.

Cloudflare’s one click option

On July 3, Cloudflare announced a new one click toggle to block AI bots, available on every plan including free. You turn on the setting labeled AI Scrapers and Crawlers under Security, then Bots, in the dashboard. Cloudflare says it will keep updating the rule as it fingerprints new bots scraping the web for model training.

The interesting part isn’t the button, it’s why it exists. Cloudflare’s post makes the same point the RFC does: robots.txt only works if the bot operator respects it and identifies itself honestly, and user agents are trivial to change. Cloudflare says its machine learning bot detection still flags crawlers that pretend to be ordinary browsers. It also shared a telling number. In June, AI bots accessed around 39% of the top one million sites on its network, but only 2.98% of those sites blocked or challenged them.

That makes the toggle an actual enforcement layer rather than a polite request, which is its big advantage. The tradeoff is that you’re trusting Cloudflare’s definition of what counts as an AI scraper, and the list will change without you editing anything. If you have specific bots you want to allow, robots.txt gives you finer, more visible control.

Training versus showing up in answers

Here’s the real decision. There are two things an AI company can do with your page. It can train on it, which bakes some statistical impression of your writing into a model with no link back to you. Or it can fetch it at question time and use it to answer someone, ideally with a citation.

Blocking training is the easier call for a lot of sites. If your content is your product, such as original reporting, tutorials you sell, or a niche reference you spent years building, you may reasonably not want it feeding a model that competes with you for the same readers.

Blocking answers is a harder call, because that’s the part that can send readers back to you. OpenAI’s ChatGPT-User page says answers built from browsing include a link to the source. But given how OpenAI treated the two agents this month, you can’t currently block GPTBot and still allow ChatGPT’s browsing to fetch your pages. Google and Apple have drawn the line differently: their Extended tokens leave search alone.

One more expectation to set. Every one of these controls is about what happens next. The docs talk about crawling going forward and about future models, and none of the pages I read say anything about pulling out content that was already collected.

A robots.txt you can start from

If your answer is “no training, keep search”, this is the shape I’d use. Each token below comes from its operator’s own documentation, and each bot gets its own group, which is how every vendor’s examples are written.

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Remember the matching rule from earlier: the named bots won’t read the * group, so if you later add rules under *, copy the ones that matter into the named groups too. Also remember that the RFC asks crawlers not to use a cached copy for more than 24 hours, so changes can take up to a day to land, and that robots.txt is per host, so blog.example.com needs its own.

My take

For most sites I’d block the training tokens if you care about how your work gets used, leave Googlebot and Applebot alone, and accept that this is a request, one that each of these companies documents and tells you how to use. If you’re seeing heavy scraping from bots that don’t identify themselves, robots.txt won’t help, and that’s where Cloudflare’s toggle or your own bot rules earn their keep. And if you’d actually like to be cited in AI answers, it’s fine to allow all of it. The worst option is copying a giant blocklist without knowing which lines cost you search traffic.