Which AI crawlers is your site blocking?
We read your robots.txt and tell you, for each AI crawler separately, whether it is allowed or blocked — and which line in the file decided it.
- No sign-up, no email
- One read of one file
- Check a specific path, not only the home page
Who each one is
This table stands on its own: the crawler name as it appears in the User-agent line, who operates it, and what for.
| User-agent | Operated by | What for |
|---|---|---|
GPTBot | OpenAI | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
OAI-SearchBot | OpenAI | IndexBuilds a search index, which AI answers are built on top of. |
ChatGPT-User | OpenAI | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
OAI-AdsBot | OpenAI | AdvertsChecks the landing pages of adverts. Blocking it affects neither answers nor the index. |
ClaudeBot | Anthropic | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
Claude-SearchBot | Anthropic | IndexBuilds a search index, which AI answers are built on top of. |
Claude-User | Anthropic | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
Claude-Agent | Anthropic | Agent in a browserA browser an AI agent drives for one person. Blocking it changes nothing a model knows — it stops your own visitors asking an assistant to do something on your site. |
Google-Extended | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. | |
GoogleOther | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. | |
Google-CloudVertexBot | IndexBuilds a search index, which AI answers are built on top of. | |
PerplexityBot | Perplexity | IndexBuilds a search index, which AI answers are built on top of. |
Perplexity-User | Perplexity | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
Applebot-Extended | Apple | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
Meta-ExternalAgent | Meta | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
Meta-WebIndexer | Meta | IndexBuilds a search index, which AI answers are built on top of. |
Meta-ExternalFetcher | Meta | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
Meta-ExternalAds | Meta | AdvertsChecks the landing pages of adverts. Blocking it affects neither answers nor the index. |
FacebookExternalHit | Meta | Link previewBuilds the title and image shown when someone shares the link. Blocking it leaves the share bare. |
Amazonbot | Amazon | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
CCBot | Common Crawl | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
AI2Bot | Allen Institute for AI | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
Diffbot | Diffbot | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
Googlebot | IndexBuilds a search index, which AI answers are built on top of. | |
Applebot | Apple | IndexBuilds a search index, which AI answers are built on top of. |
Bingbot | Microsoft | IndexBuilds a search index, which AI answers are built on top of. |
Bytespider † | ByteDance | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
cohere-ai † | Cohere | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
MistralAI-User † | Mistral AI | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
Timpibot † | Timpi | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
omgili † | Webz.io | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
† No operator page documents this token. It is listed because it appears in server logs, and kept apart from the rows that rest on an operator’s own documentation.
How many sites actually block them
On the date below we read the robots.txt of the highest-ranked sites in the markets listed here, and matched them against the table above.
Of the sites that block GPTBot, 66.5% also block an AI crawler that would have brought them visitors.
These are large, high-ranked sites, not small businesses. Among small businesses the picture is probably different, with fewer deliberate blocks and more neglect. Do not read these figures as a picture of small businesses.
| Market | Block an AI crawler that brings visitors |
|---|---|
.nl | 31.5% |
.com | 29.7% |
.fr | 25.8% |
.it | 16.2% |
.es | 15% |
.jp | 13.7% |
.de | 12.3% |
Percentages are of 528 robots.txt files actually read, not of the domains requested.
Brazil and Portugal were not in the sample. The .com figure is the global English-language slice and is not a figure for either country.
Measured on
The whole measurement, as open data: JSON · CSV
We read them again on 2026-11-09, keeping every domain this time. Of the sites that block GPTBot, 80.3% wrote a rule that names it, so the block was a decision. Of the sites that block OAI-SearchBot — the crawler behind the answers people are shown — 43.6% never chose it: the block came from a rule aimed at every bot alike.
That second reading covers 407 readable files out of 700 domains requested. 145 hosts refused this reader outright, and a host that refuses an unknown automated reader is plausibly a host that restricts AI crawlers, so this reading probably under-states blocking rather than over-stating it.
By country suffix
The same reading, cut by suffix. Every share in a row is of the files read in that suffix, not of the hundred domains requested.
| Market | Files read | Block GPTBot | Block OAI-SearchBot |
|---|---|---|---|
.it | 65 / 100 | 15 (23.1%)15 of these deliberate | 5 (7.7%)0 of these inherited |
.es | 64 / 100 | 11 (17.2%)9 of these deliberate | 5 (7.8%)2 of these inherited |
.fr | 59 / 100 | 7 (11.9%)6 of these deliberate | 4 (6.8%)1 of these inherited |
.nl | 59 / 100 | 10 (16.9%)8 of these deliberate | 4 (6.8%)2 of these inherited |
.com | 56 / 100 | 17 (30.4%)10 of these deliberate | 12 (21.4%)9 of these inherited |
.de | 55 / 100 | 8 (14.5%)7 of these deliberate | 4 (7.3%)1 of these inherited |
.jp | 49 / 100 | 8 (16.3%)6 of these deliberate | 5 (10.2%)2 of these inherited |
Every domain, with its verdict, the rule group that decided it and the exact line: JSON · CSV
What this check cannot tell you
- robots.txt is a request, not a wall. A crawler can ignore it.
- A firewall or CDN can block a crawler that robots.txt allows.
- The check describes one moment. Editing the file changes the answer at once.
- Blocking a training crawler and blocking a live-answer crawler are two entirely different decisions.
See whether AI actually mentions you
Access is only the first step. Go Top SEO checks automatically whether your business shows up in answers from ChatGPT, Gemini and Google AI, and tracks your Google and Maps rankings every month.
Try it free for 7 days, no card