Block AI crawlers in robots.txt
Short answer. Add one group for each crawler, with its User-agent token and Disallow: /. RFC 9309 says that crawlers are requested to honor these rules and that the rules are not a form of access authorization. The table links the page of each operator. OpenAI and Perplexity state that fetches a user asks for can pass over robots.txt.
Updated 2026-11-08
By Michael Lip. The sources on this page were read on 8 November 2026.
Select crawlers by operator and purpose, and select the whole site or only some paths. The generator writes one group per crawler. If you paste your current file, the merge check warns when a new named group stops a crawler from following your * group. Everything runs in your browser.
Crawlers to disallow
The tool runs in your browser. Its script makes no network request. It does not test a live server, and it cannot verify what a crawler operator does.
The crawler tokens, each with the operator's own page
Each row names a token that is on the page of its operator. The fourth column quotes that page. The list covers the operators whose pages this site read on 8 November 2026, and other operators exist.
| Operator | Token | Purpose | The operator's own words | Source (read 2026-11-08) |
|---|---|---|---|---|
| OpenAI | GPTBot | Model training | Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models. | OpenAI page |
| OpenAI | OAI-SearchBot | Search listing | Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. | OpenAI page |
| OpenAI | ChatGPT-User | User-requested fetch | Because these actions are initiated by a user, robots.txt rules may not apply. | OpenAI page |
| Anthropic | ClaudeBot | Model training | When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets. | Anthropic page |
| Anthropic | Claude-SearchBot | Search listing | Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results. | Anthropic page |
| Anthropic | Claude-User | User-requested fetch | Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search. | Anthropic page |
Google-Extended | Model training and grounding | Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. | Google page | |
| Apple | Applebot-Extended | Model training | Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results. | Apple page |
| Perplexity | PerplexityBot | Search listing | PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models. | Perplexity page |
| Perplexity | Perplexity-User | User-requested fetch | Since a user requested the fetch, this fetcher generally ignores robots.txt rules. | Perplexity page |
| Common Crawl | CCBot | Open web crawl | To prevent Common Crawl from crawling your website, include the following in your robots.txt | Common Crawl page |
Google states that Google-Extended "doesn't have a separate HTTP request user agent string". Google states that publishers can use the token to manage whether content that Google crawls may be used for training future generations of Gemini models and for grounding. Apple states that Applebot-Extended "does not crawl webpages".
The three robots.txt rules that decide which group a crawler reads
- A group is one or more
User-agentlines and then its rules. The generator writes one named group for each crawler. - Token matching is not case-sensitive. RFC 9309 states that "Crawlers MUST use case-insensitive matching to find the group that matches the product token". Google states that "Both the user-agent field name and its value are case-insensitive."
- A crawler with a matching named group does not follow the
User-agent: *group. RFC 9309 sends a crawler to the*group only "If no matching group exists". Google states that "User agent specific groups and global groups (*) are not combined." Two named groups for one crawler do combine.
Sources read on 2026-11-08 are RFC 9309 and the Google robots.txt specification.
What the generator and the check report
- For each token in the table, the check reports whether your file disallows the whole site, some paths or no path, and whether that rule comes from a named group or from the
*group. - Named groups with a token that is not in the table.
- The generator writes one group for each crawler that you select, for the whole site or for your paths.
- The merge check warns when a new named group stops a crawler from following your
*group, and copies the*rules into a group for some paths. It also warns when a group for that crawler already exists, and when an Allow rule still wins over a new Disallow rule.
The tool reads your text only. It cannot tell you whether an operator follows robots.txt. The table quotes each operator's own statement.
What robots.txt cannot do
robots.txt is a request. RFC 9309 says that crawlers "are requested to honor" these rules and that "These rules are not a form of access authorization." OpenAI states about ChatGPT-User that "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity states about Perplexity-User that "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." To keep content private, serve it behind authentication. This page does not offer legal advice.
The robots.txt tester checks whether one exact URL is allowed or blocked by a file you paste, using Google's group-selection rules.