robots.txt for AI crawlers: who to allow, and how to check

How robots.txt is read

robots.txt is a plain text file at the root of a site. It is a request, not a lock: well-behaved crawlers obey it, and it does nothing to a crawler that ignores it. It is organised in groups.

• A group starts with one or more User-agent lines naming the crawlers it applies to; * means all.
• Disallow lines list paths the group may not fetch. Allow lines list exceptions.
• A crawler follows the group that names it most specifically, and only that one. A group for GPTBot overrides the * group for GPTBot; it does not add to it.
• When Allow and Disallow both match, the more specific (longer) path wins.

That second-to-last rule causes most accidents: a site with a strict * group and no group for an AI crawler blocks that crawler, and a site with a friendly * group and one careless group for a single bot blocks that bot completely.

Which crawlers matter

Vendors run several crawlers each, for different jobs. The names below are correct at the time of writing; check each vendor’s own documentation, because they change.

User-requested fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page because a person asked for it. Some vendors document that these do not apply robots.txt the same way as their training and search crawlers — check each vendor’s documentation before relying on robots.txt to keep them out.

Blocking training use while allowing search use is a legitimate choice, and some businesses make it. The mistake is doing it by accident: a blanket rule that blocks every unfamiliar bot also blocks the ones that decide whether you are named.

A working example

User-agent: *
Disallow: /admin/
Disallow: /account/

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /account/
Allow: /

Sitemap: https://example.co.uk/sitemap.xml

Here the AI crawlers may read everything except the two private paths. Notice that the Disallow lines are repeated inside their group: a crawler follows only the group that names it, so the * group no longer applies to them, and without the repeat they would be allowed into /admin/ and /account/.

How to check in a minute

• Open yourdomain/robots.txt in a browser. Check the file exists and is not an HTML error page.
• Look for a group that names GPTBot, ClaudeBot, PerplexityBot or Googlebot with Disallow: / — that is a full block.
• If there is no group naming them, read the * group: Disallow: / there blocks every crawler you have not named.
• Check the sitemap line points to a sitemap that loads.

The free audit does this for you: its technical readiness table tests the main AI crawlers against your file and marks each Pass or Fail, along with the sitemap and llms.txt. Our own crawler obeys robots.txt, so if you block it, the report says so rather than working around it.

Mistakes that remove a business quietly

• A staging robots.txt (Disallow: /) that went live with the site.
• A security or bot-management setting that blocks unfamiliar user agents at the network edge. The robots.txt looks fine; the crawler still gets an error.
• Blocking a path that holds your prices, services or team pages.
• A robots.txt that is unreachable or slow, which some crawlers treat as a reason to skip the site.