Robots.txt Checker
See which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — your robots.txt lets in, with the exact rule that decides each one.
AI crawler access
Which AI crawlers can access your site?
What this robots.txt checker tests
It fetches your robots.txt, validates the syntax, and reports — for each of the AI crawlers that matter — whether your site lets them in: GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini training), CCBot (Common Crawl), Applebot-Extended and Meta-ExternalAgent. Each verdict shows the exact rule that decided it, so nothing is a mystery.
Rule precedence, done correctly
Most robots.txt tools get precedence wrong. The real rules: a crawler uses the most specific User-agent group that matches it and ignores * once a specific group exists; within a group, the longest matching path wins, not the first; and when an Allow and a Disallow match equally, Allow wins. That means Allow: /blog/ genuinely can punch a hole through a broader Disallow: / — and a specific User-agent: ClaudeBot group with Allow: / overrides a blanket block. This checker applies those semantics exactly.
Should you block AI crawlers?
It’s a real trade-off. Blocking training crawlers like Google-Extended or CCBot keeps your content out of model training but has no effect on search rankings. Blocking retrieval crawlers like ChatGPT-User or PerplexityBot means AI assistants can’t cite you when users ask questions your site answers — which is rapidly becoming a real traffic source. Whatever you choose, make it deliberate: an accidental Disallow: / under User-agent: * blocks everything, including the crawlers sending you customers.
Fixing what this finds
Rules before any User-agent line are ignored entirely — move them under a group. Unknown directives are silently skipped, so typos like Dissallow fail open. And if you have no robots.txt at all, everything is allowed — a valid choice, but make it on purpose.
More free tools
- Site Answerability Check See which visitor questions your website can answer.
- Sitemap Validator Check your XML sitemap against the protocol.
- Sitemap Extractor Pull every URL out of any sitemap.
- Sitemap Checker Find every sitemap a site exposes and whether they validate.
- LLMS.txt Checker Check your llms.txt follows the standard, with AI suggestions.
- LLMS.txt Generator Generate an llms.txt file from your sitemap.
Related guides
Crawling and discovery
Your robots.txt file is where crawlers start. These guides cover what happens next.
- How to find the sitemap of a website — robots.txt is the reliable way to locate one
- How to check a sitemap — including pages blocked in robots.txt but listed in the sitemap
- Sitemap for WordPress — where WordPress puts yours
- Answerability check — what an AI can answer from what it is allowed to crawl