Free Tool
Is your site blocking AI search crawlers?
Check your robots.txt in seconds. See exactly which AI bots — GPTBot, ClaudeBot, Google-Extended — can access your site, and which you're accidentally blocking.
What this tool checks
- ✓Whether GPTBot (ChatGPT) can crawl your site
- ✓Whether ClaudeBot (Anthropic) has access
- ✓Whether Google-Extended (Google AI) is allowed or blocked
- ✓Whether PerplexityBot can index your pages
- ✓Whether CCBot (Common Crawl, used by many AI models) is permitted
- ✓Whether Bytespider (ByteDance/TikTok AI) is blocked
- ✓Your wildcard (*) rule and how it affects all crawlers
- ✓Any Disallow paths that affect AI visibility
See how we've helped UK businesses get found. View case studies →
Frequently asked questions
- What is a robots.txt file?
- A robots.txt file sits at the root of your website (e.g. yourdomain.com/robots.txt) and tells search engine crawlers and AI bots which pages they can and cannot access. It's a standard protocol followed by Google, Bing, ChatGPT, and most other major crawlers.
- Why does it matter which AI bots can access my site?
- AI tools like ChatGPT, Perplexity, and Google AI Overviews learn from web content. If your robots.txt blocks their crawlers, you're invisible in AI-generated answers — even if you rank well on Google. As AI search grows, being cited in AI answers is a significant source of traffic and trust.
- Should I block GPTBot and other AI crawlers?
- For most UK businesses, no. Blocking AI crawlers means you won't appear when potential customers ask ChatGPT or Perplexity who to hire in your area. Some businesses (publishers, data-sensitive sites) have reasons to block certain bots, but for service businesses the default should be open access.
- Does blocking AI bots affect my Google rankings?
- Blocking AI crawlers (GPTBot, ClaudeBot, etc.) does not directly affect your standard Google Search rankings — those use Googlebot, which is separate. However, blocking Google-Extended may reduce your visibility in Google AI Overviews and Gemini-powered features.
- How do I fix a robots.txt that's blocking AI crawlers?
- To allow all AI crawlers, ensure your robots.txt either has no specific rules blocking them, or includes explicit Allow directives. The safest approach is to keep a clean robots.txt that only blocks paths you genuinely don't want indexed (e.g. /admin/, /checkout/). We can review and fix this as part of our SEO Management programme.
- Can I trust these results?
- This tool fetches your live robots.txt file and parses it using standard rules. Results are accurate for the rules your file currently contains. However, robots.txt parsing can be complex — some crawlers interpret rules differently. For a definitive audit, we recommend a full technical SEO review.
Want this handled for you?
Book a free discovery call and we'll show you exactly where you stand — and how we'd fix it.