Free Tool
Generate a clean robots.txt file
Build a correct robots.txt in seconds — control search crawlers and AI bots, point to your sitemap, and avoid the mistakes that accidentally hide your site.
Your generated code
User-agent: *
Disallow:
Copy & download your code
Enter your email to copy the code, download robots.txt, and get a plain-English guide to adding it to your site.
What this tool checks
- ✓Allow or block search engine crawlers
- ✓Allow or block AI crawlers (GPTBot, ClaudeBot, Google-Extended…)
- ✓Add your sitemap location
- ✓Block private paths (/admin/, /checkout/)
- ✓Standards-compliant syntax
- ✓The safe defaults most businesses should use
See how we've helped UK businesses get found. View case studies →
Frequently asked questions
- What does robots.txt do?
- robots.txt sits at the root of your site and tells crawlers which pages they may access. Get it wrong and you can accidentally block your whole site from Google or AI search.
- Should I block AI crawlers?
- For most service businesses, no — blocking GPTBot, ClaudeBot and similar means you won't appear when buyers ask AI tools who to hire. This generator lets you choose, with 'allow' as the recommended default.
- Where do I put the file?
- Upload it to the root of your domain so it's reachable at yourdomain.com/robots.txt. We include instructions for common platforms with your download.
- Can a bad robots.txt hurt my rankings?
- Yes — a single stray 'Disallow: /' can remove your entire site from search results. That's why this generator uses safe defaults and shows you exactly what each option does.
Related free tools
Want this handled for you?
Book a free discovery call and we'll show you exactly where you stand — and how we'd fix it.