Free Robots.txt Generator
Create a correct robots.txt: choose what to block, add your sitemap, and decide how to treat AI crawlers like GPTBot and ClaudeBot.
Runs in your browser · nothing is uploaded
Before you publish
- Add your sitemap URL so crawlers find every page.
robots.txt
User-agent: *
Allow: /How to use the free robots.txt generator
- 01
Choose what crawlers may access.
- 02
Add your sitemap URL and any AI crawler rules.
- 03
Save the output as robots.txt at the root of your domain.
A robots.txt that doesn't hurt you
Most sites need very little in robots.txt: allow everything, block admin and cart pages, and point to the sitemap. Over-blocking is a common cause of pages vanishing from Google, especially blocking CSS and JavaScript folders Google needs to render pages.
The AI options let you opt out of training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) while still allowing the search crawlers that put you in ChatGPT, Claude and Perplexity answers.
AI crawlers and what they do
The user agents the robots.txt tools know about. Blocking a training crawler keeps your content out of model training; blocking a search crawler keeps you out of that assistant's answers.
| User agent | Company | Used for |
|---|---|---|
| GPTBot | OpenAI | Training AI models |
| ChatGPT-User | OpenAI | Fetching pages a user asks about |
| OAI-SearchBot | OpenAI | AI search answers |
| ClaudeBot | Anthropic | Training AI models |
| Claude-User | Anthropic | Fetching pages a user asks about |
| Claude-SearchBot | Anthropic | AI search answers |
| PerplexityBot | Perplexity | AI search answers |
| Google-Extended | Training AI models | |
| Applebot-Extended | Apple | Training AI models |
| CCBot | Common Crawl | Training AI models |
| Bytespider | ByteDance | Training AI models |
| meta-externalagent | Meta | Training AI models |
Tips for using the robots.txt generator
Keep it short
Most sites only need to block admin, cart and internal search pages, and list the sitemap.
Test before uploading
After uploading, check specific URLs with the robots.txt checker.
Think before blocking AI search
Blocking OAI-SearchBot or PerplexityBot removes you from those assistants' answers. Most businesses want to be there.
Use wildcards carefully
"Disallow: /*?" blocks every URL with parameters, which may include pages you want crawled.
Skip the manual work: get an article published to your site every day.
Rankfly finds the keywords your site can actually win, writes a full SEO article for each one, adds images, and publishes it to WordPress, Shopify, Webflow, Wix and more. Enter your website to see your free 30-day content plan.
3-day free trial · $66/month for 30 articles
Robots.txt Generator FAQ
Where do I upload robots.txt?
To the root of your domain, so it's at https://yourdomain.com/robots.txt. On WordPress, plugins like Yoast let you edit it from the dashboard.
Does blocking a page remove it from Google?
No. Use a noindex tag for that. robots.txt only stops crawling.
Is anything uploaded?
No. The file is generated in your browser.
What's the difference between Allow and Disallow?
Disallow blocks a path; Allow re-opens part of a blocked path. Google applies the most specific rule.
Does Google respect Crawl-delay?
No. Bing and Yandex do. Google adjusts its crawl rate automatically.
How do I block ChatGPT from training on my site?
Block GPTBot. Choose "Block AI training, allow AI search" to keep appearing in ChatGPT search answers.
Related free SEO tools
All 87 free toolsCanonical Tag Checker
Enter a URL to check its canonical tag: whether it exists, points to itself, is the only one, and points to a working page.
Broken Link Checker
Enter a URL to test every link on the page and find the broken ones (404s and errors) and the ones that redirect.
Robots.txt Checker
Enter a URL to fetch the site's robots.txt, check it for errors, and see whether Googlebot, Bingbot and AI crawlers like GPTBot may crawl that URL.
XML Sitemap Validator
Enter a sitemap URL (or just your domain) to validate its structure, URLs and lastmod dates, and check that the first pages listed actually load.
Noindex Checker
Enter a URL to find out whether Google can index it: meta robots, X-Robots-Tag header, robots.txt, status code and canonical, all in one check.
HTTP Header Checker
Enter a URL to see its HTTP response headers, with checks on HTTPS, caching, compression, X-Robots-Tag and key security headers.
All free technical seo tools
Let Rankfly do your SEO for you.
Enter your website for a free 30-day content plan: the keywords your site can win and the articles Rankfly would write and publish. About three minutes.
3-day free trial · $66/month after · Cancel anytime