Free Robots.txt Checker
Enter a URL to fetch the site's robots.txt, check it for errors, and see whether Googlebot, Bingbot and AI crawlers like GPTBot may crawl that URL.
Checks the live page · free, no sign-up
How to use the free robots.txt checker
- 01
Enter any URL on the site.
- 02
Read whether each crawler may fetch it, and which rule decides.
- 03
Fix syntax errors, and make sure important pages aren't blocked.
What robots.txt controls
robots.txt sits at the root of your site and tells crawlers which paths they may fetch. One wrong line ("Disallow: /") can block your whole site from Google. It's also where you decide whether AI crawlers like GPTBot and ClaudeBot may read your content.
robots.txt controls crawling, not indexing: a blocked page can still appear in Google if other sites link to it. Use noindex to keep a page out of results.
AI crawlers and what they do
The user agents the robots.txt tools know about. Blocking a training crawler keeps your content out of model training; blocking a search crawler keeps you out of that assistant's answers.
| User agent | Company | Used for |
|---|---|---|
| GPTBot | OpenAI | Training AI models |
| ChatGPT-User | OpenAI | Fetching pages a user asks about |
| OAI-SearchBot | OpenAI | AI search answers |
| ClaudeBot | Anthropic | Training AI models |
| Claude-User | Anthropic | Fetching pages a user asks about |
| Claude-SearchBot | Anthropic | AI search answers |
| PerplexityBot | Perplexity | AI search answers |
| Google-Extended | Training AI models | |
| Applebot-Extended | Apple | Training AI models |
| CCBot | Common Crawl | Training AI models |
| Bytespider | ByteDance | Training AI models |
| meta-externalagent | Meta | Training AI models |
Tips for using the robots.txt checker
Never block CSS or JavaScript
Google needs them to render your pages. Blocking them can hurt how pages are understood.
List your sitemap
A Sitemap: line helps every search engine find your pages. Validate it with the XML sitemap validator.
Use noindex to hide pages
robots.txt stops crawling, not indexing. To keep a page out of Google, use noindex and check it with the noindex checker.
Build a clean file
Start from a correct template with the robots.txt generator.
Skip the manual work: get an article published to your site every day.
Rankfly finds the keywords your site can actually win, writes a full SEO article for each one, adds images, and publishes it to WordPress, Shopify, Webflow, Wix and more. Enter your website to see your free 30-day content plan.
3-day free trial · $66/month for 30 articles
Robots.txt Checker FAQ
How does Google decide which rule applies?
The most specific (longest) matching rule wins, and Allow wins a tie. This checker applies the same logic.
Should I block AI crawlers?
Blocking training crawlers (GPTBot, Google-Extended) keeps your content out of model training. Blocking search crawlers (OAI-SearchBot, PerplexityBot) keeps you out of AI answers, which costs visibility.
Is it free?
Yes, with no sign-up. Checks are limited to 60 a day per person.
Where is robots.txt located?
Always at the root of the domain: https://example.com/robots.txt.
What happens if robots.txt is missing?
Crawlers assume everything may be crawled. That's fine, but you lose the chance to point to your sitemap.
Does every crawler obey robots.txt?
Reputable ones do. Malicious bots ignore it, so it isn't a security tool.
Related free SEO tools
All 87 free toolsCanonical Tag Checker
Enter a URL to check its canonical tag: whether it exists, points to itself, is the only one, and points to a working page.
Broken Link Checker
Enter a URL to test every link on the page and find the broken ones (404s and errors) and the ones that redirect.
Robots.txt Generator
Create a correct robots.txt: choose what to block, add your sitemap, and decide how to treat AI crawlers like GPTBot and ClaudeBot.
XML Sitemap Validator
Enter a sitemap URL (or just your domain) to validate its structure, URLs and lastmod dates, and check that the first pages listed actually load.
Noindex Checker
Enter a URL to find out whether Google can index it: meta robots, X-Robots-Tag header, robots.txt, status code and canonical, all in one check.
HTTP Header Checker
Enter a URL to see its HTTP response headers, with checks on HTTPS, caching, compression, X-Robots-Tag and key security headers.
All free technical seo tools
Let Rankfly do your SEO for you.
Enter your website for a free 30-day content plan: the keywords your site can win and the articles Rankfly would write and publish. About three minutes.
3-day free trial · $66/month after · Cancel anytime