Robots.txt Validator
Enter your domain to instantly check if your robots.txt is properly configured, with detailed explanations of any issues found.
Enter a domain above to validate its robots.txt file
What is robots.txt?
A robots.txt file tells search engine crawlers which pages or sections of your site should or shouldn't be crawled. It's placed at the root of your domain (e.g., example.com/robots.txt) and is one of the first files search engines check.
A properly configured robots.txt helps ensure search engines can efficiently crawl your important pages while avoiding duplicate content, admin areas, and other pages you don't want indexed.
It is also the single most dangerous file on your site. A stray Disallow: / left over from a staging environment will quietly remove an entire site from search results, and it can take weeks to recover once noticed. This is worth checking after every site launch, redesign, or platform migration.
What this validator checks
That the file exists and is really plain text. A surprising number of sites serve an HTML 404 page at /robots.txt. Search engines won't treat that as valid, so whatever rules you believe are in force aren't being applied at all.
Sitewide blocks. A Disallow: / under User-agent: * blocks your whole site. If that shows up and you didn't intend it, stop and fix it before anything else on your marketing list.
Sitemap declarations. Listing your sitemap here gives every crawler a direct route to your URLs without any manual submission.
AI crawler directives. The file shows how you're treating GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. For a local business hoping to be recommended when someone asks an AI assistant for a nearby provider, blocking these means being invisible in a channel that keeps growing.
One thing to keep clear: robots.txt controls crawling, not indexing. If you need a page kept out of search results entirely, allow it to be crawled and add a noindex meta tag — blocking it here can leave a bare URL showing in results with no description underneath.
Frequently Asked Questions
- Where does robots.txt need to live?
- At the root of your domain, exactly at example.com/robots.txt. Crawlers only look there. A file at example.com/pages/robots.txt or example.com/Robots.txt will be ignored, and subdomains need their own file — blog.example.com does not inherit the one at example.com.
- Does Disallow stop a page appearing in Google?
- No, and this trips up a lot of people. Disallow stops Google crawling the page, not indexing it. A blocked URL with links pointing at it can still appear in results with no description. To genuinely keep a page out of the index, allow crawling and use a noindex meta tag instead.
- Should I block AI crawlers like GPTBot and ClaudeBot?
- That depends on your goals. Blocking them keeps your content out of AI training and out of AI-generated answers. For a local business trying to be recommended when someone asks an assistant for a plumber nearby, being blocked means being invisible. Most local businesses should allow them.
- Do I need to list my sitemap in robots.txt?
- It's not required, but it's a free win. A Sitemap directive gives every crawler a direct path to your URL list without you submitting anything anywhere. You can list more than one line if you have multiple sitemaps.
- What happens if I have no robots.txt at all?
- Crawlers assume everything is fair game and crawl freely, which is fine for most small sites. You'll see a 404 in your server logs, but nothing breaks. You only need the file when you want to restrict something or point to a sitemap.
- Why is my robots.txt returning an HTML page?
- Your server is serving a 404 page or a catch-all route instead of a plain text file. Search engines will not treat that as valid, so any rules you think you have aren't being applied. Check that the file is genuinely reachable and returns content-type text/plain.
More Free Tools
Explore our other browser-based utilities — 100% free, no sign-up.
Meta Tag Analyzer
Audit title, description, OG & Twitter tags.
OG Preview Tool
Preview how links appear on social platforms.
Heading Analyzer
Check H1–H6 hierarchy & accessibility.
Robots.txt Validator
Validate crawl directives & rules.
Sitemap Validator
Verify sitemap structure & URLs.
pSEO CSV Generator
Generate keyword CSVs for programmatic SEO.
Email Signature Generator
Create professional HTML email signatures.