Is my site blocking search engines?
Robots.txt Generator and Tester
Shows which search and AI crawlers your live robots.txt lets in, tests any address against it, and builds a new file rule by rule.
Runs in your browser. The file or text you give it is not uploaded. Free, no account.
Finding the problem is the first half.
RankBrain checks your robots.txt on every audit and flags a rule that blocks pages you want found.
RankBrain works on websites built in code, through the AI tool you already use (Cursor, Claude Code, Codex, ChatGPT).
How it works
You want to know whether your site is shutting out search engines or AI crawlers. Type your website and this tool loads the live robots.txt, shows who may read your home page and tests any address against the file, naming the rule that decides. A second tab builds a new file rule by rule.
- 1
Enter your website. We load its robots.txt and show, crawler by crawler, who may read your home page: the main search crawlers and the AI crawlers. Lines a crawler would misread are listed.
- 2
The tester applies the published matching rules: a crawler obeys the most specific group for its name, the longest matching path wins, and Allow wins a tie.
- 3
Loading the live file is the only step that uses our server. The second tab builds a new file: add groups of rules for all crawlers or for specific ones, with presets for the main search and AI crawlers.
- 4
When the live file has a fault, a prompt under the result tells your AI coding tool what to change.
What it does not do
robots.txt controls crawling, not indexing. A blocked page can still appear in results as a bare link; use noindex to keep a page out.
It is a request, not a lock. Well-behaved crawlers follow it; others ignore it.
When a source is busy or a site does not answer, this tool says so. It never fills the gap with a guess.
Questions
Why does my page still show in Google when robots.txt blocks it?
robots.txt controls crawling, not indexing. A crawler blocked from fetching a page can still list it as a bare link, without a title or description, because it found the address somewhere else. To keep a page out of results, let it be crawled and add a noindex rule to the page itself.
What happens when an Allow and a Disallow rule match the same address?
The tester applies the published matching rules: the longest matching path wins, and Allow wins when the two paths are the same length. So Disallow: /private/ beats a shorter Allow rule, and Allow: /private/public/ beats Disallow: /private/. To open an exception inside a blocked folder, write the Allow rule with the longer path.
Can I check a site that is already live?
Yes, and that is where the tool starts. Type the website and it loads the live robots.txt, shows which search and AI crawlers may read the home page, and lets you test any address against the file. Loading the file is the only step that uses our server. Remember that robots.txt is a request, not a lock. Well-behaved crawlers follow it, others ignore it.
Tools that answer the next question
- Can Google index my page?Google Index CheckerChecks common technical indexing blockers for up to 10 addresses, with an optional search lookup for the first three.
- Create a sitemap for my websiteXML Sitemap GeneratorCrawls your site from the homepage and builds a sitemap.xml from the pages that can be indexed.
- Make an llms.txt file for my sitellms.txt GeneratorDrafts an llms.txt file for your site from the real titles and descriptions of up to 80 of your pages.