Robots.txt Tester

How to Use the Robots.txt Tester

Enter Your Domain or Paste the File — Type your website URL (e.g. https://example.com) to fetch the live robots.txt file, or paste your robots.txt content directly into the text area. The tester reads the file exactly as Googlebot would.
Enter the URL Path to Test — Type the path you want to check — for example /blog/my-post, /admin/, or /wp-login.php. The tester checks every applicable rule in your robots.txt file against that path.
Select a User-Agent — Choose Googlebot, Googlebot-Image, Bingbot, or * (all bots) from the dropdown. Most robots.txt files use * to target all crawlers at once, but some sites define crawler-specific rules.
Read the Result — The tester returns Allowed, Blocked, or No Specific Rule. 'Allowed' means the path is open to crawling. 'Blocked' means a Disallow rule matches. 'No Specific Rule' means no rule applies — the path is crawlable by default.

What Is a Robots.txt File?

A robots.txt file sits at the root of your domain — https://yourdomain.com/robots.txt — and tells search engine crawlers which pages to crawl and which to skip. It follows the Robots Exclusion Protocol, a standard Google, Bing, and other major search engines have respected since 1994.

Robots.txt is advisory, not a security mechanism. It instructs well-behaved crawlers; it does nothing to stop bad actors or scrapers. Its real job is protecting crawl budget: by steering Googlebot away from admin interfaces, parameter-generated duplicates, and internal search results, you concentrate crawl attention on pages that actually need to be indexed.

Why You Should Test Your Robots.txt Before Publishing

A single typo in a robots.txt file can accidentally block your entire site from Google. The most common disaster is Disallow: / without a specific user-agent target — this tells all crawlers to skip every URL on the domain. It sounds extreme, but this exact mistake happens on real sites during staging-to-production migrations when someone forgets to swap out the staging robots.txt.

Even subtler mistakes cost rankings. A missing trailing slash on a directory rule, an extra space before a directive, or a rule targeting the wrong user-agent can silently block content for months. Our free robots.txt checker catches these issues before Googlebot does.

Robots.txt Directives Explained

Every robots.txt file is built from a handful of directives. Understanding what each one does is essential for validating your file correctly.

User-agent: Specifies which crawler the following rules apply to. User-agent: * targets all bots. User-agent: Googlebot targets only Google's main web crawler. A robots.txt file can contain multiple user-agent blocks with different rules for each.
Disallow: Tells the specified crawler not to crawl the listed path. Disallow: /admin/ blocks everything under /admin/. Disallow: / blocks the entire site. Disallow: with no value allows everything (often used as a reset after a previous block).
Allow: Overrides a Disallow for specific paths. If you block a whole directory but want one subfolder to be crawlable, Add an Allow rule for that path. The more specific rule takes precedence when rules conflict.
Crawl-delay: Requests that the crawler waits the specified number of seconds between fetches. Google officially ignores Crawl-delay — if you need to throttle Googlebot, use the crawl rate settings in Google Search Console instead.
Sitemap: Points crawlers to your XML sitemap. Can appear anywhere in the file, outside any user-agent block. Using an absolute URL is required: Sitemap: https://yourdomain.com/sitemap.xml

The Mistake That Blocks Entire Sites

The most dangerous robots.txt configuration looks like this:

User-agent: *
Disallow: /

This tells every crawler to avoid every URL on the domain. It is the correct setup for a staging environment that should never be crawled. It is catastrophic if accidentally deployed to production. Google will drop all your pages from the index within days of seeing this configuration. Always run a robots.txt checker before pushing changes to production.

A subtler version of this problem: a staging robots.txt is left in place during a CDN or DNS migration. The main domain redirects to the new server, but the robots.txt file from the old server gets cached by Google for hours or days before it fetches the new version. Check your robots.txt file after any infrastructure change.

Robots.txt vs. Noindex: When to Use Which

Robots.txt and the noindex tag solve different problems. Confusing them is one of the most common technical SEO mistakes.

Use robots.txt to prevent Googlebot from crawling a URL entirely — this saves crawl budget but does not guarantee the URL stays out of the index. If other sites link to a Disallowed URL, Google can still discover and index it (as a URL-only result) without ever crawling the page content.

Use noindex (via a meta robots tag or X-Robots-Tag HTTP header) when you want to prevent a page from appearing in search results. This requires the page to be crawlable — Googlebot must fetch the page to see the noindex instruction. Never block a URL in robots.txt and also noindex it. If you block crawling, Googlebot cannot read the noindex tag.

The practical rule: use robots.txt for pages that should never be crawled (admin, staging, session parameters). Use noindex for pages that can be crawled but should not be indexed (thank-you pages, internal search results, print versions).

How Google Processes Robots.txt Rules

Google follows a specific order of priority when evaluating which rule applies to a URL. The most specific matching rule wins — not the first matching rule. If you have:

Disallow: /blog/
Allow: /blog/category/

Google will crawl everything under /blog/category/ while blocking the rest of /blog/. The Allow rule is more specific and takes precedence. When two rules match with equal specificity, the Allow rule wins (Google's tiebreaker favors crawling over blocking).

Rules are case-sensitive. Disallow: /Admin/ does not block /admin/. Wildcards use * for any sequence of characters and $ to anchor a pattern to the end of a URL. Disallow: /*.pdf$ blocks all URLs ending in .pdf.

How to Verify Your Robots.txt Is Working

Beyond our free robots.txt tester, there are two other checkpoints worth running:

Google Search Console robots.txt report — Under Settings → Crawl stats → Open report, you can see how often Googlebot fetches your robots.txt and any fetch errors. GSC also has a robots.txt inspector in the legacy tools section where you can test specific URLs against your live file.
Direct URL fetch — Open https://yourdomain.com/robots.txt in a browser. If you get a 404, Google will treat the file as missing and crawl everything. A blank file is valid; a 404 is equivalent to no file.

Our robots.txt checker fetches the live file directly — what you see here is what Googlebot sees. There are no caches or assumptions.

Frequently Asked Questions

What is a robots.txt tester?

A robots.txt tester is a tool that reads a website's robots.txt file and checks whether a specific URL path is allowed or blocked for a given crawler. You enter a URL and a user-agent (such as Googlebot), and the tester evaluates every matching rule to tell you the result: allowed, blocked, or no specific rule found.

How do I check my robots.txt file?

The fastest way: go to https://yourdomain.com/robots.txt in a browser to verify the file exists and looks correct. To check whether specific rules block specific URLs, use a robots.txt checker like this one — enter your domain or paste your file content, add the URL you want to test, select your target user-agent, and read the result.

What is the difference between a robots.txt tester and validator?

A robots.txt tester checks whether a specific URL is allowed or blocked by your current rules — it tests your rules against real paths. A robots.txt validator checks the file's syntax: correct directive names, proper formatting, valid URL patterns. Our tool does both: it validates the file structure while simultaneously testing specific paths.

How do I validate my robots.txt file?

Paste your robots.txt content into this tool and test paths you know should be allowed (like your homepage and key blog posts) and paths that should be blocked (like /admin/). If the results match your expectations, your file is valid. If not, check for missing User-agent lines, incorrect path formatting, or conflicting rules.

Can I test robots.txt online without installing anything?

Yes — this tool runs entirely in your browser. Enter your domain URL to fetch the live file, or paste content directly. No downloads, no account, no plugins required.

How does this compare to Google's own robots.txt tester?

Google Search Console includes a robots.txt inspector, but it requires you to log in and have verified ownership of the site. Our tool works on any publicly accessible robots.txt file — your own site or a competitor's — without authentication. It fetches the live file and evaluates rules using the same priority logic Google uses.

What does "no specific rule" mean in a robots.txt test?

It means no rule in the file explicitly allows or blocks the tested path for the selected user-agent. The default behavior is allowed — Google will crawl any URL that is not explicitly disallowed. A missing rule is not the same as a blocking rule.

Why is my robots.txt blocking Googlebot from pages I need indexed?

The most common cause is an overly broad Disallow rule. Check for Disallow: / under User-agent: *, which blocks everything. Also watch for rules like Disallow: /blog that accidentally match all URLs beginning with /blog (including /blog-post-title) instead of just the /blog/ directory. Adding a trailing slash — Disallow: /blog/ — limits the block to the directory itself.

Should I block my admin panel in robots.txt?

You can block /admin/, /wp-admin/, and similar admin paths in robots.txt to save crawl budget, but this is not a security measure. Admin pages should be protected by authentication, not by robots.txt. Use robots.txt to manage crawl priority, not to hide sensitive content.

How often does Google re-fetch the robots.txt file?

Google typically caches your robots.txt for up to 24 hours. After publishing changes, it may take a day for the new rules to take effect across all of Google's crawlers. You can request a faster re-fetch through Google Search Console's crawl settings, but there is no guaranteed instant update. Plan robots.txt changes in advance — especially before migrations.

Does a missing robots.txt file block crawling?

No. If Googlebot requests your robots.txt and gets a 404 (not found), it treats this as if no file exists and proceeds to crawl all pages normally. A 500 server error, however, causes Google to temporarily stop crawling your site while it retries. Make sure your robots.txt returns either a 200 with content or a 404 — never a 5xx error.

Is this robots.txt tester free?

Yes. The robots.txt tester on SEO Scout is completely free — no sign-up, no usage limits, no premium tier. All our tools work in your browser and do not require an account.