Robots.txt Guide: Examples, Rules and Common Mistakes

Robots.txt is a small text file at the root of your website that tells search engine crawlers which parts of the site they may crawl. It is simple, but a single wrong line can stop Google from crawling your entire site. This guide explains the rules, gives copy-ready examples and lists the mistakes to avoid.

Where robots.txt lives

The file must be called robots.txt and sit at the root of the host it applies to, for example https://example.com/robots.txt. A file in a subfolder is ignored. Each subdomain and protocol needs its own file, so blog.example.com needs a separate robots.txt from example.com.

Google limits robots.txt files to 500 KiB and ignores content beyond that. Most sites need only a few lines.

What robots.txt does and does not do

Robots.txt controls crawling, not indexing. This is the most misunderstood point about the file.

• If you block a URL in robots.txt, Google will not crawl it, but it can still index the URL without its content if other pages link to it. In Search Console this appears as "Indexed, though blocked by robots.txt".
• To keep a page out of Google, allow crawling and add a noindex meta tag or X-Robots-Tag header instead. Google has to crawl the page to see the noindex.
• Robots.txt is not a security tool. The file is public, and badly behaved bots can ignore it. Protect private content with a login.

The core rules

A robots.txt file is made of groups. Each group starts with one or more User-agent lines followed by rules.

• User-agent: names the crawler the group applies to. An asterisk means all crawlers. Googlebot means Google's main crawler.
• Disallow: a path the crawler should not crawl. An empty Disallow means nothing is blocked.
• Allow: a path that may be crawled even inside a disallowed folder.
• Sitemap: the full URL of your XML sitemap. It can appear anywhere in the file and is not tied to a group.

Paths are case-sensitive and match from the start of the URL path. Google also supports two wildcards: an asterisk matches any sequence of characters, and a dollar sign marks the end of the URL.

When Allow and Disallow rules conflict, Google uses the most specific rule, meaning the one with the longest matching path. If they are equally specific, Google uses the least restrictive rule, which is Allow.

A crawler follows only the most specific group that matches its name. If you have a group for Googlebot and a group for all crawlers, Googlebot follows only its own group and ignores the general one.

Unsupported rules

• Crawl-delay is ignored by Google. Some other search engines, such as Bing, support it.
• Noindex inside robots.txt is not supported by Google. Use a meta robots tag instead.
• Comments start with a hash sign and are ignored by crawlers, which makes them useful for notes to your team.

Ready-to-use examples

Each example below shows the lines of the file, one directive per line.

Allow everything (good default for most small sites)

• User-agent: *
• Disallow:
• Sitemap: https://example.com/sitemap.xml

Block an admin area and internal search

• User-agent: *
• Disallow: /admin/
• Disallow: /search
• Sitemap: https://example.com/sitemap.xml

WordPress

• User-agent: *
• Disallow: /wp-admin/
• Allow: /wp-admin/admin-ajax.php
• Sitemap: https://example.com/sitemap_index.xml

The Allow line keeps admin-ajax.php crawlable because some themes and plugins use it on public pages.

Block URLs with filter parameters

• User-agent: *
• Disallow: /*?sort=
• Disallow: /*?filter=

Useful for online stores where sorting and filtering create thousands of near-duplicate URLs that waste crawling.

Block one file type

• User-agent: *
• Disallow: /*.pdf$

The dollar sign makes the rule match only URLs that end in .pdf.

Block an AI crawler but not search engines

• User-agent: GPTBot
• Disallow: /

Many AI companies publish their crawler names and say they respect robots.txt. Check each company's documentation for the current user-agent name. Google-Extended is the token Google uses for control over use of content in some of its AI products; it does not affect Google Search.

Common mistakes

1. Leaving "Disallow: /" from a staging site. Developers often block everything during development and forget to remove it at launch. This blocks your whole site. Check robots.txt on launch day; it is part of our new website SEO checklist.
2. Blocking CSS and JavaScript. Google renders pages like a browser. If it cannot load your styles and scripts, it may misunderstand the page.
3. Using robots.txt to remove pages from Google. Blocked pages can stay in the index. Use noindex, then let Google recrawl.
4. Blocking a page and adding noindex together. Google cannot see the noindex if it cannot crawl the page.
5. Wrong case or missing slash. Disallow: /Admin does not block /admin. Disallow: /blog also blocks /blog-tips, because matching starts from the beginning of the path.
6. Serving robots.txt with an error. If the file returns a server error, Google may treat the whole site as temporarily disallowed. A 404 for robots.txt is fine; Google treats it as no restrictions.

How to test your robots.txt

Search Console has a robots.txt report under Settings. It shows which robots.txt files Google found for your property, when they were last crawled and any parsing warnings or errors. To check a specific page, use the URL Inspection tool and look for whether crawling is allowed. Our Google Search Console guide explains where to find both.

After you fix a blocking rule, use Request indexing on your most important pages. External links can also help Google rediscover them; a run with Backlink Generator creates public discovery mentions of your URL, though it cannot override a robots.txt block and does not guarantee indexing.

FAQ

Does every website need a robots.txt file?

No. If the file is missing and returns a 404, Google crawls everything it can find. Having one is still useful for listing your sitemap and blocking low-value sections such as internal search results.

How long does Google take to see robots.txt changes?

Google generally caches robots.txt for up to 24 hours, so changes are usually picked up within a day. You can ask for a faster recrawl from the robots.txt report in Search Console.

Can robots.txt hurt my SEO?

Yes, if it blocks pages or resources Google needs. A single "Disallow: /" under "User-agent: *" blocks the entire site. Check the file after every launch, migration or redesign.