Google’s robots.txt parser recognises four fields: user-agent, allow, disallow and sitemap. If it encounters a line it does not understand, it usually ignores it and moves on.
In the 21 October 2026 episode of Google’s Non-Canonical Cabinet series, Gary Illyes and Cherry Prommawin from Google Search Relations answered questions about unusual robots.txt rules, including emojis, ASCII art, non-English URL paths and attempts to control AI crawlers.
Their advice was straightforward: keep your robots.txt file simple.
Google does not necessarily warn you when a rule is invalid. If Google ignores a line, the pages you intended to block may still be crawled. If your server returns the wrong response for /robots.txt, Google may behave as though the file does not exist at all.
This can lead to unnecessary crawling, problems with important pages and confusion about which AI systems can access your content.
This guide explains how Google reads robots.txt, the mistakes that can cause your rules to fail, and how SEO, development, marketing and legal teams can manage the file properly. It also covers the different rules for AI training and AI search crawlers, and includes examples your developers can use when checking a robots.txt file.
1. What Happened?
Google explains which robots.txt rules it understands
On 21 October 2026, Google Search Central published the second episode of Non-Canonical Cabinet, a video series in which Gary Illyes and Cherry Prommawin answer unusual technical questions from the search community.
The first episode, published on 14 October, covered domain names and URL paths. The second focused on robots.txt, including whether the file can contain emojis, ASCII art and non-English characters, and whether it can be used to control AI systems.
The answers all came back to the same principle: Google’s robots.txt parser follows the Robots Exclusion Protocol, defined in RFC 9309, and recognises a limited set of fields.
It reads the file line by line and ignores instructions it does not support.
For example, this is not a valid Google robots.txt instruction:
Don't crawl: /secret-pageA person can understand what it means, but Google does not recognise Don't crawl as a supported field. The instruction is ignored, and Google may crawl the page.
The episode also highlighted a distinction that matters increasingly for AI search: robots.txt controls crawling, not how content is used after it has been accessed.
Blocking a crawler tells it not to fetch certain pages. It does not automatically control what happens to content the system has already collected.
What Google said about unusual robots.txt rules
The following table summarises the questions covered in the episode.
| Question | Google’s answer | What it means for your website |
|---|---|---|
| Can robots.txt contain ASCII art? | Yes. Text in comments is ignored when the line starts with #. | Comments are fine, but large amounts of decorative text waste space and make the file harder to maintain. |
| What happens to formatting mistakes or custom rules? | Google ignores fields it does not recognise. | A rule that looks sensible to a person may do nothing. |
| Can a user-agent token contain emojis or non-English characters? | Google supports a limited set of characters in crawler tokens: letters, underscores and hyphens. | Use the exact crawler name published by the relevant company. |
| Can URL paths contain emojis or non-English characters? | Google can match raw UTF-8 and percent-encoded paths. Other crawlers may behave differently. | Use encoded paths when you need to support different crawlers consistently. |
| Can robots.txt stop AI companies from training on your content? | Not by itself. It is a set of instructions about crawling. | Choose the correct, published crawler token for the type of access you want to control. Invented rules do nothing. |
A file could contain 97% comments or decorative material and only 3% actual rules. That would not necessarily stop Google’s parser from working, but it would make the file unnecessarily large and harder to maintain.
The presenters joked about an imaginary “chicken bot”. The point was that inventing a crawler name does not create a new rule or control a real crawler.
The main rules Google follows
Google’s documentation, How Google interprets the robots.txt specification, was updated on 31 August 2026. It explains the behaviour discussed in the episode.
| Topic | What Google does |
|---|---|
| Supported fields | Recognises user-agent, allow, disallow and sitemap. Other fields, including crawl-delay, are not supported by Google. |
| Invalid lines | Ignores lines it cannot understand. A byte order mark at the start of the file can also cause problems. If the server returns HTML instead of a robots.txt file, Google tries to extract any valid rules and ignores the rest. |
| Character encoding | Expects UTF-8. Unsupported characters can cause parts of the file to be ignored or rules to stop matching. |
| Non-English URL paths | Treats raw UTF-8 and percent-encoded paths as equivalent when matching URLs. Other crawlers may not behave the same way. |
| Maximum file size | Reads up to 500 KiB. Anything after that limit is ignored. |
| Capital letters | Field names and user-agent values are case-insensitive, but URL paths are case-sensitive. |
| Comments | Ignores everything after # on a line. |
| Choosing a crawler group | Uses the most specific matching user-agent group. A crawler with its own group does not also use the general * group. |
| Choosing a rule | The longest matching path wins. If an allow and a disallow rule are equally specific, the less restrictive allow rule wins. |
| Where the file applies | Each robots.txt file covers one host, protocol and port. A subdomain needs its own file. |
| Caching | Google usually caches the file for up to 24 hours, although it may take longer to refresh when the file cannot be fetched. |
These rules explain why robots.txt should be kept simple and checked carefully. A small mistake can change how Google treats a large part of your website.
Why AI crawlers make robots.txt more important
Businesses now need to think about several different types of AI crawler.
Google’s Google-Extended token controls whether content Google crawls can be used to train Gemini models and to help generate answers in the Gemini app and Vertex AI. It does not control Google Search.
Google also introduced a separate Search generative AI control in Search Console. Following a requirement from the UK’s Competition and Markets Authority (CMA) on 3 June 2026, Google began providing a separate way to control the use of content in AI Overviews, AI Mode and generative AI features in Discover. Google said the control was available worldwide by 31 August 2026.
These are different controls. Google-Extended is a published crawler token, while the Search generative AI control is a Search Console setting.
You cannot replace either of them with an invented robots.txt instruction such as No-AI-Training: /. Google will ignore that line.
2. Why Do Robots.txt Mistakes Matter?
Robots.txt is a small file, but it can affect how many crawlers access your website and which parts they visit.
The biggest risk is that mistakes are often silent. Your website may continue working normally while crawlers follow different rules from the ones your team intended.
1. A rule Google cannot read is not applied
Google ignores unsupported fields and invalid lines rather than treating them as instructions.
The dangerous mistakes often look perfectly normal to a person reading the file.
For example:
A full-width colon
Some Chinese and Japanese input methods produce a full-width colon (：) rather than the standard colon (:).
This line may look correct:
Disallow：/account/But the punctuation is wrong. Google may not recognise it as a valid Disallow instruction.
Invisible characters
A non-breaking space or zero-width character copied from a document or content management system can interfere with a rule.
Unsupported fields
Google does not support noindex: or crawl-delay: as robots.txt instructions. Adding these lines will not have the effect some website owners expect.
An invalid crawler name
An emoji or an unsupported user-agent token will not match the crawler you intend to target.
A # in the wrong place
In robots.txt, # starts a comment. This means the following rule is dangerous:
Disallow: /#privateGoogle treats everything after # as a comment. The effective rule is:
Disallow: /That blocks the entire website for crawlers following the group.
If an important rule stops working, Google may spend its time crawling filter combinations, internal search pages, basket URLs or other addresses you intended to keep out of crawling.
On a large website, that can lead to hundreds of thousands of unnecessary requests.
2. The wrong response from your server can disable crawling rules
The response your server returns for /robots.txt matters as much as the instructions inside the file.
| Server response | How Google generally treats it |
|---|---|
2xx | Uses the file returned by the server. |
3xx | Follows redirects, up to a limit of five consecutive redirects. If the file still cannot be retrieved, Google treats it as unavailable, effectively like a 404. |
4xx, except 429 | Treats the website as though no robots.txt file exists, so there are no robots.txt restrictions. |
429 | Treats the response as a sign of rate limiting and may slow down crawling. |
5xx, timeouts or network errors | Initially pauses crawling, uses a previously cached good file where possible, and may eventually treat the file as unavailable if the problem continues. |
Two situations deserve particular attention.
A firewall blocks Googlebot.
If your firewall or bot protection returns 403 Forbidden for /robots.txt, Google treats the file as unavailable. That can remove the restrictions your team intended to apply.
A JavaScript website returns its normal app page for robots.txt.
Some websites return the same HTML application shell for every unknown path, including /robots.txt, with a 200 OK status. Google may try to extract rules from the HTML, but your intended robots.txt instructions may not be available.
Developers should make sure /robots.txt is handled explicitly by the server or CDN, before any catch-all route for the JavaScript application.
3. Non-English URL paths must match the URLs your website creates
Google can match raw UTF-8 characters and their percent-encoded forms in robots.txt. Other crawlers may not do this consistently, which is why using encoded paths is often the safer choice.
For example, a path containing the chicken emoji:
/🐔-gear
can be written in percent-encoded form as:
/%F0%9F%90%94-gear
There is another detail to consider: the same visible character can have different Unicode representations.
For example, the accented letter é can be represented as one character or as the letter e followed by a combining accent. Their encoded forms differ:
- Single character:
%C3%A9 - Letter followed by combining accent:
e%CC%81
If your website uses one form and your robots.txt rule uses the other, some crawlers may not match the rule as you expect.
This matters for websites with Polish, German, French, Arabic, Chinese, Japanese, Hindi and other non-English URL paths.
The rule should match the URL your website actually generates. Check live URLs before deciding which encoded version to use.
4. A separate crawler group can override your general rules
Google applies the most specific matching user-agent group to each crawler.
Suppose you start with a general group:
User-agent: *
Disallow: /basket/
Disallow: /checkout/
Disallow: /account/Later, someone adds a separate Googlebot group:
User-agent: Googlebot
Allow: /blog/Googlebot follows the Googlebot group rather than combining it with the general * group. The three restrictions may therefore no longer apply to Googlebot.
If the specific group should also block the basket, checkout and account pages, those rules must be repeated in that group.
Other crawlers have their own rules. Apple says Applebot follows Googlebot’s rules when robots.txt does not mention Applebot. Google’s special-case crawlers, including AdsBot, may ignore the general * group and need to be named separately if you want to restrict them.
5. Robots.txt does not guarantee that a page stays out of Google
A page blocked by robots.txt can still appear in search results if Google discovers its URL through other links. It may appear without a normal description because Google cannot fetch its content.
There is also an important difference between blocking crawling and removing a page from search.
If you block a page in robots.txt, Google cannot crawl it to see a noindex tag on that page.
If you want a page removed from Google’s index, it generally needs to remain crawlable until Google can read the noindex instruction. Alternatively, protect private content with authentication.
Do not use robots.txt as a replacement for access controls or page-removal instructions.
6. Choosing the wrong AI crawler can cost you search visibility
AI companies use different crawler names for training, search discovery and requests made on behalf of individual users.
Confusing these crawlers can have direct consequences.
- Blocking
OAI-SearchBotcan stop your website from being included in ChatGPT search answers, along with the referral traffic those answers may generate. - Blocking
GPTBotcontrols a different activity. It does not automatically stop ChatGPT search from finding your website. - Blocking
Google-Extendeddoes not remove your website from Google Search, AI Overviews or AI Mode. - Blocking Googlebot can remove your website from Google Search altogether, affecting both traditional results and Google’s AI search features.
- Blocking
Perplexity-UserorChatGPT-Usermay not work as intended because user-initiated fetchers may not follow robots.txt in the same way as search crawlers.
For businesses measuring how often AI answers cite their websites, blocking the wrong crawler can mean losing visibility in an assistant’s results.
Our guide to the ChatGPT UTM tracking gap explains why AI referral traffic can already be difficult to measure. Our answer capture ratio research looks at how often AI answers cite a website.
7. Rules at the end of a large file may never be read
Google reads up to 500 KiB of robots.txt. Anything after that limit is ignored.
Files can become unnecessarily large when plugins generate thousands of individual URL rules or append long lists of addresses.
If the file exceeds the limit, the rules near the end may stop working.
Use directory rules and carefully chosen wildcards where appropriate, rather than maintaining enormous lists of individual URLs.
The main risks at a glance
| Problem | Common cause | Possible result |
|---|---|---|
| Rules no longer work | Incorrect punctuation, unsupported fields or # used inside a path | Pages you meant to block may be crawled |
| No rules are applied | 403 response or HTML returned instead of a proper robots.txt file | Google may treat the file as unavailable |
| Crawling pauses | 5xx responses, timeouts or network errors | Google may pause crawling or rely on an older copy of the file |
| Rules fail to match URLs | Non-English paths use a different encoded form | The intended pages may not be blocked |
| Googlebot follows the wrong rules | A specific group does not repeat the restrictions in the general group | Googlebot may crawl URLs that other crawlers cannot |
| AI search visibility falls | A search crawler is blocked while the business intends only to block training | Fewer citations and referrals from that assistant |
| Rules stop part-way through the file | Robots.txt exceeds 500 KiB | Rules beyond Google’s reading limit are ignored |
3. Which Websites Are Most at Risk?
Every website with a robots.txt file can be affected by these problems. The risk increases when the file is complicated, several teams edit it or the website relies on firewalls, plugins and CDNs that change how the file is served.
International and multilingual websites
Websites with non-English URL slugs need particular care.
Examples include Polish paths such as żółw, German words containing ö, and URL paths written in Arabic, Chinese, Japanese or Hindi.
The team needs to understand how its CMS stores these URLs and use matching paths in robots.txt. A rule that works for Google may not work in the same way for every other crawler.
E-commerce and marketplaces
Retail websites often use robots.txt to control crawling of filter combinations, internal search results, basket pages and account areas.
A single broken rule can allow crawlers to spend time on large numbers of filter URLs rather than the product and category pages that matter for search.
Retailers should also review the rules that apply to Google’s Storebot-Google crawler and Applebot. The former is associated with Google Shopping, while Applebot may follow Googlebot’s rules if it is not named separately.
Publishers and news websites
Publishers have to decide which AI training crawlers to allow, which AI search crawlers to allow and whether to use Google’s Search generative AI control.
They should also review crawler-specific groups. For example, a Googlebot-News group can replace the general Googlebot rules for Google News crawling, so it needs to contain the rules the team intends Google News to follow.
SaaS and B2B websites
Application, documentation, help-centre and status subdomains can each require their own robots.txt file.
Staging websites are another risk. A rule such as Disallow: / may be appropriate for a staging environment, but it can cause a major problem if the same file is accidentally deployed to production.
Staging sites should be protected with authentication rather than relying only on robots.txt.
Healthcare, finance and other regulated industries
Robots.txt is public. It is not a security feature.
Listing private folders in the file can tell other people where those areas exist. Protect private records, internal tools and sensitive information with proper authentication and authorisation.
Use robots.txt to manage crawling, not to protect confidential information.
Websites behind a CDN or bot protection
Firewalls, bot protection and CDN settings can affect how crawlers access robots.txt.
If Googlebot receives a 403 or 5xx response, the file may not be applied as intended.
Cloudflare’s managed robots.txt can also add rules to the file your team has created. Check the combined file that the CDN serves, rather than only the file stored on your server.
JavaScript and headless websites
A single-page application may return the same HTML shell with a 200 OK status for every unknown path.
If /robots.txt is handled by the same catch-all route, Google may receive the application’s HTML instead of the intended rules.
The file should be served directly by the server or CDN before the application handles other routes.
Large organisations
Robots.txt can become difficult to manage when SEO, development, security and marketing teams all make changes without one agreed process.
Common problems include contradictory rules, repeated crawler groups, copied instructions and invisible characters introduced by documents or editors.
Assign a clear owner and require changes to be checked before deployment.
Small websites
A standard CMS robots.txt file is often sufficient for a small website.
However, the basics still matter: make sure the file returns the correct status, decide how AI crawlers should be handled, and confirm that important pages are not accidentally blocked.
4. How to Audit and Fix Your Robots.txt File
The aim is to keep robots.txt short, clear and easy to check.
Every important host should serve the right file, someone should own it, and the business should have a written policy for AI crawlers.
The following six steps provide a practical way to achieve that.
Step 1: Find every robots.txt file your website serves
Owners: SEO lead and development team
Each robots.txt file applies to one host, protocol and port. A large website may therefore have many files rather than just one.
List all your hostnames, including:
wwwand non-wwwversions where both exist.- Subdomains such as
shop,blog,help,app,imagesandcdn. - Separate hosts used for international websites.
- Any hosts running on non-standard ports.
Internationalised domain names also need to be considered. Google treats a domain written with certain non-English characters and its punycode form as the same host.
For each robots.txt file, record:
- The HTTP status code.
- Any redirects.
- The content type.
- The file size.
- Who last changed it and why.
Then compare the results with Google’s view.
Use Search Console’s robots.txt report. Under Settings, it shows the files Google has found for your property, when they were last fetched and any problems reported when reading them. You can also request a fresh fetch when a change is urgent.
Check your server logs. These show what crawlers actually receive. Verify requests claiming to be Googlebot using reverse DNS checks or Google’s published crawler IP information. Do not trust the user-agent string alone, because it can be copied by other bots.
Step 2: Make sure the file is served correctly
Owner: Development team
Before editing the rules, confirm that the server or CDN is returning the right file.
| Check | What you want | Common fix |
|---|---|---|
| HTTP status | 200 OK for a real file, or 404 Not Found if no file is deliberately provided | Exempt robots.txt from bot challenges, rate limits, geographical blocks and maintenance pages |
| Redirects | No redirect, or a simple redirect where necessary | Serve the file directly from every host whenever possible |
| Content | Plain-text robots.txt rules | Add an explicit route before the JavaScript application’s catch-all handler |
| Content type | text/plain; charset=utf-8 | Set the correct type in the server or CDN configuration |
| Encoding | UTF-8 without a byte order mark | Save the file as UTF-8 and remove any BOM during the build |
| File size | Well below 500 KiB | Replace long lists of individual URLs with appropriate folder rules and wildcards |
| Update time | Changes reach Google after it fetches the new file | Plan changes ahead and request a fresh fetch through Search Console when urgent |
A 404 response can be appropriate if you deliberately have no robots.txt file and want no robots.txt restrictions. For a website with specific crawling rules, however, make sure the file is available and returns the intended content.
Do not allow bot protection to block access to the file. If Google cannot read it, your rules may not be applied.
Also check the response from the CDN, not just from the origin server. The two can differ.
Step 3: Keep the file simple and use the rules correctly
Owners: SEO lead and development team
Follow these rules when creating or updating robots.txt.
1. Use the fields Google supports.
Use user-agent, allow, disallow and sitemap. Remove unsupported fields and invented instructions.
Some other crawlers support additional directives such as crawl-delay, but Google ignores them.
2. Keep crawler groups clear.
Start with a general User-agent: * group where appropriate.
If you create a specific group for Googlebot or another crawler, repeat the general restrictions that crawler should still follow. Do not assume that a specific group automatically inherits the * rules.
3. Use the exact crawler names published by each company.
Write tokens in plain ASCII, using the correct letters, underscores and hyphens. Do not add emojis, descriptions or invented names.
4. Make sure paths match your actual URLs.
Start path rules with /. For non-English paths, use percent-encoded UTF-8 characters where appropriate. Check how your website actually represents the URL, because visually identical characters can have different encodings.
Write spaces as %20 when necessary and do not include URL fragments introduced by #.
5. Use wildcards carefully.
The * character matches a sequence of characters, while $ marks the end of a URL path when placed at the end of a rule.
URL paths are case-sensitive. The longest matching rule wins, and an allow rule wins when it is equally specific as a disallow rule.
6. Keep comments useful.
Use comments to record who owns the file, why a group exists and how changes should be approved.
Avoid adding large amounts of ASCII art or decorative text.
7. Put sitemap declarations in the right place.
Use a full sitemap URL, including the protocol and hostname. Put sitemap declarations clearly outside crawler groups, ideally at the end of the file.
A sitemap line placed between crawler definitions can affect how the rules are grouped by parsers.
8. Never list secret folders as a security measure.
Robots.txt is publicly accessible. Use authentication to protect private areas.
9. Do not block a page that needs to be removed from Google until Google can see the removal instruction.
If you want Google to process a noindex tag, leave the page crawlable until the tag has been read.
10. Do not block resources needed to display your pages.
Avoid blocking the CSS, JavaScript and other resources Google needs to understand the content.
Example of a simple robots.txt file
The following example shows one possible setup for a UK retail website. It is an illustration, not a recommendation that every website should block all AI training crawlers.
# robots.txt for https://www.example.co.uk/
# Owner: SEO team. Changes require an approved ticket.
# Last reviewed: 22 October 2026
User-agent: *
Disallow: /basket/
Disallow: /checkout/
Disallow: /account/
Disallow: /search
Disallow: /*?filter=
Disallow: /pl/zam%C3%B3wienia/
# AI training policy: approved by the business
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://www.example.co.uk/sitemap-index.xmlIn this example, the general group blocks selected basket, checkout, account, search, filter and Polish order URLs.
The training-crawler group blocks the named tokens from the entire site. Because those crawlers have their own group and are blocked from every path, there is no need to repeat the general group’s rules.
AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot are not included in the training group. They therefore follow the general rules, unless the vendors’ current documentation or another group changes the behaviour.
The decision to block training crawlers is a business choice. It should be made deliberately and reviewed alongside the effect on search visibility and the way your content may be used.
Step 4: Decide your AI crawler policy in writing
Owners: Leadership, marketing and legal
Robots.txt controls crawling, not every use of content. Each AI company may have separate crawlers for training, search discovery and requests made on behalf of individual users.
Decide what your business wants before adding rules.
| Business goal | What to consider | What will not achieve the goal |
|---|---|---|
| Appear in ChatGPT search answers | Allow OAI-SearchBot | Blocking GPTBot does not automatically block ChatGPT search; these are separate tokens |
| Keep content out of OpenAI model training | Disallow GPTBot | Blocking ChatGPT-User, which handles certain user-initiated fetches |
| Appear in Claude search results | Allow Claude-SearchBot and Claude-User, according to Anthropic’s current documentation | Blocking ClaudeBot, which controls a different activity |
| Keep content out of Anthropic model training | Disallow ClaudeBot | Assuming this also removes the website from Claude search |
| Appear in Perplexity results | Allow PerplexityBot | Relying on robots.txt to control user-initiated fetchers that may not follow it |
| Limit use of content for Gemini training and grounding in the Gemini app and Vertex AI | Disallow Google-Extended | Assuming it controls Google Search, AI Overviews or AI Mode |
| Stay out of Google’s AI Overviews, AI Mode and covered generative AI features in Discover | Use the Search generative AI control in Search Console | Blocking Googlebot, which can remove the site from Google Search altogether |
| Keep content out of Apple’s model training | Disallow Applebot-Extended | Assuming this prevents Applebot from crawling the website for other purposes |
| Keep content out of the Common Crawl dataset | Disallow CCBot | Assuming this stops every other crawler from accessing the same content |
| Stop crawlers that ignore robots.txt | Use suitable bot management, firewall rules and verified crawler lists | Robots.txt alone, because it relies on crawlers choosing to follow its instructions |
Vendor crawler names and policies can change, so check the latest documentation before deploying rules.
Before blocking AI search crawlers, measure what they contribute to your business. Our ChatGPT UTM tracking gap guide explains how to review AI referral traffic, while our answer capture ratio research looks at how often AI answers cite your website.
Review the policy with legal and marketing teams every quarter.
Step 5: Control changes to robots.txt
Owner: Development team
Robots.txt should be managed like other important configuration files. A small change can affect the entire website, so make sure it is reviewed and tested before release.
Keep the file in version control.
Store the file in your code repository and build the correct version for each environment.
Protect staging sites with authentication instead of relying only on Disallow: /. Make sure a staging file cannot accidentally replace the production file.
Check every change before deployment.
Run a linter and test a list of URLs that must remain allowed or blocked.
Your test list should include important pages such as the homepage, product and category pages, articles, CSS and JavaScript. It should also include URLs that should normally be blocked, such as basket, checkout, account, internal search and filter pages.
Monitor the live file after a release.
Check the file on every host. Set alerts if the status code changes, the content changes unexpectedly or the file size jumps.
Use Search Console’s robots.txt report and Crawl Stats after significant changes.
Check what the CDN serves.
A managed robots.txt feature, such as one provided by Cloudflare, may add rules to the file you have created. Review the combined file visitors and crawlers receive, not just the version stored in your repository.
Use the correct publishing method for your platform.
WordPress SEO plugins and Shopify’s robots.txt.liquid template may generate the file. Always check the result served by the live website rather than assuming the setting you changed is the final output.
Measure the effect.
If the rules block unnecessary crawling, requests for those paths should fall in your server logs over time. Search Console data exported to BigQuery can also help you monitor whether the visibility of important pages remains stable.
Step 6: Make responsibilities clear across the business
Robots.txt affects SEO, development, marketing, content and legal decisions. Each team needs to understand what it controls.
| Team | Main responsibilities | First actions |
|---|---|---|
| Content | Understand URL rules for new sections and languages. Avoid publishing important pages under blocked paths. | Tell development teams before launching a new section or language and agree how URLs will be written in each market. |
| Marketing | Own the AI crawler policy with legal and measure how the decision affects visibility. | Complete the AI policy table, make a deliberate decision about Google’s Search generative AI control and report AI referrals and citations monthly. |
| Development | Own the robots.txt file, the way it is served and checks before deployment. | Check every host, fix server responses, add the checker to the deployment process and exempt robots.txt from bot protection. |
| Leadership and legal | Approve the AI crawler policy and monitor relevant rules and commitments. | Sign off the policy and keep track of the CMA’s requirements for Google in the UK. |
5. Developer Toolkit: Check Your Robots.txt File
The original draft describes a checker called robots_check.py that can help developers test robots.txt files before deployment.
It is designed to do four things:
| Command | Purpose |
|---|---|
lint FILE-or-URL | Reports invalid characters, unsupported fields, invalid crawler names, path problems, group issues, oversized files and other potential mistakes. When given a URL, it also checks the response and redirects. |
test FILE --agent NAME URL... | Shows whether a URL is allowed for a particular crawler and which rule determines the result. |
ai FILE | Summarises how a list of AI crawler tokens is handled by the file. |
encode PATH | Shows percent-encoded versions of a path so that they can be compared with the URLs generated by the website. |
A deployment test can use commands such as:
python3 robots_check.py lint build/robots.txt
python3 robots_check.py test build/robots.txt \
--agent Googlebot \
--urls robots-expectations.txt
python3 robots_check.py test build/robots.txt \
--agent OAI-SearchBot \
--urls robots-expectations-ai-search.txtIn this setup, the build should fail when the linter finds a serious problem or when a URL behaves differently from what the test file expects.
The checker described in the original draft was tested against 50 path-matching cases from Google’s documentation, six rule-precedence scenarios, examples involving group selection and merging, and tests for common errors. The draft reports 62 checks in total.
Important implementation note: the source text supplied for this article includes the checker description, example output and parts of its matching code, but it does not contain the complete robots_check.py programme. The commands above are therefore examples of the intended workflow, not a standalone tool that can be run without the full script. Contact Szymaniak Digital for any questions!
Example: a robots.txt file with several hidden problems
Consider this example for an online shop:
# Acme Shop robots.txt
User-agent: *
Disallow: /basket/
Disallow: /*?filter=
Disallow：/account/
Disallow: /#checkout
Crawl-delay: 5
Noindex: /old-range/
User-agent: Googlebot
Allow: /blog/
User-agent: 🐔
Disallow: /chicken-diary/
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow: /pl/żółw-sklep/
No-AI-Training: /
Sitemap: /sitemap.xmlAt first glance, it looks as though the file blocks basket and filter URLs, the account area, a checkout page, an old range and several AI crawlers.
But several lines do not work as expected.
A checker based on Google’s documented rules should flag the following issues.
| Line or setting | Problem | What to change |
|---|---|---|
Disallow：/account/ | The colon is a full-width character rather than the normal ASCII colon. | Retype the line with Disallow: /account/. |
Disallow: /#checkout | The # starts a comment, so the rule effectively becomes Disallow: / and blocks the whole site for that group. | Remove the fragment and write the actual path that needs to be blocked. |
Crawl-delay: 5 | Google does not support this instruction. | Remove it if the only aim is to control Googlebot. Other crawlers may have their own support for this directive. |
Noindex: /old-range/ | Google ignores noindex in robots.txt. | Use a noindex tag on a crawlable page, or another suitable removal method. |
User-agent: 🐔 | This does not identify a valid crawler token. | Remove it or replace it with a real, published crawler name. |
Second User-agent: * group | Repeated groups can be handled differently by different parsers, and the file is harder to maintain. | Keep the general rules together in one clear group. |
Disallow: /pl/żółw-sklep/ | The path contains non-ASCII characters. Google can encode them, but other crawlers may behave differently. | Use the encoded form that matches the website’s actual URLs. |
No-AI-Training: / | This is not a supported Google robots.txt field. | Use the appropriate published crawler token or platform control. |
Sitemap: /sitemap.xml | The sitemap address is not a full URL. | Use the full URL, such as https://acme.example/sitemap.xml. |
The example demonstrates why robots.txt needs to be tested rather than judged only by how readable it looks.
A specific Googlebot group can change the outcome
The Googlebot group in the example contains only one rule:
User-agent: Googlebot
Allow: /blog/Googlebot uses the specific group rather than combining it with the general * group. As a result, the restrictions listed in the general group may not apply to Googlebot.
The other crawlers may follow the general group, including its unintended Disallow: / caused by the #checkout line.
This is the kind of mistake that can go unnoticed until server logs show crawlers visiting pages that the team believed were blocked.
How to read a robots.txt test report
The original draft reports that the example checker produced five errors, six warnings and two notes for the example file.
A test against Googlebot also showed that the basket and filter URLs were not blocked by the Googlebot group as intended. A test against Bingbot showed that the erroneous Disallow: / rule blocked the blog and product URLs.
The key lesson is to test the effective rules for each important crawler, not just the general * group.
The core matching logic uses three principles:
- Select the most specific user-agent group that applies to the crawler.
- Compare the URL path with the rules in that group.
- Use the longest matching rule, with
allowwinning when the matching rules are equally specific.
The original draft includes the following excerpts from that logic:
def canon(path):
"""Google's canonical form: non-ASCII as percent-encoded UTF-8."""
out = "".join(
c if ord(c) < 128
else "".join("%%%02X" % b for b in c.encode("utf-8"))
for c in path
)
return re.sub(
r"%([0-9a-fA-F]{2})",
lambda m: "%" + m.group(1).upper(),
out
)
def matches(pattern, target):
anchored = pattern.endswith("$")
core = pattern[:-1] if anchored else pattern
rx = "".join(
".*" if c == "*" else re.escape(c)
for c in core
)
return re.match(
rx + (r"\Z" if anchored else ""),
target,
re.S
) is not None
def decide(rules, target):
"""Longest matching rule wins; allow wins a tie."""
if target == "/robots.txt":
return True, None
best = None
for rule in rules:
if matches(rule.canon, target):
key = (len(rule.canon), rule.kind == "allow")
if best is None or key > best[0]:
best = (key, rule)
return (
(True, None)
if best is None
else (best[1].kind == "allow", best[1])
)These are excerpts, not a complete programme. The full checker described in the original draft would also need the file parser, crawler-group selection, lint rules, URL fetching and command-line interface.
Your developers should test the live robots.txt file against the most important URLs and crawlers, then compare the results with what Google reports in Search Console and what appears in server logs. A local test alone cannot show whether a firewall treats genuine crawlers differently.
6. What We’re Watching Next
Robots.txt was designed to tell crawlers which pages they should fetch. Businesses increasingly want to control what happens to their content after it has been fetched, especially when AI systems are involved.
Several developments could change how these controls work.
A possible standard for how content may be used
The Internet Engineering Task Force’s AI Preferences working group is developing proposals for expressing how online content may be used, rather than only whether a crawler may fetch it.
The original draft refers to two working group documents: one covering a vocabulary for content-use preferences, version 8 dated 13 September 2026, and another covering how to attach those preferences to content, version 5 dated 18 August 2026.
Earlier versions discussed ways to include content-use preferences in robots.txt and in HTTP headers.
These are proposals, not a published RFC, and Google has not documented support for them. For now, a new, unsupported line in robots.txt will be ignored by Google.
We will watch for the drafts to progress and for any announcements from Google about support. Until then, businesses should rely on the controls that the relevant platforms actually document.
The UK’s rules for Google’s AI features
The CMA’s conduct requirement of 3 June 2026 gave Google nine months to implement a range of changes.
These include controls for whether content appears in AI features such as AI Overviews, the ability to opt out of AI model fine-tuning following consultation, and clearer links to sources. Google is also expected to provide compliance reports every six months during the first year.
Google has said that the features covered by the Search generative AI control may change over time.
We will watch for more detailed controls and whether Google adds any new options beyond Search Console.
AI crawlers that do not follow robots.txt
OpenAI and Perplexity have said that some of their user-initiated fetchers may not follow robots.txt in the same way as search crawlers because a person has requested a particular page.
As AI assistants and agents fetch more content on behalf of users, robots.txt may control a smaller share of automated requests.
Businesses may need to rely more on bot management, verified crawler lists and server-side rules for traffic that does not follow robots.txt.
This makes cooperation between SEO, development and security teams increasingly important.
More crawler names and more opportunities for mistakes
Major AI companies use separate crawlers for training, search and user requests, and those arrangements can change.
One common risk is blocking AI search crawlers while trying to prevent model training. That can stop an assistant from citing a website or sending referral traffic, even though the business only intended to restrict training.
A quarterly review of the AI crawler policy is a simple way to reduce that risk.
Further guidance from Google
Google’s robots.txt documentation was last updated on 31 August 2026. The Non-Canonical Cabinet series is also continuing, with the next episode set to cover the 404 status code and how Google treats a robots.txt file that does not exist.
We will update this guide when Google publishes further guidance or changes how its parser works.
7. About Szymaniak Digital
Szymaniak Digital is a UK enterprise SEO and AI search consultancy.
We review robots.txt files and crawling behaviour for large and international websites, help marketing and legal teams decide how to handle AI crawlers, and work with developers to put checks in place before changes go live.
A single incorrect line can affect crawling across an entire website. The goal is to make sure the file is clear, serves the right rules to the right crawlers, and matches the business’s decisions about search and AI visibility.
If you are unsure what your robots.txt file currently tells Google, ChatGPT, Claude or Perplexity, we can help review the files served across your website, test the rules against important URLs and explain the available AI crawler choices in plain English. Contact Szymaniak Digital to discuss a review.
