Google is developing AI agents that can browse websites and carry out tasks for people. These tasks can involve searching for products, comparing options, checking availability, making bookings and adding items to a basket.
But there’s a problem. Many websites use firewalls and bot protection to block automated traffic. These security systems can sometimes block Google’s agents as well as unwanted bots.
On 17 November 2026, Google published new guidance explaining how website owners can allow its Google-Agent traffic through a firewall, content delivery network (CDN) or security system without removing their existing protection.
I used this guidance as the basis for a review of an anonymised client website. The main question was straightforward: could a genuine Google agent access the website and complete the actions a customer might ask it to perform, while requests pretending to be Google-Agent remained blocked?
This article explains what Google-Agent is, how to recognise genuine requests and how I would approach the same checks on your website.

1. What Is Google-Agent?
Google-Agent is the name Google uses for agents running on its infrastructure that browse websites and carry out actions when someone asks them to.
Unlike Googlebot, which crawls websites for Search, Google-Agent is a user-triggered fetcher. It visits a website because a person has asked a Google product to do something.
For example, someone might ask an AI assistant to find a product, compare prices, check availability or make a booking. The agent may need to visit several pages and interact with the website to complete the task.
This is an important difference for website owners.
Google’s user-triggered fetchers generally ignore robots.txt because their visits are initiated by a user. That means robots.txt isn’t the right way to control Google-Agent traffic. Your CDN, firewall and bot protection settings are where you need to manage access.
How Google-Agent identifies itself
Google provides three ways to check whether a request is genuine.
| Method | What to check | How reliable is it? |
|---|---|---|
| User-Agent | The request header contains Google-Agent. Google has mobile and desktop versions. | Easy to check, but anyone can copy the text. Never trust it on its own. |
| IP address | The request comes from an IP range listed in Google’s user-triggered-agents.json file. | A strong way to check that the request comes from Google’s published infrastructure. |
| Web Bot Auth | A cryptographic signature identifies the request as coming from agent.bot.goog. | Provides stronger verification, but Google is still testing it and only some requests are signed. |
The important point is that you should not allow a request simply because its user agent contains the word Google-Agent.
A scraper can use exactly the same text.
Google recommends checking the user agent against its published IP ranges. Where a request also includes a valid Web Bot Auth signature, you can verify that as an additional check.
Google’s published IP ranges
Google provides a JSON file called:
user-triggered-agents.json
It contains the IP ranges used by Google-Agent.
On 18 November 2026, the file contained 20 network ranges:
- 12 IPv4 ranges.
- 8 IPv6 ranges.
These ranges can change, so don’t copy them into a firewall rule and assume they’ll remain correct indefinitely.
I recommend checking Google’s published file regularly and updating your firewall or CDN allowlist whenever the ranges change.
What is Web Bot Auth?
Google is also testing a protocol called Web Bot Auth.
The idea is to give automated agents a way to prove who they are using a cryptographic signature, rather than relying only on a user-agent string and IP address.
Google’s experimental implementation uses the identity:
https://agent.bot.goog
A participating request includes three HTTP headers:
Signature-Agent: g="https://agent.bot.goog"
Signature: ...
Signature-Input: ...
The Signature and Signature-Input headers contain the information needed to verify the request.
The process uses HTTP Message Signatures, published as RFC 9421. Google also provides a public key set that websites can use to check the signatures.
Web Bot Auth is still experimental. Google doesn’t sign every Google-Agent request, so you shouldn’t rely on it as your only verification method.
For now, Google’s recommendation is to continue checking IP addresses, reverse DNS and user-agent strings alongside any signature verification you implement.
Why are websites blocking Google-Agent?
In most cases, businesses aren’t deliberately blocking genuine Google agents. The problem comes from security settings designed to stop unwanted automated traffic.
Here are some common examples.
1. A setting that blocks AI bots
Some CDNs and security plugins provide options to block AI crawlers or automated agents.
These settings can catch Google-Agent requests if the system doesn’t distinguish between an agent acting for a person and a bot scraping a website.
2. Blocking entire cloud or data centre networks
Some firewalls block traffic from broad groups of IP addresses, including cloud and data centre networks.
Google-Agent uses Google’s infrastructure, so these rules can block genuine requests as well.
3. CAPTCHAs and JavaScript challenges
A website might allow an agent to load the homepage but then show a CAPTCHA when it tries to search, filter products, submit a form or reach checkout.
The agent may not be able to complete that step, even though the first page loaded successfully.
4. Rate limits based on IP address
Google’s agents can use shared IP addresses for requests made on behalf of different users.
A rate limit designed for one visitor may therefore affect several customers using agents at the same time.
5. Proxy and load balancer settings
A load balancer or proxy can change the information passed between the visitor and your website.
If it removes the original IP address or important request headers, your security system may no longer have enough information to verify Google-Agent.
2. Why Does Allowing Google-Agent Matter?
Think about the difference between someone asking an AI assistant for information and asking it to complete a task.
If someone asks for three product recommendations, the assistant may only need to find and read the relevant pages.
But if the person asks it to check stock, compare prices or make a booking, the agent needs to interact with the website.
It may need to search, use filters, open product pages, fill in forms or add something to a basket.
If your firewall blocks the agent halfway through, the task may fail.
The customer doesn’t see your firewall settings. They simply see that the assistant couldn’t complete the request.
Where a firewall can interrupt the customer journey
| What goes wrong | What it means for the customer | Who is likely to notice? |
|---|---|---|
| A blanket AI bot setting blocks Google-Agent | The agent can’t access the website or start the task. | Often nobody, unless the failure is reported. |
| The homepage loads but search or checkout is blocked | The agent starts the task but cannot complete the important step. | Customer service may receive complaints about failed bookings or purchases. |
| Per-IP rate limits are too restrictive | Several customers’ agents may be blocked because they share Google’s IP pool. | Development teams may notice an increase in rate-limit responses. |
| The firewall trusts the user agent alone | Other bots can pretend to be Google-Agent and avoid security checks. | The security team may notice unexpected traffic or scraping. |
| Google’s IP list becomes outdated | Genuine requests from newly added IP ranges may be blocked. | This can go unnoticed until someone checks the firewall or agent activity. |
Allowing Google-Agent doesn’t mean turning off your security
I wouldn’t recommend switching off bot protection to make a website accessible to AI agents.
The better approach is to create a specific exception for requests that you can verify are genuinely from Google-Agent.
The normal security rules should continue to apply to other visitors and automated traffic.
For example, a request should only receive special treatment when its user agent and IP address match Google’s published information. If it also has a Web Bot Auth signature, that can provide an additional check.
You should also make sure the exception covers the actions the agent needs to perform, not just the initial page load.
Google’s new agent features make this more relevant
On 17 November 2026, Google announced that Tasks in Gemini was becoming available free of charge across six countries: the United States, Japan, South Korea, Canada, Australia and India.
Google described tasks such as scheduling appointments, managing bills and shopping. Its announcement also explained how Gemini can call businesses, check options and help users complete purchases.
Google hasn’t explicitly confirmed that every Tasks feature uses Google-Agent, so I wouldn’t assume that all of this activity will appear in your server logs under that name.
However, it shows the direction in which AI assistants are moving: from answering questions to taking action on behalf of users.
For businesses, that makes it worth checking whether their websites allow verified agents to access the journeys that matter.
Googlebot and Google-Agent are different
This is important for SEO and development teams.
| Googlebot | Google-Agent |
|---|---|
| Crawls websites for Google Search. | Visits websites to carry out actions requested by users. |
| Helps Google discover and understand pages for its index. | Helps an AI agent browse and interact with a website. |
| Follows robots.txt rules. | As a user-triggered fetcher, generally ignores robots.txt. |
| Uses Google’s published crawler IP ranges. | Uses the separate user-triggered-agents.json IP ranges. |
| Needs access to pages and resources for crawling and rendering. | Needs access to the pages, forms, searches and actions required to complete a task. |
Being accessible to Googlebot doesn’t automatically mean your website is accessible to Google-Agent.
You need to review both separately.
3. Which Businesses Are Affected?
Any business using a CDN, firewall, managed hosting security or a bot-protection plugin could be affected.
The impact is likely to be greater for websites where customers need to complete actions rather than simply read information.
| Industry | What an AI agent may need to do | Where security can block it |
|---|---|---|
| Ecommerce and retail | Search products, apply filters, compare prices, check stock and add products to a basket. | Bot rules on search and product filters, or CAPTCHAs during checkout. |
| Travel, hotels and restaurants | Check availability, compare options and make a booking or reservation. | Booking systems hosted on another domain, or rate limits during busy periods. |
| Healthcare and local services | Find a provider, check services, book an appointment or submit an enquiry. | CAPTCHAs on forms and security plugins on WordPress websites. |
| Finance and insurance | Compare products, use calculators and request quotes. | Strict IP restrictions and JavaScript challenges on quote tools. |
| SaaS and B2B | Find pricing, compare features, search documentation and book a demonstration. | Demo forms, API endpoints and GraphQL requests behind bot protection. |
| Publishers | Access articles and search the website. | Paywalls and anti-scraping rules that cannot distinguish a genuine agent from a scraper. |
Which website setups are most at risk?
I would pay particular attention to four situations.
Websites with a blanket AI bot blocking setting
A one-click setting in a CDN or security plugin can block several types of automated traffic. Check whether Google-Agent is included in the blocked category.
Websites that use third-party booking or checkout platforms
Your own firewall rules won’t necessarily control the booking system or payment provider.
If an agent needs to use a separate booking engine or subdomain, that provider may need to allow verified Google-Agent requests too.
Websites with several layers of security
A website might use Cloudflare, a hosting firewall and a WordPress security plugin.
Allowing Google-Agent through one layer doesn’t help if another layer blocks it afterwards.
You need to check every system that can challenge or refuse the request.
Websites behind a load balancer or proxy
If your infrastructure doesn’t preserve the original visitor IP address and the relevant headers, your firewall may not be able to verify Google-Agent correctly.
This needs to be checked before creating an allow rule.
4. How to Let Google-Agent Through Your Firewall
I would approach this as a controlled security change, not a request to allow all AI bots.
The aim is to identify genuine Google-Agent requests, allow them through the necessary security checks and make sure everything else remains protected.
Here is the process I would follow with a client’s development, hosting and security teams.
Who should be involved?
| Team | Main responsibility |
|---|---|
| Development and IT security | Review firewall rules, CDN settings, bot protection, rate limits and proxy headers. |
| Content | Identify the pages, forms and journeys an agent needs to use. |
| Marketing and SEO | Decide which customer journeys matter and monitor whether agents can access them. |
Step 1: Check what happens to Google-Agent requests today
Owner: Development and SEO · Day 1
Start by reviewing your website’s access logs or CDN logs for requests containing Google-Agent in the user-agent header.
For each request, record:
- The date and time.
- The requested URL.
- The user-agent string.
- The source IP address.
- Whether the request was allowed, blocked, challenged or rate limited.
- The response status.
- Which stage of the customer journey the request reached.
Compare the source IP addresses with Google’s published user-triggered-agents.json ranges.
Separate requests from verified Google IP addresses from requests that only claim to be Google-Agent.
Also check your CDN or firewall event logs. A request blocked at the edge may never reach your website’s own server logs.
The aim is to understand where the agent is getting stopped.
For example, can it load the homepage but not use the search function? Can it open a product page but not add the item to a basket?
Those details will help you identify the rule that needs changing.
Step 2: Check whether your security provider already verifies Google-Agent
Owner: Development · Day 1
Before writing your own rules, check whether your CDN, firewall or bot-management provider already supports Google-Agent verification.
Some security providers can recognise verified Google agents and allow them through their protection systems.
Some are also beginning to support Web Bot Auth.
If your provider offers a verified-bot option, use that rather than building your own verification system unnecessarily.
Check the provider’s documentation to understand exactly what the setting allows. You still need to make sure it skips the relevant bot challenges and applies to the pages and actions your agents need to access.
Step 3: Create an allow rule using both the user agent and IP address
Owner: Development · Week 1
If your provider doesn’t verify Google-Agent automatically, you may need to create your own rule.
Google’s guidance recommends checking two things:
1. The user-agent header contains Google-Agent.
This identifies the type of request being made.
2. The source IP address is included in Google’s published user-triggered-agents.json file.
This confirms that the request comes from Google’s published IP ranges.
Both checks are important.
Don’t create a rule that allows every request containing Google-Agent in its user-agent string. Anyone can copy that text.
If a request also includes Web Bot Auth headers, verify the signature using Google’s public keys. A failed signature should never be treated as proof that a request is genuine.
Remember that not every Google-Agent request is signed, so IP-based verification is still needed.
Step 4: Make sure verified requests don’t receive CAPTCHAs or other challenges
Owner: Development · Week 1
This is an easy part to miss.
An allow rule might stop a firewall returning a 403 response, but the agent could still encounter a JavaScript challenge or CAPTCHA.
For a verified Google-Agent request, check that the relevant rules also skip:
- JavaScript challenges.
- Interactive CAPTCHAs.
- Bot challenges.
- CAPTCHA widgets that appear inside search or form submissions.
For example, an agent might be able to access your homepage but get stuck when it submits a search or enquiry form.
The rule needs to cover the action, not just the initial page request.
Step 5: Put the allow rule before broad blocking rules
Owner: Development · Week 1
Check the order in which your security rules run.
Google specifically recommends making sure the verified Google-Agent rule is evaluated before broader rules that might otherwise block it.
These could include:
- Cloud or data centre IP blocks.
- Managed bot protection rules.
- Rules that block AI agents.
- General rate-limiting policies.
If you use several layers of security, add the necessary exception at each one.
An allow rule in your CDN won’t help if your hosting firewall or WordPress security plugin blocks the same request afterwards.
Step 6: Test the complete customer journey
Owner: Development and content · Weeks 1–2
Don’t stop after checking whether the homepage loads.
An agent may need to visit several pages and interact with different website components.
For an ecommerce website, I would test:
- Homepage access.
- On-site search.
- Product filtering.
- Opening product pages.
- Checking price and stock.
- Adding a product to the basket.
- Accessing the checkout process.
- The underlying XHR, GraphQL or API requests.
For a service business, the tests might include:
- Finding a location.
- Checking availability.
- Opening a service page.
- Completing an enquiry form.
- Starting an appointment booking.
If the website uses a third-party booking or payment provider, ask that provider whether it supports verified Google-Agent requests too.
You want to identify the entire journey, including any system outside your own website.
Step 7: Review rate limits and proxy settings
Owner: Development · Week 2
Google points out that multiple users’ agents can share the same Google IP pool.
This can create problems with per-IP rate limits.
For example, a rule designed to allow a certain number of requests from one IP address might block legitimate traffic when several users’ agents are using that same address.
Where appropriate, exempt verified Google-Agent requests from these limits or set a suitable higher threshold.
Also check the information passed through your load balancers and proxies.
They need to preserve:
- The original visitor IP address.
- The
User-Agentheader. - The
Signatureheader, where provided. - The
Signature-Inputheader, where provided. - The
Signature-Agentheader, where provided.
If these headers are removed or changed, your firewall may not be able to verify the request.
Don’t change proxy settings without checking the trusted proxy configuration. You should only trust original IP information supplied by systems you control.
Step 8: Keep Google’s IP ranges up to date
Owner: Development · Ongoing
Don’t rely on a static copy of Google’s IP ranges.
Google updates its user-triggered-agents.json file, so an allow rule can become outdated.
I recommend automating the process:
- Download Google’s latest file on a regular schedule.
- Compare it with the previous version.
- Identify any new or removed IP ranges.
- Update your CDN, firewall or server allowlists.
- Log the change so the team knows when the rules were updated.
At a minimum, check the list weekly.
If you’re managing several websites, make this part of your central security maintenance process.
Step 9: Check that genuine requests get through and fake requests don’t
Owner: SEO and development · End of week 2
The final test needs to check both sides of the rule.
Genuine requests must be allowed.
Review the logs and confirm that verified Google-Agent requests can access the pages and actions you tested.
Fake requests must remain blocked.
Send a test request using a browser or tool that copies the Google-Agent user-agent string but comes from an ordinary IP address outside Google’s published ranges.
It shouldn’t receive the special treatment given to verified Google-Agent traffic.
This is a useful way to check that your rule isn’t trusting the user agent alone.
Don’t carry out tests that place real orders, make bookings or submit live customer information. Use safe test pages and controlled journeys.
5. Example Firewall Rules
The exact configuration depends on your security provider. The following examples show the general approach, but you should check the current documentation for your platform before deploying anything.
Cloudflare
One approach is to create an IP list containing Google’s published Google-Agent ranges and then use a custom rule to check both the user agent and source IP.
An illustrative expression is:
(http.user_agent contains "Google-Agent" and ip.src in $google_agent_ips)
Here, google_agent_ips represents the IP list you’ve populated with Google’s current ranges.
The action should use Cloudflare’s appropriate Skip or verified-bot controls for the rules that would otherwise block the request.
Check that the exception covers the relevant challenges, managed rules and rate limits. Put it in the correct position so a broader blocking rule doesn’t override it.
Don’t use a blanket setting that disables unrelated security checks.
AWS WAF
For AWS WAF, create an IP set using Google’s published ranges.
The rule should check both:
- Whether the source IP matches the IP set.
- Whether the user-agent header contains
Google-Agent.
You then need to consider the rule priority and action carefully.
An Allow action can stop further rule evaluation, potentially bypassing more protection than you intended. Work with your security team to choose the right rule order and exception so verified agents can pass the necessary bot checks without removing unrelated protections.
Nginx and Apache
If you manage your own server, the verification needs to happen before the relevant blocking or rate-limiting rules.
For Nginx, the configuration should only mark a request as verified when both the user agent and source IP match Google’s published information.
For Apache, the same approach can be used to set an environment variable for verified Google-Agent requests, which your security rules can then recognise.
If your website sits behind a load balancer, configure the trusted proxy and real visitor IP settings first.
For example, an Nginx configuration may use:
set_real_ip_from 10.0.0.0/8;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
The IP range above is only an example. Replace it with the actual trusted addresses of your own load balancers. Never trust forwarded IP headers from arbitrary sources.
I wouldn’t copy a server configuration from another website without adapting and testing it. A mistake in these rules can affect every visitor, not just Google-Agent.
6. Who Does What?
Allowing Google-Agent through a firewall isn’t just a development task. The website also needs to provide clear information and working journeys for the agent to use.
I would split the work across four teams.
| Team | What they should do | How to check the work is complete |
|---|---|---|
| Development and security | Review logs, configure verification, skip challenges for genuine requests, adjust rate limits and maintain Google’s IP ranges. | Verified Google-Agent requests can complete the agreed journeys, while fake requests remain subject to normal protection. |
| Content | Identify the pages and customer journeys agents need. Make prices, stock, availability, opening hours and policies clear. | The agent can find the relevant information, and it matches what a human visitor sees. |
| Marketing | Decide which agent-led journeys matter and monitor requests and business outcomes. | Regular reports show requests, successful journeys and any bookings or orders that can be attributed to agents. |
| SEO | Check the implementation before and after firewall changes. Keep Googlebot crawling rules separate from Google-Agent access rules. | Each release is tested, and the rules for search crawlers and user-triggered agents are documented separately. |
Don’t forget the content on your website
Allowing an agent to access your website is only half the job.
The agent still needs to find clear, accurate information.
For ecommerce websites, check that product prices, stock, delivery information and product details are easy to access.
For service businesses, make sure opening hours, availability, locations, prices and booking instructions are clear.
Keep structured data such as Product, Offer and LocalBusiness accurate and consistent with the information people can see on the page.
An agent should not need to guess what a business offers or how much a service costs.
7. The Toolkit: Google Agent Access Checker
I’ve also put together a Python script called google_agent_access.py to help with the checks.
It runs on Python 3 and doesn’t require additional packages.
The script has four commands.
The ranges command
The ranges command downloads Google’s current user-triggered-agents.json file and compares it with the previous version.
It identifies IP ranges that have been added or removed and produces files that can be used when configuring:
- A standard IP allowlist.
- Nginx.
- Apache.
- CDN IP lists in CSV format.
Run it regularly so your security rules don’t rely on an old copy of Google’s IP ranges.
The logs command
The logs command reads Nginx and Apache access logs, including compressed .gz files, and CSV exports from supported CDN logs.
It separates requests that match Google’s published Google-Agent IP ranges from requests that only use the same user-agent string.
It also records signed requests and shows which requests were:
- Served successfully.
- Blocked.
- Challenged.
- Rate limited.
Where the log data allows, it groups the results by the different stages of a customer journey, such as the homepage, search, filtered listings, product pages, basket, checkout, booking forms and APIs.
The probe command
The probe command helps test whether your website is trusting the user-agent string alone.
It requests selected pages using a normal browser and a simulated Google-Agent user agent from your own computer.
A fake request should not receive the same special treatment as a verified Google-Agent request because its IP address won’t match Google’s published ranges.
The ip command
The ip command checks an individual IP address against Google’s published ranges.
This is useful when investigating a request in your server or CDN logs.
Running the script
Use the following commands in your terminal:
python3 google_agent_access.py ranges
python3 google_agent_access.py logs access.log
python3 google_agent_access.py probe urls.txt
python3 google_agent_access.py ip 136.122.10.20
The script is designed to make the checks easier to repeat. You should still review the output and confirm the firewall changes with your development or security team before deploying them.
8. What We’re Watching Next
Google-Agent and other AI agents are changing the way websites receive automated visits.
Until recently, many businesses treated most automated traffic as something to block. That approach becomes more complicated when an agent is acting on behalf of a real customer.
There are several developments I’ll be following.
How many Google-Agent requests are completing customer journeys?
The important measure isn’t simply the number of agent requests.
I want to understand how many can access the website, use search, view products or services and reach the booking or checkout stage.
Server and CDN logs are particularly useful here. Don’t assume that your normal analytics reports will capture every agent request or action. Check how your own analytics setup handles this traffic.
Will Web Bot Auth become more widely used?
Google is currently signing only some Google-Agent requests.
If adoption increases, cryptographic signatures could make it easier to identify genuine agents without relying entirely on IP ranges.
For now, keep the established verification methods in place and monitor changes to the Web Bot Auth specification.
Which Google products use Google-Agent?
Google hasn’t published a complete list of the products and features that generate Google-Agent requests.
The launch of Tasks in Gemini shows that Google is giving its AI assistant more ways to complete activities on behalf of users.
I’ll be monitoring client logs to understand how much Google-Agent traffic reaches websites and which parts of the customer journey it accesses.
Will Google’s IP ranges change?
Google’s published file contained 20 IP ranges on 18 November 2026.
I’ll continue checking for additions and removals so that allowlists remain up to date.
How will CDNs and security plugins respond?
I expect more security providers to offer ways to recognise verified AI agents.
There is also a need to distinguish between an agent acting for a person and a crawler collecting information.
The two types of automated traffic have different purposes, and website owners may want to treat them differently.
What about other AI assistants?
Google isn’t the only company developing agents that browse websites and carry out tasks.
Other AI platforms are also working on ways to identify their automated traffic.
The same approach will be useful as more providers publish IP ranges, verification methods or cryptographic keys.
The objective is to recognise genuine agents, decide which journeys they should be allowed to complete and keep the rest of your website protected.
9. About Szymaniak Digital
Szymaniak Digital is an Enterprise AI SEO Consultancy led by Konrad Szymaniak.
We research how Google and other AI platforms access websites, find information and interact with online businesses.
Our work brings together technical SEO, AI search visibility and website development. We help businesses understand whether search crawlers and AI agents can access the pages and features that matter to their customers.
The important thing is not to allow every bot through your firewall. It’s to understand which automated visitors are genuine, what they need to access and how to let them through without weakening your website’s security.
Konrad Szymaniak
Enterprise AI SEO Consultant
