Chrome DevTools can now show which advertising scripts are slowing down your pages. Its new Ads section in the Application panel lists advertising scripts, measures their CPU and network use, highlights ads on the page and displays four experimental ad metrics collected through the Chrome User Experience Report (CrUX).
This is useful for improving page performance. However, publishers and SEO teams have a different question: can advertising also affect how often Google crawls their websites?
Could advertising scripts cause Googlebot to crawl fewer pages, or take longer to revisit important URLs?
The answer depends on where the cost of serving those ads falls. If the work happens entirely on an advertising network’s servers or in a visitor’s browser, it may have little or no direct effect on your site’s crawl budget. If it increases the work your own servers must do, or makes your pages slower to respond to Googlebot, the situation could be different.
Google’s documentation does not say that advertising automatically reduces crawl budget. The only reliable way to establish whether your advertising setup affects crawling is to test it.
This guide explains what Chrome DevTools can tell you, how Google manages crawl budget, and how to run a controlled experiment using server logs, DevTools and Search Console.
It also includes an example of a Python analysis tool to help compare crawling before and after a change.
1. What Has Changed in Chrome DevTools?
Google’s October 2026 New in DevTools round-up, covering Chrome 153 and 154, introduced an Ads section in the Application panel.
This follows the addition of four experimental advertising metrics to the Chrome User Experience Report, announced on 15 September 2026.
Chrome now releases updates twice a month, while DevTools also provides monthly round-ups of its latest features.
What the New Ads Section Shows
The Ads section brings together several details about advertising on a page.
- A dashboard displaying the four experimental CrUX advertising metrics.
- CPU and network usage broken down by individual ad elements.
- A list of advertising scripts running in the main frame, with links that let you open them in the Sources panel.
- A control that highlights advertising elements on the page.
This makes it easier to identify which scripts use the most resources and which parts of the page contain advertising.
For publishers, the information can help identify opportunities to improve page performance without immediately removing advertising altogether.
The Four Experimental CrUX Advertising Metrics
| Metric | What it measures |
|---|---|
| Ad count | The average number of ads visible in the viewport at the same time. |
| Ad density | The average proportion of the viewport covered by ads. |
| Ad weight: network | The number of bytes of network resources used by ads. |
| Ad weight: CPU | The amount of processing time, measured in milliseconds, used by ads. |
Google says these metrics are not part of Core Web Vitals and do not have recommended targets or thresholds.
They are experimental metrics collected from real Chrome users, like other CrUX data.
They provide additional information about advertising performance, but they are not official SEO scores and do not directly measure crawl budget.
How Google Calculates Crawl Budget
Google’s crawl budget documentation focuses mainly on very large or frequently changing websites.
It gives rough examples of sites that may need to think about crawl budget: websites with around a million or more pages that change weekly, or websites with at least 10,000 pages changing daily.
Google describes crawl budget in terms of two main factors.
Crawl capacity limit
This is the amount of crawling Google’s systems can carry out without overloading your server.
It can increase when your server responds quickly and consistently. Slow responses, server errors such as HTTP 5xx responses and rate-limiting responses such as HTTP 429 can reduce the rate at which Google crawls your site.
Crawl demand
This is how much Google wants to crawl your website.
It depends on factors such as the number of pages, how frequently they change, their quality and relevance, and their popularity.
Even if your server can handle more crawling, Google will not necessarily use that capacity if there is little demand for your pages.
Google also explains that faster loading and rendering may allow it to read more content from a website.
Another important detail is that Google treats each hostname as a separate site for crawl budget purposes.
In its December 2024 Crawling series, Google also explained that its Web Rendering Service tries to cache JavaScript and CSS resources it fetches. These resources can remain cached for up to 30 days, regardless of your HTTP caching rules.
Google recommended serving resources from a different hostname, such as a CDN, and being careful with cache-busting parameters that prevent cached files from being reused.
Google’s documentation does not say that advertising itself reduces crawl budget. The question is whether the way your site serves advertising increases the number of requests hitting your own hostname, slows responses to Googlebot or affects how efficiently Google can render your content.
That is what the experiment below is designed to test.
2. Why Does This Matter?
Advertising generates revenue for many publishers, so claims that ads are hurting SEO should be tested rather than accepted without evidence.
The key question is where the cost of serving the ads falls: on the visitor’s device, on the advertising network’s servers or on your own servers.
Only some of these costs are likely to affect your site’s crawling directly.
Where the Advertising Cost Falls
| Where the cost falls | Example | Could it affect your crawl budget? |
|---|---|---|
| The visitor’s device | Ad scripts using CPU, causing layout shifts or slowing interactions | Not directly. These problems affect user experience, but Googlebot’s page fetch does not automatically have to wait for the same work that a visitor’s browser performs. |
| Advertising network hostnames | Scripts and ad creatives served directly from the network’s own domains | Generally not your site’s crawl budget. Google treats each hostname as a separate site, so requests to the network’s hostname count against that hostname rather than yours. |
| Your own hostname | Ad configuration files, header-bidding wrappers, first-party ad proxies, refresh calls and self-hosted scripts | Potentially. Requests to your hostname use its crawl capacity, and cache-busting parameters can prevent Google’s rendering service from reusing cached resources. |
| Your server’s response time | Server-side ad decisions, personalisation or ad insertion that delays the HTML response | Potentially. Slower responses can reduce the crawl capacity available to Googlebot. |
| Page rendering | Heavy scripts competing for resources needed to render content | Possibly. Google says faster loading and rendering may allow it to read more content from your site. |
These differences explain why reducing the amount of advertising JavaScript in a visitor’s browser does not automatically free up crawl budget.
A script can be expensive for the visitor without putting any additional load on your own server during Google’s initial request for the page.
Likewise, moving an advertising resource to a separate hostname will only help if that change actually reduces the crawling or server work affecting your main hostname.
Why This Needs an Experiment Rather Than Guesswork
There are several reasons why simply removing ads and watching what happens is not enough.
Other factors can explain the results. Websites with many ads are often large, frequently updated sites with other crawling problems, such as faceted navigation, duplicate URLs or low crawl demand.
Before-and-after comparisons can be misleading. Google rankings and crawling have been unusually volatile since late September 2026. If crawling increases after you change your advertising setup, the improvement might be caused by wider changes in Google’s behaviour rather than your intervention.
Removing ads can cost revenue. Publishers need evidence that a change provides enough benefit to justify any financial impact.
Crawl changes may take weeks to become clear. Google’s crawl capacity can fall quickly but may take much longer to recover. This means a short test lasting only a few days may miss the effect you are trying to measure.
A controlled experiment helps separate the effect of your advertising change from other things happening at the same time.
Why It Is Worth Finding Out
For large publishers, retailers and marketplaces, crawling affects how quickly new and updated pages can be discovered and refreshed.
If unnecessary requests or slow server responses are limiting crawl capacity, important changes to articles, prices and stock availability may take longer to be reflected in Google’s systems.
On the other hand, if your advertising setup has no measurable effect on crawling, that is also a useful finding. It means your team can focus on advertising changes that improve user experience and revenue rather than spending time trying to solve a crawl problem that does not exist.
3. Which Websites Are Most Likely to Be Affected?
The experiment is most useful for websites where both advertising and crawling are significant operational concerns.
Ad-Funded Publishers and News Websites
Publishers often have large websites, frequent content updates and complex advertising setups.
Header bidding, refreshed ad slots and advertising endpoints running on the publisher’s own hostname could all be worth investigating.
This group also has the most revenue at stake if an advertising change affects earnings.
Recipe, Review, Comparison and Affiliate Websites
These websites often have long pages with ads placed between content sections.
They may benefit from checking how much advertising work happens in the browser compared with how much is handled by their own servers.
Retailers and Marketplaces with Retail Media
Retailers increasingly run sponsored product placements and advertising systems alongside their main website.
If their own servers or hostnames handle advertising requests, there may be a reason to investigate whether those requests affect crawling.
However, advertising scripts served entirely by external networks should not automatically be treated as a drain on the retailer’s crawl budget.
Forums and Community Websites
Forums may contain hundreds of thousands of pages, with advertising displayed across threads and discussions.
The combination of a large URL inventory, relatively low demand for some pages and advertising across multiple templates makes them worth investigating if crawl capacity is already a concern.
Large Websites Already Struggling with Crawling
If Google has discovered large numbers of URLs but is not indexing them, investigate the cause before blaming advertising.
For example, a site with many URLs in the Discovered – currently not indexed category may have problems involving crawl demand, duplicate content, site quality or internal URL structures.
Advertising is only one possible factor to test.
Smaller Websites
Google says that most websites below its rough crawl budget thresholds do not need to worry about crawl budget.
For smaller businesses, the new advertising metrics are more likely to be useful for understanding user experience, page performance and revenue than for investigating crawling.
4. How to Test Whether Ads Affect Your Crawl Budget
Run the experiment before changing your advertising strategy for SEO reasons.
The proposed test takes six weeks: two weeks to establish a baseline, one day to make the change, and four weeks to measure what happens afterwards.
It requires server logs and two comparable groups of pages.
The important principle is to change one thing in one group while leaving a similar group unchanged. This gives you a control group against which to measure the result.
The Experiment at a Glance
| Element | Plan |
|---|---|
| Main question | Does reducing advertising-related work on our own servers increase how often Googlebot crawls our pages? |
| Hypothesis 1 | Moving first-party advertising requests away from the main hostname and removing unnecessary cache-busting parameters frees capacity for page crawling. |
| Hypothesis 2 | Removing advertising work from the server’s response process reduces response times to Googlebot, allowing more crawling. |
| Useful null result | Changes that only reduce advertising work in the visitor’s browser have no measurable effect on crawling. |
| Design | Two matched groups of pages: change one group and leave the other unchanged. Compare the difference between the groups before and after the change. |
| Duration | Two weeks of baseline data, one day for the change, then four weeks of measurement. |
| Primary metric | Googlebot HTML page requests per URL per day in each group. |
| Secondary metrics | The share of Googlebot requests used by resources, response times to Googlebot and the time taken for updated pages to be recrawled. |
| Guardrails | Advertising revenue per 1,000 sessions, Core Web Vitals and the four experimental CrUX advertising metrics. |
| Success criteria | Set these before the test. The proposed threshold is at least a 10% relative increase in page crawling compared with the control group, with a 95% confidence interval above zero and revenue falling by no more than 3%. |
The thresholds above are proposed criteria for this experiment, not standards set by Google.
Step 1. Measure the Cost of Your Advertising Setup
Who: Advertising operations and developers
Before changing anything, establish how advertising currently works on your website.
1. Inspect representative pages in Chrome DevTools.
Open representative pages from each template and go to Application > Ads.
Enable ad highlighting and review the advertising elements on the page.
2. Record the advertising scripts.
Export or record the available advertising script details. For each script, note its hostname, CPU use, network use and whether it appears across every page.
This helps identify resources that could affect many pages rather than a single template.
3. Find advertising requests to your own hostname.
Open the Network panel and filter requests by your website’s domain.
Identify advertising-related requests, including configuration files, header-bidding wrappers, proxies, refresh calls and tracking pixels.
Record any cache-busting parameters, such as timestamp-based values added to query strings.
4. Record the existing CrUX advertising metrics.
Where the data is available, collect the four CrUX advertising metrics for your origin.
These provide a baseline for advertising performance. They do not measure crawl budget directly, but they help you check whether the advertising changes affect the experience of real Chrome users.
Step 2. Measure What Googlebot Does Today
Who: Developers and SEO
Use your server logs to establish how Googlebot is currently using your website.
1. Count Googlebot requests.
Use requests verified through reverse DNS or Google’s published IP ranges.
Count requests per day and separate HTML pages from resources. Within the resource requests, identify advertising-related requests and other resources.
2. Record response times.
Measure Googlebot’s response times from server logs rather than relying only on averages for all visitors.
This matters because crawlers may experience different response times from human visitors, particularly when server load or request handling differs.
3. Compare the findings with Search Console.
Review the Crawl Stats report for information such as:
- Requests by file type and purpose.
- Average response time.
- Host status.
Use this information alongside your logs to understand what Googlebot is requesting and how your server is responding.
If Googlebot makes almost no requests to your own advertising endpoints and response times are already healthy, the first hypothesis has little support. You could concentrate on the second hypothesis or stop the crawl-specific experiment and focus on improving user experience instead.
Step 3. Choose Two Comparable Groups of Pages
Who: SEO lead
Choose two groups that are as similar as possible.
For example, you might compare two content sections, such as /news/ and /sport/, or two halves of a category structure.
Your groups should have similar:
- Page templates and layout.
- Number of URLs.
- Content update frequency.
- Existing crawling levels.
- Traffic patterns.
Use whole sections or URL patterns so you can apply the change to one group cleanly and separate the results in your logs.
Avoid pages with planned migrations, campaigns or major redesigns during the experiment.
Write down the group definitions before starting and keep them unchanged for the full test. Changing the groups halfway through makes the results harder to interpret.
Step 4. Make One Change to the Test Group
Who: Developers and advertising operations
Choose the change that matches your hypothesis. Do not combine several different changes in a single test, or you will not know which one made the difference.
Option A: Reduce Advertising Requests to Your Main Hostname
Use this option to test the first hypothesis.
- Serve advertising configuration files, bidding wrappers and other first-party advertising resources from a separate hostname, such as an advertising subdomain or CDN hostname.
- Replace unnecessary cache-busting timestamps with versioned filenames that change only when the file changes.
- Stop unnecessary advertising refresh calls from hitting the main hostname.
The aim is to reduce requests affecting your main hostname, rather than simply changing where the scripts appear in the browser.
Option B: Reduce Advertising Work Before the HTML Response
Use this option to test the second hypothesis.
- Move advertising decisions and insertion out of the server’s response process where practical.
- Return the main HTML without waiting for advertising systems to finish their work.
- Load advertising in the browser after the main content has been returned.
This tests whether reducing work on the server’s response path improves its ability to respond to Googlebot.
Keep User Experience Changes Separate
You may also want to lazy-load ads below the fold or reduce ad density.
Those can be sensible performance improvements, but run them as a separate test. Otherwise, you will not know whether the observed result came from reducing server load, changing hostnames or changing what visitors see.
Do not use a robots.txt block as the test intervention. That would introduce a crawling restriction into a performance experiment and could affect the rendering of resources needed to understand the page.
Step 5. Run the Experiment Without Other Major Changes
Who: Everyone
Once the test is ready, make the change on a single day and record the exact date.
The analysis should exclude the day of the change itself.
Then follow the planned schedule:
- Use two complete weeks before the change as your baseline.
- Measure four complete weeks after the change.
- Avoid changing anything else in either group during the test.
- Record any unavoidable changes, releases or incidents.
- Check Google’s Search Status Dashboard for confirmed updates during the experiment.
A Google update could affect both groups. The control group helps account for changes that affect the wider site, but any major event should still be recorded when interpreting the results.
Step 6. Analyse the Results Properly
Who: Analyst
Do not compare the test group before and after the change in isolation.
Instead, compare how much the test group changed with how much the control group changed over the same period.
This is called a difference-in-differences analysis.
For example, suppose the number of Googlebot page requests increases in your test group after the change. If crawling also increases by a similar amount in the control group, the advertising change may not have caused the increase.
Report the estimated difference and its 95% confidence interval, not just the average figures.
If the interval includes zero, the test has not detected a clear effect at the chosen confidence level. That does not prove that the true effect is exactly zero, but it means the observed data do not provide strong enough evidence of a measurable difference.
The developer tool described below performs this analysis using server logs.
Then check the guardrails for both groups:
- Advertising revenue per 1,000 sessions.
- Core Web Vitals.
- The four CrUX advertising metrics.
More crawling is not necessarily a win if it comes at an unacceptable cost to advertising revenue or user experience.
Step 7. Decide What to Do Based on the Result
Who: Leadership and advertising operations
| Result | What to do |
|---|---|
| Crawling increases and revenue remains steady | Roll out the change gradually, then measure its effect on another template. |
| Crawling increases but revenue falls | Compare the value of faster crawling with the revenue loss. Test a smaller change before rolling it out widely. |
| Crawling does not change, but user metrics improve | Keep the change if it benefits users and revenue. Stop treating advertising as a crawl budget problem. |
| Neither crawling nor user metrics improve | Look for other constraints, such as duplicate URLs, faceted navigation, low-value pages or server capacity. |
Whichever result you get, there are still good reasons to review your advertising setup.
Keep first-party advertising resources on an appropriate hostname, avoid unnecessary cache-busting parameters, reduce advertising work on the server’s response path where possible, and reserve space for ads to prevent unexpected layout shifts.
If the experiment suggests that advertising is not the constraint, investigate crawl demand and technical health instead.
Step 8. Give Each Team Clear Responsibilities
| Team | Role in the experiment | First actions |
|---|---|---|
| Advertising operations | Owns the advertising change and revenue limits. | Record the DevTools advertising data and agree how much revenue loss, if any, is acceptable. |
| Development | Implements the change and supplies server logs. | Map first-party advertising requests, make the hostname or response-path change, and provide logs with response times where available. |
| SEO | Owns the test design, page groups and analysis. | Document the protocol and success criteria before the experiment begins. Analyse the results once the measurement period finishes. |
| Content | Keeps the page groups stable. | Avoid restructuring, migrations and large-scale changes in either group for the six-week test. |
| Leadership | Decides whether the change should be rolled out. | Agree in advance what result would justify a wider change. |
5. Developer Toolkit: crawl_experiment.py
The proposed crawl_experiment.py tool analyses the experiment using standard web server access logs and Python’s standard library.
It is designed to:
- Read Apache or NGINX combined access logs.
- Identify Googlebot requests, with optional DNS verification.
- Assign URLs to the test and control groups using rules in a text file.
- Separate HTML page requests from resource requests, including scripts, stylesheets, JSON, images and fonts.
- Identify advertising-related endpoints using an additional regular expression.
- Measure Googlebot response times if the log includes a suitable response-time field.
- Calculate difference-in-differences estimates for three metrics.
- Produce bootstrap 95% confidence intervals using daily results.
The tool expects response times to be recorded in a format it can interpret, such as NGINX’s $request_time or Apache’s %D. Check the log format before running the analysis.
Example Command
Create a group definition file:
printf "treatment ^/news/\ncontrol ^/sport/\n" > groups.txtThen run the analysis:
python3 crawl_experiment.py access.log --groups groups.txt --change 2026-11-03 \
--baseline-days 14 --after-days 28 --resource-regex "/ads/"The date and URL patterns above are examples. Replace them with the actual date of your test and the paths used by your website.
Example Results: With and Without a Measurable Effect
The following output shows two synthetic tests designed to illustrate how the analysis works.
In the first test, the data was set up so that the treatment group’s page crawling increased by 15% after the change, advertising configuration requests fell and response times improved.
In the second test, there was no meaningful difference between the treatment and control groups.
Synthetic log WITH a planted effect
Metric Treat before Treat after Ctrl before Ctrl after Effect 95% interval
html_per_url 2.722 3.261 2.757 2.879 +0.418 [+0.034, +0.794] increase
resource_share 0.232 0.107 0.230 0.231 -0.126 [-0.131, -0.121] decrease
resp_ms 350.849 299.813 350.104 350.005 -50.936 [-52.750, -48.932] decrease
Synthetic log WITHOUT an effect
Metric Treat before Treat after Ctrl before Ctrl after Effect 95% interval
html_per_url 2.722 2.886 2.757 2.900 +0.021 [-0.364, +0.399]
resource_share 0.232 0.231 0.230 0.230 -0.001 [-0.007, +0.005]
resp_ms 350.849 349.963 350.104 349.763 -0.544 [-2.290, +1.165]These numbers are illustrative results from synthetic logs, not findings from a live website.
The second test highlights why a control group matters.
The treatment group’s page crawling increased from 2.72 to 2.89 requests per URL per day. Looking only at the treatment group, you might think the change had worked.
However, the control group’s crawling increased by a similar amount. After accounting for that change, the estimated difference between the two groups was close to zero.
A simple before-and-after comparison could have led to the wrong conclusion. The control group helps distinguish the effect of your change from wider movements in crawling.
The Core Statistical Function
The following code excerpt shows the difference-in-differences calculation and its bootstrap confidence interval.
It is the core calculation only, not a complete standalone version of the log analysis tool. The full tool also needs the log readers, group assignment rules, data preparation and supporting definitions.
def did(t_before, t_after, c_before, c_after, n_boot=2000, seed=7):
"""Difference in means, with a bootstrap 95% interval over days."""
est = (
(statistics.mean(t_after) - statistics.mean(t_before))
- (statistics.mean(c_after) - statistics.mean(c_before))
)
rng = random.Random(seed)
sims = []
for _ in range(n_boot):
pick = lambda xs: [rng.choice(xs) for _ in xs]
sims.append(
(statistics.mean(pick(t_after)) - statistics.mean(pick(t_before)))
- (statistics.mean(pick(c_after)) - statistics.mean(pick(c_before)))
)
sims.sort()
return (
est,
sims[int(0.025 * n_boot)],
sims[int(0.975 * n_boot) - 1]
)The point of the analysis is not to produce a more complicated chart. It is to give the business a better basis for deciding whether the advertising change made a measurable difference.
6. What We’re Watching Next
Several developments could help establish how useful the new advertising metrics are and whether advertising changes have a measurable effect on crawling.
Will the CrUX Advertising Metrics Become More Important?
Google says the metrics are experimental and are not part of Core Web Vitals. It has not provided targets or thresholds.
If Google adds recommended thresholds or includes these metrics in more reporting tools, advertising performance could become a more standard part of website performance reviews.
Publishers would benefit from establishing their own benchmarks before that happens.
How Does Chrome Identify Advertising?
Google’s announcement does not explain exactly how advertising is identified for these metrics.
This matters for publishers using native advertising or their own first-party advertising formats, which may not be identified in the same way as conventional advertising scripts.
Clearer guidance on what is counted would help website owners interpret the metrics more accurately.
What Will Real-World Experiments Show?
The results are likely to vary between websites.
Where advertising runs entirely on external network hostnames, there may be little or no effect on the publisher’s own crawl budget.
Where advertising requests use the publisher’s hostname or delay server responses, a measurable effect is possible.
The controlled experiment described in this article is a way to find out rather than assuming the same answer applies to every site.
How Will DevTools Support AI Agents?
The same DevTools release introduced controls for AI coding agents, including ways to inspect pages without running JavaScript.
As automated agents are used more widely for performance analysis and SEO audits, these tools may become increasingly useful for investigating complex websites.
How Should Crawling Be Tested During Volatile Periods?
Google’s search results have been unusually unsettled since late September 2026.
If crawling also changes during that period, it becomes especially important to separate the effect of your own changes from wider shifts in Google’s systems.
A control group is valuable in any experiment, but it is particularly important when the wider environment is changing.
About Szymaniak Digital
Szymaniak Digital is a UK enterprise SEO and AI search consultancy. We design and run controlled SEO experiments for large publishers and retailers, analyse Googlebot activity using server logs, and help SEO and advertising teams make changes that protect both revenue and search visibility.
If you need to establish whether your advertising setup is affecting crawling, the starting point is a clear test with a reliable baseline, a control group and agreed success criteria. The results can then guide decisions about advertising delivery, server performance and SEO priorities.
