Nearly 1 in 4 Cloudflare-Protected Domains Block Verification Bots.

The September Default Will Widen the Gap.

The Web Wants AI Agents. Its Firewalls Disagree.

By Thorsten Litzki  |  27 August 2026

Tranco Top 1,000 scan, August 2026. SOVPBot/1.0, 15-second timeout.

The Number

22.8% of Cloudflare-protected domains in the Tranco Top 1,000 block a standards-compliant verification bot on first contact.

Among all other providers: 4.7%.

Factor of five.

The Scan

I scanned the 1,000 most popular global domains, ranked by the Tranco research list (Le Pochat et al., NDSS 2019). One HTTP HEAD request per domain. User-Agent: SOVPBot/1.0. Fifteen-second timeout.

305 of the 1,000 returned silence. DNS errors, TLS failures, infrastructure endpoints with nothing to say on the apex domain. CDN plumbing. Tracking pixels. API routers serving content through subdomains only.

695 answered. 55 of those 695 turned the bot away.

The Deadline

That measurement is from late August 2026. On September 15, 2026, Cloudflare changes its default configuration for all new domains and all existing free-tier accounts. The new default blocks an entire category of bots by design.

What Cloudflare Announced

On July 1, 2026, Cloudflare published „Content Independence Day: Taking Control of Your Content in the Age of AI” (blog.cloudflare.com/content-independence-day-ai-options). The title alone tells you where this is headed.

Three Categories Replace One Toggle

The old system had one switch: „Block AI Bots.” The new system splits bot traffic into three categories. „Search” means crawlers that index content for search engines (Googlebot, BingBot). „Training” means crawlers that feed model training pipelines (GPTBot, ClaudeBot). „Agent” means anything acting on behalf of a human: placing an order, verifying a certificate, checking a security header.

The Default After September 15

Search stays open. Training and Agent get blocked. Free-tier accounts move to these settings automatically.

The mechanism uses a new directive in Cloudflare’s managed robots.txt:

User-agent: *

Content-Signal: https://example.com/.well-known/content-signal

  use=reference

The „use=reference” value grants permission to cite and summarize. Bots in the Agent category need this signal to pass. A verification bot that checks structured data, validates DNS, or audits security headers fits the Agent definition perfectly. So does a compliance scanner. So does an accessibility checker.

All three face the same default block after September 15.

The Measurement

The Tranco list aggregates four data sources: Chrome UX Report, Cisco Umbrella, Majestic, and Cloudflare Radar. The list I used carries the ID 46W9X, generated on August 26, 2026 (tranco-list.eu).

The scanner recorded three things per domain: HTTP status code, presence of a cf-ray or server: cloudflare header (identifying the domain as Cloudflare-protected), and challenge-page behavior.

The Response Table

695 domains answered. Only 115 of those (16.5%) responded with a clean 2xx. Here is the full breakdown:

The redirect mass (486) is www-to-apex and apex-to-www canonicalization. Three domains responded with HTTP 418, a status code from RFC 2324. The teapot response exists as a joke. In the wild, it serves a specific purpose: the server sees the bot, recognizes it, and says no.

The Provider Split

The 55 blocks (403, 503, challenge pages, the three teapots) make up 7.9% of all responding domains. Split by provider, the asymmetry is sharp:

Figure 1: WAF block rate by provider. Cloudflare: 28 of 123 blocked (22.8%). All others: 27 of 572 blocked (4.7%).

The Collateral Damage

The verification bot I sent identifies itself by name. It states its purpose in the User-Agent string. It waits 15 seconds. It reads one page. That is the entire interaction.

The Fire Inspector Problem

A firewall that blocks this bot is a smoke detector that locks out the fire inspector. The sprinklers still work. You just lost the only person who can confirm it.

The same blocking mechanism hits every bot in the Agent category. Compliance scanners verifying regulatory requirements. Accessibility checkers testing WCAG conformance. Infrastructure auditors measuring machine readiness. After September 15, all of them face the default block on Cloudflare’s free tier and on every new domain added to existing accounts.

Common Crawl and the Open Web

This reaches further than one CDN provider. The Common Crawl Foundation operates CCBot, the crawler behind the largest open web corpus available to researchers. Over 3.5 billion pages. Academic studies, search index alternatives, large-scale web analysis: they all draw from this corpus.

Cloudflare’s new taxonomy places CCBot alongside training crawlers. As more domains apply that classification by default, the corpus shrinks. The domains that drop out of Common Crawl still exist. They become opaque to measurement.

Stephen Burns maintains the publicly available AI Visibility Audit (audit.stephenburns.ai). His tool checks robots.txt permissions for GPTBot, ClaudeBot, CCBot, and Googlebot-Extended. It measures CDN and firewall blocking patterns, JavaScript-free rendering, and Common Crawl presence. His measurements and mine land on the same structural finding. Domains that want autonomous systems to find them are, at the firewall level, turning those systems away.

The Compliance Scenario

Picture a compliance team at a regulated company. They deploy a verification scanner to confirm that suppliers provide valid structured data and signed attestations. The scanner identifies itself, sets a timeout, requests one page. The supplier runs Cloudflare on default settings. The response: 403. The compliance team gets a blank. The supplier may be fully compliant. The firewall made that impossible to confirm.

What the Blocked Domains Are Hiding

A second measurement layer scanned all 1,000 Tranco domains with 183 deterministic signals: structured data quality, DNS configuration, security headers, protocol support, and machine-readable metadata.

I compared the 55 blocked domains against the full-sample averages. Infrastructure: 54.2 vs. 59.1. Content quality: 51.3 vs. 53.9. Domain anchoring: 82.6 vs. 80.2.

Within five points. Across every category.

These domains maintain functional infrastructure. Their structured data exists. Their DNS resolves correctly. Their security headers are in place. The firewall hides all of it. A verification bot that receives a 403 or a JavaScript challenge sees a wall. Behind that wall sits a fully equipped domain. The bot sees the wall. A compliance pipeline sees the wall. An AI agent evaluating data sources sees the wall. The result is the same as a domain with zero infrastructure.

The safest wall is the one that knows what it lets through.

The Other Gap: 11.28 Out of 100

Figure 2: Infrastructure scores, Tranco Top 1,000 average. AI Readiness sits at 11.28 out of 100.

The firewall story has a second layer. Even among the domains that do let the verification bot through, the machine-readable infrastructure for AI systems barely exists.

The average AI Readiness score across all 1,000 Tranco domains is 11.28 out of 100. Domain anchoring scores 80.2. Infrastructure scores 59.1. Content quality scores 53.9. AI Readiness: 11.28.

That score measures the signals an autonomous system needs to evaluate a domain: llms.txt files, structured AI metadata, agent-facing configuration, and machine-readable permission boundaries. The top 1,000 global domains provide almost none of it.

A broader dataset confirms the pattern. Across 2,677 domains in regulated industries (finance, healthcare, government, critical infrastructure, e-commerce), the AI Readiness average is 11.1 out of 100. Readiness verdicts: 0% ready. 8% partial. 92% failed. The Tranco sample and the sector-specific sample land on the same number. This is a structural deficit, visible across geographies and industries alike.

Two Problems, One Result

1. 22.8% of Cloudflare-protected domains turn verification bots away at the door. The bot sees a wall.
2. The domains that open the door have, on average, 11.28 out of 100 in AI-facing infrastructure behind it. The bot walks in and finds an empty room.

The firewall blocks the front door. The missing infrastructure empties the house. An AI agent hitting either problem reaches the same conclusion: this domain has nothing to offer.

Before September 15: Three Steps

1. Open the Cloudflare dashboard. Find the three new bot management categories. The toggles for Search, Training, and Agent default to blocking Agent and Training crawlers. Decide deliberately which categories your domain permits.
2. Add the Content-Signal directive to your robots.txt. The „use=reference” value tells compliant bots that your domain allows citation and summarization. One line. Machine-readable permission boundary.
3. Run a verification scan against your own domain. The AI Visibility Audit does this. A 183-signal infrastructure verification engine does this. Any tool that sends a HEAD request with a named User-Agent and reads the response code does this. If your firewall returns a 403, your domain is opaque to every verification system that respects your firewall.

The Question

Before September 15, send a HEAD request to your own apex domain. Use a named User-Agent. Set a 15-second timeout.

Read the response code.

If your firewall turns away a bot that identifies itself, states its purpose, and waits for an answer, ask what exactly you are protecting. And from whom.

AI Joux invites researchers and practitioners who write for us technology articles grounded in original data and reproducible methods. Contributor details at aijoux.com.

Methodology

Domain list: Tranco Top 1,000 (list 46W9X, generated August 26, 2026). Le Pochat, V., Van Goethem, T., Tajalizadehkhoob, S., Korczyński, M., & Joosen, W. (2019). „Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation.” NDSS 2019. https://doi.org/10.14722/ndss.2019.23386. List: https://tranco-list.eu/list/46W9X.

Bot admission check: HTTP HEAD request per domain, User-Agent: SOVPBot/1.0, 15-second timeout, August 27, 2026. Cloudflare identification via cf-ray and server headers.

Infrastructure scan: 183-signal deterministic scan per domain, August 26–27, 2026. Scores: infrastructure, content quality, domain anchoring, AI readiness.

Two measurement layers: The bot admission check (HEAD request) and the infrastructure scan (183 signals) ran independently. The admission check measures if a domain lets a verification bot through. The infrastructure scan measures what the bot finds once inside. This article reports both layers separately.

About the Author

Thorsten Litzki is the founder of Litzki Systems LLC (St. Petersburg, FL). He builds cryptographically signed infrastructure verification tools and maintains an IETF Individual Draft on sovereign validation. His scanner has audited over 2,600 domains across multiple measurement cycles. This article is part of AI Joux’s write for us technology contributor program. litzki-systems.com