There is a question most executives have never been asked, and it now determines whether an AI model can recommend you: which machines are allowed to read your website?
Until recently the answer was "all of them," by default, forever. That default is gone.
What changed#
On July 1, 2025, Cloudflare introduced pay per crawl — infrastructure that lets a site owner charge AI crawlers for access, or refuse them outright. Cloudflare paired it with a shift to blocking AI crawlers by default for new domains. Its AI Crawl Control documentation is now standard tooling, not an experiment.
Meanwhile Google spent 2025 publishing an unusually detailed Robots Refresher series explaining exactly how exclusion rules are interpreted — a strong signal that crawl control was about to matter to a much larger audience.
The trap#
Publishers had a legitimate grievance: models were consuming their work and returning fewer visitors than ever. Blocking made sense for them.
It usually does not make sense for you.
"If nobody on your leadership team can say which machines are allowed to read your site, that decision is being made for you."
Andrew Bethel — COO, Perfectus Labs
If you sell a service, an AI model that cannot read your site cannot cite your site. You are not protecting inventory — you are removing yourself from the shortlist. The publisher's economics (ad impressions per pageview) and the operator's economics (one qualified booking is worth thousands of pageviews) point in opposite directions.
Yet we routinely find service businesses blocking AI user agents. Not deliberately. It came bundled: a security setting, a WAF default, a managed-hosting toggle flipped by a vendor who never asked what it would cost in demand.
What to actually do#
1. Read your own robots.txt out loud. List every AI user agent you disallow and say why, in one sentence, per agent. If you cannot finish the sentence, you have a policy you did not choose.
2. Separate the two decisions. "Should bots train on my content?" and "Should assistants be able to cite my content when a buyer asks?" are different questions with different answers. Many crawlers are documented and separable — treat them individually rather than with one blanket rule.
3. Check the layer above robots.txt. Rate limiting, bot-management rules, JavaScript challenges, and geographic blocks can deny access silently even when robots.txt permits it. Robots.txt is a request. Your edge is an enforcement.
4. Verify with logs, not assumptions. Server logs tell you which assistants actually fetched which pages, and when. That record is the only ground truth in this space.
Where Cognos fits#
Crawl access is the first thing Cognos checks, because everything downstream is worthless without it. We inventory which AI agents can reach which templates, flag silent blocks at the edge, and then measure whether reachable pages are actually being retrieved.
It is the least exciting slide in the deck and the most common single point of failure we find.
The strategic read#
Access to content is becoming a negotiated market rather than a public good. Large publishers will license. Everyone else will make a quieter choice: be readable and be recommended, or be private and be absent.
For a business whose customers ask an assistant "who should I hire for this," being absent is not neutral. It is a decision to hand the answer to a competitor.
Sources: Cloudflare, Jul 1 2025; Cloudflare AI Crawl Control docs; Google Robots Refresher, 2025.
