On March 31, 2026, Google did something it rarely does: it published the plumbing. The post, Inside Googlebot: demystifying crawling, fetching, and the bytes we process, is the clearest official description of crawl mechanics in years.
Two things in it deserve your attention.
One: "Googlebot" isn't one robot#
Google confirmed that Googlebot is a client of a centralized crawling platform, not a standalone program. Dozens of other clients, including Google Shopping and AdSense, route requests through the same infrastructure under different crawler names. When you see Googlebot in your logs, you're looking at Google Search specifically.
Why this matters: crawler policy is no longer a single decision. Different Google products fetch your pages for different purposes, with different settings, and you can allow or deny them separately.
Two: the 2MB cutoff#
The concrete number, direct from the post:
- Googlebot fetches up to 2MB per individual URL, including the HTTP header. PDFs get 64MB. Crawlers that don't specify a limit default to 15MB.
- If your HTML exceeds 2MB, Google doesn't reject the page. It stops the fetch exactly at the cutoff and passes that partial file to indexing and the Web Rendering Service as if it were complete.
- Bytes past the threshold "aren't fetched, they aren't rendered, and they aren't indexed."
- Referenced resources are fetched separately with their own byte counters, so they don't count against the parent page.
Google's own framing is reassuring for most of the web. 2MB of HTML is enormous. But it names the exact failure mode we see in real audits: pages that begin with "bloated inline base64 images, massive blocks of inline CSS/JavaScript, or start with megabytes of menus."
In Google's words: if those crucial bytes aren't fetched, "to Googlebot, they simply don't exist."
"Page weight needs an owner, the same way a budget line does."
Andrew Bethel, COO, Perfectus Labs
Why this is an AEO problem, not just an SEO problem#
Answer engines are downstream of retrieval. A model can only cite text that something managed to read. So a truncated page isn't merely ranked lower. It is unavailable as evidence. Your pricing table, your service-area list, your JSON-LD block, your proof: if they load after two megabytes of framework noise, they're invisible to the systems now deciding who gets recommended.
Modern component frameworks make this easier to do than you would like. Inlined critical CSS, hydration payloads, base64 hero images, and enormous mega-menus rendered before content all push the good part of the page further down the wire.
The five-minute check#
- Fetch a key page with
curl -s URL | wc -c. That's your raw HTML byte count. - If you're anywhere near 2,000,000 bytes, find out what is consuming it.
- Confirm your structured data and primary answer content appear early in the document, not after the navigation and inline scripts.
- Move base64 images to real image URLs. They are fetched separately and don't count against the page.
- Re-check templates, not just the homepage. Byte bloat is a template problem.
What we do about it#
Cognos treats payload order as a ranking input for answer visibility. We check where the substantive content sits in the byte stream, whether structured data survives the cutoff, and whether the first meaningful content block appears before the machinery. It's engineering work, not copywriting, and it's frequently the cheapest visibility win available.
Great content that arrives at byte 2,100,000 isn't content. It's a rounding error.
Source: Google Search Central, Mar 31 2026. Limits quoted as published; verify current values before relying on them.
