Indexation, AI-Bot, and Agent Readiness

Chapter 11 (Crawler Access) sets the robots.txt posture that lets AI systems reach your content. This chapter keeps that content current in the retrieval index and makes sure the right bots can prove who they are once they arrive. It covers three layers. Indexation is the fast path from publish to retrieval index: IndexNow for the Bing-and-beyond engines, and Google's Indexing API for the Google side, so a refresh reaches Search, AI Overviews, Gemini, and Copilot without waiting on the standard crawl. AI-bot readiness is verifying and admitting legitimate AI crawlers while blocking impersonators (Web Bot Auth), pointing them at your machine-readable resources (Link headers), and declaring how your content may be used (Content Signals and the emerging IETF standard). Agent readiness is the honest horizon scan: the agent-operability and agent-commerce standards the industry is building, and why most of them are a different discipline from getting cited. The through-line is Inclusion: reached, current, verified, and correctly read.

Why This Technique Matters

Recency is a real citation signal. AirOps's August 2025 research on content freshness showed something clear. Pages not updated in over a year are more than 2 times as likely to lose ChatGPT citations, and more than 70% of cited pages had been updated within the past 12 months. The recency signal feeds AI retrieval through the search-engine indexes. Faster index inclusion means that recency lift lands sooner on refreshed pages.

A second, larger study says the same. Ahrefs measured content age across 16.975 million cited URLs on seven AI platforms and found AI-cited content is 25.7% fresher on average than the URLs in organic search results, roughly 2.9 years old versus 3.9 (Ahrefs, 2026). ChatGPT skews newest of all. Freshness carries more weight for AI citation than it ever did for organic ranking, which is exactly why the speed of getting a refresh indexed is worth engineering.

Standard crawl cycles add delay between publish and index inclusion. Major search engines crawl high-authority domains frequently. They crawl mid-tier domains less often. New content sits unindexed until the next crawl. It cannot earn citations during that window. Refreshed content has the same delay. The recency boost only kicks in after the index updates.

IndexNow closes this delay. The protocol lets your brand notify retrieval engines direct when a URL publishes or updates. The engines take the notification as a signal that the URL changed and decide when to recrawl. They tend to fetch fast for high-authority domains. They fetch a little slower for newer domains. The recency boost from the refresh kicks in right away. It does not wait for the next standard crawl.

The advantages compound with Chapter 6's refresh cadence. Chapter 6 calls for quarterly refresh on benchmarks. It calls for annual on frameworks. It calls for reactive on outside events. Brands that follow Chapter 6 without IndexNow lose recency-boost time on every refresh. They wait for standard crawls to pick up the change. Across many refresh events, the lost time adds up to real lost citation lift.

Where Indexation Sits in the Stack

Before the mechanics, the order of operations, because indexation is the easiest place in this pillar to over-invest. Getting cited by an AI system depends, in order, on reach (a crawler can fetch you, which is Chapter 11's robots.txt posture), verification (the crawler is who it claims to be and you admit it, covered in Web Bot Auth below), correct interpretation (your content and entities read cleanly, Chapters 9 and 10), and governance (you have not quietly told the engines they may not use your content). The two indexing protocols in this section sit on top of all of that. They speed how fast a change reaches the index once everything above is already true. They are worth setting up, and they are not the lever. A site with a clean robots posture, verified bot access, and citable content gets found and cited on the standard crawl cycle. A site that nails IndexNow but blocks the retrieval bots stays invisible. Read the indexing sections as the freshness accelerator they are, then spend the real attention on the access, verification, and governance work that follows.

What IndexNow Is, and What It Is Not

Most coverage of IndexNow overstates it, so keep the mechanism straight. IndexNow is not a citation lever. It does not make a page more likely to be cited, and it does nothing for Google, which has never joined the protocol. What it does is narrower and still worth doing: it is freshness plumbing. It carries the recency signal AirOps measured to the Bing index fast, which feeds Microsoft Copilot directly and reaches ChatGPT and Perplexity indirectly through Bing-indexed content. The lever is the freshness of the content itself. IndexNow is the pipe that delivers that freshness before the next crawl would.

That distinction changes how you treat it. Do not sell IndexNow, internally or to anyone else, as a way to earn more AI citations. Treat it as infrastructure for the Chapter 6 refresh discipline: when a refresh lands, IndexNow is what puts the updated page in front of the Bing-and-beyond engines without waiting on a crawl cycle.

How IndexNow Works

The protocol runs in three steps and the whole cycle finishes in seconds.

You generate a unique key (32 characters is typical) and host it as a plain text file at your domain root. The key is not a secret; hosting it there is the proof that you control the domain. On publish or update you POST a small JSON body (host, key, key location, and a urlList of up to 10,000 URLs) to an endpoint. Submit to api.indexnow.org and it forwards to every participating engine (Bing, Yandex, Naver, Seznam, and Yep); submit to www.bing.com/indexnow or yandex.com/indexnow to reach one directly. The engine fetches your key file, confirms the match, and takes the URLs as a signal to recrawl. It is a signal, not a guarantee: high-authority domains get picked up fast, newer domains slower, and a failed submission (a bad key, URLs outside your verified domain, a tripped rate limit) returns an error and never enters the crawl queue.

The Google Indexing API

Google runs an indexing API of its own. It is not part of IndexNow, and Google has shown no plans to join. It is the fast-inclusion path on the Google side, which is Search, AI Overviews, and Gemini retrieval.

Official Scope, and the Gray Area

Google built the Indexing API for two schema types, JobPosting and BroadcastEvent, and the official docs limit it to URLs that carry one of them. The stated reason is time-sensitive value: job listings expire, events happen on set dates, so they earn a fast track. General URLs are outside the official scope.

In practice, general URL submission works. The endpoint accepts any content type, checks ownership against Search Console, and pushes the URL to Google's priority crawl queue. Cody C. Jensen has used it for general URLs on Searchbloom's own sites and on partner sites and seen fast SERP changes, matching the pattern in his information gain SEO breakdown. It is common among technical SEO operators, and it is outside Google's stated terms. Be honest about that. Google could pull access at any time if it chose to enforce the scope. It has not, as of mid-2026, but the risk is real and dormant, not retired.

Setting It Up, and Rationing It

The build is a Google Cloud project with the Indexing API enabled, a service account, and, the step that trips almost everyone, the service account added as an OWNER of the Search Console property, not a user, or every call returns a 403 with no useful hint. Submissions POST to indexing.googleapis.com/v3/urlNotifications:publish with an OAuth 2.0 bearer token; use Google's official SDK (google-auth for Python, googleapis for Node) rather than hand-signing the JWT. Wire it into the same publish event as IndexNow, from one hook point, so the two never drift apart and one fails silently while the other keeps firing.

The two protocols reward opposite behavior, and that decides how you use each. IndexNow is forgiving: no credential to lose, high limits, no per-URL gating, so you submit every change and over-submission is at worst ignored. Google's Indexing API is a scoped, quota-limited tool, not a bulk indexer, and the default of 200 requests per day is a ceiling, not a target. Point it at the whole sitemap on a schedule and that is the misuse that draws enforcement: Google can throttle the project or pull the service account with no warning. Ration it. Submit only genuinely new or substantively refreshed URLs that matter, and let the standard crawl carry the rest.

Building this is an engineering task (service-account credentials, OAuth, Search Console ownership), so it usually falls to the Technical Lead from Chapter 15 or a technical SEO partner. WordPress plugins that claim Indexing API support still need the same Google Cloud setup underneath; the plugin is a thin layer on top, not a shortcut around it.

Wiring It Up

The rule for where to attach the submission is the same on every stack: fire it at the layer that first knows a URL is publicly live. Submit before the page renders and the crawler gets a 404; submit after the deploy and you keep the recency window. On WordPress that layer is the publish event, and it is a toggle: the official Bing plugin, or Rank Math, Yoast, or All in One SEO. On a headless CMS it is a publish webhook to a small function that resolves the canonical URL and submits. On a static site it is a post-deploy hook that diffs the new sitemap against the last and submits the changed URLs. Run IndexNow and the Google Indexing API from that one hook point so the two never drift apart, and filter to published status so drafts never fire.

Refresh-Cadence Integration with Chapter 6

The full value of both protocols shows up when you pair them with Chapter 6's refresh cadences. Every refresh, quarterly on benchmarks, annual on frameworks, reactive on outside events, fires a submission to both IndexNow and the Google Indexing API for the changed URLs, so the recency boost lands on both halves of the ecosystem at once instead of waiting on the next crawl. Reactive refreshes gain the most, because that is when the query surge is live and the window is short.

The operating pattern is one line in the Chapter 6 checklist: update the page, confirm it is live, submit to both protocols, and log both, since a run of non-200 responses is the tell that a submission path broke, and the Bing Webmaster Tools IndexNow history shows what Bing actually received. Single-protocol coverage leaves the other half on the standard crawl; dual coverage is the working standard. Treating index freshness this way is one operational layer of the Corpus Engineering discipline, keeping the brand's body of content current in the retrieval index on purpose, not left to crawl-cycle chance.

The Indexing Lag Reduction Score

The point of both protocols is to compress time-to-index, and the Indexing Lag Reduction Score is a Searchbloom-coined diagnostic that puts the improvement on one number: the percentage cut in median time-to-first-citation against a pre-integration baseline, tracked monthly per content type.

ILRS = ((baseline median time to first citation) - (post-protocol median time to first citation)) / (baseline median) x 100

Read it in bands. Above 70% is a strong reduction, with both protocols firing on publish. 30 to 70% usually means one protocol is broken or unused, so pull both response logs. Below 30% means the site was already crawled fast, or submissions are firing on the wrong events. Zero or negative almost always means the integration was built but never actually enabled at the deploy level. It pairs with refresh velocity: the same content earns a higher citation share when refreshes are both on time and promptly indexed.

From Indexation to AI-Bot Readiness

Indexation gets your content into the retrieval index fast. It does nothing about which bots you let in, whether they are who they claim to be, or how you tell them your content may be used. Chapter 11 set the robots.txt posture. The rest of this chapter covers the layer on top of it: verifying the AI crawlers that arrive, pointing them at your machine-readable resources, and declaring your content-use terms. These are newer, and the standards are moving fast, so the goal here is to separate what is worth building now from what is worth watching.

Bot Verification: Web Bot Auth

A robots.txt rule names a bot by its user-agent string. Anyone can set that string. A scraper can call itself OAI-SearchBot and your access rules cannot tell the difference. IP allowlists help, but they are brittle and go stale as the vendors change infrastructure. Web Bot Auth fixes the identity problem at the request level. A legitimate agent signs each request with a cryptographic key, and the site, or the CDN in front of it, verifies the signature before applying policy.

The mechanism rests on a published standard. The agent holds an Ed25519 key pair and signs outbound requests using HTTP Message Signatures (RFC 9421). The request carries a Signature, a Signature-Input, and a Signature-Agent header that points at the agent's public-key directory, served at a well-known path. The receiving server fetches the directory, verifies the signature, and confirms the bot is who it claims to be. From there the site makes a real decision: admit verified AI search crawlers, and treat unsigned traffic that claims the same identity as suspect.

This is the most production-ready of the emerging bot standards. Cloudflare turned on signed-agent verification in 2025 and folded it into Verified Bots, and named signing agents include OpenAI's ChatGPT agent. An IETF working group is standardizing it, and as of mid-2026 it is running in production ahead of a finished RFC, so cite RFC 9421 and the active protocol draft rather than the earlier architecture draft, which has expired. For Inclusion the point is direct. Verification keeps the door open for the crawlers that cite you while closing it to impersonators, instead of the blunt alternative, an accidental blanket block that makes you invisible to the model.

Discovery Pointers and Content-Use Governance

Two low-cost, standards-clean moves round out the AI-bot layer. Neither is a growth lever on its own. Skipping them leaves easy signal on the table. Overselling them repeats vendor hype.

Link Headers

An HTTP Link header (RFC 8288) lets a server advertise related resources in the response headers themselves. A bot doing a single request learns where your machine-readable resources live before it parses any HTML. Point the retrieval crawlers at the sitemap, at feeds, and at any alternate representation of a page with typed relations such as rel=sitemap and rel=alternate. It is a stable standard, it costs a header, and it improves discovery, which is upstream of being understood and cited.

The llms.txt Question, Answered Honestly

llms.txt is a curated markdown map of a site, proposed as a discovery aid for language models and marketed as robots.txt for AI. Searchbloom does not build one, and does not recommend paying for one. The evidence is one-sided: origin-log studies, and Searchbloom's own measurement across partner sites, show the answer-engine crawlers that drive citations, GPTBot, ClaudeBot, PerplexityBot, and Google, effectively do not fetch llms.txt; the requests that arrive come almost entirely from SEO audit tools, and no major LLM provider has documented support for it. If a vendor is leading with llms.txt as an AI-search deliverable, treat it as a tell. The higher-return move is the one that has always worked: serve clean, well-structured HTML that the crawlers already read.

Content-Use Signals

Content-use declarations state how automated systems may use your content, separate from whether they may crawl it. Cloudflare's Content Signals adds directives to robots.txt (search, ai-input, ai-train, each set yes or no), and the IETF AIPREF working group is standardizing the same idea under a neutral name for eventual cross-vendor use. Understand two things. First, these are voluntary legal preferences, not technical blocks: a compliant crawler honors them, a non-compliant one ignores them. Second, they can suppress citations when set to restrict. A brand behind a CDN that applies a restrictive default can be telling the answer engines not to use its content without knowing it. The Inclusion action is defensive: audit the signals so you are not opting out of the retrieval you want.

The CDN Default-Block Check

The most common silent citation-killer is not a setting the brand chose. It is a CDN default. Providers are moving toward blocking or tolling AI crawlers by default, and one major CDN is introducing a default that blocks mixed-use crawlers on any page that carries ads. If a partner sits behind that CDN, the retrieval bots that would cite them can be blocked at the edge, above anything in robots.txt. Confirm the CDN is admitting the AI search crawlers before assuming an access problem lives in the site.

Meta Robots and AI Snippet Controls

robots.txt governs crawling; the meta robots tag and the X-Robots-Tag header govern what happens after the fetch, at the page level, and two of their directives bear directly on AI. noindex keeps a page out of the index entirely, so it cannot be cited, which makes an accidental noindex on a template one of the quieter ways to disappear from AI answers. The snippet controls are the ones people miss. max-snippet caps how much text an engine may show, and nosnippet, along with the data-nosnippet attribute on a section, suppresses the snippet altogether. Because AI Overviews and the answer engines quote from that same snippet layer, a tight max-snippet or a nosnippet can keep your text out of the generated answer even when the page ranks. For AI visibility the default max-snippet:-1, no limit, is usually what you want. Some publishers also add emerging noai and noimageai signals; support is uneven, so treat them as a preference like the content-use signals above, not a guarantee.

The Agent-Readiness Horizon

The industry is building a much larger set of standards around AI agents, and a single vendor dashboard now scores a domain on its agent readiness. Most of that set is a different discipline from getting cited, and it is worth naming plainly so the line stays clear. Two layers sit beyond Inclusion.

The first is agent operability: standards that let an external agent discover and operate your site as a tool. Anthropic's Model Context Protocol, Google's Agent2Agent, and the browser-native WebMCP proposal all live here. They make your site callable by an agent. They do not make a model more likely to name your brand in an answer.

The second is agent commerce: standards for an agent to transact and pay. OpenAI and Stripe's Agentic Commerce Protocol, Google's Agent Payments Protocol, and Coinbase's x402 sit here. They govern the checkout and the payment, downstream of any recommendation.

All of it is early, most of it has near-zero real adoption, and the specs are churning. The honest line is the one to keep: agent-ready, meaning an agent can transact with you, is a different question from AI-search visible, meaning a model cites and recommends you. This chapter, and MERIT, answer the second question. The agent-operability and agent-commerce layer is worth watching, and it may grow into its own body of work, but it is not an Inclusion action item today.

Common Mistakes That Defeat This Layer

1. Build it, then never verify. You install a plugin or write a function and assume it fires; months later an audit shows submissions have been failing silently. Counter-test: did the last five publish events trigger submissions that returned a 200?

2. The key file is not publicly reachable. Cloudflare bot rules, server auth, or CMS access controls stop the engines from fetching your hosted key, so every submission fails the check. Counter-test: does a curl of the key-file URL with a plain user-agent return it without auth?

3. Firing on the wrong events. The hook fires on drafts, autosaves, and cosmetic saves, so you submit URLs that are not published or not really changed, and the engines de-prioritize you. Counter-test: does the build require published status and a real content change before it submits?

4. Treating IndexNow as Google coverage. Google does not participate in IndexNow, so an IndexNow-only setup leaves the larger Google surface, Search, AI Overviews, and Gemini, on the standard crawl. Counter-test: does a publish or refresh also submit to the Google Indexing API, or to Search Console at low volume?

5. Trusting the user-agent string. You admit AI crawlers on robots.txt rules alone, with no way to tell a real retrieval bot from a scraper wearing its name. Counter-test: can your stack tell a Web Bot Auth signed OAI-SearchBot from an unsigned request that claims the same identity?

6. Blocking Google-Extended to stay out of AI Overviews. Google-Extended only governs Gemini training. AI Overviews and AI Mode serve from the regular Googlebot index, with no selective opt-out. Counter-test: do you know that only a Googlebot block removes AI Overviews, and it takes all of Google Search with it?

7. Letting a CDN default block the retrieval crawlers. The brand never chose to block AI search bots, but its CDN did, at the edge, above robots.txt. Counter-test: have you confirmed your CDN admits OAI-SearchBot, Claude-SearchBot, and PerplexityBot rather than blocking or tolling them by default?

8. Restricting content-use by accident. A restrictive Content Signals or meta robots default, often set by a CDN or a template, tells the engines not to use or index your content while you assume you are visible. Counter-test: do your content-use signals and meta robots allow the AI retrieval you actually want?

Questions & Answers

What is IndexNow, and does it help ChatGPT and Perplexity? An open protocol that tells Bing, Yandex, Naver, Seznam, and Yep about new or changed URLs right away, with no crawl-cycle wait. It feeds Microsoft Copilot directly through the Bing index and reaches ChatGPT and Perplexity indirectly through Bing-indexed content. The direct lift is modest: it is freshness plumbing, not a citation lever.

How is it different from Google's Indexing API? IndexNow covers the Bing-and-beyond ecosystem and is simple, a hosted key file and a POST. Google does not participate; its Indexing API covers Search, AI Overviews, and Gemini, is officially scoped to JobPosting and BroadcastEvent, and needs Google Cloud, a service account, OAuth, and Search Console ownership. Mature operators run both from one hook point.

How do I set it up? For IndexNow on WordPress: the Bing plugin, an SEO plugin (Rank Math, Yoast, AIOSEO), or a custom save_post function. For the Google Indexing API: a Google Cloud project with the API enabled, a service account added as an OWNER in Search Console, then a POST to the urlNotifications:publish endpoint with an OAuth token. Plugins still need that setup underneath.

How do I verify submissions work? Log every response code (200 is accepted; a run of non-200 is a real problem), and check the Bing Webmaster Tools IndexNow history, which shows what Bing actually received. For Google, use the Search Console URL Inspection tool.

What is Web Bot Auth and do I need it? Web Bot Auth lets an AI agent prove its identity by signing each request with a cryptographic key, using HTTP Message Signatures (RFC 9421). It matters because a user-agent string can be spoofed by anyone. Cloudflare runs it in production and OpenAI's ChatGPT agent signs its requests. You do not build it yourself; you verify signed agents at the CDN or server, so you admit real AI crawlers and turn away impostors.

Does llms.txt help me get cited? No, and Searchbloom does not build one. No major LLM provider has documented support for llms.txt, origin logs show the citation-driving crawlers do not fetch it, and Searchbloom's own measurement across partner sites found effectively no AI retrieval of it. Treat it as an unproven experimental signal, not a deliverable; clean, well-structured HTML that ordinary crawlers already read is what earns citations.

Does blocking Google-Extended keep me out of AI Overviews? No. Google-Extended only controls Gemini model training. AI Overviews and AI Mode serve from the regular Googlebot index, and there is no selective opt-out. Blocking Google-Extended does nothing to AI Overviews. Only blocking Googlebot removes them, and that removes all Google search at the same time.

What about MCP, A2A, and the agent-commerce standards? Those are agent operability and agent commerce, a different discipline from getting cited. They let an agent operate or pay your site. They do not make a model recommend your brand. Most have near-zero real adoption and are still changing fast. Worth watching, not an Inclusion action item today.