Agent Readiness Checker
See whether the 32 AI crawlers behind ChatGPT, Claude, Perplexity, Google AI and the other AI platforms can reach your site, and whether it meets the 23 standards AI agents look for. Free, no sign-up, about 15 seconds.
Is your site agent ready? Two questions decide it.
Can the AI crawlers reach your pages? And once they do, does the site publish the standards AI agents look for? This checker answers both in one run.
Crawler access is the live test. ChatGPT, Claude, Perplexity, Google and the other AI platforms each run separate crawlers: one indexes pages for search, one opens a page when a person asks about it, one collects training data. A robots.txt rule, a CDN or firewall setting, or a security plugin can turn any of them away, and you won't see a firewall block in robots.txt.
Agent standards are the files and headers an AI agent checks before it uses a site: rules for AI crawlers, Content Signals, a sitemap, markdown on request, Link headers and DNS records, plus the MCP, OAuth, A2A and agent skills documents for sites with an API. The checklist covers the same 23 checks as Cloudflare's agent readiness scanner, run from our own server.
How to use the agent readiness checker
- Enter your website. One run fetches your robots.txt and homepage as a real browser, as two control bots and as each of 32 AI crawlers, then reads the 23 agent standards. It takes about 15 seconds.
- Read the two scores. Agent readiness is the share of standards checks that pass, with a level from 0 to 5. AI crawler access scores, from 0 to 100, the crawlers that decide whether you are found and cited. Choose Content site or API or app; a content site is not scored on the API, auth and MCP documents.
- Fix from the top. Every failed check opens to the goal, the fix, a snippet written for your site and a prompt for a coding agent such as Claude Code or Cursor. Copy all fixes hands every failed check to your agent as one prompt. Crawler blocks get a fix plan for the layer doing the blocking.
Which crawlers each AI platform uses
Blocking one says nothing about the others. The first column decides whether a platform can find and cite you, the second whether it can open your page when a person asks, the third is training only (your choice, never scored).
| Platform | Finds and cites you | Opens your page when asked | Training |
|---|---|---|---|
| ChatGPT | OAI-SearchBot | ChatGPT-User | GPTBot |
| Claude | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | none |
| Google AI Overviews, AI Mode, Gemini | Googlebot | Google-Agent, Google-GeminiNotebook | Google-Extended (robots.txt token) |
| Microsoft Copilot | bingbot | uses the Bing index | none published |
| Siri and Apple Intelligence | Applebot | uses Applebot | Applebot-Extended (robots.txt token) |
| Meta AI | meta-webindexer | meta-externalfetcher | meta-externalagent |
| Mistral Le Chat | MistralAI-Index | MistralAI-User | MistralAI-Training |
| Alexa | Amzn-SearchBot | Amzn-User | Amazonbot |
| DuckDuckGo AI answers | DuckAssistBot | same crawler | none |
| You.com | YouBot | same crawler | none |
Also reported, never scored: CCBot (Common Crawl, the dataset many models train on), Bytespider (ByteDance), AI2Bot, Diffbot, GoogleOther and Google-CloudVertexBot.
Why robots.txt checkers get AI crawler access wrong
A robots.txt checker reads the file and reports "allowed". Cloudflare, Vercel, Akamai and the other edge networks block AI crawlers with firewall rules that never appear in that file, and Cloudflare's defaults for new domains block AI training and agent crawlers on pages with ads (a change it set for September 15, 2026). This tool makes the live request as each crawler and compares it with what a browser and an unknown bot get. If the AI crawler is refused while the others get in, a rule names that crawler, and the tool says which layer.
On October 22, 2026, theverge.com answered a browser, our unknown bot and Googlebot with HTTP 200. Its robots.txt disallowed Claude-SearchBot on line 101 and PerplexityBot on line 293, and the live requests for both came back 403. Access score 62, partly hidden. On agent standards the same run scored Level 1: robots.txt and a sitemap in place and AI crawlers named, with no Content Signals, no markdown, no Link headers and no DNS-AID records.
The honest limit: our requests come from Searchbloom's server, not from OpenAI's or Anthropic's published IP ranges. An edge that verifies crawlers by network identity will challenge our test while letting the real crawler through. When that happens, the tool reports "could not test" and leaves those crawlers out of the score instead of calling them blocked. The standards checks follow the same rule: a request the site's bot filter refused reads "Could not check", never "Fail".
The 23 agent standards, in plain English
| Check | What it does | Who needs it |
|---|---|---|
| Discoverability | ||
| robots.txt | The first file every crawler reads: the access policy and the sitemap pointer. | Every site |
| Sitemap | Lists the real URLs so crawlers and agents do not have to guess. | Every site |
| Link headers | Point an agent to machine-readable resources from the HTTP headers alone. | Every site |
| DNS-AID | DNS records that tell an agent the site has agent endpoints before it loads a page. | Every site |
| Content | ||
| Markdown for agents | Returns markdown when an agent asks for it, which costs far fewer tokens to read than HTML. | Every site |
| Bot access control | ||
| AI bot rules | robots.txt rules that name the AI crawlers, so each gets a deliberate policy. | Every site |
| Content Signals | A robots.txt line that says whether content may be used for search, AI answers and training. | Every site |
| Web Bot Auth | A signing-key directory that lets other sites verify your own bots. | Sites that send bots (not scored) |
| API, auth, MCP and skills | ||
| API catalog | Lists the site's APIs at a standard address. | Sites with an API |
| OAuth discovery | Tells an agent how to get a token. | Sites with sign-in for agents |
| OAuth protected resource | Links a protected API to its authorization server. | Sites with a protected API |
| auth.md | A readable page that tells agents how to register and authenticate. | Sites with agent sign-up |
| MCP server card | Describes a Model Context Protocol server and its tools. | Sites that run an MCP server |
| A2A agent card | Advertises an agent that other agents can talk to. | Sites that run an agent |
| Agent skills index | Lists skill documents an agent can load to work with the site. | Sites that publish skills |
| WebMCP | Registers the site's real actions, such as search or contact, as tools a browser agent can call. | Sites with actions to expose |
| ARD manifest | One index of every agent interface the site has. | Sites with agent interfaces |
| Personal Agent Protocol | A draft standard for agents acting for a signed-in user. | Early adopters (not scored) |
| Commerce | ||
| x402, MPP, UCP, ACP, AP2 | Declare that the site accepts payments from agents. | Stores that take agent payments (not scored) |
A card or manifest for an interface that does not exist sends agents to a dead end, which is worse than no card. That is why a content site is scored on the first seven checks only, and why nothing in the commerce row ever costs points.
What the scores mean
Agent readiness levels
- Level 0 Hard for agents to find. robots.txt or the sitemap is missing.
- Level 1 Crawlable. robots.txt and a sitemap are in place.
- Level 2 AI-aware. robots.txt states a policy for AI crawlers and Content Signals for AI use.
- Level 3 Agent-readable. Agents can ask for markdown and find resources from Link headers.
- Level 4 Agent-discoverable. DNS-AID records and a discovery document advertise what agents can use.
- Level 5 Agent-operable. Agents can call a real interface: an MCP server, an A2A agent or WebMCP tools.
The percentage next to the level is the share of scored checks that pass. Checks the site's bot filter kept us from reading are left out, never counted as failures.
AI crawler access
- 95-100 Fully visible. Nothing stands between the AI crawlers and the site.
- 80-94 Mostly visible. One or two fetchers are turned away.
- 60-79 Partly hidden. A generic bot filter or a shared rule catches AI crawlers.
- 30-59 Largely hidden. A CDN, firewall or robots.txt rule blocks most AI search crawlers.
- 0-29 Hidden from AI search. The platforms that find and cite sites cannot reach this one.
Each crawler carries a weight for how much it matters to being found and cited, so blocking OAI-SearchBot costs more than blocking a small platform's crawler. Training crawlers carry no weight.
Ready for agents is step one. Being recommended is step two.
Once crawlers and agents can reach and read the site, the question becomes what the AI platforms say about you. The AI Visibility Checker asks five buyer questions on ChatGPT, Google AI Mode and Gemini and shows every answer, including whether your brand was named and your site cited. Moving those numbers is the work of answer engine optimization and generative engine optimization: a clear entity (look yours up in the Knowledge Graph Explorer), complete structured data, an AI-information page, consistent facts across the web, and third-party corroboration.
Questions and Answers
What does the Agent Readiness Checker test?
Two things in one run. First, it requests your robots.txt and homepage from our server as a real browser, as two control bots and with the exact user agents of 32 AI crawlers, and reports whether each one gets in, which layer turned it away, and how to fix it. Second, it reads the 23 agent standards (robots.txt AI rules, Content Signals, sitemap, Link headers, DNS-AID, markdown, the API, OAuth, MCP, A2A and agent skills documents, WebMCP and the commerce documents) and shows the evidence behind each result.
How do I check if a website is AI-agent ready?
Enter the site above. A site is ready when the AI crawlers that find and cite pages can reach it, and when it publishes the standards agents read: a robots.txt with AI rules and Content Signals, a sitemap, markdown on request, Link headers and DNS-AID records. Sites with an API or an agent also publish the matching cards. The checker scores both and lists the fix for every gap.
What is an AI crawler?
A bot an AI company runs to fetch web pages. Most companies run three kinds: a search crawler that indexes pages for the assistant's answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), a fetcher that opens a page live when a person asks about it (ChatGPT-User, Claude-User), and a training crawler that collects data for future models (GPTBot, ClaudeBot). Each one can be allowed or blocked on its own.
What is GPTBot?
GPTBot is OpenAI's training crawler. It collects public pages that may be used to train OpenAI's models. ChatGPT search runs on a different crawler, OAI-SearchBot, and ChatGPT-User opens pages when a person asks. Blocking GPTBot keeps content out of training without removing a site from ChatGPT's answers. Blocking OAI-SearchBot or ChatGPT-User does remove it.
What is the Google-Extended user agent?
Google-Extended is a robots.txt token, not a separate crawler: Google's existing crawlers do the fetching. Disallowing it tells Google not to use your pages to train Gemini models or to ground answers in the Gemini apps. According to Google's crawler documentation, it does not affect inclusion in Google Search, and AI Overviews and AI Mode are part of Search.
Can AI crawlers execute JavaScript?
Mostly no. Vercel's study of AI crawler traffic found that none of the major AI crawlers render JavaScript; they download the scripts but do not run them. Gemini (which uses Googlebot's infrastructure) and Applebot are the exceptions. Content that only appears after JavaScript runs is invisible to ChatGPT, Claude and Perplexity, which is why the checker tests whether your text is in the raw HTML.
How do I allow AI crawlers in robots.txt?
Add one group that stacks the User-agent lines for the AI crawlers you want (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and the rest), then any Disallow lines for private paths, then Allow: / and a Content-Signal line. Then check the edge: a CDN or firewall can still block crawlers that robots.txt allows. The checker tests both, and its fix gives you the group to paste.
Is my website blocking ChatGPT?
Check three crawlers, not one. OAI-SearchBot decides whether ChatGPT search can find and cite your pages, ChatGPT-User opens a page live when someone asks ChatGPT about it, and GPTBot only collects training data. The checker shows what each of the three actually gets from your site, not just what robots.txt says.
Can my website block AI crawlers even if robots.txt allows them?
Yes, and a robots.txt checker cannot see it. Cloudflare, Vercel, Akamai and other edge providers block AI crawlers with firewall rules that never appear in robots.txt, and WordPress security plugins do the same at the server. This tool tests the live response and compares it with what a browser gets.
Does my site need an MCP server card or an A2A agent card?
Only if it runs an MCP server or an A2A agent. Those cards, the OAuth documents and auth.md describe interfaces an agent can call, and a card for a server that does not exist sends agents to a dead end. That is why the checker scores a content site without them. Choose API or app to score them.
How accurate are the results?
Exact for robots.txt and for rules keyed on the user-agent string, which is how Cloudflare AI Crawl Control, Vercel's AI bot rules, .htaccess rules and most firewall rules work. Our requests come from our server, not the AI companies' IP ranges, so a rule keyed on verified identity shows as "could not test" and is left out of the score. The standards are read the way an agent reads them. WebMCP is read from the page source and its scripts, so a tool registered by code we cannot read would be missed.
Is the checker free and do I need to sign up?
Free, no sign-up, no limit, and results are never gated. Every result has a share link that works for 30 days, plus JSON and CSV downloads.
Searchbloom opens your site to the AI crawlers and agents, then gets your brand into their answers. Get a free action plan.
