Free robots.txt checker for AI and search bots

Type your website to see whether its robots.txt lets Google, ChatGPT and other AI bots in. A robots.txt file tells bots which pages they may visit.

Example result for medium.com

This example checks medium.com's home page against a copy of its robots.txt file saved on October 21, 2026.

Google: Allowed No robots.txt rule blocks this page for Googlebot.
Bing and Copilot: Allowed No robots.txt rule blocks this page for Bingbot.
ChatGPT: Allowed No robots.txt rule blocks this page for OAI-SearchBot.
Claude: Allowed No robots.txt rule blocks this page for Claude-SearchBot.
Perplexity: Allowed No robots.txt rule blocks this page for PerplexityBot.
Gemini: Allowed No robots.txt rule stops Google from using this page for Gemini (Google-Extended).

This example checks robots.txt, not whether medium.com's server lets these bots through.

AI training bots: 4 of 5 blocked. Blocking them is the site's choice, and it does not affect Google or Bing.

Help with these results

Each row is decided by one robots.txt token, named here.

Google: Googlebot: Allowed. It reads pages for Google Search, including AI Overviews and AI Mode.
Bing and Copilot: Bingbot: Allowed. It is Bing's main crawler. Bing says Copilot's search relies on the same crawling and indexing.
ChatGPT: OAI-SearchBot: Allowed. It surfaces websites in ChatGPT's search features.
Claude: Claude-SearchBot: Allowed. It crawls the web to improve search results for Claude users.
Perplexity: PerplexityBot: Allowed. It surfaces and links websites in Perplexity search results.
Gemini: Google-Extended: Allowed. It tells Google whether pages it crawls may be used to train Gemini and to ground answers in Gemini Apps. It is not a crawler of its own.

See the robots.txt file

User-Agent: *
Disallow: /m/
Disallow: /me/
Disallow: /@me$
Disallow: /@me/
Disallow: /*/edit$
Disallow: /*/*/edit$
Disallow: /media/
Disallow: /p/*/share
Disallow: /r/
Disallow: /trending
Disallow: /search?q$
Disallow: /search?q=
Disallow: /*/search?q=
Disallow: /*/search/*?q=
Disallow: /*/*source=
Allow: /_/api/users/*/meta
Allow: /_/api/users/*/profile/stream
Allow: /_/api/posts/*/responses
Allow: /_/api/posts/*/responsesStream
Allow: /_/api/posts/*/related
User-Agent: Amazonbot
User-Agent: Applebot-Extended
User-Agent: Bytespider
User-Agent: ClaudeBot
User-Agent: FacebookBot
User-Agent: GoogleOther
User-Agent: GPTBot
User-Agent: meta-externalagent
Disallow: /
Allow: /about
Allow: /business
Allow: /earn
Allow: /gift
Allow: /membership
Allow: /partner-program
Allow: /verified-authors
Sitemap: https://medium.com/sitemap/sitemap.xml
License: https://medium.com/license.xml

See all 65 bots

Search engine bots: 14 allowed, 2 unclear.

Bot	Company	Result	Why
`Amzn-SearchBot`	Amazon	Allowed	No rule covers this page.
`Applebot`	Apple	Allowed	No rule covers this page.
`Baiduspider`	Baidu	Unclear	RFC 9309: allowed. No rule covers this page. Baidu's documentation: either. Why they differ
`Baiduspider-image`	Baidu	Unclear	RFC 9309: allowed. No rule covers this page. Baidu's documentation: either. Why they differ
`Bingbot`	Microsoft	Allowed	No rule covers this page.
Brave Search crawler	Brave	Allowed	No rule covers this page.
`Diffbot`	Diffbot	Allowed	No rule covers this page. Diffbot's crawls follow robots.txt by default, including disallow and crawl-delay, but can ignore it in specific cases, typically under an agreement with the crawled site.
`DuckDuckBot`	DuckDuckGo	Allowed	No rule covers this page.
`Googlebot`	Google	Allowed	No rule covers this page.
`Googlebot-Image`	Google	Allowed	No rule covers this page.
`Googlebot-News`	Google	Allowed	No rule covers this page.
`Googlebot-Video`	Google	Allowed	No rule covers this page.
`Storebot-Google`	Google	Allowed	No rule covers this page.
`YandexBot`	Yandex	Allowed	No rule covers this page.
`YandexImages`	Yandex	Allowed	No rule covers this page.
`YandexVideo`	Yandex	Allowed	No rule covers this page.

AI search bots: 6 allowed.

Bot	Company	Result	Why
`Claude-SearchBot`	Anthropic	Allowed	No rule covers this page.
`DuckAssistBot`	DuckDuckGo	Allowed	No rule covers this page.
`meta-webindexer`	Meta	Allowed	No rule covers this page.
`MistralAI-Index`	Mistral AI	Allowed	No rule covers this page.
`OAI-SearchBot`	OpenAI	Allowed	No rule covers this page.
`PerplexityBot`	Perplexity	Allowed	No rule covers this page.

AI training bots: 1 allowed, 4 blocked.

Bot	Company	Result	Why
`Amazonbot`	Amazon	Blocked	Line 30: `Disallow: /`
`ClaudeBot`	Anthropic	Blocked	Line 30: `Disallow: /`
`GPTBot`	OpenAI	Blocked	Line 30: `Disallow: /`
`meta-externalagent`	Meta	Blocked	Line 30: `Disallow: /`
`MistralAI-Training`	Mistral AI	Allowed	No rule covers this page.

Bots that open a page when a person asks: 5 allowed, 11 not decided by robots.txt.

Bot	Company	Result	Why
`Amzn-User`	Amazon	Allowed	No rule covers this page. Amazon says that because its actions can be user-initiated, Amzn-User may not follow all robots.txt directives.
`ChatGPT-User`	OpenAI	Not decided by robots.txt	OpenAI documents no robots.txt token for it. OpenAI says robots.txt rules may not apply because these actions are user-initiated, and names only OAI-SearchBot and GPTBot as robots.txt tags. ChatGPT-User is not used to decide Search inclusion.
`Claude-User`	Anthropic	Allowed	No rule covers this page.
`Diffbot-User`	Diffbot	Allowed	No rule covers this page. Diffbot says its crawls follow robots.txt by default but can ignore it under an agreement with the crawled site. It lists Diffbot-User among the user agents to name in robots.txt.
`FeedFetcher-Google`	Google	Not decided by robots.txt	Google documents no robots.txt token for it. Google says Feedfetcher ignores robots.txt because its requests come from explicit user action; it suggests serving a 404, 410 or other error to the Feedfetcher-Google user agent instead.
`Google-Agent`	Google	Not decided by robots.txt	Google says fetchers that act for a person generally ignore robots.txt. Google gives them no robots.txt token, so we show the name each fetcher sends when it visits.
`Google-CWS`	Google	Not decided by robots.txt	Google says fetchers that act for a person generally ignore robots.txt. Google gives them no robots.txt token, so we show the name each fetcher sends when it visits.
`Google-GeminiNotebook`	Google	Not decided by robots.txt	Google says fetchers that act for a person generally ignore robots.txt. Google gives them no robots.txt token, so we show the name each fetcher sends when it visits.
`Google-Pinpoint`	Google	Not decided by robots.txt	Google says fetchers that act for a person generally ignore robots.txt. Google gives them no robots.txt token, so we show the name each fetcher sends when it visits.
`Google-Read-Aloud`	Google	Not decided by robots.txt	Google documents no robots.txt token for it. Google says you cannot opt out with robots.txt because Read Aloud is user-initiated; it offers the nopagereadaloud meta tag instead. The older agent name google-speakr is deprecated.
`Google-Site-Verification`	Google	Not decided by robots.txt	Google says fetchers that act for a person generally ignore robots.txt. Google gives them no robots.txt token, so we show the name each fetcher sends when it visits.
`GoogleMessages`	Google	Not decided by robots.txt	Google says fetchers that act for a person generally ignore robots.txt. Google gives them no robots.txt token, so we show the name each fetcher sends when it visits.
`GoogleProducer`	Google	Not decided by robots.txt	Google says fetchers that act for a person generally ignore robots.txt. Google gives them no robots.txt token, so we show the name each fetcher sends when it visits.
`meta-externalfetcher`	Meta	Allowed	No rule covers this page. Meta says it may bypass robots.txt because it performs fetches requested by the user.
`MistralAI-User`	Mistral AI	Allowed	No rule covers this page.
`Perplexity-User`	Perplexity	Not decided by robots.txt	Perplexity says this fetcher generally ignores robots.txt because a user requested the fetch, although the same page lists Perplexity-User among the robots.txt tags webmasters can use. The rules would allow it.

AI opt-outs: 1 opted out, 2 allowed.

Bot	Company	Result	Why
`Applebot-Extended`	Apple	Opted out	Line 30: `Disallow: /`
`Google-Extended`	Google	Allowed	No rule covers this page.
`YandexAdditionalBot`	Yandex	Allowed	No group names it, and it skips the * group.

Other bots: 13 allowed, 3 blocked, 3 not decided by robots.txt.

Bot	Company	Result	Why
`AdIdxBot`	Microsoft	Allowed	No rule covers this page.
`AdsBot-Google`	Google	Allowed	No group names it, and it skips the * group. Ignores the global user agent (*) with the ad publisher's permission; rules addressed to AdsBot-Google apply.
`AdsBot-Google-Mobile`	Google	Allowed	No group names it, and it skips the * group. Ignores the global user agent (*); rules addressed to AdsBot-Google-Mobile apply.
`APIs-Google`	Google	Allowed	No group names it, and it skips the * group. Ignores the global user agent (*); rules addressed to APIs-Google apply.
`BingPreview`	Microsoft	Allowed	No rule covers this page.
`BingVideoPreview`	Microsoft	Allowed	No rule covers this page.
`CCBot`	Common Crawl	Allowed	No rule covers this page.
`facebookexternalhit`	Meta	Allowed	No rule covers this page. Meta says it might bypass robots.txt when performing security or integrity checks.
`Google-CloudVertexBot`	Google	Allowed	No rule covers this page.
`Google-InspectionTool`	Google	Allowed	No rule covers this page.
`Google-Safety`	Google	Not decided by robots.txt	Google documents no robots.txt token for it. Google says this user agent ignores robots.txt rules.
`GoogleOther`	Google	Blocked	Line 30: `Disallow: /`
`GoogleOther-Image`	Google	Blocked	Line 30: `Disallow: /`
`GoogleOther-Video`	Google	Blocked	Line 30: `Disallow: /`
`iTMS`	Apple	Not decided by robots.txt	Apple says the iTMS user agent does not follow robots.txt because it is not a general search crawler. Apple documents no robots.txt token for it.
`Mediapartners-Google`	Google	Allowed	No group names it, and it skips the * group. Ignores the global user agent (*); rules addressed to Mediapartners-Google apply.
`meta-externalads`	Meta	Allowed	No rule covers this page.
`MicrosoftPreview`	Microsoft	Allowed	No rule covers this page.
`OAI-AdsBot`	OpenAI	Not decided by robots.txt	OpenAI documents no robots.txt token or rules for OAI-AdsBot. It says the bot visits only pages submitted as ads.

Next: check whether AI mentions your business

After reviewing your robots.txt rules, you may want to track what AI says about your business. Discoverable combines monitoring with a list of suggested tasks and draft content. Otterly is another option for regularly checking a chosen set of questions.

Our sister site PriceFern compares their cheapest plans.

The crawler list says what each bot does.

Still blocked?

A bot your robots.txt allows can still be turned away. These fixes cover common causes:

403 and 429 errors for Googlebot and Bingbot Cloudflare blocks a crawler you meant to allow robots.txt returns errors, redirects or HTML