Technical
Which crawlers to let in
Blocking AI crawlers is a legitimate choice. Blocking them by accident, while trying to be found by AI, is the one to avoid.
These are the readers your website is being written for. Whether they can reach you is one line in robots.txt — and the most common reason a site is invisible.
Several AI companies operate crawlers, and they do not all do the same job. Some collect data for training. Some fetch pages live when a user asks a question. Blocking the second kind is what removes you from answers.
The main user-agents
| User-agent | Operator | Roughly what it does |
|---|---|---|
GPTBot | OpenAI | Crawls for model training |
OAI-SearchBot | OpenAI | Fetches pages for search results in ChatGPT |
ClaudeBot | Anthropic | Crawls for training |
PerplexityBot | Perplexity | Indexes for its answer engine |
Bingbot | Microsoft | Indexes for Bing Search and feeds Copilot answers |
Google-Extended | Controls use in Gemini and AI training, not Search | |
Applebot-Extended | Apple | Controls use in Apple AI training |
CCBot | Common Crawl | Open crawl used by many downstream datasets |
meta-externalagent | Meta | Crawls for Meta AI |
The distinction that matters most: Google-Extended does not affect your normal Google Search ranking. Blocking it removes you from Gemini-related uses only. People block it thinking it protects Search, which it does not do and does not need to do.
Copilot: get into Bing, not into robots.txt
There is no CopilotBot. Microsoft does not run a separate AI crawler — Copilot answers are built from Bing's index, gathered by Bingbot. That single fact changes what you should do about it.
To be visible in Copilot, work on being visible in Bing. Practically:
- Sign up to Bing Webmaster Tools. Free, and it is the only place you see Bing's view of your website.
- Submit your sitemap. Bing does not find pages as aggressively as Google does, so this matters more here than it does there.
- Import from Google Search Console if you already have it. Bing Webmaster Tools can pull your verification and sitemaps across in a couple of clicks rather than making you start again.
- Use IndexNow for anything time-sensitive. It pings Bing the moment a page changes instead of waiting for a crawl. Most CMSs have a plugin, and Cloudflare can switch it on for you.
- Check the crawl and index reports for pages Bing has not picked up. A page Bing cannot see is a page Copilot cannot cite.
This is the least glamorous item in this guide and one of the highest-return, because Bing gets a fraction of the attention Google does while feeding an assistant sitting inside every Microsoft 365 subscription.
If you want the opposite
Microsoft's AI controls are page-level meta tags rather than robots.txt rules, so blocking Bingbot is a blunt instrument — it removes you from Bing Search and Copilot together.
<!-- out of Copilot answers and Microsoft AI training, still in Bing Search --> <meta name="robots" content="noarchive"> <!-- Copilot may use your URL, title and snippet, but not the full page --> <meta name="robots" content="nocache">
Content marked noarchive or nocache still appears in Bing search results.
Microsoft 365 Copilot's agentic features use ordinary Edge and Chromium user-agent strings with no bot signal. In your logs that traffic looks like a person using Edge, so no robots.txt rule reaches it either way.
The recommended configuration
If your goal is to be found and cited, allow the lot:
User-agent: * Allow: / User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Bingbot Allow: / User-agent: Google-Extended Allow: / Sitemap: https://example.com/sitemap.xml
The honest trade-off
There is a real argument on the other side, and we are not going to pretend otherwise.
Reasons to block: if your content is your product — a paid archive, a research library, original photography — you may not want it absorbed into a training set you get nothing from. Publishers with licensing deals block precisely so they can charge.
Reasons to allow: if your website exists to bring you customers, being absent from the answer layer costs you more than the training use costs you. A local business gains nothing from being excluded.
A middle path: allow the retrieval crawlers that put you in answers (OAI-SearchBot, PerplexityBot) and block the pure training crawlers (GPTBot, CCBot). This is a defensible position, though the line between the two blurs and the user-agent list changes.
Open yoursite.com/robots.txt. A stray Disallow: / under a wildcard, or a security plugin blocking unknown agents, is the most common cause of a genuinely invisible site.
Two things robots.txt cannot do
It cannot enforce anything — it is a request, and badly-behaved crawlers ignore it. And it cannot make a slow, JavaScript-dependent page readable. Permission is necessary, not sufficient.
Build your file with the AI robots.txt builder.
Common questions
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended governs use in Gemini and AI training. Normal Google Search crawling and ranking is controlled by Googlebot and is unaffected.
How do I get into Microsoft Copilot?
Through Bing. There is no separate Copilot crawler — Copilot answers are built from Bing's index, gathered by Bingbot. So sign up to Bing Webmaster Tools, submit your sitemap, import your setup from Google Search Console if you have it, and use IndexNow for time-sensitive pages.
How do I keep Copilot from using my content?
Not through robots.txt, because blocking Bingbot removes you from Bing Search too. Microsoft's controls are page-level meta tags: noarchive keeps you out of Copilot answers and Microsoft AI training while staying in Bing Search, and nocache limits Copilot to your URL, title and snippet.
Will blocking GPTBot remove me from ChatGPT answers?
Not necessarily, because ChatGPT's live search uses OAI-SearchBot. Blocking both removes you from both training and retrieval.
Do AI crawlers respect robots.txt?
The major operators state that they do. It is a voluntary convention, so it offers no technical enforcement.
Take this to your assistant
Paste it into ChatGPT, Copilot, Claude or Gemini and apply it to your own website.
Nothing is sent anywhere. The text is copied to your clipboard.