A few years ago, the only bots you had to worry about were search engine crawlers. You wanted Googlebot and Bingbot on your site — they sent you traffic. Everything else was noise.
That’s not true anymore.
Today, dozens of AI companies run crawlers. They scrape your articles, product pages, and blog posts. Some feed model training. Others power real-time answer engines.
Some bots ask nicely. Some don’t ask at all.
Unlike Googlebot, most send you nothing back. No clicks. No citations. No traffic. Just a server bill.
Maybe you’ve noticed traffic spikes in your logs. Maybe you’re tired of seeing your writing paraphrased in a chatbot answer.
This guide shows you how to take control. We’ll start with simple fixes you can do in ten minutes. Then we’ll cover server-level defenses that stop bots that ignore the rules.
First, Understand What You’re Actually Blocking
Not all AI bots do the same job. Lumping them together leads to mistakes.
AI bots fall into three broad categories:
- Training crawlers scrape your site to build the dataset a model learns from. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Meta-ExternalAgent (Meta), Bytespider (ByteDance).
- Search/retrieval crawlers index your pages so an AI assistant can cite or link to them in a live answer. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- User-triggered fetchers grab a specific page the instant a person asks an AI assistant about it. Examples: ChatGPT-User, Perplexity-User, Claude-User.
Here’s what people get wrong. Blocking a training bot doesn’t affect your AI search visibility. They’re different crawlers with different names.
Example: block GPTBot but leave OAI-SearchBot alone. OpenAI can’t train on your content. But ChatGPT Search can still cite your page.
Mix these up and you’ll make a mistake. You might leak content you meant to protect. Or you might erase yourself from AI search results you wanted to be part of.
Decide what you actually want before you start blocking anything:
- Want out of training data entirely, but still cited in AI answers? Block training bots, allow search/retrieval bots.
- Want zero AI presence, period? Block all three categories.
- Only worried about specific companies? Target their bots individually rather than using a blanket rule.
Step 1: Set Up Your robots.txt File Correctly
robots.txt is still your starting point. It’s a plain text file at yoursite.com/robots.txt.
This file tells well-behaved bots what they can and can’t touch. Every reputable AI company confirms this in their docs — OpenAI, Anthropic, Google, Apple, and Perplexity all check this file before they crawl.
A basic block for the major training crawlers looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
A couple of things worth knowing before you copy-paste this:
- Google-Extended and Applebot-Extended are training opt-outs, not search opt-outs. Block them, and you stop your content from training Gemini or Apple Intelligence. Your regular Google or Siri search visibility stays the same. Blocking these won’t hurt your rankings.
- Never block plain Googlebot or Bingbot. That’s a different bot from Google-Extended. Block it and you disappear from search results.
- Spelling has to be exact. Write “GPT-Bot” instead of “GPTBot” and the rule does nothing. The crawler just ignores a rule that doesn’t match its name.
Don’t want to hand-write this? Use a robots.txt generator instead.
It lets you set a custom rule for each crawler by name. You add your sitemap and copy out a clean, correctly formatted file. No guessing at syntax.
This matters more than it sounds. A single misplaced slash in a hand-written file has taken entire sites out of Google’s index by accident.
Once you have the file, upload it to your site’s root directory. It needs to be reachable at yoursite.com/robots.txt — not inside a subfolder. Crawlers won’t find it otherwise.
Then check Google Search Console’s robots.txt tool. Confirm your existing pages aren’t caught in the rules by accident.
Step 2: Understand That robots.txt Is a Request, Not a Lock
Here’s the part a lot of guides skip: robots.txt works purely on trust. It’s a sign on the door, not a locked door.
A bot has to choose to read the file. Then it has to choose to obey it.
OpenAI, Anthropic, Google, and Apple have all publicly committed to respecting robots.txt. Independent monitoring backs this up — they generally do.
Others don’t. Bytespider, ByteDance’s crawler, has a long history of crawling sites that explicitly disallow it. Some crawlers even rotate IP addresses and user-agent strings just to dodge blocklists.
So if a bot is costing you bandwidth, or your content keeps showing up somewhere you blocked, robots.txt alone won’t stop it. You need a second layer — one that enforces the rule instead of just requesting it.
Step 3: Block Bots at the Server Level
This is where you go from asking politely to actually stopping the request.
Server-level blocking checks each incoming request. It refuses to serve the page. The bot gets a 403 Forbidden error instead of your content.
If you’re on Apache, add this to your .htaccess file:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|CCBot|Bytespider|ClaudeBot) [NC]
RewriteRule .* - [F,L]
If you’re on Nginx, add this inside your server block:
if ($http_user_agent ~* (GPTBot|CCBot|Bytespider|ClaudeBot)) {
return 403;
}
Both approaches inspect the User-Agent header on every request. They reject anything matching your list before your site even loads.
It’s a five-minute change. And it works even against bots that never bothered to read your robots.txt file.
One caveat: user-agent strings can be faked. A determined scraper can simply claim to be Chrome on Windows.
For that level of persistence, you need the next step.
Step 4: Use a Firewall or CDN for Bots That Lie About Who They Are
Running your site through Cloudflare, Sucuri, or a similar service? You already have tools that go beyond user-agent matching.
These services fingerprint traffic based on behavior. They look at request rate, header patterns, and TLS signatures — not just what a bot claims to be.
Cloudflare, for example, has a dedicated AI bot-blocking feature. It maintains its own updated list of known AI crawlers. One toggle blocks them all, independent of your robots.txt.
It also catches bots that spoof their user-agent, because it looks at deeper signals than the header alone.
Not on a service like this? Even a basic Web Application Firewall (WAF) rule helps.
Set a rate limit — say, 60 requests per minute per IP. This stops most scraping tools. Real visitors never come close to that pace.
Step 5: Verify Bots Are Actually Who They Say They Are
Here’s a trick worth knowing: anyone can send a request with User-Agent: GPTBot in the header. That doesn’t mean OpenAI sent it.
Real AI companies publish the actual IP ranges their crawlers use. Check these to verify a request is genuine before you trust it — or block it.
- OpenAI publishes JSON files listing GPTBot, OAI-SearchBot, and ChatGPT-User IP ranges.
- Google publishes IP lists for its crawlers, including Google-Extended.
- Common Crawl publishes CCBot’s ranges.
Your logs show a spike from “GPTBot,” but the IP doesn’t match OpenAI’s published range? That’s an impersonator.
It’s likely a scraper hiding behind a trusted name to slip past simple user-agent blocks. Block that specific IP outright.
Step 6: Add a “Do Not Train” Signal in Your Content
Robots.txt and firewalls stop bots from fetching your pages. But some sites want something different. They want to allow crawling for search visibility, while stating clearly that the content isn’t licensed for AI training.
A few emerging tags help with this. Enforcement still depends on the crawler choosing to honor them, though.
- A
noaiornoimageaimeta tag in your page’s<head>signals that you don’t want the content or images used for AI training. - Your Terms of Service page can state clearly that scraping or reuse for model training is prohibited without a license. This won’t physically stop a bot. But it matters if you ever pursue a legal claim — it establishes clear notice.
Think of this as a paper trail, not a wall. It strengthens your position. It doesn’t replace robots.txt or server blocking.
Step 7: Check Your Logs Regularly
None of this is “set and forget.” New AI crawlers show up several times a year. Companies rename or split their bots, too.
Anthropic, for example, uses separate user-agents for training (ClaudeBot), search indexing (Claude-SearchBot), and live user requests (Claude-User). A blocklist you wrote in 2024 is already missing names.
Set a recurring reminder — monthly is reasonable — to:
- Pull your server logs and search for known AI user-agent strings.
- Check whether any of your blocked bots are still getting through (a sign they’re spoofing).
- Check whether bots you wanted to allow have quietly disappeared. This happens when a security plugin updates and silently re-enables a “block all AI” default.
Plenty of site owners install an SEO or security plugin and forget about it. Months later, they discover a default setting quietly blocked every AI crawler — including the ones sending them real traffic through AI search citations.
Audit your file regularly. Catch this before it costs you visibility you actually wanted.
Putting It All Together
Want a realistic setup instead of an all-or-nothing approach? Here’s the order of operations that works:
- Decide your policy — training bots blocked, search/citation bots allowed, or everything blocked.
- Build your robots.txt file with the correct bot names for that policy.
- Back it up with server-level rules (.htaccess or Nginx) so non-compliant bots can’t just ignore the file.
- Add firewall or CDN protection if you’re on a platform that offers it, to catch spoofed traffic.
- Verify suspicious traffic against published IP ranges before assuming a bot is who it claims to be.
- Review your setup monthly, because this landscape moves fast.
No single step here is bulletproof. Together, they raise the cost of scraping your site.
Compliant bots move on because they respect the rules. Non-compliant bots move on because it’s no longer worth the effort. Either way, they look for easier targets.
Your content took real time and expertise to create. It’s reasonable to decide who gets to use it next.
