How to Block All AI Crawlers in robots.txt

There are over a dozen AI crawlers roaming the web right now, each collecting content for different AI companies. Blocking one isn't enough if the others are still helping themselves to your content. Here's the complete list and the exact robots.txt rules to block them all.

The Complete AI Crawler List (2026)

Every known AI crawler that respects robots.txt, who operates it, and what it does:

The Copy-Paste Block (All AI Crawlers)

Add this to your robots.txt. It blocks every known AI crawler while keeping search engines fully allowed:

# ==========================================
# AI Crawlers: BLOCKED
# ==========================================

# OpenAI
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

# Anthropic
User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

# Google AI (not search)
User-agent: Google-Extended
Disallow: /

# Perplexity
User-agent: PerplexityBot
Disallow: /

# Common Crawl
User-agent: CCBot
Disallow: /

# ByteDance
User-agent: Bytespider
Disallow: /

# Meta
User-agent: meta-externalagent
Disallow: /

# Apple
User-agent: Applebot-Extended
Disallow: /

# Cohere
User-agent: cohere-ai
Disallow: /

# Diffbot
User-agent: Diffbot
Disallow: /

# Webz.io
User-agent: Omgilibot
Disallow: /

# ==========================================
# Search Engines: ALLOWED
# ==========================================

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Understanding the Trade-offs

Blocking all AI crawlers is the maximum-protection option. But it's not free -- here's what you're giving up:

• AI visibility -- ChatGPT, Claude, and Perplexity won't reference your content accurately when users ask about topics in your space.
• AI search traffic -- Perplexity and similar AI search tools won't link to your pages in their answers.
• Future AI products -- As AI becomes more integrated into how people find information, being absent from training data means being absent from answers.

What you keep:

• Content control -- Your work isn't used to train models without your consent.
• Search rankings -- Google and Bing rankings are completely unaffected.
• Server resources -- Several AI crawlers are aggressive. Blocking them reduces unnecessary server load.

Why You Can't Use a Single Wildcard Rule

A common question: "Can't I just add one rule to block all AI bots?"

No. robots.txt doesn't have an "AI crawler" category. The wildcard User-agent: * blocks everything, including search engines. Each AI crawler uses a unique user-agent string, so you need a separate rule for each one.

That's why using a robots.txt generator with AI presets saves time. One click generates all the rules correctly.

The CCBot Wildcard

CCBot deserves special attention. Common Crawl maintains the largest public web dataset, and many AI companies (including some that don't have their own crawlers) use Common Crawl data for training. Blocking CCBot is like closing a back door -- even if you block GPTBot directly, OpenAI might still access your content through Common Crawl's dataset if CCBot was allowed.

Note: Common Crawl data that was already collected before your block remains in their archive. The block only prevents future crawls.

Beyond robots.txt: Additional Protection

robots.txt is the minimum. For stronger protection:

• HTTP headers -- Some crawlers respect X-Robots-Tag: noai or similar headers, though there's no universal standard yet.
• IP blocking -- Major AI companies publish their crawler IP ranges. You can block these at the firewall level for hard enforcement.
• Rate limiting -- If you want to allow some crawling but limit volume, implement rate limiting at the server level.
• TDM Reservation Protocol -- The EU's TDM (Text and Data Mining) protocol via tdmrep.json provides a legal framework for content usage policies.

Keeping the List Updated

New AI crawlers appear regularly. The list above is current as of April 2026, but it will grow. Strategies to stay current:

1. Monitor your server access logs for unfamiliar bot user-agents
2. Check our AI Readiness Checker periodically -- we update the crawler list as new ones appear
3. Follow announcements from major AI companies about new crawlers