AI Crawler Guides

Practical, no-nonsense guides for controlling AI bots in your robots.txt. Each guide includes copy-paste code, verification steps, and the trade-offs you need to know about.

How to Block GPTBot (OpenAI)

Block OpenAI's crawler from collecting training data. Covers GPTBot and ChatGPT-User, with full robots.txt examples and verification steps.

OpenAI

How to Block ClaudeBot (Anthropic)

Block Anthropic's crawler from your site. Covers ClaudeBot and anthropic-ai, crawl-delay configuration, and comparison with GPTBot behavior.

Anthropic

How to Block Google-Extended (Gemini)

Block Google's AI training without losing search rankings. How Google-Extended differs from Googlebot, and the AI Overviews nuance most guides miss.

Google

Block All AI Crawlers (Complete List 2026)

The full block list: GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider, CCBot, Meta, Apple, Cohere, and more. One copy-paste to block them all.

All Crawlers

Allow AI Crawlers (And Why You Should)

The strategic case for letting AI crawl your site. How to appear in ChatGPT, Claude, and Perplexity answers. Covers llms.txt, GEO, and selective access.

Strategy

How to Create an llms.txt File (Step-by-Step)

Build an llms.txt file so AI models like ChatGPT, Claude, and Perplexity can accurately describe and recommend your site. Full format spec, templates, and common mistakes.

llms.txt

robots.txt for WordPress: Block AI Bots, Allow Search Engines

The complete WordPress robots.txt guide. Block GPTBot, ClaudeBot, and 15+ AI crawlers while keeping Google and Bing. Plugin, manual, and code methods.

WordPress

Complete List of AI Crawlers in 2026 (With User-Agent Strings)

Every known AI crawler with exact user-agent strings, owning companies, purpose, and robots.txt compliance. Updated monthly. Copy-paste blocking rules included.

Reference

Check Your AI Crawler Configuration

Scan your site to see which AI crawlers are blocked or allowed, check for llms.txt, and get your AI readiness score.

AI Readiness Checker Robots.txt Generator