AI Crawler Controls
Choose training and data preferences separately from AI search and user fetchers
Choose the scope you want
ProRank adds robots.txt preferences for the crawler groups you select. In 1.6.4, the training switch covers training and data crawlers. AI search and user fetchers are included only when you select their separate category.
This lets you request a training opt-out while keeping AI discovery available. Restricting search crawlers may reduce your visibility and referrals from those services.
Robots.txt expresses crawl preferences; it does not prevent access to a public page. Compliance depends on the provider and the type of request. Use authentication or server access controls for private content.
Set your preferences
- Open Technical SEO → Robots & Indexing → Robots.txt.
- Enable Block AI/ML Training Bots via Robots.txt to select the training and data groups, or leave it off and choose individual groups.
- Review AI search and user fetchers separately. Leave this category unselected if you want to allow those services under your other robots.txt rules.
- Save, then check the public
/robots.txtoutput.
A physical robots.txt file or another tool may take precedence over WordPress output. Check the public file after saving, including any custom rules you already use.
Crawler groups in ProRank 1.6.4
| Group | Robots.txt tokens |
|---|---|
| OpenAI training | GPTBot |
| Google model training and grounding | Google-Extended |
| Anthropic training | ClaudeBot |
| Apple AI training | Applebot-Extended |
| Meta model training | Meta-ExternalAgent |
| AI search and user fetchers — separate selection | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User |
| Image and media data crawlers | Bytespider, Diffbot, img2dataset |
| Research and training datasets | AI2Bot, CCBot, omgilibot |
These are the tokens included in this plugin version. Provider behavior can change; review the linked provider documentation when choosing your policy.
Example training preferences
Selecting OpenAI and Anthropic training adds these groups. The AI search category has its own selection and is not added by this example.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /Provider differences
- OpenAI separates GPTBot training from OAI-SearchBot search. It says robots.txt rules may not apply to user-initiated ChatGPT-User requests. OpenAI crawler documentation
- Anthropic documents separate ClaudeBot, Claude-SearchBot and Claude-User controls. Anthropic crawler documentation
- Perplexity distinguishes its search crawler from Perplexity-User, which generally ignores robots.txt for user-requested fetches. Perplexity crawler documentation
- Google-Extended is separate from Googlebot and is not a switch for Google Search AI features. Review Google's Search controls for those features. Google Search AI guidance
Image and content safeguard settings
Image Optimisation's No AI Training setting adds uploads rules for named image and data crawlers. It does not add a blanket uploads restriction for every crawler. Check your other rules before assuming images are crawlable.
Content Safeguard can also add noai and noimageai signals. These are voluntary preferences with limited adoption, not an access restriction or a guarantee that a provider will exclude content from training.
Verify the result
- Open your public robots.txt and confirm the intended groups and paths.
- Use the robots tester to review a specific crawler and URL together.
- Check custom rules and server or CDN settings if the public output differs.
A successful page request using a bot's user-agent does not show that robots.txt is failing: the file gives instructions to the crawler, rather than denying HTTP requests.