Robots.txt is four lines of text that can hide an entire website.That asymmetry — trivial to write, catastrophic to get wrong — is why testing tools exist for a file most people edit twice a year. Here is how to test rules before they ship, the WordPress baseline worth copying, and how the file’s job changed once AI crawlers arrived.
“robots txt” — 9,900 searches/mo (Google, US)Average monthly US search volume. Source: Google autocomplete demand via AnswerThePublic, 2026.
The baseline worth copying
Most generated robots.txt files are padded with directives Google ignores. A sane WordPress file is short:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlThat is genuinely it for most sites. Block the admin area, allow the AJAX endpoint some front-end features need, and point crawlers at your sitemap. Everything else should be added for a reason you can articulate.
The three tools you actually need
- A tester, before deploying. Paste the rules and a URL; it tells you allowed or blocked, and which line decided. Ten seconds that prevents the classic launch-day disaster.
- An editor you can reach. On WordPress the file is virtual, so a dashboard editor is the practical route — especially on managed hosting with no file access.
- An audit, afterwards. The one nobody runs. Search Console reports pages it found blocked; a crawl catches blocked CSS, JavaScript, and images that break rendering even when your pages are allowed.
What testers cannot tell you
- Whether blocking was the right call. A tester confirms the rule works, not that hiding that section was wise.
- That Disallow does not deindex. Blocked URLs can still appear in results as bare links — and because Google cannot crawl them, it cannot see a
noindexeither. - Which bots you are actually talking to. Rules addressed to crawlers that do not exist are common and do nothing; real-world audits routinely find a dozen of them.
The AI-crawler layer
Robots.txt now carries decisions it never used to: GPTBot, Google-Extended, PerplexityBot and peers all read it. Retrieval bots put you in AI answers; training-only bots learn from you without sending anyone. That makes it a per-bot judgement rather than one blanket rule — and a place where a careless copy-pasted block can quietly remove you from assistant answers your buyers read. The strategy is in the GEO guide.
The full explanation of directives, precedence, and the mistakes that deindex sites is in what is robots.txt.
Frequently asked questions
- How do I test a robots.txt file?
- Test before you ship: paste your proposed rules into a robots.txt tester along with a URL, and it tells you whether that URL would be allowed or blocked, and which rule decided. After deploying, Search Console reports blocked URLs it encounters, and a crawl shows which of your own pages the rules are actually stopping.
- How do I edit robots.txt in WordPress?
- WordPress generates a virtual robots.txt, so there is often no physical file to edit over FTP. SEO plugins expose an editor in the dashboard that writes the virtual file safely, which is also the only practical route on managed hosting where file access is restricted.
- Is there a robots.txt generator I should use?
- Generators are fine for producing a first draft, but the useful WordPress baseline is four lines and included below — most generators pad it with directives Google ignores. What matters more than generation is testing before deployment and auditing afterwards, because the damage from a wrong line is total and silent.
- How do I check whether robots.txt is blocking my pages?
- Search Console flags "Blocked by robots.txt" in the indexing report, which is the authoritative answer for pages Google tried to reach. A crawl-based audit catches the wider problem — including blocked CSS, JavaScript, and images that stop Google rendering pages correctly even when the pages themselves are allowed.
- How do I block AI crawlers in robots.txt?
- Name each bot explicitly — GPTBot, Google-Extended, PerplexityBot, ClaudeBot and others each read their own user-agent — and decide per bot rather than applying a blanket rule. Blocking training crawlers keeps content out of model training; blocking retrieval crawlers can remove you from AI answers your buyers read. Never let a broad rule catch Googlebot.
