Robots.txt is a plain-text file at the root of your site that tells crawlers which parts they may fetch.It is the first thing every bot requests, one wrong line in it can make an entire site vanish from search — and since AI assistants started reading the web, it has quietly become the file where you decide who may learn from your content. Here is what belongs in it, and what absolutely does not.
“what are robots txt files” — 480 searches/mo (Google, US)Average monthly US search volume. Source: Google autocomplete demand via AnswerThePublic, 2026.
How to read one (60 seconds)
Four directives cover nearly every real file: User-agent names the bot the following rules apply to (* means everyone), Disallow asks it not to fetch a path, Allow carves an exception back out, and Sitemap points crawlers at your XML sitemap. A sensible WordPress file is short:
# https://example.com/robots.txt
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# Training-only AI crawlers — a per-bot decision
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlWhat robots.txt is for — and the trap it hides
Its real job is crawl management: keeping bots out of admin areas, infinite parameter spaces, and duplicate paths so crawl budget goes to pages that matter. The trap is using it to hide pages from Google: Disallow blocks crawling, not indexing.A blocked URL can still show up in results as a bare link — and because Google can’t crawl it, it can’t even see a noindex tag on it. To remove a page from search, do the opposite: let it be crawled, mark it noindex.
The mistakes that deindex sites
Disallow: /left over from staging — the classic launch-day disaster; one character hides the whole site.- Blocking CSS and JavaScript — Google renders pages; blind rendering reads as broken pages.
- Listing private paths as “protection” — the file is public; that is a treasure map, not a lock.
- Inventing directives — crawl-delay and friends are ignored by Google; fantasy bot names do nothing but bloat the file. An audit of real-world files finds plenty of rules addressed to crawlers that do not exist.
- Forgetting the sitemap line — the cheapest discovery hint you can give.
The 2026 job: AI crawler policy
Robots.txt is now where the AI decisions land: GPTBot, Google-Extended, PerplexityBot, ClaudeBot and the rest all check it. The strategic frame — retrieval bots put you in AI answers, training-only bots just learn from you — is covered in the GEO guide; the mechanical rule is: name each bot explicitly, decide each deliberately, and audit the file so a blanket rule never silently blocks the crawlers your traffic depends on.
Robots.txt is one half of crawl control — the other half is what you invite crawlers to fetch: your sitemap and your internal links. Check all three in one pass with the free audit.
Frequently asked questions
- Where do I find my robots.txt file?
- It always lives at the root of the domain: yoursite.com/robots.txt — type that into a browser and you are reading the live file. On WordPress there is often no physical file; WordPress generates a virtual one, and SEO plugins let you edit its contents safely from the dashboard.
- Does robots.txt stop a page from appearing in Google?
- Not reliably — this is the most common misunderstanding. Disallow blocks crawling, not indexing: a blocked URL can still appear in results (as a bare link) if other sites link to it. To keep a page out of the index, allow it to be crawled and mark it noindex — blocking it in robots.txt actually prevents Google from seeing the noindex.
- What should robots.txt block on a WordPress site?
- Very little. Block /wp-admin/ (while allowing admin-ajax.php, which some front-end features need), plus any genuinely private or infinite-space paths like internal search results with parameters. Do not block CSS, JavaScript, or images — Google needs them to render your pages, and blocking them hurts you.
- Should I block AI crawlers like GPTBot in robots.txt?
- Make it a per-bot decision, not a blanket one. Blocking training-only crawlers keeps your content out of model training; blocking retrieval crawlers can remove you from AI answers your buyers actually read. List the specific bots you mean, decide each deliberately, and never block Googlebot while trying to block AI.
- Is robots.txt a security feature?
- No — it is a public request, not a lock. Anyone (including bad bots that ignore it) can read the file, and listing "secret" paths in it is literally publishing a map to them. Protect private content with authentication; use robots.txt only for crawl management.
