LLM Optimization (llms.txt)
Future-proof your website for AI tools like ChatGPT, Claude, and Gemini with the llms.txt standard
Overview
ProRank SEO's LLM Optimization feature helps you create and manage an llms.txt file - a new standard that helps AI assistants understand your website better. Located under Technical SEO → LLM Optimization, this feature ensures AI tools like ChatGPT, Claude, and Google Gemini can accurately represent your business when answering questions about it.
Why LLM Optimization Matters
AI Can't Read Everything
Unlike search engines, AI tools have limited context windows and can only process small portions of your site
Control Your Narrative
Guide AI to your most important, accurate, and up-to-date content instead of letting them guess
Reduce AI Hallucinations
Provide clean, structured content to minimize incorrect or misleading AI responses about your brand
Future-Proof Your SEO
Prepare for the shift from traditional search to AI-powered discovery and recommendations
As more users turn to AI assistants for information, having an llms.txt file ensures your business is accurately represented in AI-generated responses, potentially leading to better visibility and more qualified traffic.
Key Features
🤖 Automatic Page Selection
Intelligently identifies your most important pages including homepage, about, contact, privacy policy, and top content based on your site structure and analytics data.
✏️ Manual Page Selection
Take full control by manually selecting which pages to include, perfect for highlighting specific products, services, or content you want AI to prioritize.
📝 Clean Markdown Conversion
Automatically converts your HTML content to clean, readable markdown that AI tools can easily process, removing navigation, ads, and other clutter.
🔄 Weekly Auto-Updates
Keeps your llms.txt file fresh with automatic weekly regeneration, ensuring AI tools always have access to your latest content.
How to Set Up LLM Optimization
Step 1: Navigate to the Feature
Go to WordPress Admin → ProRank SEO → Technical SEO and click on the LLM Optimization tab.
Step 2: Enable the Feature
Toggle the "Enable llms.txt file feature" switch to activate the functionality. This will start generating your llms.txt files.
Step 3: Choose Selection Mode
Let ProRank SEO intelligently select your most important pages based on best practices and site structure.
Hand-pick specific pages you want AI tools to focus on for complete control over your AI presence.
Step 4: Configure Settings
If using automatic mode, you can configure:
- Include blog posts (recommended for content-heavy sites)
- Include pages (essential for service/product information)
- Include products (for WooCommerce sites)
- Maximum pages to include (5-100, default: 20)
Step 5: Generate Files
Click "Generate Now" to create your llms.txt files. You can preview them before publishing to ensure everything looks correct.
Generated Files
llms.txt (Index File)
Located at yoursite.com/llms.txt
Contains a structured index with links to your key content, site description, and metadata. This is the main file AI tools will look for.
llms-full.txt (Complete Documentation)
Located at yoursite.com/llms-full.txt
Contains the complete content of all selected pages in one file, useful for AI tools that can process larger context windows.
Best Practices
Enable weekly auto-updates or manually regenerate after major content changes.
Always include About, Contact, Privacy Policy, and Terms pages for credibility.
Prioritize timeless content over news or time-sensitive information.
After setup, test by asking ChatGPT or Claude about your business using your llms.txt URL.
Manual Page Selection Guide
When using manual selection mode, you can specify pages for different categories:
Essential Pages
- About Us: Your company story and mission
- Contact: How to reach you
- Terms: Terms of service
- Privacy Policy: Data handling practices
- Shop: Main products/services page (if applicable)
Content Pages
Add your most valuable content such as comprehensive guides, product descriptions, service pages, or cornerstone content that best represents your expertise.
How AI Tools Use llms.txt
Currently, AI tools don't automatically discover llms.txt files. You need to actively provide them:
- Direct Link Method:
Paste your llms.txt URL directly in your prompt: "Learn about my business from example.com/llms.txt"
- Copy Content Method:
Copy the llms.txt content and paste it into your conversation for immediate context
- File Upload Method:
Download and upload the file to AI tools that support file attachments
Troubleshooting
Files Not Accessible
If your llms.txt files return 404 errors:
- Ensure the feature is enabled in settings
- Click "Generate Now" to create the files
- Flush permalinks: Settings → Permalinks → Save Changes
- Check for conflicting plugins or security rules
Content Not Updating
If changes aren't reflected in llms.txt:
- Manually regenerate files after major changes
- Check if weekly auto-update is enabled
- Clear any caching plugins or CDN cache
- Verify the correct pages are selected
Pages Missing in Automatic Mode
If important pages aren't included automatically:
- Switch to manual mode for full control
- Ensure pages are published (not draft)
- Check the maximum pages setting
- Verify page slugs match common patterns (about, contact, etc.)