ProRank SEO

Robots & Indexing Troubleshooting

Common issues and solutions for robots.txt, indexing, and content protection

Common Issues Quick Reference

Robots.txt Issues

Physical File Conflict

Problem:

Warning message: "A physical robots.txt file exists in your site root"

Solution:

  1. Access your site via FTP or file manager
  2. Navigate to your WordPress root directory (where wp-config.php is located)
  3. Delete or rename the robots.txt file
  4. Clear any caching plugins
  5. Verify at yoursite.com/robots.txt
# Via SSH/Terminal
cd /path/to/wordpress/root
mv robots.txt robots.txt.backup
# Or remove it
rm robots.txt

Rules Not Taking Effect

Common Causes:

  • Caching at server or CDN level
  • Virtual editor not enabled
  • Syntax errors in rules
  • Conflicting plugins

Solutions:

  1. Clear all caches (WordPress, CDN, browser)
  2. Verify "Enable Virtual Robots.txt Editor" is ON
  3. Check rules syntax:
# Correct syntax
User-agent: *
Disallow: /private/

# Incorrect syntax
User-agent *  # Missing colon
Disallow /private/  # Missing colon
  1. Deactivate other SEO plugins temporarily
  2. Check .htaccess for conflicting rules

Sitemap Not Appearing in Robots.txt

Checklist:

  1. Ensure XML Sitemaps are enabled in ProRank SEO → Technical SEO → Sitemaps
  2. Verify sitemap exists at yoursite.com/sitemap_index.xml
  3. Check "Auto-add sitemap to robots.txt" is enabled
  4. Save robots.txt settings again to regenerate

Manual Addition:

If auto-add fails, manually add to your robots.txt rules:

# Add at the bottom of your rules
Sitemap: https://yoursite.com/sitemap_index.xml

AI Bots Still Crawling Despite Blocking

Verification Steps:

  1. Check robots.txt shows AI bot rules:
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /
  1. Verify meta tags are present in page source:
<meta name="robots" content="noai, noimageai" />
  1. Check server logs for bot user agents
  2. Note: Some bots may not respect robots.txt immediately

Additional Measures:

  • Enable Content Safeguard for noai directives
  • Consider IP blocking at server level for persistent bots
  • Use .htaccess rules as additional layer

Indexing API Issues

IndexNow Not Working

Common Issues:

  • Missing API Key:Generate a new key in settings and save
  • Key file not accessible:Verify key file exists at yoursite.com/[your-key].txt
  • Invalid URL format:Ensure URLs use https:// and are publicly accessible
  • Rate limiting:IndexNow has daily limits - check submission log

Debugging Steps:

# Test IndexNow manually
curl -X POST &quot;https://api.indexnow.org/indexnow&quot; \
  -H &quot;Content-Type: application/json&quot; \
  -d &apos;{
    &quot;host&quot;: &quot;yoursite.com&quot;,
    &quot;key&quot;: &quot;your-api-key&quot;,
    &quot;urlList&quot;: [&quot;https://yoursite.com/test-page&quot;]
  }&apos;

Response Codes:

  • 200 - Success
  • 202 - Accepted, processing
  • 400 - Invalid format
  • 403 - Invalid key
  • 429 - Rate limited

Google Indexing API Errors

Service Account Issues:

  1. Verify JSON key file is valid:
{
  &quot;type&quot;: &quot;service_account&quot;,
  &quot;project_id&quot;: &quot;your-project&quot;,
  &quot;private_key_id&quot;: &quot;...&quot;,
  &quot;private_key&quot;: &quot;-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n&quot;,
  &quot;client_email&quot;: &quot;indexing@your-project.iam.gserviceaccount.com&quot;
}
  1. Add service account email to Search Console as owner
  2. Enable Indexing API in Google Cloud Console
  3. Check API quotas (200 URLs/day limit)

Common Error Messages:

Permission denied→ Service account not added to Search Console
Invalid grant→ JSON key file corrupted or expired
Quota exceeded→ Daily limit reached (200 URLs)
API not enabled→ Enable API in Google Cloud Console

Meta Tags & Noindex Issues

Pages Still Appearing in Search After Noindex

Understanding the Delay:

Search engines need to recrawl pages to discover noindex tags. This can take days or weeks.

Speed Up Deindexing:

  1. Use Google Search Console URL Removal tool for urgent cases
  2. Submit URL to IndexNow for faster recrawling
  3. Update sitemap to trigger recrawl
  4. Ensure page returns 200 status (not 404) with noindex tag

Verification:

# Check if noindex is present
curl -I https://yoursite.com/page | grep -i &quot;x-robots-tag&quot;
# Or check HTML source for:
# <meta name="robots" content="noindex" />

Meta Robots Tags Not Appearing

Troubleshooting Checklist:

  1. Theme Compatibility:Ensure theme has wp_head() function in header.php
  2. Plugin Conflicts:Disable other SEO plugins that might override tags
  3. Caching Issues:Clear all caches and check in incognito mode
  4. Settings Not Saved:Re-save settings in ProRank SEO
  5. Page Builders:Some page builders may interfere - check their SEO settings

Debug Mode:

Add to wp-config.php to enable debug output:

define(&apos;PRORANK_DEBUG&apos;, true);
define(&apos;WP_DEBUG&apos;, true);
define(&apos;WP_DEBUG_LOG&apos;, true);

Content Safeguard Issues

AI Protection Not Working

Verification Steps:

  1. Check page source for noai/noimageai tags:
<!-- Should see this in <head> -->
<meta name="robots" content="index, follow, noai, noimageai" />
  1. For Pro+ users, check HTTP headers:
curl -I https://yoursite.com/page
# Should include:
# X-Robots-Tag: noai, noimageai

If Tags Missing:

  • Verify Content Safeguard is enabled
  • Check license tier (Core+ or Pro+ required)
  • Clear all caches
  • Check for theme/plugin conflicts

Performance Issues

Slow Loading or Timeouts

Large Robots.txt File:

If blocking many AI bots, robots.txt can become large. Optimize by:

  • Using wildcard patterns where possible
  • Combining similar bot rules
  • Removing outdated bot entries

API Rate Limiting:

Prevent API exhaustion:

  • Batch IndexNow submissions (max 10,000 URLs per request)
  • Implement submission queue for Google API (200/day limit)
  • Use cron jobs for scheduled submissions

Diagnostic Tools

Useful Testing Tools

ProRank SEO Built-in

  • • Robots.txt preview
  • • IndexNow submission log
  • • Meta tag inspector
  • • Conflict detector

External Tools

  • • Google Search Console robots.txt tester
  • • Bing Webmaster Tools
  • • Technical SEO crawlers (Screaming Frog)
  • • Browser DevTools Network tab

Quick Diagnostic Commands:

# Check robots.txt
curl https://yoursite.com/robots.txt

# Check meta tags
curl -s https://yoursite.com | grep -i &quot;meta.*robots&quot;

# Check HTTP headers
curl -I https://yoursite.com | grep -i &quot;x-robots&quot;

# Test IndexNow key file
curl https://yoursite.com/[your-key].txt

Emergency Fixes

Critical Issues Quick Fixes

Site Blocked from Search Engines:

  1. Immediately disable virtual robots.txt editor
  2. Remove any physical robots.txt file
  3. Clear all caches
  4. Submit sitemap to Search Console
  5. Use URL Inspection tool to request indexing

Complete Feature Reset:

If all else fails, reset the module:

-- Backup first, then run in database:
DELETE FROM wp_options WHERE option_name LIKE &apos;prorank_robots%&apos;;
DELETE FROM wp_options WHERE option_name LIKE &apos;prorank_indexing%&apos;;
-- Then reactivate and reconfigure in admin

Always backup your database before running SQL commands. After reset, you'll need to reconfigure all Robots & Indexing settings.

Getting Help

Before Contacting Support

  • ✓ Check WordPress and plugin versions
  • ✓ Test with default theme
  • ✓ Disable other SEO plugins
  • ✓ Clear all caches
  • ✓ Check error logs
  • ✓ Document steps to reproduce

Information to Provide

  • • ProRank SEO version and license
  • • WordPress version
  • • Active theme and plugins
  • • Server environment (PHP version)
  • • Error messages or logs
  • • Screenshots of settings

Related Documentation

Overview

Main Robots & Indexing guide

Robots.txt

Virtual editor configuration

Indexing APIs

IndexNow and Google API setup