Robots.txt

The Robots.txt report retrieves the root robots.txt file for your project website and categorizes recognized user agents as Allowed, Partial, or Blocked. It evaluates a declared crawling policy, not actual request logs or an enforced security boundary.

Analyze the fileCopy Anchor Link

1. From GEO Optimization, choose Analyze robots.txt.
2. Confirm whether the file was found and review the allowed, partial, and blocked counts.
3. Filter AI Bots by status to focus on relevant entries. Each card identifies its matched agent and any displayed blocked paths.
4. Review Reference Bots separately, then inspect Recommendations.
5. If present, inspect robots.txt Content. Copy copies that fetched text for review; it does not publish a replacement.

Interpret the categoriesCopy Anchor Link

The analyzer chooses the most specific matching user-agent token, merging groups of equal specificity. No matching group or no nonempty disallow rule results in Allowed. An unrestricted root disallow can result in Blocked; other nonempty disallow cases are categorized as Partial.

This is a summary classifier rather than an exhaustive URL-by-URL robots rules simulator. An Allowed card does not guarantee access to every URL, and a Partial card does not list every affected page. Inspect the source rules and test important URLs independently before changing policy.

If no file is found, the report shows default-allow categories and missing-file guidance. A network error or inability to retrieve a file is not proof that the site's access policy permits crawling. Also consider HTTP responses, authentication, firewalls, and meta or header directives.

Score and scopeCopy Anchor Link

Only the AI bot catalog contributes to the access score. Each Allowed entry contributes 100, Partial contributes 50, and Blocked contributes 0; the average is rounded. Reference Bots do not contribute to that score.

A restrictive policy can be intentional. Do not allow unwanted crawlers just to increase the score. robots.txt is a public instruction for cooperating crawlers, not a way to protect private content.

Make and verify a policy changeCopy Anchor Link

1. Decide which crawlers and paths should be permitted for your organization.
2. Edit the actual root robots.txt through your website's deployment or hosting process.
3. Confirm that the published file contains the intended rules and sitemap references.
4. Run Analyze robots.txt again to store a new result, then compare the bot categories and source text.

Refresh rechecks the current project website in the open report. Export downloads its displayed JSON. Share copies the current report URL without granting access. See saved-report behavior.

Use AI Agents for observed user-agent activity and Discoverability for additional indexing hints.