AOGSIAOGSI

Free Robots.txt Generator

Choose crawler rules, then copy or download your file.

Interactive Robots.txt Generator Interface

Configure Robots Directives

AI Scraper Access
robots.txt
# Robots.txt generated for https://example.com/sitemap.xml

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

What you can do

Paths to exclude

Enter one path per line to add Disallow rules under the default user-agent.

Named crawler rules

Use the GPTBot and ClaudeBot switches to add separate Disallow rules for those user-agents.

Sitemap reference

Add a sitemap URL, inspect the generated text, then copy it or download robots.txt.

Before you use the result

  • Review every rule before replacing your existing file. Check that pages and assets you want crawled are still accessible.
  • Do not use a public crawler-rules file to protect private data. Keep private pages behind proper access controls.

Questions

Other tools

View all tools

Website & API Health Checker

Send a request to a website or API and inspect its response.

AI Meta Tag & OpenGraph Generator

Turn a page topic into draft metadata you can review.

AI Search & GEO Audit

Check a URL for structured data and AI search visibility.

Have a project in mind?

Call AOGSIWhatsApp