Configuration
`*` targets all crawlers.
Optional delay directive supported by some crawlers.
Enter one path per line for directories or URLs you want crawlers to access.
Enter one path per line for directories or URLs you want blocked from crawling.
Optional directive sometimes included in robots.txt generators.
Guidance
What this file does
robots.txt files commonly declare crawler access rules using directives such as `User-agent`, `Allow`, `Disallow`, and `Sitemap`, helping search engines understand which paths should or should not be crawled.
Typical use cases
- Block admin or private sections from crawler access.
- Expose your sitemap URL for discovery.
- Control crawler behavior for specific directories and file paths.
Important note
robots.txt is a crawler instruction file, not a security mechanism, so sensitive content should still be protected by proper authentication and server controls.
Allow Rules
0
Disallow Rules
0
Total Lines
0
User-agent
*
Generated robots.txt
How it works
Robots.txt generators commonly produce plain-text crawler directives such as `User-agent`, `Allow`, `Disallow`, `Crawl-delay`, and `Sitemap`, helping website owners build a deployable robots file without writing syntax manually. These tools are typically used as part of SEO setup alongside sitemaps and other indexing controls.
- `User-agent: *` is commonly used to target all crawlers.
- `Disallow` rules are typically used for admin, private, temporary, or filtered paths.
- `Sitemap` directives help point crawlers to sitemap discovery URLs.