Robots.txt Generator

Generate a ready-to-use robots.txt file with user-agent, allow, disallow, crawl-delay, and sitemap rules.

Configuration

Crawler Rules + Sitemap
`*` targets all crawlers [web:671].
Optional delay directive supported by some crawlers [web:677].
Enter one path per line for directories or URLs you want crawlers to access [web:671].
Enter one path per line for directories or URLs you want blocked from crawling [web:671][web:677].
Optional directive sometimes included in robots.txt generators [web:677].

Guidance

Quick Rule Summary

What this file does

robots.txt files commonly declare crawler access rules using directives such as `User-agent`, `Allow`, `Disallow`, and `Sitemap`, helping search engines understand which paths should or should not be crawled [web:671][web:677].

Typical use cases

  • Block admin or private sections from crawler access [web:671][web:677].
  • Expose your sitemap URL for discovery [web:671].
  • Control crawler behavior for specific directories and file paths [web:677].

Important note

robots.txt is a crawler instruction file, not a security mechanism, so sensitive content should still be protected by proper authentication and server controls [web:671][web:677].

Allow Rules
0
Disallow Rules
0
Total Lines
0
User-agent
*

Generated robots.txt

0 characters

How it works

Robots.txt generators commonly produce plain-text crawler directives such as `User-agent`, `Allow`, `Disallow`, `Crawl-delay`, and `Sitemap`, helping website owners build a deployable robots file without writing syntax manually [web:671][web:677]. These tools are typically used as part of SEO setup alongside sitemaps and other indexing controls [web:671].

  • `User-agent: *` is commonly used to target all crawlers [web:671].
  • `Disallow` rules are typically used for admin, private, temporary, or filtered paths [web:671][web:677].
  • `Sitemap` directives help point crawlers to sitemap discovery URLs [web:671].