Configuration
Guidance
What this file does
robots.txt files commonly declare crawler access rules using directives such as `User-agent`, `Allow`, `Disallow`, and `Sitemap`, helping search engines understand which paths should or should not be crawled.
Typical use cases
- Block admin or private sections from crawler access.
- Expose your sitemap URL for discovery.
- Control crawler behavior for specific directories and file paths.
Important note
robots.txt is a crawler instruction file, not a security mechanism, so sensitive content should still be protected by proper authentication and server controls.
Generated robots.txt
About the Robots.txt Generator
robots.txt is the first file every well-behaved crawler reads on your site — a plain text note at the root (example.com/robots.txt) declaring which paths crawlers may or may not fetch. It is how you keep admin panels, cart pages, and internal search results out of crawl budgets, and how every major search engine is told where your sitemap lives.
This generator builds correct robots.txt from simple choices: allow-all for public sites, block specific paths, or target individual user-agents (Googlebot, Bingbot, GPTBot-style AI crawlers) with different rules. Crawl-delay, sitemap declaration, and common-sense presets are built in, and the output is plain text you drop into your site root unchanged.
The tool’s most valuable guardrails are the warnings: robots.txt cannot hide a page (use noindex for that), and blocking CSS/JS in the file famously breaks Google’s rendering — both traps are called out before you deploy.
How to use the Robots.txt Generator
- 1 Pick the overall posture: allow all, or disallow selected paths.
- 2 Add Disallow paths for directories crawlers should skip (/admin, /cart, /search).
- 3 Optionally target specific crawlers with separate rule groups or set a crawl-delay.
- 4 Add your sitemap URL, then download and place the file at your domain root.
Key features
Preset postures
Public site, staging lockdown, or custom path rules in one click.
Per-bot rules
Separate groups for Googlebot, Bingbot, or AI crawlers as needed.
Sitemap declaration
Sitemap: line emitted so discovery needs no Search Console step.
Trap warnings
Guidance against blocking CSS/JS and misusing Disallow as noindex.
Who uses the Robots.txt Generator?
- New sites publishing a correct robots.txt before launch day.
- E-commerce excluding cart, checkout, and filter URLs from crawling.
- Staging environments requesting full exclusion until go-live.
- Publishers managing AI-bot crawling separately from search engines.
- Audits rebuilding a file after a "why is Google crawling that?" surprise.
Tips and best practices
- Disallow only crawls, not indexing: for a page to vanish from Google, use a noindex meta tag — a Disallowed page can still appear as a bare URL.
- Never block CSS and JavaScript files; Google renders pages, and blocking assets degrades how it evaluates them.
- Test in Search Console’s robots report and URL Inspection before trusting the file in production.
- Paths are case-sensitive and prefix-matched: /admin blocks /admin and /admin-login both — plan for surprises.
- Keep one robots.txt at the root for the whole domain; subdomains need their own file.
Frequently asked questions
A plain-text file at your site root (example.com/robots.txt) that tells crawlers which paths they may fetch, plus where your sitemap lives.
No — it prevents crawling, not indexing. Blocked pages can still appear as URLs. To de-index, use a noindex meta tag on a crawlable page.
A policy choice, not an SEO necessity. The generator supports separate user-agent groups so search engines and AI bots can follow different rules.
It requests a pause between requests. Google ignores it (use Search Console settings); Bing and some others honour it. Use only for genuinely strained servers.
It is public — Disallow paths advertise hidden directories to attackers. Exclude sensitive areas by auth, and let robots.txt manage crawl efficiency.