Robots.txt Generator
Build a correct robots.txt file to guide search engine crawlers — block private sections, list your sitemap and, if you want, opt out of AI training crawlers.
Result
Choose your rules to generate robots.txt.
How to use this tool
- Choose whether search engines may crawl the site.
- List paths to block, one per line, such as
/admin/, and any exceptions to allow. - Add your sitemap URL, then download robots.txt and upload it to the root of your site.
How it works
Rules are grouped by User-agent. Disallow blocks paths that start with the given text; Allow makes exceptions inside a blocked path; * matches anything and $ marks the end of a URL. An empty Disallow: allows everything. The AI option adds a separate group blocking GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider and meta-externalagent.
Example
Hiding an admin area
Blocking /admin/ with an exception for /admin/public/ gives:
User-agent: *
Disallow: /admin/
Allow: /admin/public/Privacy and limitations
- Everything runs in your browser; nothing you enter is uploaded or stored.
- robots.txt is a public request to well-behaved crawlers, not security. Protect private pages with a login.
- Blocking crawling does not remove a page from search results; use a noindex meta tag (with crawling allowed) for that.
Frequently asked questions
Where does robots.txt go?
In the root of your domain, so it is available at https://www.example.com/robots.txt. Each subdomain needs its own.
Does Google follow Crawl-delay?
No. Google ignores it; Bing and some other crawlers respect it.