How to use Robots.txt Generator
- Choose a starting preset.
- Add the paths to allow or disallow, and your sitemap URL.
- Copy the file and upload it as /robots.txt.
About this tool
robots.txt tells crawlers which parts of your site they may request. It is a public request, not a security measure, and a blocked page can still appear in results if other sites link to it. Use a noindex tag or authentication for pages that must stay private.
The optional AI crawler block adds rules for GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and others. Well-behaved crawlers follow these rules; not all bots do.
Frequently Asked Questions
- Where does robots.txt go?
- At the root of your domain, for example https://example.com/robots.txt.
- Does Disallow remove a page from Google?
- No. It only stops crawling. To remove a page use noindex or the removal tool.
- Does Google use crawl-delay?
- No. Googlebot ignores it, although some other crawlers respect it.
Related tools
- Sitemap XML GeneratorPaste your URLs and download a ready-to-upload sitemap.xml.
- Meta Tag GeneratorFill in your page details and copy ready-to-use meta tags.
- Disavow File GeneratorTurn a list of spammy domains and URLs into a correctly formatted disavow file.
- SERP Snippet PreviewSee your page as it may appear in Google results.
- Keyword Density CheckerSee which words and phrases dominate your text.
- 301 Redirect GeneratorCreate redirect rules for moved pages.