Robots.txt Generator
Build robots.txt files with crawl rules and sitemap references
About Robots.txt Generator
Assembles a robots.txt file from one or more User-agent blocks, each with its own Disallow and Allow paths and an optional Crawl-delay, plus an optional Sitemap line at the end. Empty paths are skipped and a blank user agent becomes *. Paths are written exactly as typed: wildcards such as * and $ pass through unchanged and nothing is validated, so double-check the rules before uploading the file.
Category: SEO.
Frequently asked questions
What is a robots.txt file?
▾
A plain text file at the root of your website that instructs web crawlers which pages or sections they can or cannot access. It is a convention — not a security mechanism — and is publicly visible.
Does robots.txt prevent indexing?
▾
Disallow prevents crawlers from fetching a URL, but it does not prevent indexing if other pages link to it. To keep a page out of search results, use a noindex meta tag or X-Robots-Tag HTTP header and don't block that page in robots.txt, or crawlers will never see the noindex.
How do I block all crawlers from a specific folder?
▾
Add a User-agent: * block with Disallow: /folder/. Rules are prefix matches, so the trailing slash limits the rule to that directory; without it, /folder also blocks /folder.html, /folder-old/ and any other path that starts with /folder.
Does Google respect Crawl-delay?
▾
No. Googlebot ignores the Crawl-delay line, so the value only affects crawlers that support it, such as Bingbot. Google adjusts its crawl rate automatically based on how your server responds.