Back to tools
Robots.txt Generator
Build robots.txt files with crawl rules and sitemap references
About Robots.txt Generator
Robots.txt Generator lets you build robots.txt files with crawl rules and sitemap references.
Category: SEO.
Frequently asked questions
What is a robots.txt file?
▾
A plain text file at the root of your website that instructs web crawlers which pages or sections they can or cannot access. It is a convention — not a security mechanism — and is publicly visible.
Does robots.txt prevent indexing?
▾
Disallow prevents crawlers from fetching a URL, but it does not prevent indexing if other pages link to it. To prevent indexing, use the noindex meta tag or X-Robots-Tag HTTP header.
How do I block all crawlers from a specific folder?
▾
Add: User-agent: * followed by Disallow: /folder/. Use a trailing slash to block a directory; without it, /folder blocks only a file named exactly 'folder'.