All free tools
Robots.txt generator
Build a valid robots.txt in seconds — choose a default policy, add disallow/allow paths, set a crawl-delay, and link your sitemap.
Use
* for all crawlers, or a specific bot like Googlebot.Note: Google ignores crawl-delay; Bing & others honour it.
robots.txt
Heads up: robots.txt controls crawling, not indexing. A blocked page can still appear in search if other sites link to it. To keep a page out of results, use a
noindex meta tag instead — and don’t block it in robots.txt (or Google can’t see the noindex).
Is your live robots.txt actually correct?
atlookup validates your robots.txt and sitemap during a crawl — catching blocked pages, conflicts, and missing references. Free to start.
What is robots.txt?
The robots.txt file lives at the root of your domain (example.com/robots.txt) and tells search-engine crawlers which parts of your site they may or may not request. It’s the first file most crawlers check before exploring your site.
The directives this tool writes
- User-agent — which crawler the rules apply to (
*= all). - Disallow — paths the crawler should not request.
- Allow — exceptions inside a disallowed path.
- Crawl-delay — seconds between requests (honoured by Bing and others; ignored by Google).
- Sitemap — the absolute URL of your XML sitemap.
How to use it
Set your rules on the left; the file builds on the right. Click Copy or Download, then upload it to your site root so it’s reachable at /robots.txt. Everything runs in your browser.
Related guides
Dig deeper in our docs, glossary, and fix library.
Frequently asked questions
Quick answers about this tool and the topic.
Where does robots.txt go?
At the root of your domain, reachable at https://yoursite.com/robots.txt. It must be a plain-text file.
Does robots.txt stop a page from being indexed?
No — it controls crawling, not indexing. A blocked page can still appear in search if other sites link to it. To keep a page out of results, use a noindex meta tag and do not block it in robots.txt.
Does Google honor crawl-delay?
No. Google ignores the crawl-delay directive, but Bing and some other crawlers respect it.