Generate a valid robots.txt file in seconds. Set rules for search engine crawlers, add your sitemap, and download a ready-to-upload file — free, no sign-up.
Upload this file to the root of your website (e.g. example.com/robots.txt).
How to use the Robots.txt Generator
- Choose a default rule: allow all crawlers, block all crawlers, or define custom rules per user-agent.
- Add your sitemap URL and an optional crawl-delay to be polite to your server.
- Click Download robots.txt and upload the file to the root folder of your website.
Frequently asked questions
What is a robots.txt file?
It is a small text file that tells search engine crawlers which parts of your site they may or may not visit. It lives at your domain root, for example example.com/robots.txt.
Does robots.txt keep pages private?
Not really — it is a request, not a lock. Well-behaved crawlers obey it, but anyone can still open the URL directly. Never use robots.txt to hide sensitive content; use proper authentication instead.
What is crawl-delay?
It asks crawlers to wait a number of seconds between requests, which protects small servers from being overwhelmed. Most sites can leave it empty.