Robots.txt Generator: Create Crawl Rules

Robots.txt Generator

Default - all robots are:	Allowed Blocked
Crawl delay:	Default - no delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave empty if you don't have one)
Search robots:	Google	Same as default Allowed Blocked
	Google Image	Same as default Allowed Blocked
	Google Mobile	Same as default Allowed Blocked
	MSN Search	Same as default Allowed Blocked
	Yahoo	Same as default Allowed Blocked
	Yahoo MM	Same as default Allowed Blocked
	Yahoo Blogs	Same as default Allowed Blocked
	Ask/Teoma	Same as default Allowed Blocked
	GigaBlast	Same as default Allowed Blocked
	DMOZ Checker	Same as default Allowed Blocked
	Nutch	Same as default Allowed Blocked
	Alexa/Wayback	Same as default Allowed Blocked
	Baidu	Same as default Allowed Blocked
	Naver	Same as default Allowed Blocked
	MSN PicSearch	Same as default Allowed Blocked
Blocked directories:	Paths are relative to the root and must end with a slash "/"

Next, create a robots.txt file in your site's root folder and paste the text above into it.

About Robots.txt Generator

A robots.txt file sits at the root of your site and tells well-behaved crawlers which areas they may visit. A small typo can block an entire site, so a generator helps you get the syntax right.

How to use it

1. Choose whether all robots are allowed or refused by default.
2. Optionally set a crawl delay and add your sitemap address.
3. Adjust individual search bots and list any folders to keep out.
4. Generate, copy the text and save it as robots.txt in your root folder.

Important to know

Robots.txt is a request, not a lock. Private content still needs a password. Blocking a page from crawling also doesn't guarantee it stays out of search results; use a noindex tag for that. After uploading, open yoursite.com/robots.txt in a browser to confirm it's live.