Robots.txt and XML Sitemaps: A Beginner's Guide

---

By Justin Zwick for Free Micro Apps · Published September 13, 2026

A robots.txt file and an XML sitemap work as a pair: one tells crawlers where not to go, the other lists the pages you'd like them to find.

Two small files help search engines find their way around your website: robots.txt and the XML sitemap. They're often mentioned together and sometimes confused. This guide explains what each one does, how to create them with free tools on Free Micro Apps, and the mistakes that cause real problems.

XML sitemap and robots.txt rules worth knowing

Both files have documented limits. Google's sitemap documentation says a single sitemap can hold up to 50,000 URLs or 50 MB uncompressed, and that submitting one is only a hint, not a guarantee of crawling. For robots.txt, the official standard, RFC 9309, says the file must be UTF-8 encoded, that crawlers should parse at least the first 500 KiB, and that robots.txt is not a form of access control. Anything truly private needs a password or a server-level block.

What robots.txt does

robots.txt is a plain text file at the root of your site, for example https://example.com/robots.txt. It gives instructions to crawlers (the automated programs search engines use to read the web) about which parts of the site they may visit.

A simple robots.txt looks like this:

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

This says: all crawlers (*) should stay out of /admin/ and /cart/, and the sitemap is at the address given.

What robots.txt is for

• Keeping crawlers out of areas with no search value, like admin panels, carts, internal search results or endless filter combinations
• Reducing wasted crawling on large sites
• Pointing crawlers to your sitemap

What robots.txt is not for

• Hiding private information. The file is public, and anyone can read it. Listing a secret folder in robots.txt actually advertises it. Protect private areas with passwords instead.
• Removing pages from search results. Blocking a page in robots.txt stops crawlers from reading it, but the URL can still appear in results if other sites link to it. To keep a page out of results, let it be crawled and use a noindex meta tag instead.

What an XML sitemap does

An XML sitemap is a file that lists the pages you want search engines to know about, often with the date each page was last updated. It's like a table of contents for crawlers.

A sitemap helps most when:

• Your site is new and has few links from other sites
• Your site is large
• Some pages aren't well linked from other pages
• You add or update content often

A sitemap doesn't guarantee that every listed page gets indexed. It's a strong hint, not a command.

Creating a robots.txt file

The Robots.txt Generator builds a file for you. Choose which crawlers to address, add any folders you want to block, enter your sitemap address and copy the result.

Then:

1. Save it as a plain text file named exactly robots.txt.
2. Upload it to the root of your site, so it's available at yourdomain.com/robots.txt.
3. Open that address in your browser to confirm it loads.

Many website platforms (WordPress, Shopify and others) generate a robots.txt automatically. In that case, edit it through the platform's settings or an SEO plugin rather than uploading a new one.

Creating an XML sitemap

The XML Sitemap Generator crawls your site and produces a sitemap file you can download. Upload it to your site's root, typically as sitemap.xml.

Again, many platforms create sitemaps automatically. WordPress, for example, has a built-in sitemap and most SEO plugins add their own. If yours already exists, you don't need a second one. Just make sure it's listed in robots.txt.

Once your sitemap is live, submit it in Google Search Console and Bing Webmaster Tools. This lets you see how many listed pages were indexed and spot errors.

Seeing your site the way a crawler does

The Search Engine Spider Simulator shows what a crawler sees on a page: the text, the links and the meta information, without the visual design. It's a useful way to check that important content isn't hidden in places crawlers struggle with.

To check whether a page has been indexed at all, try the Google Index Checker.

Common mistakes

Blocking the whole site by accident

This line blocks everything:

User-agent: *
Disallow: /

It's sometimes left over from a development site and copied to the live site during launch. It's one of the most damaging and most common mistakes. After any launch or migration, check your robots.txt.

Blocking CSS and JavaScript

Search engines render pages much like a browser. If your robots.txt blocks the folders containing your CSS and JavaScript files, they may not see the page correctly. Don't block these resources.

Listing pages you don't want indexed in the sitemap

Your sitemap should only include pages you'd like to see in search results: no redirects, no error pages, no duplicate versions, no pages marked noindex.

Mixing http and https or www and non-www

Sitemap URLs should match the version of your site that's live. If your site uses https://www.example.com, every URL in the sitemap should too. The WWW Redirect Checker confirms that the other versions redirect properly.

Forgetting to update

If your sitemap is a static file, it won't update itself when you add pages. Regenerate it after significant changes, or use a platform or plugin that updates it automatically.

Quick checklist

• robots.txt exists at the root and loads in a browser
• It doesn't block the whole site, CSS or JavaScript
• Private areas are protected by passwords, not robots.txt
• XML sitemap lists only pages you want indexed
• Sitemap URLs match your live domain version
• Sitemap is listed in robots.txt and submitted to search consoles

These files take a few minutes to set up and very little effort to maintain, and they remove a common source of crawling problems. If you have questions, contact us.

Key takeaways

• robots.txt controls crawling, not indexing or access
• Never use robots.txt to hide private content
• An XML sitemap lists the canonical pages you want found
• Reference your sitemap in robots.txt and submit it in Search Console
• Split sitemaps that go over 50,000 URLs

Frequently asked questions

Do I need a robots.txt file?

Not strictly, but it's useful. Without one, crawlers assume they can crawl everything.

Does blocking a page in robots.txt remove it from Google?

Not necessarily. A blocked URL can still be indexed if other pages link to it. Use a noindex tag on a crawlable page instead.

Where should my sitemap live?

Usually at the root of your site, such as /sitemap.xml, and referenced in robots.txt with a Sitemap line.

Should every page be in my sitemap?

Only the canonical pages you want indexed. Leave out duplicates, redirects and pages marked noindex.

Can I generate these files for free?

Yes. Free Micro Apps has free robots.txt and XML sitemap generator tools.