Robots.txt and sitemap.xml generator

Build a robots.txt with the rules your site needs, and a sitemap.xml from a list of pages. Both files are generated as you type, ready to download and upload to your site.

  • Free
  • No sign-up
  • Runs in your browser
robots-sitemap-generator

100% private — the files are generated in your browser and nothing is sent to any server.

How it works

Choose robots.txt or sitemap.xml

Two tabs, two small files most sites need.

Fill in the rules or the pages

Presets cover the common cases.

Download and upload

Both files go in the root of your site.

Two small files, one job: guiding crawlers

Search engines and other automated crawlers read two files before anything else: robots.txt, which tells them what they may and may not fetch, and sitemap.xml, which lists the pages worth visiting. Neither is required, but a small site without them leaves crawlers to guess, and a badly written robots.txt can accidentally hide the whole site from search results. Both files are plain text and easy to get wrong by hand; this tool builds them from a form.

robots.txt

This file sits at the root of your domain, such as example.com/robots.txt, and lists rules by user agent: which crawler the rule applies to, which paths are disallowed and which are explicitly allowed. It is a request, not a lock: well-behaved crawlers follow it, but it does not stop a page from being accessed directly or protect private data. Use it to keep crawlers away from admin areas, search result pages, shopping carts and duplicate content, and to point to your sitemap. Some site owners now also want to block the crawlers that collect training data for AI models, such as GPTBot or Google-Extended, which this tool can add with one click, separately from the rules for search engines.

sitemap.xml

A sitemap lists your pages in a simple XML format so that a crawler can find them without following every link, which helps with new or poorly linked pages. Paste your page paths, or full addresses, and the site address once, and the tool builds the file, adding an optional last modified date, a change frequency and a priority. These last three are hints, not guarantees; search engines increasingly ignore change frequency and priority and rely on their own crawling patterns, but they cost nothing to include and some tools still read them.

Limits

A single sitemap file should not list more than 50,000 addresses; larger sites split their pages into several sitemap files listed in a sitemap index, which is beyond what this simple generator produces. Robots.txt rules are advisory for compliant crawlers only. Everything here is built in your browser; nothing is uploaded until you place the files on your own server.

Frequently asked questions

Where do robots.txt and sitemap.xml go?
Both belong at the root of your domain: example.com/robots.txt and example.com/sitemap.xml.
Does robots.txt stop a page from being indexed?
Not reliably. It asks crawlers not to fetch a page, but the page can still appear in results if other sites link to it. Use a noindex meta tag on the page itself to keep it out of search results.
Should I block AI crawlers?
That is your choice. Blocking them stops your content from being used to train some AI models, but it is a separate decision from blocking search engines, and this tool lets you do one without the other.
Do I need a sitemap for a small site?
It helps but is not essential for a handful of well-linked pages. It matters more for large sites or pages that are hard to discover by following links.
What do priority and changefreq actually do?
They are hints. Major search engines have said they give them little weight and rely on their own signals, but they do not hurt to include.
Is my sitemap data uploaded anywhere?
No. Both files are built in your browser.

Related tools

More utilities that also run without leaving your browser.

Screen size checkerSee your screen resolution, window size and pixel ratio, live. SVG optimizerClean up SVG code and shrink its file size. SVG to PNG converterConvert an SVG to PNG, at one size or several sizes at once. Clean URLRemove tracking parameters from links before you share them. Broken link checkerPaste a list of URLs and see which ones fail to respond. Email signature generatorCreate a professional HTML email signature and copy it into your email program.