robots.txt Generator

Create a robots.txt file for your website: specify what to block from search engines, select the search engine—and you'll get a ready-to-use file. Everything is generated in the browser; nothing is sent anywhere.

Why Do We Need a robots.txt File?

This is a text file in the root directory of the website that tells search engine bots which pages to avoid. It conserves the crawl budget: the bot doesn’t waste time on service URLs, search pages, and endless filter combinations, but instead focuses on what really needs to be in the index.

What robots.txt Does NOT Do

The most common misconception is to think of it as a means of protection. That's not the case:

• Does not hide the page from people. The file is open: anyone can access it your-website/robots.txt and will see a list of what you're hiding. Don't include paths to admin panels or restricted sections there—you'll just be revealing them to an attacker.
• Does not guarantee that it will no longer be available. If a page contains external links, Google may display it in search results without a description. To remove the page from the index, you need to noindex in the meta description, rather than a ban in robots.txt.
• Does not protect against bots. Bona-fide crawlers follow the rules. Vulnerability scanners and content parsers simply ignore them.

Common Mistakes

Disallow: /

A single character can block the entire site. This usually happens after development—the site was launched, but someone forgot to remove the block. If search traffic suddenly disappears, check this first.

Proprietary CSS and JavaScript

In the past, they were hidden so as “not to drain the budget.” Today, Google renders a page just like a browser: without styles and scripts, it sees a broken layout and may conclude that the site isn’t optimized for mobile.

Blocking Instead of "noindex"

The page is already indexed; if you block it in robots.txt, the crawler will no longer visit it and, therefore, won’t realize it needs to be removed. It remains in the search results. The correct procedure is: first noindex, wait until it is removed from the index, then, if necessary, block it in robots.txt.

There is no link to the site map

Line Sitemap: — the easiest way to let search engines know about all your pages. Our generator adds it automatically when you enter your website's URL.

On the Ban on AI Bots

A separate clause restricts access by GPTBot, ClaudeBot, CCBot, Google-Extended, and other data collectors used for training models.

The decision isn't clear-cut. A ban reduces the use of your texts for training, but some of these bots also ensure that your site appears in AI responses—with a link to the source. By blocking them, you lose this channel. We’ve analyzed how this affects traffic in the article About Google's AI-powered responses.

Where to save the file

Only in the website's root directory: example.com/robots.txt. It won't work in a subfolder. On WordPress, the file is often generated on the fly by a plugin—in that case, the physical file in the root directory will override it, and you'll have to edit that file instead.

After uploading it, verify that it opens at the direct URL, and submit the site for reindexing in Google Search Console.