About Robots.txt Generator
A robots.txt file is the first thing well-behaved search engine crawlers look for when they visit a site. It sits at the root of the domain, for example example.com/robots.txt, and tells bots which areas they may crawl and where to find your sitemap. This generator builds the file from your choices: apply the rules to all crawlers or to a specific one such as Googlebot or GPTBot, list the paths to disallow and allow one per line, add a crawl delay and your sitemap if you like, or block the whole site for those crawlers. Then copy the result or download it as robots.txt.
It suits owners of new websites, bloggers moving to their own domain, developers launching a client site, and anyone whose platform does not create a robots.txt automatically. Any path that does not start with a slash is pointed out. The output follows the standard directives that Google, Bing and other major crawlers understand. Remember that robots.txt guides crawlers; it does not hide pages from people or protect private content.
Why use it
- Produces a valid robots.txt in standard syntax, ready to copy into a plain text file.
- Includes a Sitemap line so search engines can find all your pages faster.
- Allows crawling of the whole site by default, which is the right setting for most public websites.
- Blocks the common /admin/ path as a sensible starting rule you can keep or edit.
- Runs in your browser, with a copy button and a button to download the finished file.
How to use
Choose the crawlers
Pick All crawlers (*) or type the name of a specific crawler such as Googlebot or GPTBot.
List the paths
In Disallow, put each path to block on its own line, such as /admin/ and /cart/, and in Allow anything to permit inside them. Every path starts with a slash.
Add your sitemap
Paste the full address of your XML sitemap, such as https://example.com/sitemap.xml, then press Generate robots.txt.
Download and upload
Press Download robots.txt and upload the file to the root of your domain, so it opens at yourdomain.com/robots.txt.
Examples
A new WordPress blog
Put /wp-admin/ in Disallow and /wp-admin/admin-ajax.php in Allow, add https://myblog.com/sitemap_index.xml, and the file keeps crawlers out of the dashboard while pointing them to the sitemap index.
Online store with a private area
A store lists /cart/ and /checkout/ under Disallow so crawlers skip pages with no search value.
Blocking an AI crawler
Choose a specific crawler, type GPTBot and tick Block the whole site; the file gets User-agent: GPTBot with Disallow: /.
Checking the result
After uploading, the owner opens example.com/robots.txt in a browser to confirm the lines appear exactly as generated.
| Line | Meaning |
|---|---|
| User-agent: * | The rules below apply to all crawlers |
| Allow: / | Crawlers may access every path on the site |
| Disallow: /admin/ | Crawlers should not request anything under /admin/ |
| Sitemap: https://… | Full address of your XML sitemap |
Tips
- The file must be named robots.txt in lowercase and placed at the domain root; crawlers do not look for it in subfolders.
- Disallow stops crawling but does not guarantee a page stays out of search results. Use a noindex meta tag for pages that must not be indexed.
- Never list secret URLs in robots.txt; the file is public, and anyone can read it.
- Do not block CSS or JavaScript folders that your pages need, or search engines may not render your site correctly.
- Each subdomain needs its own robots.txt, such as shop.example.com/robots.txt.
The file is generated in your browser; the URL you enter is not sent to any server.
Frequently asked questions
What is a robots.txt file?
It is a plain text file at the root of a website that tells search engine crawlers which parts of the site they may visit. It follows the Robots Exclusion Protocol, which major search engines respect.
Where do I put robots.txt?
In the top-level folder of your domain, so it is reachable at https://yourdomain.com/robots.txt. A file inside a subfolder is ignored.
What rules does this generator create?
A User-agent line for the crawlers you chose, a Disallow line for each blocked path (or Disallow: / when you block the whole site), an Allow line for each allowed path, plus Crawl-delay and Sitemap when you fill them in. The defaults block /admin/ and /cart/.
Do I have to enter a sitemap URL?
The field is optional, but you should enter your own. If you leave it empty, the result uses Adwaty's sitemap address as a placeholder, which you would need to replace.
My site has no /admin/ folder. Is that a problem?
No. A Disallow rule for a path that does not exist has no effect. You can also delete the line for a tidier file.
How do I block another folder?
Add a new line under the existing rules, for example Disallow: /private/. Use one Disallow line per path.
Does robots.txt remove pages from Google?
Not reliably. It stops crawling, but a blocked URL can still appear in results if other sites link to it. To keep a page out of the index, allow crawling and use a noindex tag.
Is robots.txt a security measure?
No. It is a public request that polite crawlers follow. Malicious bots can ignore it, so protect private areas with passwords or server rules.
How do I save the result as a file?
Press Download robots.txt to save it with the right name and encoding, or copy the text into a plain text editor and save it as robots.txt in UTF-8.
How can I test my robots.txt?
Open the file's URL in your browser to confirm it loads, then check it in Google Search Console, which reports how Google reads your robots.txt.
Can I have different rules for different bots?
Yes. Add another block that starts with a specific User-agent line, such as User-agent: Googlebot, followed by its own rules.
How long until search engines notice changes?
Crawlers usually re-read robots.txt within a day or so, though timing varies by search engine.
Last updated: