DEVELOPER TOOLS / ONLINE UTILITY

Robots.txt Generator

Build a simple robots.txt file and review the crawler directives before publishing it at your website root.

FREE ONLINE TOOL

Robots.txt Generator

Runs in your browser
Robots.txt is not access control. Never use it to protect confidential pages or files.
OVERVIEW

What is Robots.txt Generator?

The robots.txt file provides crawler instructions for paths on a host. Search engines may use it to decide which URLs they should crawl, but it does not prevent users from accessing a URL and does not guarantee de-indexing.

What can you use it for?

  • Create a basic crawler policy for a website.
  • Add disallow rules for non-public utility paths that should not be crawled.
  • Include the absolute sitemap URL in the robots.txt file.

How to use this tool

  1. Choose a user-agent, usually * for all crawlers.
  2. Enter one path per line in Allow and Disallow fields.
  3. Optionally enter an absolute sitemap URL.
  4. Generate the file and save it as robots.txt in the website document root.

Example input

These example paths are placed in the Disallow field. Review them carefully before publishing.

/admin/\n/search/\n/tmp/

Tips and things to know

  • Robots.txt is public and must not contain secrets or private URLs you need to hide.
  • Disallow does not provide access control and crawlers may still index URLs discovered elsewhere.
  • Avoid blocking CSS or JavaScript files needed to render important pages.

Frequently asked questions

Does Disallow hide a page from search results?

Not necessarily. It asks compliant crawlers not to crawl a path; a URL may still be indexed if discovered elsewhere.

Where should robots.txt be uploaded?

It is normally placed at the root of the host, for example https://example.com/robots.txt.

Can I block private admin pages with robots.txt?

Do not rely on robots.txt for security. Protect admin pages with authentication, authorization, and server-side access controls.