What is Robots.txt Generator?
The robots.txt file provides crawler instructions for paths on a host. Search engines may use it to decide which URLs they should crawl, but it does not prevent users from accessing a URL and does not guarantee de-indexing.
What can you use it for?
- Create a basic crawler policy for a website.
- Add disallow rules for non-public utility paths that should not be crawled.
- Include the absolute sitemap URL in the robots.txt file.
How to use this tool
- Choose a user-agent, usually * for all crawlers.
- Enter one path per line in Allow and Disallow fields.
- Optionally enter an absolute sitemap URL.
- Generate the file and save it as robots.txt in the website document root.
Example input
These example paths are placed in the Disallow field. Review them carefully before publishing.
/admin/\n/search/\n/tmp/
Tips and things to know
- Robots.txt is public and must not contain secrets or private URLs you need to hide.
- Disallow does not provide access control and crawlers may still index URLs discovered elsewhere.
- Avoid blocking CSS or JavaScript files needed to render important pages.
Frequently asked questions
Does Disallow hide a page from search results?
Not necessarily. It asks compliant crawlers not to crawl a path; a URL may still be indexed if discovered elsewhere.
Where should robots.txt be uploaded?
It is normally placed at the root of the host, for example https://example.com/robots.txt.
Can I block private admin pages with robots.txt?
Do not rely on robots.txt for security. Protect admin pages with authentication, authorization, and server-side access controls.