robots.txt Generator | Crawl Rules and Sitemap Path
Create robots.txt rules for user agents, allowed or blocked paths, crawl delay, sitemap location, and common crawler directives.
Default Rules for All Crawlers (*)
Specific Search Robots
What robots.txt Controls
A robots.txt file gives crawl instructions to search engine bots, including allowed paths, disallowed directories, crawl delay hints, and sitemap location.
Use Crawl Rules Carefully
Blocking the wrong path can keep important pages out of search. Check admin paths, private folders, duplicate areas, and sitemap URL before publishing the file.
robots.txt Is Not Security
robots.txt is a crawler instruction, not an access-control system. Do not rely on it to hide private files, secrets, staging content, or sensitive directories.
About This Tool
Robots.txt Generator creates a valid robots.txt file from rules you configure in a form. It lets you allow or disallow specific crawlers from accessing parts of your site and adds your sitemap location, without needing to remember the exact syntax.
When to Use It
Use this when launching a new site and you need to create a robots.txt file, when you need to block a specific bot from crawling certain directories, or when adding a sitemap directive to an existing file.
How to Use
- Select which user agents to configure: all bots or specific ones.
- Set disallow rules for paths you want to block.
- Add allow rules for paths within blocked directories.
- Enter your sitemap URL.
- Copy the generated robots.txt content.
Frequently Asked Questions
Does blocking a path in robots.txt prevent it from being indexed?
No. robots.txt tells crawlers not to visit a URL, but if other pages link to it, Google may still index it without crawling it. Use a noindex meta tag to prevent indexing.
Should I block all bots from the admin directory?
Yes. Directories like /admin, /wp-admin, and /private should always be disallowed in robots.txt to reduce crawler load, even though they should also require authentication.
Where should robots.txt be placed?
robots.txt must be placed at the root of your domain at example.com/robots.txt. Placing it in a subdirectory makes it ineffective.