Robots.txt Generator

Build a perfect robots.txt file visually. Control which bots can crawl which parts of your site.

Configuration

Generated robots.txt

How to Use This Tool

  1. 1Choose a preset (Allow All, Block All, WordPress, eCommerce) or start from scratch
  2. 2Set your sitemap URL
  3. 3Add custom User-agent rules if needed
  4. 4Preview the generated robots.txt
  5. 5Copy and upload to the root of your website as robots.txt

Frequently Asked Questions

Where do I put my robots.txt file?

robots.txt must be in the root directory of your domain. It must be accessible at https://yourdomain.com/robots.txt.

Can robots.txt block my entire site from Google?

Yes — Disallow: / for Googlebot blocks all crawling. Be careful with blanket rules.

Does robots.txt prevent content from being indexed?

No. Robots.txt only controls crawling, not indexing. To prevent indexing use <meta name="robots" content="noindex">.

Free Consultation