Free Robots.txt Generator
Generate a robots.txt file with crawl rules, sitemap location, and optional blocked paths.
- Crawler rules
- Sitemap directive
- Blocked path formatting
Robots.txt inputs
Robots.txt Generator
Robots.txt file
Generated result
User-agent: * Allow: / Disallow: /admin Disallow: /api/private Sitemap: https://seobase.co/sitemap.xml
Where this tool fits
Robots.txt Generator FAQ
Is the robots.txt generator free?
Yes. The robots.txt generator is free to use on Seobase with no signup required for the on-page robots.txt output.
Can I use the robots.txt generator for SEO content?
Yes. The tool is built for search-focused workflows and pairs best with human review before publishing.
Does the generated robots.txt include a sitemap directive automatically?
Yes. Enter your site URL and the output includes a Sitemap: line pointing at the standard /sitemap.xml location alongside your allow/disallow rules, so search engines can find both in one file.
What happens if I don't list any paths to block?
You'll get a permissive robots.txt that allows crawling of the whole site plus the sitemap reference, a reasonable default for most sites that don't have admin panels or private sections to hide.
Can robots.txt keep a page out of Google's index?
Not reliably, disallowing a URL stops crawling but Google can still index a blocked page if it's linked to elsewhere, just without a description. Use a noindex meta tag on the page itself if the goal is to keep it out of search results entirely.
How do I block Ahrefs, Moz, and Majestic's crawlers?
Add User-agent rules for AhrefsBot, rogerbot (Moz), and MJ12bot (Majestic) with a Disallow: / each — this generator's "paths to block" field applies to the wildcard User-agent by default, so add these three as separate blocks if you want to keep the site fully crawlable to Google while blocking competitor backlink-index bots specifically. This stops those tools from indexing your link profile; it has no effect on Google, Bing, or AI crawlers like GPTBot.
Related free tools
Enter a page URL and get every image it references — img tags, responsive srcset variants and the Open Graph image — as a plain list of direct links.
Enter a site and get every icon it declares — favicon, apple-touch-icon, mask-icon and each declared size — as direct links you can open or save.
Paste a page's HTML to check its canonical link, or enter a URL to generate a canonical tag in your browser. Find missing, duplicate, relative, or conflicting canonical URLs before they confuse search engines.
Your next customer is searching right now.
Start today and your first articles can be live this week, on your site and in your voice, quietly picking up keywords while you get on with everything else.
3 days free · Cancel anytime