Free Robots.txt Generator
Generate a robots.txt file with crawl rules, sitemap location, and optional blocked paths.
- Crawler rules
- Sitemap directive
- Blocked path formatting
Robots.txt inputs
Robots.txt Generator
Robots.txt file
Generated result
User-agent: * Allow: / Disallow: /admin Disallow: /api/private Sitemap: https://seobase.co/sitemap.xml
Where this tool fits
Robots.txt Generator FAQ
Is the robots.txt generator free?
Yes. The robots.txt generator is free to use on Seobase with no signup required for the on-page robots.txt output.
Can I use the robots.txt generator for SEO content?
Yes. The tool is built for search-focused workflows and pairs best with human review before publishing.
Does the generated robots.txt include a sitemap directive automatically?
Yes. Enter your site URL and the output includes a Sitemap: line pointing at the standard /sitemap.xml location alongside your allow/disallow rules, so search engines can find both in one file.
What happens if I don't list any paths to block?
You'll get a permissive robots.txt that allows crawling of the whole site plus the sitemap reference, a reasonable default for most sites that don't have admin panels or private sections to hide.
Can robots.txt keep a page out of Google's index?
Not reliably, disallowing a URL stops crawling but Google can still index a blocked page if it's linked to elsewhere, just without a description. Use a noindex meta tag on the page itself if the goal is to keep it out of search results entirely.
Related free tools
Enter a page URL and get every image it references — img tags, responsive srcset variants and the Open Graph image — as a plain list of direct links.
Enter a site and get every icon it declares — favicon, apple-touch-icon, mask-icon and each declared size — as direct links you can open or save.
Paste a page's HTML to check its canonical link, or enter a URL to generate a canonical tag in your browser. Find missing, duplicate, relative, or conflicting canonical URLs before they confuse search engines.
Related articles
If you need to configure your WordPress robots.txt file or verify that search engines can crawl your site, start by checking the file directly. The robots.txt file acts as a plain-text instruction set for web crawlers. It tells user agents which areas they can access and which paths they must avoid, such as /wp-admin/ . WordPress creates a virtual version automatically, which means your site applies default directives even before you upload a physical .txt file to your server root.
AhrefsBot is Ahrefs’ web crawler. It fetches pages to build Ahrefs’ search and backlink indexes. Use the user agent in your server logs to identify it, verify the requester’s IP and reverse DNS before blocking, and treat robots.txt as a crawl instruction—not access control. Read the official AhrefsBot documentation , then use the checks below to allow, rate-limit, or block Ahrefs bot without confusing it with Googlebot.
Short answer: Technical SEO makes a page reachable, crawlable, and understandable to search engines. On-page SEO makes the page the best answer for a searcher. Fix technical blockers first, then improve the page’s intent match, structure, and evidence. Doing more of one cannot compensate for a failure in the other.
Your next customer is searching right now.
Start today and your first articles can be live this week, on your site and in your voice, quietly picking up keywords while you get on with everything else.
Start your SEO workflow3 days free · Cancel anytime