Build clean, search-friendly robots.txt files. Set user-agents, allow/disallow paths, and insert your XML sitemap URL.
User Guide
Robots.txt tells search bots where they can and cannot crawl. This preserves crawl budget and blocks indexing of private or administrative directories.
Highly rated by 140+ developers and search consultants. 100% free, processed client-side.
FAQ
It is a simple text file placed in your website's root directory that instructs search engine crawlers which pages or folders they should not crawl.
Not reliably. To keep a page out of search results, use a "noindex" meta tag rather than blocking crawling in robots.txt.
You can check your rules using the Robots Testing Tool in Google Search Console to verify that important pages are not accidentally blocked.
Get a professional, hand-reviewed website speed, SEO audit, and ranking opportunity framework from our New York team.
Delivered in 3 business days. No spam, no obligation.
