Robots.txt Generator
Build a robots.txt with allow/disallow rules and sitemap pointer.
Build a valid robots.txt by adding allow/disallow rules per user agent, plus a sitemap URL. Copy the output to /robots.txt at your site root.
Common use cases
- Block staging or admin paths from being crawled
- Allow Googlebot but block scraping bots like AhrefsBot or SemrushBot
- Add a sitemap pointer so crawlers find your XML sitemap
- Re-enable crawling after a launch by replacing a 'Disallow: /' file
Frequently asked questions
Does robots.txt block crawling?▾
It tells well-behaved crawlers what not to fetch. It is not a security control — sensitive content should be protected by auth, not just by robots.txt.
Where do I put the robots.txt file?▾
At the root of your domain — exactly /robots.txt. It must be accessible at https://yoursite.com/robots.txt to be honored by crawlers.
Related tools
🏷
Meta Tag Generator
Generate complete <head> meta tags for SEO and social sharing.
🔗
Open Graph Preview
Preview how a URL will look when shared on social media.
🔍
Keyword Density Checker
Analyze single-word, 2-gram and 3-gram frequency in any text.
🗺
XML Sitemap Generator
Build a valid sitemap.xml from a list of URLs with lastmod, changefreq and priority.