Robots.txt Generator
Decide which crawlers may visit your site — search engines and AI bots alike. Set a rule per crawler, add your sitemap, then copy or download the finished file and upload it to your site’s root as robots.txt.
AI crawler presets
“Block AI training only” lets AI crawlers visit your pages (so you can still appear in AI answers) while signalling that your content should not be used to train models.
Crawler rules
| Crawler | Rule | Disallowed paths |
|---|
Your robots.txt
# robots.txt — generated with SEO Audit Wizard
Upload this file to the root of your website (for example https://example.com/robots.txt) and test it in Google Search Console.
Frequently asked questions
Does robots.txt hide my pages from Google?
No — blocking a page in robots.txt stops Google from crawling it, but the URL can still appear in search results if other sites link to it. To remove a page from search results, use a noindex tag instead.
What is the __usage_ai-training signal?
It is an emerging robots.txt field announced by Google that lets publishers say their content should not be used to train AI models. It is separate from blocking a crawler: a bot can still visit (and show your content in AI answers) while being told not to train on it. Support is still rolling out, so treat it as an extra signal rather than a guarantee.
Where do I put the robots.txt file?
At the root of your domain — for example https://example.com/robots.txt. One file per domain (or subdomain).
Related tools
Free Website SEO Audit · Meta Tag Generator · XML Sitemap Generator