robots.txt generator and checker
Choose what crawlers may fetch and get a robots.txt, or paste one to see what it does for a path.
What this tool is for
A robots.txt file sits at the root of a site and asks crawlers which paths to leave alone. Compliant crawlers follow it; it is a request, not a lock. Use the generator to write one, and the checker to paste an existing file and see, in plain words, whether a given crawler may fetch a given path.
Everything runs in your browser. What you type is not sent anywhere and is not saved.
Write a robots.txt
For example /private/ or /drafts
Asks 15 named AI crawlers, such as GPTBot and ClaudeBot, to fetch nothing.
For example https://example.com/sitemap.xml
Check a robots.txt
For example /blog/my-post
For example Googlebot or GPTBot. Empty means any crawler.
Rules follow RFC 9309: the longest matching rule wins, and Allow wins a tie.
Want this done for you?
A site published with Galascribe generates its robots.txt for you, with an option to block AI crawlers.
See what Galascribe does