robots.txt generator and checker

Choose what crawlers may fetch and get a robots.txt, or paste one to see what it does for a path.

What this tool is for

A robots.txt file sits at the root of a site and asks crawlers which paths to leave alone. Compliant crawlers follow it; it is a request, not a lock. Use the generator to write one, and the checker to paste an existing file and see, in plain words, whether a given crawler may fetch a given path.

Everything runs in your browser. What you type is not sent anywhere and is not saved.

Write a robots.txt

What may crawlers fetch?

Asks 15 named AI crawlers, such as GPTBot and ClaudeBot, to fetch nothing.

For example https://example.com/sitemap.xml

    Check a robots.txt

    For example /blog/my-post

    For example Googlebot or GPTBot. Empty means any crawler.

    Rules follow RFC 9309: the longest matching rule wins, and Allow wins a tie.

    Want this done for you?

    A site published with Galascribe generates its robots.txt for you, with an option to block AI crawlers.

    See what Galascribe does