Make and test a robots.txt file

Make a robots.txt that blocks the folders you choose, decides which AI crawlers can use your site, and points to your sitemap. Then test any URL against it, or against your own file, for Googlebot, Bingbot, GPTBot, ClaudeBot and others.

Make a robots.txt

Search engines
AI crawlers

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent: these collect pages to train AI models. Blocking them doesn’t affect Google Search or Apple search results.

OAI-SearchBot, Claude-SearchBot, PerplexityBot: these show and link your pages in ChatGPT, Claude and Perplexity answers, so blocking them can cost you visitors.

Put it at the root of your site, e.g. https://www.example.com/robots.txt. It controls crawling, not indexing: to keep a page out of Google, use a noindex tag and let Google crawl it.

Test a URL

robots.txt to test

    Tested the way Google reads robots.txt (RFC 9309): the most specific group for the crawler, the longest matching rule, and Allow winning a tie. ChatGPT-User and Perplexity-User fetch pages when a person asks and generally don’t follow robots.txt.

    Runs in your browser. Nothing is fetched or sent anywhere.

    How it works

    1. Choose what to block, which AI crawlers to keep out, and add your sitemap: the file is written as you go.
    2. The tester reads a robots.txt the way Google does (RFC 9309): it finds the group for the crawler, then the longest rule that matches the URL.
    3. It shows whether the URL is allowed, and which line decides it.

    What you enter stays on your device

    This tool runs entirely in your browser. Nothing you type is sent to our servers, so there is nothing for us to store, see or delete.

    To check, open your browser’s developer tools, switch to the Network tab and use the tool: you will not see what you type being sent anywhere.

    Questions

    Does blocking a page in robots.txt keep it out of Google?

    Not reliably. robots.txt stops crawling, but a blocked page can still be listed if other sites link to it. To keep a page out of results, let Google crawl it and add a noindex tag.

    Should I block AI crawlers?

    It’s your choice. Training crawlers such as GPTBot, ClaudeBot and Google-Extended collect pages to train AI models; blocking them doesn’t affect search. AI search crawlers show and link your pages in AI answers, so blocking those can cost you visitors.

    Is what I enter saved or sent anywhere?

    No. Everything runs in your browser; nothing is fetched or sent to our servers.