About this website
Go Top SEO publishes a free check of how a website treats AI crawlers. Enter a domain and the page reads its robots.txt, then shows which AI user-agents are allowed and which are blocked. Behind the checker sits an open reference table of 26 AI crawler tokens, from GPTBot and ClaudeBot to Google-Extended, PerplexityBot and CCBot. Every row records what the operator's own documentation says the crawler is for, whether that is collecting training data, indexing for search, fetching a page to answer a live question, or serving ads. Each row also carries the operator's documentation URL and the date that page was read, so nothing is inferred from server logs. The dataset is available as JSON or CSV under CC BY 4.0, with an OpenAPI 3.1 definition, and needs no key and no signup.
Key features
- Checks any domain's robots.txt for AI crawler rules
- Open reference table of 26 AI crawler user-agent tokens
- Each row cites the operator's documentation and the date it was read
- Separates training crawlers from search, answer and ads crawlers
- JSON and CSV under CC BY 4.0 with an OpenAPI 3.1 endpoint
Who is this website for?
Site owners, SEO specialists and developers who need to decide which AI crawlers to allow or block on their own sites.
What can visitors do here?
Paste your domain to see whether GPTBot or Google-Extended is currently allowed by your robots.txt, then open the dataset to read what each crawler is documented to do.
What makes it different?
Every row is sourced from the crawler operator's own documentation, with the URL and the date it was read recorded in the data, rather than inferred from server logs.
