The crawler check reads your site's robots.txt and works out, for each AI crawler, whether robots.txt permits it to fetch the home page. That is permission, not proof of access: the check does not test whether a crawler reaches the site or what an assistant gets from it.
Running it
- An owner or admin sets the site's address in Settings, under Site.
- Anyone but a viewer runs the check from the Overview. A workspace can run it up to 12 times an hour.
- The full table is on Crawlers.
The crawler check does not use credits.
Reading the table
Each crawler shows its access as a text tag: allowed, blocked, unknown or not checked. Answer crawlers are listed with the user-triggered fetchers and training-only crawlers, which are there for the record.
robots.txt is read the way crawlers read it: the group naming the crawler wins over the * group, the longest matching path wins, Allow wins a tie, and no robots.txt means everything is allowed.
Googlebot is the normal search crawler. Blocking it stops Google crawling the site, but a blocked page can still appear in Google Search. To take a page out of Google Search, allow crawling and give the page a noindex rule.
Access shown is what robots.txt says, not what a crawler does.