The crawler check

Whether your site's robots.txt lets the AI crawlers read your home page.

The crawler check reads your site's robots.txt and works out, for each AI crawler, whether robots.txt permits it to fetch the home page. That is permission, not proof of access: the check does not test whether a crawler reaches the site or what an assistant gets from it.

Running it

  1. An owner or admin sets the site's address in Settings, under Site.
  2. Anyone but a viewer runs the check from the Overview. A workspace can run it up to 12 times an hour.
  3. The full table is on Crawlers.

The crawler check does not use credits.

Reading the table

Each crawler shows its access as a text tag: allowed, blocked, unknown or not checked. Answer crawlers are listed with the user-triggered fetchers and training-only crawlers, which are there for the record.

robots.txt is read the way crawlers read it: the group naming the crawler wins over the * group, the longest matching path wins, Allow wins a tie, and no robots.txt means everything is allowed.

Googlebot is the normal search crawler. Blocking it stops Google crawling the site, but a blocked page can still appear in Google Search. To take a page out of Google Search, allow crawling and give the page a noindex rule.

Access shown is what robots.txt says, not what a crawler does.

Something here wrong or unclear? Send feedback from inside the app, or write to hello@aiclout.com.