A crawler shows access unknown

The check could not read the site's robots.txt, so it can't say which crawlers are allowed.

unknown means the crawler check could not fetch the site's robots.txt, so it can't say which crawlers are allowed. The warning under the check gives the reason.

Why can't the check read robots.txt?

  • The site address is wrong. Check it in Settings, under Site.
  • The server answered with an error (for example HTTP 500 or 429). Try again later.
  • Redirects: too many redirects, or a redirect to an address the check does not follow.
  • The address can't be reached from the internet, or it isn't a web address.

A site with no robots.txt at all (for example a 404) is not "unknown": no robots.txt means every crawler is allowed.

"not checked"

not checked means the check hasn't been run yet. Run it from the Overview. A workspace can run it up to 12 times an hour.

Something here wrong or unclear? Send feedback from inside the app, or write to hello@aiclout.com.