Your robots.txt decides whether an AI crawler may read your pages at all. Everything else on this site assumes the answer is yes — so check it first.
Illustrative results for fictional Northstar Analytics. These are sample data, not live AI answers or measured customer outcomes.
AI crawler check · sample
Suggested change: Allow OpenAI’s search crawler while keeping your training preference.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Merge into your existing rules, then re-scan to check access. An allowed rule does not guarantee a citation.
Check my crawler access →Allowed
A rule names this crawler, or a catch-all group lets it through.
Blocked
A rule disallows it. That crawler will not read the page.
Unspecified
No rule mentions it. In practice that means allowed — but nobody decided it.
Does Allow: / make ChatGPT cite me?
No. It only means that crawler may read the page. Being cited is a different problem — that is what the visibility checker is for.
What if there is no robots.txt at all?
Crawlers treat that as allowed. The snippet still gives you explicit groups, so the decision is written down rather than assumed.
Is this a different parser from the scan?
No — the same one, on the same eight user-agents. This page is just the door, without the rest of the entity card.