A founder blocked ChatGPT’s crawler for two years — and nothing looked wrong

The short version: a SaaS founder's robots.txt silently blocked GPTBot for two years. Google kept crawling normally, rankings never moved, no dashboard showed a symptom. After fixing one line, their docs started appearing in ChatGPT answers within weeks. The failure mode is common, invisible by design, and takes 30 seconds to check.

The story

This is a field note — a real problem from a real thread, with the mechanism and the fix. In a discussion about AI visibility, a founder shared their server logs and this:

“We had GPTBot blocked in robots.txt for like 2 years without realizing. Wildcard rule caught it by accident. Fixed that and suddenly our docs started showing up in ChatGPT answers within weeks.”

Two years. No error, no warning, no traffic dip. Just a quiet absence in a channel they weren't watching yet.

Why nothing looked wrong

This is the part worth understanding, because it's why the problem survives for years: robots.txt groups are exclusive. A crawler follows its own named group if one exists — and only falls back to the User-agent: * catch-all if it doesn't.

So the classic “allowlist” pattern, written back when the only bots that mattered were search engines:

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Disallow: /

…works exactly as intended for Google and Bing. But every crawler invented after that file was written — GPTBot (2023), ClaudeBot, PerplexityBot, OAI-SearchBot — has no named group, falls into the catch-all, and gets silently blocked.

And here's the trap: the only symptom is invisibility in a channel you weren't measuring. Your rankings don't flinch, because Googlebot has its own group. Your analytics look normal, because the traffic that's missing never shows up as missing. If the same rule had caught Googlebot, you'd have noticed within a week — your organic traffic would have collapsed. The fact that this founder went two years without noticing is itself proof of how asymmetric the failure is.

The 30-second check

Open yoursite.com/robots.txt and read it line by line. You're looking for two things:

1. Named groups: do GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot have explicit Allow rules — or any named group at all?
2. The catch-all: if they don't, what does your User-agent: * group do? If it disallows anything significant, every unnamed AI crawler inherits that block.

If you find the allowlist pattern, the fix is adding the AI crawlers as named groups (or loosening the catch-all, if that fits your policy). One commit:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Does unblocking actually get you cited?

Honest answer: being crawlable is necessary, not sufficient. This founder saw docs in ChatGPT answers within weeks — a real, log-backed outcome, but one anecdote, not a study. What the broader evidence supports: AI engines can only cite what they can read, none of the major AI crawlers execute JavaScript (they read your raw HTML and move on), and access + server-rendered content + answer-shaped structure is where the measurable levers are. Contrast that with llms.txt — the fix everyone recommends — where Ahrefs found 97% of files never get read at all. Effort should follow evidence: the robots.txt check is 30 seconds and provably load-bearing.

If you're new to this whole channel, start with SEO vs. GEO vs. AEO and why AI Overviews aren't optional anymore — this check is step zero of both.

Check yours now

Our free scan checks exactly this — each AI crawler resolved against your actual robots.txt group rules, plus server-rendering and schema — in about 30 seconds. Or just open the file and read it. Either way: today, not someday. The whole point of this story is that nothing will ever remind you.