When bot protection locks out your own AI assistant

We run more than Asktopus. One of our other sites is companydata.dk, a Danish company data service, and it uses Asktopus to answer visitors' questions. This summer companydata.dk turned on Cloudflare's Super Bot Fight Mode to keep scrapers away. It worked. It also kept our own AI assistant away.

We noticed when the assistant didn't know about a new plan we'd just launched. The knowledge base looked fine at a glance: the last crawl had run and fetched 72 pages. But every one of those 72 pages was the same thing, Cloudflare's "Just a moment..." verification page. There wasn't a single sentence of real content to learn from.

If you use bot protection on your site, the same thing can happen to any AI chatbot that learns from your pages. Here's what we learned fixing it, and what we changed in Asktopus so you don't have to go through the same detour.

Why bot protection blocks AI chatbots

Bot protection looks at each request and challenges anything that doesn't behave like a person in a normal browser. That's the point of it. But an AI chatbot that learns from your website has to read your pages automatically, so it gets the challenge page instead of your content.

The frustrating part is that nothing looks broken. The crawl "succeeds", pages are counted, and the assistant quietly answers from stale or empty knowledge.

Why allowing our IP address didn't help

Our first fix was the obvious one: add a rule that lets the server running Asktopus skip the bot check. Nothing changed.

The reason is how pages get read. Asktopus fetches your sitemap and robots.txt from its own server, but it reads the pages themselves in a cloud browser on Cloudflare's network. That's what lets it read sites built with JavaScript correctly. Those page requests don't come from our server's IP address at all. They come from Cloudflare's network, from addresses that change and are shared with everyone else who uses the same service.

So an IP allowlist either misses the traffic that matters or, if you allow the whole network, lets far more than your AI assistant through.

The workaround that worked, and why we didn't stop there

What finally got companydata.dk crawled again was a Cloudflare rule that matched two things: requests from Cloudflare's own network, and a header that Cloudflare's cloud browser adds to every request. The next crawl read all 72 pages properly and the assistant knew about the new plan.

It worked, but it has two problems. It only works because both the site and the cloud browser happen to be on Cloudflare. And it lets through anyone who uses the same cloud browser service, not just Asktopus.

Later, for other reasons, companydata.dk added another condition to the same rule: skip the bot check for requests that carry a secret header only we know. That turned out to be the general answer. A secret header works on any firewall that can read request headers, it doesn't depend on IP addresses, and only requests with your exact value get through.

The fix: a secret header you control

So we built it into Asktopus. Every site now has a Bot protection setting where you add a header name and a secret value. Asktopus sends that header with every request it makes to your site when it scans and reads it, and you add one rule to your firewall that lets requests with it through.

We were careful about where the secret goes:

• Only to your domain. The header is sent to your site's domain and its subdomains, never to other domains, and not along when a page redirects somewhere else.
• Nothing from other domains. While a header is set, the cloud browser doesn't load anything from other domains when it reads your pages. Analytics, ad scripts and third-party widgets are blocked, so the secret can't leak to them. The trade-off: if your site loads its actual content from another domain, those pages may come out incomplete.
• Never shown again. After you save a value, the dashboard only shows its last four characters.

How to set it up

1. In Asktopus, open your site and go to Settings › Bot protection.
2. Enter a header name such as X-Asktopus-Verify, click Generate for a random secret, copy it, and click Add header.
3. Add a rule to your firewall that lets requests with that header and value skip the bot check.
4. On the Knowledge tab, click Scan for new pages and then Update knowledge.

What the firewall rule looks like depends on your platform:

• Cloudflare with Super Bot Fight Mode (Pro and up): a custom rule with the Skip action and All Super Bot Fight Mode rules selected, matching an expression like any(http.request.headers["x-asktopus-verify"][*] == "your-secret").
• Cloudflare with Bot Fight Mode (Free plan): this one can't be skipped by any rule, so a header won't help. Cloudflare's own advice is to turn Bot Fight Mode off or upgrade to Super Bot Fight Mode.
• Vercel Firewall: a custom rule with a condition on your header and the Bypass action, placed at the top. Custom rules run before Vercel's Bot Protection ruleset.
• Anything else: any firewall or proxy that can match a request header, from AWS WAF to your own nginx config, can do the same.

The bot protection guide has the exact steps for each platform.

Asktopus now tells you when it's blocked

The worst part of our own experience was that nothing told us. The crawl reported success while the knowledge base filled up with verification pages.

That's fixed too. When Asktopus reads your pages and only gets bot verification pages back, such as Cloudflare's "Just a moment..." or the Vercel Security Checkpoint, Asktopus no longer adds them to the knowledge base. The Knowledge tab says that your site's bot protection blocked Asktopus and links straight to the setting.

Frequently asked questions

Does this weaken my bot protection?

Only requests that carry your exact secret skip the bot check, and only the bot check. Keep the value private, make it long and random, and keep your other rules such as rate limits in place.

Can other services see the secret?

The header is only sent to your own domain and its subdomains, and while it's set, Asktopus doesn't load anything from other domains when it reads your pages.

Which plans have the setting?

All of them. Owners and editors on a workspace can add and remove headers.

Do I have to use Cloudflare or Vercel?

No. Any firewall or proxy that can match a request header works the same way. The guide covers Cloudflare, Vercel and AWS WAF, and the general approach for everything else.