# Sites with bot protection If your site uses bot protection such as Cloudflare or Vercel Firewall, it can stop Asktopus from reading your pages. A secret header lets Asktopus through without switching the protection off. ## Why bot protection blocks Asktopus Bot protection looks at each request and challenges anything that doesn't behave like a person in a normal browser. Asktopus reads your site automatically, so it gets the challenge instead of your content: a verification page such as Cloudflare's "Just a moment..." or the Vercel Security Checkpoint. There's nothing to learn from those pages. When Asktopus reads your pages and only gets verification pages back, Asktopus doesn't add them to the knowledge base. The Knowledge tab tells you that your site's bot protection blocked Asktopus and links to the fix below. ## Why an IP allowlist isn't enough Asktopus fetches your sitemap and robots.txt from its own server, but it reads the pages themselves in a cloud browser on Cloudflare's network, so pages built with JavaScript come out right. Those requests don't come from one fixed IP address, and the addresses they do come from are shared with everyone else who uses the same service. Allowing them would let far more than Asktopus through. A secret header is precise: only requests that carry your value get past. ## How the header works You choose a header name and a secret value. Asktopus then adds the header to every request it makes to your site when it scans and reads it: - Sitemap and robots.txt fetches from Asktopus's server. - Page reads in the cloud browser. - **Only your domain** — The header is sent only to your site's domain and its subdomains, for example example.com, www.example.com and blog.example.com. It's never sent to other domains, including when a page redirects somewhere else. - **Nothing from other domains** — While a header is set, the cloud browser loads nothing from other domains when it reads your pages. Analytics, ad scripts and third-party widgets are blocked, so your secret can't leak to them. If your site loads its actual content from another domain, those pages may come out incomplete while a header is set. - **Never shown again** — After you save a value, it's never shown in full again. The dashboard shows the last four characters of values that are 12 characters or longer. ## Set it up in Asktopus 1. Open your site in the dashboard and go to Settings › Bot protection. 2. Enter a header name, for example X-Asktopus-Verify. 3. Click Generate to create a random 32-character secret, or paste your own. Copy it now: you need it for your firewall rule, and it isn't shown in full again. 4. Click Add header. 5. Add a rule to your firewall that lets requests with this header and value through. The platform guides below show how. 6. Go to the Knowledge tab and click Scan for new pages, then Update knowledge. New and changed headers take effect from the next scan or update. You can add up to 5 headers. Names can be up to 64 characters long and use the standard header-name characters; values can be up to 1,024 printable ASCII characters. To change a value, add the same name again and it replaces the old one. Remove deletes a header. Owners and editors can manage headers, on every plan. Some names can't be used because the browser or the network controls them: Host, Content-Length, Content-Type, Content-Encoding, Transfer-Encoding, Connection, Keep-Alive, Upgrade, TE, Trailer, Expect, Cookie, User-Agent, Accept-Encoding, and any name that starts with cf-, sec- or proxy-. ## Cloudflare ### Super Bot Fight Mode (Pro, Business and Enterprise) Create a custom rule that skips Super Bot Fight Mode for requests with your header: 1. In the Cloudflare dashboard, open your domain and go to Security › Security rules. 2. Select Create rule › Custom rules and give the rule a name, for example Allow Asktopus. 3. Under When incoming requests match, select Edit expression and enter the expression below with your own header name and secret. 4. Under Then take action, choose Skip and select All Super Bot Fight Mode rules. 5. Select Deploy. ``` any(http.request.headers["x-asktopus-verify"][*] == "your-secret") ``` Write the header name in lowercase in the expression, because Cloudflare stores header names in lowercase. The value has to match exactly. Custom rules run in order, and a rule that blocks or challenges stops the evaluation. Place the skip rule above any of your own rules that could block Asktopus. If rate limiting rules or managed rules also stop Asktopus, you can select them in the same Skip rule. ### Bot Fight Mode (Free plan) Bot Fight Mode can't be skipped. It doesn't run in Cloudflare's rules engine, so Skip, Bypass and Allow rules have no effect on it, and a header won't help. If it blocks Asktopus, either turn Bot Fight Mode off (Security › Settings, filtered by Bot traffic) or upgrade to Pro and use Super Bot Fight Mode with the skip rule above. ## Vercel Firewall Vercel's Bot Protection ruleset shows the Vercel Security Checkpoint to traffic that is unlikely to come from a browser. A custom rule with the Bypass action lets requests with your header through. Custom rules run before managed rulesets such as Bot Protection. 1. In your Vercel dashboard, open the project and select Firewall in the sidebar. 2. Select ⋯ › Configure, then Add New… › Rule, and give the rule a name. 3. Add an If condition on the request header: your header name, matched exactly against your secret. 4. Set Then to Bypass and select Save Rule. 5. Select Review Changes, then Publish. Keep the rule at the top of your custom rules, because a bypass only skips the rules after it. Bypass doesn't cover Vercel's system-level mitigations such as DDoS mitigation. ## Other firewalls Any firewall or proxy that can match a request header can use the same approach: make a rule that matches your header name and exact secret, and let it skip the bot check. - **AWS WAF** — add a rule with a string match on your header, set to Exactly matches string, with the Allow action. Allow ends the evaluation, so place the rule before your Bot Control rule group. - **Your own server** — in nginx, Apache or your application code, skip the bot check when the header carries your secret. ## Keep the secret safe Treat the value like a password. Use a long random value (Generate makes one), don't reuse it anywhere else, and keep it out of public code. The header only gets Asktopus past the bot check, so keep your other protection, such as rate limits, in place. To change the secret, let your firewall rule accept both the old and the new value, add the new value in Asktopus under the same header name, and then remove the old value from the rule. ## Check that it works After the update, the Knowledge tab should show your pages and no bot protection warning. Your firewall's event log shows which rule handled each request: in Cloudflare under Security › Analytics › Events, and in Vercel on the Firewall overview page, where you can group traffic by your custom rule. If requests with the header are still challenged, check that the name and value match exactly and that the rule sits above any rule that blocks.