Knowledge sources and crawling

Four knowledge sources

Every answer the assistant gives comes from one of four sources you control:

• Crawled pages — Pages we fetch automatically from your website. Good for product pages, FAQs, blog posts — anything that's already public.
• Uploaded documents — PDF, Word (.docx), plain text (.txt) and Markdown (.md) files you upload, up to 50 MB each, and CSV files of questions and answers (the question in the first column, the answer in the second; each pair becomes its own answer). Good for price lists, terms, internal handbooks, anything that isn't on the public site. Their pages count toward your plan's pages, like website pages. To update a document, choose Replace next to it and upload the new version: the old version keeps answering until the new one is ready, and stays if the new file can't be read.
• Articles you write — Knowledge articles you author directly in the dashboard. Good for filling gaps the crawl missed or for answers you want phrased a specific way. Under Knowledge → Articles, the dashboard lists questions from the last 30 days your assistant couldn't answer or visitors marked as not helpful, with a button to write the article that answers each one.
• Shop products — The product catalogue of your webshop, synced automatically with prices, variants and availability. See Shop products below.

One site, multiple sources

An Asktopus site can pull content from more than one place. The site has a primary domain (set when you create it), and you can add more crawl sources: other public websites, such as a subdomain, another domain you own or a public help centre, and single pages. All sources feed the same knowledge base — visitors get answers from any of your content regardless of which source it came from.

• Primary domain — The URL you used to create the site. Cannot be removed; the widget can always be embedded here.
• Additional domain — Any public website: a subdomain like blog.acme.com, another domain you own, a help centre on acme.zendesk.com, or a public Notion or Confluence site. Each one is scanned independently and gets its own include/exclude patterns. Asktopus reads the pages anyone can open, so content behind a login can't be added. Headers you set under Settings → Bot protection are only ever sent to your site's own domain and its subdomains, never to a source on another domain. The widget can also be embedded on subdomains of your own domain that you add.
• Single page — One specific URL — useful for a page the scan missed, a freshly published page you don't want to wait for the next full re-crawl to pick up, or a high-importance page outside your normal patterns.

All sources share your plan's pages per site, together with uploaded documents and synced products. Pro's 5,000 pages are split however you like across the primary domain, other domains, single pages, documents and products.

Scan vs. crawl

When you add a site, Asktopus first runs a scan: it reads your sitemap and a sample of pages to suggest URL patterns to include or exclude. The scan is fast and doesn't fetch every page. The crawl is the actual fetch — it follows the patterns you've approved and downloads each page's content. Both can be re-run from the dashboard whenever you change your site.

Include and exclude patterns

After the scan you'll see URL patterns grouped by structure — for example, /produkter/*, /blog/*, /admin/*. Tick the ones to include in the crawl and untick the ones to skip. Common things to exclude: admin pages, search-result pages, paginated archives, customer-account pages. The crawl only fetches pages that match an included pattern and don't match any excluded one.

Shop products

If your site is a webshop, Asktopus syncs its product catalogue into the assistant's knowledge. Each product becomes its own answer source with its name, brand, price (and previous price), variants and availability, SKU, categories, description and a link to the product page. Visitors can ask “Do you have it in size M?” or “What does it cost?” and get the current answer with a link to the product.

• Read automatically — Asktopus looks for a shop when it scans your site, so there's nothing to set up. Shopify, WooCommerce and Adobe Commerce (Magento) are read from what the platforms publish. Wix, Squarespace, PrestaShop, BigCommerce, Ecwid, Shopware and most other shop systems are read from the product data their product pages publish for search engines (schema.org).
• Product feed — Any shop that publishes a Google Shopping product feed can paste the feed's address instead. Many shop systems can produce one: it's the same feed Google and Meta use.
• How often — Shopify, WooCommerce, Magento and product feeds sync every hour on Pro and Business, and daily on the free plan and Starter. Shops read from their product pages sync daily on every plan. Sync now updates the products at any time, on any plan. Only changed products are updated, and the assistant keeps answering about your products while a sync runs.
• Pages — Each product counts as one page of your plan's pages per site: 50 on the free plan, 500 on Starter, 5,000 on Pro and 25,000 on Business. Product pages are then left out of the website update, so nothing is counted twice. If your catalogue has more products than your pages leave room for, only the products that fit are synced.

On the Knowledge page, the Products tab shows the shop Asktopus found, the sync status and the synced products. That's where you turn on the sync with one click, or paste the address of a product feed.

robots.txt and blocking

Asktopus respects robots.txt. If your robots.txt blocks our crawler, the dashboard will tell you and link to a fix — usually removing a Disallow rule that targets all crawlers. While the crawl is blocked you can still build a knowledge base from uploaded documents and hand-written articles; the assistant just won't know about your live pages.

Bot protection from Cloudflare, Vercel and similar services can also stop the crawler by showing it a verification page instead of your content. Let Asktopus through with a secret header

What happens after a crawl

Once content is fetched (whether from the crawl, a document upload, or an article), it goes through the same three-step processing pipeline:

• The text is split into overlapping chunks — roughly 1,500 characters each, with 200-character overlap so context isn't cut at chunk boundaries.
• Each chunk is converted into a 1,536-dimensional embedding vector using OpenAI's text-embedding-3-small model.
• Vectors are stored in our search index. When a visitor asks a question, we embed their question the same way and retrieve the closest chunks, which the language model uses to write the answer.

Per-plan limits

Your plan's pages cap how much knowledge a site can hold. Website pages, document pages and products count together: a PDF counts its pages, a Word, text, Markdown or CSV file one page per 2,400 characters, and a synced product one page. A document that doesn't fit in the pages left is refused with a note of how many are left. The knowledge sources limit caps how many uploaded documents and hand-written articles you can have per site combined.

If your site has more pages than your plan allows, use exclude patterns to skip low-value sections (search results, archives, account pages) so the crawl focuses on the pages that matter for visitor questions.

Keeping the assistant up to date

On the Pro and Business plans, each site updates its knowledge automatically, every week on Pro and every 3 days on Business: it re-reads your sitemap, picks up new pages in the sections you chose, and replaces the old content without a gap. You can switch this off per site on the Knowledge page. On Starter, or right after you publish something important, click Update knowledge. To update a document, choose Replace next to it and upload the new version; the old one keeps answering until the new one is ready. Editing an article updates it in place. Synced products follow their own schedule, described under Shop products.