Which pages AI assistants read on B2B websites: 11 days of our own crawler logs

Most advice about AI crawlers is written without looking at a single log line. We wanted to see what the bots actually request on a real B2B site, so we looked at our own: planconversion.com, a multilingual agency site with service, pricing, comparison, location and blog pages in English, French and Italian.

Our AI crawler monitor reads the site's Netlify request logs once a day, classifies each request by user agent into a bot family, and writes one row per family per day to a tracking sheet: total requests, requests per agent, key files fetched (robots.txt, sitemap.xml, llms.txt), errors and, on some days, the five most requested URLs. This article uses every row from October 30 to November 9, 2026: 112 rows across 11 bot families.

Three limits matter. First, this is one site over 11 days, which is a small sample. Second, the page-level data comes only from the top five URLs per family on the days the monitor pulled them, 1,771 requests in total, so page shares are a sample, not a census. Third, user agents can be faked. We saw scanners requesting credential files under bot names, which is why the steps below include checking IP ranges, and the last day in the sheet may not be a complete day.

Every major assistant vendor visited every day. The table shows totals for the five families we focus on, with the agents each vendor documents. The other six families (other AI search tools such as Exa and ShapBot, Amazon, Apple, Meta, ByteDance and Common Crawl) added 12,968 requests.

The split inside each vendor is the useful part. OpenAI documents OAI-SearchBot as the crawler that surfaces sites in ChatGPT search, GPTBot as the crawler for training data, and ChatGPT-User as the agent that visits a page when a person asks ChatGPT something. On our site, the search crawler made 94% of OpenAI requests and GPTBot about 2%. Perplexity says PerplexityBot surfaces sites in its results and is not used for foundation model training; it made 99% of Perplexity requests. Anthropic describes ClaudeBot as collecting content that may be used for training, Claude-SearchBot as crawling for search quality, and Claude-User as fetching pages for user questions.

Google needs a caution. Googlebot made only 2,712 requests. Most Google traffic came from GoogleOther, which Google describes as a generic crawler for product teams and research crawls, and Google-InspectionTool, which powers Search Console's URL inspection and has no effect on Search. Inspection requests come from testing tools such as URL inspection, so do not read them as demand.

In the URL samples, AI crawlers read the pages that describe the company and its offer far more than the blog. Language versions are grouped with their English page.

Each vendor had a clear habit. In OpenAI's samples, the home pages, service pages and pricing led, with pricing alone at 61 requests on one day. Anthropic's samples were mostly housekeeping: robots.txt and sitemap.xml made up 68% of them, then contact pages. Perplexity's sampled requests went to contact pages 60% of the time, then resources. Bing spread its requests across contact, blog, comparison, team and case study pages.

One technical detail surprised us. Our site is built on Next.js, and 87% of OpenAI's sampled requests were for the framework's .txt data files (for example /pricing.txt) rather than the HTML page. Our monitor noted HTML /pricing at only 2 requests on a day when /pricing.txt drew 61. If your site uses a modern JavaScript framework, check which version of each page the crawler actually receives.

Yes, but not by the assistants most B2B buyers use. llms.txt is a proposal for a Markdown file that gives language models a curated overview of a site. Our monitor logged it as fetched on 9 of the 11 days, in 12 family-day rows: 7 from other AI search tools (almost always Exa, once ShapBot), 2 from Amazon, 2 from Anthropic (November 2 and November 9) and 1 from ByteDance. In the URL samples it was 4 requests out of 1,771.

In the same period, robots.txt appeared as a fetched key file in 33 family-day rows, and Anthropic, OpenAI and Microsoft regularly fetched sitemap.xml. We never saw OpenAI, Perplexity, Google or Bing request llms.txt. Google states that you do not need AI text files or special markup to appear in its AI features.

Our view: keep llms.txt, because it is cheap to maintain and some agents read it, but spend your effort on robots.txt, the sitemap and the pages themselves. That is where the requests went.

• Volume is volatile. OpenAI ranged from 165 to 6,602 requests a day, and Google from 412 to 7,225. One quiet day says nothing.
• Trends differ by vendor. Perplexity averaged 414 requests a day in the first five days and 746 in the last six, with its two highest days at the end. OpenAI averaged 2,779 and then 1,605 over the same halves.
• Anthropic fell for four days in a row (240, 125, 28, 21) and then returned at 404 on November 9. A drop is not a block; check before you change anything.
• User-triggered fetches are visible. ChatGPT-User made 388 requests on November 8, against 33 to 78 on every other day, and Claude-User outnumbered ClaudeBot that day, 38 to 10. These are visits made while someone is asking a question.
• Crawlers follow your hreflang links. ClaudeBot hit 70 errors on November 4 and PerplexityBot about 160 on November 8, all on French and Italian alternates of pages that only exist in English.
• Bot names are borrowed. Requests for credential files and server paths arrived under crawler user agents. Vendors publish IP lists so you can tell the real crawler from an impostor.

The pages assistants read most are the ones that answer "who is this company, what do they sell, for whom, and how do I reach them". Rewrite those first, not the blog archive.

• Home page: state what you sell, to whom and where in the first two sentences, in text, not in an image or animation.
• Service pages: open each with a two to four sentence answer covering the service, the buyer it fits and the outcome, then the detail.
• Pricing: publish at least your pricing model and what drives cost. Crawlers request it, and an assistant with no pricing page fills the gap from review sites.
• Contact and location pages: list offices, regions served and languages plainly. Perplexity and Bing requested contact pages more than any other page type in our samples.
• Comparison and about pages: keep facts consistent with the home page, because assistants read several pages and conflicting facts produce wrong answers.

Blog posts still matter as cited sources for specific questions, but they did not draw most crawler attention. When an assistant already describes your company wrongly, the guide on how to fix wrong AI answers about your company shows how to trace the error back to its source.

You do not need a data team for this. A first pass takes an afternoon, and a daily script keeps it running.

1. Export request logs from your host, CDN or firewall (Netlify, Vercel, Cloudflare, AWS or your server) for at least 14 days, with timestamp, URL, status code, user agent and IP.
2. Filter by the documented user agents: OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, GoogleOther and bingbot.
3. Verify identity: match each request IP against the vendor lists (openai.com/searchbot.json and gptbot.json, claude.com/crawling/bots.json, perplexity.ai/perplexitybot.json) and drop the impostors.
4. Group URLs into page types (home, services, pricing, contact, blog, guides, robots.txt, sitemap.xml, llms.txt) and count requests per vendor and per type.
5. List every 4xx and 5xx response by agent, and look for patterns such as missing language versions or old URLs still linked.
6. Check robots.txt and your firewall or CDN bot settings for each search agent, because bot protection can block crawlers that robots.txt allows.
7. Repeat weekly and compare like with like: same agents, same page groups, same number of days.

If a search agent is missing entirely, fix access first. OpenAI says OAI-SearchBot is needed to appear in ChatGPT search answers, and Perplexity asks site owners to allow PerplexityBot and its IP ranges in the firewall. Both say changes can take about 24 hours. If a crawler hits errors, fix the links that cause them; our own hreflang errors came from a language switcher pointing to pages that did not exist. If the crawlers reach your money pages, make those pages answer-first, then measure whether assistants start describing and recommending you correctly. Crawling is only the input; measuring share of answer tells you whether it turned into visibility.

Most of the fixes behind these findings are covered in our technical SEO checklist for B2B websites in AI search: crawl access, sitemaps, language versions and how pages render. Our generative engine optimization service runs this log review, the page rewrites and the monthly answer tracking for B2B companies in English and French. If you want to know what AI assistants read on your site and what they say about you, talk to our team.