Skip to content

Digital growth & optimisationTechnical guide

AI crawlers hit our site 3x harder than Google: 14 days of server logs

We counted every bot request to mooshfy.com for a fortnight. GPTBot made 40,690 requests. Googlebot made 12,292. Here are the raw numbers, what they do and do not prove, and what we changed because of them.

Quick answer

AI crawlers hit our site 3x harder than Google: 14 days of server logs

Over 14 days, AI crawlers requested pages on mooshfy.com far more often than Googlebot did: GPTBot made 40,690 requests and PerplexityBot 40,588, against 12,292 from Googlebot. Crawl volume is not the same as being cited, so the useful lesson is operational: check that every crawler you want is actually being served pages, bot by bot.

What you will learn

  • GPTBot and PerplexityBot each made more than three times as many requests to mooshfy.com as Googlebot over the same 14 days.
  • About 99.7% of AI-crawler requests to mooshfy.com were answered with a 200 status, so a small share still failed.
  • A high crawl count from an AI crawler shows interest in a site, not that the site is being cited in AI answers.
  • Google says no special AI text files or schema are needed to appear in AI Overviews; an llms.txt file is not a Google ranking tool.
  • Checking crawl success per user agent catches blocks that a sitewide error rate hides.
Contents 7 sections

Over 14 days, AI crawlers requested pages on mooshfy.com far more often than Googlebot did. The figures below come straight from nginx access logs on mooshfy.com, not from a third-party tool.

How many requests did AI crawlers make compared with Googlebot?

GPTBot made 40,690 requests to mooshfy.com in 14 days and PerplexityBot made 40,588, while Googlebot made 12,292.

CrawlerRequests (14 days)Served 200
GPTBot40,69040,622
PerplexityBot40,58840,492
Amazonbot31,06830,954
Bingbot25,39924,993
Googlebot12,29212,161

About 99.7% of all AI-crawler requests to mooshfy.com were answered with a 200 status.

GPTBot and PerplexityBot each made more than three times as many requests as Googlebot. Together with Amazonbot, the three AI-focused crawlers accounted for roughly 112,000 requests against Googlebot's 12,000. That split is the opposite of how most SEO effort is allocated.

Does more AI crawling mean more AI citations?

No, a high crawl count does not mean a site is being cited. Server logs show that an AI crawler fetched a page; server logs cannot show whether ChatGPT, Perplexity or any other engine later quoted that page in an answer. Citation tracking needs a separate method, such as running a fixed set of prompts in each AI engine and recording which sources each answer links to.

What do these server-log numbers not prove?

The mooshfy.com log counts are a measurement of one site, not a rule for every site. Four caveats apply:

  • One site, one fortnight. A news site, a large store or a forum would see a different mix of crawlers.
  • Crawl volume is not citation volume. Being fetched 40,000 times does not mean being quoted once.
  • Site size inflates totals. A site with tens of thousands of URLs gives a crawler more to fetch than a ten-page site, so request rate is a fairer comparison than raw totals.
  • Googlebot crawls more efficiently. Googlebot has crawled mooshfy.com for years and refetches less, so a lower request count from Googlebot does not mean lower interest from Google.

Does Google need an llms.txt file to show a site in AI Overviews?

No. Google says you do not need to create new machine-readable files, AI text files or markup to appear in its AI features, and that there is no special structured data to add (Google Search Central). Mooshfy still keeps an llms.txt file, because the two crawlers requesting mooshfy.com most often belong to engines that have not said the same, but the file is no longer counted as work done for Google.

What did Mooshfy change after reading the server logs?

Mooshfy made three small changes after reviewing the 14 days of logs:

  • Crawl success is now checked per crawler, not in aggregate. A 0.3% error rate across 150,000 requests is still a few hundred failed fetches, and a block that only affects one user agent disappears inside a sitewide average.
  • llms.txt is treated as optional. The file stays for engines that may read it, with no expectation that it affects Google.
  • Page structure gets as much attention as rankings. AI engines retrieve individual passages, so every section on mooshfy.com now opens with a direct answer to its heading.

How do you count AI crawler requests in your own server logs?

Counting AI crawler requests takes one command per crawler if you can read your access logs. For example, run grep -c "GPTBot" /var/log/nginx/access.log and repeat it for Googlebot over the same date range.

Compare the totals, then check status codes for each crawler rather than only the totals. The user agents worth counting today are GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Amazonbot, Bingbot, Applebot-Extended and Googlebot. OpenAI and Perplexity publish the user-agent strings their crawlers use.

The useful result is not a ranking of crawlers. The useful result is finding the one crawler that is being served errors.

Frequently asked questions

Should a Shopify store block AI crawlers like GPTBot?

Blocking AI crawlers stops those engines from reading your pages, which also removes the chance of being cited in their answers. Most stores that want visibility in ChatGPT or Perplexity should allow GPTBot, OAI-SearchBot and PerplexityBot, and decide case by case for crawlers used only for model training. OpenAI and Perplexity document their user agents.

Does an llms.txt file help a site appear in Google AI Overviews?

No. Google states that you do not need to create new machine-readable files, AI text files or markup to appear in its AI features, including AI Overviews and AI Mode (Google Search Central). An llms.txt file may still be read by other services, which is the only reason to keep one.

How often should server logs be checked for AI crawler errors?

Checking AI crawler status codes once a month is enough for most sites, plus after any change to the firewall, CDN or robots.txt. Those changes are the usual cause of a single crawler suddenly receiving errors while the rest of the site looks healthy.

Want more of the right people to find you?

Send us your site. We check what is holding it back in search and quote the work to fix it.

Start a project on WhatsApp