Over 14 days, AI crawlers requested pages on mooshfy.com far more often than Googlebot did. The figures below come straight from nginx access logs on mooshfy.com, not from a third-party tool.
How many requests did AI crawlers make compared with Googlebot?
GPTBot made 40,690 requests to mooshfy.com in 14 days and PerplexityBot made 40,588, while Googlebot made 12,292.
| Crawler | Requests (14 days) | Served 200 |
|---|---|---|
| GPTBot | 40,690 | 40,622 |
| PerplexityBot | 40,588 | 40,492 |
| Amazonbot | 31,068 | 30,954 |
| Bingbot | 25,399 | 24,993 |
| Googlebot | 12,292 | 12,161 |
About 99.7% of all AI-crawler requests to mooshfy.com were answered with a 200 status.
GPTBot and PerplexityBot each made more than three times as many requests as Googlebot. Together with Amazonbot, the three AI-focused crawlers accounted for roughly 112,000 requests against Googlebot's 12,000. That split is the opposite of how most SEO effort is allocated.
Does more AI crawling mean more AI citations?
No, a high crawl count does not mean a site is being cited. Server logs show that an AI crawler fetched a page; server logs cannot show whether ChatGPT, Perplexity or any other engine later quoted that page in an answer. Citation tracking needs a separate method, such as running a fixed set of prompts in each AI engine and recording which sources each answer links to.
What do these server-log numbers not prove?
The mooshfy.com log counts are a measurement of one site, not a rule for every site. Four caveats apply:
- One site, one fortnight. A news site, a large store or a forum would see a different mix of crawlers.
- Crawl volume is not citation volume. Being fetched 40,000 times does not mean being quoted once.
- Site size inflates totals. A site with tens of thousands of URLs gives a crawler more to fetch than a ten-page site, so request rate is a fairer comparison than raw totals.
- Googlebot crawls more efficiently. Googlebot has crawled mooshfy.com for years and refetches less, so a lower request count from Googlebot does not mean lower interest from Google.
Does Google need an llms.txt file to show a site in AI Overviews?
No. Google says you do not need to create new machine-readable files, AI text files or markup to appear in its AI features, and that there is no special structured data to add (Google Search Central). Mooshfy still keeps an llms.txt file, because the two crawlers requesting mooshfy.com most often belong to engines that have not said the same, but the file is no longer counted as work done for Google.
What did Mooshfy change after reading the server logs?
Mooshfy made three small changes after reviewing the 14 days of logs:
- Crawl success is now checked per crawler, not in aggregate. A 0.3% error rate across 150,000 requests is still a few hundred failed fetches, and a block that only affects one user agent disappears inside a sitewide average.
- llms.txt is treated as optional. The file stays for engines that may read it, with no expectation that it affects Google.
- Page structure gets as much attention as rankings. AI engines retrieve individual passages, so every section on mooshfy.com now opens with a direct answer to its heading.
How do you count AI crawler requests in your own server logs?
Counting AI crawler requests takes one command per crawler if you can read your access logs. For example, run grep -c "GPTBot" /var/log/nginx/access.log and repeat it for Googlebot over the same date range.
Compare the totals, then check status codes for each crawler rather than only the totals. The user agents worth counting today are GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Amazonbot, Bingbot, Applebot-Extended and Googlebot. OpenAI and Perplexity publish the user-agent strings their crawlers use.
The useful result is not a ranking of crawlers. The useful result is finding the one crawler that is being served errors.
Frequently asked questions
Should a Shopify store block AI crawlers like GPTBot?
Blocking AI crawlers stops those engines from reading your pages, which also removes the chance of being cited in their answers. Most stores that want visibility in ChatGPT or Perplexity should allow GPTBot, OAI-SearchBot and PerplexityBot, and decide case by case for crawlers used only for model training. OpenAI and Perplexity document their user agents.
Does an llms.txt file help a site appear in Google AI Overviews?
No. Google states that you do not need to create new machine-readable files, AI text files or markup to appear in its AI features, including AI Overviews and AI Mode (Google Search Central). An llms.txt file may still be read by other services, which is the only reason to keep one.
How often should server logs be checked for AI crawler errors?
Checking AI crawler status codes once a month is enough for most sites, plus after any change to the firewall, CDN or robots.txt. Those changes are the usual cause of a single crawler suddenly receiving errors while the rest of the site looks healthy.