Shoal Research Says Bots Account for 60% of Cloudflare HTML HTTP Requests

Shoal Research said bots account for about 60% of HTML content HTTP requests on Cloudflare’s network, while humans account for 38.2%. According to ChainCatcher, Cloudflare CEO Matthew Prince had previously predicted bot traffic would exceed human traffic in 2027, and later said agent traffic growth had already pushed bots ahead of human traffic. The report said AI applications keep users inside chat interfaces by summarizing content instead of sending them back to original sites. It added that AI crawlers fetch and revisit pages at a much higher rate than Googlebot, with ClaudeBot averaging about 4,200 pages fetched per revisit over the past year. Pew research found AI summaries reduced click-through rates from 15% to 8%. The report also said the current model of using robots.txt and content in exchange for traffic is difficult to sustain. It described lawsuits and private licensing as temporary measures, and said pay-per-request models are split between closed markets with keys and open protocols using HTTP 402 and stablecoins, such as x402 and MPP. Most websites, it said, still either provide access for free or block crawlers.