Between the afternoon of 16 September and the morning of 27 September, AI bots made 12,396 requests to Newswire that we could see. In the same period, AI assistants sent 194 human visits back. That works out at roughly 64 machine requests for every person who clicked through, and it is the first number anyone running a content site in 2026 should probably know about their own traffic.
We know this because the site runs Forge AI Bot Log, a WordPress plugin that records every request from a known AI crawler, checks it against the IP ranges or reverse DNS the vendor publishes, and labels it verified, spoofed or unverified. Its Pro version also records people who arrive from ChatGPT, Gemini, Perplexity, Claude and the other assistants. We reported on the plugin’s release in August.
We are publishing the numbers openly, including the ones that are awkward for us, because most of this traffic is invisible in Google Analytics and very little real-world data from small publishers is public. Here is what 11 AI bots from eight companies did on a New Zealand news site over 11 days.
The first surprise is the mix. The story most publishers have been told is that AI companies are hoovering up their work for training. On Newswire, training crawlers accounted for just 16% of logged requests. The biggest group, 44%, was live retrieval, meaning an assistant fetching a page at that moment because someone asked it something. Search indexing, bots building the indexes AI answers draw on, made up the other 40%. OpenAI’s training crawler, GPTBot, made only 36 requests in 11 days. Its live fetcher, ChatGPT-User, made 2,823.
That distinction matters if you are deciding what to block. OpenAI’s crawler documentation treats GPTBot, OAI-SearchBot and ChatGPT-User as separate controls, and Anthropic’s equivalent page does the same for ClaudeBot and Claude-User. A blanket block on “AI” would have cut off the traffic that actually sends readers, while barely touching training.
The strangest pattern in the log belongs to our tsunami warning page. ChatGPT-User requested it 424 times, 15% of all its traffic to the site. The requests arrived roughly every half hour, day and night, with a median gap of 32.5 minutes, and every one passed verification against OpenAI’s published addresses. OpenAI says ChatGPT-User visits pages when users ask a question and is not used to crawl the web automatically. Nobody asks about tsunamis at 3am every 30 minutes, so our reading is that someone has set ChatGPT up to check New Zealand’s tsunami status on a schedule, whether through a recurring task, a custom GPT or an agent. We cannot see who. It is a small glimpse of AI assistants being used as monitoring tools, not just answer engines, and it shows how an ordinary web page can quietly become somebody’s data feed.
Claude-User, Anthropic’s live fetcher, produced a different puzzle. It made 2,557 requests, and all 2,373 that came from Anthropic’s own network arrived from a single IP address, covering 1,159 different URLs. ClaudeBot used the same address for 171 of its 186 requests, so it looks like a shared Anthropic exit point rather than one customer. Its taste leaned heavily towards business and economics. One story on ANZ’s OCR forecast and BNZ’s mortgage rate rise was fetched nearly 40 times, and the OCR tracker, the fuel prices page and the business category page were each fetched nine times. Nearly three-quarters of the requests were for URLs without the trailing slash WordPress uses, which suggests the links came from somewhere other than our own pages. From the server side, we cannot tell whether that is many Claude users asking about the economy or a handful of automated workflows built on Claude that read a lot of New Zealand economic news. It is a reminder that “verified” confirms which company sent a request, not why.
Apple’s crawler topped the chart but read surprisingly few articles. Of Applebot’s 3,397 requests, 78% went to plumbing, including the news sitemap, RSS feeds, the WordPress REST API and robots.txt. It fetched the news sitemap 482 times. Amazonbot went the other way, spreading 1,642 requests across 1,444 different URLs, broad and shallow, one or two visits per page. OpenAI’s search indexer fetched robots.txt 75 times, about seven times a day, which suggests it takes the file seriously. GoogleOther spent more than a third of its 118 requests on old URLs dating from 2008 to 2017.
Speed is where the training label gets misleading. Of the 70 articles Newswire published from 20 September, Anthropic’s ClaudeBot fetched 50, with a median delay of just under four hours after publication and the quickest within 33 minutes. For a crawler officially described as collecting material for model training, that is close to news-crawler freshness.
Now the humans. The 194 visits from AI assistants came from 179 different visitors. ChatGPT sent 96, Google’s Gemini 67, Perplexity 16, Microsoft Copilot 9, Claude 5 and Meta AI 1. Set those against the bot numbers and the picture is lopsided. Claude-User fetched 2,557 pages and Claude sent five people. ChatGPT’s bots made more than 4,000 requests between them and sent 96. Gemini sent the second-largest group of readers even though Google’s only AI-labelled bot, GoogleOther, made just 118 requests. Gemini appears to draw on Google’s ordinary search index, which is built by Googlebot and is not counted as AI traffic at all.
The single most popular landing page from AI assistants was our interactive Ice Age globe, with 18 visits from ChatGPT. In the same 11 days the log recorded no AI bot requests to that page at all. The link must come from something the assistant already knew or found through a search index, which is a useful caution against reading crawl logs as a direct map of what AI will recommend.
There is an old web development lesson hiding in here too. Many AI fetchers do not run JavaScript. The plugin’s Pro version compares a page’s raw HTML with what a browser renders, to show what bots miss. Of the 12 pages it has checked so far, 11 lost nothing that mattered without scripts, including the tsunami page at 2,248 words. The exception was our fuel prices tracker. Only about 5% of its words are added by JavaScript, but those words include the current pump prices, 25 numbers in all. The part of the page people actually want is the part a bot cannot see. We will be fixing that. Any site that renders prices, stock levels or live figures on the client side probably has the same gap.
Impersonation turned up, but on a small scale. Eleven requests claiming to be ChatGPT-User came from addresses outside OpenAI’s published ranges and were labelled spoofed, which is exactly the kind of traffic that user-agent-only analytics would count as OpenAI. The more interesting false alarm came from developers. Claude Code, Anthropic’s coding agent, fetches web pages under a Claude-User label that includes its version number, and it made 184 requests to Newswire from 148 different IP addresses, many of which appear to be New Zealand broadband connections, with others in Australia, Europe and the United States. Some are probably ours, since Newswire’s articles are written with AI tools, as the note at the foot of every article says. Because Claude Code runs on the user’s own machine rather than Anthropic’s servers, the plugin flagged every one of those requests as spoofed or unverified. Technically that is correct, since the requests did not come from Anthropic’s network, but they are genuine Anthropic software acting for a real person. As coding agents and browser agents spread, “comes from the vendor’s IP range” will describe less and less of the legitimate AI traffic a site receives, and verification labels will need a human to read them.
One important limit applies. The logger runs inside WordPress, and Newswire sits behind a full-page cache, so a bot served a cached copy never reaches the logger. The bot counts here are a minimum, and the real ratio of machine requests to human visitors is probably higher than 64 to one. The referral figures come from a small script in the page, which does run on cached pages.
For anyone building or running sites, the takeaways are fairly practical. Look before you block, because the bot you think is the problem may not be the one doing the reading. Treat AI retrieval as a real audience with its own habits, such as clockwork polling, sitemap-heavy crawling and a strong appetite for data pages. Put the figures that matter in the HTML. And measure the return, because at the moment the AI companies take a lot more from sites like ours than they send back.
Do you track AI bots on your own site, and do your numbers look anything like ours? Tell us in the comments below.
This article was written by AI, briefed to report the facts, hopefully without some of the bias people bring to the job 🙂
Leave a Reply