Every day, software owned by OpenAI, Anthropic, Google, Perplexity and a dozen other companies quietly reads New Zealand websites. Some of it collects pages to train AI models. Some indexes content so it can be quoted in AI search results. Some fetches a page live because a person just asked a chatbot about it. Most site owners have no idea it is happening, and the tools that claim to show them have a flaw that is easy to miss. They simply believe whatever the bot says about itself.
An Auckland digital agency has released a free WordPress plugin that tries to close that gap. Forge Creative announced Forge AI Bot Log on Thursday, describing it as a tool that shows which AI crawlers are visiting a site, checks whether each one is genuine, and lets the owner block any of them.
The problem it targets is identity. Web crawlers announce themselves through a line of text called a user agent string, and that text can say anything at all. There is nothing stopping a scraper from labelling itself GPTBot and appearing in a website’s logs as though it belonged to OpenAI.
“Any script can call itself GPTBot and most tools will log it as OpenAI,” said Roger McSaveney, General Manager at Forge Creative. “We built this for our own clients. They kept asking whether AI companies were taking their content, and the honest answer was that nobody could tell them, because the tools out there just take bots at their word. When you verify the traffic properly, you find some of what looks like OpenAI or Anthropic is actually something else using their name.”
The plugin checks each visit against the crawler’s own published infrastructure using forward-confirmed reverse DNS, a method that looks up the name attached to the visiting address, checks that the name points back to that same address, and confirms the name belongs to the company the bot claims to work for. It is the technique Google, OpenAI and Anthropic themselves recommend for confirming their crawlers are real. Every hit is then labelled Verified, Spoofed or Unverified.
Alongside that, the plugin offers a dashboard covering the last 24 hours, 7 days and 30 days, a searchable log with CSV export, a plain English directory of 23 bots from 15 companies including Amazon, Apple and Meta, and one-click blocking that writes the robots.txt rules for the owner. For crawlers known to ignore robots.txt, it supplies a harder block through an .htaccess snippet. Human visitors are not logged at all, and there are settings for IP anonymisation and how long data is kept.
One detail is more consequential than it first appears. The plugin treats the different jobs a single company’s bots perform as separate things. OpenAI’s training crawler, its search indexer and its live page fetcher are listed as three different bots, which means a publisher can refuse to have its pages used for model training while still appearing in AI search answers. Google-Extended and Applebot-Extended are presented as what they actually are, opt-out switches for AI training that have no effect on ordinary Google Search rankings. Site owners have often conflated the two and blocked themselves out of search results by accident.
That distinction is about to matter to a much wider group of website owners. Cloudflare, which sits in front of a large share of the world’s websites, has set a deadline of 15 September for AI companies to separate crawlers used for search from those used for training and AI agents. From that date it will block mixed-use crawlers by default on pages that carry advertising, a change that applies to new customers and to all existing free-tier customers. Announcing the policy on 1 July, chief executive Matthew Prince said that now the majority of traffic on the internet is non-human, the company had to act faster so a sustainable ecosystem could emerge, as TechCrunch reported.
Forge Creative is not arguing that New Zealand businesses should shut the door. “We’re not telling anyone to block AI,” McSaveney said. “For a lot of New Zealand businesses, getting cited by ChatGPT or Perplexity is where the next customer comes from. But you should get to make that call yourself, and right now most site owners can’t, because they can’t see the traffic. Have a look at what’s actually visiting first, then decide.”
That is a reasonable framing of where the argument has landed. For a publisher whose revenue depends on people arriving at the page, an AI system that answers the question without sending the reader anywhere is a straightforward loss. For a plumber, an accountant or a small retailer, being the business an AI assistant names when someone asks for a recommendation is closer to free advertising. The two positions are not reconcilable in the abstract, which is why the practical question is who gets to choose, and whether they can see enough to choose sensibly.
Forge AI Bot Log is free and open source under the GPL, and is available now from the official WordPress plugin directory. It requires WordPress 5.8 or later. It follows LeadSource and TrueConversion, two lead tracking plugins the agency released earlier this year.
Disclosure. Forge Creative is a related party of Newswire.
Have you ever checked which AI bots are reading your website, and would you block them or welcome them? Tell us in the comments below.
This article was written by AI, briefed to report the facts, hopefully without some of the bias people bring to the job 🙂
Leave a Reply