Bots Passed Humans This Year. Here Is What the AI Ones Actually Do on a Store.
Cloudflare, Akamai, Fastly, HUMAN and DataDome all published edge data on AI bots in 2026. Put together, the numbers show retail absorbing nearly half of all AI bot traffic, agents spending 77% of their time on product and search pages, and Shopify's own reports hiding the whole thing.
Product Gale Team
Bots Passed Humans This Year. Here Is What the AI Ones Actually Do on a Store.
On 3 June 2026 Cloudflare's CEO posted a chart showing that automated systems had made 57.5% of HTML requests on its network over the previous week, against 42.5% for humans. He had expected the crossover in 2027. Cloudflare's September review made it official: "for the first time, more than half of Internet traffic wasn't human," and daily requests from AI agents had grown by more than 1,700% in a year.
A Shopify merchant reads that and asks a reasonable question: what are these things doing on my store? Five edge and bot-security vendors published measurements in 2026 that answer it. We have put them side by side, kept their denominators straight (most coverage does not), and worked out what the figures mean for one storefront.
1. Retail is where the AI bots go
Every vendor that broke traffic down by industry found the same thing. AI bots over-index on commerce by a wide margin.
| Measurement | Figure | Source, period |
|---|---|---|
| Share of all AI bot traffic that landed on commerce | 47.9% | Akamai, Jul to Dec 2025 |
| Share of agentic AI traffic going to retail and ecommerce | 46.6% | HUMAN, 2025 |
| Share of AI crawler activity on retail plus software | just over 40% | Cloudflare, Oct 2025 |
| Share of AI bot POST requests hitting ecommerce | 44.8% | Akamai, 30-day window, Sep 2026 |
| North American retail alone | 33 billion AI bot hits, 16% of global AI bot activity | Akamai, 2025 |
A caution on the first row, because it is widely misquoted. 47.9% is the share of AI bot traffic that lands on commerce sites, not the share of commerce traffic that is AI bots. Akamai says plainly that "AI bots currently account for a small fraction of the total volume across our network." DataDome puts AI's share of all web requests at 1.7% in June 2026, up from 1.2% a year earlier. Cloudflare had AI bots at an average 4.2% of HTML requests in 2025.
So the picture is: a small slice of the web's traffic, but a slice that is concentrated on stores.
2. What they do once they arrive
HUMAN's 2026 benchmark is the only report that classifies agentic activity by the kind of page it touched. Across 2025:
| Page type | Share of agentic AI activity |
|---|---|
| Product and search pages | 77% |
| Account pages | 8.8% |
| Authentication flows | 5% |
| Content pages | 3.6% |
| Checkout | 2.3% |
| Other | 3.3% |
That distribution matters for how a merchant should feel about it. Three quarters of agent activity is a machine reading product pages and running searches, which is to say, doing the comparison a shopper used to do on the category page. Checkout is 2.3%, which HUMAN calls "small in relative terms but significant in kind." DataDome's cut of the same question across its own customers is consistent: of 29 billion AI bot requests in the first half of 2026, 97.9% went to ordinary content. Carts and payment flows together drew 0.27% of them.
The purpose mix also matters. Akamai classified the AI bots hitting commerce sites by what they were for.
| Purpose, commerce sites | Share of AI bot activity |
|---|---|
| Training crawlers | 71.0% |
| Fetchers (a user asked for the page) | 14.8% |
| Search crawlers | 13.6% |
| Agents | 0.4% |
Seven in ten AI fetches of a product page were a model builder collecting training data. Fewer than three in ten were anything that could send a shopper back. And the live, user-directed share is the one growing fastest: Cloudflare counted ChatGPT-User, the fetcher that runs when someone asks ChatGPT about your product, at up to 16x its early-2025 volume by December.
Who is doing the fetching? Akamai's commerce ranking: OpenAI 37%, ByteDance 28%, Anthropic 18%, Meta 14%, Perplexity 1%. HUMAN, whose panel skews retail, had OpenAI's bots at roughly 62% to 69% of all AI-driven traffic, with ChatGPT-User alone at 31.8%.
3. Crawls per referral: the price of being read
Cloudflare's crawl-to-refer ratio is the number of HTML pages a company's bots fetch for every visit its products send back. It is the clearest measure of what the arrangement costs a site.
| Operator | Jan 2025 | Jul 2025 | Q3 2026 (Radar data as read by TechnologyChecker) |
|---|---|---|---|
| Anthropic | 286,930 : 1 | 38,066 : 1 | 988 : 1 |
| OpenAI | 1,217 : 1 | 1,091 : 1 | 309 : 1 |
| Perplexity | 55 : 1 | 195 : 1 | 1,083 : 1 |
| 3.8 : 1 | 5.4 : 1 | 4.9 : 1 |
Two things changed in 21 months. Anthropic's ratio fell by more than 99.8%, from the most extractive operator to one of the middle ones, as Claude started sending real traffic. Perplexity went the other way and by the third quarter of 2026 was fetching more pages per referral than anyone. Google's ratio barely moved, and that stability is the benchmark: a search engine crawls a handful of pages per visit it sends.
Retail-specific numbers exist too. Botify's data across about 200 retail and ecommerce sites in 2025, published by Retail Economics, put the ratio at one visit per 198 OpenAI crawls, against one per six for Google. Apply that to a store receiving 100 ChatGPT referral visits a month and OpenAI fetched about 19,800 of its pages to produce them. Google would have fetched about 600 for the same result.
Those fetches are not cheap in the way search crawls are. Fastly measured that over 51% of automated AI requests reach the origin server, against under 9% of human requests, because AI bots want the current page rather than the cached one. Cloudflare adds that over half of AI crawl traffic is spent re-fetching pages that have not changed.
4. Most of it you cannot trust, and most of it you cannot see
Two measurement problems sit on top of all this.
Identity. DataDome sent a request claiming to be ChatGPT-User to 698,214 live sites. 79.7% did not block or challenge it. Its wider finding is that 80% of AI agents do not properly identify themselves, and that spoofing of user-prompted agents rose 45% between February and July 2026. The fix is Web Bot Auth, where the operator cryptographically signs each request; by September 2026 OpenAI, Google and AWS were signing. Everyone else is a user-agent string that anyone can type.
Visibility. Shopify handles bot management for every store and publishes no figure for the bot share of storefront traffic. Since 7 October 2025 its analytics has classified sessions as human or bot, and since late September 2026 identified bot sessions are filtered out of session reports by default, with no way to adjust the filter on the home page or Live View. That is the right default for a sales report. It also means a merchant looking at Shopify Analytics will never see a crawler, a fetcher or an agent, and cannot tell how often an assistant read a product page before deciding whether to mention it.
A JavaScript tracker does not help here either. Crawlers and fetchers do not run JavaScript, so a pixel on the page records none of them. The only place this traffic is observable is at the edge, in the request log, before the page renders.
5. What Shopify tells the bots
Shopify's managed robots.txt, as served on 10 October 2026, is worth reading because it is the merchant's policy whether they wrote it or not. It opens:
# Shopify storefront. Public product, collection, page, blog, policy, cart, and localized HTML is crawlable.
# Agent instructions: https://www.example.com/agents.md
# UCP discovery: https://www.example.com/.well-known/ucp
# Agents should use UCP/MCP for catalog, cart, and checkout. Payment requires buyer approval.
#
# Checkouts are for humans. Do NOT complete checkout, payment, or order placement automatically
Three things to notice.
- It names no AI user agent and blocks none. GPTBot, ClaudeBot, PerplexityBot and the rest are allowed by default, which is consistent with Shopify's policy that blocking crawlers "affects only open-web discoverability" and does not stop Shopify Catalog from sending product data to the agentic storefronts anyway.
- It points agents to a machine-readable catalog, cart and checkout (UCP and MCP endpoints, and a shop.app skill) and tells them to keep the human in the loop at payment. This is the "discovery in the assistant, purchase on the store" model in one file.
- A store with its own
robots.txt.liquidtemplate does not get any of it. Those stores still serve the older rule set with no agent instructions. Shopify warns that editing the template is unsupported and that "incorrect use of the feature can result in loss of all traffic."
For contrast, across the web 17% of Cloudflare sites now block AI training, and between 15% and 20% of the top 1,000 domains fully block GPTBot. Among those GPTBot blockers, 41 of 76 in one census never mention OAI-SearchBot, the crawler that decides whether a site appears in ChatGPT search. They have blocked the training bot and left the search bot open, or in some cases the reverse, without apparently knowing the two are different.
What a merchant can do with this
- Get a request log. Shopify Analytics will not show you bots. Product Gale's tag records them at the edge, alongside the human visits that follow. Thirty days is enough to see which operators read your catalog and how often.
- Count crawls per referral for your own store. Divide each operator's fetches by the visits it sent. If an assistant reads your pages 500 times a month and sends nobody, that is a finding: it is looking and not recommending.
- Check which robots.txt you serve. If you have a
robots.txt.liquidtemplate, you have opted out of Shopify's agent instructions without deciding to. Fetch/robots.txtand/agents.mdand read them. - Watch the 2.3%. Checkout attempts by agents are rare today and growing. A log that can distinguish a signed ChatGPT agent from a spoofed string is the difference between a sale and a chargeback.
Sources
- Cloudflare, the crawl-to-click gap, 29 Aug 2025: blog.cloudflare.com
- Cloudflare, Radar 2025 year in review, 15 Dec 2025: blog.cloudflare.com
- Cloudflare, the agentic web, 30 Sep 2026: blog.cloudflare.com
- Cloudflare, accountable mixed-use crawlers, 15 Sep 2026: blog.cloudflare.com
- Cloudflare Radar readings for 2026, TechnologyChecker, updated 1 Oct 2026: technologychecker.io
- Tom's Hardware on the June 2026 bot crossover: tomshardware.com
- Akamai, Securing the Agentic Storefront, 15 Jul 2026: akamai.com
- Help Net Security on Akamai's POST request analysis, 22 Sep 2026: helpnetsecurity.com
- Fastly, AI traffic grew 6.5x faster than human traffic, 9 Jun 2026: fastly.com
- HUMAN Security, 2026 State of AI Traffic and Cyberthreat Benchmark Report (PDF): humansecurity.com
- DataDome, State of Bot and Agent Security 2026, 22 Sep 2026: datadome.co
- Retail Economics with Botify and DataDome, The Future of Search and Discovery, 2026: retaileconomics.co.uk
- Forage.ai, AI crawler blocking in the top 1,000, 4 Sep 2026: forage.ai
- Shopify, bot filtering in analytics: help.shopify.com; session measurement update, Sep 2026: help.shopify.com
- Shopify, products in agentic storefronts: help.shopify.com; editing robots.txt: help.shopify.com
- OpenAI, bots documentation: developers.openai.com
Keep reading
AI Referrals Went From the Worst Retail Channel to the Best in Twelve Months. Here Is the Full Series.
Adobe has published AI referral data for US retail every quarter since late 2024. Lined up end to end, the series shows a channel that grew 14x, flipped from converting 38% worse to 60% better, and is still mostly invisible to the stores receiving it.
Every Shopify Store Is in ChatGPT by Default. The Numbers Say the Sale Still Happens on Your Site.
Shopify switched Agentic Storefronts on for millions of merchants in March 2026, the same week OpenAI wound down Instant Checkout. The published figures from Shopify, Walmart, Etsy, Google and Gartner all point the same way: assistants discover, stores convert.