The web wasn't really built for agents was it?. Two decades of bot blocking prove it.

Everyone is arguing about payment rails.

One major payment platform shipped over 250 products at their developer conference last month. An agentic payment token from one of the card networks currently requires three to nine months of KYC and a $250M revenue minimum to qualify. A crypto platform reports 69,000 active agents already running. Big numbers. Real investment. And underneath all of it, a problem nobody seems to want to say out loud:

We've spent the last year scanning hundreds and hundreds of retailer pages across spirits, beauty, supplements, and consumer electronics. Real products, real pages, real WAFs. Security said no most of the time (or was it just doing its job?) What we found makes the payment infrastructure conversation feel like the wrong starting point. The web spent over two decades building systems designed to keep automated access out. That infrastructure is still there, still running, and it has no concept of a legitimate agent.

How we got here

From voluntary etiquette to ML-based threat scoring: the web's two-decade war on automation, and why legitimate agents look identical to the threats it was built to stop.

The full history runs from Archie in 1990 to the ML-detection era, and it's worth reading on its own. The short version: the web built its security infrastructure in direct response to bad-faith scrapers, and that infrastructure is exceptionally good at what it was designed to do.

robots.txt in 1994 was etiquette, not enforcement. The price comparison wars of the early 2000s ended that. Rule-based WAFs, CAPTCHA farms, JavaScript fingerprinting, behavioural ML. Each layer was built to stop one category of abuse, and each layer also blocked every subsequent category of legitimate automation, because the security industry has never developed a concept of a trusted automated visitor. By 2017, ML-based bot detection services were scoring mouse entropy and TLS fingerprints at the CDN edge. A headless browser on a server is detectable in milliseconds. A legitimate shopping agent looks identical to a hostile scraper. The WAF doesn't know the difference, and nobody ever asked it to learn.

What we actually see - it ain't pretty let us tell you

The largest spirits retailer in the US, serving an ML-based "press and hold" human verification challenge to every non-human visitor. What every AI agent hits before seeing a single product.

We scan retailer pages at scale. Can an agent reach the page? Does it contain structured product data? If a product is there, can it be identified with any confidence?

The pattern that comes back is uncomfortable. I've looked at this data a lot and I still find it somewhat absurd.

The retailers with the best-structured product data are the ones running the most aggressive bot blocking. In spirits alone: a leading UK specialist retailer runs ML-based bot detection at the edge. Another sits behind an enterprise challenge platform. The largest spirits retailer in the US serves a "press and hold" human verification challenge to any visitor that doesn't look like a human. These aren't mid-tier operations with patchy infrastructure. They've invested heavily in their data, and just as heavily in keeping automated traffic out.

Flip it around: retailers without enterprise WAF protection tend to have thinner product pages. Less structured data. Missing fields. Availability information that's stale. These are the pages AI agents can reach. They're also the pages where agents can't reliably confirm what they're looking at.

The identity problem nobody is talking about - new phone, who dis?

Bot accessibility: 30/30. Schema quality: 0/40. Accessible to AI agents, but no Schema.org Product markup means product identity relies entirely on name-matching, and name-matching breaks.

Before any payment conversation, there's a more basic problem: product identity.

GTIN is the barcode system physical retail has run on since 1974. When a logistics network and a warehouse and a point-of-sale terminal need to agree on what a product is, they use it. That problem has been solved for fifty years.

Digital retail never fully adopted it.

Schema.org introduced product markup in 2011, pushed by the major search engines to give crawlers structured data. Retailers adopted it, but they added the fields that moved the needle for SEO: product name, price, availability, star ratings. GTIN is optional in the spec. Shopping platforms use it and reward merchants who include it, but plenty of retailers skip it, particularly outside categories where that commercial pressure has real weight.

So when we scan a product page and look for a GTIN, we frequently don't find one. Not hidden behind authentication. Not blocked by a WAF. Just absent. The product is listed. The price is there. The inventory status is there. The one field that would let an agent say "I found exactly what I was looking for, not just something with a similar name" is missing.

Without GTIN, agents fall back on name matching. "Aged Blanco 750ml" on one retailer. "Aged Blanco Tequila 75cl" on another. Same product. Different string. A name-fuzzy match gets this right most of the time. Not all of the time. In commerce, "most of the time" is the same as broken.

In our scanning data we categorise matches by confidence method:

Across hundreds of pages, gtin_exact matches are rare. Not because the products aren't stocked. Because the data was never built for machines to read.

What brands don't know

Brands have no visibility into any of this.

A spirits brand knows which retailers stock their products. They negotiate listings, set MSRP, run trade marketing. What they don't know is which of those retailers has their products with GTIN in Schema.org markup, which are blocking all AI agent traffic at the edge, and which pages are so structurally bare that an agent would misidentify the product or miss it entirely.

There's something genuinely strange about this gap. These brands spend significant money on retail relationships, trade marketing, and shelf placement analytics. And yet the question "can an AI agent find and correctly identify our products on this retailer's site" has never had an answer. The data hasn't existed.

What we build from scanning gives brands a view they've never had: not just "are we stocked at this retailer" but "what can an AI agent actually do when it gets there." The answer comes back in four states. Blocked. Name-matched. Brand-detected but unconfirmed. GTIN-confirmed and actually transactable.

The debate at the wrong layer

There's an argument going around about llms.txt files and markdown versions of content pages. Add the file, make your content more legible to agents, watch your AI visibility improve. That's the pitch.

Simon Heaton at Buffer ran the test. They published a markdown version of their pricing page specifically for agent audiences. The HTML version gets roughly 6,000 to 7,000 agent visits a week. The markdown file got about a dozen. 201 total citations since launch. Josh Grant ran the same experiment across two dozen sites. SE Ranking looked at 300,000 domains. Nobody found a measurable relationship between the file and appearing in AI answers.

Weekly agent traffic to buffer.com/pricing vs buffer.com/pricing.md across AI indexing, training, and agent categories. The markdown file barely registers. Chart: Simon Heaton, Buffer.

The agents went to the main URL. That makes more sense once you split the problem in two.

We probe WAF status on retailer pages. That's the discoverability question, and it has nothing to do with files in the root directory. The largest spirits retailer in the US fails it completely. Every agent identity we test gets a challenge page before it sees a single product. Nothing in the webroot changes that.

When an agent does get through, it still needs something to read. We score each product page for Schema.org markup, GTIN, price data. "Aged Blanco Tequila 75cl" with no GTIN is technically reachable. But from there it's name-matching, and name-matching breaks often enough to matter.

The markdown file at Buffer got 12 visits. The HTML page got 6,000. The agents went to the main URL, got what they needed or hit a wall, and left. The markdown file wasn't part of that.

The llms.txt conversation is two layers above the actual problem.

So what does this mean

A post went around this week arguing that the hard problem in agentic commerce isn't moving money between agents, it's coordination: verifying what was done, establishing trust, settling outcomes. That framing is right, but it's still missing a step.

Coordination requires identity first. Agents need to agree on what the product is before they can coordinate around it. They need to know what they ordered before they can verify an outcome. The payment and coordination infrastructure being built right now assumes that step is solved. It isn't.

GTIN solves this in physical goods. It always has. Digital retail just never took it seriously, because human shoppers don't need barcodes. They have eyes.

The people building agent payment rails are solving real problems. But they're building on a web that spent two decades getting extremely good at one specific thing: treating automated access as a threat. That infrastructure doesn't disappear because the use case changed. The WAF blocking a scraper in 2004 and the WAF blocking a legitimate shopping agent in 2025 are running the same logic.

Until the security industry develops a concept of trusted automation, and until retailers start treating GTIN as infrastructure rather than an optional SEO field, agent commerce will run on name-matching and guesswork.

That's not a foundation you can build a transaction layer on. And it's the thing nobody in the payment infrastructure conversation seems to have hit yet, because they haven't spent a year watching agents bounce off WAFs and land on pages with no structured data.

We have.