A Measurement-First Architecture for AI Shopper Observability

On two live product pages of a well-known footwear brand, our scan came back with a photograph of the page and not a single marker on it.

That is not a failure of the scan. It is the finding. The price, the material, the shipping terms: every fact the model came away with, it had read out of the page's JSON-LD block. None of those characters were rendered anywhere on the page. The store publishes excellent structured data, so the machine was well fed. But the answer an AI shopper would give about those products was assembled entirely from a layer no human customer will ever see, and could not be checked against the page itself.

Most tools would report that as a clean pass. We report it as a picture with nothing on it, because those are different things and confusing them tells a merchant something false about their own store.

This post is about the machine that produces that distinction, what it costs to run, and the three times it lied to us before we got it honest.

What a scan actually is

You give us a store URL and a shopper's question. We open a real browser at your product pages, and drive it the way an AI shopping assistant would: check what the store permits, load the page, read what is legible, follow a lead if the answer is not there, and then answer the shopper.

Then we report what such an agent could and could not establish. Critically, we also report where on the page each answer came from.

One scan covers one store and produces many traces. One trace is one product page: the fields the agent resolved, the answer it produced, a verdict, and a photograph of the page with the position of every fact that was literally written on it.

The budget: fifteen turns, five cents, sixty seconds

The agent runs against three independent ceilings, per page:

Three, and independent, because they fail differently. A model can burn sixty seconds in two turns on a slow page, or fifteen turns in four seconds going in circles. Any one of them can be the binding constraint on a given store.

Hitting a ceiling is not an error. The run ends with what it has, and the trace is still written. A page that ran out of budget and a page that refused us are different findings, and a report that collapses them is a report you cannot act on.

The agent has nine tools: check robots, measure access, read the rendered page, read the fields, answer the shopper, list links, fetch a page, check a fact elsewhere, and finish. There is no fixed sequence. It decides.

Two temperatures, and a line the model cannot cross

Here is the design decision we would defend hardest.

The agent's decisions, meaning which tool to reach for next, run at temperature 0.3. That layer is allowed to vary. A decision layer pinned to zero is just a pipeline with extra round trips.

Every measurement runs at temperature 0 with the tool pinned by name: reading the fields off a page, answering the shopper, checking a fact elsewhere. Comprehension was measured at 5 identical results out of 5 runs under those conditions.

And no tool the agent can call exposes temperature, model, or tool choice. The creative layer physically cannot reach into the measuring layer. That is not a policy we ask a model to respect; it is absent from the schema.

Two guards in the same spirit, both added after a real store found the hole:

• The access check enforces robots.txt itself rather than trusting the agent to have checked first. A rule that holds only when the agent remembers is not a rule, it is a hope with a tool schema.
• finish refuses while the agent still owes the shopper an answer. That one came from a live store producing a trace with a finding and no verdict.

A position is a measurement, not a guess

The comprehension call reads text. It knows nothing about layout. Ask it for coordinates and you get plausible numbers with nothing behind them.

So positions are measured during the render, every visible run of text with its real box, and matched to the model's answers afterwards, with no model involved in the matching. The governing rule:

If the value is not literally on the page, the fact gets no position.

That rule is expensive on purpose. It costs us the marker on every fact the model paraphrased. If the page says "90-day returns" and the model reports "90 days", that is a correct reading and an unlocatable one, and it gets no marker. We eat that cost because the alternative is a cursor pointing at a place where the words are not.

Matching folds case and whitespace and nothing else. Nothing strips punctuation, folds hyphens, or drops currency symbols. Where several boxes contain the value, the tightest one wins: the box a person pointing at the page would point at.

What we will not claim

We measure the first 400 runs of text, down to 4,000 pixels, at a 1280x900 viewport. Real product pages run to 9,000 pixels and beyond.

So a returns policy sitting at y=6,000 on a 9,137-pixel page is genuinely on that page and will still have no marker from us. It is past the clip, and quite possibly past the 400-box limit as well.

This is why nothing in our product ever tells you a word is absent from your page. The provable claim is narrower: a page was measured, and nothing matched among what was measured. Every vendor in this space is one careless sentence away from telling a merchant their page is missing something it plainly contains. We wrote the limit into the copy instead.

The part where we were wrong three times

A scan took about 17.7 seconds before anything appeared on screen, and nobody could say where the time went. What follows is what happened when we stopped guessing and measured.

Guess one: the message broker. That was the standing theory. Measured through a real RabbitMQ: 4 to 5 milliseconds.

Guess two: the first model turn. Our own written plan named it as the prime suspect, on the reasoning that it is the last thing to happen before the first line of output. Measured: 679 to 814 milliseconds.

Both wrong, and wrong in the same way. They were arrived at by subtracting known times from the total and pointing at the gap.

The actual answer was a segment neither document contained. An observability decorator, wrapping the agent's entry point, was capturing the function's arguments as trace input before the body ran. Those arguments include a live browser session and an HTTP client. It walked and serialised them: 9.27 seconds per page, consistently.

It was not the trace exporter. With observability disabled entirely, it was still 9,271 milliseconds. Our monitoring was the single most expensive thing in the scan, and it was invisible precisely because it was the thing we were monitoring with.

Same rig, same store, before and after:

Roughly 46x faster to the first line, and half the total.

Two things we deliberately did not fix. Chromium's first launch in a fresh process is 1,827 ms against 190 ms warm. Real, and an order of magnitude below what mattered, so it stays. And the storefront access check is still 4.1 to 6.9 seconds, because it is genuinely three HTTP round trips and a render against your homepage. Making it shorter is a real piece of work and it deserves its own before-and-after, not a paragraph in someone else's.

Subtraction is not measurement. We had that written down before we started, and still needed to learn it twice on the way through.

The bug that shipped, because it looked right

The worst one was quieter.

Chromium does not raise an error if you ask it to photograph a region taller than the viewport. It just hands back a shorter image. So for a stretch of releases we published a page height of 4,000 pixels next to a JPEG that was 900 pixels tall, for documents that were 9,137 pixels long.

Every coordinate was then scaled into a space the image did not have. Every marker was off by 340%.

It was silent because the output was plausible. A picture of a product page with markers on it looks correct at a glance, and you have to know what the right answer is to notice that the marker is on the wrong paragraph. It shipped in one ticket and stood until another.

Two guards exist now. The backend drops evidence with a missing image or a zero dimension, whole rather than partially, because a picture with no dimensions is as undrawable as a coordinate with no picture. And the frontend compares the JPEG's aspect ratio against the measured space and refuses to draw anything if they disagree by more than 1%. Aspect ratio rather than size, so a capture at a different device pixel ratio still matches.

The fix cost us: median screenshot time 19 ms to 60 ms, and picture size 26 KB to roughly 150 KB. Worth every byte.

The other lesson has a longer tail: every performance figure measured while that bug was live is wrong by the same factor. We had a documented cost-per-picture number that was four times too low for exactly this reason, and it had been quoted in planning documents for months.

Why any of this should matter to you

Because you are going to be shown a report about your store by someone, and the only question that matters is whether the instrument that produced it is honest about its own limits.

Ours will tell you it took no picture. It will tell you a fact was resolved but could not be located, and which of six reasons applies. It distinguishes the store refused us from we reached it and the page failed from our own defect, because a report that blames a merchant for a vendor's bug is worse than no report.

And it will hand you a photograph of your product page with nothing marked on it, and tell you plainly that this means the machine's entire answer came from somewhere your customers cannot see.

That last one is the strongest finding this product produces. We kept it as its own state rather than folding it into "no data", which is what it would have been in almost any other pipeline.

---

Want this run against your own store, on your hardest category? Ask us for a scan. You get the traces, the photographs, and the markers, including the pages where there are none.

---

All performance figures are our own measurements, taken on a single test rig against a live store, and are reported here as measured rather than averaged or projected. Segment timings come from the scan's own instrumentation; where a segment could not be measured, it is not estimated.