How we measure AI shopping recommendations

Surfaces and conditionsPilot only

Every observation records:

• Surface and reported model version, when the surface exposes it
• Date and time of observation
• Locale and market
• Account state (signed in or out, memory on or off, prior turns)
• Grounding and citation status, when the surface exposes it
• Whether shopping-specific rendering appeared

Why it matters: the same question asked from two locales, or with memory on versus off, can return different products. An observation without this metadata isn’t a measurement.

Stated limitation: providers can change models and retrieval behaviour without exposing version identifiers. Where a surface does not report its version, we record that fact rather than inferring one.

Journeys, not promptsPilot only

We test multi-turn shopping journeys with stated constraints — budget, compatibility, use case — because that’s how people actually shop in an assistant. A single-turn brand prompt tells you almost nothing about a catalog.

Journey sets are built from your onsite search terms, support tickets and return reasons where available, and from a curated category library where they aren’t. We publish the journey set with every report. If you can’t see the questions, you can’t judge the answer.

Repeated observation and variancePilot only

Assistants are non-deterministic. A product that appears in one run may be absent in the next with no underlying change.

• Every journey is run a fixed number of times per observation window — the exact run count is reported alongside every figure we publish
• We report distributions, not single results
• We predefine a minimum effect of interest and report whether an observed change can be distinguished from ordinary run-to-run variation under the selected comparison design

Product identity and matchingPilot only

Assistants refer to products in ways your catalog doesn’t: shortened names, previous generation names, retailer naming, incorrect variant labels, informal aliases.

We resolve mentions to SKUs through GTIN, MPN, brand and model tokens, variant attributes and an alias set built per catalog. Every resolution carries a confidence score. Low-confidence resolutions are flagged for review rather than counted silently.

Stated limitation: ambiguous variant references are the largest source of error in this category. Where we can’t resolve to a variant with confidence, we report at the parent-product level and say so.

Product truthPilot only

A price or availability claim is only correct or incorrect relative to a specific merchant, market and moment. We hold product truth as a temporal record:

product × variant × merchant × market × timestamp

Every discrepancy we report names its truth source and the timestamp of that truth alongside the observation timestamp. Without both, a legitimate regional price or a valid retailer price looks identical to a hallucination.

Discrepancy classificationPilot only

We classify observed product claims as:

We additionally record whether the discrepancy appeared in the assistant’s explicit explanation, and whether that explanation was used to reject or deprioritise the product.

A note on terminology. We say answer rationale or explicit explanation. What we can inspect is the visible answer text and any justification it contains. We have no access to a model’s internal processing and do not claim any.

What we claim, and what we don’tPilot only

We claim: a discrepancy was observed; it appeared in the answer’s explicit explanation; the product was not recommended in that answer; the association occurred in [n] of [m] runs.

We describe this as: a likely recommendation-affecting discrepancy.

We do not claim: that the discrepancy caused the loss, or that correcting it will recover the recommendation — until a test says so.

We do not attribute revenue to an AI recommendation unless the available exposure, transaction and comparison data support that conclusion. Reports separate directly observed commercial activity from estimated or unmeasured downstream effects.

Comparison design for recovery testsPlanned

A before-and-after comparison alone is not sufficient for causal attribution. Model versions change, competitors change their catalogs, categories move seasonally. Before-and-after observations are evidence — they are simply weak evidence for causality on their own.

Depending on the intervention we use untreated journey groups, untreated markets, comparable catalog subsets, staggered intervention dates, or difference-in-differences against a comparison set. Every recovery report names the design used.

Stated limitation: matched control SKUs are imperfect controls, because products compete inside the same answer. Improving a treated product can directly suppress a control product. Where that risk is material we use untreated journeys or markets rather than untreated SKUs, and we say which.

Result statesPlanned

Every recovery test resolves to one of three states, defined before the test runs:

• Recovered — the predefined outcome improved beyond the decision threshold
• No detected recovery — the observed change was too small or too uncertain to distinguish from ordinary variation
• Inconclusive — the comparison design, sample or concurrent changes prevented interpretation

We do not collapse the second and third. A test that found nothing and a test that could not tell you anything are different results.

UncertaintyPilot only

We report uncertainty ranges rather than point estimates, and we use the term uncertainty range rather than confidence interval unless the underlying procedure is a confidence interval as formally defined.

Corrections

We publish corrections to this methodology and to any published finding. Changelog below, most recent first.

• 2026-07-30 Initial publication. All methods labelled with their current status.