The Conversion Rate Comparability Framework

CONVERSION RATE COMPARABILITY REVIEW — original Growthcraft worksheet
Decision being considered / owner / deadline:
Baseline and comparison periods / time zone / data cutoff:
Source exports and versions:

1. DEFINITION GATE — analyst + instrumentation owner
Session identity and timeout / human-bot filter / consent coverage:
Qualifying conversion / count once per session / observation maturity:
Measurement releases, attribution and segment-definition changes:
Evidence of common definition: pass / unresolved / incompatible
If incompatible, reconstruct comparable data or start a new baseline. Do not approve a performance claim.

2. POPULATION GATE — analyst
Segment A / Segment B: mutually exclusive, exhaustive definitions:
Unknown classifications: retained where? Missing identity exceptions:
Baseline A sessions / converting sessions:
Baseline B sessions / converting sessions:
Comparison A sessions / converting sessions:
Comparison B sessions / converting sessions:
Reconciles to scoped source totals? Evidence ID:

3. DESCRIPTIVE BRIDGE — analyst
Observed rate change (percentage points):
Symmetric traffic-mix component / within-segment component:
Comparison rate at baseline weights (separate reference, not additive):
Both components sum to observed change? Test output:
Small cells / unobserved segments / hidden composition:

4. DECISION HANDOFF — growth owner + reviewer
Permitted claim / prohibited causal interpretation:
Next evidence or controlled experiment / owner / review date:
Spend or product changes require separate approval:
Status: hold for comparable data / descriptive review / separate experiment
Sources, assumptions and unanswered questions:

A rate change is the start of a review

This original Growthcraft workflow is for D2C acquisition leads, consumer-app teams and startup analysts deciding whether a headline conversion-rate movement warrants investigation or a change in strategy. It separates measurement continuity, population reconciliation, arithmetic description and decision authority. It is not a causal identification method or a universal scorecard.

Use it when you have two exports with sessions and sessions that converted, classified into the same two groups. Do not use it to validate a landing-page test after assignment has failed, to compare orders with sessions from another platform, or to explain a change in revenue. Those are different questions requiring different evidence.

Why the definition gate comes first

Shopify's September 21, 2026 update warns that session-based metrics can change without a change in shopper behaviour and calls for a new baseline. That is a concrete reason to inspect definition continuity before describing a rate movement as performance. Shopify session-measurement update. This framework does not claim to reverse or emulate that update.

Inputs and ownership

The analyst owns the count reconciliation and calculation. The instrumentation owner confirms session identity, filtering and conversion rules. The growth owner states the actual decision, such as whether to investigate a channel or propose an experiment. A named reviewer checks claims before a budget or product change. One person can hold multiple roles, but missing responsibility must remain visible.

Bring dated source exports, the query or report configuration, segment definitions, a release log, consent coverage notes and a data-cutoff record. No personal identifiers need to enter the worksheet. Preserve anonymous aggregate counts and evidence IDs; keep raw customer data in its authorized system.

Four gates with explicit stopping conditions

1. Definition: verify the same outcome, denominator, session boundary, bot treatment, coverage and observation maturity. A documented discontinuity blocks a behavioural interpretation. Rebuild a common definition where legitimate data supports it or establish a new baseline. An algebraic adjustment cannot repair missing evidence.
2. Population: assign every eligible session to exactly one group using a stable rule. Confirm group totals equal the scoped population and converting counts are subsets of session counts. Resolve unknown labels explicitly. A zero-session cell has an unobserved rate, not a zero rate; stop this two-group bridge rather than invent one.
3. Bridge: calculate the observed rates, weights and symmetric components with the companion calculator. Confirm components reconcile to the total change. Review counts and cell rates alongside the result; a clean identity is not proof of reliable instrumentation.
4. Handoff: write the narrowest defensible claim, next evidence request and accountable owner. Descriptive arithmetic supports a diagnostic question. It does not automatically authorize a budget shift or demonstrate incremental sales.

No arbitrary pass percentage is assigned. The gates follow logical requirements: a denominator must represent the intended population; a decomposition must reconcile; a causal statement needs more than observational aggregates.

Worked example with a deliberately cautious decision

Synthetic, not client data. Baseline A has 8,000 sessions and 160 converting sessions; B has 2,000 and 200. Comparison A has 2,000 and 60; B has 8,000 and 880. The headline rate rises from 3.6% to 9.4%, a 5.8 percentage-point change. Both group rates rise by one percentage point, while the higher-rate B group becomes a larger share of traffic.

The symmetric bridge assigns 4.8 percentage points to mix and 1.0 to within-group movement. At the old weights, the comparison rate is 4.6%. These are alternate descriptive views of the same counts. If the two periods straddle an unresolved measurement update, the worksheet stays on hold despite correct arithmetic. Request like-for-like exports or establish a new baseline. If continuity is verified, the permitted claim is that mix explains most of this arithmetic movement, not that reallocating budget to B will reproduce it.

Where teams get misled

Dropping an unknown channel can manufacture improvement. Assigning a session to every channel it touched double-counts the denominator. A broad group can hide shifts in country, device, intent or new-versus-returning visitors; a within-group term is not a pure UX effect. Tiny cells can produce unstable rates. Repeated sessions from one person are not independent experimental units. Keep these limitations in the handoff rather than solving them with an invented confidence score.

GA4 documents session identity and explains that its reporting and exported-data calculations can differ. Choose a controlling source and reproduce its definition; do not join numerator and denominator simply because both are labelled sessions. Google Analytics session documentation.