Tracking Migration: Why Matching Conversion Totals Is Not Enough

Your old purchase collector reports 90 conversions. The replacement reports 90. It looks like a clean migration until you join the event identities: only 80 purchases appear in both. Ten disappeared from the new route and ten different purchases appeared. The totals cancel; the operational problems do not.

This guide is for ecommerce growth teams, consumer subscription operators and startup engineers replacing a tracking integration. It builds a bounded, reproducible evidence packet before a cutover. The method compares observed presence within an independently defined eligible reference. It does not estimate causal conversion lift, certify a platform integration or authorize processing without permission. All worked data is synthetic.

Four checks before trusting the new total

• Define the business event, eligible reference and arrival cutoff before comparing collectors.
• Join identities and retain both one-sided groups; a net zero change can conceal substantial disagreement.
• Check duplicates, payload values and destination processing separately from presence.
• Make unresolved evidence and rollback ownership explicit; do not turn a descriptive percentage into an automatic release decision.

Use the migration review framework to assign owners, the paired coverage calculator for aggregate arithmetic and the evidence-review prompt to challenge the proposed conclusion. The workflow extends marketing data contracts with a concrete old-versus-new identity comparison.

Why this matters during the current Shopify transition

Shopify's August 24, 2026 changelog states that script-tag creation and updates stop working from October 1 across API versions. Existing storefront script tags continue running until Shopify stops injecting them on March 1, 2027. These are different deadlines: the October restriction does not mean every existing integration stopped collecting that day. Source: Shopify's script-tag deprecation announcement.

Shopify directs analytics-only or conversion-only scripts toward web pixels, while other script-tag functionality may require app embed blocks. Its Web Pixels API documents controlled browser interfaces and customer-event subscriptions inside sandboxes. A replacement is therefore not simply an old script pasted into a different box. Source: Web Pixels API. Confirm the actual app's migration path with its vendor; this article does not modify a Shopify store.

The transition is a current engineering reason to review measurement continuity, not proof of keyword volume or a claim that all merchants use deprecated script tags. The same paired-reference method is useful for other collector replacements, but platform event semantics, identity and consent behavior must be checked independently.

Define what both pipelines should observe

Start with a business decision: can a specific downstream report continue supporting acquisition or lifecycle decisions after a named collection route changes? Write down which event and destination matter. Do not compare an old paid-order event with a new checkout-completed browser callback and assume the labels describe identical behavior.

The unit might be one eligible purchase, identified by an internal transaction key. Define cancellation, test-order, retry and multi-currency rules. A browser callback can fail even when a backend order exists; a backend record can also represent an event the browser collector was never expected or permitted to capture. The reference must reflect the agreed scope rather than every transaction in the company.

Use an independent source to enumerate that eligible reference. Record its extraction version, event-time interval, timezone and late-arrival cutoff. A half-open interval includes the start and excludes the end, which avoids double counting a boundary event across adjacent exports. Freeze a settled snapshot before comparison. A new pipeline with a shorter observation delay will otherwise look artificially worse.

Keep eligibility unknowns separate. If permission or routing evidence is absent, do not silently classify the event as expected and missing, or exclude it merely to improve coverage. Ask the responsible owner to resolve the boundary. Never use measurement parity as a justification for circumventing consent or capturing more personal information.

Build the four-cell presence ledger

For every unique reference identity, record two Boolean values: observed by old, observed by new. That produces four mutually exclusive cells: both, old-only, new-only and neither. Let their counts be B, O, N and Z. The fixed reference total is T = B + O + N + Z.

The old collector covers B + O reference events; the new collector covers B + N. Their coverage rates divide these counts by T. The coverage change is 100 × (N − O) / T percentage points. The net change deliberately cancels gains and losses, so always show O and N alongside it.

Discordant events equal O + N. Their rate is (O + N) / T. Shared/union overlap, often called Jaccard overlap, is B / (B + O + N). It asks how much of the observed union is shared, excluding joint omissions. Agreement including joint absences is (B + Z) / T. Agreement and coverage answer different questions.

If the reference is built from the union of the two collectors, Z is unknowable. A zero neither count then follows from construction, not successful tracking. Similarly, an empty reference yields undefined coverage, not zero-percent failure or perfect accuracy. The companion calculator returns null for zero-denominator ratios and labels that state in the interface.

Work through the equal-total trap

Synthetic example: 100 eligible purchase events have completed the same observation window. Both collectors see 80; old-only contains 10; new-only contains 10; neither contains zero. Old and new each report 90 reference events. Both coverage rates are 90%, giving zero percentage-point change, but 20% of the reference disagrees and shared/union overlap is only 80%.

Presence group	Count	First investigation
Both	80	Compare payload and duplicate evidence
Old only	10	New-route gaps, identity mapping or late arrivals
New only	10	Old-route gaps, changed semantics or eligibility
Neither	0	Check independent reference completeness

None of the one-sided group labels identifies the cause. A new-only observation may be a useful recovery, a different event definition or a faulty identity translation. An old-only observation may be a genuine regression or evidence that the old route collected something it should not. Root-cause review determines the interpretation.

Now add 900 eligible events that neither collector observes. Coverage falls to 9% for each, but agreement including joint absences rises to 98%. Shared/union overlap remains 80%. Reporting only agreement would reward a common failure mode. This counterexample belongs in the review packet, not just the test suite.

Run a local identity reconciliation

The following dependency-free JavaScript runs in Node.js and uses synthetic string keys. It accepts a unique reference and raw old/new observation arrays. It records duplicate extras before converting observations into sets, and lists outside-reference identities separately. Do not upload production identifiers to the public calculator or a general-purpose assistant.

function reconcile(reference, oldRows, newRows) {
  const valid = xs => Array.isArray(xs) && xs.length <= 1000000
    && xs.every(x => typeof x === 'string' && x.length > 0 && x.trim() === x);
  if (![reference, oldRows, newRows].every(valid)) throw new Error('Invalid identities');
  const ref = new Set(reference), old = new Set(oldRows), next = new Set(newRows);
  if (ref.size !== reference.length) throw new Error('Reference must be unique');
  const groups = {both: [], oldOnly: [], newOnly: [], neither: []};
  for (const id of ref) {
    const key = old.has(id) ? (next.has(id) ? 'both' : 'oldOnly')
      : (next.has(id) ? 'newOnly' : 'neither');
    groups[key].push(id);
  }
  return {
    groups,
    counts: Object.fromEntries(Object.entries(groups).map(([k,v]) => [k,v.length])),
    duplicateExtras: {old: oldRows.length-old.size, next: newRows.length-next.size},
    outside: {old: [...old].filter(id => !ref.has(id)),
      next: [...next].filter(id => !ref.has(id))}
  };
}
const ids = Array.from({length: 100}, (_,i) => 'synthetic-'+i);
const oldRows = ids.slice(0,90);
const newRows = [...ids.slice(0,80), ...ids.slice(90)];
console.log(reconcile(ids, oldRows, newRows).counts);
// {both:80, oldOnly:10, newOnly:10, neither:0}

This reference implementation intentionally treats identifiers as case-sensitive and refuses leading or trailing whitespace instead of silently normalizing it. A real pipeline needs an explicit canonical key contract. Changing case, trimming prefixes or joining by email can merge distinct events. Solve that upstream and version the mapping; the code cannot infer business identity.

Duplicate extras count rows beyond the first occurrence of each key across the whole supplied export. They are not the number of affected customers or the amount of duplicated revenue. Outside-reference observations are unique keys and need their own eligibility and timing investigation. Neither category is folded into the four reference cells.

Presence is only one layer of correctness

For matched events, compare required fields, value, currency, event type and business timestamp. Use declared rounding and currency conventions rather than a broad percentage tolerance that conceals wrong units. A purchase sent as cents in one route and currency units in another can pass presence reconciliation while corrupting revenue reporting.

Separate collection, transport acceptance, destination processing and reporting. Google's Measurement Protocol documentation explains that malformed events do not necessarily produce HTTP error codes and recommends validation before production. It also states that validation-server events do not appear in reports. A successful validation request is therefore not evidence of report inclusion. Source: Google event validation documentation, updated September 30, 2026.

Test vendor-specific deduplication deliberately in an approved test environment. The local set operation removes repeated identities only for analysis; it does not remove duplicates from an analytics destination. Do not dual-fire two live purchase collectors into production and assume a shared identifier guarantees deduplication across every platform.

Review material segments such as checkout route, device family and consent path when those attributes are already appropriately available. Aggregate coverage can hide a complete loss in a smaller route. Keep small segments descriptive; a handful of observations does not justify a precise population failure-rate estimate.

Validate the arithmetic and the review logic

Use the known 80/10/10/0 fixture, then deliberately break its comforting headline. Swapping old and new must swap coverage and reverse the net change without changing discordance or overlap. Multiplying every cell by the same positive integer must preserve rates. Adding joint absences must reduce coverage while leaving shared/union overlap unchanged.

Test an empty reference, all jointly missing events, complete overlap and wholly disjoint observations. Reject blank, negative, fractional, non-finite and out-of-range count inputs. The interactive tool accepts at most one million per cell, which keeps counts exact in JavaScript; this is an implementation limit, not a benchmark or recommended sample size.

For the local join, verify duplicate reference rejection, unchanged cell counts when an observation row repeats, and unchanged cells when an outside-reference key is added. The separate duplicate or outside counter must change. These tests catch mistakes that a calculator-only unit test cannot, because aggregation can be correct while identity preparation is wrong.

Create a cutover packet, not a green badge

Agree business-specific tolerances and hold conditions before reviewing results. Explain why a condition matters: an unverified consent boundary, missing purchase currency or unsupported checkout path can block approval even with high aggregate coverage. There is no universal overlap percentage that certifies a safe migration.

Assign each unresolved finding an owner, required evidence and review time. Preserve the old and new configuration versions, destination mapping and last known-good state. The release owner needs a rollback trigger and a way to confirm rollback worked. The evidence-review prompt can organize this packet, but must not approve deployment or modify an account.

After an approved cutover, continue bounded monitoring at a consistent maturity window. Distinguish a collector change from campaign performance: fewer observed conversions can reflect a changed measurement system, while more observed conversions do not prove incremental sales. Record the measurement break in reporting rather than rewriting historical comparisons as if nothing changed.

Start with the review worksheet, reproduce the example in the coverage calculator, then use the prompt to surface missing evidence. For hands-on help defining the contract and implementation scope, explore consumer growth consulting. This article is AI-assisted editorial work with synthetic examples; it does not claim a client migration or Akshay's personal review.