Skip to main content

​
At a glance

Automation split test report: contacts, buyers, net revenue per contact and recovery rate for each call center arm What it does: a Split Test step in an automation sends each contact down one of its arms. Every step becomes a split test of its own on the Split Tests page, next to your page split tests, and each arm gets its own results. Typical use: send half of your abandoned checkouts to one call center and half to another (two Send to call center steps after the split), then keep the one that brings in more money after refunds. Main metric: net revenue per contact, which is the revenue from the contacts in an arm, minus their refunds and chargebacks, divided by the number of contacts in that arm.

​
Set it up

  1. Open Automations and edit the automation (or create one with a trigger such as On Abandoned Cart).
  2. Add a Split Test step from the General group of steps.
  3. Add an arm for each path and give it a weight. Weights are percentages of contacts, so two arms at 50 split contacts evenly.
  4. Connect each arm to the steps that path should run.
  5. Save the automation.
Saving creates the split test. It shows up on Split Tests with the type Automation, named after the automation and the step. Click View results on the step in the builder to open its report.
An arm is identified by its name. Renaming an arm starts a new arm with no results; the old one stays in the report marked “(removed)” so its numbers are not lost.

​
How contacts are split

  • One arm per contact. A contact is recognised by email, or by phone when there is no email. When the same contact enters the step again (a second abandon, for example), it goes down the same arm as before.
  • Weights apply to new contacts. Contacts already in the test keep their arm when you change the weights.
  • Paused test: every contact goes down the first arm and nothing is counted.
  • Winner declared: every contact goes down the winning arm.
  • Step deleted: the split test is marked deleted and keeps its results.
  • Test runs from the builder and orders from a merchant in test mode are never counted.

​
How results are counted

An order counts for an arm when the contact places it within the attribution window after entering the step:
  • Every order in the window counts, upsells included. The order that started the automation does not.
  • Refunds and chargebacks of a counted order are taken off that arm’s revenue when they arrive.
  • A contact counts in its arm even when nobody reached them. That keeps the arms comparable: an arm that only counted the contacts it reached would look better than it is.

​
Reading the report

Recovery rate is shown next to net revenue per contact on purpose. An arm that wins more orders but loses more of them to refunds can have the higher recovery rate and still bring in less money.