Skip to content
Documentation
DropUI homepage
Topics

Compare two Web Push versions with an A/B test

Unsure whether “Your serum is back” or a benefit-led title works better? An A/B test compares versions across separate recipient groups. Change one thing so the result is interpretable.

Set up the comparison

  1. Open A/B tests in the Web Push installation and create a test.
  2. Prepare a base message and at least two variants with recognizable names.
  3. Change only the element under test, such as the title. Keep the offer and destination the same.
  4. Set the test audience share, variant weights, winner criterion and minimum sample.
  5. Review the summary before starting. Starting sends variants to the selected sample.
Two title variants in an A/B test draft; this example contains no campaign results.

Let the test collect enough evidence

The sample receives separate variants. Weights determine each variant's share; equal weights mean equal shares. A small number of deliveries or clicks may not support a reliable decision.

Do not declare a winner after the first click. Check the minimum sample and required confidence. Automatic selection does not guarantee a conclusive result for every test.

Example of the audience split

With 1000 active subscriptions and a 20% test, 200 subscriptions enter the test. Two equally weighted versions get about 100 each. The remaining group is outside that initial sample. This illustrates the split, not guaranteed deliveries.

Minimum deliveries per variant does not mean the size of the entire database. A small sample or unsuccessful deliveries may leave the threshold unmet. With a small list, reviewing one ordinary campaign can be more useful than forcing a winner.

A/B test settings for the sample, metric and minimum deliveries.

After the result

If sending the winner to remaining recipients, do not also send a separate campaign to the whole list. Sample recipients should not receive another version merely because the test ended.

Review results and record a finding about the tested element, not a general promise of higher sales.

Read the first results

Demo test in progress: 3 deliveries per variant, one click for A and none for B. The minimum sample is 100; no winner has been selected.

Variant A has one click, but three deliveries per variant are not enough to call it better. This example requires 100 deliveries per variant and has automatic winner selection turned off. Check the sample and completeness of the data before ending the test and choosing a version.

Have questions after reading this guide?

Get in touch. We’ll answer your questions and help you with setup, free of charge.

Patryk Surmacz

I’ll walk you through the settings, step by step.