Unsure whether “Your serum is back” or a benefit-led title works better? An A/B test compares versions across separate recipient groups. Change one thing so the result is interpretable.
Set up the comparison
- Open A/B tests in the Web Push installation and create a test.
- Prepare a base message and at least two variants with recognizable names.
- Change only the element under test, such as the title. Keep the offer and destination the same.
- Set the test audience share, variant weights, winner criterion and minimum sample.
- Review the summary before starting. Starting sends variants to the selected sample.
Let the test collect enough evidence
The sample receives separate variants. Weights determine each variant's share; equal weights mean equal shares. A small number of deliveries or clicks may not support a reliable decision.
Do not declare a winner after the first click. Check the minimum sample and required confidence. Automatic selection does not guarantee a conclusive result for every test.
Example of the audience split
With 1000 active subscriptions and a 20% test, 200 subscriptions enter the test. Two equally weighted versions get about 100 each. The remaining group is outside that initial sample. This illustrates the split, not guaranteed deliveries.
Minimum deliveries per variant does not mean the size of the entire database. A small sample or unsuccessful deliveries may leave the threshold unmet. With a small list, reviewing one ordinary campaign can be more useful than forcing a winner.
After the result
If sending the winner to remaining recipients, do not also send a separate campaign to the whole list. Sample recipients should not receive another version merely because the test ended.
Review results and record a finding about the tested element, not a general promise of higher sales.
Read the first results
Variant A has one click, but three deliveries per variant are not enough to call it better. This example requires 100 deliveries per variant and has automatic winner selection turned off. Check the sample and completeness of the data before ending the test and choosing a version.

