The GrowthMinds Quick Guide to A/B tests

Experiment with A/B testing

 In this guide, you’ll find out:

  • Some ways to decide what to A/B test
  • How to decide if your A/B test is worth doing using statistical significance
  • Plus some great tools that help you to test
  • And some inspiring results.

Why you need A/B testing

Want to make your marketing more effective? Of course you do.

A/B testing helps you find the most effective route – it’s a way to experiment and compare two different approaches before you release the winning approach to your entire audience.

A/B Testing is great for landing pages, ads, email and more: run an experiment, see which approach brings the results, run another, choose the best combinations and use those to drive growth!

Where to start

It’s always a good idea to try different approaches and experiment with your marketing communications.

  • Maybe a landing page with an urgent call-to-action to “Buy today!” drives more sales than “See What’s New”.
  • Or an email subject line with high urgency works better at the end of the day, whereas more informal, chatty subject lines work best at lunch?

But how can you test it?

Well, this guide will show you how, so you can choose all the best elements that work together to really build growth and results.

1. Test one thing at a time

You can do any number of tests, but with A/B testing, just make sure to vary one factor at a time (the thing you’re testing). Keep everything else the same, and then you’ll know what difference that one factor has made, before you move on to the next test.

test examplesFinding which approach works best is worth the effort. True A/B testing always has a knock-on effect – if one call-to-action performs better than another for example, and you roll out the better call-to-action, you’re likely to get a lot more clicks. What if these clicks were adding product to a basket? Or completing a purchase? You could have more sales. That’s the bottom line. A/B testing helps bring results and it can be done on a small budget.

Basic A/B testing can be relatively straightforward to implement. Depending on what system you use for each channel, there are usually ways to A/B test with minimal technical knowledge.


2. Check your tests results are statistically significant

Here’s how:

Imagine you run an email test. You create two emails that are exactly the same, except for the subject line. Each email is sent to 5000 people.

These are the results:


In this test, two different subject lines were used, resulting in a 6% difference in the open rate. The content in the email was the same apart from the subject line and we see that the click thru rates were the same. However, because more people opened email B, the result was 12 more people visiting your site.

But is 12 people enough of the sample size to be a significant result?

Always use a statistical significance calculator to see whether the results are significant.

We love this one: AB Test Calculator


For the test above, visits generated is your conversion in the calculator. There was a 0.8% conversion rate for email B versus 0.56% for email A. That doesn’t sound like a lot, but it’s actually an uplift of 43%.

Running the conversions through the test calculator shows you it’s a statistically significant result and therefore it’s worth using email B’s subject line in the future – email B will drive 43% more traffic to your site than email A.

A/B testing here has delivered a meaningful result which can influence your decision on which email to send out to the wider audience.

Just think of the knock-on effect on your conversions – it’s worth it!

Kate Fairhust, GrowthMinds CEO and co-founder

3. Things to check before you test

Standard A/B testing involves testing the effect of varying one thing (the headline, or the location of a CTA for example). Usually taking a sample of the whole audience for a particular channel, and using the results to inform future activity.

Look at your audience size first.

If the audience is too small, you might not be able to test and get a statistically significant result, or it might not be worth running from a revenue perspective.

For example, if your email list is less than 1000 people, it might not be worth your while running a test to 10% or 100 people, because the difference in results might not be significant. Instead, you might have to go for a 50/50 test to your whole audience. Then as your email list grows, you can reduce the percentage of your audience that receives a test.

Always plan your tests.

Hypothesise what the results might be using current performance where you can.

  • Will the expected results lead to a decent uplift in traffic, leads, orders or revenue?
  • Is it the best test to do first or would another test deliver better results?
  • If the test works, how will you build on it or scale it?

 Don’t forget your budget

A/B tests and experiments don’t have to blow your budget, but when you’re planning, if your test involves paid advertising, make sure to check you have the budget to get the reach you need to prove the results are significant.

And don’t forget to work that into your plan. Maybe you don’t make a return on investment from the experiment. That’s OK. You’re proving a hypothesis. What’s crucial is that it delivers ROI when you scale it. If it doesn’t, it’s probably not the best test to run.


If you’re unsure: give us a call. We’re always happy to help.

And we can definitely help with more sophisticated A/B testing requirements.

4. Tools to help you test

There’s a bunch of tools out there to help your A/B testing. We’ve listed a few here to get you going here for websites, emails and ad campaigns.

For website tests:

These tools give you on-site testing with inbuilt reporting.

For email tests:

Most email systems have inbuilt tools to help you set up A/B tests. Some (like MailChimp) offer simple 50/50 tests, whereas more expensive systems (like Hubspot or Marketo) give you the opportunity to run more complex tests with holdout groups and automated winning email roll outs.

For landing page tests:

  • Unbounce – build and test landing pages, popups and sticky bars
  • Leadpages – optimise landing pages and lead generation campaigns

For ads:

5. Finding what works

You could spend hours messing about with different colours for your Call-to-Action buttons, but you’re probably more likely to see an uplift by changing the wording or focusing your audience on the action you want them to take by simplifying a page.

What works for one company might not work for another.


Take a look at the pages on your site that aren’t converting as well as you’d like and run a test.

Tim Hanson, GrowthMinds Growth Hacker

Here’s one we ran for our client, GeoSpock:

Optimising a landing page with heat map technology

GeoSpock approached GrowthMinds to develop and test some custom landing pages. We developed a new page to test, which featured a video alongside the lead form. Using heatmap technology from Hotjar, we could instantly see the difference.

The images below show the original page (left) and the test page (right) and the heat marks represent where a visitor spends time looking at the page.

Visitors scan across the page.
Visitors focus on the video

The video focused audience attention on the page and drove engagement.

And the results were impressive:

The original page (without the video) converted at 3.63% (below average)  The new page (with the video) converted at 8.27% – a 128% improvement!

GeoSpock are now driving traffic to the new page.

Need some inspiration?

Take a look at these test examples. Maybe some of them can help your business grow.

Or subscribe by email for more guides and inspiration on growth hacking techniques.


Share on FacebookShare on Google+Tweet about this on TwitterShare on LinkedIn

If you liked this guide, maybe you'll also like these: