No Longer an Option: What the 2014 State of Online Testing Report Reveals About Businesses Like Yours

There are lots of things that we know we SHOULD be doing, but we just kinda don’t do them. Like getting your oil changed on time, or flossing twice a day. (Y’all know who you are.)

Sometimes, it’s because doing those things doesn’t seem THAT necessary. They kind of fall into that grey area of “it would be nice, but I just can’t right now,” or “I get why other people do it, but it doesn’t seem necessary for ME.”

And while certain things can stay in that zone basically forever, other things may start out there and then become things you really, really NEED to do. They stop being optional, and become absolute musts.

Turns out, online testing is one of those things.

Remember online testing?

Ah yes, TESTING. Remember A/B testing? (I’ve written about it on here before.)

The CliffsNotes version is, this is when you actually have two different versions of a given page on your site go live at the same time. Maybe it’s a landing page, maybe it’s a sales page, whatever.

The pages are similar, except for maybe one key difference – a button is a different color, or the copy is different, or you chose a different photo to use in a certain place. (Sometimes they might be two tooootally different versions of the same page, but that can get more complicated.) Some of the people who visit the page get one version, while everyone else gets the other version. Your testing tool keeps track of who does what on which version of the page, so you can determine which one works the best, and switch to using just that version for everyone.

This is one of those things that admittedly used to be kind of a luxury, because it took time and effort that not everyone can spare when they’re busy running a business. If you could do it, great. If not, c’est la vie.

But now?

Well, it turns out that this kind of testing is quickly going from an option to a necessity – and there’s a lot of data to back that up.

Introducing the State of Online Testing 2014

Not too long ago, WhichTestWon published its second State of Online Testing report. This report breaks down the results of a massive online survey of businesses, which asked questions about their online testing efforts in 2014.

The report includes data regarding testing habits, what these businesses look for in their results, how effective they find testing to actually be, and a whole lot more – and you’re definitely gonna want to know what this data means for you and YOUR business.

So, what’s the skinny on the stuff that matters most? I’ve got three big takeaways for you:

1. Online testing is growing – especially for small business and solopreneurs.

Turns out that online testing isn’t just a fad – in fact, it’s getting more popular by the year.

What are marketers testing?

As you can see, landing pages and emails have been and continue to be the most popular things to test, but errrrvrything gets more popular as time goes on. The chart above only shows the difference between this year and last year, too – if you look further back, you can see that the increase in popularity is actually pretty dramatic.

According to the report, in their 2010 study, only 37% of those surveyed had tested a web page – making for a 78% increase over four years. That’s a LOT of new businesses starting to run tests on their websites.

It isn’t just big businesses with lots of resources who are increasing their testing, either.

Testing in marketing

Notice anything? The two types of organizations whose likelihood of testing have increased the most are those with more than 1000 employees and those with fewer than 20 – the biggest and the smallest, respectively.

What gives?

Well, it comes down to two things. First of all, more big businesses are increasing their testing because it’s become clearer that it works, and is worth the investment (but we’ll get to that later). They have the resources to hire people whose sole job is performing this kind of task – not the worst way to spend your money, if you can afford it!

More importantly, though, businesses with fewer than 20 employees are also becoming likelier to perform tests. Why? Well, that’s the REALLY good news: testing is getting cheaper and easier all the time.

Because testing is growing more popular for everyone, your options for performing online tests are getting more competitive. The technology keeps getting more accessible, so you don’t necessarily NEED a massive investment of time or resources to perform super-valuable online tests. Whether you have a teeny-tiny team or none at all, this is within your grasp. (Nice!)

2. Testing works – IF you test the right things!

So we know that more and more businesses are performing online tests – but what do they think of how useful those tests actually ARE? (After all, it doesn’t matter HOW many people are doing something if none of them think it’s actually helping.)

There are two important statistics to know. First of all, a sort-of-insane 91% of marketers said that testing helped them achieve moderate-to-big results. (The number of marketers who said it wasn’t worth doing? Only .5%.)

Here’s the second big stat to know:

Successful testing in marketing

As of last year, less than half of online sales marketers said that testing was “significantly worth it.” Now? More than 3/4 – that’s a 60% jump! Basically, if you’re considering using testing on things like sales pages and sales emails, more than 3 out of 4 people doing the same exact thing say it’s a darn good idea.

(Why the big change from one year to the next? Could be any number of reasons – people honing their strategies as time goes on, for example. You might even factor in what I mentioned before, about how testing itself is cheaper and easier than it used to be – the figurative and literal cost is lower, which may make more marketers see it as a high-value opportunity.)

Of course, the usefulness of testing depends on WHAT you’re testing, and here’s another helpful piece of wisdom from the crowd:

Marketing testing

This chart may look simple, but it teaches a verrrrrry important lesson: testing conversions can be a LOT more valuable than testing clicks. For example, if you’re testing the efficacy of a sales page, you don’t necessarily want to just test for which version of that page got people to click through to the shopping cart – you want to test which version got people to actually BUY. Clicks are great and all, but unless those clicks add up to something – purchases, sign-ups for your newsletter, whatever – they might not be worth tracking.

3. If you haven’t started yet, you’re not alone. (And it’s never too late to start.)

Plot twist! Not everyone who took the survey about online testing is ACTUALLY performing online testing!

WhichTestWon asked its survey respondents who aren’t testing why not, and if they plan on starting in the future. Here’s what they learned:

Marketing testing

Turns out that the majority of respondents who aren’t testing DO plan to start – they just haven’t gotten around to it yet. Second most popular response? Just don’t have the time! (Together, those two reasons account for more than half of all businesses that aren’t testing.)

The good news for that 56% of non-testing businesses actually takes us back to where we started – online testing is growing in popularity even for small businesses and solopreneurs. While it used to be something that only SOME businesses had the time, money, or other resources to support, it’s increasingly becoming something well within reach of businesses both big AND small.

So, what does all this mean for you?

Like I mentioned back at the beginning – it looks like online testing is quickly going from one of those “it would be nice” things to one of those “you’d better get on this” things. Fortunately, testing is relatively easy, inexpensive, AND effective – and not a lot of things in this world are all three!

What’s your experience with online testing been like? Got any questions? Feedback? Share it in the comments below!

Need more actionable advice?
Get your FREE weekly marketing “to-do” list
straight to your inbox every Wednesday:
About Laura Roeder

Laura Roeder is a social media marketing expert who gives businesses of all sizes the tools they need to make their mark on the web. She is the creator of the social media scheduling software Edgar, as well as social media marketing web courses like Creating Fame and Social Brilliant.


  1. “Online testing is quickly going from one of those ‘it would be nice’ things to one of those ‘you’d better get on this’ things.”

    Ooooh, man, you don’t have to tell me twice; what’s really helpful here is being able to see which kinds of tests made an impact in the end, and which didn’t. One of the main reasons our testing has been so minimal thus far is the uncertainty of ROI, so this will be great to refer back to as we get up and running with A/B content.