These tests often fail because, be it the designer or the analyst, the groups fail to come together. The designer doesn’t want his trade and experience to be questioned by a numbers geek. The analyst doesn’t want some artist telling him how to analyze user experience.

I’d like to see a post about the flip side that shows examples of how designers going with their “instincts” caused thousands of dollars in lost revenue. If you need examples of this, I have seen it many times, I would be more than happy to share.

Its is far too rare when I talk to people about their test results and read published tests, that significance is taken into account. Results are almost always published with no reference to sample size or level of significance. I have to conclude its because it’s not been considered.

I fear testing is going to get a bad name if common practice doesn’t change, as worse performing versions are picked in some cases.

In the A/B example given above the significance is 80%. Weak but worthy of a further test with larger sample to verify or otherwise.

A/B test also suffer from assuming total independence of the elements under test. If you change a page headline too much and not the supporting text and images then its not a fair test. Multi-variate testing should become more widespread to address this.

Another issue often not considered is fairness of test. Are there environmental factors that means the traffic or data in use will have bias?

There’s no such thing as perfect data. Statistics simply doesn’t bend that way. All I care about is good enough data, and the numbers say that your data doesn’t pass that threshold. If that doesn’t concern you, no problem. It’s your site and your opinion, as this is mine.

first of all, I’d have appreciated if you had linked to media.io so people could check out the site instead of two old screenshots.

You are right that the numbers I presented at ABTests.com might not have been statistically relevant.

What is important to me is that I decided to stop the test and implement the changes and that the conversion rate increased (~25 % better if I’m not wrong). And this is what I’m interested in. I don’t care about the theory behind MV tests.

Also, you can search for perfect data as long as you want, but on the web, it’s a futile effort that prevents you from making decisions.

Johann

YES, yes, and yes! I’d also add that another, perhaps even more fundamental, problem with A/B tests is that they presume that there is in fact a single, binary variable to be tested; or that the variable in question is the right one to test.

Here, as you say, is where the designer’s instinct comes into play. Or, perhaps, intuition is a better word — as this is not innate ability, but the thoughtful application of experience. ABtests.com is full of examples of designers asking absolutely the wrong question. Why fret about nuances like the color or position of a button, when typography is so bad, or hierarchy so confusing, that a user might have no idea why to push it?

Great post. I understand what you’re saying and enthusiastically agree with it.

Having said that, I personally feel the much bigger problem is that way too few people are conducting A/B tests or multi-variate tests. I am a huge believer the value of such tests. (This is no doubt influenced by the fact that I grew up listening to rapturous soliloquies on the value of Design of Experiments at the dinner table from my dad, William G. Hunter, one of the co-authors of “Statistics for Experimenters”).

At any rate, if I had the choice of using a website designed by (a) people who used A/B testing (without any concept of statistical significance) or (b) people who designed the website without any use of A/B testing, I would choose “choice a” in a heartbeat.

Having said that, if I had a chance to speak to the designers running the A/B tests, would I point out the significant issue of statistical significance to ensure they didn’t draw conclusions too quickly from random variation? Sure, I’d probably even point them to this article.

I’ve written a blog post summarizing the best web video presentation I’ve seen on A/B testing and multi-variate testing (given by Kohavi). If anyone is dealing with skeptics at their place of work who do not feel A/B testing is worth pursuing, I’d recommend it as a fantastic source of useful, eye-opening examples. http://hexawise.wordpress.com/2009/08/18/learning-using-controlled-experiments-for-software-solutions/

- Justin