Wednesday, January 24, 2007

A/B vs multivariate testing

Referring to Wikipedia, multivariate statistical analysis, or multivariate testing (MVT) "describes a collection of procedures which involve observation and analysis of more than one statistical variable at a time". The key is "more than one".

A/B testing generally aims to compare various versions of one statistical element. In it's simplest incarnation, and what most web analytics solutions offers out of the box, A/B testing is based on two time frames.

When we refer to "4 different version's of a website homepage", the element of comparison is the whole page. What if instead we used each sub-elements such as graphics, layouts and text to evaluate the best combination of all those elements to get the winning home page? The first example can't tell if one element of a page is better than another, it tells us if the whole page is better. With the second example, we could find that "text A" combined with "layout B" and "graphics C" is, in fact, much better than "text A" with "layout A" and "graphic A".

Another important difference between A/B and multivariate testing is that MVT provides quantitative data, while A/B testing often a branch of usability testing that also encompass qualitative data. Both quantitative and qualitative information are important to take the best decisions.

Finally, when you want to test two variations of an element, you end up with two tests. Fairly easy! If you want to test three variables (text, layout, graphics), and say each one as two variations, you end with 3^2 tests to do. This quickly gets out of hand as the number of possibilities increases. MVT statistical methods use a smaller number of tests and uses inference analysis to find out which combination of individual elements is "optimal."

The demos at Offermatica are pretty good at simplifying the differences between A/B and multivariate.