Testing for optimization has been an essential scientific method, but it has its own requisites that have to be taken care of. Testing two variations can solve many confusions in the form of A/B Testing But there is also the need of testing website properties by varying more than two components at a time, which is referred to as Multivariate Testing or MVT. It is also worth noting that it is simpler to run an A/B Test compared to run an MVT. The ease of setting up an A/B Test makes it a better choice to test major variation changes which can be limited to the variation of two types. In fact, testing of more than two largely disparate variations is also generally referred to as A/B Tests, though they happen to be A/B/“n” Tests. These would be applicable if all the variations are significantly different from each other
MVT is majorly useful if there are multiple options of variations possible on a page, within a page and conversion flow or within a navigation flow. The primary advantage of running an MVT compared to an A/B test is the potential to test more than two variations at a time. However, this comes at the cost of the duration until when an MVT has to be run; also making multiple versions requires a lot more investment in terms of time and creativity. And the worst case scenario may be that your test results into an inconclusive winner after all the time, money and effort invested.
So what impacts the duration of an MVT? Here are two elemental factors
- Traffic Volume – a medium to low volume traffic can suffer a lot in arriving at statistically significant results out of an MVT
- Innumerable Test combinations – opt to test those combination variations which can have an impact on the goals of the test
In short, more the number of variables that are going to be tested in an MVT, more data will be required to infer any results out of the myriad of variable combinations. In other words, more data indicates, a higher volume of traffic would be required to display the various combinations of the multiple variables in the test. For the part of statistical significance, ideally a 99% level of confidence is maintained and that means a good chunk of data. For example, if there are 6 distinct variables that are chosen for an MVT and each variable have 2 versions, then there have to be 26 combinations that have to be tested without bias, i.e.
26 = 2x2x2x2x2x2 = 64 combinations
There have to be sufficient visitors who visits the 64 variations of this test and finally arrive at statistically significant results to be able to declare the winning combination. Also making multiple versions requires a lot more investment in terms of time and creativity.
Since knowing the time estimates becomes very important, there are testing solutions that have come up with tools by employing the concept of statistical significance for this purpose. Tools like such help find out the optimum time till when a test can be expected to be run based on confidence level and goal conversion expectations. One of the examples is the Split Test duration calculator by Visual Web Optimizer. This tool calculates an estimate of the time required to run a test on a web page based on certain input parameters. For example, if the existing conversion rate is 3% and we want to take it to 4% with just 6 variations; channelizing even 50% of the average number of daily visitors of 200,000, it would still take about 310 days to come to a conclusion. For a business, investing such a long time period may not be feasible. Hence, a pre-calculated method of estimating the number of days based on the test and variations to be tested would be very handy.
Following are the methods of running an MVTs and the how time can affect these methods:
(a) Full Factorial Design: This gives equal weight to all the possible combinations without making any assumptions.
In this method, (say) there are ‘n’ combinations of the same page
Then the total traffic is divided into groups of ‘1/n’ and channelized through each variation.
This gives a detailed understanding of how the factors impact. But as said above, more the number of combinations, more would be the time required to complete the test, depending upon the volume of visitors. In certain cases, MVTs may take 6 months or more to complete and it may well be that by the time the test is over, the scenarios would have change. Results in such a situation would have no use at all.
(b) Partial/Fractional Factorial Design: In this method, a selective choice of the combinations to be tested out are made; i.e. (say) there are ‘n’ combinations
Then ‘n-k’ (n>k) combinations would be tested, while, for the untested ones, assumptions would be drawn from the tested ones.
This method requires less time to be tested as the combinations are less but is less reliable compared to a Full Factorial Design because of the many assumptions that are made.
(c) Taguchi Method of Testing: This method is named after Genichi Taguchi, which belongs to the genre of Fractional Design. It uses statistical theory to shortlist and prioritize the combinations that need to be tested out. Though widely used, there is significant controversy about the theoretical stronghold of this method. One of its disadvantages is that it may cause less efficient combinations to be wrongly declared as winners. Hence, it is recommended by most to stay away from this testing method.
Any workaround / Solution / Advice
With the various methods of testing and unsure nature of those methods which can take shorter time compared to another, it becomes very important to choose the right method for the right scenario. Many organizations have adopted performing Multivariate testing and found certain workarounds.
One of the best-fit solutions is provided in certain testing tools itself – Auto Segmentation. But how does it help? Auto Segmentation can reduce the time required for a test significantly by automatically understanding the visitor behavior and thereby reduce the overall time required to analyze and arrive at statistically significant results. Segmenting the visitor data helps to segregate useful sources from the unrequired segments; for example, between actual leads against ones who have no intent of conversion, concentrate on channels that have higher priority or geo locations that make more impact on your business. This will save resources from running blindly on a whole gamut of the heterogeneous dataset.
If your site doesn’t get a high volume of traffic, don’t worry about running an MVT. A better and systematic planning of the required test can be executed in the form of an A/B test, which can provide valuable data, even if with the constraints.
As part of your Multivariate Testing process, are you facing any of these or other such challenges? If yes, don’t forget to contact the Nabler Testing & Optimization team.