That’s the most popular question we have faced during our Testing and Optimization efforts with our clients. This simple sounding but very important question asks a lot of itself. And the answer is that the duration depends upon the achievement of goals of the test and also upon the confidence level attained so far.
Let me put forward a simple question – What are the odds that a new visitor to a website would land on the Home Page first? Furthermore, what are the odds that a returning visitor to the website enters the site on the Home Page? Well, it’s 50-50.
Depending upon the nature of the business and website type, approximately 85-90% traffic to E-commerce and lead generation websites begins from the Home Page. Banking websites, though, have a lower traffic originating at the Home Page compared to other customer login pages. In whichever case, the Home Page gets high respect, both by humans and automated crawlers. Hence, it’s no surprise that conducting a test on the Home Page is a major decision.
Any page optimization process requires an in-depth analysis of visitor behavior on that page and paths traversed around that page. Before deciding the duration of running a test, one should keep in mind that optimizing the website Home Page by running an A/B test needs additional parameters to keep note of, compared to, say, Landing Page Optimization. Here is a list of some of the parameters to consider before running a test, which can help in deciding the duration of a test:
1. Visitor Intent – The website Home Page is seen by a variety of audience, either by intent or by chance. The visitors can be grouped into target audience, existing customers, new prospects, or random obsolete visitors. Identifying goals of this heterogeneous mixture of visitors gets more complicated. The non-conformity is multiplied when the traffic influx spikes.
2. Content Type – Home Page, being a major real estate of the website, provides only a limited space for sites with an extensive content to showcase. The hero banner has become a premium space for which several departments have fought aggressively. To resolve the conflict, businesses have played around with a dynamic Home Page to display more than one hero banner in a round robin fashion. But the problem still persists as CTRs don’t seem to improve and CPA gets marginally increased even further.
3. Content Seasonality – This is not applicable to all websites, but in majority of the sites such as E-commerce and Content sites, merchandise and content finds popularity during certain seasonal times compared to other times. For instance, during the holiday season which spans across Thanksgiving, Black Friday, Cyber Monday up to Christmas and New Year, there are sites which get 80% more traffic than normal times; E-commerce, Travel, Hotel & Hospitality, Content and Comparison Engines are some website types that fall in this category.
One of the major requests by site owners has been to improve engagement from the website Home Page and thereby bring down the Home Page bounce rate. A test is devised after considering the above factors. Additionally, the longevity of a test also depends upon –
- Monitoring visitor behavior based on the pre-defined segments
- Targeting the test on desired segments
- Percentage of visitors and statistical significance of the test variations
- Number of alternate content experiences and percentage of visitors’ exposure to the content
- Objective of the test strategy
- The currently prevailing conversion rate and desired conversion rate
In simple words, how long a test should be run is a function of the six factors listed above. To make it simpler, there are online calculators available to make an estimation of the time required. However, since every website is unique in its own way, use the calculators as a guideline and observe how the various parameters weigh out.
In our experience, running an A/B test takes lesser time when compared to a multivariate tests (MVT) which is of notoriously long duration. Websites with approximately million visitors or more are better off running a MVT. Among the different types of MVT’s, running a partial factorial multivariate tests could be easier compared to a full factorial multivariate tests. It serves as an added advantage to run a test on the Home Page of a site since it enjoys a lion’s share of the site’s traffic. But do not forget that participation of the Home Page in conversion plays a crucial role.
So, next time when this question arises, remember to look back at the Visitor Intent, Content Type, and the Content Seasonality and then focus on the six deciding factors.