7 Steps to Formulate a Strong A/B Testing Hypothesis

Hypothesis is defined as “A tentative assumption made in order to draw out and test its logical or empirical consequences.” When it comes to landing page optimization, the test hypothesis is the assumption you want to test against what you already have. In order to increase the probability of success of your A/B tests, you need to form a strong hypothesis through researching and observation.

Expert Marketers don’t go about testing random ideas hoping to stumble upon a working solution. Instead, they use solid test hypothesis.

A hypothesis provides a solid and targeted A/B testing roadmap that will generate lifts and provide vital lessons. Thus, every test should start with a hypothesis that you are trying to prove or disapprove. So formulating a hypothesis is the quicker, more accurate and less expensive way of solving a problem.

Here are 7 steps to take to formulate a strong A/B testing hypothesis

1. Define your problem

 Defining your problem is the first thing that needs to be done. What is it that you want to test or solve? Is it to double your sales or to increase the number of opt-ins? Whatever your goals are, they need to be clearly defined, quantifiable, and measurable. This should give you a clear idea of what your new design should solve including the process that will be followed to achieve the results.

2. Find out the reasons behind the numbers

Now that you have defined your problem and you have a clear picture of what it is you want to achieve, the next thing that follows is an in-depth statistical analysis of the current problem. This can be equated to sharpening the ax. Basically, you want to take as much time as possible to learn the reasons behind your numbers.

You won’t be able to form an accurate hypothesis without studying what is happening in the website where you want to test your A/B test. Now that you are already looking for better variables to improve your conversion rates, it is only logical that you find the reasons that brought you to this current situation. Why are you experiencing high bounce rate? Why aren’t you seeing more conversions? Why are most of your customers failing to complete the payment process? These are obviously some of the reasons that may push you to improve your website.

The only way to discover areas of improvement in your website is to study your target market. You also need to get customer feedback through comments, social media, surveys and email.

3. Talk to your visitors

It is important to get real feedback from your visitors. One way is to use surveys—both entry surveys and exit surveys that are used to discover your visitor’s objectives and determine whether their goals have been met respectively this is aimed at understanding what they want or what their desires are.

Knowing the reasons behind their decisions and actions is the most important part of the survey. Therefore, do not hesitate to ask them to give reasons for their actions in the survey. For instance, you can place an exit survey at the end of a buying process to ask them why they bought your product. You could also place an exit survey immediately they abandoning a buying process to understand why they did so.

You could also use analytics tools to gather quantitative data such as location, devices used, bounce rates and number of visitors and so on. In other words, both surveys and analytics tools can complement each other when it comes to gathering information about the customer.

4. Use segmentation to get actionable data

In statistics, averages don’t tell you the whole story. Segments do, and that is why segmentation is an important step in the formulation of hypothesis. For example, an experiment may show that a certain product is not performing well, but upon further analysis, it may be discovered that majority of people who buy the product are women aged between 18 and 29 years. Upon further investigation, it may turn out that ads for the product were being targeted to the general population.  So when you do segmentation, it may eventually occur to you that you should concentrate your marketing efforts on the women who fall in the 18-29 age bracket. In other words, segmentation gives you actionable data, which would otherwise be useless without it. There are many approaches to segmentation. Examples include:

Source segmentation: involve separating the visitors who come to your site based on their sources e.g desktop or mobile, android or iOS, email or social media. Find all the metrics associated with all these segments such as bounce rates, number of visitors, conversion rates and regions.

Behavioral segmentation : See what elements your customers focus most in your website. These days there are eye-tracking technologies that make it possible to study the elements that attract the most attention. Find out who are the 20% of your segment that bring the 80% of revenue (refer to Pareto principle). Another way one can gather data is through usability testing where one observes the customer behavior and records the observations.

Outcome segmentation: here, you focus on different types of products that have been purchased, the number of people who did not complete purchase, where most of your orders are coming from etc.

In short, the aim of segmentation is to find out where your most profitable segment comes from and focus your efforts on that segment.

5. Articulate a Hypothesis for your test

Now that you have gathered enough evidence to show what or where the problems is, it is time to state why you think the problem occurs.

Your hypothesis should have the following characteristics:

• It is goal oriented—it clearly states what needs to be accomplished
• It can be tested—it can easily be implemented
• It is insightful—looking at the hypothesis, one should learn something about the problem.

An example of hypothesis

Problem: less than 5% of visitors buy the mobile app

Hypothesis: The text in the CTA button does not provide a clear message to the customer. The text needs to be changed from “Get it” to “Download you app now”.

6. Test substantial variations based on your Hypothesis

We can call this the brainstorming stage. After determining the problem and articulating a hypothesis. The next thing that follows is coming up with substantial variations based on your hypothesis.  Taking the above example, the hypothesis states that “The text in the CTA button does not provide a clear message to the customer.” The substantial variations could include things like changing the color of the button, changing position of the CTA on the landing page, changing the wordings, creating different icon etc. The substantial variations of your hypothesis are meant to bring you closer to the solution as quickly as possible and provide you with insights.

7. Analyze results to validate your hypothesis and Repeat

Once you’ve managed to articulate your hypotheses and test substantial variations, it’s time to analyze results to validate your hypothesis. You need to have sufficient test results in order to analyze and compare. When you are analyzing your tests with the aim of implementing solutions, you should bear in mind that revenue is the ultimate measurement of improvement. Customer feedback and analytics are tools you can use. You should look at the data your customers have left to help you choose the elements that need to be analyzed. The various elements you could test include:

• CTAs—colors, texts, size
• Images—placement size
• Headlines—size, length, style, tone text color
• Testimonials—placement, number, length
• Videos—number, with or without videos
• Forms—files type, color, number of fields
• Shopping cart—icon, text, number of steps
• Copywriting—long text or short, style, tone

After learning from your results, you should start the process all over because there is always room for improvement. In marketing, you never hit the perfect solution and even there is no way to know that you have reached perfection. Constant improvement is the name of the game in this field. So, it is a continuous process.

About the Author:

Steven Sanchez has been actively involved in SEO and Internet marketing since 1999. Steven’s knowledge and experience have made him one of the most respected and referenced SEO’s in the industry and his passion for innovation and growth has led his company, Internet Marketing Invesp, to become one of the world’s leading online marketing firms.

FAQ

What is an example of a hypothesis for AB testing?

In the context of A/B testing, a hypothesis is an educated guess about what changes will improve a particular metric on your website. For instance, you might hypothesize that “Changing the call-to-action button from green to red on our product page will increase the click-through rate by 15%.” This hypothesis is based on your understanding of visitor behavior, data science, and the performance of your existing site. The effectiveness of the hypothesis is then tested via A/B or split testing.

What is null and alternative hypothesis in AB testing?

In A/B testing, the null hypothesis states that there is no significant difference between the performance of version A (the control) and version B (the variant). For example, “Changing the call-to-action button color will not have an impact on the click-through rate.” The alternative hypothesis, on the other hand, posits that there is a significant difference. A statistical analysis is then performed, using a testing tool and machine learning if available, to determine if the null hypothesis can be rejected based on user behavior and the results of the multivariate testing.

How do you validate an AB test?

To validate an A/B test, you need to collect and analyze data to see if the changes made in the variant (B) have led to a significant improvement over the control (A). This involves determining your sample size, running the test until you have sufficient data, and then using statistical analysis to determine if there’s a significant difference. This process is part of your conversion rate optimization program, and a statistically significant result provides an excellent starting point to validate your proposed solution.

How do you write a hypothesis test?

A hypothesis for a test is written as a statement that predicts a potential outcome. The statement should specify the change being tested, the metric it is expected to impact, and the anticipated result. For example, “Increasing the font size of product prices on our website will increase user engagement by 10%.” This hypothesis is then validated through A/B testing. It’s also helpful to supplement A/B tests with other forms of data gathering, such as user testing or customer surveys, to understand why certain changes may or may not be effective.