Gearing for Growth during Beta Testing
Understand how real users use your product and experiment for future growth
Months of preparation have led to this. Your product development team (UI/UX designers & engineers) have put in countless hours into crafting the final product and got past the Alpha Phase. Your business functions (marketing, customer support, sales) have drawn up (& redrawn) plans for launch and even participated as testers in the Alpha Phase. However, there is just one final phase that almost all product has to go through before the official launch: the Beta Test.
What is a Beta Test?
Beta Test refers to the phase when a product is released to a selected pool of real-life customers over a period of time. A product is ready to be beta-tested when it has all (most) of its promised features complete and the internal quality assurance team are looking for areas where the product doesn’t function correctly.
Why do Beta Testing?
But hey, isn’t the job of the internal quality assurance team to flash out all the defects and bugs? If so, why is there a need to invest time and money in beta testing instead of going for a full launch?
Real People, Real Environment, Real Product are the three R’s of Beta Testing, and the most important question that arises here in Beta Testing is “Do Customers like the Product?”
Beta Testing provides a complete overview of the true experience gained by the end users while experiencing the product. It is performed by a wide range of users and the reasons for which the product is being used varies highly. While much has been done by the product development team during the initial phase, product managers identify the potential market segment and focus on the target market’s opinion to decide which features to include, while a usability engineer focuses on product usage and easiness.
However, at the end of the day, it is the actual perception of the end users that matter as they are the ones paying for the product. Known Issues, which are accepted by the product development team, might be vehemently rejected when the end user faces the same issue and may not be comfortable while using the product. In such cases, Beta Testing helps to uncover the business impact of known issues on the entire product as user experience gets hampered. The results can help the product team reprioritize features and fixes in order to help the business succeed.
Beyond all practical reasons to run a Beta program to ensure high product quality, it is also used as a growth tactic to create a sense of exclusivity too. Pinterest uses beta testing by invitation to build an aura of exclusivity around their product. Initially, Pinterest was invitation-only, with a wait-list of interested users excitedly waiting to get a chance to try it. By artificially restricting the supply of your product in this way just make people want it more and drive the demand for it. Everybody wants to be a VIP!
So what role does the Growth Team play during Beta?
The primary role of a Growth Team is to scale the usage of a product that has product-market fit at a sustainable rate. This means concentrating on getting as many people to experience the core value of the product in the fastest way while ensuring a healthy retention rate. Otherwise, exponential growth means nothing if you can’t retain your customers beyond their first experience.
During Beta-Testing, the Growth Team have access to a group of users who are industry-savvy. They are excited to be the first adopter and takes pride in providing feedback to make your product better. This is the best time to dig deep to understand why users are returning to use your products while others do not after the first try.
To do so effectively, the Growth Team needs to gather both quantitative data and qualitative data. Quantitative data often only reveals ‘what the users are doing’ while qualitative data reveal ‘why they do it’. For example, a funnel analysis shows how users are dropping off at each stage of the registration process.
Just by looking at the funnel, we are not able to identify the reasons why users are dropping off. It takes qualitative research methods such as interview and focus group to gather insights from talking to customers to uncovers the ‘Why’.
Usually, in a mature organization, this is owned by a Product Research team. The Growth Team owns the quantitative research methods such as funnel and cohort analysis. Together, they marry the insights to provide a comprehensive view of the gaps in the product that helps with the prioritization of new features to improve customer experience.
Running Experiments
This is also a good time to run experiments that will help you determine your growth plans at launch. Even with the relatively small user population, learnings from these experiments are still useful. They are less likely to affect many users (if anything goes wrong) and cost less compared to a larger user population at launch. While I won’t go into the technicalities of designing and running an experiment, I will use a real-life example to explain what makes up for a good business hypothesis.
A good business hypothesis typically starts out with a goal, meaning knowing what specific business metrics you want to move. To better define goals, I used Objective and Key Results (OKRs), a goal system practiced in Google and LinkedIn for better alignment and engagement around measurable goals. For example, let’s say I have an OKR of
Objective: Increase the % of New User Conversion to First Purchase
Key Results: by 20%
With a measurable goal in mind, I move on to the next step which is to define a Problem Statement. For example,
50% of New Users did not make their first purchase within 2 weeks even though they know they have a discount voucher.
I know that this is a puzzling behavior as the Beta users made the effort to sign up to try the product and were even given a discount voucher. So why aren’t they making their first purchase? After much brainstorming, perhaps, sending them a push notification that directs them to the page that shows the vouchers will remind and motivate them to complete their first transaction.
This brings me to the next step where I form 2 hypotheses; the null hypothesis and the alternative hypothesis. These hypotheses contain opposing viewpoints.
H0: The null hypothesis: Sending a push notification that directs users to the page that shows discount vouchers will not motivate a user to make a purchase
Ha: The alternate hypothesis: Sending a push notification that direct users to the page that shows discount vouchers will motivate a user to make a purchase
It is important to ensure that the null and alternative hypotheses are contradictory in order to examine the evidence to reject the null hypothesis or not. In this case, we run the experiment and examine the “% of New User Conversion to First Purchase” for 3 days. Surprisingly, results from the experiment show that people who were direct to the vouchers page from the push notifications are 2x more likely to make a purchase. Thus, we accept the alternate hypothesis and reject the null hypothesis.
In the absence of qualitative data, we can posit that it is likely due to 2 reasons:
- As users are people going through the motion of life bombarded with many distractions, they simply forgot the vouchers
- Seeing the vouchers itself entices them more to make their first purchase
This is just one of the many examples of experiments that a Growth Team could run to improve business metrics significantly without incurring too much cost or development effort.
So that’s it. Even during the Beta Phase, Growth team can use data and qualitative research to identify gaps in a product to help in features prioritization for the official launch. On top of that, you can run experiments to further optimize your onboarding or conversion flow to get more users to experience the values that you created. Lastly, be sure to make your Beta Testers feel heard and incorporate their feedback into product development. This will help you win them over from tester to long-term loyal customer who can’t stop raving about your product to their friends.
