Testing is an integral part of the product
design cycle. Different phases of the cycle require different types of testing.
In this post, I’ll explore the type of testing that
can reduce the risk of product failure through customer validation – beta
Although the practice of beta testing is
fairly common among product teams, there is still a lot of misunderstanding on
what beta testing really is and what benefits it offers to product design
teams. We will fill the gaps by providing best
practices for beta testing and things you need to consider when running them.
What is beta testing?
Beta testing is a type of user acceptance
testing where the product team gives a nearly finished product to a group of
target users to evaluate product performance in the real world.
There is no standard for what a beta test
should look like and how to set up beta testing. The actual testing procedure
should be relevant to your testing goals. However, there are a few requirements
that a product needs to comply with in order to be ready for beta testing:
- The product should be in the
“Feature complete” state (it should have all the features that are planned for
the release version).
- The product should be stable
(test participants should not face unpredictable crashes).
- Test participants should belong
to the product’s target audience.
- Test participants should
complete real-world scenarios while interacting with an app, and they should do
it in the real environment (not in lab environments).
Alpha testing vs. beta testing: what is the difference?
Before discussing different types and
approaches to beta testing, it’s essential to differentiate it from alpha
Alpha testing is the testing phase that
precedes the beta test phase. Unlike the beta version, the alpha version is usually
less stable and might have a limited feature set. Alpha testing is done by an
in-house team of developers, designers, and QA specialists. Alpha testers
usually mix black-box (testing method in which the
internal structure, design, implementation of the item, product, and feature
being tested is not known to the tester) and white-box testing (testing method in which the internal structure, design,
and implementation of the item being tested is known to the tester)
techniques to discover bugs and crashes. Alpha testing can be controlled
activity (meaning, it’s possible to reach test participants and ask specific
questions about their experience).
In the modern software development processes, beta testing should be an integral part of the product development cycle.
Beta testing is usually black-box testing, meaning test participants don’t know anything about the backend and don’t have access to source code. Since beta testing happens most of the time at the end user’s side, it cannot be a controlled activity. Beta testing can provide extremely valuable insights – genuine scenarios of interactions with a product.
For example, during beta testing, it’s possible to match the expected user journey with the actual user journey, by gathering behavioral habits (i.e. whether the user prefers to use mouse or keyboard during the interaction with a software) and emotional responses on interactions (i.e. how your design decisions make your users feel). It’s also possible to create an empathy map based on the actual data from your users.
The role of A/B testing in product design: phases of testing
In the modern software development
processes, beta testing should be an integral part of the product development
cycle. Ultimately, the moment for conducting a beta test in a development cycle
depends on the nature of a product you’re developing. Some teams reserve beta
testing only for major releases, while other teams conduct beta testing for
both minor and major releases.
While the phases of testing vary based on
the needs of the organization, generally the procedure of testing looks like
- Pre-alpha testing (in-house
testing done by the product team during the initial development stages)
- Alpha testing (in-house testing
done by the product team in a lab setting with a more developed prototype; it’s
possible to invite a small group of trusted users to participate in the alpha
- Beta testing (out-of-house
testing where subjects use the product to get initial reactions of finished
- Release testing (final testing
before shipping the product to the market)
Every phase should have clear exit
criteria. For instance, for beta testing the exit criteria can be zero open
high-priority issues in a bug tracking system.
Types of beta testing
Below are a few common types of beta
testing. Note that beta testing can include several different beta tests of
Closed beta testing vs. open beta testing
Closed beta version (also known as private beta) released to selected people who test its features. Usually, the number of test participants is limited. For example, if you plan to ship a product on the market in the next few months and have a landing page that allows visitors to leave their emails in order to get the latest updates about the product, you can choose beta testers from people who signed up for product updates. Closed beta tests are better suited for beta tests with a limited scope (only core features of the future product). And since the number of unique issues found by each tester decreases as the number of testers increase, it makes sense to limit the total number of test participants.
On the other hand, open beta tests do not
restrict access and allow anyone to sign up for beta testing. Open beta tests
usually follow a closed beta testing phase. This type of beta testing is great
when you want to collect quantitative data about your target users and their interaction
patterns. This type of testing also provides some insights on how well your
infrastructure scales – whether your backend can handle a real number of users. However, it can be hard to analyze the results
of open beta testing, especially if you have a large number of test
participants. Google for example, beta tested Gmail
and it took five years until the product was completely usable and fully
Technical beta testing
A technically focused beta test is beta
testing with a group of tech-savvy users (usually, it’s an internal group in
the organization). The goal of this testing is to uncover complex bugs and
provide high-quality test execution reports to the engineering team. The fact
that tech-savvy test participants have a tolerance to minor issues makes the
testing more focused. Test participants are more willing to complete the
testing despite the problems they face along the way.
Focused beta testing
Focused beta testing is used when a product
team needs to gather feedback on specific features. In order to collect the
feedback, the team releases the product to the market.
Marketing beta testing
Marketing beta testing has a goal to get media attention for your product. Generally, this type of beta testing can help you evaluate your marketing channels. But it’s also possible to use beta testing to understand customer’s reactions to new product features. Some products like Instagram have both active goals in their marketing beta testing.
What you need to run beta testing
Like any other testing process, beta
testing also begins with proper planning. During this stage, the team prepares
a testing strategy. Here are a few points that an organization needs to
Define the goals in advance
First and foremost, it’s important to know
what goal (or goals) you aim to achieve from your beta test. What exactly do
you want to test? Is there a particular user flow or specific feature? Based on
your goal, you will be able to select the testing scope and find the most
relevant type of beta testing.
Recruit the right test participant
Identifying and recruiting the right
participants is the major challenge. The fact that test participants should
have the necessary skills to use your product can be a problem for some groups
of products. An improper number of participants is among the common reasons why
beta testing fail.
But how do you calculate how many beta testers you need? Since all projects are different, it’s hard to provide a one-size-fits-all formula for calculating the number of test participants but you can rely on the triple constraint concept to get this understanding. This concept says that cost is a function of time and scope and that these three factors are related in a defined and predictable way. If we apply this concept to beta testing, we will see that when we increase the scope, we also have to increase time or cost to preserve quality. In terms of the number of test participants, the right amount of testers provide enough feedback to prove the statistical significance of your findings. The article “How to RightSize Your Mobile App Beta Program” shows how to do it in the context of a mobile app.
Define the length of the testing period
Both too short and too long test periods
lead to non-representative test results. It’s important to decide how many days
you want to keep the beta version available for beta testers before starting
the testing. The timeline can be determined by a set goal you have put in
place, or by utilizing a set portion of a budget, among other factors.
Write product documentation
The product should have How-to-Setup and
How-to-Use instructions. The instructions should be detailed, jargon-free, and reviewed
for correctness. Test participants should be able to access the instructions
whenever they want.
Share the information about known issues with test participants
If your product has some known issues that
have a negative impact on user experience, it’s better to share the list of
issues with test participants before starting beta testing. By doing that you
make test participants more tolerant of the issues.
Share results of beta testing with the product team and stakeholders
Design team, development team, QA, product
management – all should receive results of beta testing.
Create a clear procedure for collecting feedback
While some information about user behaviour
should be collected automatically (i.e. your product should have a built-in
mechanism of sending crash reports and other system information), it’s also
important to create a clear channel for communication between test participants
and your product team. Test participants should be able to share their thoughts
on product features and submit requests for changes in design.
Use tools for beta testing
Logging bugs, measuring productivity, and
collecting feedback from users are typical things you do during beta testing.
It’s important to have a tool (or a set of tools) that will simplify the
process of beta testing. Here are a few tools that can help you run effective
Beta testing is a fantastic practice that
allows you to test your product with real users before it reaches the market.
Though the purpose of beta testing may vary depending on the product, the
ultimate goal remains the same – create products that will have an excellent