Testing is an integral part of the product design cycle. Different phases of the cycle require different types of testing. In this post, I’ll explore the type of testing that can reduce the risk of product failure through customer validation – beta testing.

Although the practice of beta testing is fairly common among product teams, there is still a lot of misunderstanding on what beta testing really is and what benefits it offers to product design teams. We will fill the gaps by providing best practices for beta testing and things you need to consider when running them.

What is beta testing?

Beta testing is a type of user acceptance testing where the product team gives a nearly finished product to a group of target users to evaluate product performance in the real world.

There is no standard for what a beta test should look like and how to set up beta testing. The actual testing procedure should be relevant to your testing goals. However, there are a few requirements that a product needs to comply with in order to be ready for beta testing:

  • The product should be in the “Feature complete” state (it should have all the features that are planned for the release version).
  • The product should be stable (test participants should not face unpredictable crashes).
  • Test participants should belong to the product’s target audience.
  • Test participants should complete real-world scenarios while interacting with an app, and they should do it in the real environment (not in lab environments).

Alpha testing vs. beta testing: what is the difference?

Before discussing different types and approaches to beta testing, it’s essential to differentiate it from alpha testing.

Alpha testing is the testing phase that precedes the beta test phase. Unlike the beta version, the alpha version is usually less stable and might have a limited feature set. Alpha testing is done by an in-house team of developers, designers, and QA specialists. Alpha testers usually mix black-box (testing method in which the internal structure, design, implementation of the item, product, and feature being tested is not known to the tester) and white-box testing (testing method in which the internal structure, design, and implementation of the item being tested is known to the tester) techniques to discover bugs and crashes. Alpha testing can be controlled activity (meaning, it’s possible to reach test participants and ask specific questions about their experience).

In the modern software development processes, beta testing should be an integral part of the product development cycle.

Beta testing is usually black-box testing, meaning test participants don’t know anything about the backend and don’t have access to source code. Since beta testing happens most of the time at the end user’s side, it cannot be a controlled activity. Beta testing can provide extremely valuable insights – genuine scenarios of interactions with a product.

For example, during beta testing, it’s possible to match the expected user journey with the actual user journey, by gathering behavioral habits (i.e. whether the user prefers to use mouse or keyboard during the interaction with a software) and emotional responses on interactions (i.e. how your design decisions make your users feel). It’s also possible to create an empathy map based on the actual data from your users.

The role of A/B testing in product design: phases of testing

In the modern software development processes, beta testing should be an integral part of the product development cycle. Ultimately, the moment for conducting a beta test in a development cycle depends on the nature of a product you’re developing. Some teams reserve beta testing only for major releases, while other teams conduct beta testing for both minor and major releases.

Hand turning a process knob. Concept of software or app development. Composite image between a photography and a 3D background.
Image by Olivier Le Moal

While the phases of testing vary based on the needs of the organization, generally the procedure of testing looks like this:

  • Pre-alpha testing (in-house testing done by the product team during the initial development stages)
  • Alpha testing (in-house testing done by the product team in a lab setting with a more developed prototype; it’s possible to invite a small group of trusted users to participate in the alpha test)
  • Beta testing (out-of-house testing where subjects use the product to get initial reactions of finished product)
  • Release testing (final testing before shipping the product to the market)

Every phase should have clear exit criteria. For instance, for beta testing the exit criteria can be zero open high-priority issues in a bug tracking system.

Types of beta testing

Below are a few common types of beta testing. Note that beta testing can include several different beta tests of different types.

Closed beta testing vs. open beta testing

Closed beta version (also known as private beta) released to selected people who test its features. Usually, the number of test participants is limited. For example, if you plan to ship a product on the market in the next few months and have a landing page that allows visitors to leave their emails in order to get the latest updates about the product, you can choose beta testers from people who signed up for product updates. Closed beta tests are better suited for beta tests with a limited scope (only core features of the future product). And since the number of unique issues found by each tester decreases as the number of testers increase, it makes sense to limit the total number of test participants.

Graph of test users leveling out with the number of problems found.
More users do not equate to more problems being uncovered.

On the other hand, open beta tests do not restrict access and allow anyone to sign up for beta testing. Open beta tests usually follow a closed beta testing phase. This type of beta testing is great when you want to collect quantitative data about your target users and their interaction patterns. This type of testing also provides some insights on how well your infrastructure scales – whether your backend can handle a real number of users. However, it can be hard to analyze the results of open beta testing, especially if you have a large number of test participants. Google for example, beta tested Gmail and it took five years until the product was completely usable and fully publicly accessible.

Technical beta testing

A technically focused beta test is beta testing with a group of tech-savvy users (usually, it’s an internal group in the organization). The goal of this testing is to uncover complex bugs and provide high-quality test execution reports to the engineering team. The fact that tech-savvy test participants have a tolerance to minor issues makes the testing more focused. Test participants are more willing to complete the testing despite the problems they face along the way. 

Focused beta testing

Focused beta testing is used when a product team needs to gather feedback on specific features. In order to collect the feedback, the team releases the product to the market.

Marketing beta testing

Marketing beta testing has a goal to get media attention for your product. Generally, this type of beta testing can help you evaluate your marketing channels. But it’s also possible to use beta testing to understand customer’s reactions to new product features. Some products like Instagram have both active goals in their marketing beta testing.

What you need to run beta testing

Like any other testing process, beta testing also begins with proper planning. During this stage, the team prepares a testing strategy. Here are a few points that an organization needs to consider:

Define the goals in advance

First and foremost, it’s important to know what goal (or goals) you aim to achieve from your beta test. What exactly do you want to test? Is there a particular user flow or specific feature? Based on your goal, you will be able to select the testing scope and find the most relevant type of beta testing.

Recruit the right test participant

Identifying and recruiting the right participants is the major challenge. The fact that test participants should have the necessary skills to use your product can be a problem for some groups of products. An improper number of participants is among the common reasons why beta testing fail.

But how do you calculate how many beta testers you need? Since all projects are different, it’s hard to provide a one-size-fits-all formula for calculating the number of test participants but you can rely on the triple constraint concept to get this understanding. This concept says that cost is a function of time and scope and that these three factors are related in a defined and predictable way. If we apply this concept to beta testing, we will see that when we increase the scope, we also have to increase time or cost to preserve quality. In terms of the number of test participants, the right amount of testers provide enough feedback to prove the statistical significance of your findings. The article “How to RightSize Your Mobile App Beta Program” shows how to do it in the context of a mobile app.

Drawing on a triangle showing scope, cost and time at each point.
Use the triple constraint concept to determine how many beta testers you need.

Define the length of the testing period

Both too short and too long test periods lead to non-representative test results. It’s important to decide how many days you want to keep the beta version available for beta testers before starting the testing. The timeline can be determined by a set goal you have put in place, or by utilizing a set portion of a budget, among other factors.

Write product documentation

The product should have How-to-Setup and How-to-Use instructions. The instructions should be detailed, jargon-free, and reviewed for correctness. Test participants should be able to access the instructions whenever they want.

Share the information about known issues with test participants

If your product has some known issues that have a negative impact on user experience, it’s better to share the list of issues with test participants before starting beta testing. By doing that you make test participants more tolerant of the issues. 

Share results of beta testing with the product team and stakeholders

Design team, development team, QA, product management – all should receive results of beta testing.

Create a clear procedure for collecting feedback

While some information about user behaviour should be collected automatically (i.e. your product should have a built-in mechanism of sending crash reports and other system information), it’s also important to create a clear channel for communication between test participants and your product team. Test participants should be able to share their thoughts on product features and submit requests for changes in design.

Use tools for beta testing

Logging bugs, measuring productivity, and collecting feedback from users are typical things you do during beta testing. It’s important to have a tool (or a set of tools) that will simplify the process of beta testing. Here are a few tools that can help you run effective beta testing:

Conclusion

Beta testing is a fantastic practice that allows you to test your product with real users before it reaches the market. Though the purpose of beta testing may vary depending on the product, the ultimate goal remains the same – create products that will have an excellent user experience.