What is guerrilla testing?
Guerrilla testing (also known as hallway usability testing) is a relatively fast and informal way to test ideas, to get high-level feedback, and potentially uncover user experience problems. It can be done pretty much anywhere: a coffee shop, a shopping center, or on the street. Ideally, you’ll test between 8–10 users as part of your hallway test, with each usually lasting for around 10 mins.
While the technique is one that is fairly straight-forward, getting the most out of it requires some structure and preparation.
No research is ever quite complete. It is the glory of a good bit of work that it opens the way for something still better, and this repeatedly leads to its own eclipse.
Mervin Gordon, Software/UI Developer at Mezlad
A quick caveat: It’s not for every situation
Guerrilla testing won’t work for everyone, and it’s not liked by everyone, either. Its usefulness very much depends on your situation and how you do it. If done well, it has value; if not, you’re wasting your time.
What guerrilla testing is NOT
Firstly, guerrilla testing should not replace formal user testing. Don’t do it just to jump through research quicker; this’ll screw things up, trust me on this one. You might get some good little bits of feedback with guerrilla testing, but better results can be gleaned from taking the time to do fuller research. Secondly, guerrilla testing is not detailed testing. It’s not thorough research, and therefore you need to be very cautious about making significant decisions based on the insights you derive from these tests.
The good things about guerrilla testing
- You can move fast.
- If you have no research budget, it’s better than no testing at all.
- It’s super cheap compared to more formal research.
- It can be a great way to validate/invalidate early assumptions.
- It can be an excellent way to identify areas to do deeper research on.
- It can work well with small-iterations type work.
The not so good things about guerrilla testing
- Not a lot of time spent with participants, so you’re restricted to a small part of the flow.
- It may be more difficult to convince stakeholders about the insights you generate.
- The people you chat with may not be the types of people that will use your product.
Tips on how to do guerrilla testing right
Successful hallway usability tests require strategy and preparation. Here are a few tips to get the most out of your testing:
Have a goal
Why are you doing this? Have a sensible goal, bearing in mind that this is a 10-minute test. Can people find a specific part of your website? Can people log into the app without any trouble? Do people know what’s happening when they reach a particular part of the app or website? You need to know what you expect with this research.
Have an easily digestible sales pitch (with incentive) when introducing yourself
You’ll get 20–30 seconds to pitch yourself to your potential test participants. If they’re in a coffee shop, buying them a coffee/cake for their time is an excellent idea. They’re in there to have a drink or to eat something, so it’s a bonus for them to talk to you if they have the time. Make the incentive worth their while.
A board such as the one in the picture above is a great idea. It does a lot of the work for you in getting your intro across (including incentive). It also means you have a base to set yourself up with the gear you need for testing.
Prioritize diversity when selecting participants
This can be tough, especially if limiting your selection process to a single setting. If I set myself in Sydney’s CBD, I may find a fairly consistent type of participant. If I test with four people there, then test four more people in another location, then four in yet another area, that may help in getting a more diverse group of people. It pays to move around.
Be sure to ask people to participate of different ages, cultures, genders, etc. It’s very important to get diverse points of view.
Have qualifying questions and be sure to tell participants what the heck is going on
Don’t assume that people know what’s happening even if you have a sign. You need to make clear who you are, what your and your colleagues’ names are, what you’re asking of them, and the incentive you’re offering. An example dialog would be:
You: “For a coffee and cake, we’d love 10 minutes of your time to get feedback on a few parts of our app. You will need to qualify with a couple of super quick questions.”
Them: “Ok, I have time.”
You: “Do you have an iPhone?”
Them: “Yes.”
You: “Great. Have you paid a tradesman (plumber, electrician, etc.) in the past 6 months?”
Them: “Yes, I’ve used a locksmith in the past month.”
You: “Do you have 10 minutes? Do you need to be anywhere soon?”
You could also add qualifying questions on your “pitch board,” as mentioned above, if it helps qualify people early on.
Be clear what you need from the participant
You may want them to look at a prototype, talk out loud, and tell you if they can locate something. Be extra clear and articulate this.
Make sure to ask if they need to be somewhere soon
You do not want to chat with someone who needs to rush off before completing the process. Make sure to ask them if they need to rush off. It will be a waste of your time if they can’t see the session through to completion.
Have a partner in crime
It’s useful to have a colleague. This can be helpful for recruiting people, buying incentives (like coffee, for example, if you’re in a coffee shop), taking notes, and helping keep you on track.
Have a plan for capturing feedback
One route for capturing feedback is to have a notebook or a piece of paper. Add the date, time, and location at the top, along with you and your colleague’s name.
Write a few notes on the intro you may need to say to the participant. This will remind you to tell the participant exactly what you’re expecting of them and what they’ll be doing if they agree to help you. You may have said all of this when you met them, but it may have gone in one ear and out the other. Have a section outlining all the details about them that you need; for example, their age, gender, what they do, where they’re from, where they live, etc.
Then, make sure to have the questions you want to ask written down. In between each question, have a chunk of space to scratch out your answers. I use Google docs and print the document so I can write on it during the sessions.
You need all of the above for each participant. If this goes over into more than one piece of paper, be sure to have a mini stapler by your side so you can put the pieces of paper together when you’re done. You could do this before you begin, if it helps what you test. If the participant consents you can also consider recording the conversation to refer to afterwards.
When it comes to visual aids and collecting feedback, you can use a production-ready app or website of your product to show the participant and collect notes on specific aspects of an experience. You can also use a prototype, or you can use printed-out sheets. Each has pros and cons, but for experiences requiring connectivity, remember that you’ll need access to a fast and stable internet connection. It’s always an option to also test competitors’ products, if your product is in an early stage and you don’t have a prototype to use.
Create a simple scenario
For your test, you’ll also likely want to make sure to create and frame a simple scenario that the participant can understand. Overly complex scenarios will make task analysis more difficult and less reliable. Write down the tasks that the participant will need to go through, and make sure to only choose the most essential of them. There will not be enough time to cover an exhaustive list.
For each task, create a simple scenario. You need to present the participant with a problem and no indication of how to solve it. You can work through the scenarios with colleagues before going out in the field. This kind of cognitive walkthrough is a great way to make sure it’ll make sense for the participants.
Have set questions in-line with your goal/scenario
It’s important to have set questions you ask each person you test. From this, you can get clear takeaways and themes that arise from multiple interviews. If you go rogue and stray from these questions, even if it seems like an interesting rabbit hole to go down, you may wind up with different answers to different questions. Without consistent questioning, you won’t have reliable results.
Be sure to listen
Don’t talk over your participants; be sure to listen to them and don’t be leading. This may seem obvious but testing can be exciting and you can get carried away being too chatty.
Conduct sessions with 8–10 participants
The old adage is that, after you successfully interview five participants, you will start to get diminishing returns in terms of findings. That works for me, but in this type of testing, I think you should allow for a few off-key people within the group (people that aren’t your ideal users). In essence, what we want is to get five very solid people out of 8–10 successful interviews.
Stick to your allotted time
Don’t ask participants to commit to more time than was requested. You’ll need to make sure you set up scenarios that fit the allotted time. If you can’t test certain scenarios with certain people, you’ll miss out on getting the set of answers from all participants.
Be strategic about the locations you choose
We’ve talked about coffee shops, but you don’t have to choose a coffee shop for your testing. Coffee shops are just good places to get people who are stopping in one place for a period of time. Plus, coffee as an incentive works nicely in this environment.
Another solid idea is to offer to buy someone lunch in a food court. The key is to find a place that is not super noisy and distracting. You want people to feel relaxed during the session.
Take photos, record the test
Take a selfie of you and the participant after the test. This is a great way of recording the moment so you can easily reflect back on their answers. You can also video the session. Watching it back a few times will help see anything you may have missed. Just remember to have a fully charged phone and take a plug or portable battery with you. A phone stand is also a great idea to free up your hands.
Analysis
So, you’ve finished all of your sessions, and you have 8–10 sheets of notes related to each person. Make sure that they’re organized by participant, and don’t mix them up. One at a time, go through the notes for each session. This is good to do shortly after your testing session so that it’s fresh in your mind. For each scenario, jot down additional findings on post-it notes (make sure to put the corresponding participant’s name on each post-it note). This can be useful if you have repeat comments from the same participants.
Once you have notes for each scenario from each participant, find a wall to stick your post-its to. Split them out for each scenario, then go through the videos (if you recorded them) to see if you’ve missed anything. This can be an excellent way to slow down your process and make sure you have all your findings organized together.
Now that you have a bunch of post-it notes for each scenario, cluster them by any themes you find. Then, go through this process a second time to see if you can find more themes. Do this as many times as makes sense.
Once you’ve reached this stage there will no doubt be larger themes and smaller ones. Larger themes are ones that came from the majority of participants. Smaller themes are ones that came from fewer people. While these edge-cases may be less common they could still offer important insights.
Some insights may need more digging to find out the ‘why’ behind the problem. Some may be more obvious, and easily improved. Ideally tackle the more important stuff first.
Hallway usability testing takeaways
- Hallway usability tests should not replace more formal user research, but if you have no budget for formal testing, then this could be a promising option.
- See if guerrilla testing suits your product. There is a good chance you won’t be able to find people that are similar to your customers. If so tread carefully.
- Have a clear goal.
- Have a master plan: A good pitch, scenarios, questions, note-taking set-up, a partner in crime, sensible location, diverse participants, recording, photos, etc.
- Have qualifying questions. Don’t just accept anyone who simply says yes to participating. This will help you get the right people. Also, make sure they have enough time to participate. Be diverse with your choices.
- Follow your script and be time conscious.
- Analysis.
- Take action. Do more digging on areas where the ‘why’ is not clear.
If you’re working somewhere that does not want to do research this can be a great way to pitch it to them. It’s small steps to get them to create a budget.
Your old site is the best prototype of your new site.
Hoa Lorangers, Vice President at Nielsen Norman Group
Guerrilla research is up for debate. Many see it as dangerous. Purists will spit in its general direction. As someone who’s had his own business with little to no budget, I can safely say there is a place for it. It will only get you so far, but provided you have a plan and can talk to people who are not too different from potential customers, then give it a crack.
I’ve never regretted speaking to people about products I’ve designed for, even if my process has not been perfect. I doubt you will either.