Qualitative usability testing, also known as qualitative user testing, is a popular UX methodology that can provide rich insights. One of the biggest advantages of qualitative usability tests is that they can be carried out cheaply and quickly. But there are a few things I learned by preparing and conducting my first few qualitative usability testing sessions.
Do qualitative usability tests now and often
1. Test the current design to learn, gain confidence and get buy-in more easily.
It is a good idea to carry out qualitative usability tests on the current design. There are always usability problems that the client is not aware of. Confronting the client with all the findings and recommendations about their current product can be a game changer. By conducting these user tests, you gain stakeholder’s trust and/or get easier buy-in for more research, especially if the UX maturity is low. In addition, gathering insights from the current design is important so you don’t make the same mistake in the new design.
2. Test early and often, not just the approved design or redesign.
Qualitative usability testing can be done at any stage of the UX process. Whether you follow an industry standard process or your own approach, qualitative usability testing is one of the easiest and richest qualitative methods to test the user experience of an interface. There are several reasons when you should do qualitative usability testing:
- Exploration. Testing the current usability or preliminary concepts.
- Evaluation. Evaluate specific features during implementation.
- Comparison. Assessing your design against others.
- Validation. To certify specific features that meet the expected standards.
If you wait for qualitative usability testing until the approved version, you risk getting into trouble. Imagine testing an approved design and it turns out that your users have trouble using the interface, or worse, you find out that you have designed a concept that doesn’t work at all.
Start recruiting early
3. Recruiting participants on your own is harder than you think.
Recruiting participants can be a challenge. If you have never recruited before, it is very difficult to get started. If you have decided to recruit participants yourself, here are some ways to do it:
Colleagues. Are easy to recruit, but will give you biased data, which is a horrible idea. Basically, you should never do this.
Friends and/or family. Are a little harder than recruiting your own colleagues, but still pretty easy. They can go wide because you can ask colleagues of friends or family. But they can have an impact on your data because they might know you or are related to the field.
Target audience. Are of great value and can be recruited via marketing and customer support resources. Recruiting them yourself can be a time consuming and painful process.
As you can see from the list above, it can be difficult to recruit participants on your own and it can put your collected data at risk. Always go for participants who represent the target audience. However, it is always 100% better to conduct usability tests with someone than with no one.
4. Use recruiters and provide a recruitment screener.
The best solution for participant recruitment is to leave it in the hands of recruitment professionals or someone in your company/agency who has connections with marketing or customer support. These people know exactly where and how to get in touch with potential participants.
If you can give the responsibility to others, make sure you give them a recruitment screener. This is a document that provides guidelines setting out the criteria for test participants. A recruitment screener should contain the following information:
- Number of participants. How many test participants do you need?
- Time and place. When do you need the test participants? Where do you need them? Online or offline?
- Demographic criteria. Only relevant information. Mix of ages is often a good idea.
- Product knowledge. What is their history with digital products? How tech-savvy should they be?
- Duration. How long will it take for each participant?
- Payment. What and how will they be paid?
- Location and contact info. For both online and offline qualitative usability tests it’s a good idea to give some kind of information when recruiting participants.
5. Recruit early, and enough.
Recruit early! Waiting for other factors before you start recruiting can cause you a lot of stress during a project. It will take time to find some participants. If you have a high UX maturity, it will go faster. If not, be prepared and start recruiting as soon as possible. Start recruiting at least 2 weeks in advance. If it’s your or your team’s first time, start recruiting even earlier. Test ways to recruit, test different channels, talk to people internally or even talk to your friends or family.
If you start recruiting early, it is also a good idea to recruit participants for more usability testing sessions in the future. Be prepared if you need extra participants, or if you might need to test different iterations soon. With a few extra participants, you can do more usability testing and reduce stress during the different phases of the UX process.
Keep a structured living document
6. Document your research project and method from the start
Try to create one source of truth at the beginning of your project. This may come in handy when you have to lead the usability research project or when you are responsible for designing the usability test sessions. A structured document can contain the following parts:
- Brief summary. Start with a summary to give some background information for yourself or for the person reading the document.
- Methodologies. Explain the methodologies used for this research project, in this case qualitative usability testing. Give an overview of the participants and demographic information and an overview of the task scenarios you’re planning to use for the test sessions.
- Test results. Include the correct paths, plus the paths participants chose during the usability test sessions.
- Findings and recommendations. Make sure you write down all findings and recommendations for the usability problems you want to improve. These can be grouped by pages, tasks, functions and so on.
Keeping such a document is useful for keeping track of things as the project progresses. In addition, this document can be used as a template deliverable for the client. When you decide to deliver a report with this structure in mind, write as much as possible in bullets. Your clients are busy people, they want to digest, discuss and remember this information quickly.
Qualitative usability tests go hand in hand with interviews
7. Complement you qualitative usability test sessions with in-depth interviews
Conducting qualitative usability sessions gives you the opportunity to talk to your users. Therefore, if needed you can add an additional method: in-depth interviews or user interviews. Usability testing sessions can be done within 30–60 minutes. If you think there is room for more, adding a short user interview focused on goals and context only takes 20–30 minutes of extra time from the participants.
Combining qualitative usability tests with in-depth interviews can provide richer insights, such as a better understanding of user goals and the context of use. These richer insights can provide data that can be used for mapping experiences. Mapping alignment diagrams gives an overview across channels and touchpoints which can lead to new and innovative ideas.
Focus on top tasks or key features
8. If you have a tight budget, prioritise your tasks to test what is most important
If you need to design or redesign an entire product or service, make use of the Pareto Principle. This means prioritising 20% of your product/service that are responsible for 80% of the use cases. Most products or services can do a lot, and in order to focus on the most important things, we use these 20% of use cases as a guide, also known as top tasks, for your qualitative usability testing sessions.
If all tasks seem important to the client, it is our job to do an analysis to identify the top tasks. If you don’t have the budget, time or resources to do a proper top task analysis, start by discussing which features are important. Involve stakeholders, webmasters, analytics experts, customer support in discussions about which features are important. Discuss features that are:
- used often;
- new;
- frequently used;
- considered hard to use;
- considered important to users.
Discussing these features will help you focus and hopefully lead to realistic scenarios that can be used to set up your qualitative usability test script. Realistic scenarios will feel much more natural to your target audience than scenarios thought up from scratch. Moreover, discussing top tasks can lead to new questions and gaps in knowledge, which can lead to easier buy-in and/or extra budget to carry out more methods. The opportunity to conduct more research methods can in turn open doors to triangulation, to increase the credibility and validity of your research.
Let’s recap
There is no time to waste, conduct qualitative usability test sessions now and often. You’ll gain confidence, get easier buy-in and gather rich insights about the current design or iteration.
Start recruiting your participants right now! Once you’ve got a clear picture of who your target audience is, start looking for them and don’t forget to provide your recruitment screener. Always go for people who are or at least represent the target audience. But remember, it is always 100% better to test with someone than with no one.
Keep a structured document to plan and capture your research progress. In addition, this document can be used as an artefact for your client and/or stakeholders.
If needed, combine your qualitative usability test sessions with an in-depth interview. It only costs your participant a few extra minutes, but in return you get much richer insights about the user’s goals and context.
If money for the project is tight, prioritise your tasks so that you can focus on what is most important to your target audience. This way, you can test the most critical parts of your product or service.