Lessons I Learned from my first few Qualitative Usability Tests

Photo by Frederick Medina on Unsplash
Photo by Frederick Medina on Unsplash

Qualitative usability testing, also known as qualitative user testing, is a popular UX methodology that can provide rich insights. One of the biggest advantages of qualitative usability tests is that they can be carried out cheaply and quickly. But there are a few pitfalls I have encountered when preparing and conducting my first few qualitative usability testing sessions.


Do qualitative usability tests now and often

1. Test the current design to learn, gain confidence and get buy-in more easily.

It is a good idea to carry out qualitative usability tests on the current design. There are always usability problems that the client is not aware of. Confronting the client with all the findings and recommendations about their current product can be a game changer. By conducting these user tests, you gain stakeholder’s trust and/or get easier buy-in for more research, especially if the UX maturity is low. In addition, gathering insights from the current design is important so you don’t make the same mistake in the new design.

2. Test early and often, not just the approved design or redesign.

Qualitative usability testing can be done at any stage of the UX process. Whether you follow an industry standard process or your own approach, qualitative usability testing is one of the easiest and richest qualitative methods to test the user experience of an interface. There are several reasons when you should do qualitative usability testing:

  • Exploration. Testing the current usability or preliminary concepts.
  • Evaluation. Evaluate specific features during implementation.
  • Comparison. Assessing your design against others.
  • Validation. To certify specific features that meet the expected standards.

If you wait for qualitative usability testing until the approved version, you risk getting into trouble. Imagine testing an approved design and it turns out that your users have trouble using the interface, or worse, you find out that you have designed a concept that doesn’t work at all.

Keep a structured living document

3. Document your research project and method from the start

Try to create one source of truth at the beginning of your project. This may come in handy when you have to lead the usability research project or when you are responsible for designing the usability test sessions. A structured document can contain the following parts:

  • Brief summary. Start with a summary to give some background information for yourself or for the person reading the document.
  • Methodologies. Explain the methodologies used for this research project, in this case qualitative usability testing. Give an overview of the participants and demographic information and an overview of the task scenarios you’re planning to use for the test sessions.
  • Test results. Include the correct paths, plus the paths participants chose during the usability test sessions.
  • Findings and recommendations. Make sure you write down all findings and recommendations for the usability problems you want to improve. These can be grouped by pages, tasks, functions and so on.

Keeping such a document is useful for keeping track of things as the project progresses. In addition, this document can be used as a template deliverable for the client. When you decide to deliver a report with this structure in mind, write as much as possible in bullets. Your clients are busy people, they want to digest, discuss and remember this information quickly.

Qualitative usability tests go hand in hand with interviews

4. Complement your qualitative usability test sessions with user interviews

Conducting qualitative usability sessions gives you the opportunity to talk to your users. Therefore, if needed you can add an additional method: in-depth interviews or user interviews. Usability testing sessions can be done within 30–60 minutes. If you think there is room for more, adding a short user interview focused on goals and context only takes 20–30 minutes of extra time from the participants.

Combining qualitative usability tests with in-depth interviews can provide richer insights, such as a better understanding of user goals and the context of use. These richer insights can provide data that can be used for mapping experiences. Mapping alignment diagrams gives an overview across channels and touchpoints which can lead to new and innovative ideas.

Let’s recap

There is no time to waste, conduct qualitative usability test sessions now and often. You’ll gain confidence, get easier buy-in and gather rich insights about the current design or iteration.

Keep a structured document to plan and capture your research progress. In addition, this document can be used as an artifact for your client and/or stakeholders.

If needed, combine your qualitative usability test sessions with an in-depth interview. It only costs your participant a few extra minutes, but in return you get much richer insights about the user’s goals and context.