Usability Testing
- Andrew Brown (Deactivated)
Before we dive into usability testing, let’s step back and look at usability or the measurement of how well users can successfully and efficiently achieve their goals when using a product. After we’ve determined we can solve the users’ problems, we must make sure the feature is usable. That’s where usability testing comes in.
What is usability testing and why is it important?
Usability testing is an important part of the iterative design process. It is a way to gain deep insight about our users’ behavior and expectations of our product and its features.
There are two types of data that can be collected from usability testing:
Qualitative data - findings observed during testing that help identify areas of the product that are easy or hard to use
Quantitative data - a metrics-driven approach that tells the story of how well or poorly users were able to complete defined tasks
Qualitative Data
Qualitative data is a direct approach to assessing the usability of a product, generally used to identify design problems in the early stages. Data is collected through observing a user's behaviors, emotions, and experiences; not the numbers. Moderators are able to ask follow-up questions and change the narrative to gain more information; generally done through Moderated User Tests.
Quantitative data
Quantitative data is an indirect approach to assessing the usability of a product, generally used to collect data after a feature is released. As usability testing tools have evolved, we can now take a quantitative approach during the design process. Data is collected based on the user's performance of the defined tasks. To put it simply, it’s all about the numbers; generally done through Unmoderated User Tests.
Examples of quantitative data
Bounce rates
Click patterns (tracked via heat maps)
Myth-busting
A common myth is that usability testing slows down shipping features. This is not true and doesn’t need to be the case. Usability testing runs in parallel with other activities and doesn't require a large number of participants. Actually, it’s recommended to only test with five users, more on that below. Conducting user tests early in the design process allows us to fail early and often before we touch a line of code saving us time and money in the end.
Understanding our users
Usability testing is a methodology that provides psychological and behavioral evidence of our user’s needs. It’s a fast and proven way for us to better understand our users, listen to their feedback, and understand their needs and expectations. The insights we gain will help the business meet the needs of our customers.
You only need to test with 5 users
Testing with too many users is a waste of time and resources. The best results come from testing with no more than 5 users and running as many small tests as we have time for, and can afford (if we are compensating users for their time).
I’m not going to get super granular here, since the NN/g (Neilsen Norman Group), an authority on research-based user experience, have done that nicely here.
Simple chart illustrating how insights can change by adding a single user test
Usability Testing Methods
There are a variety of usability testing methods. We’ll focus on the following methods to help better understand our users and their needs.
Heuristic evaluations
Moderated User tests
Moderated usability testing is when users complete a set of tasks while being observed/recorded by a moderator (researcher or designer) in real-time.
Things to consider
It’s important to build trust with our test users, the more they trust us the more comfortable and open they’ll be when sharing their thoughts and expressing their feelings. During moderated testing, it’s easier to want to guide the user through tasks, dig deeper into their problems while receiving real-time feedback. It’s best to try and avoid interacting with the user until they’ve completed the tasks. Use your best judgment when jumping to not bias the tester.
Pros
Ask a question during the test to gather real-time feedback
Improvise when needed, ie add an additional task to gain better insight
Observe emotional responses
Cons
Can be more time consuming to recruit, conduct tests, analyze, and interpret and report the findings
Moderation can influence or alter the results
Unmoderated User tests
Similar to moderated user tests, unmoderated usability testing is when users complete a set of tasks without being observed/recorded by a moderator. These tests can be completed by the user at their own pace, anywhere they choose.
Things to consider
It’s important that with unmoderated tests, we are very clear with the tasks that need to be completed since we won’t be with the users’ to help if they become stuck. We need to make sure that the tools we choose, Maze Design, continues to provide the right value when capturing and recording data throughout the tests.
Pros
Saves time not having to plan to test with individual users
No moderator is needed for users to take the test
Users can take the test at their own pace, anywhere they’d like
Cons
No one can help if the user gets stuck
No way to observe the users emotional responses
We could ask users to record themselves with Loom?
Five Second tests
Five-second tests are a popular and quick way to capture a user's first impressions. These tests generally focus on the copy-writing or discoverability of a new element in the UI. Setting up five-second tests is quick but it provides valuable insights.
Things to consider
FIve-second tests are truly meant to be fast, with low effort to set up. If you find yourself spending too much time creating these tests, consider pairing back the test or converting to an unmoderated user test or another method.
Pros
Fast and low effort
Cons
TBD
Card Sorting
Card sorting is a popular usability testing method used to understand how users find information, group the information, and navigate through a product or website. This is generally reserved for information architecture but is also a great way to learn how well labels and terminology resonate with users.
Things to consider
There are two methods for card sorting tests, Open and Closed. Open card sorting allows testers to freely organize / group content that makes sense to them. While Closed card testing, asks users to organize / group content into pre-set categories.
Pros
Fast and low effort
Cons
Words can be hard, especially when considering internationalization
A/B Testing
A/B or split testing is a commonly used method of testing two versions of a website page design, a control (live version) and a variation (proposed version), and seeing which performs better. Seems simple enough, you take your designs and give half to one group and half to another.
This isn’t commonly used in the world of product design, but can be a powerful method, especially when using moderated or unmoderated user tests. We can take our live tests and run A/B tests with users to validate our hypothesis on areas of improvement we think will improve the user experience.
Things to consider
It’s important to remember when performing an A/B test with product design, the control and variation should never be two completely separate designs. They should be similar designs with subtle changes. The designs need to be randomly and equally distributed to groups of users. If you have an odd number of users, you can either add or drop users to ensure the count can be equally divided. Lastly, don’t A/B test too much as it’s not always the whole story and can leave out important information as to why something worked better or how users felt about it.
Pros
Higher confidence level in changes to designs because the users are randomly generated/distributed
Cons
Can be harder to understand the ‘why’
Can be harder to observe emotional responses
Resources