Testing and Validation
What, why and when.
Some people think that usability is very costly and complex and that user tests should be reserved for the rare web design project with a huge budget and a lavish time schedule. Not true. Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford. Sometimes however, there is a clear need to test with more users. Just keep reading 🙂
Test as soon as possible, especially for new products that require a POC.
Where possible, the designers should help designing the test but not be part of the testing process. In the perfect combination, testing specialists should design, test and interpret the tests.
Designer/Facilitator Service Staff Users Experts Stakeholders
Tests should be designed to cover the most important features and aim to understand if the experience is consistent. It does not have to be perfect.
As soon as you collect data from a single test user, your insights shoot up and you have already learned almost a third of all there is to know about the usability of the design. The difference between zero and even a little bit of data is astounding.
When you test the second user, you will discover that this person does some of the same things as the first user, so there is some overlap in what you learn. People are definitely different, so there will also be something new that the second user does that you did not observe with the first user. So the second user adds some amount of new insight, but not nearly as much as the first user did.
The third user will do many things that you already observed with the first user or with the second user and even some things that you have already seen twice. Plus, of course, the third user will generate a small amount of new data, even if not as much as the first and the second user did.
As you add more and more users, you learn less and less because you will keep seeing the same things again and again. There is no real need to keep observing the same thing multiple times, and you will be very motivated to go back to the drawing board and redesign the site to eliminate the usability problems.
After the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new.
Testing the experience for all users is important, always test with the service support team in the areas they are touching the service. Make sure they understand what the end-customer goes through so they can empathise and offer better support.
What happens to everyone touched by the experience in between the interactions with the system(s), it is crucial.
How they understand a completed or an abandoned journey is highly relevant to the product/service.
Also the successful or failure of implemented solutions, are highly dependant on context. Always add the context note to the analysis of the result.
The behaviour of users is highly dependant on their personality and context.
When To Test More Users
You need to test additional users when a website has several highly distinct groups of users. The formula only holds for comparable users who will be using the site in fairly similar ways.
If, for example, you have a site that will be used by both children and parents, then the two groups of users will have sufficiently different behavior that it becomes necessary to test with people from both groups. The same would be true for a system aimed at connecting purchasing agents with sales staff.
Even when the groups of users are very different, there will still be great similarities between the observations from the two groups. All the users are human, after all. Also, many of the usability problems are related to the fundamental way people interact with the Web and the influence from other sites on user behavior.
In testing multiple groups of disparate users, you don’t need to include as many members of each group as you would in a single test of a single group of users. The overlap between observations will ensure a better outcome from testing a smaller number of people in each group. I recommend:
3–4 users from each category if testing two groups of users
3 users from each category if testing three or more groups of users (you always want at least 3 users to ensure that you have covered the diversity of behavior within the group)
Methods and activities
A/B Testing ›
Validating a service concept ›