The main reason is that it is better to distribute your budget for user testing across many small tests instead of blowing everything on a single, elaborate study. Let us say that you do have the funding to recruit 15 representative customers and have them test your design. Great. Spend this budget on 3 studies with 5 users each!

You want to run multiple tests because the real goal of usability engineering is to improve the design and not just to document its weaknesses. After the first study with five participants has found 85% of the usability problems, you will want to fix these problems in a redesign.

After creating the new design, you need to test again. Even though I said that the redesign should “fix” the problems found in the first study, the truth is that you think that the new design overcomes the problems. But since nobody can design the perfect user interface, there is no guarantee that the new design does in fact fix the problems. A second test will discover whether the fixes worked or whether they didn’t. Also, in introducing a new design, there is always the risk of introducing a new usability problem, even if the old one did get fixed.

Also, the second study with 5 users will discover most of the remaining 15% of the original usability problems that were not found in the first round of testing. (There will still be 2% of the original problems left — they will have to wait until the third study to be identified.)

Finally, the second study will be able to probe deeper into the usability of the fundamental structure of the site, assessing issues like information architecture, task flow, and match with user needs. These important issues are often obscured in initial studies where the users are stumped by stupid surface-level usability problems that prevent them from really digging into the site.

So the second study will both serve as quality assurance of the outcome of the first study and help provide deeper insights as well. The second study will always lead to a new (but smaller) list of usability problems to fix in a redesign. And the same insight applies to this redesign: not all the fixes will work; some deeper issues will be uncovered after cleaning up the interface. Thus, a third study is needed as well.

The ultimate user experience is improved much more by 3 studies with 5 users each than by a single monster study with 15 users.

Important!

As recommended in the testing and validation – the iteration activity needs to be documented to describe the reasons for change and the context. In time, this data will become very important, especially if the system relies on prediction.