In general, we think lab based moderated user testing produces the best outcomes for five main reasons:
- You observe in real time – you see users’ behaviours – ‘what they do’ rather than ‘what they say’, which can be quite different. You hear what they say but also get their whole body language. It is clear when what they say conflicts with what they think. You don’t get this using any other approach
- You can understand users’ goals better – often we want a tester to explore a site in line with their own interests: the moderator will draw out the tester’s goals so they can frame relevant tasks
- You can probe – testers do things you don’t anticipate. They also do things and it’s not clear why. A moderator can then probe to gain insight into what’s causing the problem and possible solutions. Only a moderator working in real time with a tester can get these insights.
- You can guide (if necessary) – Sometimes a tester can misinterpret a task, not find content on which feedback is required, or may need explanation (say on a prototype). A moderator can manage these issues.
- It gets the stakeholders together – the biggest issue on most projects is getting stakeholders to agree what the issues are and what they want to do about them (getting the user evidence is often the easier bit). Lab based testing provides an environment where stakeholders are ‘forced’ to confront user evidence and this is the most effective way to get agreement to change.
For hard to reach testers, who cannot come to the lab, we can conduct moderated research remotely using screen sharing software.
However, moderated research isn’t the cheapest approach, and there are circumstances where remote unmoderated testing can offer real benefits, for example:
- Larger tester numbers – even though we know most usability issues will be surfaced with only a few users (i.e. 3-6 see ‘how many testers are enough‘), in order to have greater credibility within their organisation clients sometimes want to test with more users. Remote unmoderated testing is a good way to build up numbers especially when used in conjunction with lab based testing which provides the insights not available from remote testing.
- Site usability monitoring – after a site is launched there is often a desire for a low cost way to monitor usability on an ongoing basis. If issues are identified these can then be investigated, in more detail, using lab based testing
- Benchmarking – clients sometimes need a measure of a site’s usability against which they can assess subsequent performance. Larger tester numbers are required to make this meaningful. Unmoderated remote testing is a cost effective way to do this.
- Speed – When quick feedback is required some types of remote testing can be done within hours by using existing panels of users
- Low budgets – if you really can’t afford lab based testing then small scale remote unmoderated testing has a definite role: any testing is better than no testing
Unmoderated testing works best when there are clear straightforward user journeys to be tested. Where a more exploratory user-led form of testing is required, we would recommend moderated lab based testing as the better approach.
With lab based moderated testing…
…we adopt a user-led goal-oriented approach to usability testing. We develop a facilitation guide for each project that encourages a user led journey within the client’s project objectives; this means the tasks we ask users to undertake are relevant to them. We ask testers about their circumstances and needs at the start of each testing session. This provides valuable user insight on the goals they wish to achieve on the site, and helps guide the site exploration, by enabling us to frame tasks and make them relevant to the testers whilst exploring areas of the site that the site owner wishes to investigate. Making the testing relevant to the testers’ ‘real world’ increases their engagement in the testing and the quality of the outcomes. This way we can surface both strategic issues (how users want to interact with the site) and tactical issues (the usability issues that enable or inhibit user goals being achieved). The research data that results from this process will provide more insightful and relevant results.
The facilitation approach Web Usability adopts will depend on the project objectives and elements of the site under test – either real time think aloud or retrospective think aloud, or a combination of the two; there are pros and cons to both:
- Real time think aloud requires the tester to think out loud whilst they are using the site; a stream of consciousness of what they think and how they feel about the site at that point in real time, not subsequently, when testers might post-rationalise their actions. This approach is appropriate when exploring looser scenario based user journeys through a site, understanding attitudes to the content as well as the ease, or otherwise, of finding that content. The think aloud protocol enables a better understanding of the tester’s mental model and interaction with the site – how the tester approaches the site, their expectations of the site and their perceptions of the experience. It also helps to inform an understanding of the language and terminology the tester expects and comprehends. However, it can arguably interrupt the user journey, as it provides the tester with pauses and the opportunity to take an alternative path
- Retrospective think aloud is more appropriate when testing a process where timing or continuous flow might be important. With this approach, testers work through a task uninterrupted and are then asked to talk through a replay of the task; this might be particularly important when testing transactional elements, such as the check-out process, and where perceived speed through the process may be a major contributor to user satisfaction.
Web Usability would agree with the client the most appropriate methodology depending on the project objectives and the interface to be tested.
Web Usability facilitators adopt an open, non-directional form of facilitation. The former prevents users giving yes/no answers enabling greater qualitative insight, and the latter means the testers are not asked leading questions.
We recommend the use of eye tracking for lab based usability testing with desktop, laptop and tablet as this provides greater insights into users’ behaviours. For testing with smartphone devices we use screen mirroring to project and capture the screen, and appropriately placed overhead cameras to capture the tester’s hand movements and gestures.
With remote unmoderated testing…
…we adopt one of two principal approaches:
- Think aloud – testers review a site and talk out loud about what they think. They can be prompted with a series of questions and tasks. What the tester views and says is recorded. The think aloud approach provides qualitative data and is better for understanding usability issues with a site and to gain insights into how to address these; for example, if there’s a period of inactivity, the recording will help reveal if the tester is interested and reading, or if they are confused.
- Survey based – Testers are given a series of tasks on a site. They attempt to complete the tasks. They can also be asked questions about the tasks. Outputs are task success rates and tester’s written comments. The survey based approach is quantitative and so better for monitoring and benchmarking projects and is best when used with straightforward tasks.
In both cases people test the web sites in their own homes, workplaces or ‘out and about’ using their own devices. There is no facilitator so testers are ‘on their own’: in some ways this is great because it’s more natural – but equally it requires absolute clarity about objectives and tasks to maximise the value of the research. Our experience ensures that testers will be able to work through the test.
We choose the appropriate software to run the research. There are now a large number of products on the market, all with their pros and cons, some easy to use, others less so. Having evaluated them all we know which one will be best for your requirements.
During observation of the testing, observers are asked to record issues using cognitive mapping techniques for subsequent discussion, in order to capture their immediate reactions to the testers’ experiences. Subsequently, a Web Usability consultant facilitates a discussion of the issues to identify the priority areas to be fixed. This discussion means that the development team takes ownership of the research results, and develops a collective view of the priority issues to be rectified that enables the identification of actionable results.
Web Usability consultants bring to these discussions their experience of watching a large number of testers on a range of different websites and devices, as well as a detailed understanding of usability issues. As the situation demands, we draw on this experience to ensure the discussions are guided by this knowledge.
We plan the research in consultation with the client so the client gets the right solution – not an ‘off the shelf’ one. Prior to the research, we agree with the client’s project manager:
- the key aims of the testing, tester numbers and profiles, testing locations, timescales, etc.
- the most appropriate research methodology
- for lab based moderated testing – the facilitation approach to be adopted, and prepare a facilitation guide for the testing
- for remote unmoderated testing – tasks and attitudinal questions to be set: it is critical that these are specific, clear and unambiguous so testers will be able to work through these quickly on their own without any clarification, understand when they have completed a task, and be able to complete the test without assistance.
- the screener to be used in tester recruitment
- the devices to be used for testing: we can undertake testing on desktop and laptop, and on iOS and Android tablets, phablets and smartphones
Testers and recruitment
The number of testers required in any usability testing programme depends on a range of factors e.g. what it is wished to achieve, the budget and time available and the site’s target audiences. However, our experience of undertaking testing on hundreds of websites shows that typically a relatively small proportion of usability problems account for the bulk of the bad user experience on a site: these problems can be identified with a small number of testers (e.g. 3 per user type – see ‘How many testers…’). It is seldom particularly valuable or cost effective to test with more than 12 users, even on sites with several different audiences, in order to identify usability issues to be fixed.
However, the exception is with a benchmarking study when at least 100 testers may be required.
We usually undertake our own recruitment. We believe this allows us to recruit more accurately to our client’s specification, provide greater flexibility, and treat testers in a decent and ethical way. We are experienced at recruiting testers from a wide range of backgrounds and occupations, and with a range of experience, including testers with visual, motor or cognitive impairment.
All potential testers undergo a detailed screening to ensure they match the tester profiles. Details of the testers are sent to the client project manager for approval. Testers are paid a fee for their participation in the usability testing: this ensures their participation and demonstrates appreciation of the value of their involvement.
We have a well-established process for ensuring testers attend lab based testing or participate in remote testing; our no-show rate is less than 1% (normal market research no-show rates are about 10%). We ensure testers receive the right information (e.g. they need to be aware they are being recorded and have consented to this) and manage tester expectations to ensure they are in a positive frame of mind to take part in the research.
The ‘core’ of lab based usability testing is the day when the client stakeholders observe some testing (usually 3-4 testers) and then take part in the discussion to agree issues and actions. If more than 3-4 testers are involved in the research we will conduct these in advanced of the observed testing day and feed the outcomes into the facilitated discussion.
Testing can be undertaken a variety of locations:
- At our studio facilities – clients can watch this remotely if required
- At other studio facilities e.g. we have London labs we use on a regular basis
- Using a mobile usability lab in other suitable testing locations
- For hard to reach testers, who cannot come to the lab, we conduct the research remotely using screen sharing software.
The lab based testing sessions are conducted as follows:
- The observed testing day is split into two halves, with user testing (usually 3-4 testers) in the morning and a facilitated discussion session in the afternoon
- The testing session can be observed by up to 10 client stakeholders (ideally all those involved in implementing the research outputs should attend the observed testing including third party site developers if used)
- The testing sessions are facilitated by a Web Usability consultant using a user led approach. The observers are active participants, capturing issues as they arise. A second Web Usability consultant facilitates this process.
- A discussion session is held after the research to discuss the outcomes. This session enables the development team, within a single day, to take ownership of the research results, develop a shared view of the issues and agree solutions that fit within their constraints.
With remote unmoderated testing testers are sent instructions and a link to undertake the research and given a specified set time period to complete the test.