Lab-based usability testing on desktop, tablet and smartphones, or reaching out remotely to engage larger numbers of testers.
Why do it?
Usability testing is the best way to get detailed insights into the usability problems of a website or app, and how to fix them. Why? Because you can watch what users are actually doing on your site, rather than relying on what they say they are doing.
In general, we think lab based moderated user testing produces the best outcomes. Here’s why:
- You observe in real time and can see users’ behaviours as they happen
- You can better understand users’ goals with expert facilitation
- You can probe when testers do something unexpected
- You can guide (if necessary) when testers cannot find content
- It gets the stakeholders together and forces them to ‘confront’ user evidence
For hard to reach testers, who cannot come to the lab, remote moderated research using screen sharing software offers an effective alternative to lab-based sessions.
We are big believers in the benefits of lab based user testing. Here is more on the five reasons why we think it is the best approach:
- You observe in real time – you see users’ behaviours – ‘what they do’ rather than ‘what they say’, which can be quite different. You hear what they say but also get their whole body language. It is clear when what they say conflicts with what they think. You don’t get this using any other approach
- You can understand users’ goals better – often we want a tester to explore a site in line with their own interests: the moderator will draw out the tester’s goals so they can frame relevant tasks
- You can probe – testers do things you don’t anticipate. They also do things and it’s not clear why. A moderator can then probe to gain insight into what’s causing the problem and possible solutions. Only a moderator working in real time with a tester can get these insights.
- You can guide (if necessary) – Sometimes a tester can misinterpret a task, not find content on which feedback is required, or may need explanation (say on a prototype). A moderator can manage these issues.
- It gets the stakeholders together – the biggest issue on most projects is getting stakeholders to agree what the issues are and what they want to do about them (getting the user evidence is often the easier bit!). Lab based testing provides an environment where stakeholders are ‘forced’ to confront user evidence, and this is the most effective way to get agreement to change.
How we do it
We adopt a real time think aloud ‘lite’ facilitation approach so not to interfere with the user journey. We observe behaviours to get insights into users ‘Fast’ thinking and this allows us to ask attitudinal questions to access ‘Slow’ thinking about relevant issues e.g. is the content useful.
In order to get the best, unbiased results out of the think aloud protocol, we adopt an open, non-directional form of questioning and let the users lead the session.
Finally, a key element of our approach is to involve influential members of your organisation in the observation of research in order to develop a shared understanding of the issues and develop solutions that ‘fit’ your organisation.
Each project follows the following rough structure:
- Project preparation – We work with you to plan the research, so you get the right solution – not an ‘off the shelf’ one. We are very flexible in our approach to ensure you feel you are getting value for money.
- Testers and Recruitment – we conduct all our recruitment in house. We believe this allows us to recruit more accurately to your specification, provide greater flexibility, and treat testers in a decent and ethical way. Our tester no-show rate is less than 1% (normal market research no-show rates are about 10%).
- Usability testing – The ‘core’ of lab based usability testing is the day when the you and your stakeholders observe some testing (usually 3-4 testers) and then take part in a discussion to agree issues and actions.
What is fast and slow thinking?
Navigation, content presentation and many design issues are driven by System 1 (Fast) thinking processes (i.e. thinking that is intuitive, automatic, experience-based, and relatively unconscious). Insights into behaviours driven by fast thinking are best gained by watching what users do, not by asking them about it, as users are not good at articulating how easy a site is to use. Indeed, often if we ask users about these issues, thereby accessing their System 2 (Slow) thinking (i.e. conscious, considered and requiring cognitive effort), what they tell us they are doing often contradicts what we observe them doing.
However, gaining user feedback on the suitability and usefulness of content, whether it meets users’ goals, some design issues and attitudes does require accessing users’ ‘Slow’ thinking.
Usability testing provides insights into both ‘Fast’ and ‘Slow’ thinking by observing behaviours and asking attitudinal questions within the same testing session.
How do we run usability testing sessions?
We plan the research in consultation with the client so the client gets the right solution – not an ‘off the shelf’ one. Prior to the research, we agree with the client’s project manager:
- the key aims of the testing, tester numbers and profiles, testing locations, timescales, etc.
- the most appropriate research methodology
- for lab based moderated testing – the facilitation approach to be adopted, and the set up: either desk based or living room style
- for remote unmoderated testing – tasks and attitudinal questions to be set: it is critical that these are specific, clear and unambiguous so testers will be able to work through these quickly on their own without any clarification, understand when they have completed a task, and be able to complete the test without assistance.
- the screener to be used in tester recruitment
- the devices to be used for testing: we can undertake testing on desktop and laptop, and on iOS and Android tablets and smartphones
Testers and recruitment
The number of testers required in any usability testing programme depends on a range of factors e.g. what it is wished to achieve, the budget and time available and the site’s target audiences. However, our experience of undertaking testing on hundreds of websites shows that typically a relatively small proportion of usability problems account for the bulk of the bad user experience on a site: these problems can be identified with a small number of testers (e.g. 3 per user type – see ‘How many testers…’). It is seldom particularly valuable or cost effective to test with more than 12 users, even on sites with several different audiences, in order to identify usability issues to be fixed.
However, the exception is with a benchmarking study when at least 100 testers may be required.
We usually undertake our own recruitment. All potential testers undergo a detailed screening to ensure they match the tester profiles. Details of the testers are sent to the client project manager for approval. Testers are paid a fee for their participation in the usability testing: this ensures their participation and demonstrates appreciation of the value of their involvement.
Prior to the session, we ensure testers receive the right information (e.g. they need to be aware they are being recorded and have consented to this) and manage tester expectations to ensure they are in a positive frame of mind to take part in the research.
The ‘core’ of lab based usability testing is the day when the client stakeholders observe some testing (usually 3-4 testers) and then take part in the discussion to agree issues and actions.
If more than 3-4 testers are involved in the research we will conduct these in advanced of the observed testing day and feed the outcomes into the facilitated discussion.
Testing can be undertaken a variety of locations:
- At our studio facilities – clients can watch this remotely if required
- At other studio facilities e.g. we have London labs we use on a regular basis
- Using a mobile usability lab in other suitable testing locations
- For hard to reach testers, who cannot come to the lab, we conduct the research remotely using screen sharing software.
A typically lab based testing sessions may be conducted as follows:
- During observed testing, the day is split into two halves, with user testing (usually 3-4 testers) in the morning and a facilitated discussion session in the afternoon
- The testing session can be observed by up to 8 client stakeholders (ideally all those involved in implementing the research outputs should attend the observed testing including third party site developers if used)
- The testing sessions are facilitated by one of our consultants using a user led approach. The observers are active participants, capturing issues as they arise. A second Web Usability consultant facilitates this process in the observation room.
- We like to hold a discussion session after the research to discuss the outcomes. This session enables the development team, within a single day, to take ownership of the research results, develop a shared view of the issues and agree solutions that fit within their constraints.
- With remote unmoderated testing testers are sent instructions and a link to undertake the research and given a specified set time period to complete the test.