Lab usability testing

We are big believers in the benefits of lab based user testing. Here is more on the five reasons why we think it is the best approach:

• You observe in real time – you see users’ behaviours – ‘what they do’ rather than ‘what they say’, which can be quite different. You hear what they say but also get their whole body language. It is clear when what they say conflicts with what they think. You don’t get this using any other approach
• You can understand users’ goals better – often we want a tester to explore a site in line with their own interests: the moderator will draw out the tester’s goals so they can frame relevant tasks
• You can probe – testers do things you don’t anticipate. They also do things and it’s not clear why. A moderator can then probe to gain insight into what’s causing the problem and possible solutions. Only a moderator working in real time with a tester can get these insights.
• You can guide (if necessary) – Sometimes a tester can misinterpret a task, not find content on which feedback is required, or may need explanation (say on a prototype). A moderator can manage these issues.
• It gets the stakeholders together – the biggest issue on most projects is getting stakeholders to agree what the issues are and what they want to do about them (getting the user evidence is often the easier bit!). Lab based testing provides an environment where stakeholders are ‘forced’ to confront user evidence, and this is the most effective way to get agreement to change.

What is fast and slow thinking?

Navigation, content presentation and many design issues are driven by System 1 (Fast) thinking processes (i.e. thinking that is intuitive, automatic, experience-based, and relatively unconscious). Insights into behaviours driven by fast thinking are best gained by watching what users do, not by asking them about it, as users are not good at articulating how easy a site is to use. Indeed, often if we ask users about these issues, thereby accessing their System 2 (Slow) thinking (i.e. conscious, considered and requiring cognitive effort), what they tell us they are doing often contradicts what we observe them doing.

However, gaining user feedback on the suitability and usefulness of content, whether it meets users’ goals, some design issues and attitudes does require accessing users’ ‘Slow’ thinking.

Usability testing provides insights into both ‘Fast’ and ‘Slow’ thinking by observing behaviours and asking attitudinal questions within the same testing session.

How do we run usability testing sessions?

Project preparation

We plan the research in consultation with the client so the client gets the right solution – not an ‘off the shelf’ one. Prior to the research, we agree with the client’s project manager:

• the key aims of the testing, tester numbers and profiles, testing locations, timescales, etc.
• the most appropriate research methodology
• for lab based moderated testing – the facilitation approach to be adopted, and the set up: either desk based or living room style
• for remote unmoderated testing – tasks and attitudinal questions to be set: it is critical that these are specific, clear and unambiguous so testers will be able to work through these quickly on their own without any clarification, understand when they have completed a task, and be able to complete the test without assistance.
• the screener to be used in tester recruitment
• the devices to be used for testing: we can undertake testing on desktop and laptop, and on iOS and Android tablets and smartphones

Testers and recruitment

The number of testers required in any usability testing programme depends on a range of factors e.g. what it is wished to achieve, the budget and time available and the site’s target audiences. However, our experience of undertaking testing on hundreds of websites shows that typically a relatively small proportion of usability problems account for the bulk of the bad user experience on a site: these problems can be identified with a small number of testers (e.g. 3 per user type – see ‘How many testers…’). It is seldom particularly valuable or cost effective to test with more than 12 users, even on sites with several different audiences, in order to identify usability issues to be fixed.

However, the exception is with a benchmarking study when at least 100 testers may be required.

We undertake our own recruitment. All potential testers undergo a detailed screening to ensure they match the tester profiles. Details of the testers are sent to the client project manager for approval. Testers are paid a fee for their participation in the usability testing: this ensures their participation and demonstrates appreciation of the value of their involvement.

Prior to the session, we ensure testers receive the right information (e.g. they need to be aware they are being recorded and have consented to this) and manage tester expectations to ensure they are in a positive frame of mind to take part in the research.

Usability Testing

The ‘core’ of lab based usability testing is the day when the client stakeholders observe some testing (usually 3-4 testers) and then take part in the discussion to agree issues and actions.

If more than 3-4 testers are involved in the research we will conduct these in advanced of the observed testing day and feed the outcomes into the facilitated discussion.

Testing can be undertaken a variety of locations:

• At our studio facilities – clients can watch this remotely if required
• At other studio facilities e.g. we have London labs we use on a regular basis
• Using a mobile usability lab in other suitable testing locations
• For hard to reach testers, who cannot come to the lab, we conduct the research remotely using screen sharing software.

Typically lab based testing sessions will be conducted as follows:

• During observed testing, the day is split into two halves, with user testing (usually 3-4 testers) in the morning and a facilitated discussion session in the afternoon
• The testing session can be observed by up to 8 client stakeholders (ideally all those involved in implementing the research outputs should attend the observed testing including third party site developers if used)
• The testing sessions are facilitated by one of our consultants using a user led approach. The observers are active participants, capturing issues as they arise. A second Web Usability consultant facilitates this process in the observation room.
• We like to hold a discussion session after the research to discuss the outcomes. This session enables the development team, within a single day, to take ownership of the research results, develop a shared view of the issues and agree solutions that fit within their constraints.