Remote usability testing is a great way to test with hard to reach customers or during a global pandemic!
Remote usability testing: Why do it?
Usability testing is the best way to get detailed insights into the usability problems of a website or app, and how to fix them. Why? Because you can watch what users are actually doing on your site, rather than relying on what they say they are doing.
Traditionally, usability testing has been conducted in a research lab, however, remote usability testing is a great way to speak to hard to reach or international customers. It also allows us to continue delivering high-quality insights during a global pandemic!
Some of the key benefits of remote usability testing:
- Research can be conducted internationally or with hard to reach users
- Live streaming allows observation of research in real-time
- Testing can be conducted on live websites, app or early stage prototypes and wireframes
- Desktop, tablet or smartphone testing are all possible
How we do it
Remote usability testing is conducted using video calling and screen sharing software. Testing can be conducted on all devices and on a website or app at any stage of development. Live streams of the research are available to ensure you never miss a moment.
We adopt a real time think aloud ‘lite’ facilitation approach so not to interfere with the user journey. We observe behaviours to get insights into users ‘Fast’ thinking and this allows us to ask attitudinal questions to access ‘Slow’ thinking about relevant issues e.g. is the content useful.
In order to get the best, unbiased results out of the think aloud protocol, we adopt an open, non-directional form of questioning and let the users lead the session.
Finally, a key element of our approach is to involve influential members of your organisation in the observation of research in order to develop a shared understanding of the issues and develop solutions that ‘fit’ your organisation.
Each project follows the following rough structure:
- Project preparation – We work with you to plan the research, so you get the right solution – not an ‘off the shelf’ one. We are very flexible in our approach to ensure you feel you are getting value for money.
- Testers and Recruitment – we conduct all our recruitment in house. We believe this allows us to recruit more accurately to your specification, provide greater flexibility, and treat testers in a decent and ethical way. Our tester no-show rate is less than 1% (normal market research no-show rates are about 10%).
- Remote usability testing – depending on the number of testers, we will conduct the sessions over the course of a day or two. We encourage you to observe some of the testing via a live stream link. A debrief between sessions ensure we shape the research to give you the insights you need.
What is fast and slow thinking?
Navigation, content presentation and many design issues are driven by System 1 (Fast) thinking processes (i.e. thinking that is intuitive, automatic, experience-based, and relatively unconscious). Insights into behaviours driven by fast thinking are best gained by watching what users do, not by asking them about it, as users are not good at articulating how easy a site is to use. Indeed, often if we ask users about these issues, thereby accessing their System 2 (Slow) thinking (i.e. conscious, considered and requiring cognitive effort), what they tell us they are doing often contradicts what we observe them doing.
However, gaining user feedback on the suitability and usefulness of content, whether it meets users’ goals, some design issues and attitudes does require accessing users’ ‘Slow’ thinking.
Usability testing provides insights into both ‘Fast’ and ‘Slow’ thinking by observing behaviours and asking attitudinal questions within the same testing session.
How do we run usability testing sessions?
We plan the research in consultation with the client so the client gets the right solution – not an ‘off the shelf’ one. Prior to the research, we agree with the client’s project manager:
- the key aims of the testing, tester numbers and profiles, testing locations, timescales, etc.
- the most appropriate research methodology
- for moderated testing – the facilitation approach to be adopted, and the set up: either desk based or living room style
- for unmoderated testing – tasks and attitudinal questions to be set: it is critical that these are specific, clear and unambiguous so testers will be able to work through these quickly on their own without any clarification, understand when they have completed a task, and be able to complete the test without assistance.
- the screener to be used in tester recruitment
- the devices to be used for testing: we can undertake testing on desktop and laptop, and on iOS and Android tablets and smartphones
Testers and recruitment
The number of testers required in any usability testing programme depends on a range of factors e.g. what it is wished to achieve, the budget and time available and the site’s target audiences. However, our experience of undertaking testing on hundreds of websites shows that typically a relatively small proportion of usability problems account for the bulk of the bad user experience on a site: these problems can be identified with a small number of testers (e.g. 3 per user type – see ‘How many testers…’). It is seldom particularly valuable or cost effective to test with more than 12 users, even on sites with several different audiences, in order to identify usability issues to be fixed.
However, the exception is with a benchmarking study when at least 100 testers may be required.
We undertake our own recruitment. All potential testers undergo a detailed screening to ensure they match the tester profiles. Details of the testers are sent to the client project manager for approval. Testers are paid a fee for their participation in the usability testing: this ensures their participation and demonstrates appreciation of the value of their involvement.
Prior to the session, we ensure testers receive the right information (e.g. they need to be aware they are being recorded and have consented to this) and manage tester expectations to ensure they are in a positive frame of mind to take part in the research.
The ‘core’ of remote usability testing is the day when the client stakeholders observe some testing (usually 3-4 testers) via a live stream link and then take part in the discussion to agree issues and actions.
Typically remote usability testing will be conducted as follows:
- During observed testing, the day is split into two halves, with user testing (usually 3-4 testers) in the morning and a facilitated discussion session in the afternoon
- The testing session can be observed by as many stakeholders as you want via a live stream link (ideally all those involved in implementing the research outputs should attend the observed testing including third party site developers if used)
- The testing sessions are facilitated by one of our consultants using a user led approach. The observers are active participants, capturing issues as they arise. A second Web Usability consultant facilitates this process in via a WhatsApp group or similar.
- We like to hold a discussion session after the research to discuss the outcomes. This session enables the development team, within a single day, to take ownership of the research results, develop a shared view of the issues and agree solutions that fit within their constraints.