Usability testing

Usability testing reveals the issues that interfere with a good user experience – how users achieve their goals and the usefulness and relevance of content.

• Why and when
• Our approach
• How we usability test
• Outputs
• Fees

Download this product sheet in PDF format

Why and when

Usability testing reveals the issues with a web site, mobile site or app that interfere with a good user experience – how users achieve their goals and the usefulness and relevance of the content – surfacing both behaviours and attitudes. Improving the user experience increases user satisfaction and:

• Increases conversion rates and transaction completion rates
• Increases the successful acquisition of information
• Increases return visits and recommendations
• Helps to build brand value

It also helps site owners to understand whether to site is helping to achieve the organisation’s online goals. Also, usability testing, applied well, can help organisations become more user focused by communicating the insights gained from the research.

Usability testing can be used at all stages in a website or app’s development from paper prototypes to final designs.

Our approach

The approach we adopt brings together rigorous user-focused usability research combined with a process to produce agreed and actionable results.

User Focused research

Effective usability testing comprises two key user focussed elements:

• can users find the content they want
• is what they find useful and relevant

Users are, in the main, not good at articulating how easy it is to find content – indeed, often what they say contradicts what they do. Therefore, to assess ‘how easy it is for users to find content’ it is necessary to look at behaviours, not attitudes, which is best done individually rather than in groups.

We adopt a user-led goal-oriented approach to usability testing. We develop a facilitation guide for each project that encourages a user led journey within the client’s project objectives; this means the tasks we ask users to undertake are relevant to them. We ask testers about their circumstances and needs at the start of each testing session. This provides valuable user insight on the goals they wish to achieve on the site, and helps guide the site exploration, by enabling us to ‘frame’ tasks and make them relevant to the testers whilst exploring areas of the site that the site owner wishes to investigate.

Making the testing relevant to the testers’ ‘real world’ increases their engagement in the testing and the quality of the outcomes. Through this, we can surface both strategic issues (how users want to interact with the site) and tactical issues (the usability issues that enable or inhibit user goals being achieved). The research data that results from this process will provide more insightful and relevant results.

The facilitation approach WUP will adopt will depend on the project objectives and elements of the site under test – either real time think aloud or retrospective think aloud, or a combination of the two; there are pros and cons to both:

• Real time think aloud requires the tester to think out loud whilst they are using the site – a stream of consciousness of what they think and how they feel about the site at that point in real time, not subsequently, when testers might post-rationalise their actions. This approach is appropriate when exploring looser scenario based user journeys through a site, understanding attitudes to the content as well as the ease, or otherwise, of finding that content. The think aloud protocol enables a better understanding of the tester’s mental model and interaction with the site – how the tester approaches the site, their expectations of the site and their perceptions of the experience. It also helps to inform an understanding of the language and terminology the tester expects and comprehends. However, it can arguably interrupt the user journey, as it provides the tester with pauses and the opportunity to take an alternative path
• Retrospective think aloud is more appropriate when testing a process where timing or continuous flow might be important. With this approach, testers work through a task uninterrupted and are then asked to talk through a replay of the task; this might be particularly important when testing transactional elements, such as the check out process, and where perceived speed through the process may be a major contributor to user satisfaction.

WUP would agree with the client the most appropriate methodology depending on the project objectives and the interface to be tested.

WUP facilitators adopt an open, non directional form of facilitation. The former prevents users giving yes/no answers enabling greater qualitative insight, and the latter means the testers are not asked leading questions.

Action Oriented Process

A key element of our approach is to encourage all those in an organisation who can influence the implementation of the session’s outcomes to attend at least some of the usability testing session and a discussion facilitated by WUP. During the testing sessions, observers are asked to record issues using cognitive mapping techniques for subsequent discussion, in order to capture their immediate reactions to the testers’ experiences. Subsequently, a WUP consultant facilitates a discussion of the issues to identify the priority areas to be fixed. This discussion means that the development team takes ownership of the research results, and develops a collective view of the priority issues to be rectified that enables the identification of actionable results. The benefits of this process are:

• Shared understanding and ownership of the issues and solutions
• Solutions that ‘fit’ your organization
• Reduced need for post research debriefs and, thereby, speeding the process from research to action.

WUP consultants bring to these discussions their experience of watching a large number of testers on a range of different websites, as well as a detailed understanding of usability issues. As the situation demands, we draw on this experience to ensure the discussions are guided by this knowledge.

Eyetracking

WUP recommends the use of eye tracking for usability testing – with PCs, tablets and mobile devices – as this provides greater insights into users’ behaviours.
We have the latest in eye tracking equipment – a Tobii T60 – which is ideal for PC based eye tracking studies, and a Tobii X120 for testing with tablets and mobile devices.

The new technology enables eye tracking to be used alongside ‘real time think aloud’ and with ‘retrospective think aloud’ usability testing. Unlike earlier models it is possible to use these research approaches because the eye tracker ‘finds’ the tester’s eyes when they look away and then back at the screen. It is possible to observe the eye tracking in real time using ‘Liveviewer’ – so observers can see and hear testers, watch what they are looking at, and see the eye tracking overlaid on the screen as it happens. It also helps make observing usability testing really engaging.

Eye tracking in real time shows precisely what the user is looking at. There is far less doubt about which screen elements are being looked at or have been noticed. This can add significantly to the understanding of the usability issues on a website. This is particularly valuable when a testing session, or a video of it, is being viewed by a client as it makes explicit, without explanation, the usability issues affecting a site. The other major area where eye tracking is valuable is when trying to assess the effectiveness of a specific page or the design and location of elements on a page – for example when comparing two versions of a design e.g. promotions, navigation bars etc.

Significant additional insight can be gained by watching the users’ gaze paths in real time – and real value can be gained from small sample sizes e.g. 3-5 testers. The important thing is being able to see in real time exactly where testers are looking.

We do not tend to recommend producing heatmaps from eye tracking data as we do not believe they add value to the type of research we undertake:

• Heatmaps require large numbers of testers – increasing cost significantly
• They can lead to ambiguous outputs: cumulative eyetracking data is highly context specific, depending on the tasks and user journeys the testers are pursuing. It can also be misleading – does a long fixation mean that someone is looking at something because it is interesting to them, or because it’s very confusing and they’re having to spend time making sense of it.

How we usability test

Depending on the number of users who will test a site, our research typically has four elements:

• Project preparation
• Recruitment
• Unobserved user testing
• Observed user testing

NB the number of testers required in any usability testing programme depends on a range of factors (e.g. what it is wished to achieve, the budget and time available and the site’s target audiences based on target user goals). However, our experience of undertaking testing on hundreds of websites shows that typically a relatively small proportion of usability problems account for the bulk of the bad user experience on a site. These problems can be identified with a small number of testers (e.g. 3 per distinct target user type). It is seldom particularly valuable or cost effective to test more than 12 users, even on sites with several different audiences.

Project preparation

We plan the research in consultation with the client so the client gets the ‘right’ solution -not an ‘off the shelf’ one. Prior to the research, we agree with the client’s project manager the key aims of the testing, the tester profiles, testing locations, timescales, etc. We agree the facilitation approach to be adopted, and prepare a facilitation guide for the testing sessions for approval by the client’s project manager in advance of the testing sessions. We also agree with the project manager the screener to be used in tester recruitment.

Recruitment

We undertake our own recruitment – we do not use third parties. We believe this allows us to recruit more accurately to our client’s specification, provide greater flexibility, and treat testers in a decent and ethical way. We are experienced at recruiting testers from a wide range of backgrounds and occupations, and with a range of experience, including testers with visual, motor or cognitive impairment.

All potential testers undergo a detailed screening to ensure they match the tester profile – a profile of the testers is sent to the client project manager prior to the testing session for approval. Testers are paid a fee for their participation in the usability testing: this ensures their participation and demonstrates appreciation of the value of their involvement.

We have a well established process for ensuring testers attend – our ‘no-show’ rate is less than 1% (normal market research no-show rates are about 10%). We ensure testers receive the right information (e.g. they need to be aware they are being recorded and have consented to this) and manage tester expectations to ensure they arrive in a positive frame of mind to take part in the research.

Unobserved User Testing

The unobserved user testing is conducted as follows. The unobserved testing can be conducted in one of three ways:

1. At our studio facilities
2. Using a mobile usability laboratory in other suitable testing locations
3. Remotely using screen sharing software where suitable testers can not attend at other research premises, or where it’s important that the testing in undertaken in the tester’s ‘natural’ environment (NB Eyetracking is not available for remote testing)

• The testing sessions are facilitated by a WUP consultant using a user led approach (see ‘User Focused Research’ above)
• Following the unobserved testing, the research outputs will be used to inform the people attending the observed user testing

Observed User Testing

The observed user testing is conducted as follows:

• The observed testing day is split into two halves, with user testing (3 testers) in the morning and a facilitated discussion session in the afternoon
• The observed testing session is conducted either in our observation studios or other suitable facilities and observed by up to 8 client stakeholders (ideally all those involved in implementing the research outputs should attend the observed testing including third party site developers if used)
• The testing sessions are facilitated by a WUP consultant using a user led approach (see ‘User focused research’ above). The observers are active participants, capturing issues as they arise. A second WUP consultant facilitates this process. (See ‘Action Orientated Process’ above)
• A discussion session is held after the research to discuss the outcomes and identify the priority issues to be fixed. This session enables the development team to take ownership of the research results, and develop a collective view of the priority issues to be addressed. This approach means that the team will, within a single day, develop a shared view and have a detailed knowledge, and ownership, of the issues raised, and will agree the implications of these issues, the required actions and the priorities. (See ‘Action Orientated Process’ above)
• The sessions are recorded onto a DVD capturing the test screen outputs, a video of the tester, and audio outputs for analysis purposes and subsequent viewing by the client

Outputs

WUP undertakes a qualitative analysis of the data in line with the WUP Usability Framework. The data sources comprise the ‘real time’ issues captured by observers during the testing sessions and the DVD recordings, which are analyzed following the user research. Usability issues are sorted according to our Usability Framework, and used to support and augment the conclusions and actions agreed during the facilitated discussion session.

The outputs of the research are:

A final report covering:

• Executive summary
• Background and objectives for the specific usability testing session
• Key findings from the usability testing session covering the key usability issues
• Verbatim tester comments to support the issues identified
• Annotated screen shots
• Summary of facilitated discussion outcomes
• A list of recommendations and a prioritized action plan as agreed at the facilitated discussion that would detail timescales and responsibilities (if agreed)
• DVDs of the testing sessions showing the screen being viewed, tester head shot and the audio of the session

Client Examples

WUP have undertaken more than 300 usability testing projects for over 80 clients in both the public and private sectors. Recent work includes:

Disney	February 2011, April 2011, May 2011, July 2011
Printerland	August 2011
Boots	June 2010, October 2010, May 2011
Interflora	June 2011, August 2011
Dyson	June 2010, February 2011
Body Shop	June 2011
Orange	2007 – Present

Fees

Most of our work is tailored to individual client’s requirements and we quote an inclusive price for each piece of work. However, the following gives an illustration of typical costs:

• Unobserved user testing £450 per tester
• Observed user testing and facilitated debrief £5,450 (NB this fully inclusive price covers all WUP’s fees, tester recruitment and payment, observation studios, equipment hire, reports, DVDs and travel expenses etc.)

Terms and conditions

• VAT will be added to all charges at the currently prevailing rate. Payment will be within 30 days on the presentation of our invoice
• Additional work not identified above would be charged at WUP’s standard rates (£1,000/day for directors, £800 for consultants and £350/day for support staff). Additional expenses not shown above would be charged at cost
• Cancellation of booked sessions will incur cancellation fees as follows: cancellation within 4 weeks of the session, 50% of the fee is payable; cancellation within 2 weeks of the session, 100% of fee is payable.
• The rates quoted assume tester remuneration of £40 plus travel expenses up to a maximum of £10. If tester remuneration needs to be greater than this for any reason (e.g. securing individuals with high time value or on short lead times) then any costs above the basic charge will be invoiced at cost
• It is assumed that the client will invite the observers to research sessions

Return to top