Disabled testing

WUP offers Usability Testing with Disabled Users, to assess sites’ practical accessibility to people who have disabilities. Sites are typically tested by users with no vision, low vision, and motor impairments.
Disabled user testing

Why and when
Our approach
How we do it
Outputs
Fees

Download this product sheet in PDF format

Why and when

Conducting usability tests with disabled users is an essential part of evaluating the accessibility of your site. When done in combination with an accessibility audit, it forms a comprehensive accessibility evaluation of your site, as recommended by the W3C Web Accessibility Initiative.

The testing reveals the issues that interfere with a good user experience for disabled users – how users achieve their goals and the usefulness and relevance of the content – surfacing both behaviours and attitudes. Improving the user experience for disabled users increases user satisfaction and:

Helps you comply with your obligations under the Disability Discrimination Act (part III) and the Disability Equality Duty.
Increases your audience reach to key demographics. Accessible sites can be used by more people – people with disabilities, older people, people with low literacy, people who are not fluent in the language of the site, people with low bandwidth connections to the Internet, people with older technologies, and new and infrequent Web users
Increases conversion rates on e-commerce sites and sign-up rates for transactions on public sector sites
Increases the successful take up of information on information sites
Increases return visits and recommendations
Shows up usability problems encountered by all users

Usability testing, applied well, can also help organisations become more user focused by communicating the insights gained from the research.

Usability Testing with Disabled Users is most effective when applied to a working website, after any basic technical problems shown by an accessibility audit have been resolved.

Our approach

The approach we adopt brings together rigorous user-focused usability research combined with experience of users with disabilities and a process to produce agreed and actionable results.

We adopt a user-led goal-oriented approach to usability testing that surfaces testers’ behaviours and actions to help inform thinking about both strategic issues (how users want to interact with the site) and tactical issues (the usability issues that enable or inhibit user goals being achieved). We use a facilitation guide that encourages a user led journey, with a combination of free exploration, broad scenarios, and loosely defined tasks based on the tester’s preferences.

We do not use a highly scripted approach employing a set of tightly pre-defined tasks as this does not necessarily address the issues that users are most interested in. We believe that testers will engage more realistically in the testing process if they are asked to undertake activities in which they are interested and that are relevant to them in their ‘real’ lives. The research data that results from this process will provide more insightful and relevant results.

Action Oriented Process

A key element of our approach is to encourage all those in an organisation who can influence the implementation of the session’s outcomes to attend at least some of the usability testing session and a discussion facilitated by WUP. During the testing sessions, observers are asked to record issues using cognitive mapping techniques for subsequent discussion, in order to capture their immediate reactions to the testers’ experiences. Subsequently, a WUP consultant facilitates a discussion of the issues to identify the priority areas to be fixed. This discussion means that the development team takes ownership of the research results, and develops a collective view of the priority issues to be rectified that enables the identification of actionable results.

WUP consultants bring to these discussions their experience of watching a large number of testers on a range of different web sites, as well as a detailed understanding of usability issues. As the situation demands, we draw on this experience to ensure the discussions are guided by this knowledge.

How we test usability with disabled users

Depending on the number of users who will test a site, our research typically has four elements:

Project preparation
Recruitment
Unobserved user testing
Observed user testing

N.B. the number of testers required in any usability testing programme depends on a range of factors (e.g. what it is wished to achieve, the budget and time available and the site’s target audiences). However, our experience of undertaking testing on hundreds of websites shows that typically a relatively small proportion of usability problems account for the bulk of the bad user experience on a site. These problems can be identified with a small number of testers (e.g. 3). It is seldom particularly valuable or cost effective to test more than 12 users, even on sites with several different audiences.

Project preparation

We plan the research in consultation with the client so the client gets the ‘right’ solution – not an ‘off the shelf’ one. Prior to the research, we agree with the client’s project manager the key aims of the testing, the tester profiles, testing locations, timescales, etc. We recommend the modes of disability that should be tested, and prepare a facilitation guide for the testing sessions for approval by the client’s project manager in advance of the testing sessions.

Recruitment

We undertake our own recruitment – we do not use third parties. We believe this allows us to recruit more accurately to our client’s specification, provide greater flexibility, and treat testers in a decent and ethical way. We are experienced at recruiting testers from a wide range of backgrounds and occupations, and with a range of experience, disabilities, and assistive technologies.

All potential testers undergo a detailed screening to ensure they match the tester profile – a profile of the testers is sent to the client project manager prior to the testing session for approval. Testers are paid a fee for their participation in the usability testing: this ensures their participation and demonstrates appreciation of the value of their involvement.

Unobserved User Accessibility Testing

The unobserved user accessibility testing is conducted as follows:

The unobserved testing is conducted either at our facilities or using a mobile usability laboratory in suitable testing locations (Mobile usability laboratory consists of: Laptop running Techsmith Morae, with webcam and microphone, to record screen, tester’s face, audio). We also preload our systems with any required assistive software such as JAWS screenreader.
The testing sessions are facilitated by a WUP consultant using a user led approach (see ‘Our Approach‘ above)
Following the unobserved testing, an interim report is produced that summarises the research outputs to inform the people attending the observed user testing

Observed User Accessibility Testing

The observed user accessibility testing is conducted as follows:

The observed testing day is split into two halves, with user testing (3 testers) in the morning and a facilitated discussion session in the afternoon
The observed testing session is conducted either in our observation studios or other suitable facilities and observed by up to 8 client stakeholders (ideally all those involved in implementing the research outputs should attend the observed testing including third party site developers if used)
The testing sessions are facilitated by a WUP consultant using a user led approach (see ‘Our Approach‘ above)
The observers are active participants, capturing issues as they arise. A second WUP consultant facilitates this process. (See ‘Action Orientated Process‘ above)
A discussion session is held after the research to discuss the outcomes and identify the priority issues to be fixed. This session enables the development team to take ownership of the research results, and develop a collective view of the priority issues to be addressed. This approach means that the team will, within a single day, develop a shared view and have a detailed knowledge, and ownership, of the issues raised, and will agree the implications of these issues, the required actions and the priorities. (See ‘Action Orientated Process‘ above)
The sessions are recorded onto a DVD capturing the test screen outputs, a video of the tester, and audio outputs for analysis purposes and subsequent viewing by the client

Outputs

WUP undertakes a qualitative analysis of the data. The data sources comprise the ‘real time’ issues captured by observers during the testing sessions and the DVD recordings, which are analyzed following the user research. Usability issues are sorted according to our Usability Framework, and used to support and augment the conclusions and actions agreed during the facilitated discussion session.

The outputs of the research are:

Interim report of the outcomes of the unobserved testing
A final report covering:
- Background and objectives for the specific usability testing session
- Key findings from the usability testing session covering the key usability issues
- Verbatim tester comments to support the issues identified
- Summary of facilitated discussion outcomes
- A list of recommendations and a prioritized action plan as agreed at the facilitated discussion that would detail timescales and responsibilities (if agreed)
DVDs of the testing sessions showing the screen being viewed, tester head shot and the audio of the session

Fees

Most of our work is tailored to individual client’s requirements and we quote an inclusive price for each piece of work. However, costs are largely determined by the number of users who test a site, so approximate costs can be worked out by combining the cost of an observed session (i.e. 3 testers followed by a facilitated debrief) typically £5,450 (if conducted at our facilities) plus the cost of any additional unobserved testers at £450 per tester. Therefore, a 6 user study would cost £6,800. (N.B. this is a fully inclusive price that covers all WUP’s fees, tester recruitment and payment, observation studios, equipment hire, reports, DVDs and travel expenses etc.)