Download this product sheet in PDF format
Why and when
Benchmarking the user experience can be used to obtain:
- A user based measure about the experience on a site before and after changes to the site information architecture, navigation or content, in order to assess the impact of those changes
- A user based comparison of a site with competitor sites, in order to assess the comparative user experience
WUP recommends facilitated remote task focused testing with target users in order to obtain both objective and subjective measures of success.
WUP also recommends that ‘added value’ be obtained from the benchmarking exercise by identifying areas where usability improvements can be made to the client’s site.
Our approach
WUP bases user experience benchmarking on the usability standard ISO 9241-11: (1998) (Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability). This part of the standard deals with the extent to which a product can be used by specified users to achieve specified goals. The three key components of this standard are:
- Effectiveness – can tasks be completed by users
- Efficiency – can the task be achieved easily by the user in a ‘reasonable’ time
- Satisfaction – how satisfied or otherwise was the user with the process and content to achieve the task
WUP’s approach to user experience benchmarking comprises three elements:
Firstly, the ‘objective’ measures:
- Task success – how many testers completed the task successfully
- The time taken to complete the task
Then tester’s attitudinal feedback on the site and subjective user measures on:
- Ease of use – how easy or difficult did the tester feel it was to complete the task (based on a 5 point scale from ‘very easy’ to ‘very difficult’)
- Satisfaction – how satisfied or otherwise did the tester feel the content and process (based on a 5 point scale from ‘very satisfied’ to ‘very dissatisfied’)
- Perceived time taken to complete the task – did the tester feel that the time taken to complete the task was reasonable and acceptable (based on a 5 point scale from ‘very acceptable’ to ‘very unacceptable’)
Finally, a WUP expert assessment of the process.
Users are, in the main, not good at articulating how easy it is to use a website – indeed, often what they say contradicts what they do, so observing their actual behaviour adds another dimension to understanding the user experience. WUP rates the user experience on the following dimensions:
- Ease of use of the information architecture (IA – the chunking and labelling of content) – how easy or difficult was it for the tester to use the site IA to complete the task (based on a 5 point scale from ‘very easy’ to ‘very difficult’)
- Ease of use of the site navigation (the page layout and elements) – how easy or difficult was it for the tester to use the site navigation to complete the task (based on a 5 point scale from ‘very easy’ to ‘very difficult’)
- Usefulness of the site content – how useful or otherwise was the content in helping the tester complete the task (based on a 5 point scale from ‘very useful’ to ‘not at all useful’)
During the benchmarking sessions, WUP identifies and records usability issues encountered by the tester.
Along with undertaking the expert assessment and identifying usability issues, the other advantage of undertaking facilitated research (rather then automated research) is that the facilitator can observe what the tester is doing and can put the outcomes in context to ensure the quality of recommendations; for example, did the tester ’fail’ on a task because they did not know where to go, or because they misunderstood it.
How we benchmark
Project preparation
We plan the research in consultation with the client so the client gets the ‘right’ solution -not an ‘off the shelf’ one. Prior to the research, we agree with the client’s project manager the key aims of the benchmark testing, the tester profiles, timescales, etc. We prepare a facilitation guide with a series of agreed tasks for the benchmarking for approval by the client’s project manager in advance of the testing sessions.
The process
For each benchmarking exercise, the research process would be undertaken as follows:
- A WUP facilitator would connect individually with each tester in real time using software that allows WUP to view the tester’s computer screen, and to communicate with them either via the PC or via their mobile phone or land line
- WUP would set testers specific tasks, as agreed with the client to achieve on the site(s)
- The testers would use the site with minimal intervention from the WUP facilitator to create a naturalistic web experience as possible
When the tester believes they have completed each task, or chooses to abort the task, the WUP facilitator would:
- Record the objective measures of task success and time taken
- Review the experience with the testers to get the subjective user assessment
- Make the expert assessment
- The testing session, showing the screen being viewed, would also be recorded using Techsmith Morae.
Following all the testing sessions WUP would consolidate all the scores to produce a ‘dashboard’ of benchmarking data that would show:
- Total task success – percentage of testers to complete each task and all tasks within a specified section of the site
- Average time to complete each task and all tasks
- Average user rating by task and for all tasks
- A radar chart of the user assessment dimensions by task and for all tasks
- Average expert rating by task and for all tasks
- A radar chart of the expert assessment dimensions by task and for all tasks
As subsequent rounds of benchmarking are undertaken, the outputs would be presented in a comparison format for specified parts of the site.
Testers
WUP would recruit testers to a profile agreed with the client. For this research, it is important to have sufficient testers of each distinct user type to give adequate insight; distinct user types being defined as users with the same user goals and buying behaviour (rather than, for example, demographic factors). WUP recommends that 30 testers for each distinct user type are used in each benchmarking exercise.
Recruitment
We undertake our own recruitment – we do not use third parties. We believe this allows us to recruit more accurately to our client’s specification, provide greater flexibility, and treat testers in a decent and ethical way. We are experienced at recruiting testers from a wide range of backgrounds and occupations, and with a range of experience.
All potential testers undergo a detailed screening to ensure they match the tester profile – a profile of the testers is sent to the client project manager prior to the testing session for approval. Testers are paid a fee for their participation in the testing: this ensures their participation and demonstrates appreciation of the value of their involvement.
We have a well established process for ensuring testers receive the right information (e.g. they need to be aware they are being recorded and have consented to this) and manage tester expectations to ensure they are in a positive frame of mind to take part in the research.
Outputs
Following each benchmarking exercise, a report would be produced that would include:
Executive summary
- Background and objectives
Benchmarking data:
- Total task success
- Average time to complete each task and all tasks
- Average user rating by task and for all tasks
- Radar chart of the user assessment dimensions by task and for all tasks
- Average expert rating by task and for all tasks
- Radar chart of the expert assessment dimensions by task and for all tasks
- Tester quotes to illustrate the benchmarking data
- Usability issues identified by WUP with recommendations for action
- Annotated screen shots of key pages identifying key usability issues
We would also provide DVDs of the testing sessions, showing the screen being viewed and the audio of the session.
Fees
Most of our work is tailored to individual client’s requirements and we quote an inclusive price for each piece of work. However, the following gives an illustration of typical costs:
- 30 testers £8,500
- 60 testers £16,000
NB this fully inclusive price covers all WUP’s fees, tester recruitment and payment, equipment hire, reports, DVDs, etc.
Terms and conditions
- VAT will be added to all charges at the currently prevailing rate. Payment will be within 30 days on the presentation of our invoice.
- Additional work not identified above would be charged at WUP’s standard rates (£1,000/day for directors, £800 for consultants and £350/day for support staff). Additional expenses not shown above would be charged at cost
- Cancellation of booked sessions will incur cancellation fees as follows: cancellation within 4 weeks of the session, 50% of the fee is payable; cancellation within 2 weeks of the session, 100% of fee is payable
- The rates quoted assume tester remuneration of £40 plus travel expenses up to a maximum of £10. If tester remuneration needs to be greater than this for any reason (e.g. securing individuals with high time value or on short lead times) then any costs above the basic charge will be invoiced at cost.
- It is assumed that the client will invite the observers to research sessions
Return to top