Chatbot – user research and testing would make them better

February 1, 2018

Most current chatbots are not very good. This makes it very easy for people to be sceptical about them and ignore the whole approach. However, this is often because the user’s needs were not fully understood or because they have not been adequately tested with users.

We spend a lot of our time looking for good chatbots and understanding what makes them good or bad. Below are three chatbots which we think are an interesting starting point. Whilst none of them are perfect, we believe they offer some interesting insights.


Best Designed Chatbot (Aerie)

Aerie is a body positive lingerie & swimwear brand. We think they have one of the best designed Facebook chatbots.

What’s good about it:

  • Persona: The chatbot’s persona is truly authentic and in line with the brand values “Honest” & “Fun”. Chatting with this Chatbot is how you would expect to interact with an Aerie employee.
  • Conversation Paths: Using the chatbot is incredibly intuitive. Navigation language is always clear and it typically takes less than 4 clicks to get to the desired content
  • Rich Content: Aerie use a combination of text, pictures, videos & emojis. The picture content is exceptional and make all the products look desirable.

How it could be improved:

  • Natural Language Capabilities: Aerie uses limited Artificial Intelligence and conversations rely entirely on using buttons or links. Customers who know what they want will have to navigate to find it. Improving the Natural Language Processing would allow these customers to search without navigating.
  • Navigation Back: Navigating through a “Happy Path” is intuitive. However, it can be difficult to start conversations again or navigate to a different path. This could be resolved by regularly adding a “home” buttons or accepting natural language such as typing “start again”.
  • Handling of Errors: In human conversation there are no errors. If we do not understand something, we ask for clarification. The same applies with Conversational interfaces. Chatbots should be designed so that users can always get to what they want (Error handling is quite an extensive topic. We will be writing a separate blog on this). Unfortunately, if Aerie does not understand, it provides customers with a generic response and suggests they email customer service. This is likely to prove frustrating.


Most talked about Chatbot (1800 flowers)


The 1800 flowers Facebook chatbot has received much media coverage including a recommendation from Facebook’s CEO Mark Zuckerberg. Despite this, we don’t think it’s one of the best chatbots.

What ‘s good about it:

  • Integrated Payments: 1800 flowers allows you to order and buy flowers directly through Facebook. Customers can even pay through Facebook. This makes the purchasing experience seamless. Facebook payment is currently only available in the US.
  • Memory: 1800 flowers remembers customers who they have spoken to before. This is useful as returning customers do not have to go through unnecessary introductions for a second time.

How it could be improved:

  • Lack of Empathy: Using a chatbot should feel like a conversation. The 1800 flowers chatbot gives instructions and takes orders. This makes the conversation feel forced and unnatural. A simple way to build empathy into chatbots is to ask; “how would I deal with this customer if they called me?”
  • Response Length: Respect users’ time. Responses and questions should be short and to the point. 1800 flowers has a number of unnecessarily lengthy questions (~40 words). Our analysis found that customers typically read the first and last sentence when responses were too long.
  • Poor Photo Quality: Many of the 1800 flower pictures are stretched or grainy. This appears to be because of not adapting their website content appropriately. Unfortunately this makes many flowers unappealing.
  • Natural Language & Error Handling: This chatbot does not use Artificial Intelligence. Customers are, therefore, forced to use the buttons and links to navigate to products.


Best use of Artificial Intelligence in a chatbot (The Guardian)

The significant improvement in artificial intelligence is one of the key reasons why conversation interfaces are becoming popular, in particular there has been significant improvement in Natural Language Processing. This is the ability for a computer to understand what a customer has written or spoken and then respond in the most relevant manner. The Guardian chatbot is a good example of this.

What’s good about it:

  • Natural Language Understanding: The Guardian uses natural language understanding to help users search for content. For example, putting “sports” in front of the word “headlines” the Guardian knows that customers want to see sports. Not the news headlines. Additionally, when customers search for a specific subject, (e.g. “Brexit”) the Guardian knows you are looking for specific types of articles. This solves one of the shortcomings of the previously mentioned chatbots. Customers who know what they are looking for should be able to search for it.
  • Personalised Content: Users can sign up to personalised news updates. This allows the Guardian to proactively reach out to their audience throughout the day, week or year.

How it could be improved:

  • Error Handling: The Guardian gives a general fallback response when it doesn’t understand (i.e I don’t understand). This could be improved by asking leading questions to guide the users to their desired content.
  • Subscription Design: Users have to go to a menu option to subscribe to content. This can cause navigation challenges. It would be more useful if users were able to subscribe to content more easily. For example, by typing ‘subscribe’ below an article they have just read.

We have picked these examples because they have some great features but, we suspect, more discovery research and more user testing would have made them a whole lot better!