If you asked somebody just two years ago about chatbots, the response would have been optimistic. Apps’ retention rates were slowing down, while messaging apps were on the rise. In 2016, it was not uncommon to see articles with headlines like: ‘This is how Chatbots will kill 99% of Apps’.
And there was data to back up the hype. Facebook messenger crossed the 1 billion users mark in 2016, which is also the year it introduced its Bots API, which now boasts 100,000+ bots. Western app developers were also enamored by apps like China’s WeChat, a much more immersive messaging platform whose bots seemed capable of myriad tasks, like calling a taxi or making appointments.
However, being excited about chatbots and actually implementing a working chatbot are two very different things. A chatbot needs to be easy to use, efficient, and get the job done for users to consistently engage with it. Developers and investors constantly run into snags when launching a chatbot. To get at why, one has to revisit what the point of a chatbot is.
What is a chatbot?
A chatbot is any program that mimics real conversations. This can either be embedded in a site or through a third party messaging platform like Facebook Messenger or Slack. Chatbots rely on natural language processing (NLP), the same technology behind virtual assistants such as Google Now or Apple’s Siri.
Chatbots follow three simple steps: understand, act, respond. In the first case, the chatbot processes what the user sends, then acts according to a series of algorithms that interprets what the user said, and then finally picks from a series of appropriate responses. You can actually program your chatbot to respond the same way each time or to respond differently to messages containing certain keywords.
Why are chatbots important for customer experience?
If you had to a pick an app or a chatbot, you’d pick an app for its user interface, but you’d pick the chatbot for its simplicity. A chatbot should be easy to use, or not at all.
As Stefan Kojouharov explains, chatbots are useful in two cases:
Conversational: When an app can’t do it because multiple variable inputs are needed to solve the problem.
Simplicity: When a bot offers the most immediate and direct solution to a person’s problem.
What are the difficulties in building a chatbot?
“Any system or application that relies upon a machine’s ability to parse human speech is likely to struggle with the complexities inherent in elements of speech such as metaphors and similes.”
That’s D. Shewan over at Medium. Because chatbots so heavily rely on natural language processing, they are also constrained by NLP’s limitations. Even the best NLP does not often compare to a human. That would be fine in other use cases, but not for a chatbot. A user will only tolerate a chatbot if it speeds up communication, not hinders it. But at the current stage of NLP it’s very possible that a chatbot understand words, but not necessarily their meaning. There needs to be major improvements in NLP for users to regularly use chatbots.
The other issue is human speech itself. Chatbots try to mimic conversations, but most conversations aren’t linear: discussions restart, there are tangential topics, or multiple topics being discussed at once. This is very tough to follow algorithmically.
How can I build an effective chatbot?
So how do we get over this technological hurdle? The answer: data, data, and more data. One of the best ways to improve a chatbot is to constantly train it and get it to respond to different human interactions. The more data you feed a chatbot, the better it can adapt to human speech and all its idiosyncrasies, and the better you get at achieving that ‘human-like’ conversation level.
As per Rajaih Nuseibeh, successful ‘human-like’ chatbots rely on two things:
Context Understanding: This is the ability to remember and track different aspects of a conversation — location, time, preferences of people — and combine all of the inputs to ‘paint a picture’ of the conversation. Just like humans use surrounding context to inform their interactions, chatbots also need information that keeps the conversation going.
Intent Recognition: This is the ability to extract relevant information from each sentence, word and verb, and understand the intention and the meaning behind it. This allows the use of long complex sentences by users, because the chatbot is able to understand and extract multiple intents.
There are two services that are particularly useful for chatbot training:
Entity Annotation: This refers to extracting ‘units of information’ from sentences or unstructured data – and making it structured. These units can include names, such as people, organizations, and location names, proper nouns. It can also be used to identify numeric expressions such as time, date, money, and percent expressions. This is helpful in training a chatbot to develop ‘context’ in a conversation.
Linguistic Annotation: This refers to assessing the subject of any given sentence. Its a broad genre, but essentially it’s anything to do with analysis of text, whether that be sentiment analysis of the user, or using NLP to answer questions. Through sentiment analysis, a chatbot can understand the ‘tone’ of the user, and see if they’re comfortable, friendly, or upset (and adapt accordingly).
Ultimately, the goal of any chatbot is to maintain natural conversation fluidity, and provide consistent engagement. The more data you give to a chatbot, the smoother its NLP engine runs, and the better the AI experience is. That’s when the hype over chatbots is worth it.