Researchers estimate that by 2020 we will speak to chatbots more than we speak to our spouses. Obviously, companies that implement chatbots are doing something right. But while there are plenty of effective chatbots on the market, there are also many that don’t quite meet consumers’ needs. So how do you measure the success of your chatbot?
This is the dilemma facing an increasing number of companies that use chatbots as part of their customer experience. Eighty percent of businesses want to implement a chatbot by 2020, but many still face the challenge of gauging the efficacy of the technology.
Google’s Chatbot Analytics platform recently opened up to all, but it is still necessary for businesses to develop and understand their own chatbot success metrics to effectively use the platform. The process of defining the best KPIs for your company’s bot will depend on your business goals and the functions you want your bot to perform.
Here are seven metrics of success you can use to identify opportunities for improvement in your company’s chatbot.
Revenue growth
The first thing any prospective investor wants to know about a company is whether it makes money. Likewise, the best indicator of a chatbot’s value is its financial benefit.
There are many ways to evaluate a bot’s impact on revenue, and the most appropriate method will depend on your bot’s purpose. An interesting wrinkle is that your chatbot can have a knock-on effect in a number of areas.
For example, you can measure a customer service bot’s profitability growth by the amount of money it saves the company compared to maintaining a customer service team 24/7. But you will want to take the bot’s impact on customer service into account. If self-service rates are higher and clients are more satisfied, that will result in repeat customers and higher online sales, thus impacting top-line revenue growth.
Self-service rate
Businesses are happy the moment a user gets exactly what they want from the chatbot without any human input. If your chatbot’s goal is to change a user’s password, you would measure success by the percentage of user interactions that end with this result.
The self-service rate closely correlates with the cost savings aspect of revenue growth — in other words, how much money did your chatbot save by doing its job well?
Satisfaction rate
What better way to find out exactly how well your chatbot is doing than by asking the very people who use it?
Your chatbot can help you determine this metric by asking the key question for the Net Promoter Score: “On a scale of 1-10, how likely is it that you would recommend our chatbot to a friend/colleague?” As a lead indicator of growth, the NPS provides a crucial foundation for understanding the customer experience performance of your chatbot.
Activation rate
At this point, it’s worth reflecting on AARRR and its importance in measuring the success of your business.
In the context of a chatbot, the activation rate refers to a user responding to the chatbot’s initial message with a question or answer that is relevant to your business goals. For example, a chatbot designed to provide you with weather updates would receive an activation rate when you enter your location — thus allowing the bot to provide you with the information.
How can this KPI help? If people are not responding when the weather chatbot first reaches out to them, the botmaster can tinker with it to enable a more satisfactory outcome.
Confusion triggers
Unfortunately, even bots with the most robust natural language processing are unable to understand everything a user says. These errors are a useful indicator for measuring whether you need to improve your chatbot’s matching.
Bear in mind that there are three different triggers, each of which necessitates its own type of response.
The first is simple confusion if the bot cannot understand a comment. A basic “Sorry, I didn’t understand that. Can you ask again in a different way?” response would suffice. Second is if the user sends a number of messages that are outside the remit of your chatbot. After a couple of attempts, it would be worth programming your bot to relay a message that reminds the user of its exact purpose.
The final trigger is if the bot needs to pass a user to a customer service agent after an unsatisfactory interaction.
Each of these will tell you something different about how your chat agent is performing.
Retention rate
Once again referring to AARRR, the retention rate represents the percentage of users who return to the chatbot over a specified period of time. This timespan would vary depending on your bot’s purpose. For example, a fitness chatbot would require daily interaction and would benefit from analysis of its one-day retention.
Artificial intelligence and machine learning rate
How strong is the AI in your chatbot? You can use the percentage of user questions that are correctly understood to measure this. This leads us to the million, if not billion dollar question — can my chatbot learn independently?
Chatbots with machine learning can measure progress by comparing improvement in self-service rates over a period of time without human intervention. An agent with robust machine learning will be able to continually run its own gap analysis to highlight potential areas for improvement.
Millennials’ demand for chatbots is clear. Consumers are asking for simple and effective customer service, but not every chatbot is capable of delivering on this promise out of the gate. In a market that is becoming increasingly crowded, these KPIs can help you keep your chatbot one step ahead of the pack.
Originally poste on VntureBeat byJordi Torras, CEO and founder of Inbenta, an artificial intelligence technology company. November 2017.
Join Inbenta and learn more about AI assistants and chatbots at the AI Assistant Summit and Deep Learning Summit in San Francisco this January 25 & 26. Register now to guarantee your place.