Photo by Luke Chesser

Originally Posted On: https://clevermessenger.com/chatbot-kpi-metrics/

 

Did you know experts predict that 90% of customer interaction in banks is to be automated by the year 2022? It is vital to set Key Performance Indicators (KPIs), to gauge the success of your chatbot. In this article, you learn the metrics to assess how your chatbot is adding value to your growth strategy.

You have just deployed your very first chatbot. Now what? Well, what about knowing how it’s performing? Here are the most important KPI metrics you will need to track.

What Are Chatbot Metrics Or KPIs?

A Metric, or KPI, if you will, is a quantifiable measure used to track and assess the status of a specific business process or outcome. It could also be a measure of success, for example, how many leads generated convert to revenue?

Spending time upfront to define the KPI is critical. It is important to note that it is quite likely that the KPIs measured in the early life of the bot may change over time. These User Metrics may, in time, play less of a role than Response Metrics might as the chatbot matures. And by doing it right, the performance of your bot can be observed, and it’s efficiency improved.

It is useful to group metrics into three areas; 1.) User Metrics, 2.) Message Metrics, and 3.) Response Metrics.

About The User Chatbot KPI

When launching a chatbot, it is vital to measure the take up of the bot with the intended user base. This means the usage of your intended customer and how they’ll interact with the chatbot and its conversational abilities.

1. New Users

This Chatbot KPI is an exceptional indicator of the success of the chatbot. New Users, but cannot be read in isolation. It’s essential to support this metric with others like “Active Users” and “Engaged Users”. It is right to state that customer preferences will change over time and. As such, the amount of interaction with the chatbot will show “New User” declines if the chatbot does not keep up with those changing demands. New, engaged users will keep your chatbot base healthy. Often chatbots go viral. And new users regularly engage with the bot through recommendation by a friend, family member, or colleague. This metric indicates the level of virality by surveying new users as to how they “heard about” the bot.

2. Total Users

This metric is often referred to as a foundational metric. This metric captures the number of people who have used your chatbot even if only once. The trends measured in this metric give a sense of changes in the number of users. Plus, it shows the rate at which new users are joining the chatbot.

3. Active Users

This chatbot KPI are the number of people who interact with the chatbot. Basically measuring the sending of messages over an agreed time frame. It is worth mentioning that this is considered a base metric. And when looked at in isolation is sure to create a misread of the chatbots’ effectiveness. Knowing the number of people reading the chatbots messages is critical.

4. Engaged Users

These users are the ones who send more than one message to the chatbot. In website terms, this is sometimes referred to as “stickiness”. It’s content that keeps the user interacting with the chatbot. This critical metric provides one of the base elements for more advanced metrics. Like goal conversion and CTR. This is the metric most likely to shape decisions on the effectiveness of the chatbot. And perhaps even the ongoing commitment to mature the chatbot. If the bot is unable to sustain a conversation with the users, it’s effectiveness has to be challenged.

Message Chatbot Metrics

User metrics capture the trend in your user base. But a metric to measure individual interactions with your chatbot, are superfluous. Message metrics are the start of the effectiveness of the bot.

Conversation Starter Messages

There are two kinds of message initiations, those started by the bot, and those initiated by the user. Measuring both gives a view of the customer’s willingness to strike up a conversation. Chatbots are created with the express purpose of eliciting some sort of user response. Either by sending a message or replying to a user’s message.

1. Bot Messages

While chatbots are conversational by nature, as mentioned before, the goal of the bot is to trigger some sort of user call to action. Or to automate the process before a human takes over from the bot. Measuring the total number of messages sent by the chatbot to get that user to the point of call to action is precious. If the conversation is too long or too vague, almost certainly, the call to action will be lost. Not only is it right to measure how many messages it takes to elicit a user response, but it is also good to measure the time taken. Measuring the time taken can afford an opportunity to address whether there are any confusing messages, the wording of messages, and the positioning of those messages in the conversation.

2. User Messages

Measuring how many messages are sent by the user. Or how many times the user goes back to the start (or previous step) in the conversation. This provides a sense of the flow of the conversation. If this category is low, it may be that a chatbot is not required at all.

3. Missed Messages

Missed messages are the ones the chatbot doesn’t understand and cannot process. At this point in a conversation, the bot needs to respond in a way the risk of losing a user is reduced. One of the more difficult metrics to calculate and report on is, nonetheless, an important metric. This chatbot metric can be measured in more than one way. The most complex is to measure the time it takes for the user to abandon the conversation. The second, slightly easier, is to measure how many times the chatbot responds with an ‘I don’t know what you mean’ sentiment. This is a chatbot KPI if the bot is expected to interact in different languages, and the risk of “lost in translation” is elevated.

Total Conversation

It is excellent to know how many users get all the way to the end of the conversation. Measuring how many conversations are started and completed on a given day. This metric is weighed directly into the strategic intent of the chatbot. If it is a chatbot designed to gather user information and fault details for a call center, then the call center agents will soon tell if it is useful. If it is a lead generation bot, the measure may be how many leads have been generated, unqualified, or not. This metric, however, is not to be confused with the conversation rate, where leads are ‘converted’ to revenue.

Different Types Of Chatbots

Different businesses will find they require different types of chatbots. Ask yourself what industry you’re in, and what questions your potential customers will need to be answered. Every business is varied, and they all have different needs and goals. The Facebook Messenger chatbot that you create will need to be unique as well. Some companies have made chatbots that are outstanding for various reasons. This article will detail these differences for you.

A chatbot can even be used in your recruitment process. You can set up your chatbot in such a way that it asks all the fundamental questions. And also performs a competency test before you start to review those CVs. So you see, chatbots can be useful for you both internally and for your customers.

Measuring Response Chatbot Metrics

It’s not only important that you measure the metrics mentioned above. But the response to your chatbot also needs to be measured. Measuring your response metrics are a huge part of your overall KPI measurement. You do this in the following ways.

1. Conversation Length

Use this chatbot metric to measure the total time a user spends chatting with the chatbot. It isn’t ways the case that more extended conversations are better. When looking at this metric, consider the intended purpose of the bot. If the bot’s goal is to provide customer service, quick and efficient interaction between the bot and user is paramount.

Bots need to be conversational artists… use this metric to figure out if the conversations are too long. Is it because the user isn’t finding the conversation intuitive? Is it because the bot is asking for the wrong information or responding incorrectly? Maybe the bit is not handing over to a human representative at the correct stage in the conversation.

2. Retention Rate

Depending on the intent of the bot, the retention rate is a great way to measure the number of users who have already had a conversation with the chatbot and return. Because a high retention rate can be a signal that your chatbot is succeeding. And that the chatbots audience is satisfied. But, if the bot’s intention is to field customer complaints, then this is a metric that may indicate a problem elsewhere in the organization. Incorrectly used, this could become a destructive metric, correctly used; it is arguably the most influential metric.

3. Bounce Rate

This is a ‘new’ chatbot KPI web sites have been measuring it for a long time already. The bounce rate measures the percentage of users who engaged the chatbot but then immediately abandoned the conversation. This chatbot metric must inspire investigation as to why users are not going beyond the first step in the conversation.

4. Message Rate

A well-designed conversation leads to a good message rate. It measures the average number of messages exchanged among all conversations over time. A low message rate may mean the conversation is poorly designed, not intuitive. A high message rate may be a sign that the user’s issues and inquiries aren’t being served relevantly.

5. No Response Rate

The “no response” rate indicates whether the chatbot needs more content. Basically, it measures the number of times the chatbot suddenly drops the conversation. Not quite the same as the bot ‘not understanding’, this metric is an indication of a possible dead leg in the conversation.

6. Response Rate

The opposite of the “no response” rate these records and reports on how many times the chatbot successfully ended the conversation. Meaning that there was a positive end to the conversation, and the customer wasn’t left with unanswered questions.

7. Goal Completion Rate

Along with the retention rate, this is the most powerful metric. Not bot should ever go live without a clear, measurable intent. It is vital to set up measurement as a percentage of times the chatbot met the goal(s) set for it, as well as the number of times it failed.

8. User Satisfaction

Exit surveys where users who have interacted with the chatbot are requested to rate their experience. This metric is emotive rather than qualitative. The user may be frustrated with the product they are complaining about and may ‘take it out on the bot’.

Understanding Chatbot KPI Metrics

In summary, chatbot KPI metrics are put in place to measure the chatbot related to the intended function and intent of the bot. There are qualitative measures, quantitative metrics, and emotive issues at play in any conversation. Rating the conversation without effective blends of these measures will not provide any form or notion of the effectiveness of the chatbot.

Be guarded against too many of either and too little of another metric. Moving forward, you’re going to want to stay ahead of the future of chatbots and how they might change in the near future. If you’re setting up your first Facebook Messenger chatbot, head on over and read our article. It’s about the six pillars of setting up a successful chatbot.