RP Sanjiv Goenka Group

3 tools to evaluate your progress on chatbots

Estimated reading time : 3 Minutes

Share this post

Conversational platforms, such as chatbots, are rapidly becoming a popular medium of interaction between enterprises, customers and employees. Gartner predicts that by 2022, 70% of white-collar workers will interact with conversational platforms on a daily basis. At the same time, analyst data shows that the global use of top messaging applications has surpassed the use of top four social networks. This tectonic shift in consumer behavior requires brands across industries to recalibrate their customer experience strategy to increase customer satisfaction and reduce customer attrition. This is especially important for B2C businesses such as telecom, media, financial services, utilities, and e-commerce.

Regardless of whether you’ve just begun your chatbot journey or you’re considering more complex chatbot experiences for your customers, here are three tools to help you evaluate your progress.

Chatbot maturity matrix

Start your chatbot journey where it is most contextually relevant to your business and customer service objectives. You can use a Chatbot Maturity Matrix (see Table 1) to evaluate your readiness and target early success.

Chatbot Maturity Model Characteristics Target Chatbot Deployment Opportunities
Immature No Natural Language Processing (NLP)
No learning. The chatbot cannot learn by itself from its interactions with customers
Simple Q &A
Infant Simple NLP
No learning
Simple Q&A + actions
Mature NLP
Supervised learning
Moderate conversation
Advanced NLP + context
Intelligent conversation
Futuristic research areas NLP + context + emotion
Fluent conversation

Table 1: Chatbot maturity matrix

Four-pillar chatbot strategy framework

Delivering a memorable chatbot experience is no trivial task. It requires a multi-disciplinary approach with extensive planning around four important pillars: functionality, conversational intelligence, personality and interface (see Table 2).

Pillar Description Must Have Features
Functionality Does the chatbot do what it is supposed to do, and if so, how good is it? Are there existing alternatives (apps, websites, search engines, etc.) to accomplish the same task? Chatbots must accomplish their primary task, outperform alternatives and have adequate domain knowledge.
Conversational Intelligence Does the chatbot feel human? Is it witty or easily confused? Can it understand the contextual difference between “I am fine” and “’m going to be fined”? Can it retain and build on the conversation’s context? Chatbots must have ‘human-like’ conversational capabilities and be able to preserve context throughout the chat session.
Personality Does the chatbot have a personality? Can it indulge in humor, enter and exit gracefully? Chatbots must have a personality that’s best suited to the domain it’s deployed in. Personalities make a strong impression.
Interface Does the chatbot experience take you outside the chat window? Does the chatbot interact with buttons, text, voice or a combination of all of the above? Chatbots should have interactive elements in the interface, along with the ability to handle text input and minimal external links.

Table 2: Four Pillar Chatbot Strategy Framework

Claim your complimentary session with Firstsource chatbot experts who can help you plan around the important pillars of a chatbot strategy. Contact us.

Chatbot performance benchmarker

If you’ve already deployed a chatbot to serve your customers, you can use the scorecard shown in Figure 1 to benchmark its performance at regular intervals. This exercise will help you amplify what’s working, fix bottlenecks, and leapfrog your current chatbot capabilities to the next level.

Give yourself a “1” if you “Strongly Disagree”, a “5” if you “Strongly Agree”, or any other value in the continuum based on your best judgement.

Pillar Description Must Have Features
1 2 3 4 5
Functionality Can interpret commands accurately 1 2 3 4 5
Can execute requested tasks 1 2 3 4 5
Is able to maintain a discussion 1 2 3 4 5
Is able to activate a new topic 1 2 3 4 5
Number of services available in the chatbot 1 2 3 4 5
Trustworthiness Offers dependable information 1 2 3 4 5
Offers rating capability 1 2 3 4 5
Shows breadth of knowledge 1 2 3 4 5
Gracefully handles unexpected input 1 2 3 4 5
Offers transparency 1 2 3 4 5
Security & Privacy Protects and respects privacy 1 2 3 4 5
Is safe from intrusion/breach 1 2 3 4 5
Efficiency Is easy to use 1 2 3 4 5
Supports canned responses vs. free text 1 2 3 4 5
Is available at all times 1 2 3 4 5
Is accessible via multiple platforms 1 2 3 4 5
Graphical Appearance Offers a friendly user-interface 1 2 3 4 5
Uses emojis and pictures/gifs 1 2 3 4 5
Humanity Creates an enjoyable interaction 1 2 3 4 5
Conveys personality 1 2 3 4 5
Reads and responds to moods 1 2 3 4 5
Offers personalized options 1 2 3 4 5
Empathy Offers personalized suggestions 1 2 3 4 5
Responds immediately 1 2 3 4 5

Figure 1: Chatbot performance benchmark – scorecard

Share the results of your benchmarking exercise with a Firstsource chatbot expert for an interactive, diagnostic discussion about your organization’s chatbot roadmap. Contact us.

Getting chatbot deployment right

Chatbot deployment success requires stakeholders to: adopt a continuous experimentation mindset, objectively analyze performance metrics, and frequently iterate to close the gaps and improve performance. The upside of getting chatbot deployment right is superior customer loyalty, in turn translating to a higher share of the customer wallet in the long run.


Download Now

Simply fill out this form to download