Copilot report: track your conversational AI's Performance

If you use generative AI, this dedicated section allows you to view your main indicators for bots utilizing this technology.

 

Copilot for Shoppers indicators

b.jpg

 

This view is aggregated if you have several bots. It can be filtered to view results by bot.

 

Indicators Definitions
Turnover

Total revenue generated by conversations to which generative AI has contributed.


Two amounts are calculated :

- The turnover of fully automated conversations, where only the copilot interacted with the visitor,

- The turnover of partially automated conversations, where the copilot contributed to part of the exchanges with the visitor.

Conversion rate

Conversion rate of conversations in which generative AI was involved.

 

Two percentages are calculated :

- The conversion rate for fully automated conversations, where only the copilot interacted with the visitor,

- The conversion rate for partially automated conversations, where the copilot contributed to part of the exchanges with the visitor.

Customer satisfaction (CSAT)

Percentage of customers satisfied by conversations in which generative AI was involved.


Two percentages are calculated :

- The CSAT of fully automated conversations, where only the copilot interacted with the visitor,

- The CSAT of partially automated conversations, where the copilot contributed to part of the exchanges with the visitor.

Involvement in conversations Percentage and number of conversations including at least one response from the generative AI.
AI Response Capability

Percentage of conversations where the generative AI was unable to provide an answer at least once.

 

The AI's ability to respond measures the percentage of conversations that answered all of the customer's questions. It also measures the proportion of conversations that did not provide an answer to the customer.

Customers feedbacks on AI responses

Percentage of messages generated by the AI that have received positive or negative feedback from customers.

 

During the conversation, customers can assign a satisfaction score to the answers provided by the generative AI. This will be used to calculate the percentages of positive and negative feedback.

Human escalation rate

Percentage of conversations with the generative AI for which the copilot performed a transfer.

 

The calculation of this percentage does not distinguish between failed and successful transfers.

 

 

No Answer conversations

 

 

Details of the reasons for no answer conversations in order to analyze the end-user experience and understand how to improve the copilot's ability to respond.

 

 

Product disovery

 

 

  • Analyze the performance of the copilot’s suggestions throughout the conversation so that I can analyze how customers react to the recommendations.
  • Compare the impact of use cases vs. other types of conversations.
  • Spot the top suggested products.
  • Analyze the behavior of the Copilot.

 

Three elements deserve special attention :

  • Responses that received negative evaluations: visitors can evaluate each response given by the AI (distinct from the satisfaction survey that only occurs at the end of a conversation). In such cases, they can specify why they're unsatisfied. We advise you to review them carefully to ensure the AI is functioning as expected and to prevent any incorrect responses.
  • Unanswered questions are those the AI couldn't answer. We advise you to analyze these situations in detail to determine if you need to provide more information to your AI.
  • Product discovery allows you to analyze the performance of the copilot's suggestions. As a result, you will be able to spot the most suggested products and analyze its behavior.

 

Copilot for Agents indicators

Capture d’écran 2023-11-06 à 15.07.16.png

Indicators Definitions
AI Assisted Conversations Percentage and number of conversations in which an agent used Copilot for Agents to respond to the visitor.
Customer satisfaction (CSAT) Satisfaction rate of conversations in which the agent used the Copilot for Agents feature.
Conversation rate Conversion rate of conversations for which an agent has used the Copilot for Agents functionality and which have led to a transaction.

Analyse and improve your bot's results

To go even further in analysing the performance of your conversations with Generative AI, you can read conversations, customer feedback and unanswered questions from your bots.

 

Filter the Gen AI report by agent

You can monitor your Gen AI bot's performance using the "Agents" filter in your Automation report. In this way, you can monitor its results and statistics to make it more effective.

 

Using this filter allows you to consult the statistics for all your bots at the same time, or for one bot at a time. You can access them from your production report, or from your Generative AI report.

 

To access it from your production report, click on "Reporting" > Production > Add a filter : Choose filter.

Capture d’écran 2023-12-11 à 10.17.21.png

 

To access it from your Generative AI report, go to "Reporting" > Generative AI > Add a filter: Choose filter.

Capture d’écran 2023-12-11 à 10.17.42.png

NB : Check that you have selected the right filter to ensure the accuracy of the data in the automation report.

 

To analyse the performance of your Gen AI bots in more detail, you can use the date, project and channel filters to focus only on the statistics that interest you.

 

What to look for ?

Participation in conversations

This indicator measures the number of conversations including at least one response from the AI. You can use the "Read conversations" button to view the conversations concerned.

Capture d’écran 2023-12-08 à 16.35.34.png

You have a number of key pieces of information about closed conversations that have benefited from Copilot for your customers. Notably the NPS (Net Promoter Score), which corresponds to the rating given by the visitor to the conversation; but also an emoji representing the customer satisfaction rate.

 


When conversations have a poor NPS or low satisfaction rate (CSAT), you can consult or export the conversations concerned in order to analyse their content and provide the necessary solutions to improve your bot's performance.

Capture d’écran 2023-12-08 à 16.38.10.png

AI response capacity

This indicator calculates the percentage of conversations concerned with no-answers questions.

Capture d’écran 2024-04-25 à 16.48.32.png

With this indicator, you can explore the questions asked by your visitors that the Generative AI was unable to answer. This way, you can provide more information to your bot to improve its response capabilities.

 

Capture d’écran 2023-11-10 à 12.10.57.png

 

In the selected conversation, click on the below your bot's response to see the details of this non-response.

No Answer types

Missing information : The LLM looked into knowledge sources but didn’t find information to generate an answer. This can be due to score embeddings not good enough or embeddings good enough but without relevant information inside.

Missing knowledge source : The model did not identify any knowledge sources suitable for the question:

  • For a FAQ type question, no corresponding FAQ knowledge was found.
    For a product question, no suitable product knowledge was identified.

Product discovery disabled : The Copilot didn't answer because it detected a use case it wasn't allowed to manage. Valid only if product catalog is uploaded.

Hallucination risk : The LLM generated an answer but evaluated it wasn't correct and shouldn't be sent to the end-user.

Content filter : The end-user's or bot message was filtered by Azure, which prevented the LLM to answer.

 

Customer feedback on AI responses

This indicator reports the percentage of positive and negative feedback from customers on the answers provided by Generative AI.

Capture d’écran 2023-12-08 à 16.37.14.png

The "View" button gives you access to the questions formulated by your visitors, the answers provided by the Generative AI that received bad feedback, as well as this feedback.

You can choose to simply view or export these conversations.

Capture d’écran 2023-12-08 à 16.40.14.png

Conversations with AI assistance

This indicator calculates the number and percentage of conversations in which your agent used Generative AI to formulate answers.

Capture d’écran 2023-12-08 à 16.40.42.png

As with the "Participation in the conversation" indicator, there are several additional indicators available, such as the NPS (Net Promoter Score), the duration of the conversation and the customer satisfaction rate.

 


When conversations have a poor NPS or a low satisfaction rate (CSAT), you can consult or export the conversations concerned in order to analyse their content and provide the necessary solutions to improve your bot's performance.

 

Capture d’écran 2023-12-08 à 16.41.20.png