Chatbot personality testing

The best UX solutions for chatbots result in asking the right questions.

alt text

Is there a way to test a proof of concept (PoC) beyond to look critically at the personality designed with users ? Can we really find out if the chat bot service will make their life better? Users of your chatbot may even not exist if their technology habits have not validated. What follows are ways to fill the knowledge gap between chatbot responses and users that don’t yet exist.

Why your users don’t exist and you need validation

It is normally assumed that an ideal user base will be enticed to buy other service through the chatbot. Research can lead us to think that interacting with a chatbot can solve their problems. User Experience (UX) of Artificial Intelligence (AI) needs more than measuring system usability. Realizing this helps any product team building the chatbot technology involved and personality.

Data science to the rescue: dialog transcripts, analysis of word choices and sentiment

Data Science offers simple tools to build a matrix and gain a holistic understanding. We can analyze chat transcripts and arrive at insights for design and AI. For example, how many words and the frequency of terms in use from the transcripts lead us to change our content. We can look into word usage, positive or negative perceptions, and what is being discussed.

Usability testing transformation

The best UX solutions for chatbots result in asking the right questions. Getting videos of the user interactions is invaluable, especially if these capture voice. Equally important is the usefulness of the chatbot responses. Is our design biologically plausible? in other words is it working how people actually work, think, talk and type in real life? Initial tests tend to discover that potential users do not ask questions as scripted. So our team reshaped the personality of the chatbot and adjusted its biases. This results in extra spin cycles to change the AI to “catch up” to the real world.

Chatbot UX testing template

The scenario plans to be as real as possible meaning that it targets actual content from the domain. Make sure to capture the interactions of the users actual language. If such is not possible at least plan to send out a survey instead to capture their impressions.


Measure at what point users are getting confused and quit using your chatbot.


  1. Launch the __________________________________ chatbot

  2. Chat with the __________________________________ chatbot to find out

  3. Add more steps as needed


  1. Was the chatbot confusing to use? If so, when?

  2. At any time did you feel like quitting the bot service and find the answers somewhere else? If so, why?

  3. Did the chatbot have a personality? If so which type?

  4. Did the chatbot seem like it can help you? Motivate you in any way? Provided enough useful and timely feedback?

  5. Add more questions as needed.