When you are creating an automated agent - a chatbot - the plan for rolling out the bot to customers needs to be thought through carefully. There is nothing magic about this - it’s just like rolling out any other software system - first you need to be comfortable that the chatbot works from a functional, technical point of view (i.e. customers can access it and it doesn’t actually break down), then you need to do several rounds of progressively deeper user testing.

Several rounds of user testing are usually done to make sure you’re comfortable having this new automated team member talking to your customers.

Internal User Testing

It is normally fairly easy to conduct some testing with a group of staff or internal users, though it is often short-lived as the nominated testers can bombard the chatbot with imagined scenarios and quickly exhaust the chatbot’s knowledge.

This type of testing can generate some information on the stability of the chatbot, or misses from the initial chatbot training, but in our experience, few employees really understand how customers interact with their business so the conversations between the chatbot and tester tend to be quite forced.

That being said, this is a useful first step in flushing out any very obvious issues not revealed during technical testing.

Friendly User Testing

If you are the sort of business where each of your customers will contact you a lot, then it might be feasible to identify a group of users who are familiar with your offering and can provide decent feedback of their experience with your new agent.

On the other hand, if you have only occasional contact with each of your customers, it may be difficult to identify some friendly customers, and they may not be able to provide good feedback.

If you have previously run a friendly user trial for something else then this might be one of the options to consider, but if you haven’t done one before, a chatbot project which by its nature can be a bit open ended, might not be the one to start with.

Segmented A/B testing

Offering your new chatbot to only one segment, or group, of customers then connecting them randomly with either a human or chatbot can work really well.