How to test an AI website support assistant

Choose a range of test questions

Test the common questions the assistant should answer, but also include vague wording, follow-up questions, outdated assumptions and questions that must go to a person. Use real language from enquiries where you can do so without exposing personal information.

• Direct questions with one clear approved answer.
• Questions whose answer depends on a stated condition.
• Natural follow-ups that rely on the previous message.
• Questions your approved content cannot answer, so visitors need to contact your team.

Check the source as well as the wording

Check that the answer uses the latest approved information and includes any conditions that matter. Make sure it has not mixed up two different policies. If the original guidance is unclear, fix that first.

Check how the chat works on your site

Open the assistant on mobile and desktop, move between pages and try using it with a keyboard. Check what happens when the connection is slow or fails. Make sure contact links work and the chat window does not cover menus, navigation or cookie choices.

Decide how often to check answers

Check the first conversations closely. Then decide how often to review them based on how busy the assistant is and how much a wrong answer could matter. Test again after policy, pricing or website changes. Keep a short list of important questions and repeat those checks after each change.