Release testing
"The update broke something that used to work" is the most common complaint about software, and AI products are no exception: a small change to how answers are generated can quietly change hundreds of them.
ChatHQ treats every release as something to be proven, not assumed.
Separate environments
Changes are built and tested in environments that are separate from the live product. A change only reaches the live product once it has passed through them. Your assistants and your visitors are never the first to see a new release.
A fixed set of real questions
The team maintains a growing set of real questions, drawn from real conversations across different kinds of businesses, together with the answers the assistant is expected to give. On every release:
- The full set is run against the new version.
- Each answer is compared with the expected one.
- A release that makes answers worse is held back until the cause is fixed.
The set grows as new edge cases are found, so a problem that has been seen once is checked for on every release after it.
Versioned releases
Each major change to the engine that powers assistants is released as a numbered version. That makes it possible to compare two versions side by side and, if a release does slip through with a regression, to roll back to the previous one. See Assistant versions.
What this means for you
- Your assistant's behaviour does not drift between your publishes. The only changes to how it answers are the ones you make and publish.
- A fix for an answer-quality issue, once shipped, stays fixed: it becomes part of the question set.
- You can run your own checks the same way: keep a short list of the questions that matter most to your business and ask them in the test chat after each change you make.
Related
Was this article helpful?