Building Ari, note #3: the trust problem
Building an agent that replies to customers on its own (autonomous), and that can answer differently on different runs (non-deterministic), comes with a hard problem: trust. How do you know it does a good enough job to handle support on its own?
There are two sides to this. How we, the builders, know Ari is good, and how our customers know Ari is good. In both cases the answer is a spectrum of checks. At the simplest end you ask it questions and read the answers one by one. At the other end you build datasets and run statistical evaluations that tell you how often it answers correctly.
We run those checks constantly. But we also want to put them in our customers' hands, and we're building a series of features in that direction. The latest is Shadow Mode.
Shadow Mode runs Ari on the threads you choose, and instead of replying to the customer it records what it would have done as a timeline entry only your team sees: the answer it would have sent, or the reason it would have handed off. Two things make this powerful. You see Ari act on real customer conversations with no risk of it replying to a customer. And because it never touches the customer, you can point it at almost all of your support at once, and watch a real track record build.
Here's a short demo.
Trust matters enormously for AI systems, and nowhere more than in customer support. We want customers to have real confidence in Ari, so we're building tools to help them build it up. Shadow Mode is one of them.