P-019 · Case study
SHIPPEDReacher: AI Financial Assistant
The AI financial assistant on GoReach's SEBI-regulated advisory platform, used by 2,000+ people. Evals that grade replies against what the system did, guardrails in code, and routing that cut model costs 64%.
Case study
The problem
GoReach is a SEBI-regulated advisory platform. Its AI assistant may explain and educate, but personal advice has to come from a registered human expert. The costly failures don't show up as errors in the logs. If the assistant misreads one question, it gives regulated advice, and if it claims an action it didn't take, the user waits for help that isn't coming.
The approach
The rules that regulation depends on are written in code. Questions that need personal advice go to a registered expert, and an LLM only helps with questions the rules can't classify. Evals run on every change: checks on the line between education and advice, output-safety checks, and a scenario suite that replays real conversations against the live service and grades each reply against what the system did. Underneath, multi-provider model routing falls back to another provider when one fails.
What worked
Reacher serves 2,000+ users on iOS and Android. The replay suite caught the hardest bug: Reacher sometimes told a user "I've connected you to an expert" when no handoff had happened. The reply read well, so every text-quality check passed it. I built a guard that compares each action a reply claims with the actions the system recorded for that turn, and blocks the claims with no match. It ran in shadow mode with a counter first, so I could see how often it fired, and on which replies, before it changed anything a user saw. Once the numbers held up, I switched it on for every reply. Separately, the model routing cut Reacher's model costs 64%.
What I'd do differently
An eval that only reads the reply can't catch an assistant describing something it never did. The reply has to be graded against the actions the system recorded. I'd also ship any guard that changes what users see in shadow mode first, because the counter tells you how often it will fire before anyone is affected.
More detail
Reacher answers money questions for GoReach's customers on iOS and Android. It finds and compares funds, explains a user's own portfolio, reads the statements they upload, and hands the conversation to a registered human expert when a question needs personal advice. GoReach is a SEBI-regulated advisory platform with $10M (₹85 Cr+) in AUM. I built Reacher, its evals and guardrails, and the multi-provider model routing behind it.