Online evaluation
If the coach chatbot answered real members, how would a change be shown to help them, and not only to pass the offline eval set?
Online evaluation · Framework
How the bots would be tested with real members
No real members use the bots: the public demo is anonymous and rate-limited, so there are no online results. This page is the plan for testing the bots with real members.
1 · Offline vs online
Real chats have no answer key
Offline, each of the 64 test cases lists the facts a good reply must contain (scoring). A real message arrives with no such list.
- Stays offline
- Expected facts and the pass or fail verdict: a real message comes with no list of facts.
- The same synthetic members on every run.
- Carries over
- Code checks on every reply, such as a banned “I guarantee”.
- The LLM judge, grading by a rubric (a fixed list of questions for any reply), after its grades are checked against human reviewers.
- The skill files and guardrails.
- Only online
- Whether a reply helped: what the member did next.
- Follow-up turns that build on the new version's own replies.
- Effects at real load, such as on the human support queue.
Real chats in two steps: backtest, then test online
Logged chats can test a change before any member sees it. What a backtest measures differs from the eval set, since there is no answer key, and from the online test, since no member reacts.
- Offline
Backtest on logged chats
- How
- Replay past real messages, with personal details removed, through the current version and the change, and grade both replies with the code checks and the rubric judge.
- Measures
- Rubric pass rate; which reply the judge prefers on the same message (shown in random order, since judges favor a position); safety flags per 1,000 replies; cost and latency per reply; which skills the router picks.
- Adds
- Real typos, new topics and credit files unlike those of the 6 synthetic members, in far larger numbers. Both versions still answer the same message, so the comparison stays paired.
- Cannot tell
- Whether the facts are right (there is no answer key), how later turns would go (they were written in reply to the old version), or what members do next.
- Online
A/B test with members
- How
- Split members at random, so each sees one version (section 3).
- Measures
- What members do: asking the same question again, handoffs to human support, and later outcomes (section 2).
- Adds
- Whether members are better off, including follow-up turns and effects at real load.
- Costs
- Weeks of traffic, with real members exposed to the change, so it runs only after the backtest finds nothing wrong.
2 · What to measure
One metric decides, the others guard or explain
Each metric gets one job, set before the test starts: only the primary metric decides, a guardrail can block the launch, and the rest explain the result or are checked later.
Primary metric
ExampleDecides whether the change ships.
- Asked again within a day
- The same question within 24 hours, matched by meaning. A follow-up that builds on the answer does not count. Lower is better.
Guardrails
Can block a launch even when the primary metric improves: each must not get worse by more than a margin set in advance.
- Safety violations
- Promising a score gain, suggesting a product to a member in hardship, repeating a card number.
- Preventable handoffs
- Chats sent to human support that the coach should have solved.
- Still using the app a week later
- Members active in the app 7 days after their first coach chat. It must not fall: a change that drives members away would also lower asking again.
- Latency, 95th percentile
- 95 of 100 replies arrive within it. An average hides the slow ones.
- Cost per chat
- Before and After load different skills, so cost is measured.
Diagnostics
Explain why the primary metric moved. They never decide on their own.
- Judged quality
- The judge grades a random daily sample. Risky topics (hardship, collections, disputes) get extra samples, weighted back to their real share. A weekly human audit, blind to version, shows whether the judge can be trusted.
- Thumbs up or down
- Few members click, and those who do tend to feel strongly.
Long-term outcomes
What the coach is for, but they move over months, too slowly to decide one test.
- What members do next
- Paying a card down before the statement date, on-time payments, staying with the app.
A small holdout kept on the old version for months shows whether gains on the primary metric reach these. If they never do, the primary metric is the wrong target.
Why asking again within a day
A primary metric needs four things: it is measured on every chat without asking the member, it moves within days so a test of a few weeks can read it, one direction is plainly better, and it comes from what members do rather than from a model's opinion. Of these candidates, only asking again has all four.
| Candidate | Every chat | Moves within days | Clear direction | From members | Used here as |
|---|---|---|---|---|---|
| Asked again within a day | Yes | Yes | Yeslower | Yes | Primary metric |
| Thumbs up or down | Nofew click | Yes | Yes | Yes | Diagnostic |
| Handoffs to human support | Yes | Yes | Partlysome chats belong with a person | Yes | Guardrail (preventable ones) |
| Judge-graded quality of live replies | Partlya sample | Yes | Yes | Noa model grading a model | Diagnostic |
| Still using the app a week later | Yes | Partlya week, and one reply barely moves it | Yeshigher | Yes | Guardrail (must not fall) |
| Messages per chat | Yes | Yes | Nomore can mean engaged or confused | Yes | Not used |
| Paid down before the statement date | Noonly members with a card balance | Noa month or more | Yes | Yes | Long-term outcome |
It has weak spots too. Matching “the same question” takes a model (an LLM or text embeddings), which needs checking against human labels, as the judge did. And a member who gives up and leaves never asks again, so a change that drives members away could look like a win. The guardrail on members still using the app catches that during the test, and the long-term holdout later.
3 · A/B test
Split members at random, and fix the rules first
An A/B test splits members at random between the current version (A) and a change (B), such as Before to After or one skill edit. Chance alone decides who gets which, so a gap bigger than noise is caused by the change.
Before it starts
Fix the plan
One primary metric, a margin for each guardrail (how much worse it may get), the number of members needed (below), and a length in whole weeks that reaches it.
Why one primary metric: if a change does nothing, 20 metrics tested at 5% still show about one significant result by chance, as likely against B as for it. The others get a correction, such as the Holm method, which keeps the chance of any false win at 5% or less. Why whole weeks: weekdays and weekends differ, reactions to anything new fade (compare week one with week two), and what members ask shifts around paydays and statement dates, so a full monthly cycle is safer.
While it runs
Split members at random
By member, half and half. Each member sees one version on every visit and device, so no chat switches bots midway.
Watch for spillover: if B sends fewer chats to human support, A's queue gets shorter too, and A looks better than it would alone.
Count each member from their first coach chat
Members who never open the coach cannot be affected, and nobody sees a difference before opening it, so both groups enter at the same rate.
Every day until the planned end
Is a guardrail clearly worse?
Yes: stop early for harm
The test ends here. Stopping for harm is always allowed.
No: go on to the next check
Has it reached the planned end?
No: run another day
Back to the first check tomorrow, without reading the primary metric: stopping the first time it looks significant makes false wins far more common than 5%. Sequential testing allows planned looks.
Yes: leave the loop
After it ends
Analyze by member
The member is the unit, since one member's messages are alike. The primary metric is one value per member, adjusted with CUPED (below). Reply-level guardrails, such as 95th percentile latency, are computed over all replies, with intervals that resample whole members. The primary metric decides; each guardrail must stay within its margin.
Then
The readout in section 5: the split first, then the guardrails, then the primary metric with its confidence interval.
How many members a test needs
Power is the chance a test detects a real change of the size that matters. Halving that size roughly quadruples the members needed.
Illustration only, not a measured rate: if 20% of members asked again within a day, detecting a drop to 19% (80% power, 5% false-positive rate) would take about 25,000 members in each group.
CUPED adjusts each member's result with data from before the test: the same metric for members who used the coach before, and something every member has, such as app activity, for those new to it. That data predates the split, so it cannot bias the result. A correlation of 0.5 cuts the variance, and the members needed, by about a quarter; members with weaker data from before gain less.
4 · No random split
Match the method to how the change ships
A plain before-and-after comparison mixes a change with everything else that changed. Each method below rests on assumptions that can be checked in part, never proven.
To everyone on one date (an urgent guardrails fix)
Interrupted time series
A break at the launch date, against the earlier trend.
AssumesThe trend would have continued, and nothing else that moves the outcome changed that date.
Probe it byFake dates, and a metric the change should not touch, show no break.
In waves (a few states first)
Difference-in-differences
The change in switched members, minus the change over the same weeks in members not yet switched.
AssumesParallel trends: without the change, both groups would have moved alike.
Probe it byParallel lines before launch. A fake earlier launch date finds nothing.
To one whole state, never the others
Synthetic control
That state against a weighted mix of the others.
AssumesThe mix would have kept tracking that state, and nothing else that moves the outcome changed at the same time.
Probe it byA close fit before launch. Other states, run as fake switches, rarely show as large a gap.
Past a cutoff (hypothetical: a check-in only at 50% card utilization or more)
Regression discontinuity
Members just above against just below. Holds only near the cutoff.
AssumesMembers cannot place themselves precisely on one side, and nothing else changes at the cutoff.
Probe it byNo pile-up on either side. Traits measured before are smooth across it.
As an opt-in, with an invitation sent to a random half
Encouragement design (instrumental variables)
The gap between invited and not invited, scaled by the rise in coach use, estimates the coach's effect for members who use it because they were invited.
AssumesThe invitation acts only through coach use, and no one uses the coach less because of it.
Probe it byThe rise in use can be measured. Acting only through the coach can only be argued.
As an opt-in, with nothing random (turning on Rent Reporting)
Propensity score matching or weighting
Members who chose it against members who did not but had the same chance of choosing, judged from what was known before.
AssumesNo hidden confounders: every reason behind both the choice and the outcome is measured.
Probe it byThe groups match on measured traits. Members with no match are left out, so the result covers only those who remain. A sensitivity analysis asks how strong a hidden reason must be to erase the result.
5 · Launch
Offline gates, online confirms
Offline is cheap and repeatable, so it runs first and stops changes that break known cases. Only online shows whether members are better off.
Offline gate
OfflineRerun the cases the change can affect, as on Skill edits, then backtest it on logged chats (section 1). Go on only with a Ship decision, no guardrail misses and nothing wrong in the backtest.
Canary
About 1%A day or two, to catch crashes, slow replies, cost spikes and safety flags, with a quick rollback. It does not measure the effect.
A/B test
50/50By member, with every rule in section 3 set in advance.
Readout
Test overThe split first: if it is more uneven than chance allows (a sample ratio mismatch), something is broken and the result is not read. Then the guardrails, then the primary metric with its confidence interval (the range of effects the data are consistent with). Segment cuts are ideas for the next test, not findings. Write the decision down: ship, change or drop.
Rollout
All but a holdoutEveryone else, in steps if risky, while watching the guardrails. A small holdout stays on the old version for months.
Monitoring
Daily and weeklyCode checks, a judge sample, latency and cost every day. A human audit every week. An alert on any guardrail breach.
Back to step 1
After every launch
Turn failures into cases
A thumbs down, a repeated question, a judge or reviewer flag, or a safety catch becomes a new offline case, with personal details removed, a synthetic member attached and expected facts written by a reviewer.
Check that offline predicts online
Over many launches, offline wins should mostly become online wins, allowing for noise in both. If they do not, the eval set or the judge is measuring the wrong thing.