Before vs after
Single agent vs composable skills
Ask both bots the same question and watch which guidance each one loads. They differ in how each message is routed and how the prompt is put together. The After bot also keeps some of the previous turn's skills for one more turn, which is a Fynn Lab addition.
- Same skill files
- Same member profile
- Same answer model
What differs: Different routing
A study of how to evaluate a skill-based LLM assistant
The two bots below are the test bed. Around them: an eval set graded by an LLM judge, a regression check for every skill edit, a check of the judge against a person, and a plan for experiments on real traffic.
- Questions that need a second domain79% vs 16%of runs passed, After against Before; about even when one domain is enough
- Pre-registered held-out check39/39 vs 12/39runs passed, After against Before, on new multi-domain questions written after the skills were frozen
- LLM judge against a person92%of facts graded the same on 13 sampled replies, graded blind; in all 4 disagreements the judge was the stricter one
Starting the demo's backend in the cloud. It sleeps when nobody has used it for a while, so this can take a minute or two. If it is not up within 15 seconds, recorded replies show here while it starts.