Fynn LabCode

Skill edits

When one skill file changes on the After bot, which cases need rerunning, and is the edit safe to ship?

Skill edits · Framework

Checking a skill edit before it ships

A skill edit is like a code change: it can fix one answer and quietly break another. So before an edit ships, the eval cases it can affect are rerun and compared with the last run of the After bot.

1 · The loop

Edit, rerun, count, decide

Every edit goes through the same loop. The real skills/ folder is not touched while an edit is tested: its edited copy is laid over it for the run.

  1. Edit a skill

    Change one skill file: a topic skill's body or description, or an always-on skill.

  2. Optional

    Add test cases

    When the change covers something no case tests yet, add cases for it first.

  3. Run the test cases that depend on the change

    Only the cases the edit can affect are rerun (see What reruns).

  4. Regression report

    Compares the rerun with the last run: the cases the edit fixed, the ones it broke, and the ones that flipped by chance.

  5. Decision

    Safe to ship?

    • Yes: the edit fixed cases and broke none it could reach (Ship). Ship the edit.
    • No: iterate on the skill and go back to step 1.
The steps follow the regression-testing loop diagram in Kikoff's “Fynn at 100% scale” post, including its report of what changed with a failures overview. The eval set, the selection rules, the split into fixed, broken and chance, and the decision rule are Fynn Lab's own.

2 · What reruns

Rerun only what an edit can affect

Each run records what the router picked for every case, so the next run knows which cases depend on which skill.

  • A topic skill's body

    Reruns: The cases labeled with that skill or routed to it last time.

  • A topic skill's description

    Reruns: Every case.

  • The skill cap

    Reruns: Every case.

  • An always-on skill

    Reruns: Every case.

3 · Fixed, broken, chance

Telling the edit apart from chance

The same prompt can pass one time and fail the next: in cases these edits could not reach and whose routing did not change, about one in eight flipped between runs. The router also reorders, and now and then changes, its picks between identical runs. So a flip only counts for the edit when the edit itself could have changed that case's answering prompt (routes compare as sets, so a new order alone does not count). Descriptions and the cap are also in the router's instructions, so these rules assume such an edit acts only through the skills it adds or removes. Each kind of edit has one way in (when an edit also comes with a cap change, the cases where the cap mattered count too):

  • A topic skill's bodyA case whose answer loaded the edited skill both before and with the edit. The router never reads bodies, so when only one side loaded it, the router changed the prompt too and the flip is not put on the edit.
  • A topic skill's descriptionA case where the edited skill entered or left the routing, since only the router reads descriptions.
  • The skill capA case whose routing changed where the cap mattered: the run with the lower cap filled it, or the run with the higher cap went past it.
  • An always-on skillEvery case, since always-on skills are in every prompt.

Fixed by the edit

The edit reached the case, and it now passes.

Broken by the edit

The edit reached the case, and it now fails.

Flipped by chance

Outside the edit's reach: the edit was not in the case's answering prompt, or was not the only thing that changed it.

The decision follows from the first two: Ship when the edit fixed cases and broke none, Tradeoff when it broke some but fixed at least as many, Do not ship when it broke more than it fixed, and No effect when no case it could reach flipped. A case the edit could reach may still have flipped by chance. So the decision counts as backed only when an exact test on the cases the edit could reach is significant in the decision's direction; otherwise it is shown as a lead (for example “Lead: ship”): worth checking with more runs, not proof. A tradeoff is always a lead. With one run per case the test compares fixed against broken, so a Ship needs at least six fixed cases and none broken, and a Do not ship at least six broken and none fixed (or eight against one); with more runs it is the run-permutation test on those cases. Whether a case is reached depends on its routing, not on its result, so the cases outside the reach can be set aside for this test.

4 · The edits

3 skill edits, and a cap change

Each edit changes one skill on the After bot. The cap change lets the router pick four skills instead of three.

  • S1Topic skill body edit

    Per-card utilization in score-drop answers

    Skill: score-changes

    Adds one "How to answer" rule to score-changes: when a score drop coincides with higher reported card balances, compute each open card's utilization from the profile's accounts, give those ratios, and name the card to pay down first.

    See the S1 result: Per-card utilization in score-drop answers
  • S2Always-on skill edit

    Shorter replies in the always-on response rules

    Skill: response-rules

    Changes the length rule in the always-on response-rules skill from "under about 250 words" to "under about 120 words".

    See the S2 result: Shorter replies in the always-on response rules
  • S4Topic skill body editShipped 2026-10-22

    Closed-account reporting time in the Credit Builder skill

    Skill: credit-builder-account

    Replaces the pointer "(how long: see score-changes)" in credit-builder-account with the line itself: a closed account in good standing can stay on the report for up to 10 years.

    See the S4 result: Closed-account reporting time in the Credit Builder skill