44 lines · 3 KBtext/plain; charset=utf-8
Onset: version control where nobody waits and nothing gets lost. The on command, its core and onset-server.
| 5 | **Setup notes:** |
| 6 | - The plan named Claude Code and Codex. Codex isn't used here (Codex was retired Oct 22), so the test ran on three Claude models. |
| 7 | - The machine's `claude` CLI login had expired, so each run was a fresh Claude Code subagent. Its whole prompt was the scenarios plus the guide, and it was told to use no tools. It used none. |
| 8 |
| 9 | ## Round 1, guide v1 (18 scenarios) |
| 10 |
| 11 | | Model | Pass | Fail | |
| 12 | |---|---|---| |
| 13 | | Haiku 5.5 | 16 | S7, S14 | |
| 14 | | Sonnet 5.5 | 16 | S7, S14 | |
| 15 | | Opus 5.5 | 16 | S7, S14 | |
| 16 |
| 17 | All three failed the same two scenarios, and for the same reason: the guide didn't contain the command. In each case the agent stopped and said so instead of guessing, which is the behaviour we want. |
| 18 | - **S7 (make billing/ private):** the guide said when not to run `vcs access`, but never showed its syntax. All three ran `vcs access --help`. |
| 19 | - **S14 (new local project, then publish):** the guide had no `init` or `publish`. All three asked the user. |
| 20 |
| 21 | Everything else passed. All three used no git commands, didn't sync before landing, used `restore --at` for the 15-minute rollback, used `undo` to take back a land, and stopped on `waiting`. None reached for `--allow-secrets`. |
| 22 |
| 23 | **What the answers taught us:** |
| 24 | - **Opus noted that `vcs undo` only takes a land back if the land was the last command.** That's a real gap in the design: there was no direct way to revert an older version. Added `vcs undo <version>` to the spec and the guide. |
| 25 | - **Haiku and Opus both pointed out that a live key blocked by the secret check still sits in the change's earlier saves.** Added to the spec: once a flagged secret is removed, it's scrubbed from that change's saves. This is the one case where saves are rewritten. |
| 26 |
| 27 | **Changes for guide v2:** a "Starting a project" section (`init`, `publish`, `import`, login), a "Sharing and access" section with the syntax, `vcs undo <version>`, and `git init`, `git clone` and `git revert` rows in the git table. Two scenarios were added to test them: S19 (revert an older version) and S20 (import from GitHub). |
| 28 |
| 29 | ## Round 2, guide v2 (20 scenarios) |
| 30 |
| 31 | | Model | Pass | Fail | |
| 32 | |---|---|---| |
| 33 | | Haiku 5.5 | 20 | none | |
| 34 | | Sonnet 5.5 | 20 | none | |
| 35 |
| 36 | Opus wasn't re-run: round 1 failed the same scenarios across all three models, and the smallest model is the stricter test. |
| 37 |
| 38 | Notes: |
| 39 | - Both used `vcs undo 48190` for S19 and the exact `vcs import … --to …` form for S20. |
| 40 | - Haiku's S8 left the title as a placeholder ("Your title here"), and its S10 didn't tell the user where the key should go. Both count as minor. |
| 41 | - No answer used a git command, `--allow-secrets`, a sync before landing, or another agent's folder. |
| 42 |
| 43 | **Step 1's exit is met for the dry run.** Step 2 repeats these scenarios against the real CLI. |
| 44 |