0062. A skill's evaluation is a task, a check and a reference solution, run both ways
- Status: Accepted (Mik, 17 Nov 2026)
- Date: 2026-11-14
- Decided by: integrator (internal only), pending Mik's approval
- Planning item: WP14 (agent layer)
Context
Every skill needs an evaluation that CI can run. CI cannot run an agent, so it cannot see whether an agent following the skill succeeds; but a skill can still rot: its steps stop matching the repository, or its check stops checking anything.
Decision
Each skill has eval/task.md (the task an agent is given, with front matter naming its workspace: a copy of the repository, or a new app from a packages/create template), eval/check.ts (default export: the problems it finds in a workspace) and eval/solve.ts (a reference solution written by following the skill). pnpm skills:eval runs each check in a fresh workspace, where it must find the task undone, and after the reference solution, where it must find nothing wrong. App checks run duolith build and duolith fmt --check; contributor checks run what can run without compiling Rust (the kit's spec test, check-generated, the TypeScript compiler). pnpm skills:eval --skill <name> --workspace <folder> grades an agent's real attempt with the same check.
Alternatives considered
| Option | Why not |
|---|---|
| Checks only, run on the repository | Nothing proves a check can pass, or that it fails when the work is missing. |
| Run an agent in CI | Slow, costly and not repeatable; the same check can grade agent runs outside CI. |
| Compile Rust in contributor evaluations | Minutes per skill on a shared machine; CI compiles the real change anyway. |
Consequences
A skill change that breaks its reference solution fails CI. Evaluations copy the repository (about 5 MB, without build output) into a temporary folder and link node_modules back. skills:eval runs in pnpm check.