Duolith docs

0062. A skill's evaluation is a task, a check and a reference solution, run both ways

  • Status: Accepted (Mik, 17 Nov 2026)
  • Date: 2026-11-14
  • Decided by: integrator (internal only), pending Mik's approval
  • Planning item: WP14 (agent layer)

Context

Every skill needs an evaluation that CI can run. CI cannot run an agent, so it cannot see whether an agent following the skill succeeds; but a skill can still rot: its steps stop matching the repository, or its check stops checking anything.

Decision

Each skill has eval/task.md (the task an agent is given, with front matter naming its workspace: a copy of the repository, or a new app from a packages/create template), eval/check.ts (default export: the problems it finds in a workspace) and eval/solve.ts (a reference solution written by following the skill). pnpm skills:eval runs each check in a fresh workspace, where it must find the task undone, and after the reference solution, where it must find nothing wrong. App checks run duolith build and duolith fmt --check; contributor checks run what can run without compiling Rust (the kit's spec test, check-generated, the TypeScript compiler). pnpm skills:eval --skill <name> --workspace <folder> grades an agent's real attempt with the same check.

Alternatives considered

OptionWhy not
Checks only, run on the repositoryNothing proves a check can pass, or that it fails when the work is missing.
Run an agent in CISlow, costly and not repeatable; the same check can grade agent runs outside CI.
Compile Rust in contributor evaluationsMinutes per skill on a shared machine; CI compiles the real change anyway.

Consequences

A skill change that breaks its reference solution fails CI. Evaluations copy the repository (about 5 MB, without build output) into a temporary folder and link node_modules back. skills:eval runs in pnpm check.