Method & caveats
What this is, and what it is not.
A small benchmark of one-shot creative and technical work. Eight tasks, two variants, one model per run. The caveats are part of the result.
The shape of the test
Each run received the same sixteen prompts: eight tasks, each in a thin variant (intent only) and a thick variant (a full brief with audience, constraints and success criteria).
Subagents were asked to return exactly one self-contained file. No network requests, no external images or libraries, no build step. The orchestrator collected the reply and saved it.
The tasks are deliberately different: visual composition, interface design, critique, motion, explanation, fiction, implementation and code review. There is no single score that makes sense across all eight.
Creative work is therefore presented without a number. The parser and code-review tasks carry a short technical note because those tasks have verifiable failure modes.
Tok/s is (output + reasoning tokens) ÷ subagent elapsed time. It includes thinking and tool time, so it is end-to-end throughput, not raw decode speed.
The eight tasks
| Task | Format | Brief |
|---|---|---|
| 01 Poster | SVG | The Last Ferry, 11:40pm — a poster for a crossing between San Remo and the island of Vell. Two inks, one paper tone, a condensed grotesque and a monospace timetable. |
| 02 Interface | HTML | Inkwell — photograph a page of handwriting, get clean LaTeX back. The hero must show a handwritten page beside its rendered output. |
| 03 Critique | HTML | A shipped landing page is supplied as a fixture. Find the five most damaging problems, repair them, and record what changed in a note of at most 150 words. |
| 04 Motion | HTML | A title card for a short film about a night ferry crossing. The word must rest legibly for at least two seconds, and the loop seam must be invisible. |
| 05 Explanation | Markdown | Cover the problem DNS solves, what happens between typing an address and the page appearing, why answers are remembered, and what usually goes wrong. |
| 06 Fiction | Markdown | A workplace after hours, third person, one character. The clock must earn its place: something in the story depends on it. |
| 07 Build | MJS | parseDuration("1h30m") returns seconds. It accepts h, m and s in any order, rejects invalid input with TypeError, and runs its own tests with node:test. |
| 08 Code review | Markdown | The fixture is a small keyed limiter. The thick version asks for severity, location, problem, why it matters and a direction, in at most eight findings. |
Run conditions
The runs are not identical. “Subagent tool calls” counts every tool call made by the sixteen subagents in that run. A clean run made none. Prompt time is the subagent’s start-to-finish, including tool use. It is not wall-clock: most runs dispatched in parallel batches, and HY4 ran sequentially.
| Run | Subagent tool calls | Slowest prompt | Median tok/s | Artifact fidelity | Status |
|---|---|---|---|---|---|
| DeepSeek 4.1 FlashRun 1 | 363 | 19m 31s | 118 | 16/16 saved; run used tools | Tool-assistedSubagents used a browser to refine their work. |
| GLM 5.3Run 1 | 3 | 12m 11s | 102 | 16/16; one CSS repair | One repairOne malformed CSS line was repaired after saving. |
| GPT 6 LunaRun 1 | 30 | 1m 33s | 73 | 14/16 exact; 2 post-edits | Skill-assistedSubagents loaded skills; two files were edited after saving. |
| Claude Haiku 5.5Run 1 | 0 | 42s | 210 | 16/16; one unfenced reply | One unfenced replyOne review arrived without a fenced block. |
| Claude Haiku 5.5Run 2 | 0 | 32s | 214 | 16/16 exact; one allowed retry | Clean runNo subagent tools; one identical retry. |
| HY4 FlashRun 1 | 0 | 1m 29s | 175 | 16/16 exact | Clean runNo subagent tools; all 16 files byte-exact. |
| Muse Spark 1.3Run 1 | 5 | 1m 06s | 88 | 12/16 exact; 4 trimmed | Not comparableFour visual files were saved trimmed. |
| Muse Spark 1.3Run 2 | 1 | 1m 15s | 96 | 9/16 near-exact; 7 trimmed | Not comparableSeven files saved trimmed; three prompts re-run after wording errors. |
Muse Spark run 2 also re-ran three prompts after wording errors; the saved artifacts use the corrected prompts. Its first review attempt used a different fixture and was discarded.
What we checked
- Every expected artifact exists and parses as its declared format.
- SVG dimensions, HTML self-containment, non-empty MJS and Markdown.
- Both build tasks run their own tests with
node --test; all saved tests pass. - Word counts for the 100-word story and the 150-word explainer; note length on the critique.
- Technical notes on the two code tasks: whether the review found the planted defects, and whether the parser meets its brief.
- No subagent read another task’s output; no artifact contains an external URL.
Limitations
- Not a leaderboard. These are eight runs, not a statistically meaningful sample. One run per model except Claude Haiku 5.5 and Muse Spark 1.3.
- Two runs are not comparable. Muse Spark’s saved files were trimmed and, in run 2, not disclosed. They are included for transparency, with a “Not comparable” badge.
- Process differs. DeepSeek’s subagents used a browser and wrote working files; GPT’s loaded skills. Those runs are marked as caveats, not hidden.
- Creative work is unscored. Posters, interfaces, motion, fiction and critique are shown as submitted. The absence of a score is deliberate, not an oversight.
A note on labels
Every card names the model and the run.
Claude Haiku 5.5 and Muse Spark 1.3 have two runs each. The run badge is part of the result, not a detail to ignore.
Browse the results