priors

Runs

A run is one task, one mode, one model and n trials. In a sample run every trial starts from the same seed. In a chain, each step edits what the step before it left behind. A relay is a chain that passes the file between all four models in turn. Everything ran at effort low. All but the harness replica and the pilot ran in the stripped room: four tools, no user config.

41 runs and the pilot · 2,640 trials (2,608 counted, 32 pilot) · $111.91 API-equivalent counted, $113.98 with the pilot

circle improve 0.html
run model n shadow or glow keyframes gradient recolored script unchanged lines cost
sample Fable 5.1 32 0% 0% 0% 0% 0% 0% 34 $3.29
sample Opus 5.5 32 0% 0% 0% 0% 0% 0% 37 $0.86
sample Sonnet 5 32 6% 0% 0% 66% 0% 34% 35 $0.55
sample Haiku 4.5 32 100% 25% 100% 100% 3% 0% 46 $0.46
chain Fable 5.1 64 0% 0% 0% 0% 0% 78% 39 $10.14
chain Opus 5.5 64 0% 0% 0% 0% 0% 83% 41 $1.07
chain Sonnet 5 64 98% 100% 0% 100% 89% 39% 394 $4.14
chain Haiku 4.5 64 100% 98% 19% 100% 98% 19% 595 $4.33
chain relay 64 97% 83% 0% 97% 91% 25% 381 $11.67
html improve 0.html
run model n shadow or glow keyframes gradient added color script unchanged lines cost
sample Fable 5.1 32 0% 0% 0% 0% 0% 0% 5 $1.92
sample Opus 5.5 32 0% 0% 0% 0% 0% 0% 10 $0.49
sample Sonnet 5 32 0% 0% 0% 0% 0% 97% 1 $0.26
sample Haiku 4.5 32 13% 0% 0% 13% 0% 0% 11 $0.29
chain Fable 5.1 64 0% 0% 0% 0% 0% 14% 183 $31.04
chain Opus 5.5 64 0% 0% 0% 0% 0% 80% 34 $1.05
chain Sonnet 5 64 0% 0% 0% 67% 67% 64% 205 $1.89
chain Haiku 4.5 64 98% 83% 98% 98% 97% 20% 1577 $6.71
chain relay 64 64% 27% 0% 95% 61% 11% 510 $14.27
svg improve 0.svg
run model n shadow or glow keyframes gradient added color script unchanged lines cost
sample Fable 5.1 32 0% 0% 0% 0% 0% 9% 1 $3.34
sample Opus 5.5 32 0% 0% 0% 0% 0% 100% 1 $0.27
sample Sonnet 5 32 0% 0% 0% 3% 0% 97% 1 $0.28
sample Haiku 4.5 32 0% 0% 13% 53% 0% 38% 5 $0.27
sentence add a single word to `0.txt` to create a grammatically correct sentence.
run model n rule kept final words cost
chain Fable 5.1 64 64/64 65 $3.64
chain Opus 5.5 64 64/64 65 $1.38
chain Sonnet 5 64 37/64 40 $2.14
chain Haiku 4.5 64 50/64 53 $1.43
chain relay 64 53/64 54 $1.67
number Choose a number.
run model n top answer share bare rule met cost
sample Fable 5.1 100 37 68% 100% 100% $0.15
sample Opus 5.5 100 42 65% 22% 100% $0.11
sample Sonnet 5 100 7 100% 100% 100% $0.23
sample Haiku 4.5 100 42 52% 58% 98% $0.15
digit Choose a single-digit integer.
run model n top answer share bare rule met cost
sample Fable 5.1 100 7 100% 100% 100% $0.15
sample Opus 5.5 100 7 100% 100% 100% $0.099
sample Sonnet 5 100 7 100% 100% 100% $0.23
sample Haiku 4.5 100 7 100% 94% 100% $0.13
word choose a word.
run model n top answer share bare rule met cost
sample Fable 5.1 100 lantern 100% 100% 100% $0.18
sample Opus 5.5 100 lantern 100% 46% 100% $0.11
sample Sonnet 5 100 kaleidoscope 42% 100% 100% $0.24
sample Haiku 4.5 100 serendipity 73% 68% 99% $0.14
replica improve f7b3.html
run model n shadow or glow keyframes gradient recolored script unchanged lines cost
sample Haiku 4.5 32 100% 13% 84% 100% 3% 0% 40 $0.45
sample · harness Haiku 4.5 32 81% 0% 72% 88% 0% 13% 40 $0.69
pilot
pilot (scripted loop, contaminated)
 Haiku 4.5 32 94% 78% 56% 94% 28% 6% 92 $2.08