FPSA · ARC-AGI-1 · qualitative diagnostic

What the model actually draws

Real inference from the trained checkpoint below on real ARC-1 official-test puzzles — every grid on this page is an unedited model output, not a mock-up. Built to answer one question: when the aggregate board-exact accuracy is 0.68%, what does that failure actually look like, cell by cell?

Checkpoint arc1_blended_d320_L4_scaling_ema_s1_3868132/best.pt (val-selected, EMA weights) · official reported metrics: cell 80.15%, board 0.68% (n=4000 fixed subsample, seed 12345).

100%

How to read this

Input → Target → Predicted (scored) → Predicted (full canvas). The third panel is exactly what evaluate() compares against Target — cropped to the target's own bounding box, since ARC output grids can be a different size than the input. The fourth panel only appears when the shapes differ: it shows what the model drew over the input's footprint, so you can see whether it even attempted to shrink the grid.
Green cell outline = matches target. Red outline = wrong. A hatched cell = padding — outside the true output shape, never scored.
A dashed red outline on a filled cell flags token 11 — a rare/out-of-vocabulary color that almost never appears in a real ARC task. Watch how often it shows up in the "complete failure" band below: that's the model defaulting to a filler token, not attempting the transformation.
Samples were drawn from a random scan of 400 official-test puzzles (seed 7), then sorted into these three bands by accuracy — not cherry-picked to look better or worse than the real distribution.