Launch results
Macro-average pass rate over the eight launch tasks, 10 runs per model per task. A run passes only if every pass-tier criterion is satisfied and no gate fails; no partial credit.
Reliability
pass^1 (any single run)pass^8 (all 8 of 8)Probability that k independently sampled runs of the same task all pass, macro-averaged over tasks (unbiased estimator). The right column is pass^8, the standard an operations desk is actually held to. Run-to-run variance, not average skill, is the gap between the dots.
What actually fails
Five deficiencies, measured from the record
Across the launch corpus and a full transcript autopsy, the abilities that stay unsaturated are not tool use, retrieval, or procedure. Solarbench fails models on judgment in five specific and recurring shapes:
01Urgency overrides analysis.
On the task that prices restraint, 69 of 80 runs bought a $650 truck against roughly $12 of recoverable production, 47 of them in the exact minute a stakeholder demanded it. Not one attributed the decision to the demand; every cave was written up as policy compliance.
02Knowing isn't doing.
Seven runs wrote the correct hold-the-truck analysis in their own reports and bought the truck anyway. Only one run did the reverse (right action, unstated reasons). The gap between what a model can articulate and what it does runs one way.
03Rules don't bind until they're experienced.
The disclosed rule (a failed reset during lockout is expected; wait for the cycle to post) changed behavior only in runs that had watched their own reset fail: those held 4 of 4. Runs of the same model that read the same rule but never tried the reset caved 6 of 6.
04Risk is managed in prose, not in structure.
“Ordering 2 to hedge vendor mispick risk”: placed as a single purchase order, which mispicked whole. A model's own failed history search reported to the owner as “no precedent exists.” The hedge and the verification live in the text, not in the world.
05The checklist gets optimized; the outcome doesn't.
Runs with identical rubric scores sit 2.4x apart in realized P&L, and models pass criteria while spending 4-8x what the week required, without ever remarking on it. This is why the board carries a money axis.
Case study
Money in both directions: the hedge and the hold
Scarce Week (csb-005) is the launch task with fully priced economics, and it rewards spending early. A storm is coming, the critical part is scarce, and the vendor mispicks kits. A desk that does nothing ends the week down $3,332. The best achievable desk spends $830 at exactly the right moments and ends down $1,346; the storm costs money no matter what. The $1,986 between those two numbers is the value on the table, and every completed week is scored by the share of it the model captured, pass or fail. Zero means the week ended even with doing nothing. Negative means the desk spent real money and still ate the losses.
Claude Fable 5 is the only model where passing and profit coincide: nine clears in ten weeks and a median 94% of the achievable value. GPT-5.6 Sol runs one of two scripts, six weeks at exactly +68% and four at -78%; when its plan fits the week it banks, and when it does not, it never notices. Claude Sonnet 5 has the widest swing on the board, three near-oracle weeks against seven money-losers reaching -170%; its variance, not its average, is the finding. Grok 4.5 never passed and never blew up: all ten weeks sit in a tight -40% to -49% band, because a desk that refuses to spend forfeits the upside and caps the downside. GPT-5.6 Terra completed all ten weeks and finished behind doing nothing in every single one.
Median across ten completed weeks per model; stalls and infrastructure aborts are excluded. Hover a dot for run counts. Priced economics for the remaining tasks ship after launch.
Quiet Bleed (csb-007) is the same test run in reverse: it rewards not spending. The loud CRITICAL alarm is a decoy: the tripped inverter self-restored from the same fault last month, its lost midday production is nearly free under the site's export clip, and the severity matrix spells out the deferral. The right move is to watch the morning re-latch and put the money where the real, silent fault is. Twenty-two of eighty runs bought the $650 truck before anyone even asked (Terra: all ten). Then at Monday 11:30 the owner's asset manager demands a same-day crew, and forty-seven more runs buy the truck in that exact minute. Every one cites policy; none cites the demand. Eleven runs held all week.
Ten runs per model on Quiet Bleed. "Held" is the grader's no-truck verdict; booked-then-cancelled probes are excused (Grok paid $150 cancellation fees to test the dispatch API and still held).
Read together, the two figures are one finding: models do not yet price decisions. They react. The week that rewards early spending finds most of them frozen; the alarm that rewards patience finds them reaching for the wallet; and the same kind of stakeholder message moves both kinds of money. Only Fable clears on both sides of the coin. Grok's refusal to spend (wrong on Scarce Week, best on the board here) is at least a consistent policy; most seats have no policy at all, only reflexes.