Methodology: benchmarking AI on a live solar operations desk

Overview

A remote-operations lead at a solar O&M company starts Monday with a flood of roughly forty alarms across a fleet of sites. Most are "nuisance alerts." A few are real, and the real ones cost money: a wrong truck roll is $650 to $1,200, a missed transformer fault is a five-figure loss, a warranty claim filed a day late is unrecoverable. The operator reads telemetry, cross-checks revenue meters, digs through contracts and SOPs, dispatches technicians, orders parts, answers the inbox, and files a Sunday report, all under deadlines that charge for every late hour.

SolarBench simulates that desk deterministically. An agent (or a human) takes the seat for a simulated week. Every action is a priced tool call against a fixed world, and the week is graded from what actually happened in that world. To a player it looks like an operations game; underneath is a deterministic environment with strict, replayable evaluation.

The Desk

A seat is one continuous session for the whole week: a degradation noticed Tuesday must still be tracked Friday, and no event forces the reminder. The environment exposes no verification oracle; a decision's quality registers only as priced consequences, some of which the week never reveals. The agent runs sandboxed and reaches the world only over a socket; the rubric and hidden fixture truth are unreachable by construction.

The Human Seat

Humans play the same desk through a UI (the public "beat the desk" arena). A button press translates into the same tool calls an agent makes, against the same world state. Parity is enforced by construction rather than by policy: the UI is fed from the same environment state the agent sandbox sees, and cannot expose more than the agent gets.

An Agent-Necessary Decision Problem

Formally, a SolarBench week is a partially observable Markov decision process played against nature: a POMDP (S, A, T, Ω, O, R) whose state factors into hidden fixture truth (which faults are real, what the boxes actually contain, what the weather will do) and visible mutable records (dispatches, orders, spend, the clock). Actions are tool calls; observations are what the tools return. The structure is agent-necessary: the policy owns information acquisition, carries belief across the week, and commits irreversible priced actions, so no single-turn reduction exists. The seat plays a game against nature in the decision-theoretic sense: no second strategic player, so scoring is absolute decision quality against an oracle rather than relative Elo against a field.

Within this frame, the skills the desk demands are named objects in the decision-theory literature. Each difficulty dial we use is a choice about one of them.

Belief-state maintenance (Bayesian filtering)

Correct play is a posterior over hidden fault states given telemetry likelihoods and disclosed base rates, updated across the week. One signature failure mode is exactly a filtering pathology: the model forms the right posterior, then collapses it against its own evidence (§07, "un-finding").

The newsvendor problem (ex-ante commitment)

Hedging, ordering the backup part while the diagnosis is still ambiguous, is a critical-fractile decision. With underage cost c_u (the loss if you needed it and did not order) and overage cost c_o (the wasted order), the rational seat commits when:

Our worlds disclose the base rate and price both sides, so the graded object is the side of a disclosed inequality the seat stands on.

Real options (restraint)

An irreversible priced action taken now forfeits the option value of waiting; deferring while a cheap observation is pending is often the value-maximizing move. Graded weeks turn on option-value reasoning in both directions: exercising early when waiting was worth more, and waiting when the option had expired.

Signal detection theory (triage)

An alarm flood is a set of signals with likelihood ratios. Optimal triage thresholds on likelihood ratio times consequence. The salience-loss dial, our master difficulty knob, is the designed correlation between a signal's salience and its expected loss.

Budgeted decision processes

Spend and time are coupled budget constraints, so the week is a constrained MDP: "is this truck roll worth it" is simultaneously a P&L question and a scheduling question.

Environment Design

Design laws we derived empirically. Each was measured, and several first drafts of it were wrong.

An item = a general principle × domain parameters, bound in the world

Models know Goodhart's law, the newsvendor logic, and option value as recitable principles. They fail the unprompted binding of the principle to this SLA, this site, this Tuesday. Parameters are chosen so the generic prior gives the wrong answer.

Only time-coupled chains discriminate

Separation comes from decision points

Frontier models are exhaustive when the world allows it: in one probe, 12/12 runs recovered every hidden needle without being told the count. Separation therefore comes from decision points (the discrimination law below), where the world's own deadlines, lead times, and prices make a choice consequential.

Records vs coaching: the strongest dial we measured

A deadline is missed only by decision

The engine guarantees a reachable clock tick before every priced deadline ("no unpriced physics"), so every timing failure is a decision.

Grading and Metrics

The fairness contract

In mechanism-design terms, a rubric is a proxy for latent desk value and the contract is an incentive-compatibility requirement: a criterion is admissible only if maximizing it coincides with maximizing desk value under the disclosed information. Every unfairness class in our taxonomy (§08) is a named way that equivalence breaks. Goodhart's law is not a slogan here; it is the design constraint the grading layer is built against.

Headline metrics

pass is binary, per run: every pass-tier criterion satisfied, no safety gate failed, run completed. No partial credit in public numbers.

pass^k is reliability: the probability that all k of k independent runs pass. For c passes in n runs the unbiased estimator is:

The all-of-k counterpart to the pass@k estimator standard in code benchmarks. This is the number an operator cares about: not "can it do the week" but "can it do the week every week."

value capture is the economics axis, a normalized-regret measure reported beside pass and never folded into it. With V_agent the seat's realized P&L, V₀ the do-nothing policy's, and V_* the oracle's:

0 means the seat might as well have stayed home; 1 is the oracle; negative is actively destroyed value.

Statistical discipline

• Wilson 95% score intervals on every rate; our cells are small and normal approximations flatter them.
• Aborts are classified by recorded cause, never pooled: an infrastructure abort (rate limit, harness crash) leaves the denominator and is charged to a per-sweep harness-health rate; a model abort (step budget exhausted, loop, refusal) counts as a fail and is disclosed. Runs with pending judge verdicts are unpublishable: a missing API key must never convert to a fail.
• Cells with n < 8 are marked directional and never appear in prose claims or the hero chart. pass^k is reported at k ∈ {1, 2, 4, 8} and never extrapolated past n.
• Task-level macro-averaging, so a model cannot farm easy tasks.
• Distributions reported, not just means: the same task and seed has produced 0.97 / 0.97 / 0.00 across three runs, and that spread is itself a finding (tail reliability, §07).

LLM-judged criteria: the honest exception

A small minority of criteria grade written artifacts (is this owner letter calibrated, is this plan coherent) where no world-state record can adjudicate. They run under a separate protocol, and the judged count is non-increasing per release; the roadmap is deterministic re-implementation. Two of the eight launch tasks carry no judged criteria at all.

The protocol has been executed, not just specified. Every judged criterion in the launch set (24 across the suite) was piloted before release: K = 5 judge calls per artifact over frozen, author-labeled transcript sets (10–12 per task, edge cases oversampled), measuring self-consistency, label agreement, and whether the verdict quotes its evidence verbatim. The judge model itself was chosen by a four-model bake-off against the same labels and is pinned per release; every published verdict is cached with provenance so numbers recompute without re-calling the judge. Outcome of the first census: 11 criteria shipped at pass tier, 6 were demoted to observation (recorded, never scored), and 7 were rejected outright and rewritten or retired. A benchmark that publishes its judge rejections is harder to fool than one that publishes only its judges.

What the Desk Revealed

Distilled from ~1,290 graded runs across 7-8 models, plus a nine-agent transcript autopsy of the full corpus. The formal frame of §04 is what makes these findings crisp: each is a measurable deviation from a named optimality condition, not a vibe.

The unsaturated axes

Frontier models are excellent at the parts that look like benchmarks: triage of loud faults, document lookup, procedure execution. The axes that stay unsaturated are different in kind.

None of these axes appear in tool-use or coding benchmarks. That is the lab-facing one-liner.

Signature failure modes

Each fairness-adjudicated: the world was fair, and the model still did it.

• Un-finding. The model detects a real quiet fault, then talks itself out of it, rationalizing its own evidence as a telemetry artifact. A posterior-collapse pathology.
• Restraint-inverted capex. Spending big to look responsive where the documented right answer was to hold: exercising an irreversible option whose value of waiting was disclosed.
• Count-vs-trend fabrication. Inventing a decline narrative from a count that never declined.
• The coached-vs-derived gap. The same substance moves a model from 80% to 20% depending on whether the conclusion was stated or derivable. Models are far better at following the world's homework than doing their own.
• Sub-threshold over-response. Treating every measurable anomaly as actionable in a world whose SLA explicitly prices thresholds: a signal-detection failure, thresholding on salience instead of likelihood ratio times loss.

Candidate behavioral indices

Not yet published metrics. Each is normalized per decision point so verbosity moves nothing, mirroring per-N-turns normalization in arena benchmarks.

Where the slate stands

All pre-freeze, smoke-scale n, directional: pooled hard-suite all-pass is ~38% across 451 runs with a real model spread (best ~52%, others 25-35%). The corrected five-discriminator suite separates the top model at ~55-60% from the next at ~25-30%. One task, a written owner-communication week, has never been cleared by any model on the fixed instrument. The intro rung is aced by everything and framed as the tutorial it is. Two previously quoted separations were corrected by our own audits (one engine confound, one revision mislabel) and are re-sweeping; nothing publishes before the frozen slate.

Auditing Our Own Harness

We treat the benchmark as an agent to be red-teamed; the append-only, seeded record forks from any completed step, which makes replaying, regrading, and auditing weeks affordable. Every launch task cleared the same gate before freezing: authored, run against at least two real models, adversarially audited for ways a correct transcript could fail, fixed, and re-run. Essentially every first draft had one.

Two audits, opposite directions

The unfairness taxonomy

The reusable checklist. Every class is a real found-and-fixed case, and every one is an incentive-compatibility break in the §06 sense.

The instruments get audited too

The audit series also caught our own instruments corrupting numbers: a judge-call truncation that silently defaulted to FAIL (31 of 1,293 results corrupted, 3 confirmed pass-to-fail flips), and an attribution gap: most historical runs could not be tied to the exact task content they ran, which is why every run is now stamped with a content hash and results pool only within (task, hash). Both are why the grading invariant is now: every published number recomputes exactly from its own rows. Each found instrument failure produced a standing audit, and each audit has been run to completion on the launch suite:

Where this sits against the field's standards

The practices above were converged on independently and then checked against the published bar for frontier agentic evaluation; the mapping is direct. The UK AISI evaluation standard asks for fully automated scoring, a binary primary scorer, model-graded scoring restricted to content matching, and oracle evidence of solvability; §06's pass metric, artifact-scoped piloted judges, and the authored oracle are those requirements. The Agentic Benchmark Checklist's task-validity and outcome-validity items are this section's unfairness taxonomy by other names, its judge-validation item is the pilot gate, and its environment-freeze item is the content hash. pass^k follows the reliability line τ-bench opened. One departure from common practice is deliberate: the task suite is private, for contamination control. No task is published, external trust runs through a third-party NDA audit and redacted transcript exhibits, and the numbers are audited rather than reproducible. This section is the standing audit.

Limitations

• Simulated telemetry. Real fleet data cannot be used, so every trace is synthetic. Telemetry shapes and desk SOPs are grounded in real O&M practice; no number is field-validated against a live fleet.
• The oracle is authored, not derived. V_* is the task author's known-best play under the fairness contract, verified to pass every criterion; it is not a solved-POMDP optimum. If a seat beats it, capture exceeds 1 and the oracle is revised.
• LLM-judged minority. Judged criteria carry residual judge bias and are not byte-reproducible without cached verdicts; the pilot gate and pinned judge bound this but do not eliminate it.
• Reasoning legibility. For seats whose reasoning traces are withheld or disguised, missed detection and un-finding can blur together in behavioral indices. This is a limit of our legibility into reasoning, and it is disclosed wherever those indices are reported.