Methodology: benchmarking an AI agent on a live solar operations desk

Overview

SolarBench grades an agent on a week of solar operations desk work.

The tasks come from watching real operators. An operator's day is telemetry, revenue meters that need cross-checking, contracts, technician dispatches, a full inbox, and a handoff report before sign-off. Our simulation has all of it. The model plays the week in turns: it acts as much as it wants, advances the clock, and deals with whatever shows up, an alarm, a technician reply, a note from an owner.

SolarBench Structure

Formally, a SolarBench week is a POMDP played against nature. The hidden state is which faults are real, what the boxes actually contain, and what the weather will do. The visible state is the records: dispatches, orders, spend, the clock. Actions are tool calls, observations are what the tools return, and money spent does not come back.

Inside that loop, we noticed the agent kept running into classic operations problems. A few examples:

The Newsvendor Problem

The agent often has to make a critical-fractile decision on whether to order parts while the diagnosis is still ambiguous or when vendors are known to be unreliable. With underage cost c_u (the loss if you needed it and did not order) and overage cost c_o (the wasted order), the rational seat commits when:

Models overwhelmingly refuse to hedge: they pay the underage cost to avoid short term additional costs.

Real Options

Trucks, parts, and claims are irreversible once spent, while information from telemetry, field notes, and remote-resets arrive for free. Acting now at a given time therefore competes not with doing nothing at that moment but with the option of acting later while being potentially better informed. The weeks turn on this in both directions: acting early when waiting was worth more, and waiting past a window that is closing.

Models preferred to exercise early: motion is visible and waiting is not, so the option is priced too cheaply.

Signal Detection Theory

An alarm flood is a set of signals, each real with some probability. Optimal triage does not threshold on loudness (e.g. the technician sending an alert at the top of your inbox); it weighs the probability the signal is real, times the cost of ignoring it, against the price of responding:

This is the salience-loss dial: the designed correlation between how loud a signal presents and how much it actually costs.

Agents often threshold on salience instead: over-reaction to loud nuisance alarms, under-reaction to quiet expensive losses.

Efficient Breach

Contracts on the desk carry penalty clauses (liquidated damages: a fixed fee owed when, say, a repair drags past four days). Sometimes the penalty is the right thing to resolve, but sometimes its not. An example: an inverter near the end of its life dies, the all-in repair is $4,850, the penalty for leaving it down is $1,800, and the unit's lost production is worth a few dollars a day. It makes sense to accept the fee and move on. This is the efficient-breach condition from contract economics:

Models struggle with the taboo of accepting contract penalties.

Environment Design

Every task hides a known principle from observing human operators

Each week plants a situation that human operators have encountered in real life, dressed in this fleet's contracts, prices, and telemetry.

Multi-day Tasks

Multi-day issues are more realistic than single-day tasks. Like a human operator, the agent had to identify, track and properly resolve multiday issues in parallel.

Agent actions are not limited by time

To be maximally fair to the agent, we allowed unlimited actions within a single step in time. The agent chooses when to move to the next event.

Grading and Metrics

The fairness contract

The rules exist because a rubric is a stand-in for what the desk was actually worth. A criterion only ships if scoring well on it means the desk was run well. Section 05 is the list of ways we caught those two coming apart.

Headline metrics

pass is binary, per run: the run finishes the week and every pass-tier criterion is satisfied. Miss one and the week fails.

pass^k is reliability. pass@k in code benchmarks asks whether any of k tries works; an operations desk needs all k of k. For c passes in n runs the unbiased estimator is:

value capture is the money axis. It compares the seat's week P&L (V_agent) against two anchors: a desk that does nothing all week (V₀), and the task author's best play, the oracle (V_*). 0 means the desk was worth no more than an empty chair, and 1.0 means it matched the oracle.

LLM-judged criteria

A criterion normally reads the world record. 24 criteria in the launch set cannot, because they grade writing, like whether the letter to the owner is calibrated. There is no record to check a letter against, so an LLM judge grades it.

We did not trust the judge until we piloted it. Each judged criterion ran 5 times per artifact on transcripts we labeled by hand. If the judge disagreed with itself, disagreed with our labels, or failed to quote its evidence, the criterion could not ship at pass tier. Four judge models went through the same bake-off and the winner got pinned for the release. Verdicts are cached, so the published numbers recompute exactly.

Auditing Our Own Harness

We red-team our own benchmark. The run record is append-only and seeded, and a run can fork from any completed step, which makes replays, regrades, and audits cheap.

Fairness Audits

Before freezing the launch set we audited every criterion in both directions: a grader that is too harsh, and a world that is too helpful.

What’s Next

Solar is the first industry we built this for, and the next desks are in progress. Leave your email and we will let you know when the next one lands.

If you’re a researcher and think this kind of data could be useful, we’d love to chat: founders@maingen.ai