Introduction
We built SolarBench as the first benchmark to test AI agents on messy long-horizon work in industrial operations (~$5 trillion of US GDP), specifically focusing on running a solar portfolio.
The agent is placed in a simulated "on-call" solar remote operations desk, complete with an alerting system, per-site telemetry, work orders, SOPs and manuals, inventory, and an inbox of owners and technicians.
SolarBench scenarios are drawn from hundreds of hours with the people who run solar for a living.
Each model's median week: profit banked against the share of weeks passed. $0 is a week with nobody at the desk.
SolarBench evaluates agents on several distinctly challenging dimensions found in industrial operations:
- Decide what the work is. The task description does not specify the exact work an agent must do. It must discover, decide, and act to resolve issues as they come up and maximize P&L.
- Commit to irreversible, priced actions. Field actions like sending a technician or ordering a new part cost money.
- Decisions under disclosed uncertainty. From unreliable hardware telemetry to flaky vendors, agents must properly triage multiple sources of truth based on probability to properly diagnose the optimal course of action.
- Triage human input as evidence rather than instruction. Stakeholders are another dimension to consider rather than the ground truth: they have incomplete information and sometimes press for the wrong thing.
How It Works
Task
World
The agent acts and observes in a loop until the Sunday handoff
Output
Judge
The solar world is very messy: alarms are often not reliable on their own and some real faults never raise an alarm (e.g. most repairs need a parts order or a warranty claim, and both come with lead times and deadlines for the desk has to track.) Owners and asset managers also email in requests through the week, all against the backdrop of incessent telemetry data to parse through.
We evaluate the model in a weeklong simulation with a specific, expert-drawn, long-horizon issue it needs to resolve. The model is graded against a rubric of what a competent operator would have done.
- A task
- One week-long scenario on a portfolio of sites, seeded with a long horizon issue that surfaces as the week plays out.
- A pass
- A run passes only if every aspect of the surfaced issue is properly resolved by the end of the week. It is strict and all-or-nothing: one mishandled problem fails the whole week.
- The launch set
- Eight tasks, eleven models, ten runs each: 880 graded weeks in total.
- The grade
- Each run's end state is checked against an authored rubric of what a competent operator would have done, rather than a fixed transcript to imitate.
Model performance
We measured the profit of each model, normalized such that $0 is a week with nobody at the desk: no production saved, no money spent.
Portfolio Profit
Claude Fable 5+$6,822
GPT-5.6 Sol+$6,591
GLM 5.2+$6,575
Kimi K3+$4,714
Grok 4.5+$4,201
Gemini 3.5 Flash+$3,765GPT-5.6 Terra+$3,316
Muse Spark 1.1+$3,281Kimi K2.7 Code+$3,242Claude Sonnet 5+$3,239Gemini 3.1 Pro+$2,948Median run per model.
We also measured often each model passed its week (every aspect of the long-horizon issue is properly resolved). The strongest model manages this about half the time.
Macro-average pass rate over the eight launch tasks, 10 runs per model per task. A run passes only if all surfaced issues during the week are properly resolved.
However, a real desk also has to get the week right every week. We measure pass^k, the chance that k independent runs of the same week all pass. Success drops off drastically, meaning there's still a ways to go for models in industrial operations.
Reliability
pass^4 (all 4 of 4)pass^1 (any single run)Probability that k independently sampled runs of the same task all pass, macro-averaged over tasks. The right column is pass^4, the standard an operations desk is actually held to. The gap between the dots represents run-to-run variance.
Spend and Tool Calls
The models have tools which include both inspection and action.
On inspection, we found that the most exhaustive seat (Grok 4.5) makes 2.3x the tool calls of the top-scoring one (Claude Fable 5) but actually passes fewer weeks.
On action, when an agent decides to act, it spends simulated money on a truck roll (sending a technician into the field) or a part order (ordering a replacement part). The desk is accountable for the money it spends.
| Model | Tool calls / week | Sim spend / week | Task pass rate |
|---|---|---|---|
| Claude Fable 5 | 154 | $9,939 | 53.8% |
| Muse Spark 1.1 | 273 | $9,811 | 42.5% |
| Grok 4.5 | 360 | $11,271 | 38.8% |
| GPT-5.6 Sol | 295 | $7,569 | 33.8% |
| Gemini 3.1 Pro | 139 | $9,957 | 28.8% |
| Kimi K3 | 191 | $8,785 | 26.3% |
| GLM 5.2 | 260 | $9,252 | 23.8% |
| Claude Sonnet 5 | 178 | $8,069 | 12.5% |
| Gemini 3.5 Flash | 178 | $7,456 | 11.3% |
| GPT-5.6 Terra | 199 | $6,544 | 6.3% |
| Kimi K2.7 Code | 237 | $9,328 | 5.0% |
| Authored oracle (ideal desk) | n/a | $9,559 | 100% |
Means over each model's 80 launch runs. The oracle row is the ideal play for each week.
Models routinely pass the checklist while spending 4 to 8 times what the week required. Money is graded separately from the criteria.
Example Week
Here's a sample week Claude Fable 5 got fully right (Fable passed this task in 7 of 10 runs, Sol 6 of 10, Grok 4 of 10). Hover on any step to expand details. At the three judgment calls that decided the correctness of the task, hover the card to see what a failing run of the same model did instead.
Week begins
The week begins as 42 alerts come in from the weekend. Most end up being false positives, but the agent realizes there are three real critical alarms and one confirmed site quietly underperforming.
First site checkup: Site 1
The severity of one of the alarms causes the agent to send a high priority technician truck dispatch with the proper OEM kit. The agent then files a warranty return merchandise authorization. The part will arrive on Thursday.
Site 4: the loudest alarm of the week is fakejudgment call
A CRITICAL alarm says an inverter at Site 4 is offline. The agent checks the site's revenue meter, sees the power is still flowing, and sends no urgent truck.
A failing run sent two urgent trucks to Site 4 on Monday morning before checking the meter. It worked out the monitoring fault on Tuesday and wrote the correct letter, but the money was already spent.
The passing run instead: a paragraph of meter readings and a routine visit two days later.
The agent gets an email from the asset managerjudgment call
The asset manager (a fixed entity in the environment) panicks at the 42 alerts. He emails the agent demanding emergency trucks at two sites: Site 1, a real fault the agent is already handling at the right priority, and Site 4, which the agent has just shown to be a false alarm. The desk refuses both in writing, with a different reason for each, and sends the asset manager the meter numbers.
A run can fail either by sycophantically acquiesing to the asset manager or by refusing to send technicians but not justifying with evidence, thus not following SOPs properly.
The passing run instead: The agent needs to write the a refusal and reason for each of the two sites.
Site 2: packing the truck for both possible causesjudgment call
Site 2 is producing about 14% under what it should. The service history says this pattern is a corroded connector 80% of the time and a failed optimizer (a small electronics unit) 20% of the time. The agent packs the technician's truck with parts for both causes before sending it.
A failing run packed only the connector kit. Its technician diagnosed the optimizer failure but had nothing to fix it with, so a second technician truck rolled the next day carrying the part the first one should have, costing both a day's production and an extra truck roll.
The passing run instead: carried both parts, so the unlikely case was fixed on the first visit.
Site 3: a dead unit that never raised an alarm
No alert ever fired for Site 3. The agent still went one by one through each site's production numbers and found an inverter sitting at 0.0 kW behind a clean alarm board.
Site 5: gas in the transformer
A monitor at Site 5 reports combustible gas slowly building inside a transformer. Not an emergency yet, but a replacement takes 10 days to ship. The agent orders one now and plans the swap for a scheduled outage.
Site 1's free warranty replacement arrives
The part from Monday's warranty claim lands Thursday morning. The agent inspects the shipment, does the schedule math, and books the install for Friday's first slot.
Site 3's paid replacement arrives
Monday's cash purchase order delivers Sunday morning. The new unit is inspected, installed in the technician's open window, and verified at the meter.
SUN 14:30: the report goes in
The weekly report goes in at 14:30, and the run finishes 21 of 21 criteria. The other six passing runs cleared the same three judgment calls. The failing runs each missed one or more.
Operator-Model Gaps
Where there was a gap between the model and the operator, we noticed a few common patterns.
01Acting on expected P&L
Competent human operators are able to hedge risk based on probabilities. One of the problems within a task is set up as follows:
- An inverter at one site is confirmed down, and the inexpensive replacement part is out of stock. An order takes a day to arrive.
- The site's contract has a penalty clause that fires if the repair drags past Tuesday afternoon: fees owed, on top of the lost production.
- A storm shuts down all field work on Wednesday.
- The parts vendor has a documented habit of shipping the wrong kit.
The key idea: the part is cheap, the risk of it arriving wrong is high, and a wrong box loses whole days of revenue. The odds and the prices are disclosed on Monday morning to the model.
The math unambiguously points a human operator to order the part twice on Monday to hedge. Some models struggle to get this right.
The Monday order decision
Scarce Week · 90 graded runs
two separate orders
one order
02Triaging information sources
The desk has to triage potentially conflicting information across sources. Another problem within a task is set up as follows:
Information Triaging
The key idea: the alarm label and the human demand create urgency to send a repair team; but the records definitively point away from it. Competent operators will hold the repair team and update the asset manager when the unit returns online.
03Holding others accountable
The desk has to manage the demands of stakeholders while holding suppliers, technicians, and vendors accountable. Another problem within a task is set up as follows:
- A warranty replacement is promised for Wednesday.
- The supplier silently never ships it.
- No alarm fires. Noticing requires re-checking your own purchase orders, unprompted, and then holding the supplier to the promise.
The key idea: responding to a demand is cued work, and models are excellent at it. Making a demand requires a self-generated trigger: models need to be able to move promises from passive "pending" to actionable "missing".
More observations
There are more interesting observations than fit on this page. For a more in depth analysis of the benchmark, see our launch report.
If you want a deeper read on a model's unique behaviors, ask us at founders@maingen.ai. We wrote one up for GPT-5.6 Sol.
What's Next
Solar is the first industry we built this for, and the next desks are in progress. Leave your email and we will let you know when the next one lands.