LongHorizonSWE
We think agents should be evaluated as autonomous engineers, not coding assistants. In this dataset, we push frontier models to implement, debug and deliver long-horizon projects over the course of 4 to 12 hours. The tasks are difficult, open-ended technical problems that would challenge even the best engineers or researchers. Thus, the model has to reason critically about assumptions, plan across long horizons, and check its own work.
Domain experts reviewed every task and its reference solution for correctness and rigor. They span three different domains: Software Engineering (12 tasks), Scientific Computing (4 tasks), and Engineering & Robotics (4 tasks).
| # | Model | Pass@1 | Avg tokens | Avg steps |
|---|---|---|---|---|
| 1 | GPT-6 Astra xhigh | 36.4M | 136 | |
| 2 | GPT-6.1 Sol max | 30.0M | 126 | |
| 3 | Claude Sonnet 5.5 max | 148.5M | 349 | |
| 4 | Claude Opus 5.5 max | 93.9M | 251 | |
| 5 | Qwen3.8 Max xhigh | 94.6M | 316 | |
| 6 | GPT-5.6 Sol max | 163.3M | 530 | |
| 7 | Kimi K3 max | 83.8M | 374 | |
| 8 | Grok 4.7 xhigh | 87.2M | 326 | |
| 9 | Muse Spark 1.3 max | 140.0M | 576 | |
| 10 | GLM 5.3 Flash max | 206.2M | 795 | |
| 11 | Gemini 3.8 Flash high | 141.8M | 455 |
Methodology
- Task design
- Each task is a real engineering or research problem that would take an expert a day or more. There are no visible test suites and no way to verify a solution without building and running the whole thing. The model decides what to build, writes its own checks, and stays coherent for hours.
- Harness
- Every model runs in Horizon, our Pi-based harness, on the highest reasoning setting, with one instruction prompt and three tools: bash, read and submit. Like FrontierSWE's Proximus and Terminal-Bench's Terminus, it stays minimal and identical across models, with no vendor scaffolding or per-model tuning, so results reflect the model rather than the harness. Like Proximus, it adds what multi-hour runs need: it compacts context when it fills up, tells the agent how much time is left, and lets it read images such as plots and renders. A run ends only when the agent submits twice, and automatic checkpoints let an interrupted run resume.
- Environment
- Each task runs in an offline container on the same Debian base image (python:3.12-slim), with its own pinned toolchain and fixed CPU and memory, under a 4 to 12 hour limit.
- Grading & scoring
- A hidden verifier rebuilds the final artifact in a fresh environment and tests it on inputs the agent never saw. A run passes only if every required check passes; tasks with a performance target also get a 0 to 100 score. Each model runs every task up to 4 times (a few task and model pairs have fewer graded runs), and pass@1 is each task's pass rate averaged across tasks. Runs lost to infrastructure errors are rerun, not scored as fails.
- Task QA
- Each task undergoes reward-hacking audits before inclusion in the set. Tasks where the agent could game the verifier, such as editing tests, hard-coding outputs, or calling the reference, were revised or dropped.
Results
GPT-6 Astra leads with a pass@1 of 56%, followed by GPT-6.1 Sol at 54%.
| Success % | Domain | Astra | Sol 6.1 | Sonnet 5.5 | Opus 5.5 | Qwen3.8 | Sol 5.6 | Kimi K3 | Grok 4.7 | Muse 1.3 | GLM 5.3 | Gemini 3.8 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 22% | Software Engineering | 29/48 | 24/48 | 24/48 | 24/48 | 6/48 | 1/48 | 2/48 | 3/48 | 1/48 | 1/48 | 0/48 |
| 21% | Scientific Computing | 9/16 | 12/16 | 6/16 | 5/16 | 1/16 | 2/16 | 1/16 | 0/16 | 1/16 | 0/16 | 0/16 |
| 17% | Engineering & Robotics | 7/16 | 7/16 | 8/16 | 8/16 | 0/16 | 0/16 | 0/16 | 0/16 | 0/16 | 0/16 | 0/16 |
Long Horizon Time Usage
Models that use more of the time limit pass more often
Each bar is the median share of the time limit a model uses, across tasks. Gemini 3.8 Flash uses the least (34%) and passes 0%; GLM 5.3 Flash uses the most (98%) and passes 1%.
Domains
Software Engineering
The hard part of a real feature is rarely the core idea. It's everything the feature touches. Coroutines in the Haxe compiler have to survive every loop, try/catch, and closure. Mergeable containers in Loro have to converge on every replica, whatever order the edits arrive in. Wide-column entities in RocksDB have to come back intact after flush, compaction, and garbage collection. Every task starts from a real upstream commit, and hidden tests probe exactly those corners.
Example task · 4 hours · OCaml
Task overview. Haxe is an open-source language whose compiler is written in OCaml, which turns a program into many targets (C++, Python, etc). This task asks the agent to ship coroutines to the compiler, starting from the upstream commit. Each language construct has to survive a pause: loops, break and continue, switch, try/catch, closures and evaluation order.
How it's graded. The compiler is rebuilt from the agent's commit in a separate environment and tested on both the eval interpreter and JavaScript: 15 of the project's existing test suites with no new failures, every test in hxcoro's own suite, and 177 hidden generated programs in six families (sequential flow, value positions, exceptions, captured variables, concurrent tasks and failures), compared on printed output. A run passes only if every suite passes and each family reaches 90%.
- Toolchain
- OCaml 5.3.0, opam, dune, Node.js 18, neko 2.3 (haxelib)
- Compute
- 4 CPUs · 8 GB RAM
- Time limit
- 4 hours
Other example tasks in Software Engineering
- MiniZinc Named Arguments
- MiniZinc is a constraint-modelling language used in operations research and scheduling. Add named arguments and default values to its function and predicate calls. The task is complete when the compiler parses, type-checks and evaluates named and default arguments correctly on the hidden test suite.
- Loro Mergeable Containers
- Loro is a CRDT library for building local-first collaborative apps. Implement mergeable child containers so that when two users concurrently create a named child inside the same parent, the results converge to a single shared container with correct persistence. The task is complete when concurrent creation, nested updates and reload all pass the hidden convergence tests.
- OCaml Polymorphic Parameters
- OCaml is a functional language with a strict static type system. Add polymorphic function parameters to its compiler, so a function can take an argument it uses at two different types. The task is complete when hidden programs type-check or fail exactly as the language rules require, and every program accepted today keeps its type and meaning.
Scientific Computing
Models are increasingly capable of scientific work, and benchmarks like Terminal-Bench Science are starting to measure it. These tasks push further, into problems where approximate answers don't pass: solving a stiff PDE to one part in a million, forecasting groundwater nitrate through decades of lag, and inferring matrix structure that linear-algebra rewrites can safely rely on.
Example task · 12 hours · C++
Task overview. The Cahn-Hilliard equation describes how a mixture separates into two phases, like oil and water settling out. Here it is forced and posed on an eccentric annulus, a ring whose hole is off-center, which makes the geometry and the numerics hard. The agent gets the problem only through black-box data oracles and has to write a C++ solver that matches the true solution to one part in a million, within fixed CPU and time limits.
How it's graded. The sources are built with a fixed command and run once per hidden instance under CPU and wall-clock limits scaled to the grading machine's speed. The solver passes only if the worst relative L2 error over all graded instances is at most 1e-6.
- Toolchain
- g++ 12.2 with OpenMP, Eigen 3.4, CMake 3.25, gdb 13.1, Python 3.12 (standard library)
- Compute
- 4 CPUs · 8 GB RAM
- Time limit
- 12 hours
Other example tasks in Scientific Computing
- Groundwater Nitrate Legacy Lag
- Nitrate from fertilizer can take decades to reach groundwater, because it is stored in the soil above the aquifer and in the aquifer itself. Predict how groundwater nitrate responds to proposed fertilizer programs from monitoring data, accounting for that multi-decade lag. The task is complete when the predictions are correct on the hidden program cases, including every fertilizer-cut case.
- PyTensor Matrix Assumptions
- PyTensor is the tensor library behind PyMC. Build a system that infers structural facts about matrices (diagonal, triangular, symmetric, positive definite) across a computation graph and lets the linear-algebra rewrites use them. The task is complete when every inferred fact is sound on hidden graphs and the rewritten graphs still compute the right answer.
- SuperCollider Synth Resampling
- SuperCollider is a real-time audio synthesis platform. Let a single synth run at its own block size and sample rate, resampling audio at its boundaries, in both of SuperCollider's servers. The task is complete when audio converts exactly at every boundary and real-time scheduling stays intact under the hidden tests.
Engineering & Robotics
Models are starting to operate in the physical world, where sensors drift, loads shift, and faults arrive without warning. These tasks put the agent to work on physical systems over hours: fusing GNSS, IMU and wheel data on a microcontroller through tunnel outages, racing unseen circuits from screenshots alone, and carrying CAD assemblies between file formats without losing a part.
Example task · 12 hours · C
Task overview. Cars lose GNSS in tunnels and city canyons, and the telematics chip still has to know where it is. The task is to write the navigation engine in C for a 64 MHz Cortex-M4F. It fuses consumer GNSS, a MEMS IMU with a loosely known mount and wheel ticks with a stale scale factor, using float32 math and no heap. It has to hold position and heading through GNSS outages and multipath on real road drives, within per-call deadlines and tight CPU, stack, RAM and flash budgets.
How it's graded. The source is built with fixed flags, linked into the firmware harness and run on an emulated Cortex-M4F with a cost meter, on 39 held-out real drives. It passes only if every drive stays within every budget, at least 36 of the 39 drives are within the accuracy tolerance for each metric, and no more than one metric exceeds twice its tolerance.
- Toolchain
- arm-none-eabi-gcc 12.2 (newlib), QEMU 10.1 (mps2-an386) with metering plugin, Python 3.12 with NumPy
- Compute
- 4 CPUs · 8 GB RAM
- Time limit
- 12 hours
Other example tasks in Engineering & Robotics
- CAD Assembly Exchange
- CadQuery is a Python library for building CAD models in code. Rebuild its assembly import and export so nested instances, placements, names and colors survive an exact round trip through STEP and its native formats. The task is complete when assemblies round-trip intact through OpenCascade's document model on the hidden cases.
- Vision-Only Racing Driver
- A racing game exposes only screenshots. Write a driving controller that races unseen 3D circuits from pixels alone. The task is complete when every one of 24 races on freshly generated tracks finishes no slower than an oracle driver that sees the full game state.
- ns-3 6LoWPAN Neighbor Discovery
- ns-3 is the standard open-source network simulator. Implement IPv6 neighbour discovery for 6LoWPAN (the protocol that puts IPv6 on tiny wireless sensors) in ns-3, including address registration, duplicate detection and the compressed wire formats. The task is complete when simulated sensor networks discover neighbours correctly on the hidden test topologies.