CAD Assembly Exchange
Reconstruct metadata-preserving assembly import/export (XDE) in CadQuery.
Results
Each square represents one run. Score is the percentage of runs that pass: 0, 25, 50, 75, or 100 for four runs.
| # | Model | Rollouts | Pass@1 | Score | Time |
|---|---|---|---|---|---|
| 1 | Claude Sonnet 5.5 max | 4/4 | 3.0h | ||
| 2 | Claude Opus 5.5 max | 3/4 | 2.5h | ||
| 3 | GPT-6 Astra xhigh | 2/4 | 1.9h | ||
| 4 | GPT-6.1 Sol max | 2/4 | 2.7h | ||
| 5 | GPT-5.6 Sol max | 0/4 | 1.0h | ||
| 6 | Gemini 3.8 Flash high | 0/4 | 0.6h | ||
| 7 | Qwen3.8 Max xhigh | 0/4 | 2.5h | ||
| 8 | GLM 5.3 Flash max | 0/4 | 3.6h | ||
| 9 | Muse Spark 1.3 max | 0/4 | 2.4h | ||
| 10 | Kimi K3 max | 0/4 | 1.6h | ||
| 11 | Grok 4.7 xhigh | 0/4 | 1.0h |
Background
CadQuery is an open-source Python library for parametric CAD built on the Open CASCADE Technology (OCCT) kernel. At the starting commit it cannot read one back: Assembly.load raises NotImplementedError. The agent adds assembly import, matching upstream CadQuery #1779. Reading an assembly is much harder than writing one. XDE separates products from instances and stores names and colors at both levels, so a faithful reader must rebuild CadQuery's tree without flattening or splitting shared parts. Files from other CAD tools lay the same data out differently, adding further cases. Repeated export and load through STEP, XmlXCAF and BinXCAF must still converge.
Task
The agent adds Assembly.load and Assembly.importStep for STEP, XML and XBF, plus XBF export. A loaded assembly reproduces what was written: tree and child order, each node's name, relative placement, geometry and color, and per-node subshape names, colors and layers. Shared parts stay shared, and every component of a foreign file becomes a node. Tolerances are stated: placement entries within 1e-6, volume, area and length within relative 1e-5, and color channels within 1e-6, with XML allowed OCCT's nearest named color. Missing or malformed input raises ValueError without crashing the interpreter, and existing public behavior keeps working. Only the committed HEAD is graded.
Workspace
The offline sandbox (4 CPUs, 8 GB, 4 hours) holds a CadQuery checkout at the starting commit, installed in editable mode with development dependencies, OCP 7.8.1 (Python bindings to OCCT), Python 3.12 and pytest. Graded checks, fixtures and files imitating other CAD tools are held out.
Verification
The branch is rebuilt in a separate environment. Fixtures and foreign files are produced from a trusted starter copy and from raw OCCT before any submitted code runs; an independent XDE reader inspects every exported file. The 67 feature checks cover round trips in each format, per-instance differences, shared geometry, nested assemblies, RGBA colors, non-ASCII names, metadata, foreign files and malformed input. Twelve existing-behavior checks (STEP and exportStepMeta output, glTF colors, building and installing a wheel) and the starter's test suite form a gate. The score is the fraction of feature checks passed; 0 if any gate check fails. A pass requires every check.
Environment
- Toolchain
- Python 3.12, CadQuery, OCP/OCCT, git
- Compute
- 4 CPUs · 8 GB RAM
- Network
- Offline
- Time limit
- 4 hours
References
- CadQuery (Apache-2.0): The library the agent extends, checked out at a commit before the reconstructed change.
- CadQuery #1779: The upstream change that added assembly import; its tests, adapted to public behavior, are among the graded checks.
- Open CASCADE Technology (OCCT): The CAD kernel whose XDE document model, STEP translator and XmlXCAF and BinXCAF formats define the exchanged data.