17Engineering & Robotics11 of 44 runs pass

CAD Assembly Exchange

Reconstruct metadata-preserving assembly import/export (XDE) in CadQuery.

Results

Each square represents one run. Score is the percentage of runs that pass: 0, 25, 50, 75, or 100 for four runs.

#ModelRolloutsPass@1ScoreTime
1Claude Sonnet 5.5 max4/41003.0h
2Claude Opus 5.5 max3/475.02.5h
3GPT-6 Astra xhigh2/450.01.9h
4GPT-6.1 Sol max2/450.02.7h
5GPT-5.6 Sol max0/401.0h
6Gemini 3.8 Flash high0/400.6h
7Qwen3.8 Max xhigh0/402.5h
8GLM 5.3 Flash max0/403.6h
9Muse Spark 1.3 max0/402.4h
10Kimi K3 max0/401.6h
11Grok 4.7 xhigh0/401.0h

Background

CadQuery is an open-source Python library for parametric CAD built on the Open CASCADE Technology (OCCT) kernel. At the starting commit it cannot read one back: Assembly.load raises NotImplementedError. The agent adds assembly import, matching upstream CadQuery #1779. Reading an assembly is much harder than writing one. XDE separates products from instances and stores names and colors at both levels, so a faithful reader must rebuild CadQuery's tree without flattening or splitting shared parts. Files from other CAD tools lay the same data out differently, adding further cases. Repeated export and load through STEP, XmlXCAF and BinXCAF must still converge.

Task

The agent adds Assembly.load and Assembly.importStep for STEP, XML and XBF, plus XBF export. A loaded assembly reproduces what was written: tree and child order, each node's name, relative placement, geometry and color, and per-node subshape names, colors and layers. Shared parts stay shared, and every component of a foreign file becomes a node. Tolerances are stated: placement entries within 1e-6, volume, area and length within relative 1e-5, and color channels within 1e-6, with XML allowed OCCT's nearest named color. Missing or malformed input raises ValueError without crashing the interpreter, and existing public behavior keeps working. Only the committed HEAD is graded.

Workspace

The offline sandbox (4 CPUs, 8 GB, 4 hours) holds a CadQuery checkout at the starting commit, installed in editable mode with development dependencies, OCP 7.8.1 (Python bindings to OCCT), Python 3.12 and pytest. Graded checks, fixtures and files imitating other CAD tools are held out.

Verification

The branch is rebuilt in a separate environment. Fixtures and foreign files are produced from a trusted starter copy and from raw OCCT before any submitted code runs; an independent XDE reader inspects every exported file. The 67 feature checks cover round trips in each format, per-instance differences, shared geometry, nested assemblies, RGBA colors, non-ASCII names, metadata, foreign files and malformed input. Twelve existing-behavior checks (STEP and exportStepMeta output, glTF colors, building and installing a wheel) and the starter's test suite form a gate. The score is the fraction of feature checks passed; 0 if any gate check fails. A pass requires every check.

Environment

Toolchain
Python 3.12, CadQuery, OCP/OCCT, git
Compute
4 CPUs · 8 GB RAM
Network
Offline
Time limit
4 hours

References

  • CadQuery (Apache-2.0): The library the agent extends, checked out at a commit before the reconstructed change.
  • CadQuery #1779: The upstream change that added assembly import; its tests, adapted to public behavior, are among the graded checks.
  • Open CASCADE Technology (OCCT): The CAD kernel whose XDE document model, STEP translator and XmlXCAF and BinXCAF formats define the exchanged data.
All tasks