Measure What Was Built: Repairing Generated 3D Buildings from Their Own Data
Research in progress (October 2026) · paper and code release pending
Abstract
A language model builds a Minecraft house in one go, then gets one round to repair it. We ask what it needs to be given to fix the right things. Given the full 3D data, pictures of the house, or nothing extra, it finds few problems on its own: on 10 houses, 81 to 93 of 102 floating or wrongly attached blocks remain. Told what was measured in the 3D data, it fixes most of them, leaving 26. Pictures cover a different layer, the furniture and detail that the measurements do not describe. And a reviewer model that reads the builder's own reasons flags fewer problems than one that does not, so the measurements, not a reviewer model, decide what counts as fixed.
How it works
The point is the second box. The model is given measured facts, not instructions, and the facts come from the 3D block array the model generated. The official server is one step in that: it sets how fences, panes and stairs connect, so nothing about the game's rules is hand-written on our side. The same language model (Claude Opus 5.5) writes the brief, builds the house and makes the repairs.
The 7 kinds of fact
Click a kind to outline it on BG37 (W is drawn on the floor below the empty cells).
| Kind | What is measured |
|---|---|
| A — floating or mis-attached | A block that floats or is not held by what it should rest on. |
| O — facing the wrong way | Stairs and other blocks facing differently from their neighbours. |
| B — places nobody can walk to | A spot a player cannot walk to from the edge of the plot. |
| W — open floor nobody can reach | Open floor nobody can reach from outside. |
| C — not enough head-room | A spot with less than 1.8 blocks of head-room. |
| K — fences and panes that do not join | Fences or glass panes that the game does not join as the builder intended. |
| V — noticed in a picture | Something the model notices in a picture; never counted as a defect. |
Results
The first batch of 10 houses starts with 102 blocks that float in the air or are attached wrong. Each number is how many are still there after the model's repair. The model got either pictures of the house or its 3D data, and either had to find the problems itself or was told what was found. Most problems were fixed only when they were measured in the 3D data and told to the model.
(With nothing extra, 88 and 81 were left in two runs. Told only how many there were, 72 and 74.)
| Model looks for problems itself | Model is told what was found | |
|---|---|---|
| Pictures | 93 | 95 |
| 3D data | 86 | 26 |
Before and after, inside the house
Same camera before and after; cyan marks the blocks that changed. The last pair is from the look round, where the model sees pictures and no facts.
Before
After
BG37 — Loose pot cleared off a chest lid.
Before
After
BG42 — Jukebox moved out of the parlour doorway.
Before
After
BG44 — Post added where the pool glazing ended in mid-air.
Before
After
BG40 — Table tops made solid.
Before
After
BG54 — Table set moved off the front of the stove.
Before
After
BG47 — Staircase turned the right way.
Before
After
BG47 — Posts between the front door and the glass.
Before
After
BG41 — Display case given a lintel.
Before the look round
After the look round
BG55 — Corridor that ended in bookshelves opened up.
Findings
- Models fix what is measured in the 3D data and told to them, not what they could find on their own. In a first batch of 10 houses we tried every setting. Given the full block list, the pictures, or nothing extra, floating or mis-attached blocks go from 102 to between 81 and 93; told only how many there are, to 72–74; told the measured facts, to 26. A second batch of 10 houses, repaired with the fact list only, shows the same: across both batches it takes them from 279 to 46.
- Who tells them matters: a reviewer model is swayed by the builder's reasons. Over 1,573 groups of edits, a reviewer model (Claude Opus 5.5) flagged 68 when it could read the builder's reasons and 104 when they were hidden: real but modest, about 1.5 times. It is why our machine checks, not a reviewer model, decide what counts as fixed.
- Pictures are a separate layer, for what measurement cannot describe. Asked directly, a vision model pointed to 7 of the 102 measured defects. Run after the facts repair, the look round changed 97 and 46 cells in the two batches, mostly furnishing, and broke nothing new.
Limitations
- One builder model and one game version (26.2); 10 houses per setting, one run each (two for the no-extra and count settings).
- Pictures only was tried on 10 houses; looking before the facts repair was not tried.
- Whether another model reading the raw block list could write the facts itself was not tested.