WHAT THE DATA ACTUALLY CONTAINS.

Robot policies that perform well under evaluation degrade when they meet real conditions. The causes are established in the literature and they are all properties of data: training distributions that cover only narrow conditions, demonstration corpora that contain successes but no corrections, and brittleness to variation that was never present at training time.

Each of those is a dataset property rather than a model property. Which makes it notable that the datasets training the field's leading models are documented by the teams that built them and independently examined by nobody.

We examined one.

DROID comprises approximately 76,000 successful demonstrations across 564 scenes and 52 buildings on three continents, collected by 50 operators over twelve months using identical hardware at thirteen institutions. It sits in the training mix of flagship models including MolmoAct 2 and pi05.

We selected it because of its significance, not its weakness. It is a serious contribution by researchers whose work we respect, and it is documented with enough candour to make an independent assessment possible at all.

Two sources were held against each other: what the makers state, via the project site and paper, and what the data contains, via all 100 runs of the widely circulated public sample.

Each run was recorded on the task text displayed, the observed action, the outcome, whether text and action agreed, camera condition, setting difficulty and lighting, and whether failure or recovery behaviour was visible.

Most runs have no stated objective.

In 58 of 100 runs, no task text is displayed. Without an objective there is no way to establish what a run was attempting, and therefore no way to establish whether it succeeded. Twenty-one runs could not be scored at all as a result. An arm lifts a towel and sets it down elsewhere. A container is lifted and returned to its original position. Completed task, or abandoned attempt? The data does not say.

Failures are present and unmarked.

No run carries an outcome field. On inspection, 14 of 100 are unambiguous failures: a glass dropped during transfer to a dishwasher, a cup knocked over during pen retrieval, failed grasps of a bottle, a remote control and a yoghurt pot, an unsuccessful keypress, one recording of zero usable duration.

Nothing distinguishes them from the successful runs beside them.

Recovery is nearly absent, and invisible where it exists.

Recovery appears in 2 of 100 runs, where the arm failed an initial grasp and re-attempted successfully. No field identifies these episodes. Recovery from a failed grasp is the behaviour a deployed robot most needs, because deployment guarantees failure, and it is both rare here and unlabelled where present.

There is almost no difficulty in the data.

Ninety-one of 100 runs are brightly and evenly lit. Objects are predominantly small graspable household and office items, and the dominant structure is a single object moved between two points on a horizontal surface in a kitchen or office room.

Clutter, occluded targets, reflective and damaged objects, degraded lighting, human presence and interruption are effectively unrepresented.

Three things, recorded because they are true.

The scene coverage is a real advance: 564 scenes across 52 buildings, against ten to twenty-four for the datasets that preceded it.

The protocol discipline is unusual: identical arm, cameras and rig at all thirteen institutions, producing a corpus that is internally comparable.

And the makers documented their method honestly. The collection protocol, the operator count, the success-only design and the subsequent corrections are all stated openly. This assessment was possible because they wrote down what they did.

DROID measures short, single-object manipulation in tidy, well-lit rooms, and measures it competently. It does not measure readiness for deployment, because it contains few of the conditions that cause deployment to fail.

This is not specific to DROID. Collection protocols across the field optimise for volume, cleanliness and successful completion, because those properties are tractable to collect and straightforward to count. Difficulty is expensive to produce. Failure is unwelcome. Correction is rarely labelled.

The result is a measurement regime that certifies performance under favourable conditions, and a field that expresses surprise when models leave the room and stop working.

Episode count has stopped being a useful measure of anything. What matters is which conditions a corpus contains, whether outcomes are recorded, and whether failure and correction are present at all.

The full audit, including the 57-question framework and the complete 100-run log, is published in The Easy World. The specification we think the evidence points to is set out in the Physicore Standard v0.1, and the argument for testing in operating environments rather than on a bench is in A bench is not a building.