Beyond pass and fail.

A robot either completed the task or it did not. That is where most evaluation stops, and it is nowhere near enough. Two policies can post the same success rate and be entirely different systems, one grasping cleanly and efficiently, the other succeeding through hesitant, inefficient motion that will fail the moment conditions tighten. A pass-or-fail score cannot tell them apart, and the difference between them is exactly what determines which one survives deployment.

Real evaluation begins where the binary ends. It asks how a grasp was achieved, not merely whether it held: the efficiency of the motion, the accuracy and stability of the contact, the smoothness of the approach, the margin by which the policy avoided failure. It asks where failures cluster and why, treating them as a structured signal to be characterised by root cause rather than a count to be totalled. This is the difference between knowing a model works and knowing how it works, and only the second tells a team what to fix.

The reason this matters so much in physical AI is that failures are not evenly distributed. A policy at ninety-five percent does not fail randomly across all conditions; it fails heavily on the narrow set of cases that diverge from its training. An evaluation that reports only the aggregate hides exactly the information that would explain the gap, while an evaluation that clusters failures by cause reveals it, pointing directly at the data the model is missing. Measurement, done well, is not a verdict on a model. It is a map of what to collect next.

This is also why evaluation cannot be a generic service bolted on at the end. Judging motion quality, grasp accuracy, and failure modes requires trained assessors making fine-grained, consistent judgements against criteria defined for a specific model and task, not crowd labels applied to a rubric that fits everything and explains nothing. It is skilled work, and it is the part of the pipeline most often underbuilt, because it is harder to staff and harder to scale than collection, and its value is invisible until a model is failing in production and no one can say precisely why.

This closes the loop with a lab rather than ending the engagement. Evaluation is defined against how a particular model is meant to perform, with assessment criteria set to the dimensions that matter for its tasks and embodiment, and failures characterised in enough detail to be actionable. What that analysis surfaces, the conditions where the policy is weak, the contact regimes it mishandles, the environments it has not truly learned, feeds directly back into the next round of capture. Measurement and collection become one cycle: evaluate to find the gap, capture to close it, evaluate again.

Beyond pass and fail is where physical AI is actually understood, and a field that measures only whether a robot succeeded has chosen not to know why. Building evaluation as a discipline in its own right, one that explains a model rather than merely scoring it, is what turns a data programme from a supply of hours into a system that tells a lab where it is and where to go next.