Robot Dataset Labels: Direct Observations, Proxies and Reviewer Disagreement

A robotics dataset may include images, sensor recordings or tables with labels such as “person,” “obstacle” or “grasp succeeded.” Before treating those labels as unquestionable truth, find out who or what assigned them, what the label was intended to represent and how disagreements were handled. A well-documented label is useful evidence, but it is not automatically a perfect measurement of a physical event.

Define the target before inspecting labels

Write down the question the dataset is intended to support. A label for an object visible in a camera frame is different from a claim that the object is reachable by a robot or that contact would be safe. Similarly, a reported successful grasp may depend on a particular object, test procedure and success criterion. The label's meaning must be tied to its source instructions and measurement process.

Google's machine-learning guidance on labels distinguishes direct labels, proxy labels and human-generated labels, and emphasizes checking data quality. Its guidance is general; a robotics project still needs to define its own task, collection conditions and acceptable interpretation.

Separate direct observations from proxies

A direct observation may be closer to the desired outcome than a proxy, but it still depends on how it was recorded. A proxy is a substitute that may be easier to collect while capturing only part of the intended concept. For example, a hypothetical dataset might mark a grasp as successful when a gripper closes; that event alone does not establish that an object remained securely held through the rest of the task. The dataset documentation should state what was actually measured.

If a label was generated automatically from a sensor threshold or system log, record the rule and version. If people assigned labels, retain the instructions, available context and review process. Do not silently turn an uncertain or missing observation into a negative label; the documentation should explain how unknown cases are represented.

Review disagreement rather than hiding it

Two reviewers can reasonably interpret the same image differently, especially when an object is partially hidden or a task outcome is ambiguous. Google's discussion of categorical data issues notes that human labels can contain errors and that agreement between raters can be assessed. A high agreement figure alone does not prove the definition is appropriate, and the method and sample should accompany the result.

For a practical review, select a documented sample, have more than one qualified reviewer apply the same instructions independently and examine disagreements. Record the number and selection of examples, the original labels, the reviewers' decisions and how any final decision was reached. If the sample is small or selected for convenience, state that limitation rather than claiming the entire dataset has been validated.

Keep label changes connected to dataset versions

A corrected annotation can change a model evaluation even when the underlying image is unchanged. Preserve the dataset release, label schema, annotation revision and date used for each experiment. If a label definition changes, document whether older labels were re-reviewed or migrated and how. A file name or a download date alone may not be sufficient to reconstruct which labels were used.

When a dataset contains video or repeated views of the same scene, consider whether the evaluation design unintentionally places closely related examples in both training and test sets. The appropriate split depends on the intended use. Record the grouping and avoid claiming general performance from a test that does not represent the deployment environment.

Access and privacy rules also affect how labels can be reviewed. Images may contain people, workplace details or confidential products, so the dataset owner should define who may view the examples and how review results are stored. A reviewer can report ambiguity using an approved identifier without copying sensitive images into an uncontrolled document. If the data cannot be shared for independent review, describe that limitation and the safeguards used by the authorized team rather than implying that every user can inspect the source material.

Report what the label supports

Before publishing a result, describe the label source, intended target, known ambiguity, review method and dataset version. If the label is a proxy, say what it measures and what it does not. Do not infer a robot's safety or real-world reliability from a dataset score without the additional evidence and validation required for the application.