A buyer’s guide to human video datasets
Define the task, the evidence and the acceptance criteria before you buy footage.
Buying data · 5 min read ·
In this guide
Start with the decision your model needs to make
A video dataset is useful when its observations and labels support a specific learning objective. A thousand hours of cooking can still be a poor purchase for a model that must distinguish the instant a hand releases a container. Before comparing prices, describe the decision you want the model to make, the information available at inference time and the evaluation that would show an improvement.
This guide concerns recordings of people performing physical tasks. Human video can support activity understanding, retrieval, procedural reasoning and representation learning. Training a robot controller introduces additional requirements. Do not assume that a first-person view supplies joint positions, gripper commands or measured forces.
Write a brief with observable acceptance criteria
Replace broad categories with examples that another person can reproduce. Instead of asking for warehouse footage, specify picking an item from a shelf, checking it and placing it into a carton. Describe whether the beginning and end must be visible, whether the camera moves with the worker and which objects may leave the frame. Separate essential conditions from preferences so a supplier can explain the consequences of each constraint.
An initial brief should also name exclusions. Confidential screens, identifiable customers, incomplete tasks or staged repetitions may be unsuitable for your objective. The acceptance rule must distinguish a recording that fails the brief from a valid example of an unsuccessful task. Otherwise, a useful recovery sequence may be rejected simply because the participant made a mistake.
Evaluate a pilot before buying volume
Choose a small pilot that deliberately spans the difficult parts of the brief. For packing, include reflective wrapping, a partly occluded item, a crowded bench and different container sizes. Random selection alone can miss these conditions. Ask for the associated metadata and annotation package with each video; a polished montage cannot demonstrate whether the files work in your loader.
Run the pilot through the same preprocessing and evaluation code intended for the larger collection. Record failures by cause: decoding, missing context, vocabulary disagreement, annotation timing or task mismatch. Agree how each cause will be handled before scaling. This converts a vague judgment about quality into a list of changes that can be tested in the next batch.
Inspect what each annotation actually represents
A caption, an action interval and a contact frame answer different questions. A caption may describe a whole task without locating its steps. A temporal action label may identify lifting a box while leaving the exact grasp transition uncertain. A robot trajectory adds another layer: observations aligned with measured state and control signals. DROID’s published schema is a useful example of that distinction because it exposes robot observations and actions explicitly.
Ask which fields were declared by a contributor, measured by software, inferred by a model or checked by a person. Request uncertainty flags and the versions of the schema and processing policy. A field called confidence is not a demonstrated accuracy measure unless its relationship to observed errors has been evaluated.
Buy coverage that matches deployment
Count more than clips and hours. A collection may contain many recordings from the same room, camera and participant. Repetition can help study a controlled task, but it should not be mistaken for environmental diversity. Build a coverage table around the variation your model will encounter: objects, layouts, lighting, viewpoint, task order and participant technique.
For evaluation, keep related recordings together when assigning splits. Different clips cut from one session can share backgrounds, objects and behavior. Splitting those clips randomly between training and evaluation may overstate generalization. Where identifiers permit, reserve participants, sessions or locations rather than individual files. State which dimensions remain shared, and interpret results within those limits.
Treat provenance as part of the product
Request a documented collection process and a clear account of the permissions supporting the proposed use. Contributor declarations, participant permissions and workplace authorization are separate considerations. An automated description of a scene cannot establish any of them. Ask how exclusions, withdrawals and changes to a recording are reflected in the delivery record.
The Datasheets for Datasets research offers a useful documentation approach: describe why a dataset exists, how it was assembled and where its use has limits. Apply that principle to procurement. A clear explanation of an unavailable annotation or missing sensor is more valuable than a broad promise that hides an assumption.
Check the package, not just the extension
A format name does not establish semantic compatibility. LeRobot’s documentation, for example, distinguishes video, tabular signals and the metadata connecting them. Renaming a human-video archive or converting its table layout cannot create absent robot actions. Specify the fields, units, identifiers and time reference your code requires, then validate a complete example.
The datai package design keeps original video, contributor context and generated analysis distinguishable. Analysis JSON carries descriptions, timed actions and processing provenance; JSONL and CSV provide an index of the selected recordings. A frozen manifest records the files and annotation versions associated with a delivery. Specialized spatial labels, calibrated measurements and robot telemetry require a separate scope.
Make the purchase decision reproducible
Keep a short decision record: the objective, pilot composition, acceptance checks, known limitations and delivery version. Ask the people responsible for modeling, data loading and permissions to review the same evidence. A technically readable collection can still have the wrong task distribution, and a relevant collection can still need substantial preparation.
Compare the total work remaining after receipt. Consider inspection, conversion, relabeling, storage and the cost of repeating a failed collection. The strongest proposal is the one that makes those tradeoffs explicit and gives you a practical way to test them before committing to scale.