A practical quality checklist for video datasets

Quality is a series of decisions

A video can decode correctly and still be useless for the intended task. An accurate action description can accompany a recording with unsuitable permissions. A varied-looking catalogue can consist of repeated sessions from one environment. Evaluate these dimensions separately so a passing technical check does not hide a failure elsewhere.

Use the checklist as an acceptance plan agreed before collection or purchase. Thresholds should follow the experiment: a small component may need close detail, while a navigation task may prioritize continuous scene coverage. There is no single resolution or blur score that makes every video suitable for AI.

Gate 1: verify the files you actually received

Compare delivered files with the manifest. Check byte sizes and cryptographic hashes, then probe duration, dimensions, codec and frame-rate information from the media itself. A filename containing 4K or a contributor’s declared FPS is not a measurement. Attempt decoding across the full recording rather than checking only the first frame.

Record technical failures in a machine-readable report. A missing file, truncated stream and unsupported codec require different remedies. Preserve the original file when creating a training derivative, and document changes such as resizing or removing audio. Matching hashes establish byte identity; they do not prove that two different encodings contain different scenes.

• Every manifest entry resolves to the intended file.
• Measured duration and dimensions meet the agreed brief.
• The recording decodes without an unexplained missing tail.
• Original and derived files have distinct identities and documented roles.

Gate 2: check that the evidence is visible

Review whether the camera shows the task, not merely the room containing it. For a fastening operation, the hands may be visible while the fastener remains hidden. For sorting, the camera may show an item being picked up but miss the destination. Evaluate the complete sequence and the details needed to distinguish a correct action from a similar one.

Inspect difficult moments deliberately: rapid movement, hand-object contact, dark surfaces, reflective tools and transitions between steps. A recording can have acceptable average brightness while losing detail at the exact moment that matters. Log the interval and reason for each failure so the collector can correct framing or lighting instead of receiving an unexplained rejection.

Gate 3: audit time and annotation semantics

Check that labels refer to the correct time base and remain within the measured recording duration. Inspect segment joins, clipped final actions and repeated offsets. For multiple sensors, ask how synchronization was established. A collection with a video stream and a state table needs more than matching row counts to demonstrate useful alignment.

RLDS makes episode boundaries and step semantics explicit, including the distinction between an episode ending and a terminal state. That is a useful reminder to document what a sequence boundary means. For human video, a recording may end because the task finished, the camera stopped or an upload was cut short. Those are different training examples.

Inspect labels as hypotheses about the recording

Sample labels across task types, durations and difficult visual conditions. Check whether the named object is visible, whether the action happened and whether the outcome can be observed. Include recordings that passed automated checks; reviewing only failures cannot reveal the mistakes that the acceptance process missed.

Keep processing status separate from quality judgment. NVIDIA’s Cosmos Curator documentation, for example, distinguishes caption outcomes from quality flags and explains that flags do not themselves enforce filtering policy. A successful model response means a response was produced. Your acceptance process must still decide whether its content is suitable for the intended use.

Gate 4: measure dataset coverage and leakage

Build a coverage table for the variation you expect at deployment. Depending on the task, useful dimensions include participants, sessions, rooms, tools, object materials, camera viewpoints and successful or unsuccessful outcomes. Compare the delivered distribution with the brief, and identify important combinations that are missing rather than reporting only totals.

Keep related material together when defining evaluation splits. Clips from one recording can be near duplicates even when their file hashes differ. Exact-byte duplicate detection is useful but does not establish perceptual uniqueness. Audit train-evaluation overlap using the available session and source identifiers, then document the forms of similarity that remain unchecked.

Document permissions and uncertainty

Ask for a record of collection context, intended usage rights and the permissions relevant to the scene. Keep contributor ownership declarations separate from participant consent and workplace authorization. An image model cannot establish these relationships by recognizing an object or flagging a face. A scene with no detected person may still contain confidential information.

The Datasheets for Datasets approach encourages documentation of composition, collection and intended use. Apply it to limitations as well: missing demographic information, unverified context, unknown viewpoint or absent specialist annotations should remain visible. Missing information is a reason to qualify a claim or restrict a use, not an invitation to fill the field with a plausible guess.

Freeze the accepted selection and its evidence

Once a selection passes the agreed checks, record exactly what was accepted. Datai’s delivery design freezes recording identifiers, file references, measured durations, metadata, active action rows and analysis-package references. JSONL and CSV indexes support inspection; the per-recording JSON preserves fuller descriptions, flags and processing provenance.

Retain the acceptance report with that selection. If annotations are regenerated later, treat the new results as another version rather than silently changing an experiment’s input. A machine acceptance decision should remain identified as such. Human review, when performed, should have its own record instead of being inferred from the word approved.

Use failures to improve the next batch

Separate problems that require a new recording from those that can be corrected in metadata or annotation. Missing task visibility usually needs recapture. A vocabulary mismatch may need a revised labeling rule. A broken export reference may need packaging repair. Grouping failures by remedy prevents repeated work and reveals where the collection protocol needs to change.

Report both the result and the scope of the checks. A collection can pass your current checklist while remaining unsuitable for a different model or task. Keep that boundary explicit, and revisit the plan when the learning objective, capture setup or annotation method changes.