Video dataset formats: making MP4, JSONL and CSV work together
Build a usable video delivery with clear timestamps, structured annotations, provenance and honest checksum verification.
Data standards · 4 min read ·
In this guide
The delivery is more than a folder of videos
A video dataset needs two connected parts: media that a training pipeline can decode, and records that explain what each file contains and how it may be used. A filename such as kitchen-final-2.mp4 cannot carry reliable information about the task, contributor, annotation version and license. Those facts belong in a manifest with stable identifiers.
Agree on this delivery contract before buying a large collection. Decide which fields are required, which values may be missing, how timestamps are measured and what constitutes a new version. Test a small export through the buyer’s actual loader. A clean import on one developer’s laptop does not establish compatibility with every training environment.
Keep the container separate from the codec
MP4 and WebM describe containers. A container holds encoded video, and sometimes audio or other tracks; its extension does not fully describe the encoding inside. Mozilla’s media documentation makes this distinction explicit. Record the codec when it has been inspected, rather than inferring it from the filename.
Preserve the original uploaded media as the reference asset. If a buyer needs resized frames, another codec or a lower frame rate, make a derived version and record the transformation. Cropping may remove a hand, frame sampling may miss contact, and transcoding may change timestamps. A useful derivative needs a traceable relationship to its source, including the settings that affect interpretation.
Use JSONL for the complete recording record
JSON Lines stores one valid JSON value on each line, using UTF-8. This lets a loader process records sequentially without reading a whole collection into memory. For video data, an object per recording is a practical convention because nested metadata and multiple temporal annotations remain together.
The compact example below uses normalized export field names. Delivery URLs and optional analysis outputs are omitted for readability. It describes one twelve-second recording, with one action between 1.5 and 9.5 seconds. A real JSONL file keeps this object on one physical line and ends it with a newline. These example values explain a schema; they are not an inventory claim.
{"schema_version":"datai.video.v1","order_id":"order-001","video_id":"video-001","contributor_id":"contributor-001","split_group":"contributor:contributor-001","filename":"potting-001.mp4","duration_ms":12000,"content_type":"video/mp4","width":1920,"height":1080,"fps":30,"sha256":null,"metadata":{"task":"Fill a plant pot","viewpoint":"fixed","duration_source":"contributor_browser","duration_verified":false,"annotations_source":"contributor","sha256_verified":false},"annotations":[{"start_ms":1500,"end_ms":9500,"verb":"fill","object":"plant pot","goal":"Add potting soil","success":"success","note":"Soil is visible inside the pot at the end."}],"license":"non-exclusive-ai-training-v1"}
Use CSV for inspection without flattening away meaning
CSV is convenient for reviewing filenames, durations and category totals in a spreadsheet. RFC 4180 documents a common convention for headers, consistent field counts and quoting. A field containing a comma, quote or newline needs appropriate escaping; replacing commas inside descriptions would silently change the data.
Nested annotations need an explicit policy. One option is to serialize the annotation array as JSON inside a quoted CSV cell, matching the JSONL record. Another is a separate annotation table keyed by video ID. State which convention applies. Avoid repeating a recording’s full duration on every action row and then summing those rows as if they were separate footage.
| Information | Recommended representation |
|---|---|
| Recording identity and duration | One manifest row per recording |
| Several timed actions | A JSON array or a separate keyed table |
| Missing or unverified measurements | Explicit null values and provenance fields |
| Buyer-friendly inspection | CSV with documented escaping and column meanings |
Specify units and check annotation boundaries
Integer milliseconds make action boundaries unambiguous across an API, database and export. Each segment should satisfy zero less than or equal to start, start less than end, and end less than or equal to the recording duration. A label with an end time beyond the source file cannot be fixed by a valid JSON parser.
Write down what each semantic field means. A verb names the visible action, an object names its target, a goal describes the intended result, and success records the observed outcome. Unknown outcomes should remain uncertain. For frame-based work, preserve presentation timestamps or a documented time mapping; multiplying seconds by a nominal frame rate can be inaccurate for variable-frame-rate recordings.
Keep source claims and derived labels distinguishable
A contributor’s description is useful context, but it is not automatically an independently verified measurement. Keep duration source, annotation source and review status visible. If a model later produces captions or object tracks, store those as derived outputs with the model, version, creation time and review state. Do not overwrite the contributor’s original account with a generated description.
An order should preserve the recording and annotation versions selected at purchase. Later corrections can produce a new revision while the existing order remains reproducible. Contributor grouping also matters: related recordings from the same person can leak familiar environments into both training and evaluation. A split group provides a starting point for keeping those recordings together.
Say exactly what a checksum verifies
A whole-file SHA-256 digest identifies a byte sequence only when it has been calculated and compared at the relevant boundary. A contributor-supplied digest is a claim until independently checked. Server-verified multipart digests establish integrity for their recorded parts; they should not be relabeled as a server-verified whole-file hash.
A storage ETag is also not universally interchangeable with SHA-256, especially for multipart objects. Keep its algorithm and origin distinct. After downloading, a buyer can verify the expected part sizes and hashes, calculate a local whole-file digest, and retain the result alongside the manifest. A checksum establishes byte integrity, not consent, correct annotations or scientific suitability.
A format conversion cannot create missing observations
Packaging human video for a training workflow does not create robot joint states, calibrated depth or control actions that were never recorded. Export the observations that exist, document missing signals and let the buyer assess suitability for their objective. The most useful delivery is one whose files, labels, permissions and limitations can all be inspected without guessing.