Temporal action annotations: making video sequences useful

A timestamp is only the beginning

Temporal annotations connect an action label to a period in a recording. They let a team retrieve examples, study procedures or train a model to localize behavior. Their usefulness depends on agreement about what an action means and where it begins and ends. A perfectly valid timestamp can still describe the wrong event.

Consider someone moving a jar from a shelf to a worktop. One project may label the whole transfer. Another may distinguish reaching, grasping, lifting, carrying and placing. Neither granularity is universally correct. The useful choice is the one that matches the model’s objective and can be applied consistently to the footage.

Define the annotation unit first

Write a small set of boundary rules before labeling a large collection. Does lifting begin when the hand touches the jar, when the jar moves or when it clears the shelf? Does placing end at surface contact or hand release? Include examples where the answer is unclear, such as an object hidden behind a hand.

Research datasets distinguish these choices explicitly. Ego4D documents narration, temporal events and critical interaction frames as separate annotation resources. Ego-Exo4D’s keystep format connects intervals with a task vocabulary and step descriptions. These are useful reference patterns, but a new project still needs its own definition of the event it is measuring.

Use a vocabulary with room for uncertainty

A compact verb-object vocabulary is easier to audit than unrestricted prose alone. For a packing task, verbs might include pick, place, fold and seal. Preserve a descriptive note when the controlled label loses important context. Avoid silently merging different concepts, such as carrying a closed carton and sealing an open one, because both contain the word box.

Keep the intended goal separate from the observed outcome. A participant may try to insert a component without completing the insertion. If the final state is hidden, the outcome should remain uncertain. A model-generated sentence that sounds plausible is not sufficient evidence that the task succeeded.

Choose one time reference and preserve it

State whether timestamps refer to the original recording or a derived clip, and whether the unit is seconds, milliseconds or frame indices. If a segment begins 60 seconds into the original, a local action at 2.4 seconds maps to 62.4 seconds. Store that relationship rather than guessing it again during export.

Millisecond fields are a representation choice, not a claim of millisecond accuracy. A model inspecting sampled frames may miss a short action between samples or estimate its boundary loosely. Variable-frame-rate media also makes a simple frame-number divided by nominal FPS conversion unreliable unless the frame timestamps support that interpretation.

Make provenance visible in the package

An action record should identify its source recording and the process that produced it. Preserve the schema version, processing configuration, model identity where relevant and whether a person reviewed the result. Contributor descriptions, model observations and later corrections should not overwrite each other without a version trail.

Datai’s analysis design uses silent sampled derivatives and maps segment results back to the original timeline. It retains descriptions, objects, tools, actions and quality or privacy flags in analysis JSON. The accompanying provenance matters because full timeline coverage does not mean every source frame was inspected. Specialized contact frames, spatial tracks and calibrated motion require a separate annotation scope.

Read a record without overinterpreting it

The annotation record below shows the core fields used to describe one temporal action. The interval identifies a proposed visible action. The uncertain outcome and note preserve what cannot be concluded from the scene.

Do not convert that interval into a motor command. The label does not specify a robot pose, force, trajectory or collision-free path. It also says nothing about who owns the footage or whether workplace permission was obtained. Those questions belong to separate measurement and provenance records.

{
  "start_ms": 62400,
  "end_ms": 66100,
  "verb": "place",
  "object": "glass jar",
  "goal": "move the jar to the worktop",
  "success": "uncertain",
  "note": "The hand obscures the final release."
}

Validate structure and meaning independently

First run structural checks: required fields, allowed values, nonnegative times, start before end and intervals within the recording’s duration. Check segment offsets and gaps in processing coverage. Decide whether overlapping actions are allowed; two hands can perform different actions at once, so overlap is not automatically an error.

Then evaluate meaning on a representative sample. Look for omitted actions, wrong objects, invented outcomes and inconsistent boundary placement. Compare disagreements against the written rules. If a model and a reviewer disagree because the instruction is ambiguous, clarify the instruction before treating either label as authoritative. A JSON schema can validate structure without establishing semantic correctness.

Export enough context to reproduce the label

The delivery index should make each annotation traceable to the correct original and analysis version. Datai’s design uses per-recording analysis JSON plus JSONL and CSV indexes in a frozen selection manifest. Keep source identity, durations and annotation references together when selecting a subset; otherwise a later processing run can change the labels used by an experiment.

Finally, agree on the evaluation tolerance and intended use. Coarse intervals may work for retrieval while being inadequate for contact prediction. If the task changes, revisit granularity, sampling and validation. More decimal places or a longer description cannot repair an annotation protocol that measures the wrong event.