sBot-Datasets: evaluation details

The paired gains can differ from subtracting the rounded chart values: each contrast uses the trajectories scoreable in both configurations. The intervals resample task groups.

• Native video was stronger than contact sheets in these runs. At the matched 300-frame budget, single-camera video scored 0.513 versus 0.410 for contact sheets on L5VEL, and 0.519 versus 0.327 on RH20T. The paired gains were 0.103 (95% CI 0.042–0.183) and 0.192 (0.160–0.224), and both survived the Holm correction. On RH20T, 4.5% of video outputs failed or were unscoreable, compared with 28.2% for the 300-frame sheets. The shipped 60-frame sheet result is shown separately in Figure E1 because its input budget is smaller.
• Calibration improved timing on both sets. The table above shows gains of 0.150 and 0.155 on the L5VEL task groups, and 0.042 and 0.047 on RH20T. Both RH20T effects survived the planned 24-test Holm correction. The larger L5VEL effects did not: with only four independent task groups, their uncertainty is much wider than for the 39-task RH20T analysis.
• The ten examples transfer less reliably on RH20T. Among calibration groups accepted by the fitter, reference boundary positions varied by about 7.6–8.0% of episode duration on RH20T, compared with 2.4% on L5VEL. The calibration-to-evaluation timing shift was about 4.4–4.5% on RH20T and 0.9% on L5VEL. An exact copy of a held-out trajectory’s ordered label sequence appeared among the ten examples for only 9.7% of RH20T trajectories, compared with 94.5% on L5VEL. This weaker coverage and less stable timing are consistent with the smaller RH20T gain; the study cannot separate task diversity, annotation policy and camera observability into individual causes.
• The seeded timing controls matched the best model on temporal IoU. One control places boundaries at the median relative positions in matching examples. The other takes their median relative segment durations, normalizes them to fill the episode and accumulates them. They scored 0.695/0.696 on L5VEL and 0.591/0.592 on RH20T, compared with 0.687 and 0.566 for calibrated two-view video. These are temporal-IoU point estimates, not a win on every metric: on RH20T the calibrated model’s three-second boundary hit rate was 0.473, slightly above the duration control’s 0.471. The registered comparison did not establish an overall difference after multiple-test correction.
• We did not establish a benefit from a second camera view. The controlled comparison used uncalibrated single- and two-view inputs. The two-view point estimates were sometimes higher, but the evidence was inconclusive.

A missing predicted span scores zero overlap and its boundary counts as a miss. Missing spans can result from omitted labels, invalid timestamps or an ordering check rejecting a colliding span. These failures stay in the averages; they do not mean a reference label was deleted from the source dataset.

A repeated-label ambiguity made 23 RH20T configuration/trajectory pairs unscoreable: six for each single-video configuration, four for each two-view configuration and three for 300-frame contact sheets. Each comparison uses trajectories scoreable in both configurations. L5VEL has 757 scoreable trajectories in every configuration; RH20T has 1,299–1,305. Scores average within each task, then weight tasks equally.

The three timing controls receive the correct label order and annotated episode start and end, and always return every label. One divides the episode equally. The two seeded controls learn relative boundary positions or segment durations from the same ten examples used by calibration. With fewer than three matching examples, they use equal durations instead; that fallback covers 62 L5VEL and 253 RH20T evaluation trajectories.

These controls test whether video adds information beyond the timing pattern in the examples. They cannot discover which actions happened in a new recording.

L5VEL uses only imported source references, excluding the generated collection components. Their original authorship and human verification are not established by the import records. RoboInter’s published workflow describes ChatGPT preannotation followed by human clip segmentation, cross-checking and sampling-based validation. It does not report annotator agreement or detailed acceptance statistics.

L5VEL and RH20T use the same evaluator. The repository retains corpus_a and corpus_b as internal keys for L5VEL and RH20T, respectively; those names are not separate methods. For each task, the preparer sorts SHA-256 hashes of 1729:task:source_id:local_episode and reserves the first ten distinct annotated trajectories. This choice is fixed before model inference and does not inspect label contents or prediction quality. The ten trajectories are put in a seed-only dataset; every other trajectory is put in evaluation-only components. A failed seed is recorded and never replaced.

Each camera configuration uses those same ten identities but makes its own predictions and fits its own parameters. Fits are grouped within task by subtask count, and require at least three complete predictions with the supplied labels in the right order. The study uses count scope, so it assumes the first, second, and later positions represent comparable task phases even when their wording differs. With predicted internal boundary p_ij, reference boundary g_ij, and stitched predicted extent T_i, the position correction is:

offset_j = median_i((p_ij - g_ij) / T_i)

The fitter also stores median reference segment durations divided by T_i. Leave-one-out scoring on the retained seeds selects a duration weight from {0, 0.25, 0.5, 1, 2, 4}. The fit is recommended only when this check gains at least 0.03 temporal IoU and the raw first-boundary mean absolute error is at least 1.5 s. Because the same folds select the weight and estimate its gain, this check is optimistic rather than an independent evaluation.

Accepted parameters are frozen. On a held-out prediction, a dynamic program on a 0.1-second grid shifts all internal boundaries together, balancing corrected model times against the duration pattern while keeping positions nondecreasing. The duration fractions are not renormalized by the calibrator, and a later strict-order cleanup can reject colliding labels. Calibration cannot restore a label omitted by the model. Unsupported subtask counts, rejected fits and failed calibration attempts keep the matching raw prediction. Single-camera calibration applied to 507 of 757 L5VEL trajectories and 659 of 1,305 RH20T trajectories; every fallback remains in the published averages. The public calibration protocol documents the identity maps, hashes and runtime skip reasons.

Median duration fractions are not renormalized: accepted priors sum to 0.972–0.999 on L5VEL and 0.858–1.117 on RH20T. The same three to ten usable seeds choose the duration weight and estimate its leave-one-out benefit; the bias gate checks only the first boundary. These choices can make the fit optimistic or unstable.

We count visual tokens as Σ product(grid_thw) / merge_size², using image_grid_thw for the sheets and video_grid_thw for video. Qwen’s official video processor configuration uses 16-pixel spatial patches, merges them 2×2, and groups video frames in temporal pairs. Contact sheets also pay for a complete 5×4 canvas when their last sheet is only partly filled. Together, these choices explain the reduction here; changing the sampling rate, resolution or sheet layout can change it. Qwen’s model card documents native image and video input and configurable processor budgets. It does not itself claim that video is always cheaper than contact sheets.

We checked the metadata-derived census against decoded pixels. Six examples at fixed short, median and long frame-count ranks had processor grids and visual placeholders that matched the census exactly; both representations also completed in 12 local Qwen3.8-27B requests. This validates the counting path and payload compatibility, not total cost or throughput. Hardware and run-manifest details are in the reproduction notes.

This is a visual-token result. It does not measure generated tokens, billing, latency or annotation accuracy, and native video adds timestamp text around its visual tokens. Figure E1 reports accuracy separately. We exclude server token telemetry because the available vLLM counters treated the two modalities inconsistently and therefore cannot support a cross-format comparison. The reproduction notes describe the scripts, validation and limits.

1. Install dependencies and pin the processor

uv sync --locked --extra eval --extra serve
export ALIGN_QWEN_REV=1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
export ALIGN_QWEN_PROCESSOR="$PWD/.cache/qwen38-processor-$ALIGN_QWEN_REV"
uv run --no-sync hf download Qwen/Qwen3.8-27B \
  config.json preprocessor_config.json video_preprocessor_config.json \
  tokenizer_config.json tokenizer.json chat_template.jinja \
  --revision "$ALIGN_QWEN_REV" --local-dir "$ALIGN_QWEN_PROCESSOR"

2. Count both formats

uv run --no-sync python evaluation/scripts/measure_frame_format_tokens.py \
  <prepared_corpus_a> <prepared_corpus_b> \
  --processor "$ALIGN_QWEN_PROCESSOR" --local-files-only \
  --rows-out rows.jsonl --summary-out summary.json

3. Check sampled frames against decoded media

uv run --no-sync python evaluation/scripts/validate_frame_format_tokens.py \
  --rows rows.jsonl \
  --corpus-root corpus_a=<prepared_corpus_a> \
  --corpus-root corpus_b=<prepared_corpus_b> \
  --processor "$ALIGN_QWEN_PROCESSOR" --local-files-only --pairs-per-corpus 3 \
  --out validation.json

4. Attach validation to the count

uv run --no-sync python evaluation/scripts/measure_frame_format_tokens.py \
  <prepared_corpus_a> <prepared_corpus_b> \
  --processor "$ALIGN_QWEN_PROCESSOR" --local-files-only \
  --rows-out rows.jsonl --summary-out summary.json \
  --actual-pairs validation.json

5. Generate the plot

uv run --no-sync python evaluation/scripts/plot_frame_format_tokens.py \
  --rows rows.jsonl --summary summary.json \
  --svg figure3.svg --png figure3.png

The census reproduces the 2 fps, 300-frame, 224-pixel evaluation geometry without decoding pixels, then verifies the population against the prepared split manifests. The validator selects fixed length ranks, decodes the real frames and requires its image_grid_thw, video_grid_thw and input placeholder counts to match. Add --api-base <endpoint> --model <served_id> --gpu-index <index> to repeat the live-server check. The server should use --media-io-kwargs '{"video": {"num_frames": -1}}' so it does not cap a clip at 32 frames. For a one-episode processor check, run uv run --no-sync python -m lerobot_align.diagnostics.probe_frame_tokens <dataset_root> --episode 0 --camera observation.images.wrist. None of these cross-format calculations uses usage.prompt_tokens. The published run used processor revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. For an exact rerun, use the explicit six-file download above and verify the config hashes recorded in summary.json.

Paired per-episode wall-clock

python -m lerobot_align.diagnostics.eval_align_batch \
    <dataset_root> --formats contact_sheet video \
    --video-fallback error --camera observation.images.wrist \
    --api-base <endpoint> --model <served_id> --out ab.json

Both formats run back to back within each episode. Contact sheets run first, and the shared frame provider caches decoded frames, so video can benefit from a warmer cache. Warm both formats or counterbalance their order before interpreting the timing difference.

Keep --video-fallback error to fail the run if video encoding fails; the default can fall back to contact sheets. The reported elapsed_s includes local decoding and H.264 encoding as well as the model call.

End-to-end job telemetry

python evaluation/scripts/run_arm.py \
    --dataset <component> --source-root <dataset_root> \
    --work-dir <scratch_directory> \
    --arm align_video__udef__wrist \
    --arms-config evaluation/configs/arms.yaml \
    --base-url <endpoint> --model-id <served_id> \
    --out video-job.json

Repeat with the same inputs and --arm baseline_upstream__udef__wrist --out upstream-job.json. For a subset, give both runs the same --episodes <split.json>. Each job needs exclusive use of its model replica for the whole run so the counter differences contain only that job’s traffic.

Outputs include elapsed_seconds, n_episodes_predicted and server counters for prompt, generation and total tokens. Compare those token counters only within the same visual format. For a contact-sheet versus video token comparison, use the processor count above.