Cognition and State Modeling

query_historical_action_timeline

Queries a stored action timeline to confirm whether a task step happened and when it occurred.

Tool Introduction

Core parameters, trigger timing, and visual before/after demo references.

Short Explanation

Use this tool to check whether a previous step was already completed before the planner continues.

InputAction history + query action + time window

OutputAction occurrence times and status

Trigger TimingTriggered on demand after the required input files and configuration are prepared.

RuntimeLocal / cloud wrapper

BeforeAction history + query action + time window

Prepare the scene, image, video, sensor stream, prompt, or configuration expected by the original project.

AfterAction occurrence times and status

Read the produced visualization, prediction, map, trajectory, mask, grasp pose, or other documented artifact.

Preset Example

A quick-run style example for the documentation page.

Inputtools/query-historical-action-timeline/examples/video_001_features.npy

Promptquery_action: pour; time_window: full video

ExpectedA list of start and end frames for matching action segments.

Parameters And Output

Readable controls and the meaning of each returned artifact.

Parameter Explanation

feature_filepathtools/query-historical-action-timeline/examples/video_001_features.npy

Precomputed video features used by the timeline model or wrapper.

query_actiontext

Action label or natural-language step to look up.

time_windowtextfull

Optional frame or timestamp range to constrain the query.

sample_ratenumber1

Temporal sampling rate used when mapping model outputs back to frames.

Output Explanation

found

Boolean status indicating whether the action appears in the history.

occurrences

Detected action spans with start and end frames.

timestamp

Frame or time index for each matched action occurrence.

How To Use

Official resources, deployment steps, academic context, citation, and source-reported benchmark numbers.

Resources

GitHubhttps://github.com/yabufarha/ms-tcn Code Downloadhttps://github.com/yabufarha/ms-tcn/archive/refs/heads/main.zip Paperhttps://arxiv.org/abs/1903.01945

Deployment Notes

Prepare video features or action timeline outputs under the tool examples folder.
Configure the action label set and sample rate used by the source segmentation model.
Run the query wrapper with a repository-relative feature path.
Save the resulting timeline JSON under tools/query-historical-action-timeline/runs/.

Relative Path Example

python tools/query-historical-action-timeline/run.py --features tools/query-historical-action-timeline/examples/video_001_features.npy --query-action pour --output tools/query-historical-action-timeline/runs/timeline.json

Expected Result Shape

{
  "tool": "query-historical-action-timeline",
  "status": "ok",
  "results": [
    {
      "label": "Historical action timeline query",
      "score": 0.87,
      "output": "Action occurrence times and status"
    }
  ],
  "timing": {
    "runtime": "The submitted wrapper is described as interactive, but the upstream proxy paper primarily reports segmentation accuracy metrics rather than wrapper latency.",
    "device": "documented in source benchmark when available"
  },
  "artifacts": {
    "visualization": "tools/query-historical-action-timeline/runs/visualization.png",
    "raw_predictions": "tools/query-historical-action-timeline/runs/predictions.json"
  }
}

Paper figure

Academic Info

Paper identity and contribution summary.

TitleMS-TCN-style action timeline tool

AuthorsAdd authors

VenueSubmitted tool sheet / local wrapper

ContributionExposes temporal action segmentation outputs as a memory query interface for SOP and multi-step tasks.

Citation

@misc{queryhistoricalactiontimelineYEAR,
  title={MS-TCN-style action timeline tool},
  author={Author},
  year={YEAR},
  note={Submitted tool sheet / local wrapper},
  url={https://arxiv.org/abs/1903.01945}
}

Benchmark

Only compact, source-reported numbers are shown here.

Dataset	Metric	Value	Runtime	Source
50Salads	F1@10 / F1@25 / F1@50; Edit; Acc	76.3 / 74.0 / 64.5; 67.9; 80.7	MS-TCN 4-stage upstream model	Upstream proxy: MS-TCN paper, Table 1
GTEA	F1@10 / F1@25 / F1@50; Edit; Acc	87.5 / 85.4 / 74.6; 81.4; 79.2	MS-TCN with fine-tuning	Upstream proxy: MS-TCN paper, Table 10
Breakfast	F1@10 / F1@25 / F1@50; Edit; Acc	52.6 / 48.1 / 37.9; 61.7; 66.3	MS-TCN with I3D features	Upstream proxy: MS-TCN paper, Table 10

Artifacts

Upstream MS-TCN paper, official repository link, feature tensor shape, mock timeline output, and local deployment notes from the submitted spreadsheet.

Demo Images

Visual references from the original tool. Click any image to inspect the original size.