SASI | Yongpeng Cao

Yongpeng Cao, Masahiro Hirano, Hyuno Kim, and Yuji Yamakawa

TL;DR: SASI uses sub-action semantics to help a skeleton-based model recognize human actions earlier from incomplete observations.

Abstract

Human actions are naturally composed of smaller, meaningful sub-actions. SASI integrates those semantics with skeleton-based spatiotemporal features through a dual-branch, cross-modal fusion framework. The method is designed for robust early action recognition in human–robot interaction and runs in real time at 29 Hz. Experiments on BABEL show improved recognition from both complete and partial action sequences.

Method

Kinematic branch

A GCN backbone extracts spatial and temporal features from the skeleton sequence.

Semantic branch

Predicted sub-actions are represented as language features, exploring fine-grained semantic relationship in motion.

Cross-modal fusion

Cross-attention aligns kinematic features with sub-action semantics to recognize actions from partial observations.

Results

Video

Citation

@misc{cao2026sasi,
  title         = {SASI: Leveraging Sub-Action Semantics for Robust Early Action Recognition in Human-Robot Interaction},
  author        = {Yongpeng Cao and Masahiro Hirano and Hyuno Kim and Yuji Yamakawa},
  year          = {2026},
  eprint        = {2604.27508},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}

We used AI tools to create this page.