CoStream: Composing Simple Behaviors for Generalizable Complex Manipulation
Jun 1, 2026ยท,,,,,,,,,ยท
1 min read
Haonan Chen
Yuxiang Ma
Stephen Tian
Xiaoshen Han
Wenlong Huang
Feiyang Wu
Yunzhu Li
Jiajun Wu
Edward H. Adelson
Yilun Du
CoStream composes semantic, predictive and reactive behaviors, in contrast to pipelined and end-to-end systems.Abstract
Long-horizon, contact-rich manipulation โ such as seating a GPU into a PCIe slot โ demands both millimeter precision and generalization to new tasks. Classical pipelines achieve precise control through brittle, task-specific interfaces that need costly redesigns, while monolithic policies generalize better but lose precision on out-of-distribution tasks unless retrained. Both assume a capability, once acquired, must ship as a rigid whole rather than being freely decomposed and recomposed. We show that complex manipulation can emerge from composing simple, independent behaviors. CoStream orchestrates foundation models and diverse sensors into composable core behaviors: a semantic behavior extracting spatial constraints, a predictive behavior forecasting trajectories from imagined videos, and a reactive behavior providing high-frequency tactile corrections. On a shared SE(3) interface, these compose by right-multiplication into one pose command per control step. We demonstrate CoStream on 8 real-world tasks spanning everyday manipulation and precision assembly โ with the strongest gains in contact-rich assembly and object transfer โ and show robust recovery from manual perturbations during execution.
Type
Publication
Preprint, arXiv:2606.26423 [cs.RO]
Long-horizon, contact-rich manipulation demands both millimeter precision and generalization to new tasks โ requirements that classical task-specific pipelines and monolithic end-to-end policies each satisfy only one half of. CoStream instead treats a capability as something that can be decomposed and recomposed.
Composable behaviors
CoStream orchestrates foundation models and diverse sensors into three independent core behaviors that share a common SE(3) interface and combine by right-multiplication into a single pose command per control step:
- Semantic behavior โ extracts spatial constraints for the task.
- Predictive behavior โ forecasts trajectories from imagined videos.
- Reactive behavior โ supplies high-frequency tactile corrections.
Results
Evaluated on 8 real-world tasks spanning everyday manipulation and precision assembly:
- 96.7% mean success on contact-rich assembly, where the baselines score 0%.
- 66.7% mean success on everyday manipulation, against 11.7% for ฯ0.5.
- Drill insertion reaches 100% success with the reactive behavior, dropping to 20% without it.
- Completes a full motherboard assembly (CPU, GPU, RAM) with zero retraining between subtasks, and recovers robustly from manual perturbations during execution.
Work done with collaborators at Harvard University, Stanford University, MIT and Columbia University.