Learning from demonstration
Nakama Robotics Lab, University of Twente · Mar 2026 – Present
First-author manuscript in preparation
Abstract
A demonstration contains far more of the scene than the task depends on, and deciding which objects matter is normally done by a human drawing boxes. This vision module takes a demonstration bag and lets the robot supply its own labels: proprioceptive events mark the moments of interaction, those moments seed a frozen segmentation model, and the result is a role-tagged mask per frame written out as a JSON sidecar for the downstream learning code.
The problem
Before a policy can be learned from a demonstration, something has to decide which objects in the frame are part of the task and which are background. Doing that by hand puts a manual cost on every hour of demonstration collected, and caps the dataset at the labelling budget.
Proprioception as the annotator
Event detection runs over the merged demonstration trace and picks out grasp, release and force-contact moments, degrading gracefully when a given bag is missing either the gripper or the force-torque channel.
Each event indexes the corresponding camera frame, which seeds SAM 2, kept frozen, with a point or box prompt. Bidirectional propagation through the video then yields per-frame role-tagged masks for an arbitrary number of tracked objects. The annotation arrives free with the demonstration and scales with however much data is collected.
Rig and calibration
A Franka Research 3 with the Franka Hand gripper, a Bota SensONE wrist force-torque sensor, and a ZED Mini RGB-D camera. The camera is eye-in-hand, bracketed to the gripper rather than fixed in the world, so its pose in the base frame is a per-frame quantity derived from a single fixed bracket transform. The recorded pose is the sensor origin rather than the fingertip, which the derivation has to account for.
Pipeline
The module also converts MCAP robot recordings into synchronized multimodal traces, aligning joint states, force-torque and vision onto a common clock, and work continues on extracting 6D object poses from the segmented demonstrations.