SID: Sliding into Distribution for Robust Few-Demonstration Manipulation

01 Motivation

在低覆盖率的演示数据下，端到端视触动策略常见的失败模式是分布偏移：测试时机器人遇到的状态或观测超出了演示覆盖范围，往往表现为架构表达能力之外的失败。论文指出："a common failure mode is distribution shift—policies encounter states or observations not supported by the demonstrations—often dominating over architectural expressivity limitations." 更棘手的是，一次操作往往包含两种性质不同的阶段：全局、欠约束的 approach 阶段，和局部、对偏差敏感的 execution 阶段。用单一端到端策略同时覆盖两者，在只有少量演示时会让"pose shifts and perturbations brittle"。Under demonstration data with low coverage, a common failure mode of end-to-end visuo-tactile-motor policies is distribution shift: at test time the robot encounters states or observations that fall outside what the demonstrations cover, which typically shows up as failures beyond the limits of architectural expressivity. The paper states: "a common failure mode is distribution shift—policies encounter states or observations not supported by the demonstrations—often dominating over architectural expressivity limitations." More awkward still, one manipulation usually contains two phases of different character: a global, under-constrained approach phase, and a local, deviation-sensitive execution phase. Covering both with a single end-to-end policy makes the system "pose shifts and perturbations brittle" when only a handful of demonstrations are available.

SID 的核心思路：显式地把 approach 阶段的分布对齐（distribution alignment）与 execution 阶段的任务执行（task execution）解耦——用一个从少量演示学到的物体中心运动场负责"滑"回演示流形，再交给一个只需在这个可靠区域内工作的轻量 egocentric 策略执行任务，即 "sliding into distribution"。The core idea of SID: explicitly decouple distribution alignment in the approach phase from task execution in the execution phase — an object-centric motion field learned from a handful of demonstrations is in charge of "sliding" back onto the demonstration manifold, and a lightweight egocentric policy that only has to operate inside this reliable region then executes the task, i.e. "sliding into distribution".

02 Method

SID 由四部分组成：(i) 从少量 canonicalized 演示学到的 object-centric motion field f_θ；(ii) egocentric data augmentation 模块，用点云 reprojection 扩充 ID/OOD 观测；(iii) egocentric execution policy π_θ，用 conditional flow matching 训练，完成具体任务；(iv) 两种推理流程——open-loop 和 closed-loop。整体流程：先用 canonicalization operator C 把观测 o_t 映射为物体中心表示 x_t 及其 anchor pose；运动场在这个 SE(3) 锚定空间里工作，只覆盖演示的单调 approach 阶段。SID consists of four parts: (i) an object-centric motion field f_θ learned from a handful of canonicalized demonstrations; (ii) an egocentric data augmentation module that enlarges the ID/OOD observations via point-cloud reprojection; (iii) an egocentric execution policy π_θ, trained with conditional flow matching, which performs the concrete task; (iv) two inference procedures — open-loop and closed-loop. Overall pipeline: a canonicalization operator C first maps an observation o_t into an object-centric representation x_t together with its anchor pose; the motion field operates in this SE(3) anchored space and covers only the monotonic approach phase of the demonstrations.

物体中心运动场（distance-induced sliding field）Object-centric motion field (distance-induced sliding field)

论文把演示聚合成 approach 阶段状态集合 X_demo，用一个 pose-aligned 的 SE(3) 距离（平移距离 + 四元数测地角，按 σ_p, σ_r 加权）定义一个"到最近演示状态"的二次势能 E(x) = ½·d(x, X_demo)²。网络 f_θ 通过回归目标 sliding step v*(x)（由指向最近演示锚点的 SE(3) 位移方向与随距离衰减的步长 η(d) 构成）来学习梯度下降式的更新，训练目标是 MSE 回归：L_field = E[‖f_θ(x) − v*(x)‖²]。推理时反复执行 α·f_θ(x_t) 直到步长范数小于阈值 ε_field，即认为系统已"滑"到演示流形附近。场还可条件化在 object key k_obj 上，一个网络即可表示多物体的场。The paper aggregates the demonstrations into a set of approach-phase states X_demo and, with a pose-aligned SE(3) distance (translation distance + quaternion geodesic angle, weighted by σ_p, σ_r), defines a quadratic potential "to the nearest demonstration state", E(x) = ½·d(x, X_demo)². The network f_θ learns gradient-descent-style updates by regressing the target sliding step v*(x) (formed by the SE(3) displacement direction pointing to the nearest demonstration anchor and a step size η(d) that decays with distance); the training objective is MSE regression: L_field = E[‖f_θ(x) − v*(x)‖²]. At inference α·f_θ(x_t) is applied repeatedly until the step norm drops below the threshold ε_field, at which point the system is considered to have "slid" close to the demonstration manifold. The field can additionally be conditioned on an object key k_obj, so that a single network represents the fields of multiple objects.

Egocentric 数据增强：kinematically consistent 点云重投影Egocentric data augmentation: kinematically consistent point-cloud reprojection

为在稀疏监督下让 egocentric 策略保持鲁棒，SID 在末端执行器帧采样随机扰动，通过固定的手眼标定换算成相机帧扰动，仅对分割出的目标物体点云做重投影（gripper 点云保持不变），并同步更新相对动作以保持 action–observation 一致性。增强只应用在 approach 阶段片段，interaction-heavy 的 execution 阶段保持不变；同时从一个分离的外侧扰动范围采样 OOD 样本，作为辅助 ID-confidence head 的负例（"ID samples are generated from small bounded perturbations... OOD samples are generated from a disjoint outer perturbation range"）。To keep the egocentric policy robust under sparse supervision, SID samples random perturbations in the end-effector frame, converts them into camera-frame perturbations through the fixed hand-eye calibration, reprojects only the segmented target-object point cloud (the gripper point cloud stays unchanged), and updates the relative actions in step so as to preserve action–observation consistency. The augmentation is applied only to approach-phase segments, while the interaction-heavy execution phase is left unchanged; at the same time OOD samples are drawn from a disjoint outer perturbation range and used as negatives for the auxiliary ID-confidence head ("ID samples are generated from small bounded perturbations... OOD samples are generated from a disjoint outer perturbation range").

Egocentric execution policy（conditional flow matching）Egocentric execution policy (conditional flow matching)

策略是纯 egocentric、base-free 的（"no global views or base-frame states are inputs"）：只接收分割点云与 gripper width，用 PointNet 风格编码器提取特征，条件化任务嵌入 k_task 与 flow time τ，通过 conditional flow matching（沿线性概率路径插值噪声与目标动作）预测动作 chunk 的速度场；推理时从高斯噪声积分学到的 ODE 得到动作序列。共享编码器上还挂了一个 ID-confidence head c_φ（对辅助头做 stop-gradient，避免影响主任务表征），用于 closed-loop 推理中判断当前观测是否仍在策略可靠支持区域内。The policy is purely egocentric and base-free ("no global views or base-frame states are inputs"): it receives only the segmented point cloud and the gripper width, extracts features with a PointNet-style encoder, conditions on the task embedding k_task and the flow time τ, and predicts the velocity field of an action chunk through conditional flow matching (interpolating noise and target action along a linear probability path); at inference the action sequence is obtained by integrating the learned ODE from Gaussian noise. An ID-confidence head c_φ is also attached to the shared encoder (stop-gradient is applied to the auxiliary head so that it does not affect the main-task representation), used in closed-loop inference to judge whether the current observation still lies inside the policy's reliable support region.

两种推理流程Two inference procedures

Open-loop：先反复执行运动场直到其步长范数低于阈值 ε_field，再切换到执行策略直到 termination 信号触发。Closed-loop：用 ID confidence p_ID 决定是执行策略还是回退到运动场重新对齐；当 p_ID 过低时，进一步用位姿置信度 p_pose 判断是重新滑动对齐，还是执行一个手工设计的 RECOVERY 动作（把末端移到预设安全观测视角，重新获取分割与位姿估计）。closed-loop 变体因此能在 execution 阶段也持续纠正分布漂移。Open-loop: the motion field is applied repeatedly until its step norm falls below the threshold ε_field, then control switches to the execution policy until the termination signal fires. Closed-loop: the ID confidence p_ID decides whether to run the execution policy or to fall back to the motion field for re-alignment; when p_ID is too low, the pose confidence p_pose further decides whether to slide into alignment again or to execute a hand-designed RECOVERY action (moving the end-effector to a preset safe observation viewpoint and re-acquiring segmentation and pose estimation). The closed-loop variant can therefore keep correcting distribution drift during the execution phase as well.

03 Experiments

论文在六个真实机器人任务上评估：Open Drawer、Pour Water、Hang Tape、Hang Cup、PnP-Box、Multi-PnP-Box（涉及 articulated objects、可变形容器、以及 prehensile / non-prehensile 技能，如 approaching、grasping、non-prehensile nudging、hanging、pouring、placing）。SID 每任务只用 2 条原始演示，通过增强扩展到 100 条训练样本；对比的检索式基线 MT3、Ret-BC 给 10 条演示（因为检索方法依赖更大的演示记忆），其余可训练基线（ACT、DP3、π0.5）用 100 条演示训练，以保证公平比较下的稳定学习。评测覆盖五个问题：workspace-wide 泛化（OOD 初始位姿/视角）、动态扰动鲁棒性、杂乱场景鲁棒性、long-horizon 多步执行、以及跨任务技能复用与组合。The paper evaluates on six real-robot tasks: Open Drawer, Pour Water, Hang Tape, Hang Cup, PnP-Box and Multi-PnP-Box (involving articulated objects, deformable containers, and prehensile / non-prehensile skills such as approaching, grasping, non-prehensile nudging, hanging, pouring and placing). SID uses only 2 raw demonstrations per task, expanded through augmentation to 100 training samples; the retrieval-based baselines MT3 and Ret-BC are given 10 demonstrations (because retrieval methods rely on a larger demonstration memory), and the remaining trainable baselines (ACT, DP3, π0.5) are trained with 100 demonstrations, so as to guarantee stable learning under a fair comparison. The evaluation covers five questions: workspace-wide generalization (OOD initial pose/viewpoint), robustness to dynamic disturbances, robustness in cluttered scenes, long-horizon multi-step execution, and cross-task skill reuse and composition.

Q1 — Workspace-wide 泛化（静态 ID/OOD）Q1 — Workspace-wide generalization (static ID/OOD)

论文总结："By comparison, both SID-open and SID-closed remain robust in OOD, and outperform these baselines not only in OOD but also relative to their ID performance."（ACT、DP3、π0.5 在 OOD 下均出现明显退化，例如 Multi-PnP-Box 上 ACT/DP3/π0.5 的 OOD 成功率均为 0–0/0；而 MT3、Ret-BC 虽用检索式对齐规避了部分退化，但整体成功率仍低于 SID。）The paper summarizes: "By comparison, both SID-open and SID-closed remain robust in OOD, and outperform these baselines not only in OOD but also relative to their ID performance." (ACT, DP3 and π0.5 all degrade markedly under OOD; on Multi-PnP-Box, for example, the OOD success rates of ACT/DP3/π0.5 are all 0–0/0. MT3 and Ret-BC do sidestep part of the degradation with retrieval-based alignment, but their overall success rates remain below SID.)

Q2 — 动态扰动下的鲁棒性Q2 — Robustness under dynamic disturbances

Task	ACT	DP3	π0.5	Ret-BC	MT3	SID-O	SID-C
Hang Tape	6	14	80	12	4	44	84
Hang Cup	2	8	68	4	0	36	88
PnP-Box	2	10	74	8	2	52	86
Pour Water	4	10	78	6	2	40	82

TABLE II（动态设置，扰动场景聚合后 50 次试验/任务的成功率 %）。SID-closed 在所有四个任务上大幅领先。论文解释："SID-closed remains robust across both approach and execution phases; its OOD detection module provides a closed-loop safeguard that effectively handles OOD events induced by disturbances."（相比之下 SID-open 在 execution 阶段一旦被扰动推出 ID 区域，"the execution policy alone often fails to recover"。）TABLE II (dynamic setting; success rate % over 50 trials per task after aggregating the disturbance scenarios). SID-closed leads by a wide margin on all four tasks. The paper explains: "SID-closed remains robust across both approach and execution phases; its OOD detection module provides a closed-loop safeguard that effectively handles OOD events induced by disturbances." (By contrast, once SID-open is pushed out of the ID region by a disturbance during the execution phase, "the execution policy alone often fails to recover".)

Q3–Q5 — 杂乱场景、长时序任务与技能复用Q3–Q5 — Cluttered scenes, long-horizon tasks and skill reuse

Task（cluttered, with distractors）Task (cluttered, with distractors)	ACT	DP3	π0.5	Ret-BC	MT3	SID-O	SID-C
Hang Cup	42	24	72	58	72	86	88
PnP-Box	44	28	74	40	54	84	90

TABLE III：加入 distractor 物体、场景布局不再精心摆放后，SID 两个变体仍保持强表现，论文将此归因于 vision foundation models（SAM2/SAM3）提供的显式目标分割："these methods are less sensitive to layout curation and remain reliable even with non-canonical object arrangements."TABLE III: once distractor objects are added and the scene layout is no longer carefully curated, both SID variants still perform strongly, which the paper attributes to the explicit target segmentation provided by vision foundation models (SAM2/SAM3): "these methods are less sensitive to layout curation and remain reliable even with non-canonical object arrangements."

长时序任务 & 技能复用（Q4/Q5）Long-horizon tasks & skill reuse (Q4/Q5)

在两个超过三个顺序阶段的长时序任务上，SID 同样持续优于基线："SID achieves consistently strong performance on two long-horizon tasks with more than three sequential stages, outperforming the baselines."（Q4，归因于分解设计：运动场负责子任务间的分布过渡，执行策略只需在每个子任务的窄分布内工作。）此外，SID 的两个组件都很轻量，可以在新任务上按阶段加载之前训练好的子技能模型（Q5）：On two long-horizon tasks with more than three sequential stages, SID likewise consistently outperforms the baselines: "SID achieves consistently strong performance on two long-horizon tasks with more than three sequential stages, outperforming the baselines." (Q4; attributed to the decomposed design: the motion field takes care of the distribution transition between sub-tasks, while the execution policy only has to work inside the narrow distribution of each sub-task.) In addition, both SID components are lightweight, so previously trained sub-skill models can be loaded stage by stage on a new task (Q5):

04 Limitations

Pose estimation dependency（依赖位姿估计）Pose estimation dependency

"SID relies on an off-the-shelf 6D pose estimator; when pose estimates are unreliable, such as for transparent objects or rapid motion, the motion field can degrade." 作者提出未来可通过直接条件化在 object-centric 输入（如分割点云或 RGB）上来降低这种依赖。"SID relies on an off-the-shelf 6D pose estimator; when pose estimates are unreliable, such as for transparent objects or rapid motion, the motion field can degrade." The authors suggest that this dependence could be reduced in the future by conditioning directly on object-centric inputs (such as a segmented point cloud or RGB).

Limited scene awareness and approach feasibility（场景感知与可达性有限）Limited scene awareness and approach feasibility

"Our method does not explicitly model obstacles or collision constraints, and its motion-field alignment assumes that the demonstrated approach manifold remains observable and reachable." 当可行的 approach 区域被遮挡、阻挡或被杂乱环境约束时，性能可能下降。"Our method does not explicitly model obstacles or collision constraints, and its motion-field alignment assumes that the demonstrated approach manifold remains observable and reachable." When the feasible approach region is occluded, blocked or constrained by a cluttered environment, performance may degrade.

Limited cross-object generalization（跨物体泛化有限）Limited cross-object generalization

"We do not yet demonstrate strong generalization to unseen objects or categories; more object-agnostic representations are a natural next step."