SymSkill：符号与技能协同发明，用于数据高效的反应式长视野操作SymSkill: Symbol and Skill Co-Invention for Data-Efficient and Reactive Long-Horizon Manipulation

01 动机Motivation

当前机器人操作领域的两大主流范式——模仿学习与任务规划——分别存在根本性缺陷：前者缺乏可组合性，后者规划延迟高达数十秒，无法在动态环境中实时恢复故障。SymSkill 旨在融合两者的优势。The two dominant paradigms in robot manipulation — imitation learning and task planning — each suffer from a fundamental defect: the former lacks compositionality, while the latter incurs planning latency of tens of seconds and cannot recover from failures in real time in dynamic environments. SymSkill aims to fuse the strengths of both.

"Recent imitation learning approaches excel at reproducing skills from large datasets but tend to learn monolithic policies rather than reusable skills and predicates that compose into multi-step plans."

02 方法Method

SymSkill 的核心思想是利用操作任务中普遍存在的以物体为中心的相对位姿结构。系统在相对坐标系下学习谓词和技能，仅离线调用视觉语言模型（VLM）确定参考物体，从而同时保证数据效率和在线反应速度。The core idea of SymSkill is to exploit the object-centric relative pose structure that pervades manipulation tasks. Predicates and skills are learned in relative frames, and a vision-language model (VLM) is queried offline only, to determine the reference object, so that data efficiency and online reactivity are secured at the same time.

阶段一：演示分段与参考系选择Stage 1: demonstration segmentation and reference frame selection

根据速度阈值将每条演示分为预运动段（premotion，仅末端执行器运动）和运动段（motion，末端执行器 + 操作物体同时运动）。对运动段，使用 VLM（Gemini 2.5-Pro）在 n 个均匀采样帧上查询参考物体 o_ref，输出限制为已知场景物体集合，最大程度抑制幻觉。A velocity threshold splits each demonstration into a premotion segment (premotion, end-effector motion only) and a motion segment (motion, end-effector and manipulated object moving together). For motion segments, a VLM (Gemini 2.5-Pro) is queried over n uniformly sampled frames for the reference object o_ref, with its output restricted to the set of known scene objects so as to suppress hallucination as far as possible.

阶段二：相对位姿谓词学习Stage 2: relative pose predicate learning

谓词 ψ 建模为相对位姿的正态分布：平移分量 p_ee^o ~ N(μ_pos, Σ_pos)，旋转分量用对数映射 log(R) ~ N(μ_ori, Σ_ori) 表示。当 Mahalanobis 距离低于阈值 ε_pos 和 ε_ori 时谓词激活。运动段还额外学习物体-物体相对位姿谓词 ψ_o_int^o_ref，并在运动结束后约 2 秒的稳定窗口内增强鲁棒性。A predicate ψ is modeled as a normal distribution over relative pose: the translation component p_ee^o ~ N(μ_pos, Σ_pos), and the rotation component expressed through the log map log(R) ~ N(μ_ori, Σ_ori). The predicate activates when the Mahalanobis distance falls below the thresholds ε_pos and ε_ori. Motion segments additionally learn an object-object relative pose predicate ψ_o_int^o_ref, and robustness is strengthened over a stabilization window of about 2 seconds after the motion ends.

阶段三：算子学习Stage 3: operator learning

通过在分段边界处评估谓词，将演示转化为抽象状态序列，识别具有相同效果的重复转换组。算子 α = <params, pre, eff, maintain, skill> 的前条件、效果和维持条件分别通过集合交集运算自动提取，无需人工标注。Evaluating predicates at segment boundaries turns demonstrations into abstract state sequences and identifies repeated transition groups with identical effects. The preconditions, effects and maintain conditions of an operator α = <params, pre, eff, maintain, skill> are each extracted automatically by set intersection, with no manual annotation.

阶段四：SE(3) LPV-DS 技能学习Stage 4: SE(3) LPV-DS skill learning

每个算子关联一个在相对坐标系下的 SE(3) Linear Parameter Varying Dynamical Systems（LPV-DS）技能。位置控制使用线性时不变系统的混合 v = Σ γ_k(x) A_k(x - x*)，稳定性约束通过半正定规划（SDP）保证。技能的收敛反馈场使其对大范围初始位姿仍能鲁棒追踪目标。Each operator is associated with an SE(3) Linear Parameter Varying Dynamical Systems (LPV-DS) skill expressed in the relative frame. Position control uses a mixture of linear time-invariant systems v = Σ γ_k(x) A_k(x - x*), and stability constraints are guaranteed through semidefinite programming (SDP). The convergent feedback field of the skill lets it track the target robustly even from a wide range of initial poses.

阶段五：在线执行监控与三重恢复机制Stage 5: online execution monitoring and a triple recovery mechanism

在线执行时系统持续检查维持条件和预期效果。故障触发以下三种恢复机制之一：During online execution the system continuously checks maintain conditions and expected effects. A failure triggers one of the following three recovery mechanisms:

• 障碍物规避（Obstacle Avoidance）：通过特征值分解对策略进行调制，绕过非操作物体Obstacle Avoidance: the policy is modulated through eigenvalue decomposition in order to detour around non-manipulated objects
• 失败后重采样（Resampling after Failure）：当目标效果未满足（如物体掉落），从效果分布中重新采样策略吸引子并变换策略Resampling after Failure: when the target effect is not satisfied (for example the object is dropped), the policy attractor is resampled from the effect distribution and the policy is transformed accordingly
• 实时符号重规划（Real-Time Replanning）：在符号层进行 A* 重规划，延迟 <100ms，使机器人能响应动态环境变化Real-Time Replanning: A* replanning is carried out at the symbolic level with <100ms latency, letting the robot respond to dynamic environment changes

03 实验Experiments

实验在两个平台上验证：(1) RoboCasa 仿真环境中的 12 个单步任务及多步骤组合；(2) 真实 Franka 机械臂，仅从 5 分钟无结构 play data 中学习。每个任务仅使用 5–10 条演示，并与多个基线方法对比。The experiments are validated on two platforms: (1) 12 single-step tasks and their multi-step compositions in the RoboCasa simulator; (2) a real Franka arm, learning from only 5 minutes of unstructured play data. Each task uses only 5–10 demonstrations and is compared against several baseline methods.

表一：与相关工作的对比（Table I，论文原始数据）Table 1: comparison with related work (Table I, original data from the paper)

表二：RoboCasa 单步仿真结果（Table II，每任务 10 次测试）Table 2: RoboCasa single-step simulation results (Table II, 10 trials per task)

多步骤任务组合（StoreCheese）Multi-step task composition (StoreCheese)

在 RoboCasa 中定义新任务 "StoreCheese"，由四步组成：开门 → 抓起奶酪 → 放到台面 → 关门，共串联六个技能。无需任何额外数据，直接复用已学谓词/算子/技能完成多步规划，并在执行过程中成功触发多次符号层错误恢复。A new task "StoreCheese" is defined in RoboCasa, composed of four steps: open the door → pick up the cheese → place it on the counter → close the door, chaining six skills in all. Without any additional data, the already-learned predicates/operators/skills are reused directly for multi-step planning, and symbolic-level error recovery is triggered successfully several times during execution.

真实机器人实验Real-robot experiments

在真实 Franka 机械臂上，从 5 分钟 play data 中发现 11 个算子（Op1–Op11），涵盖拿放积木、操作锅盖、移动盘子等语义上有意义的操作原语。系统自动发现了诸如 "Thing-in-container"（物体在容器内）此类数据集特有的前条件结构，展示了无监督符号发现的能力。实验验证了三种恢复机制均有效工作。On a real Franka arm, 11 operators (Op1–Op11) are discovered from 5 minutes of play data, covering semantically meaningful manipulation primitives such as picking and placing blocks, manipulating a pot lid and moving plates. The system automatically discovered dataset-specific precondition structures such as "Thing-in-container", demonstrating the capacity for unsupervised symbol discovery. The experiments verify that all three recovery mechanisms work effectively.

消融实验分析Ablation analysis

关键消融结论：Key ablation findings:

• 监控机制的重要性：去掉监控后平均成功率从 85.0% 下降至 65.0%，证明实时故障检测与恢复的价值Importance of the monitoring mechanism: removing monitoring drops the average success rate from 85.0% to 65.0%, proving the value of real-time failure detection and recovery
• SE(3) LPV-DS 远优于 Diffusion Policy：在相同少量数据条件下，Diffusion Policy 平均成功率仅 3.3%，因为"预运动技能从宽泛初始位姿出发但测试状态迅速落出分布"，而 LPV-DS 的收敛反馈场克服了这一问题SE(3) LPV-DS far outperforms Diffusion Policy: under the same small-data setting, Diffusion Policy attains an average success rate of only 3.3%, because "premotion skills start from broad initial poses but test states quickly fall out of distribution", whereas the convergent feedback field of LPV-DS overcomes this problem
• NSIL 基线失效：重新实现的 NSIL 由于"频繁选择能解释演示但对规划无语义意义的伪相对位姿谓词"而失败，在包含干扰物体时尤为明显Failure of the NSIL baseline: the reimplemented NSIL fails because it "frequently selects spurious relative pose predicates that explain the demonstrations but carry no semantic meaning for planning", which is especially pronounced when distractor objects are present

04 局限性Limitations