Vision-Language-Action Models with Plug-and-Play Safety Constraint Layer
Songqiao Hu, Zeyi Liu, Shuang Liu, Jun Cen, Zihan Meng, Xiao He
在已有 VLA 模型的动作输出后面，外接一个即插即用的安全约束层（AEGIS），通过 VLM 感知识别障碍物、CBF-QP 求解最小偏移量，使机器人在保留原始任务意图的同时实时避免碰撞，且无需重新训练底层 VLA 模型。A plug-and-play safety constraint layer (AEGIS) is attached behind the action output of an existing VLA model: a VLM perceives and identifies obstacles and a CBF-QP solves for the minimal deviation, so that the robot avoids collisions in real time while preserving the original task intent, and without retraining the underlying VLA model.
VLA 模型在语义指令跟随方面表现出色，但在真实世界部署中缺乏明确的安全保障。碰撞可能导致硬件损坏、人员受伤或财产损失。现有安全方法大多依赖强化学习中的软约束，缺乏在推理时强制执行安全边界的显式机制。VLA models excel at following semantic instructions, but lack explicit safety guarantees in real-world deployment. Collisions may cause hardware damage, human injury or property loss. Most existing safety approaches rely on soft constraints in reinforcement learning and lack an explicit mechanism for enforcing safety boundaries at inference time.
"Safety stands as a prerequisite for the real-world deployment of VLA models, as collisions in unstructured environments can lead to hardware damage, human injury, or property loss."
标准 VLA 模型（左）：专注于完成任务指令，但在执行过程中可能与环境障碍物发生碰撞。A standard VLA model (left): it concentrates on completing the task instruction, but may collide with environmental obstacles during execution.VLSA/AEGIS（右）：在 VLA 动作输出后加入安全约束层，实时修正可能导致碰撞的动作，同时保留原始任务意图。VLSA/AEGIS (right): a safety constraint layer is added after the VLA action output, correcting in real time the actions that may lead to a collision while preserving the original task intent.
+59.16%障碍物回避率（CAR）提升Gain in collision avoidance rate (CAR)
+17.25%任务成功率（TSR）提升Gain in task success rate (TSR)
0.356 ms安全层额外计算延迟Extra computation latency of the safety layer
~1.86%占总推理延迟比例Share of the total inference latency
02方法Method
AEGIS（Action Execution Guarded by Invariant Safety）是一个即插即用的安全约束层，放置在任意 VLA 模型的动作输出之后，由两个功能模块构成：基于视觉语言的安全评估模块，以及基于动作驱动的安全保证控制模块。整个系统无需重新训练底层 VLA 模型。AEGIS (Action Execution Guarded by Invariant Safety) is a plug-and-play safety constraint layer placed after the action output of an arbitrary VLA model. It consists of two functional modules: a vision-language based safety assessment module, and an action-driven safety-guaranteed control module. The whole system requires no retraining of the underlying VLA model.
AEGIS 完整流程：① VLA 模型生成原始动作 u_vla；② VLM 根据任务指令和视觉观测识别最危险的障碍物；③ GroundingDINO 精确定位障碍物并融合点云信息；④ CBF-QP 求解器计算最小偏移量，输出安全修正动作 u_safe。The complete AEGIS pipeline: ① the VLA model generates the raw action u_vla; ② the VLM identifies the most dangerous obstacle from the task instruction and the visual observation; ③ GroundingDINO localizes the obstacle precisely and fuses point-cloud information; ④ the CBF-QP solver computes the minimal deviation and outputs the safety-corrected action u_safe.
利用 VLM 根据任务指令和视觉观测，识别场景中最可能阻碍机器人运动的障碍物。随后使用 GroundingDINO 进行开放集目标检测，结合深度信息（点云融合），将语义层面的风险转换为物理空间中的回避约束。这一模块实现了从自然语言描述到物理空间安全要求的跨模态转换。A VLM is used to identify, from the task instruction and the visual observation, the obstacle in the scene most likely to impede the robot's motion. GroundingDINO then performs open-set object detection and, combined with depth information (point-cloud fusion), converts semantic-level risk into avoidance constraints in physical space. This module realizes a cross-modal transformation from natural-language description to safety requirements in physical space.
视觉语言安全评估模块：左侧为语义风险识别阶段（VLM + 任务指令 → 危险物体名称），右侧为空间定位阶段（GroundingDINO 检测 + 点云深度融合 → 障碍物三维位置和几何形状）。Vision-language safety assessment module: on the left is the semantic risk identification stage (VLM + task instruction → name of the hazardous object), on the right is the spatial localization stage (GroundingDINO detection + point-cloud depth fusion → 3D position and geometry of the obstacle).
模块二：基于 CBF-QP 的安全保证控制Module Two: CBF-QP based Safety-Guaranteed Control
核心优化问题：以最小偏移量修正 VLA 原始动作，同时保证机器人末端执行器不与障碍物碰撞。数学上将末端执行器和障碍物分别建模为椭球体（MVEE），通过在单位球面上引入虚拟辅助状态计算最小有符号距离，构造 Control Barrier Function h(x)，并将安全约束表达为：The core optimization problem: correct the raw VLA action with the minimal deviation while guaranteeing that the robot end-effector does not collide with the obstacle. Mathematically, the end-effector and the obstacle are each modeled as an ellipsoid (MVEE); a virtual auxiliary state on the unit sphere is introduced to compute the minimal signed distance, a Control Barrier Function h(x) is constructed, and the safety constraint is expressed as:
"minimize ||u − u_vla||² subject to ḣ(x) ≥ −α(h(x))"
其中 u_vla 为 VLA 原始输出动作，u 为安全修正动作，α 为扩展 class-K 函数。该 QP 问题可在毫秒级内求解，额外延迟仅约 0.356 ms。Here u_vla is the raw action output by the VLA, u is the safety-corrected action, and α is an extended class-K function. This QP problem can be solved within milliseconds, adding only about 0.356 ms of latency.
最小体积包围椭球体（MVEE）拟合：左图为末端执行器点云拟合椭球体，右图为障碍物点云拟合椭球体。CBF 通过计算两椭球体之间的最小有符号距离定义碰撞安全集合。Minimum-volume enclosing ellipsoid (MVEE) fitting: the left figure fits an ellipsoid to the end-effector point cloud, the right figure fits an ellipsoid to the obstacle point cloud. The CBF defines the collision-safe set by computing the minimal signed distance between the two ellipsoids.
定理（安全保证）Theorem (Safety Guarantee)
假设感知准确、障碍物表示完整，"the AEGIS framework guarantees that the entire robot end-effector will not collide with the obstacle."（Theorem 1）Assuming accurate perception and a complete obstacle representation, "the AEGIS framework guarantees that the entire robot end-effector will not collide with the obstacle." (Theorem 1)
03实验Experiments
作者构建了新基准 SafeLIBERO，包含 4 个 LIBERO 子任务套件、16 个任务、32 个场景、共 1,600 个 episode，涵盖不同空间复杂度和障碍物干扰级别。基线模型为 OpenVLA-OFT 和 π₀.₅，指标包括：障碍物回避率（CAR）、任务成功率（TSR）、执行步数（ETS）。The authors build a new benchmark, SafeLIBERO, comprising 4 LIBERO task suites, 16 tasks, 32 scenes and 1,600 episodes in total, covering different levels of spatial complexity and obstacle interference. The baseline models are OpenVLA-OFT and π₀.₅; the metrics include collision avoidance rate (CAR), task success rate (TSR) and execution time steps (ETS).
主要量化结果（Table 1 — SafeLIBERO 全套件均值）Main quantitative results (Table 1 — mean over all SafeLIBERO suites)
方法Method
CAR ↑ (%)
TSR ↑ (%)
ETS ↓
OpenVLA-OFT
15.13
22.81
323.16
π₀.₅
18.69
50.88
278.24
AEGIS（本文）AEGIS (ours)
77.85
68.13
262.30
分套件性能Per-suite performance
套件Suite
CAR ↑ (%)
TSR ↑ (%)
ETS ↓
SafeLIBERO-Spatial
75.50
73.25
188.20
SafeLIBERO-Goal
81.50
75.25
179.60
SafeLIBERO-Object
74.75
80.25
201.26
SafeLIBERO-Long
79.63
43.75
480.12
SafeLIBERO 基准中的 16 个操作任务场景示例，覆盖 Spatial、Goal、Object、Long 四个套件，每个场景均包含不同位置和数量的障碍物干扰。Examples of the 16 manipulation task scenes in the SafeLIBERO benchmark, covering the four suites Spatial, Goal, Object and Long; every scene contains obstacle interference differing in position and number.
消融分析（Ablations）Ablation analysis (Ablations)
实验表明，VLM 驱动的障碍物识别模块和 CBF-QP 安全控制模块均不可或缺：去掉 VLM 识别后，系统无法正确确定需要回避的目标物体；去掉 CBF 约束后，安全保证在数学上不再成立，碰撞率显著上升。SafeLIBERO-Long 套件的 TSR（43.75%）明显低于其他套件，反映了长序列任务中分布偏移问题对策略性能的影响。Experiments show that both the VLM-driven obstacle identification module and the CBF-QP safety control module are indispensable: without VLM identification, the system cannot correctly determine which object should be avoided; without the CBF constraint, the safety guarantee no longer holds mathematically and the collision rate rises markedly. The TSR of the SafeLIBERO-Long suite (43.75%) is clearly lower than that of the other suites, reflecting the impact of distribution shift in long-horizon tasks on policy performance.
04局限性Limitations
说明：以下局限性均为作者在论文中明确陈述（stated by the authors）。Note: all limitations below are explicitly stated by the authors in the paper (stated by the authors).
AEGIS 无法达到 100% CAR 的主要原因在于上游感知失败：VLM 可能误识别障碍物、GroundingDINO 的边界框可能不准确、点云滤波可能低估障碍物的实际几何形状。此外，系统当前未对机器人"未建模的运动学部件（unmodeled kinematic components，such as the penultimate robot arm）"进行碰撞检测，这些部件同样可能发生碰撞。The main reason AEGIS cannot reach 100% CAR lies in upstream perception failures: the VLM may misidentify the obstacle, the bounding box of GroundingDINO may be inaccurate, and point-cloud filtering may underestimate the actual geometry of the obstacle. Moreover, the system currently performs no collision checking for the robot's "unmodeled kinematic components, such as the penultimate robot arm", which may collide as well.
分布偏移（Distribution Shift）Distribution Shift
当安全约束迫使机器人末端执行器绕开障碍物移动到训练数据中未见过的区域时，底层 VLA 策略的性能会出现退化："the robot, after successfully avoiding an obstacle, fails to complete the task"，即避障成功但无法继续完成原始任务。When the safety constraint forces the robot end-effector to move around the obstacle into a region unseen in the training data, the performance of the underlying VLA policy degrades: "the robot, after successfully avoiding an obstacle, fails to complete the task", i.e. obstacle avoidance succeeds but the original task cannot be completed.
仅平移约束，未扩展至完整 6-DoF（Rotational Constraints）Translation-only constraints, not extended to full 6-DoF (Rotational Constraints)
当前评估将末端执行器限定为纯平移运动。作者指出，引入完整 6-DoF（含旋转）虽然可能不会显著提升 CAR，但"would likely greatly boost the task success rate"，因为更大的运动自由度可以让机器人在绕障后更顺利地回归任务路径。The current evaluation restricts the end-effector to purely translational motion. The authors note that introducing full 6-DoF (including rotation), although it may not significantly improve CAR, "would likely greatly boost the task success rate", because a larger freedom of motion lets the robot return to the task path more smoothly after detouring around an obstacle.
