当前 VLA（Vision-Language-Action）模型在机器人操作基准上报告了令人印象深刻的成功率，但这些高分可能掩盖了模型在真实部署场景中的脆弱性。本文系统构建了 LIBERO-Plus 基准，覆盖 7 个扰动维度、21 个子维度，共 10,030 个任务，对 10 款主流 VLA 模型进行深度评估，发现成功率可从 95% 骤降至 30% 以下。Current VLA (Vision-Language-Action) models report impressive success rates on robotic manipulation benchmarks, but these high scores may mask their fragility in real deployment scenarios. This work systematically builds the LIBERO-Plus benchmark, covering 7 perturbation dimensions and 21 sub-dimensions with 10,030 tasks in total, and evaluates 10 mainstream VLA models in depth, finding that success rates can plummet from 95% to below 30%.
VLA 模型在标准机器人操作基准上屡创新高，然而这些基准场景高度受控，难以反映真实部署环境中的变化。当摄像头视角、场景光照、背景纹理或语言描述稍作调整，模型性能会如何变化？现有研究缺乏对多维度扰动的系统性探索。VLA models keep setting records on standard robotic manipulation benchmarks, yet these benchmark scenes are highly controlled and hardly reflect the variability of real deployment environments. How does model performance change when the camera viewpoint, scene illumination, background texture or language description is slightly adjusted? Existing work lacks a systematic exploration of multi-dimensional perturbations.
"Visual–Language–Action (VLA) models report impressive success rates on robotic manipulation benchmarks, yet these results may mask fundamental weaknesses in robustness."
图1：LIBERO-Plus 基准框架。共包含 7 个一级扰动维度（对象布局 O、背景纹理 B、光照条件 L、摄像头视角 C、机器人初始状态、语言指令 R、传感器噪声 N）和 21 个子维度，并按 5 个难度级别（Level-1 最易至 Level-5 最难）分层构建测试集，共生成 10,030 个任务。Figure 1: The LIBERO-Plus benchmark framework. It contains 7 top-level perturbation dimensions (objects layout O, background textures B, light conditions L, camera viewpoints C, robot initial states, language instructions R, sensor noise N) and 21 sub-dimensions, and builds the test set in a stratified way over 5 difficulty levels (Level-1 easiest to Level-5 hardest), yielding 10,030 tasks in total.
95%→<30%典型模型在摄像头视角扰动下的性能骤降范围Range of the performance collapse of typical models under camera viewpoint perturbation
10参与评估的最新 VLA 模型数量Number of recent VLA models evaluated
10,030LIBERO-Plus 基准任务总数Total number of LIBERO-Plus benchmark tasks
现有机器人操作基准（如 LIBERO、AGNOSTOS、COLOSSEUM 等）在扰动维度覆盖、细粒度分析和自动化生成等方面存在明显不足。LIBERO-Plus 正是为弥补这一空白而设计：在标准 LIBERO 场景基础上，自动化地注入 7 类扰动并按难度分级，从而构建可重复、可扩展的鲁棒性评估框架。Existing robotic manipulation benchmarks (such as LIBERO, AGNOSTOS and COLOSSEUM) fall clearly short in perturbation-dimension coverage, fine-grained analysis and automated generation. LIBERO-Plus is designed precisely to fill this gap: on top of the standard LIBERO scenes, it automatically injects 7 kinds of perturbations and grades them by difficulty, thereby building a reproducible and extensible robustness evaluation framework.
02方法Method
LIBERO-Plus 在 LIBERO 模拟环境基础上，系统设计了 7 个扰动维度的注入机制，并引入 5 级难度分层和组合扰动分析框架，用于评估模型的单维度鲁棒性与跨维度泛化能力。Building on the LIBERO simulation environment, LIBERO-Plus systematically designs injection mechanisms for 7 perturbation dimensions and introduces a 5-level difficulty stratification together with a compositional perturbation analysis framework, in order to evaluate the single-dimension robustness and cross-dimension generalization of models.
7 个扰动维度设计Design of the 7 perturbation dimensions
视觉类扰动Visual perturbations
Objects Layout (O1-O2)：在场景中随机添加 1–5 个干扰物体（从 416 个物体库中采样），或对目标物体施加位姿扰动（x/y/z 位移 + pitch/yaw/roll 旋转）Objects Layout (O1-O2): randomly add 1–5 distractor objects to the scene (sampled from a library of 416 objects), or apply pose perturbations to the target object (x/y/z translation + pitch/yaw/roll rotation)
Background Textures (B1-B2)：替换整体场景主题（950 种纹理库）或桌面/地面材质Background Textures (B1-B2): replace the overall scene theme (a library of 950 textures) or the tabletop/floor material
Light Conditions (L1-L4)：调整漫反射颜色（RGB 通道）、平行光源方向、镜面高光强度及阴影开关Light Conditions (L1-L4): adjust the diffuse color (RGB channels), the directional light orientation, the specular highlight intensity and the shadow on/off switch
Camera Viewpoints (C1-C3)：摄像头距离缩放（1.01×–2.00×）、球坐标位置偏移（方位角/仰角 15°–75°）及朝向旋转（yaw/pitch/roll 2°–10°）Camera Viewpoints (C1-C3): camera distance scaling (1.01×–2.00×), spherical-coordinate position offset (azimuth/elevation 15°–75°) and orientation rotation (yaw/pitch/roll 2°–10°)
控制与语义类扰动Control and semantic perturbations
Robot Initial States：对机器人关节角度施加 0.1–0.5 幅度的随机扰动，测试运动初始化的鲁棒性Robot Initial States: apply random perturbations of magnitude 0.1–0.5 to the robot joint angles, testing the robustness of motion initialization
Language Instructions (R1-R3)：将原始指令改写为更长的对话形式（R1）、常识替换（R2）或推理链变体（R3），保留语义意图但改变表达形式Language Instructions (R1-R3): rewrite the original instruction into a longer conversational form (R1), a commonsense substitution (R2) or a reasoning-chain variant (R3), preserving the semantic intent while changing the surface form
Sensor Noise (N1-N5)：向图像注入 5 类噪声（运动模糊、高斯模糊、缩放模糊、雾化、玻璃模糊），每类 5 个强度级别（L1–L5）Sensor Noise (N1-N5): inject 5 types of noise into the image (motion blur, Gaussian blur, zoom blur, fog, glass blur), each with 5 intensity levels (L1–L5)
5 级难度分层5-level difficulty stratification
每个扰动维度按扰动强度由弱到强划分为 5 个难度等级（Level-1 至 Level-5）。Level-1 代表最小可感知扰动，所有参考模型均可通过；Level-5 代表最强扰动，所有模型均失败。难度分级保证了基准的区分度，同时便于分析各模型的"鲁棒性边界"。Every perturbation dimension is divided by perturbation strength, from weak to strong, into 5 difficulty levels (Level-1 to Level-5). Level-1 denotes the minimal perceptible perturbation, which all reference models can pass; Level-5 denotes the strongest perturbation, on which all models fail. The difficulty grading guarantees the discriminative power of the benchmark and makes it convenient to analyze the "robustness boundary" of each model.
图2：各扰动维度的难度级别（L1–L5）任务比例分布。饼图展示了 7 个维度下不同难度任务的占比。从中可以看出，摄像头视角（Camera）和机器人初始状态（Robot）维度的高难度任务比例更高，而光照（Light）和背景（Background）维度相对容易。Figure 2: Distribution of task proportions over difficulty levels (L1–L5) for each perturbation dimension. The pie charts show the share of tasks of each difficulty under the 7 dimensions. The camera viewpoint (Camera) and robot initial state (Robot) dimensions have a higher share of hard tasks, whereas the light (Light) and background (Background) dimensions are relatively easy.
为量化不同扰动间的交互效应，本文引入组合泛化差距（Compositional Generalization Gap）指标：To quantify the interaction effects between different perturbations, the paper introduces the Compositional Generalization Gap metric:
Δᵢⱼ = Cov(Dᵢ, Dⱼ | Y=1)
通过对 2,000 次双维度扰动的试验进行协方差分析，发现一致性负差距（negative gap），说明扰动并非独立叠加，而是存在耦合交互效应，共同暴露了模型表征中的纠缠问题（entanglement in the learned representations）。A covariance analysis over 2,000 two-dimension perturbation trials reveals a consistently negative gap, showing that perturbations do not superpose independently but interact in a coupled manner, jointly exposing entanglement in the learned representations.
03实验Experiments
实验评估了 10 款主流 VLA 模型：OpenVLA、OpenVLA-OFT（3个变体）、π₀、π₀-fast、Nora、WorldVLA、UniVLA、RIPT-VLA，涵盖自回归、扩散、世界模型和强化学习等不同训练范式。每个模型在各扰动维度的多个子任务上运行 100 次。The experiments evaluate 10 mainstream VLA models: OpenVLA, OpenVLA-OFT (3 variants), π₀, π₀-fast, Nora, WorldVLA, UniVLA and RIPT-VLA, covering different training paradigms including autoregressive models, diffusion models, world models and reinforcement learning. Each model is run 100 times on multiple sub-tasks of every perturbation dimension.
关键发现一：视觉扰动是最大威胁Key finding 1: visual perturbations are the biggest threat
图3：对象布局鲁棒性分析（Figure 1）。左侧为添加干扰物体（confounding objects）时各模型性能，π₀、π₀-fast、RIPT-VLA 等高性能模型降幅有限；右侧为目标物体位移（target displacement）时的性能，几乎所有模型均大幅下降。这表明模型依赖记忆的位置先验，而非真正理解物体语义。Figure 3: Object layout robustness analysis (Figure 1). Left: model performance when confounding objects are added, where high-performing models such as π₀, π₀-fast and RIPT-VLA degrade only mildly; right: performance under target displacement, where almost all models drop sharply. This indicates that the models rely on memorized position priors rather than truly understanding object semantics.
摄像头视角变化和机器人初始状态扰动导致成功率大幅下降，平均降幅远超其他维度。π₀ 在摄像头扰动下降幅高达 -78.4%，在机器人初始状态扰动下降幅达 -87.6%。相比之下，配备腕部摄像头的 OpenVLA-OFT 在摄像头扰动下仅下降 -37.4%，说明多视角输入可显著提升视觉鲁棒性。Camera viewpoint changes and robot initial state perturbations cause large drops in success rate, with average declines far exceeding those of the other dimensions. π₀ falls by as much as -78.4% under camera perturbation and by -87.6% under robot initial state perturbation. By contrast, OpenVLA-OFT, which is equipped with a wrist camera, falls only -37.4% under camera perturbation, showing that multi-view input can significantly improve visual robustness.
关键发现二：模型几乎忽视语言指令Key finding 2: models almost ignore the language instruction
图4：语言指令分析实验（Figure 3）。(a) 将语言输入置空（blank instruction）：OpenVLA-OFT 成功率几乎无变化，证明其实际上是一个 VA（视觉-动作）模型而非 VLA 模型。(b) 目标对象替换实验：将指令中的目标从"alphabet soup"改为"tomato sauce"后，模型仍继续执行原始动作，成功率接近零，说明模型不具备跨对象指令跟随能力。Figure 4: Language instruction analysis experiments (Figure 3). (a) Blanking the language input (blank instruction): the success rate of OpenVLA-OFT hardly changes, proving that it is in fact a VA (vision-action) model rather than a VLA model. (b) Target object substitution: after the target in the instruction is changed from "alphabet soup" to "tomato sauce", the model still carries out the original action and the success rate approaches zero, showing that the model lacks cross-object instruction-following ability.
"VLA models do not possess strong cross-object instruction-following generalization… they rely on fixed vision–action mappings rather than fully exploiting language signals in task decision-making."
语言扰动平均仅导致 -25.3% 的下降（各维度中最小），但这并非因为模型具有语义鲁棒性——空指令实验证明模型根本未有效利用语言输入。语言维度的"低敏感性"实为语言信号被忽视的体现。Language perturbations cause an average drop of only -25.3% (the smallest among all dimensions), but this is not because the models are semantically robust — the blank-instruction experiment proves that they do not effectively use the language input at all. The "low sensitivity" of the language dimension is in fact a manifestation of the language signal being ignored.
关键发现三：光照鲁棒性差异揭示腕部摄像头的作用Key finding 3: differences in illumination robustness reveal the role of the wrist camera
图5：光照鲁棒性分析（Figure 2）。极端消融实验：全黑输入（all-black）下所有模型性能趋近于零；而仅将第三人称视角图像置黑（3rd-black）时，OpenVLA-OFT、RIPT-VLA、π₀ 仍分别维持 43.6%–67.3% 的成功率，证明腕部摄像头提供了对光照变化鲁棒的近距离几何线索。Figure 5: Illumination robustness analysis (Figure 2). Extreme ablations: under an all-black input the performance of every model approaches zero; when only the third-person view image is blacked out (3rd-black), OpenVLA-OFT, RIPT-VLA and π₀ still sustain success rates of 43.6%–67.3% respectively, proving that the wrist camera supplies close-range geometric cues that are robust to illumination change.
微调实验：数据增强显著提升鲁棒性Fine-tuning experiment: data augmentation markedly improves robustness
在 20,000+ 条覆盖多种扰动的轨迹数据上对 OpenVLA-OFT 进行微调后（8× A100，100,000 步，学习率 5×10⁻⁴），LIBERO-Plus 总成功率从 67.9% 提升至 79.5%（+11.6pp）。摄像头维度改善最显著（55.6% → 92.8%，+37.2pp），机器人状态维度改善相对有限（21.7% → 30.3%，+8.6pp）。After fine-tuning OpenVLA-OFT on 20,000+ trajectories covering a variety of perturbations (8× A100, 100,000 steps, learning rate 5×10⁻⁴), the overall LIBERO-Plus success rate rises from 67.9% to 79.5% (+11.6pp). The camera dimension improves the most (55.6% → 92.8%, +37.2pp), while the robot state dimension improves relatively little (21.7% → 30.3%, +8.6pp).
图6：各难度级别（Level-1 至 Level-5）的模型性能曲线（Figure 5）。展示了 4 款代表性模型在不同扰动维度下随难度增加的成功率变化趋势，体现出各模型鲁棒性边界的差异。Figure 6: Model performance curves across difficulty levels (Level-1 to Level-5) (Figure 5). It shows how the success rates of 4 representative models evolve with increasing difficulty under different perturbation dimensions, revealing the differences between the robustness boundaries of the models.
组合扰动分析Compositional perturbation analysis
在双维度组合扰动实验（2,000 次试验）中，实际联合成功率始终低于独立假设下的预期值（Δᵢⱼ < 0），证明不同类型的扰动存在负向交互效应。这意味着模型泛化能力具有内在不可分解性（intrinsically non-decomposable），单维度评估无法预测真实场景中的综合性能。In the two-dimension compositional perturbation experiments (2,000 trials), the actual joint success rate is consistently below the value expected under an independence assumption (Δᵢⱼ < 0), proving that different types of perturbation interact negatively. This means that the generalization ability of the models is intrinsically non-decomposable, and single-dimension evaluation cannot predict the overall performance in real scenarios.
04局限性Limitations
Note：以下局限性部分来自论文明确说明，部分为从方法设计中推断（已标注）。Note: some of the limitations below are stated explicitly in the paper, others are inferred from the method design (marked as such).
局限于模拟环境，真实世界迁移性待验证Confined to simulation; real-world transfer remains to be verified
LIBERO-Plus 完全在 LIBERO 模拟环境中构建，所有扰动（光照、背景、传感器噪声等）均通过仿真生成。真实机器人场景中的物理噪声、光学畸变和未见物体的多样性远超仿真范畴，模拟结论能否迁移至实物部署仍需进一步验证。（论文明确指出为未来工作方向）LIBERO-Plus is built entirely inside the LIBERO simulation environment, and all perturbations (illumination, background, sensor noise and so on) are produced by simulation. The physical noise, optical distortion and diversity of unseen objects in real robot scenes go far beyond what simulation covers, so whether the simulated conclusions transfer to physical deployment still needs further verification. (explicitly noted in the paper as a direction for future work)
评估范围限于 LIBERO 任务集，泛化性受限Evaluation is restricted to the LIBERO task suite, which limits generality
所有测试任务均基于 LIBERO 原有的桌面操作场景，场景多样性（物体类别、任务结构、桌面配置）相对有限。对于更复杂的双臂操作、移动机器人或开放世界任务，本基准的结论未必适用。（从任务设计推断）All test tasks are based on the original tabletop manipulation scenes of LIBERO, whose scene diversity (object categories, task structure, tabletop configuration) is relatively limited. For more complex bimanual manipulation, mobile robots or open-world tasks, the conclusions of this benchmark may not hold. (inferred from the task design)
微调数据分布与测试分布高度重叠The fine-tuning data distribution overlaps heavily with the test distribution
提升鲁棒性的微调实验使用了与测试集同源的 20,000+ 条扰动轨迹。在分布外（out-of-distribution）场景下，这种数据驱动的鲁棒性提升效果可能大幅缩水，且不能说明模型习得了真正可组合的泛化能力。（从训练设置推断）The fine-tuning experiment that improves robustness uses 20,000+ perturbed trajectories drawn from the same source as the test set. Under out-of-distribution scenarios this data-driven robustness gain may shrink drastically, and it does not show that the model has acquired truly compositional generalization. (inferred from the training setup)
语言维度评估未触及细粒度语义理解The language dimension evaluation does not reach fine-grained semantic understanding
现有的语言扰动（改写/替换/推理链）主要测试句式层面的鲁棒性，对更复杂的语义组合（新颖物体-动作组合、空间关系理解、多步骤指令）覆盖不足。模型究竟在何种语义粒度上失败仍不明确。（从实验设计推断）The existing language perturbations (rewriting / substitution / reasoning chain) mainly test robustness at the level of sentence form, and cover more complex semantic compositions (novel object-action combinations, spatial relation understanding, multi-step instructions) insufficiently. At which semantic granularity the models actually fail remains unclear. (inferred from the experiment design)
