Learning What Matters: Adaptive Information-Theoretic Objectives for Robot Exploration
自适应信息目标 QOED：让机器人自己找出"值得探索"的参数方向，而不是被无关维度带偏Adaptive information objective QOED: let the robot itself find out which parameter directions are "worth exploring", instead of being led astray by irrelevant dimensions
Youwei Yu, Jionghao Wang, Zhengming Yu, Wenping Wang, Lantao Liu
经典 Bayesian Optimal Experimental Design (BOED) 依赖专家预先指定"关键参数"，但高维机器人系统里很多参数方向弱可观测甚至不可辨识，简单丢弃又会让被忽略的方向反过来扭曲信息度量。本文提出 Quasi-Optimal Experimental Design (QOED)：先对 Fisher information matrix (FIM) 做特征空间分析自动挑出可辨识的关键参数方向，再用 Schur complement 构造"抑制冗余参数干扰"的自适应信息目标，并给出相对完整 BOED 目标的常数因子近似保证。Classical Bayesian Optimal Experimental Design (BOED) relies on experts to pre-specify the "critical parameters", but in high-dimensional robotic systems many parameter directions are weakly observable or even unidentifiable, and simply discarding them lets the omitted directions in turn distort the information measure. This paper proposes Quasi-Optimal Experimental Design (QOED): it first runs an eigenspace analysis of the Fisher information matrix (FIM) to automatically pick out the identifiable critical parameter directions, then uses the Schur complement to build an adaptive information objective that "suppresses the interference of redundant parameters", and gives a constant-factor approximation guarantee with respect to the full BOED objective.
信息论探索目标希望引导机器人收集能"降低模型参数不确定性"的数据，但现实中往往并不清楚收集到的数据究竟能揭示哪些信息。论文指出："Many parameter directions are weakly observable or unidentifiable, and even when identifiable directions are selected, omitted directions can still influence exploration and distort information measures." 换句话说，专家手工挑选"关键参数"这件事本身就很脆弱——选少了会漏掉真正重要的耦合方向，选多了又会被弱可观测方向带偏探索策略。Information-theoretic exploration objectives aim to guide a robot toward data that "reduces the uncertainty of the model parameters", but in reality it is often unclear what information the collected data can actually reveal. The paper points out: "Many parameter directions are weakly observable or unidentifiable, and even when identifiable directions are selected, omitted directions can still influence exploration and distort information measures." In other words, having an expert hand-pick the "critical parameters" is itself fragile: picking too few misses the truly important coupled directions, while picking too many lets weakly observable directions lead the exploration policy astray.
"This work addresses the challenge of adaptively designing learnable information-theoretic objectives. We focus on identifying critical parameters online and reducing the influence of nuisance parameters on the exploration objective."
Figure 1：对 26 个代表性机器人平台的物理参数做 Fisher information matrix 分析（环形图中心百分比 = 信息含量高的"关键参数"占全部参数的比例）。论文的观察是——不同颜色代表不同参数类型，环的大小代表信息含量，"The large variation in information distribution indicates that information can be concentrated in only a few critical physical parameters." 大多数机器人上，真正值得探索的参数方向只占很小一部分（多数环形图中心百分比在 3%–13% 之间）。Figure 1: Fisher information matrix analysis of the physical parameters of 26 representative robot platforms (the percentage at the centre of each donut = the share of all parameters that are high-information "critical parameters"). The observation of the paper is that different colours denote different parameter types and ring size denotes information content, "The large variation in information distribution indicates that information can be concentrated in only a few critical physical parameters." On most robots the parameter directions truly worth exploring make up only a small fraction (the centre percentage of most donuts lies between 3% and 13%).
35.23%QOED 相比完整 BOED 的动力学预测 RMSE 改善Dynamics-prediction RMSE improvement of QOED over full BOED
21.98%QOED 相比 QOED-Agnostic 的动力学预测 RMSE 改善Dynamics-prediction RMSE improvement of QOED over QOED-Agnostic
89%真机 rod balancing 任务成功率（QOED）Success rate of the real-robot rod balancing task (QOED)
QOED 分两步解决论文提出的 Q1（如何自动挑出关键参数子空间）和 Q2（如何设计目标以补偿被丢弃参数带来的失真）：先对 Fisher information matrix (FIM) 做特征分解定位"可观测子空间"并挑出一组互不冗余的可辨识参数坐标，再用 Schur complement 把这些关键参数对应的信息量中"能被其余参数线性预测的部分"剔除掉，从而得到一个对完整 BOED 目标有理论保证的近似目标。QOED solves in two steps the Q1 (how to automatically pick out the critical-parameter subspace) and Q2 (how to design an objective that compensates for the distortion caused by the discarded parameters) raised in the paper: it first eigendecomposes the Fisher information matrix (FIM) to locate the "well-observed subspace" and select a set of mutually non-redundant identifiable parameter coordinates, then uses the Schur complement to remove from the information carried by these critical parameters "the part that can be linearly predicted by the remaining parameters", thereby obtaining an approximate objective with a theoretical guarantee with respect to the full BOED objective.
Figure 2：以 box pushing 任务的 mass θ₁ / friction θ₂ 为例。上排：Fisher information landscape 与探索轨迹；下排：动力学局部几何。三种目标依次为 Vanilla BOED（左，Eq.5）、Agnostic QOED（中，Eq.9）、QOED（右，Eq.12）。论文指出："Optimizing information in both parameters can drift off the identifiable ridge. Restricting the objective to selected coordinates can stay closer to the ridge but may still excite non-identifiable directions. Our QOED method... adds constraints/regularization to suppress these directions and keep trajectories near the ridge."Figure 2: Taking the mass θ₁ / friction θ₂ of the box pushing task as an example. Top row: Fisher information landscape and exploration trajectories; bottom row: local geometry of the dynamics. The three objectives are, in order, Vanilla BOED (left, Eq.5), Agnostic QOED (middle, Eq.9) and QOED (right, Eq.12). The paper notes: "Optimizing information in both parameters can drift off the identifiable ridge. Restricting the objective to selected coordinates can stay closer to the ridge but may still excite non-identifiable directions. Our QOED method... adds constraints/regularization to suppress these directions and keep trajectories near the ridge."
Step I–III：挑选关键（可辨识）参数Steps I–III: selecting the critical (identifiable) parameters
先用 cross-entropy method (CEM) 从观测轨迹估计当前参数 φ̂，并近似更新参数不确定性协方差（类似 extended Kalman filtering 的更新）。接着对 FIM 做特征分解 𝓕_φ = WΛW⊤，用一个相对阈值 δ_eig（结合 α_eig·max diag(Λ) 做尺度鲁棒化）划出"well-observed subspace" W_o。最后在 W_o 内用贪心算法（lazy-greedy）选出一组预算为 |k|≤n 的参数坐标 k：目标是最大化 log det(W_ko W_ko⊤) 以覆盖尽量多方向，同时约束 |cos(r_i, r_j)| ≤ δ_cos 避免选到高度相关、彼此冗余的参数。First, the cross-entropy method (CEM) estimates the current parameters φ̂ from observed trajectories and approximately updates the parameter-uncertainty covariance (an update similar to extended Kalman filtering). Next, the FIM is eigendecomposed as 𝓕_φ = WΛW⊤, and a relative threshold δ_eig (made scale-robust by combining it with α_eig·max diag(Λ)) delineates the "well-observed subspace" W_o. Finally, a greedy algorithm (lazy-greedy) selects inside W_o a set of parameter coordinates k with budget |k|≤n: the goal is to maximize log det(W_ko W_ko⊤) so as to cover as many directions as possible, while the constraint |cos(r_i, r_j)| ≤ δ_cos avoids picking highly correlated, mutually redundant parameters.
QOED 目标：用 Schur complement 抑制 nuisance 方向The QOED objective: suppressing nuisance directions with the Schur complement
把 FIM 按关键参数 k 与其余参数 k̄ 分块，直接使用 tr(𝓕_kk)（Agnostic QOED，Eq.9）会高估关键参数的真实信息量——因为这部分"表观信息"可能主要来自与 nuisance 方向 k̄ 的相关性 𝓕_k k̄。QOED 转而使用 nuisance-adjusted Fisher information（Eq.11–12）： 𝓘_{k|k̄} = 𝓕_kk − 𝓕_k k̄ 𝓕_k̄k̄⁻¹ 𝓕_k̄k，其等价于"用 k̄ 的最佳线性预测剔除后，k 方向残余 score 的协方差"。论文证明（Theorem 1）：在 nuisance 影响有界、关键/非关键方向耦合有限的条件下，QOED 对完整 BOED 目标存在常数因子近似：tr(𝓕^π̂) ≥ (1−β)/(1+η) · tr(𝓕^π*)。Partitioning the FIM into the critical parameters k and the remaining parameters k̄, directly using tr(𝓕_kk) (Agnostic QOED, Eq.9) overestimates the true information of the critical parameters, because this "apparent information" may come mainly from the correlation 𝓕_k k̄ with the nuisance directions k̄. QOED instead uses the nuisance-adjusted Fisher information (Eq.11–12): 𝓘_{k|k̄} = 𝓕_kk − 𝓕_k k̄ 𝓕_k̄k̄⁻¹ 𝓕_k̄k, which is equivalent to "the covariance of the residual score along k after removing the best linear prediction from k̄". The paper proves (Theorem 1) that, under bounded nuisance influence and limited coupling between critical and non-critical directions, QOED admits a constant-factor approximation to the full BOED objective: tr(𝓕^π̂) ≥ (1−β)/(1+η) · tr(𝓕^π*).
接入 Model-Based Policy OptimizationPlugging into Model-Based Policy Optimization
为把 QOED 用作探索奖励，论文训练了一个物理参数条件化的可微动力学模型 q_θ(s_{t+1}|s_t,a_t,φ)——用 shortcut model（单步 transport map，六层 Transformer + AdaLN）实现，既能做想象轨迹 rollout，又能给出可用于估计 FIM 的可行密度。整体流程遵循 MBPO/Robotic World Model 范式：仿真中用 domain randomization 预训练策略与动力学模型，部署后在线采集数据、用 shortcut-model 目标更新动力学、再用 Alg. 1（QOED 目标）增广 reward 更新 PPO 策略，直到参数后验协方差与预测误差都收敛到阈值以下。To use QOED as an exploration reward, the paper trains a physical-parameter-conditioned differentiable dynamics model q_θ(s_{t+1}|s_t,a_t,φ), implemented with a shortcut model (a single-step transport map, a six-layer Transformer + AdaLN), which can both roll out imagined trajectories and provide a tractable density for estimating the FIM. The overall pipeline follows the MBPO / Robotic World Model paradigm: the policy and the dynamics model are pre-trained in simulation with domain randomization; after deployment, data are collected online, the dynamics are updated with the shortcut-model objective, and the PPO policy is updated with a reward augmented by Alg. 1 (the QOED objective), until both the parameter posterior covariance and the prediction error converge below threshold.
03 Experiments
仿真部分覆盖 MuJoCo 中 7 个环境（locomotion + manipulation）：Unitree Go1-Flat/Rough（54 维物理参数）、Humanoid G1-Flat/Rough（119 维）、Clearpath Jackal-Flat/Rough（31 维）、Inspire-FTP Hand-Rotate（104 维），每个场景 25 个随机种子、三种噪声水平（1σ/2σ/3σ）。真实世界部分在 Franka Emika Panda（rod balancing）与 Clearpath Jackal（wild wheeled navigation，含电机故障 / 对抗性拖拽负载 / 森林场景）上评测。对比基线包括 SAC-Adapt、Disagreement（ensemble 不确定性）、Domain-Random，以及 QOED 的两个消融 QOED-Agnostic（只用关键参数）与 BOED（用全部参数）。The simulation part covers 7 environments in MuJoCo (locomotion + manipulation): Unitree Go1-Flat/Rough (54-dimensional physical parameters), Humanoid G1-Flat/Rough (119-dimensional), Clearpath Jackal-Flat/Rough (31-dimensional) and Inspire-FTP Hand-Rotate (104-dimensional), with 25 random seeds per scenario and three noise levels (1σ/2σ/3σ). The real-world part is evaluated on a Franka Emika Panda (rod balancing) and a Clearpath Jackal (wild wheeled navigation, including motor failure / adversarial towed load / forest scenes). The baselines compared against include SAC-Adapt, Disagreement (ensemble uncertainty) and Domain-Random, plus the two ablations of QOED: QOED-Agnostic (critical parameters only) and BOED (all parameters).
Figure 3：7 个机器人环境下的归一化 episodic task reward（10 个随机种子均值）。论文写道："QOED-PHYSICS with ground-truth physics consistently outperforms baselines, validating our adaptive information objective. QOED with learned dynamics also performs well, highlighting the promise of learned models for exploration." BOED（贪婪追求全参数信息量）策略较弱；QOED-Agnostic 因忽略被丢弃参数的耦合影响，同样不及 QOED；G1-Rough 上 QOED 性能有所下降，论文将其归因于学习到的动力学模型在训练早期未能可靠捕捉关键参数。Figure 3: Normalized episodic task reward across 7 robot environments (mean over 10 random seeds). The paper writes: "QOED-PHYSICS with ground-truth physics consistently outperforms baselines, validating our adaptive information objective. QOED with learned dynamics also performs well, highlighting the promise of learned models for exploration." BOED (greedily pursuing information over all parameters) yields a weaker policy; QOED-Agnostic, by ignoring the coupling effect of the discarded parameters, likewise falls short of QOED; on G1-Rough the performance of QOED degrades somewhat, which the paper attributes to the learned dynamics model failing to reliably capture the critical parameters early in training.Figure 4 场景：Franka Panda 需要辨识三个立方体的 mass、inertia、friction，挑出摩擦力最大的一个并堆叠平衡。论文报告：QOED 达到 89% 成功率，而 QOED-Physics、BOED、QOED-Agnostic 分别只有 42%、8%、17%。 失败多因质心偏移（仅 2cm 误差即可导致堆叠倒塌）；QOED 采取"先辨识 mass/inertia，再估计 friction"的自适应策略，参数收敛明显更快。Figure 4 setting: the Franka Panda has to identify the mass, inertia and friction of three cubes, pick the one with the largest friction and stack it in balance. The paper reports: QOED reaches an 89% success rate, whereas QOED-Physics, BOED and QOED-Agnostic reach only 42%, 8% and 17% respectively. Failures are mostly caused by centre-of-mass offset (an error of just 2cm can topple the stack); QOED adopts an adaptive strategy of "first identify mass/inertia, then estimate friction", and its parameters converge markedly faster.
为什么动力学预测提升，但参数估计误差相近？Why does dynamics prediction improve while parameter-estimation error stays comparable?
论文做了事后归因分析：固定所有非关键参数为真值，测量识别出的关键参数集合能解释多少动力学预测误差——结果分别为 65.9%（Go1）、53.1%（G1）、51.9%（Jackal）、47.2%（Hand）。以 G1 为例，QOED 仅选出 16 个参数（占全部参数的 13.4%），却能解释相当大比例的预测误差，说明一个紧凑的可辨识子集足以主导动力学表现。The paper performs a post-hoc attribution analysis: fixing all non-critical parameters to their ground-truth values, it measures how much of the dynamics-prediction error the identified set of critical parameters can explain; the results are 65.9% (Go1), 53.1% (G1), 51.9% (Jackal) and 47.2% (Hand) respectively. Taking G1 as an example, QOED selects only 16 parameters (13.4% of all parameters) yet explains a considerable share of the prediction error, showing that a compact identifiable subset is enough to dominate the dynamics behaviour.
超参数鲁棒性与关键参数识别（Ablations）Hyperparameter robustness and critical-parameter identification (Ablations)
论文对 δ_eig、α_eig、δ_cos 做网格搜索：在 Jackal 环境上 QOED 的动力学预测 RMSE 为 2.42±0.11，而 QOED-Agnostic 为 4.77±2.51，表明 QOED 在较宽的超参数范围内保持稳健。The paper grid-searches δ_eig, α_eig and δ_cos: on the Jackal environment the dynamics-prediction RMSE of QOED is 2.42±0.11, whereas that of QOED-Agnostic is 4.77±2.51, indicating that QOED stays robust over a fairly wide hyperparameter range.
Figure 6：随着学习迭代次数增加，通过学习到的动力学模型识别关键参数的 miss rate（三种噪声水平）在四个平台上均持续下降，说明 QOED 对关键参数子空间的估计会随在线数据积累而不断改善。Figure 6: As the number of learning iterations grows, the miss rate of identifying critical parameters through the learned dynamics model (three noise levels) keeps decreasing on all four platforms, showing that the estimate QOED makes of the critical-parameter subspace keeps improving as online data accumulate.
真实世界导航实验中，QOED 把 52.5% 的探索时间花在 mass 与 friction 上，而 QOED-Agnostic 和 BOED 分别花费 87.5% 和 89.5%——论文认为这拖慢了探索、也降低了对电机相关效应和对抗性外力的估计质量。In the real-world navigation experiment QOED spends 52.5% of its exploration time on mass and friction, whereas QOED-Agnostic and BOED spend 87.5% and 89.5% respectively; the paper argues that this slows exploration down and also degrades the estimation quality of motor-related effects and adversarial external forces.
04 Limitations
Note：论文在 §VI-C "Discussions and Limitations" 中明确讨论了以下两点；第三点为论文实验部分明确报告的现象（本身即被论文归因和讨论，非纯推断）。Note: The paper explicitly discusses the following two points in §VI-C "Discussions and Limitations"; the third point is a phenomenon explicitly reported in the experimental part of the paper (attributed and discussed by the paper itself, not pure inference).
依赖显式物理参数化，不如 SAC 类方法通用Relies on an explicit physical parameterization, hence less general than SAC-style methods
论文原话："Although the SAC-based baselines perform poorly in our scenarios, they are usually more general than our MBPO setup because we require physics parameterization to define the exploration objective." 也就是说 QOED 的信息目标是围绕"物理参数 φ"定义的，需要一个可解释的参数化空间；论文提出一种缓解思路——用类似 VAE 的方式学习隐参数化，在隐空间中做探索，但这仍是未来工作，本文未实现。In the words of the paper: "Although the SAC-based baselines perform poorly in our scenarios, they are usually more general than our MBPO setup because we require physics parameterization to define the exploration objective." That is, the information objective of QOED is defined around the "physical parameters φ" and needs an interpretable parameterization space; the paper suggests one mitigation, namely learning a latent parameterization in a VAE-like manner and exploring in the latent space, but this is still future work and is not implemented here.
真机训练时间有限，BOED 类目标本身难以用 RL 学好Real-robot training time is limited, and BOED-style objectives are themselves hard to learn with RL
论文原话："In the real world, limited training time also leads to less refined behavior than in simulation: BOED-style objectives can be difficult to learn with RL, as observed in prior work. This creates a tension between objective complexity and exploration efficiency, which remains an important direction for future work." 即目标设计得越精细（如 QOED 的 Schur complement 修正），对策略优化本身的学习难度也可能越高，这与探索效率之间存在权衡，论文未给出解决方案。In the words of the paper: "In the real world, limited training time also leads to less refined behavior than in simulation: BOED-style objectives can be difficult to learn with RL, as observed in prior work. This creates a tension between objective complexity and exploration efficiency, which remains an important direction for future work." That is, the more refined the objective design (such as the Schur-complement correction of QOED), the harder policy optimization itself may become; this trades off against exploration efficiency, and the paper offers no solution.
学习到的动力学模型早期不可靠时，QOED 性能会下降（论文报告的现象）When the learned dynamics model is unreliable early on, QOED performance degrades (a phenomenon reported by the paper)
在 G1-Rough 仿真环境中，论文观察并说明："QOED performance drops in G1-Rough because the learned dynamics model does not reliably capture the critical parameters during early learning." 也就是说 QOED 的关键参数识别质量本身依赖动力学模型的准确度，在训练早期动力学模型尚未收敛时，这种耦合可能拖累策略表现。In the G1-Rough simulation environment the paper observes and explains: "QOED performance drops in G1-Rough because the learned dynamics model does not reliably capture the critical parameters during early learning." That is, the quality with which QOED identifies the critical parameters itself depends on the accuracy of the dynamics model, and early in training, when the dynamics model has not yet converged, this coupling may drag down policy performance.
