An Efficient Vision-Language-Action Model for Combat Tasks in 3D Action Role-Playing Games
Peng Chen, Pi Bu, Yingyao Wang, Xinyi Wang, Ziming Wang, Jie Guo, Yingxiu Zhao, Qi Zhu, Jun Song, Siran Yang, Jiamang Wang, Bo Zheng · Alibaba Group
CombatVLA 是一个专为3D动作角色扮演游戏（ARPG）战斗任务设计的高效 Vision-Language-Action 模型。仅3B参数，通过三阶段渐进式学习和 Action-of-Thought（AoT）推理链，在保持通用视觉理解能力的同时，实现比现有方案快 50 倍的实时战斗响应，并在专属战斗理解基准 CUBench 上超越 GPT-4o、Gemini 等所有现有大模型。CombatVLA is an efficient Vision-Language-Action model designed specifically for combat tasks in 3D action role-playing games (ARPG). With only 3B parameters, it combines three-stage progressive learning with an Action-of-Thought (AoT) reasoning chain to deliver real-time combat responses 50 times faster than existing solutions while preserving general visual understanding, and it surpasses GPT-4o, Gemini and all other existing large models on the dedicated combat understanding benchmark CUBench.
现有大型视觉语言模型（VLM）无法满足复杂3D游戏战斗场景对"秒级响应、高分辨率感知和动态战术推理"的实时要求，而专用小模型又缺乏足够的通用视觉理解能力。Existing large vision-language models (VLM) cannot meet the real-time requirements that complex 3D game combat scenarios impose on "second-level responses, high-resolution perception and dynamic tactical reasoning", while dedicated small models lack sufficient general visual understanding.
"Real-time decision-making in complex 3D environments … demand second-level responses, high-resolution perception, and tactical reasoning under dynamic conditions."
图1：CombatVLA 综合对比。左侧：在 CUBench 战斗理解基准上超越 GPT-4o、Qwen2.5-VL；中间：推理延迟仅 1.85s，比 Cradle（61.68s）快33倍，比 VARP（90.23s）快50倍；右侧：实际战斗任务成功率超越人类基线。Figure 1: Overall comparison of CombatVLA. Left: it surpasses GPT-4o and Qwen2.5-VL on the CUBench combat understanding benchmark. Middle: inference latency is only 1.85s, 33 times faster than Cradle (61.68s) and 50 times faster than VARP (90.23s). Right: success rates on real combat tasks exceed the human baseline.
50×比 VARP 快（推理延迟）faster than VARP (inference latency)
63.61%CUBench 平均准确率（第一名）CUBench average accuracy (1st place)
1.85s单次推理延迟single-pass inference latency
3B模型参数量model parameters
现有方案的核心瓶颈在于：The core bottlenecks of existing solutions are:
通用 VLM（GPT-4o, Gemini）：需要60–90秒推理，完全无法用于实时战斗控制。General-purpose VLMs (GPT-4o, Gemini): require 60-90 s of inference, entirely unusable for real-time combat control.
框架型方案（Cradle, VARP）：多次模型调用（5–10次）叠加延迟，1次战斗指令需61–90秒；且高度依赖提示工程。Framework-based solutions (Cradle, VARP): repeated model calls (5-10 of them) accumulate latency, so a single combat instruction takes 61-90 s; they also depend heavily on prompt engineering.
专用小型模型：缺乏对复杂战斗场景的感知和推理能力。Dedicated small models: lack the perception and reasoning ability required by complex combat scenarios.
CombatVLA 的目标是在单次1.85秒推理内完成"观察→思考→行动"的完整闭环，同时保持通用视觉能力不大幅退化。The goal of CombatVLA is to close the full "observe to think to act" loop within a single 1.85-second inference, while keeping general visual ability from degrading substantially.
02方法Method
CombatVLA 以 Qwen2.5-VL-3B 为骨干，冻结视觉编码器，对语言模型全参数 SFT。核心创新是三阶段渐进式训练策略与 Action-of-Thought（AoT）推理链，以及针对动作优先级的自适应加权损失函数。CombatVLA uses Qwen2.5-VL-3B as its backbone, freezes the vision encoder and applies full-parameter SFT to the language model. Its core innovations are a three-stage progressive training strategy, the Action-of-Thought (AoT) reasoning chain, and an adaptive weighted loss function tailored to action priority.
图2：CombatVLA 完整系统流程。(a) Action Tracker 以毫秒级精度同步记录键盘/鼠标/截图；(b) 三种 AoT 训练数据类型（视频AoT、帧AoT、截断AoT）；(c) CUBench 基准的三类任务；(d) 模型训练约束；(e) 在实际PC上部署运行。Figure 2: The complete CombatVLA system pipeline. (a) Action Tracker synchronously records keyboard, mouse and screenshots with millisecond precision. (b) Three types of AoT training data (video AoT, frame AoT, truncated AoT). (c) The three task categories of the CUBench benchmark. (d) Training constraints of the model. (e) Deployment on a real PC.
Action Tracker：高精度数据采集Action Tracker: high-precision data collection
自研 Python 工具，以毫秒级精度同步记录键盘、鼠标操作与游戏截图，将动作通过时间戳协议对齐到最近的未来帧。共采集 25,000 张游戏截图（分辨率 1008×560），标注 5,000 条高质量 AoT 序列，覆盖10种动作类型（移动 WSAD、闪避 space、治疗 R、定身 1、轻/重攻击、冲刺等）。A self-developed Python tool synchronously records keyboard and mouse operations together with game screenshots at millisecond precision, aligning each action to the nearest future frame through a timestamp protocol. In total 25,000 game screenshots (resolution 1008×560) were collected and 5,000 high-quality AoT sequences annotated, covering 10 action types (movement WSAD, dodge space, heal R, immobilize 1, light/heavy attack, sprint and so on).
输入完整视频（n=20帧，m=10 fps），学习粗粒度战斗理解与完整 AoT 推理链。训练 3 个 epoch。Takes the complete video as input (n=20 frames, m=10 fps) to learn coarse-grained combat understanding and the full AoT reasoning chain. Trained for 3 epochs.
Stage 2 · Fine-Grained Frames-AoT
输入帧序列（k=4 回溯帧），实现细粒度时序对齐，捕捉战斗动作的帧级依赖。训练 1 个 epoch。Takes a frame sequence as input (k=4 look-back frames) to achieve fine-grained temporal alignment and capture frame-level dependencies of combat actions. Trained for 1 epoch.
Stage 3 · Frames-Truncated-AoT
引入特殊 ⟨TRUNC⟩ token，允许模型在完成动作预测后立即截断输出，跳过冗余推理文本。平均 token 数从 116.57 降至 43.10，推理速度提升约2倍。训练 3 个 epoch。Introduces a special ⟨TRUNC⟩ token that lets the model truncate its output immediately after finishing action prediction, skipping redundant reasoning text. The average token count drops from 116.57 to 43.10, yielding roughly a 2× inference speed-up. Trained for 3 epochs.
图3：Action-of-Thought 推理链可视化。⟨TRUNC⟩ token 使模型在推理完成时提前截断输出，平均减少输出 token 数从 116.57 降至 43.10，实现约2倍推理加速，同时保持完整的战略推理语义。Figure 3: Visualization of the Action-of-Thought reasoning chain. The ⟨TRUNC⟩ token lets the model truncate its output early once reasoning is complete, reducing the average number of output tokens from 116.57 to 43.10 and achieving roughly a 2× inference speed-up while preserving the complete strategic reasoning semantics.
自适应动作加权损失（Adaptive Action-Weighted Loss）Adaptive Action-Weighted Loss
损失函数结合三项：语言建模损失 ℒ_lang、动作对齐损失 ℒ_align（优先级感知匹配，权重序列 α_i = 2^(k−i−1) 指数衰减并归一化至 [0.1, 1.0]）与模态对比损失 ℒ_con。三项联合优化，保证动作预测精度与视觉-语言模态对齐。The loss combines three terms: the language modeling loss ℒ_lang, the action alignment loss ℒ_align (priority-aware matching whose weight sequence α_i = 2^(k−i−1) decays exponentially and is normalized to [0.1, 1.0]), and the modality contrastive loss ℒ_con. Optimizing the three jointly guarantees both action prediction accuracy and vision-language modality alignment.
CUBench：战斗理解评测基准CUBench: a combat understanding benchmark
图4：CUBench 三类任务示例。Gathering（单图采集）：从单帧判断当前状态；Comprehension（多图理解）：跨帧序列理解战斗事件；Reasoning（多选推理）：面对动态情境进行多步战术推理。共914条标注样本，经10名标注员交叉验证。Figure 4: Examples of the three CUBench task types. Gathering (single-image): judge the current state from one frame. Comprehension (multi-image): understand combat events across a frame sequence. Reasoning (multiple-choice): perform multi-step tactical reasoning in dynamic situations. The benchmark holds 914 annotated samples, cross-validated by 10 annotators.
CUBench 共 914 条标注样本（39.4% Gathering / 22.3% Comprehension / 38.3% Reasoning），覆盖《黑神话：悟空》（BMW）和《只狼：影逝二度》（Sekiro）两款游戏，评测战斗场景下的感知、理解与战术推理能力。CUBench contains 914 annotated samples (39.4% Gathering / 22.3% Comprehension / 38.3% Reasoning) spanning the two games Black Myth: Wukong (BMW) and Sekiro: Shadows Die Twice (Sekiro), and evaluates perception, comprehension and tactical reasoning in combat scenarios.
03实验Experiments
在 CUBench 战斗理解基准和13项实际战斗任务上进行评测，与 GPT-4o、Gemini-2.0-flash、Claude3.5-Sonnet、Qwen2.5-VL 系列及 VARP、Cradle 等基线对比。Evaluation is carried out on the CUBench combat understanding benchmark and on 13 real combat tasks, compared against baselines including GPT-4o, Gemini-2.0-flash, Claude3.5-Sonnet, the Qwen2.5-VL series, VARP and Cradle.
CombatVLA 超越第二名 Gemini-2.0-flash 5.71 个百分点；在 Reasoning 子任务上比 Claude3.5-Sonnet 高出 14.28 个百分点。CombatVLA outperforms the runner-up Gemini-2.0-flash by 5.71 percentage points, and on the Reasoning subtask it exceeds Claude3.5-Sonnet by 14.28 percentage points.
通用基准保持能力（Table 2）Retention of general benchmark ability (Table 2)
基准Benchmark
Qwen2.5-VL-3B（基线）Qwen2.5-VL-3B (baseline)
CombatVLA-3B
MME
2157
2141
VideoMME
61.5
58.7
OCRBench
797
741
专项微调后通用能力小幅下降，但仍保持竞争力，表明 AoT 训练策略有效缓解了灾难性遗忘。After the task-specific fine-tuning, general ability drops slightly but stays competitive, showing that the AoT training strategy effectively mitigates catastrophic forgetting.
图5：定性结果示例。展示实际战斗中 AoT 推理过程、Python 动作指令生成和帧序列对应关系——模型能准确识别敌人攻击模式并生成对应的闪避/反击指令。Figure 5: Qualitative examples. They show the AoT reasoning process during real combat, the generation of Python action commands and the correspondence with the frame sequence — the model accurately recognizes enemy attack patterns and produces the matching dodge or counterattack commands.
消融实验Ablation studies
渐进式学习阶段消融（Table 4）：Ablation over the progressive learning stages (Table 4):
全三阶段（完整模型）：平均 63.61%，延迟 1.85s（Stage 3 在 Reasoning 上比 Stage 2 提升 +6.85 个百分点）All three stages (full model): 63.61% average, 1.85s latency (on Reasoning, Stage 3 improves over Stage 2 by +6.85 percentage points)
损失函数消融（Table 5）：Ablation over the loss function (Table 5):
去除 ℒ_con：61.58%Removing ℒ_con: 61.58%
去除 ℒ_align：61.64%Removing ℒ_align: 61.64%
完整损失（三项联合）：63.61%Full loss (all three terms jointly): 63.61%
结果表明三个损失分量各自贡献显著，缺少任一均导致准确率下降约2个百分点。The results show that each of the three loss components contributes significantly; dropping any one of them lowers accuracy by about 2 percentage points.
在《黑神话：悟空》10项任务和《只狼：影逝二度》3项任务（共13项）上进行实际游戏测试，涵盖简单到极难不同难度级别。Easy 任务达到与人类相当的成功率（70–90%）；Hard/Very Hard 任务上持续超越所有基线和人类水平，成功率稳定在70%以上。此外，模型在未见过任务上表现出较强的零样本泛化能力。Real in-game testing was run on 10 tasks in Black Myth: Wukong and 3 tasks in Sekiro: Shadows Die Twice (13 in total), spanning difficulty levels from easy to extremely hard. On Easy tasks the success rate matches human level (70-90%); on Hard and Very Hard tasks the model consistently surpasses all baselines and human performance, holding a success rate above 70%. The model also shows strong zero-shot generalization to unseen tasks.
04局限性Limitations
说明：以下局限性均为作者在论文中明确陈述的内容（stated by authors）。Note: every limitation below is explicitly stated by the authors in the paper (stated by authors).
任务定义较为简化Task definitions are somewhat simplistic
论文原文指出："Task definitions are somewhat simplistic" given evolving VLA capabilities。当前13项战斗任务的划分粒度和复杂度，随着 VLA 能力的进步可能需要重新设计，无法完全覆盖真实游戏中更丰富的战术场景。The paper states: "Task definitions are somewhat simplistic" given evolving VLA capabilities. The granularity and complexity of the current 13 combat tasks may have to be redesigned as VLA capabilities advance, and they cannot fully cover the richer tactical scenarios of real games.
泛化场景受限Limited generalization scenarios
论文明确指出："Research tested within BMW and SSDT games and has not extended to other scenarios"。模型目前仅在《黑神话：悟空》和《只狼》两款游戏上验证，跨游戏、跨类型的泛化能力尚未系统评估。The paper states explicitly: "Research tested within BMW and SSDT games and has not extended to other scenarios". The model has so far been validated only on Black Myth: Wukong and Sekiro; cross-game and cross-genre generalization has not been systematically evaluated.
感知能力仍有提升空间Perception ability still has room for improvement
论文指出现有 VLM 和 VLA 在感知任务上"room for improvement"——当前模型在高速战斗中对细粒度视觉信号（如敌人细微动作、血量变化）的感知精度仍存在不足，通用基准（MME、VideoMME）上的小幅下降也印证了这一点。The paper notes that existing VLMs and VLAs still leave "room for improvement" on perception tasks — in high-speed combat the current model is still not precise enough at perceiving fine-grained visual signals (such as subtle enemy movements or health-bar changes), a point confirmed by the slight drops on the general benchmarks (MME, VideoMME).
