CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking

📄 arXiv: 2607.15004v1 📥 PDF

作者: Ruilong Ren, Songsheng Cheng, Yunpeng Zhou, Hanxuan Chen, Xiangyue Wang, Tianle Zeng, Shuai Yuan, Binbo Li, Hanzhong Guo, Ji Pei, Da Zhang, Kangli Wang

分类: cs.RO

发布日期: 2026-07-16

💡 一句话要点

提出CosFly-VLA以解决UAV动态目标追踪中的遮挡问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 无人机追踪 视觉-语言-动作 空间感知 动态目标 强化学习 遮挡恢复 多模态学习

📋 核心要点

  1. 现有的VLA策略在目标被遮挡时容易失去目标状态，导致错误的追踪行为。
  2. CosFly-VLA通过空间感知模型联合定位目标和估计可见性，生成连续飞行动作，提升追踪精度。
  3. 实验结果显示，CosFly-VLA在多个测试集上显著降低了平均位移误差，并提高了成功率。

📝 摘要（中文）

动态目标追踪对于在复杂城市环境中运行的无人机（UAV）至关重要，然而现有的视觉-语言-动作（VLA）策略在目标被建筑、植被或路边物体遮挡时，性能往往会下降。为此，本文提出了CosFly-VLA，一个空间感知的VLA模型，能够联合定位目标、估计其可见性，并通过结构化预测接口生成连续的飞行动作。该模型通过大规模多样化数据源进行训练，采用空间基础的持续预训练（CPT）和三阶段的课程监督微调（SFT）过程，最终通过闭环强化学习（RL）阶段优化追踪行为。实验结果表明，相较于OpenVLA，CosFly-VLA在开放环节平均位移误差（ADE）上分别降低了34.1%和35.3%，闭环优化提高了成功率（SR）29.8%和2.5%。

🔬 方法详解

问题定义：本文旨在解决无人机在复杂城市环境中动态目标追踪时，由于遮挡导致的目标状态丢失和错误追踪的问题。现有方法在目标被遮挡时性能显著下降，无法有效恢复目标状态。

核心思路：CosFly-VLA模型通过空间感知机制，联合目标定位、可见性估计和飞行动作生成，旨在提高在遮挡情况下的追踪能力。该设计使得模型能够在复杂环境中保持对目标的持续关注。

技术框架：该模型的整体架构包括三个主要阶段：首先是空间基础的持续预训练（CPT），其次是三阶段的课程监督微调（SFT），最后是闭环强化学习（RL）优化。每个阶段都针对不同的数据特性进行优化。

关键创新：CosFly-VLA的核心创新在于其空间感知能力和结构化预测接口，使得模型能够在遮挡情况下进行有效的目标追踪。这与现有方法的单一视觉输入处理方式形成了明显对比。

关键设计：模型采用了多头热身策略和两阶段课程学习，结合自然和长遮挡数据进行微调。此外，闭环强化学习阶段通过多组件奖励机制优化追踪行为，涵盖了追踪距离、定位质量、避免碰撞和任务成功等多个方面。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在实验中，CosFly-VLA相较于OpenVLA在开放环节平均位移误差（ADE）上分别降低了34.1%和35.3%，而闭环优化阶段成功率（SR）提高了29.8%和2.5%。这些结果表明该模型在动态目标追踪中的显著性能提升，尤其是在复杂遮挡情况下的有效性。

🎯 应用场景

CosFly-VLA模型在无人机动态目标追踪中具有广泛的应用潜力，尤其是在复杂城市环境中。该技术可以用于智能交通监控、无人机配送、搜索与救援等领域，提升无人机在动态环境中的自主决策能力和任务执行效率。未来，该模型的进一步优化和推广将可能推动无人机技术的广泛应用。

📄 摘要（原文）

Dynamic target tracking is essential for Unmanned Aerial Vehicles (UAVs) operating in complex urban environments, where both the target and the camera viewpoint change continuously. Existing Vision-Language-Action (VLA) policies can track visible targets effectively, but their performance often degrades when buildings, vegetation, or roadside objects block the line of sight. During sustained occlusion, a policy may lose the target state, execute actions toward an incorrect region, and amplify this error through subsequent observations until re-acquisition becomes impossible. To this end, we present CosFly-VLA, a spatially aware VLA model that jointly grounds the target, estimates its visibility, and generates continuous flight actions through a structured prediction interface. To train this policy, we use a large-scale recipe over diverse data sources. Spatially Grounded Continued Pretraining (CPT) on a 500k mixed pool injects UAV-view depth, distance, and 3-D spatial reasoning. A three-stage Curriculum-based Supervised Fine-Tuning (SFT) process then specializes the tracker through multi-head warm-up followed by two-stage curriculum learning over natural and hard / long-occlusion data. Chain-of-Thought (CoT) training subsequently teaches recovery-oriented reasoning traces before structured answers. Finally, a closed-loop Reinforcement Learning (RL) stage optimizes tracking behavior with a multi-component reward covering stand-off tracking, grounding quality, collision avoidance, and task success. Relative to OpenVLA, CosFly-VLA-0.8B reduces open-loop Average Displacement Error (ADE) by 34.1% on seen-test and 35.3% on unseen-test. Closed-loop optimization improves Success Rate (SR) by 29.8% and 2.5%, respectively. These results demonstrate progress from visible-frame imitation toward spatially grounded action-closed-loop control, evaluated under a shared oracle state history.