Asking a Multimodal LLM Whether Someone Fell: Zero-Shot Screening of Low-Resolution Thermal Video in a Laboratory and an Occupied Home

📄 arXiv: 2610.29913v1 📥 PDF

作者: Marcos Junca, Mauro Junca

分类: cs.CV, cs.AI

发布日期: 2026-10-21

备注: 13 pages, 6 figures. Data: https://senecta.ai/research/thermal-fall-llm-data.zip

💡 一句话要点

提出多模态LLM以实现低分辨率热视频的跌倒检测

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 跌倒检测 多模态学习 热成像技术 大型语言模型 老年人监护

📋 核心要点

  1. 现有跌倒检测方法在低分辨率热视频中准确性不足，尤其是在复杂环境中。
  2. 本文提出使用多模态大型语言模型（LLM）进行跌倒检测，利用热视频和声音信息进行判断。
  3. 实验结果表明，Gemini 3.8 Flash在未见样本中识别跌倒的敏感性达到69%，并且在家庭环境中未产生误报。

📝 摘要（中文）

热成像摄像头可以在不捕捉面部的情况下监控独居老人，而多模态大型语言模型（LLM）能够在没有特定任务训练的情况下判断短视频。本文评估了Gemini 3.8 Flash作为事件触发跌倒筛查器的决策阶段，使用了438个模拟跌倒和563个非跌倒窗口的数据。结果显示，该模型在未见样本中识别了69%的跌倒事件，并且在普通生活记录中未误报。通过调整提示语，模型的敏感性显著提高，且在家庭环境中成功捕捉到所有可见的跌倒事件。本文还发布了公共数据集的帧级跌倒注释和模型的所有回答。

🔬 方法详解

问题定义：本文旨在解决低分辨率热视频中跌倒检测的准确性问题。现有方法在复杂环境中容易产生误报，且对低分辨率视频的处理能力不足。

核心思路：通过引入多模态大型语言模型（LLM），结合热视频和声音信息，进行跌倒事件的判断。该设计旨在减少对特定任务训练的依赖，提高模型的通用性和适应性。

技术框架：整体架构包括数据采集、模型请求和决策阶段。数据采集使用热成像摄像头记录视频，模型请求则通过Gemini 3.8 Flash和Gemini 3.7 Flash进行处理，最后根据模型输出进行跌倒判断。

关键创新：最重要的技术创新在于使用多模态LLM进行跌倒检测，显著提高了在低分辨率视频中的敏感性和准确性。与传统方法相比，该方法在复杂环境中表现更佳。

关键设计：模型的提示语设计对结果影响显著，调整提示语后敏感性从69%提升至83%。此外，模型在跌倒事件的时间定位上表现优异，99%的时间内准确定位到注释的着陆时间。

🖼️ 关键图片

fig_0

📊 实验亮点

实验结果显示，Gemini 3.8 Flash在未见样本中识别跌倒的敏感性达到69%，在家庭环境中未产生误报。通过调整提示语，敏感性提升至83%，并成功捕捉到所有可见的跌倒事件，显示出该方法在实际应用中的有效性。

🎯 应用场景

该研究的潜在应用领域包括老年人监护、家庭安全监控和医疗健康管理。通过使用热成像技术和多模态LLM，能够在不侵犯隐私的情况下有效监测老年人的安全，减少跌倒事故的发生。未来，该技术有望在智能家居和健康监测设备中得到广泛应用。

📄 摘要（原文）

Thermal cameras can watch over older people living alone without capturing faces, and multimodal large language models (LLMs) can judge a short video without task-specific training. We evaluate a production multimodal LLM, Gemini 3.8 Flash at temperature 0, as the decision stage of an event-triggered fall screener, used exactly as a deployed prototype uses it: one request per short thermal clip, with its sound track when there is one, and its predecessor Gemini 3.7 Flash on the same requests. On 438 staged falls and 563 non-fall windows that we located by hand in a public 32x24 thermal dataset, the deployed request called 69% of the held-out subjects' falls (67% over all subjects; bed falls excluded) and none of 499 non-fall windows, including people lying on the floor and getting up. Over 76 h of ordinary life in an occupied room, recorded at 160x120 with sound, it called none of 395 clips a fall (95% upper bound 1.2 false alarms per day) and caught 4 of the 9 staged falls the camera could see. One cautionary sentence in the prompt ("a person who goes out of view ... is not a fall") accounted for many of the misses: removing it raised sensitivity on the held-out subjects from 69% to 83% on 3.8 Flash and from 80% to 92% on Gemini 3.7 Flash, and in the home from 4 to 9 of the 9 visible falls, with no false call in ordinary living but at the price of calling deliberate lie-downs a fall. Sensitivity collapsed at 8x6 pixels (34%); identical requests disagreed on 4% of windows; and when the model called a fall, its explanation placed it within a second of the annotated landing 99% of the time. Two conventional detectors trained on the laboratory data caught nearly every held-out laboratory fall but raised 14.2 and 23.9 false alarms per day in the home. We release the frame-level fall annotations of the public dataset and every model answer on it.