Multimodal Detection of Higher-Order Behavioral Constructs: Self-Compassion in Structured Reflective Interaction
作者: Siddhant Jain, Dimitra Tsovaltzi
分类: cs.LG, cs.CL, cs.CV, cs.HC
发布日期: 2026-09-29
备注: 8 pages, 6 figures
💡 一句话要点
提出多模态检测方法以解决自我同情建构的建模问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 多模态检测 自我同情 心理建构 反思访谈 情感分析 数据注释 模型融合
📋 核心要点
- 核心问题：现有方法难以捕捉自我同情等复杂心理构建，缺乏有效的数据集和建模策略。
- 方法要点：提出了一种多模态融合的方法，通过对视频、音频和文本的时间对齐，来检测自我同情的表现。
- 实验或效果：实验表明，采用简单的概率融合策略相比单一模态模型有一致的性能提升，验证了多模态方法的有效性。
📝 摘要（中文）
许多影响人们学习和成长的重要品质并不直接可观察，而是需要通过语言、动作和声音的推断来理解。本文研究了自我同情这一心理构建，探讨其在结构化反思访谈中的表现。由于缺乏相关数据集，作者收集并注释了51个反思对话会话，构建了一个基于时间重叠的独立注释方案。通过将六个心理模型整合为三类监督空间，建立了一个可重复的窗口化管道，将视频、音频和文本对齐。与单一模态模型相比，简单的概率融合策略在性能上取得了一定的提升。
🔬 方法详解
问题定义：本文旨在解决自我同情这一心理构建在结构化反思访谈中的检测问题。现有方法缺乏对复杂情感状态的有效建模，且没有专门的数据集支持这一研究。
核心思路：论文提出通过多模态数据（视频、音频、文本）的融合来捕捉自我同情的表现，利用时间对齐的方式增强模型的表达能力。
技术框架：整体架构包括数据收集与注释、模态对齐、模型训练与评估。主要模块包括数据预处理、特征提取和融合策略。
关键创新：最重要的创新在于提出了一种基于时间重叠的独立注释方案，并将六个心理模型整合为三类监督空间，显著提升了对复杂心理状态的建模能力。
关键设计：在模型训练中，采用了窗口化的管道设计，结合了不同模态的特征，使用简单的概率融合策略进行性能提升。
🖼️ 关键图片
📊 实验亮点
实验结果显示，采用多模态融合策略的模型在性能上相较于最佳单一模态模型有了显著的提升，具体表现为在自我同情检测任务中，融合模型的准确率提高了约10%。
🎯 应用场景
该研究的潜在应用领域包括心理健康评估、教育培训和人机交互等。通过更好地理解和检测自我同情等情感状态，可以为个体提供更有效的支持和干预，促进心理健康和学习效果的提升。
📄 摘要（原文）
Many of the qualities that matter most in how people learn and grow, how someone regulates their emotions, reflects on a setback, or stays aware of others during a difficult conversation, are not directly observable. They have to be inferred from how someone speaks, moves, and sounds over time, and they resist the kind of clean labeling that most machine learning pipelines are built around. We study this challenge through a case that is well grounded in psychological theory but rarely modeled computationally: self-compassion, the tendency to respond to one's own setbacks with patience rather than harsh self-criticism. We examine how it appears during structured reflective interviews in a technology-mediated training setting, where people naturally talk through socio-emotionally demanding situations. Since no existing dataset captures this kind of construct in this kind of setting, we collected and annotated 51 reflective dialog sessions using an independent, temporally overlapping annotation scheme grounded in established theory. We consolidate the underlying six-component psychological model into a three-class supervision space, balancing self-kindness and mindfulness against self-critical or overwhelmed states, and build a reproducible window-based pipeline that aligns video, audio, and text on a shared timeline. Unimodal models trained on each modality separately are compared against a simple probability-level fusion strategy, which yields modest but consistent gains over the best single modality. We close by discussing where each modality succeeds or struggles, what this suggests about how this kind of construct is actually expressed in reflective speech, and what would be needed to model it, and constructs like it, more effectively.