Bidirectional Multimodal Fusion of Sky Images and Time-Series for Solar Forecasting with Large Language Models

📄 arXiv: 2609.11135v1 📥 PDF

作者: Ken Chen, Maneesha Perera, Wei Wang, Sachith Seneviratne, Hansani Weeratunge, Saman Halgamuge

分类: cs.LG

发布日期: 2026-09-10

💡 一句话要点

提出SolCloudLLM以解决短期光伏发电预测中的多模态融合问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 光伏发电 多模态融合 时间序列预测 大型语言模型 云层预测 智能电网 可再生能源

📋 核心要点

  1. 现有的光伏发电预测方法主要依赖历史数值数据，难以有效应对云层变化带来的预测误差。
  2. 本文提出的SolCloudLLM框架通过双向多模态融合，将天空图像与时间序列数据进行有效结合，提升预测准确性。
  3. 在SIRTA和SKIPP'D数据集上的实验表明，SolCloudLLM在所有预测时间范围内均表现优异，尤其在多云条件下效果显著。

📝 摘要（中文）

短期光伏发电（PV）功率和全球水平辐射（GHI）预测对于有效调度和电网操作至关重要。现有方法主要依赖历史数值数据，难以准确预测云层变化，导致预测误差。本文提出SolCloudLLM，一个基于大型语言模型（LLM）的多模态预测框架，通过双向多模态融合将天空图像与时间序列数据结合，显著提高了预测性能。实验结果表明，SolCloudLLM在所有预测时间范围内均优于最佳基线方法，最大相对均方误差（MSE）降低达25.4%。

🔬 方法详解

问题定义：本文旨在解决短期光伏发电预测中的云层变化导致的预测误差问题。现有方法主要依赖历史数值数据，难以捕捉即将到来的云层变化，导致预测不准确。

核心思路：SolCloudLLM框架通过双向多模态融合，将天空图像与时间序列数据进行对齐和融合，生成统一的表示，从而提升预测性能。这样的设计使得模型能够充分利用视觉信息，改善对云层变化的响应。

技术框架：该框架包括两个主要模块：首先是对天空图像和时间序列数据进行特征提取和对齐，其次是通过双向融合机制生成统一的表示，最后将该表示映射到LLM的嵌入空间中进行预测。

关键创新：最重要的创新在于引入了双向多模态融合机制，使得天空图像和时间序列数据能够相互补充，克服了传统单模态方法的局限性。

关键设计：在模型设计中，采用了特定的损失函数来优化多模态融合效果，并在网络结构中引入了注意力机制，以增强模型对重要特征的关注。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

实验结果显示，SolCloudLLM在SIRTA和SKIPP'D数据集上的均方误差（MSE）显著低于最佳基线方法，最大相对MSE降低达25.4%。此外，在多云条件下，模型的性能提升尤为明显，展示了多模态融合的优势。

🎯 应用场景

该研究的潜在应用领域包括电力调度、可再生能源管理和智能电网优化。通过提高光伏发电的预测准确性，能够有效降低电网运营成本，提升可再生能源的利用效率，推动可持续能源的发展。未来，该方法还可扩展到其他气象预测和环境监测领域。

📄 摘要（原文）

Short-term photovoltaic (PV) power and global horizontal irradiance (GHI) forecasts are essential for effective dispatch, reserve scheduling, and grid operations. At these forecasting horizons, errors are predominantly driven by cloud induced ramps: relying solely on historical numerical data may struggle to anticipate an incoming cloud, making ground-based sky images a crucial complementary physical signal. Furthermore, forecast performance is highly sensitive to location and local observing conditions, creating a strong need for site-specific data that are often scarce. Recently, large language models (LLMs) have demonstrated competitive performance and high data efficiency in time-series forecasting. Despite their success, existing LLM-based forecasting methods remain predominantly unimodal, relying primarily on historical numerical time-series data. Effectively incorporating sky imagery into an LLM-based forecasting framework remains under-explored and an open challenge. In this paper, we propose SolCloudLLM, an LLM-based multimodal forecasting framework. SolCloudLLM aligns sky-image patches with time-series patches and fuses their corresponding representations through bidirectional multimodal fusion, yielding a unified representation that is subsequently mapped into the embedding space of an LLM. Extensive experiments on the SIRTA and SKIPP'D datasets demonstrate that SolCloudLLM consistently outperforms the best baseline methods in MSE across all forecasting horizons, achieving a maximum relative MSE reduction of 25.4%. Stratified analysis further indicates that the benefits of multimodal fusion are concentrated primarily under cloudy conditions. Notably, SolCloudLLM achieves the best performance in nearly all few-shot settings, whereas other deep learning baselines experience substantial performance degradation and are frequently outperformed by the non-learning physical method.