A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out
作者: Mahdi Naser Moghadasi, Faezeh Ghaderi
分类: cs.LG
发布日期: 2026-09-09
备注: 11 pages, 2 figures, 5 tables. Code, data fetchers and per-series results: https://github.com/mahdinaser/tsfm-bench
💡 一句话要点
提出时间序列模型评估新方法以解决预训练数据影响问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 时间序列预测 模型评估 预训练模型 领域熟悉度 数据集构建 金融预测 社交媒体分析
📋 核心要点
- 现有时间序列模型评估方法无法有效区分模型的强评分与其在预训练阶段对测试集的接触。
- 本文提出了一种新的评估协议，通过构建时间上后置的测试集来检验模型的真实性能。
- 实验结果表明，预训练模型在特定领域的熟悉度显著影响预测效果，且在某些情况下优于传统模型。
📝 摘要（中文）
时间序列基础模型的评估几乎完全依赖于其发布前的公共档案，因此强评分无法与测试集的预训练分离。为此，本文构建了一个新的评估协议，使用了13个预测模型在5个领域的7个组上进行测试。结果显示，预训练模型在7个组中赢得了5个，且在每日汇率预测中与季节性简单预测无显著差异。研究发现，模型的胜负与其对数据集的熟悉度密切相关，而与季节强度和谱熵无关。最终得出结论，时间上的保留可以消除对窗口的记忆，但无法消除对领域的熟悉，建议基准测试应明确与已披露语料库相关的领域保留。
🔬 方法详解
问题定义：本文旨在解决时间序列模型评估中，如何有效区分模型性能与预训练数据影响的问题。现有方法往往无法避免模型在预训练阶段接触测试集，导致评估结果失真。
核心思路：论文提出通过构建一个时间上后置的测试集来评估模型性能，确保测试数据在模型发布后才被使用，从而消除预训练对测试结果的影响。
技术框架：整体架构包括数据集构建、模型训练与评估三个主要阶段。首先，构建一个包含多个领域的后置测试集；其次，训练多种预测模型；最后，评估模型在新测试集上的表现。
关键创新：最重要的创新在于引入了时间上的保留策略，强调模型对领域的熟悉度在预测性能中的重要性，与传统方法相比，提供了更为真实的性能评估。
关键设计：在模型训练中，采用了多种预测模型，包括经典模型和预训练模型，且在评估时使用了均方根误差（RMSE）等指标来衡量模型性能。
🖼️ 关键图片
📊 实验亮点
实验结果显示，预训练模型在7个组中赢得了5个，尤其是在每周维基百科页面浏览量预测中，表现出28%的MASE降低，相较于最佳经典方法。此外，TimesFM模型在维基百科数据集上的表现优于Chronos模型，显示出领域熟悉度的重要性。
🎯 应用场景
该研究的潜在应用领域包括金融市场预测、气象预测和社交媒体数据分析等。通过提供更准确的模型评估方法，能够帮助研究人员和从业者选择适合其特定领域的模型，从而提升预测的准确性和可靠性。
📄 摘要（原文）
Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining. The obvious remedy is a hold-out that postdates the models. We build one: thirteen forecasters -- four classical, three trained per dataset, six pretrained -- on seven groups drawn from five domains, every observation published after the last model was released, and every dataset rebuildable without an API key. Under this protocol pretrained models win 5 of 7 groups, lose one to a Theta baseline, and on daily exchange rates are indistinguishable from a seasonal naive forecast, along with every other method tested. We then ask what separates the wins from the losses, and report a negative result: the two intrinsic properties one would reach for -- seasonal strength and spectral entropy, measured on the input window -- do not account for the pattern, and seasonal strength is if anything negatively associated with the advantage. What does track it is corpus familiarity. Our largest gain (28% lower MASE than the best classical method, on weekly Wikipedia pageviews) falls on Wikipedia pageviews, the domain TimesFM's authors describe as the bulk of its pretraining corpus, at the same granularities and differing only in time window. Within the pretrained family, where every model forecasts identical series so that series difficulty cancels, the TimesFM family outranks the Chronos family by -0.53 ranks on Wikipedia against -0.09 everywhere else (1,500 vs. 754 series, Mann-Whitney p < 1e-5). We conclude that a temporal hold-out removes memorisation of a window but not familiarity with a domain, that benchmarks therefore need domain hold-outs stated relative to disclosed corpora, and that the practitioner's question is less which model is better than whether their domain is one the model was raised on.