Graded-Relevance Composed Multimodal Retrieval for E-commerce Visual Search at Scale

📄 arXiv: 2609.24152v1 📥 PDF

作者: Anubhav Gupta, Hrushikesh Mohapatra, Prijith Chandra, Asish Mohapatra, Anuj Garg, Arvind Maan, Sudip Datta, Venkat Bulusu, Sitesh Kumar Jalan

分类: cs.IR, cs.AI, cs.CV

发布日期: 2026-09-21

💡 一句话要点

提出GradCIR以解决电商视觉搜索中的分级相关性问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 复合图像检索 分级相关性 视觉搜索 电商 多模态检索 深度学习 用户体验

📋 核心要点

  1. 现有复合图像检索方法将相关性视为二元，无法满足电商目录中用户查询的复杂性和多样性。
  2. 本文提出GradCIR方法，通过视觉语言模型生成分级相关性标签，改进了复合图像检索的训练过程。
  3. 在FashionIQ基准测试中，GradCIR在平均召回率上达到了0.6703，超越了现有的监督基线，提升幅度显著。

📝 摘要（中文）

在大型电商目录中，视觉搜索需处理相似性查询和由图像与文本描述的修饰符查询。现有的复合图像检索方法将相关性视为二元，无法有效处理用户查询的部分匹配情况。本文提出了一种基于分级相关性的复合图像检索训练方法GradCIR，利用视觉语言模型生成查询和四级相关性标签，并通过迭代反馈机制扩展训练集。实验结果表明，GradCIR在多个基准上显著提升了检索性能，并已在Walmart的生产环境中部署。

🔬 方法详解

问题定义：本文旨在解决电商视觉搜索中的复合图像检索（CIR）问题，现有方法将相关性视为二元，无法有效处理用户查询的部分匹配情况，导致用户体验不佳。

核心思路：提出GradCIR方法，通过视觉语言模型（VLM）生成查询和四级相关性标签，利用迭代反馈机制扩展训练集，从而实现对分级相关性的有效学习。

技术框架：GradCIR的整体架构包括三个主要模块：1) VLM用于生成训练数据和相关性标签；2) 迭代反馈机制用于挖掘困难负样本；3) 层次感知的角度目标用于直接训练检索器。

关键创新：GradCIR的创新在于将相关性视为分级而非二元，直接在四级标签上进行训练，从而更好地反映用户查询的复杂性。

关键设计：在训练过程中，使用了4级相关性标签，设计了层次感知的损失函数，并采用了PaliGemma2双编码器架构，训练数据来自于350万对标注的Walmart目录数据。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在控制实验中，GradCIR相较于传统的二元监督方法在NDCG@10上提升了4.9%-5.9%。在FashionIQ基准测试中，GradCIR达到了0.6703的平均召回率，超越了最强的监督基线，并在多个多模态编码器上实现了最高8.5%的NDCG@10提升。

🎯 应用场景

GradCIR方法在电商视觉搜索中具有广泛的应用潜力，能够提升用户的搜索体验和满意度。通过更精确的检索结果，电商平台可以提高转化率和客户留存率，未来可扩展至其他领域的多模态检索任务。

📄 摘要（原文）

Visual search on large e-commerce catalogs must serve both "similarity" queries that ask for items resembling an uploaded image and "modifier" queries that comprise an image and text describing a desired modification (e.g. a color change or style swap). The latter is the setting known as composed image retrieval (CIR). Existing CIR methods, however, treat relevance as binary and train on triplets with a single positive target - a poor fit for real catalogs where many candidates partially satisfy a user query and ranking across that partial-match spectrum drives the customer experience. We propose a methodology for training CIR retrievers on graded relevance, consisting of: (i) a VLM to curate training data, generating both queries (object detection + modifier synthesis) and 4-level relevance labels without manual annotation, (ii) an iterative relevance-feedback loop that expands the training set by mining hard negatives from the in-training retriever, and (iii) a hierarchy-aware angular objective to train the retriever directly on the graded labels rather than collapsing them to a binary split. We call this methodology GradCIR and instantiate it on a PaliGemma2 bi-encoder trained on 3.5M graded pairs curated from raw Walmart catalog data. A controlled graded-vs-binary ablation isolates the supervision granularity and shows lift of 4.9%-5.9% in NDCG@10. The same recipe applied to other multimodal encoders lifts early-fusion backbones by up to 8.5% NDCG@10. On the public FashionIQ benchmark, GradCIR (applied to PaliGemma2) reaches 0.6703 average recall when fine-tuned, slightly ahead of the strongest peer-reviewed supervised baseline we compare against, and matching or exceeding all published CLIP-L-class zero-shot CIR methods. The system is deployed in production at Walmart, where it's serving live visual-search user traffic.