VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligence
作者: Jianing Qiu, Jian Wu, Hao Wei, Peilun Shi, Minqing Zhang, Yunyun Sun, Lin Li, Hanruo Liu, Hongyi Liu, Simeng Hou, Yuyang Zhao, Xuehui Shi, Junfang Xian, Xiaoxia Qu, Sirui Zhu, Lijie Pan, Xiaoniao Chen, Xiaojia Zhang, Shuai Jiang, Kebing Wang, Chenlong Yang, Mingqiang Chen, Sujie Fan, Jianhua Hu, Aiguo Lv, Hui Miao, Li Guo, Shujun Zhang, Cheng Pei, Xiaojuan Fan, Jianqin Lei, Ting Wei, Junguo Duan, Chun Liu, Xiaobo Xia, Siqi Xiong, Junhong Li, Benny Lo, Yih Chung Tham, Tien Yin Wong, Ningli Wang, Wu Yuan
分类: eess.IV, cs.CV
发布日期: 2023-10-08
期刊: The latest VisionFM work has been published in NEJM AI, 2024
DOI: 10.1056/AIoa2300221
💡 一句话要点
提出VisionFM以解决眼科人工智能多任务问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 眼科人工智能 多模态学习 基础模型 疾病诊断 图像处理 合成数据 深度学习
📋 核心要点
- 现有眼科人工智能方法在多任务处理和疾病诊断准确性方面存在不足，难以满足临床需求。
- VisionFM通过预训练340万眼科图像，构建多模态多任务的基础模型，提升眼科AI应用的智能和准确性。
- 实验结果显示，VisionFM在多个基准测试中超越传统深度学习模型，展现出显著的性能提升和良好的泛化能力。
📝 摘要（中文）
我们提出了VisionFM，一个基础模型，使用来自560,457个个体的340万眼科图像进行预训练，涵盖广泛的眼科疾病、成像设备和人群特征。VisionFM为多种眼科人工智能应用提供基础，包括疾病筛查与诊断、疾病预后、疾病表型的子分类以及系统生物标志物和疾病预测。该模型在共同诊断12种常见眼科疾病时超越了基础和中级水平的眼科医生，并在新的大规模眼科疾病诊断基准数据库及分割检测基准数据库上表现优于强基线深度神经网络。VisionFM学习的眼科图像表示具有良好的可解释性，并展现出对新眼科成像模式、疾病谱和成像设备的强泛化能力。
🔬 方法详解
问题定义：本论文旨在解决现有眼科人工智能模型在多任务处理和疾病诊断准确性方面的不足，现有方法往往无法有效整合多种眼科数据和任务。
核心思路：VisionFM的核心思路是通过大规模预训练，利用多模态眼科图像数据，构建一个通用的基础模型，以支持多种眼科AI应用。这样的设计能够提升模型的智能水平和准确性。
技术框架：VisionFM的整体架构包括数据预处理、模型预训练、任务适配和评估四个主要模块。首先，收集和处理多种眼科图像数据，然后进行大规模预训练，最后针对特定任务进行微调和评估。
关键创新：VisionFM的主要创新在于其大规模的多模态预训练和合成数据的使用，这使得模型能够在多种眼科任务中表现出色，超越了传统方法的局限性。
关键设计：在模型设计中，采用了先进的卷积神经网络架构，结合多任务学习的损失函数，确保模型在多个任务上均能获得良好的表现。此外，合成数据的引入显著提升了模型的表示学习能力。
🖼️ 关键图片
📊 实验亮点
实验结果表明，VisionFM在共同诊断12种常见眼科疾病时超越了基础和中级水平的眼科医生，并在新的大规模眼科疾病诊断和分割检测基准数据库上表现优于强基线深度神经网络，显示出显著的性能提升。
🎯 应用场景
VisionFM在眼科人工智能领域具有广泛的应用潜力，可以用于疾病筛查、诊断、预后评估及生物标志物预测等多个场景。其高准确性和良好的泛化能力使其在临床实践中具有重要的实际价值，未来可为眼科医疗提供更高效、经济的解决方案。
📄 摘要（原文）
We present VisionFM, a foundation model pre-trained with 3.4 million ophthalmic images from 560,457 individuals, covering a broad range of ophthalmic diseases, modalities, imaging devices, and demography. After pre-training, VisionFM provides a foundation to foster multiple ophthalmic artificial intelligence (AI) applications, such as disease screening and diagnosis, disease prognosis, subclassification of disease phenotype, and systemic biomarker and disease prediction, with each application enhanced with expert-level intelligence and accuracy. The generalist intelligence of VisionFM outperformed ophthalmologists with basic and intermediate levels in jointly diagnosing 12 common ophthalmic diseases. Evaluated on a new large-scale ophthalmic disease diagnosis benchmark database, as well as a new large-scale segmentation and detection benchmark database, VisionFM outperformed strong baseline deep neural networks. The ophthalmic image representations learned by VisionFM exhibited noteworthy explainability, and demonstrated strong generalizability to new ophthalmic modalities, disease spectrum, and imaging devices. As a foundation model, VisionFM has a large capacity to learn from diverse ophthalmic imaging data and disparate datasets. To be commensurate with this capacity, in addition to the real data used for pre-training, we also generated and leveraged synthetic ophthalmic imaging data. Experimental results revealed that synthetic data that passed visual Turing tests, can also enhance the representation learning capability of VisionFM, leading to substantial performance gains on downstream ophthalmic AI tasks. Beyond the ophthalmic AI applications developed, validated, and demonstrated in this work, substantial further applications can be achieved in an efficient and cost-effective manner using VisionFM as the foundation.