Cost-Aware Abstention for LLM-Based IT Ticket Classification

📄 arXiv: 2610.31657v1 📥 PDF

作者: Sachin Gupta

分类: cs.LG, cs.AI, cs.CL

发布日期: 2026-10-22

备注: Accepted at the 38th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 2026). 9 pages, 5 figures, 9 tables

💡 一句话要点

提出成本感知的放弃策略以优化IT工单分类

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: IT工单分类 成本感知 放弃策略 机器学习 大语言模型 错误路由 逻辑回归 TF-IDF

📋 核心要点

  1. 现有的IT工单自动路由方法可能导致高昂的错误成本，尤其是在复杂的分类任务中。
  2. 本文提出了一种基于成本的放弃策略，通过在校准数据上选择路由策略来优化分类性能。
  3. 实验结果显示，所提方法在多个设置下显著降低了测试成本和错误路由率，验证了其有效性。

📝 摘要（中文）

错误的IT支持工单自动路由可能导致的成本超过人工分流的费用。本文评估了一种成本感知的放弃策略，作为一种离线的、基于成本的准备评估方法：在校准数据上选择路由策略，并在持出测试数据上评估该固定策略。在一个包含47,837个工单和八个类别的公共数据集上，比较了四种训练的非LLM分类器与四种大型语言模型在零-shot和few-shot提示下的表现。在给定的成本模型下，TF-IDF与逻辑回归结合阈值0.60的选择使测试成本从0.763降低至0.472，减少了38.1%的相对成本，同时错误路由率从15.3%降至3.5%。

🔬 方法详解

问题定义：本文旨在解决IT工单分类中的错误路由问题，现有方法在处理复杂分类时可能导致高成本的错误。

核心思路：提出了一种成本感知的放弃策略，通过在校准数据上选择路由策略，以降低错误路由的风险和相关成本。

技术框架：整体流程包括数据预处理、策略选择、模型评估和成本分析，主要模块包括校准数据集、路由策略选择和测试数据评估。

关键创新：最重要的创新在于引入了成本感知的放弃策略，能够在不同的成本条件下优化路由决策，与传统方法相比更具灵活性和适应性。

关键设计：使用TF-IDF与逻辑回归结合的模型，设置阈值为0.60，并在成本模型(c_ok, c_triage, c_wrong) = (0, 1, 5)下进行评估，确保了策略选择的有效性。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

实验结果表明，采用TF-IDF与逻辑回归的策略在校准后测试中，成本从0.763降低至0.472，减少了38.1%的相对成本，错误路由率从15.3%降至3.5%。该方法在14个设置中优于始终路由和始终放弃的策略，显示出其有效性。

🎯 应用场景

该研究的潜在应用领域包括IT支持系统、客户服务自动化和其他需要高效分类的场景。通过优化工单分类，能够显著降低企业运营成本，提高服务效率，未来可能推动更多基于成本的智能决策系统的开发。

📄 摘要（原文）

Wrong automatic routing of an IT support ticket can cost more than human triage. We evaluate cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data. On a public 47,837-ticket, eight-class corpus, we compare four trained non-LLM classifiers with four large language models under zero- and few-shot prompting. Under the illustrative cost model (c_ok, c_triage, c_wrong) = (0, 1, 5), TF-IDF with logistic regression and a threshold of 0.60 selected on calibration reduces test cost from 0.763 to 0.472, a 38.1% relative reduction (paired ratio-bootstrap 95% CI [35.7%, 40.4%]), while lowering wrong routes from 15.3% to 3.5%. Frozen calibration-selected policies also beat both always-route and always-defer on held-out test data in 14 of 16 settings in a post-hoc cost-parameter analysis. The strongest prompted configuration reaches 0.477 accuracy on a stratified 1,000-ticket test subset, versus 0.879 for fine-tuned DistilBERT on the same tickets. Isotonic lowers expected calibration error without reliably improving discrimination: GPT-4o-mini few-shot has one confidence value before and after recalibration and correctness AUROC 0.500. We therefore report correctness AUROC and tie-aware risk-coverage area alongside ECE. At the analytically derived break-even probability 0.80, seven prompted configurations defer all tickets; Haiku few-shot routes only 19/1,000. An LLM-only agreement control and margin/entropy rejection baselines further separate calibration from discrimination. The findings are scoped to this dataset, prompt design, model endpoints, and illustrative costs; the work is an evaluation framework, not a new abstention algorithm or universal deployment certification.