多元化感知元视觉提示

我们提出了一种称为“Diversity-Aware Meta Visual Prompting（DAM-VP）”的高效有效的提示方法，用于将预训练模型转移到具有冻结主干的下游任务。视觉提示中的一个挑战性问题是，图像数据集有时具有大量的数据多样性，而基于每个数据集的通用提示无法适当地处理复杂的分布偏移，对应着原始的预训练数据分布。为了解决这个问题，我们提出了一种基于数据集多样性的提示策略，其初始化是通过元提示实现的。具体来说，我们以多样性自适应的方式将下游数据集聚类成小的同质子集，每个子集都有自己的单独优化提示。这种分而治之的设计大大降低了优化难度，并显著提高了提示性能。此外，所有提示都以元提示为初始值，元提示学习自多个数据集。这是一种引导式范例，其核心观察是从先前数据集中学习的提示知识可以帮助提示更快地收敛并在新数据集上表现更好。在推断过程中，我们根据输入与每个子集之间的特征距离动态选择合适的提示。通过大量实验，我们的DAM-VP表现出卓越的效率和有效性，明显超过了不同预训练模型的一系列下游数据集中之前的提示方法。我们的代码可在以下链接处获得：https://github.com/shikiw/DAM-VP。
We present Diversity-Aware Meta Visual Prompting~(DAM-VP), an efficient and
effective prompting method for transferring pre-trained models to downstream
tasks with frozen backbone. A challenging issue in visual prompting is that
image datasets sometimes have a large data diversity whereas a per-dataset
generic prompt can hardly handle the complex distribution shift toward the
original pretraining data distribution properly. To address this issue, we
propose a dataset Diversity-Aware prompting strategy whose initialization is
realized by a Meta-prompt. Specifically, we cluster the downstream dataset into
small homogeneity subsets in a diversity-adaptive way, with each subset has its
own prompt optimized separately. Such a divide-and-conquer design reduces the
optimization difficulty greatly and significantly boosts the prompting
performance. Furthermore, all the prompts are initialized with a meta-prompt,
which is learned across several datasets. It is a bootstrapped paradigm, with
the key observation that the prompting knowledge learned from previous datasets
could help the prompt to converge faster and perform better on a new dataset.
During inference, we dynamically select a proper prompt for each input, based
on the feature distance between the input and each subset. Through extensive
experiments, our DAM-VP demonstrates superior efficiency and effectiveness,
clearly surpassing previous prompting methods in a series of downstream
datasets for different pretraining models. Our code is available at:
\url{https://github.com/shikiw/DAM-VP}.
论文链接：http://arxiv.org/pdf/2303.08138v1