Chatting Makes Perfect: Chat-based Image Retrieval

Chatting Makes Perfect: Chat-based Image Retrieval

Image and Text    search    Images

Matan Levy1   Rami Ben-Ari2   Nir Darshan2   Dani Lischinski1
1The Hebrew University of Jerusalem, Israel
2OriginAI, Israel
To appear in Neural Information Processing Systems (NeurIPS) 2023 Page map: [Abstract | The ChatIR task  | Retrieval Examples | Citation]


Chats emerge as an effective user-friendly approach for information retrieval, and are successfully employed in many domains, such as customer service, healthcare, and finance. However, existing image retrieval approaches typically address the case of a single query-to-image round, and the use of chats for image retrieval has been mostly overlooked. In this work, we introduce ChatIR: a chat-based image retrieval system that engages in a conversation with the user to elicit information, in addition to an initial query, in order to clarify the user's search intent. Motivated by the capabilities of today's foundation models, we leverage Large Language Models to generate follow-up questions to an initial image description. These questions form a dialog with the user in order to retrieve the desired image from a large corpus. In this study, we explore the capabilities of such a system tested on a large dataset and reveal that engaging in a dialog yields significant gains in image retrieval. We start by building an evaluation pipeline from an existing manually generated dataset and explore different modules and training strategies for ChatIR. Our comparison includes strong baselines derived from related applications trained with Reinforcement Learning. Our system is capable of retrieving the target image from a pool of 50K images with over 78% success rate after 5 dialogue rounds, compared to 75% when questions are asked by humans, and 64% for a single shot text-to-image retrieval. Extensive evaluations reveal the strong capabilities and examine the limitations of CharIR under different settings.

Retrieval Examples

Chat Image Retrieval (ChatIR)

A schematic view of ChatIR and the system that we propose in this paper is provided in Figure 1. The process starts with a user-provided short description of the desired image, similarly to text-to-image retrieval. However, from this point on, the retrieval system is able to progressively refine the query by actively polling the user for additional information regarding the desired result.

CoIR task

Figure 1: The pipeline consist of two stages: Image Search (IS) and Dialog Building (DB). The IS stage takes as an input the ongoing dialog, composed of image caption and a few rounds of Q&As, in order to find the target image. Note that a dialog of length 0 is solely the image caption, equivalent to Text-to-Image retrieval task. The DB stage provides the follow-up question to the current dialog.

Citation - BibTeX

    title={Chatting Makes Perfect: Chat-based Image Retrieval},
    author={Matan Levy and Rami Ben-Ari and Nir Darshan and Dani Lischinski},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},

This webpage code was adapted from this source code.