Classification-Regression for Chart Comprehension

Figures and charts play a major role in modern communication, help to convey messages by curating data into an easily comprehensible visual form, highlighting the trends and outliers.

The Chart Question Answering (CQA) task is closely related to Visual Question Answering (VQA), which is usually applied on natural images. VQA is typically treated as a classification task, where the answer is a category. In contrast, answering questions about charts often requires regression. Furthermore, a small local change in a natural image typically has limited effect on the visual recognition outcome, while in a chart, the impact might be extensive. A chart comprehension model must consider the interactions between the question and the various chart elements in order to provide correct answers.

We present an overview of our CRCT architecture for CQA in Fig. 3. In our approach, the image is first parsed by a trained object detector (see object classes in Fig. 2). The output of the parsing stage are object classes, positions (bounding boxes), and visual features. All of the above are projected into a single representation per visual element, then stacked to form the visual sequence.

Similarly, each textual element is represented by fusing its text tokens, positional encoding and class. Together with the question text tokens, we obtain the text sequence.

The two sequences are fed in parallel to a bimodal co-attention-transformer (co-transformer). The output of the co-transformer are pooled visual and textual representations that are then fused by Hadamard product and concatenation, and fed into our unified classification-regression head.