Line Chart Understanding with Convolutional Neural Network

Abstract

To understand a technical document, you need to understand not only the text, but also the various tables and charts included in the document. To understand a chart, you need to understand the various attributes of the chart and the representation of the data. In order to understand the meaning of a chart based on an image, it is necessary to be able to extract the knowledge of the chart. Previous attempts have been made to extract knowledge from images and use them. Image captioning involves understanding the meaning in an image and generating a language based on that meaning. Because of the wide variety of knowledge graphs that images have, there is a lot of diversity in creating language. Therefore, even if the language is generated from the image, it is difficult to confirm whether the logic is correctly extracted from the image. In this thesis, we define a problem that can be evaluated to obtain the nested knowledge of technical documents. In particular, it focuses on two-dimensional line charts, among other categories of charts. Line charts represent information by connecting a series of data to a line. We define properties that have meaning among the various properties of the line chart image and create an image to include each property in various ways. The generated image defines the relations that can be represented by the line chart to learn the various meanings of the chart. In addition to the relationships that one data group can represent, the definitions of the diagrams represented by the relationships of two different data groups are defined. It also generates randomly generated features with variations in the chart to create unbiased chart images. We train the convolutional neural network model through the previously generated image and compare the results to extract and analyze the factors that improve the performance of understanding the chart and suggest ways to improve performance later.