GLSTM-based Deep learning Models for Text Sentiment Prediction

Abstract

Sentiment prediction is a famous natural language processing (NLP) task that predicts emotional intensities or categories for a given sentence. The tasks seem to be simple, but scores of them are relatively low compared to other NLP tasks, such as part-of-speech (POS) tagging or question answering (QA). In this thesis, we describe two main approaches to improve a performance of the sentiment prediction. One approach is text pre-processing. The source domains of sentiment prediction tasks are usually social media such as Twitter, which is the informal and noisy environment for machine learning. Thus, it is important to handle the characteristics of the domain. The other approach is improving prediction models. We suggested better models based on long short-term memory (LSTM).

First, we improved a text pre-processor to fit our datasets. The previous pre-processor contained tweet tokenizer, word correction, addition of a special token and so on. We modified details of this preprocessor; for example, we analyzed infrequent words that could be normalized and handled them as exception cases. Also, we added a new function to process emoticon and emoji symbols.

For a body model to predict, we suggested a variation of LSTM and dropout. We divided one LSTM into small LSTMs and tied them. For dropout, neurons in layers are grouped and dropout is applied on the group level. They are called GLSTM and gdropout, respectively. Using them, we made deep learning models based on Bi-directional LSTM and attention.

Finally, we tested our models on three tasks and analyzed them. By comparing our models to other basic models, we have proven that our model is more robust to overfitting and shows higher performances. We also added visualization of neurons on deep learning model, helping us to understand the model and effects of pre-processing.