Deep Learning of Sequential Data with LSTM

This brief article presents a method for deep learning of sequential data with LSTM through the use of Keras.

“Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction,” said Yann LeCun, Yoshua Bengio and Geoffrey Hinton in their paper, ‘Deep Learning’ (https://www.nature.com/articles/nature14539).

Deep learning in sequences of data
Supervised learning with sequences is different when compared to other types of data. Here, we have to consider the sequence and its behaviour. We need to find the correct set of data for each sequence to create and train models. Shown below are the different types of sequencing patterns.

1) Sequence predictions: This is the prediction of the next sequence, based on the sequence model.
2) Sequence classification: This classifies a sequence based on the sequence model.
3) Sequence generation: This generates a new sequence based on the sequence pattern.
Here, we are limiting our focus only to sequence classification — we are going to analyse different sets of patterns and tag them into specific types. Based on this trained model, we will see what the sequence for the test sequence data is. Before getting into implementation, let us look at how we can approach this problem. Based on my analysis on sequence prediction, long short term memory (LSTM) will work better for the sequence classification problem.

Long short term memory (LSTM)
Traditional neural networks cannot remember more of the past, but only the recent past (short term memory) and this is considered as a shortcoming of these networks. Recurring neural networks (RNN) can fix this problem (more on this at http://karpathy.github.io/2015/05/21/rnn-effectiveness/). LSTM is actually a special kind of RNN which is capable of learning long term dependencies. To solve sequencing classification and prediction modelling, we actually need the neural network to remember the past sequence so that we can classify or predict the sequence. I would recommend reading more about RNN and LSTM if you don’t know much about these topics. The section below explains how we can implement LSTM and solve the sequence classification problem.

LSTM implementation by using Keras: The problem
This problem is from Jason Brownlee’s blog and it is about the classification of sentiments expressed in IMDB movie reviews, based on the comments.
The data set can be downloaded from http://ai.stanford.edu/~amaas/data/sentiment/imdb. The load_data() function of Keras gives access to the IMDB data set to us directly.
The detailed explanation and code of LSTM implementation with Keras is given at https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/