Bidirectional RNN and Deep RNN

Bidirectional RNN

Let’s say that we want to do name entity recognition. Following two examples are included in the training set; ‘He said, “Teddy bears are on sale!”’, and ‘He said, “Teddy Roosevelt was a great President!”’.

If the information flows only from left to right, our model has no way to learn that “Teddy” in the second example is a name.

To solve this kind of problem, bidirectional RNN (BRNN) is introduced.

In BRNN, there are two forward propagations; left to right and right to left. For each layer we combine activations from these two forward props to output prediction.

For each sequential layer, output is computed as following;

brnn

In natural language processing, BRNN with LSTM is the most standard model.

Deep RNN

We can make the RNN model more complex by adding layers vertically.

Deep RNN

Leave a Comment