Let’s say that we want to do name entity recognition. Following two examples are included in the training set; ‘He said, “Teddy bears are on sale!”’, and ‘He said, “Teddy Roosevelt was a great President!”’.
If the information flows only from left to right, our model has no way to learn that “Teddy” in the second example is a name.
To solve this kind of problem, bidirectional RNN (BRNN) is introduced.
In BRNN, there are two forward propagations; left to right and right to left. For each layer we combine activations from these two forward props to output prediction.
For each sequential layer, output is computed as following;
In natural language processing, BRNN with LSTM is the most standard model.
We can make the RNN model more complex by adding layers vertically.