Visualizations of RNN units

Diagrams of RNN unrolling, LSTM and GRU.

Posted by Jakub Kvita on April 14, 2016

I am currently writing my thesis and, as a part of my text, I created some very nice vector diagrams I would like to share. To give credit, I was inspired by Christopher Olah’s Understanding LSTM Networks post, but I wanted to make vector images of those, as PNG diagrams don’t look very professional in PDFs. If you are not sure about the topic of this post, please visit the mentioned Christopher’s page to get the basic knowledge about RNNs and LSTMs.

First of all, I created a visualization of RNN unrolling (svg file).

RNN Unrolling image RNN as a neural network very deep in time.

Then, LSTM with peephole connections (svg file). Peepholes can be easily removed.

A LSTM image LSTM with peephole connections.

And finally GRU (svg file).

A GRU image Gated recurrent unit.

I spent some time creating those diagrams, so feel free to save time and reuse them. If you will, send me a message, I will be glad somebody found this useful.