RNN Architecture

RNN Architecture
- Synopsis
- History
  - Hidden State
  - Attention + RNN

Synopsis

History

RNN was designed during 2010??
The attention mechanism on RNN was designed on 2014. i.e. Decoder can access different parts of input sequence apart from hidden state.

Hidden State

  Encoder consumes all words first before generating:

  -> word1 -> word2 ... -> word-n
      h1       h2            hn

  Decoder generates one word at a time updating hidden states:

  -> outWord1 -> outWord2       ...  -> outWordFinal
       hn          h (updated)

Attention + RNN

Decoder can access entire input words during each step.
It can recognize relative strengths of input words wrt current output word