Attention and Memory in Deep Learning and NLP – WildML

A recent trend in Deep Learning are Attention Mechanisms.

Attention Mechanisms in Neural Networks are loosely based on the visual attention mechanism found in humans.

If we look a bit more look closely at the equation for attention we can see that attention comes at a cost.

An alternative approach to attention is to use Reinforcement Learning to predict an approximate location to focus to.

Interpreted another way, the attention mechanism is simply giving the network access to its internal memory, which is the hidden state of the encoder.

When the networks parameter weights are tied in a certain way, the memory mechanism inEnd-to-End Memory Networks identical to the attention mechanism presented here, only that it makes multiple hops over the memory.

It’s likely that in the future we will see a clearer distinction between memory and attention mechanisms, perhaps along the lines of Reinforcement Learning Neural Turing Machines, which try to learn access patterns to deal with external interfaces.

This article was summarized automatically with AI / Article-Σ ™/ BuildR BOT™.

Original link

Sharing is caring!

Leave a Reply

Your email address will not be published. Required fields are marked *