Attention and Memory in Deep Learning and NLP – WildML

A recent trend in Deep Learning are Attention Mechanisms.

Attention Mechanisms in Neural Networks are loosely based on the visual attention mechanism found in humans.

If we look a bit more look closely at the equation for attention we can see that attention comes at a cost.

An alternative approach to attention is to use Reinforcement Learning to predict an approximate location to focus to.

Interpreted another way, the attention mechanism is simply giving the network access to its internal memory, which is the hidden state of the encoder.

When the networks parameter weights are tied in a certain way, the memory mechanism inEnd-to-End Memory Networks identical to the attention mechanism presented here, only that it makes multiple hops over the memory.

It’s likely that in the future we will see a clearer distinction between memory and attention mechanisms, perhaps along the lines of Reinforcement Learning Neural Turing Machines, which try to learn access patterns to deal with external interfaces.

This article was summarized automatically with AI / Article-Σ ™/ BuildR BOT™.

Original link

The human memory—facts and information

Whenever you say a phone number to yourself over and over to remember it, you’re using your working memory.

Another way to categorize memories is by the subject of the memory itself, and whether you are consciously aware of it.

Some of these memories are facts or “Common knowledge”: things like the capital of Portugal, or the number of cards in a standard deck of playing cards.

The nerves’ collective activity transcribes what we experience as a memory.

For a short-term memory to become a long-term memory, it must be strengthened for long-term storage, a process called memory consolidation.

As we recall a memory, many parts of our brain rapidly talk to each other, including regions in the brain’s cortex that do high-level information processing, regions that handle our senses’ raw inputs, and a region called the medial temporal lobe that seems to help coordinate the process.

The act of remembering something makes that memory temporarily malleable-letting it be strengthened, weakened, or otherwise altered.

This article was summarized automatically with AI / Article-Σ ™/ BuildR BOT™.

Original link