Archive | Attention

attention_mechanism_cover

The Attention Mechanism from Scratch

The attention mechanism was introduced to improve the performance of the encoder-decoder model for machine translation. The idea behind the attention mechanism was to permit the decoder to utilize the most relevant parts of the input sequence in a flexible manner, by a weighted combination of all of the encoded input vectors, with the most […]

Continue Reading 2
what_is_attention_cover

What is Attention?

Attention is becoming increasingly popular in machine learning, but what makes it such an attractive concept? What is the relationship between attention as applied in artificial neural networks, and its biological counterpart? What are the components that one would expect to form an attention-based system in machine learning? In this tutorial, you will discover an […]

Continue Reading 2
attention_research_cover

A Bird’s Eye View of Research on Attention

Attention is a concept that is scientifically studied across multiple disciplines, including psychology, neuroscience and, more recently, machine learning. While all disciplines may have produced their own definitions for attention, there is one core quality they can all agree on: attention is a mechanism for making both biological and artificial neural systems more flexible.  In […]

Continue Reading 7