7 Applications of Deep Learning for Natural Language Processing

The field of natural language processing is shifting from statistical methods to neural network methods.

There are still many challenging problems to solve in natural language. Nevertheless, deep learning methods are achieving state-of-the-art results on some specific language problems.

It is not just the performance of deep learning models on benchmark problems that is most interesting; it is the fact that a single model can learn word meaning and perform language tasks, obviating the need for a pipeline of specialized and hand-crafted methods.

In this post, you will discover 7 interesting natural language processing tasks where deep learning methods are achieving some headway.

Kick-start your project with my new book Deep Learning for Natural Language Processing, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

7 Applications of Deep Learning for Natural Language Processing
Photo by Tim Gorman, some rights reserved.

Overview

In this post, we will look at the following 7 natural language processing problems.

1. Text Classification
2. Language Modeling
3. Speech Recognition
4. Caption Generation
5. Machine Translation
6. Document Summarization
7. Question Answering

I have tried to focus on the types of end-user problems that you may be interested in, as opposed to more academic or linguistic sub-problems where deep learning does well such as part-of-speech tagging, chunking, named entity recognition, and so on.

Each example provides a description of the problem, an example, and references to papers that demonstrate the methods and results. Most references are drawn from Goldberg’s excellent 2015 primer on deep learning for NLP researchers.

Do you have a favorite NLP application for deep learning that is not listed?
Let me know in the comments below.

1. Text Classification

Given an example of text, predict a predefined class label.

The goal of text categorization is to classify the topic or theme of a document.

— Page 575, Foundations of Statistical Natural Language Processing, 1999.

A popular classification example is sentiment analysis where class labels represent the emotional tone of the source text such as “positive” or “negative“.

Below are a 3 more examples:

• Spam filtering, classifying email text as spam or not.
• Language identification, classifying the language of the source text.
• Genre classification, classifying the genre of a fictional story.

Further, the problem may be framed in a way that requires multiple classes assigned to a text, so-called multi-label classification. Such as predicting multiple hashtags for a source tweet.

For more on the general topic, see:

• Text categorization on Scholarpedia
• Document classification on Wikipedia

Below are 3 examples of deep learning papers for text classification:

• Sentiment analysis of rotten tomato movie reviews.
  • Deep Unordered Composition Rivals Syntactic Methods for Text Classification, 2015.
• Sentiment analysis of amazon product reviews, IMDB movie reviews and topic categorization of news articles.
  • Effective Use of Word Order for Text Categorization with Convolutional Neural Networks, 2015.
• Sentiment analysis of movie reviews, classifying sentences as being subjective or objective, classifying question types, sentiment of product reviews and more.
  • Convolutional Neural Networks for Sentence Classification, 2014.

2. Language Modeling

Language modeling is really a subtask of more interesting natural language problems, specifically those that condition the language model on some other input.

… the problem is to predict the next word given the previous words. The task is fundamental to speech or optical character recognition, and is also used for spelling correction, handwriting recognition, and statistical machine translation.

— Page 191, Foundations of Statistical Natural Language Processing, 1999.

In addition to the academic interest in language modeling, it is a key component of many deep learning natural language processing architectures.

A language model learns the probabilistic relationship between words such that new sequences of words can be generated that are statistically consistent with the source text.

Alone, language models can be used for text or speech generation; for example:

• Generating new article headlines.
• Generating new sentences, paragraphs, or documents.
• Generating suggested continuation of a sentence.

For more in language modeling, see:

• Language model on Wikipedia
• The Unreasonable Effectiveness of Recurrent Neural Networks, 2015.
• Generative Model-Based Text-to-Speech Synthesis, Lecture 10, Oxford, 2017

Below is an example of deep learning for language modeling (only):

• Language model of English texts, books and news articles.
  • A Neural Probabilistic Language Model, 2003

3. Speech Recognition

Speech recognition is the problem of understanding what was said.

The task of speech recognition is to map an acoustic signal containing a spoken natural language utterance into the corresponding sequence of words intended by the speaker.

— Page 458, Deep Learning, 2016.

Given an utterance of text as audio data, the model must produce human readable text.

Given the automatic nature of the process, the problem may also be called Automatic Speech Recognition (ASR).

A language model is used to create the text output that is conditioned on the audio data.

Some examples include:

• Transcribing a speech.
• Creating text captions for a movie or TV show.
• Issuing commands to the radio while driving.

For more on speech recognition, see:

• Speech recognition on Wikipedia

Below are 3 examples of deep learning for speech recognition.

• English speech to text.
  • Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks, 2006.
• English speech to text.
  • Speech Recognition with Deep Recurrent Neural Networks, 2013.
• English speech to text.
  • Exploring convolutional neural network structures and optimization techniques for speech recognition, 2014.

4. Caption Generation

Caption generation is the problem of describing the contents of an image.

Given a digital image, such as a photo, generate a textual description of the contents of the image.

A language model is used to create the caption that is conditioned on the image.

Some examples include:

• Describing the contents of a scene.
• Creating a caption for a photograph.
• Describing a video.

This is not just an application for the hearing impaired, but also in generating human readable text for image and video data that can be searched, such as on the web.

Below are 3 examples of deep learning for caption generation:

• Generating captions for photographs.
  • Show, Attend and Tell: Neural Image Caption Generation with Visual Attention, 2016.
• Generating captions for photographs.
  • Show and tell: A neural image caption generator, 2015.
• Generating captions for videos.
  • Sequence to Sequence – Video to Text, 2015.

5. Machine Translation

Machine translation is the problem of converting a source text in one language to another language.

Machine translation, the automatic translation of text or speech from one language to another, is one [of] the most important applications of NLP.

— Page 463, Foundations of Statistical Natural Language Processing, 1999.

Given that deep neural networks are used, the field is referred to as neural machine translation.

In a machine translation task, the input already consists of a sequence of symbols in some language, and the computer program must convert this int a sequence of symbols in another language. This is commonly applied to natural languages, such as translating from English to French. Deep learning has recently begun to have an important impact on this kind of task.

— Page 98, Deep Learning, 2016.

A language model is used to output the destination text in the second language, conditioned on the source text.

Some examples include:

• Translating a text document from French to English.
• Translating Spanish audio to German text.
• Translating English text to Italian audio.

For more on neural machine translation, see:

• Neural machine translation on Wikipedia

Below are 3 examples of deep learning for machine translation:

• Translation of text from English to French.
  • Sequence to Sequence Learning with Neural Networks, 2014.
• Translation of text from English to French.
  • Neural Machine Translation by Jointly Learning to Align and Translate, 2014.
• Translation of text from English to French.
  • Joint Language and Translation Modeling with Recurrent Neural Networks, 2013.

6. Document Summarization

Document summarization is the task where a short description of a text document is created.

As above, a language model is used to output the summary conditioned on the full document.

Some examples of document summarization include:

• Creating a heading for a document.
• Creating an abstract of a document.

For more on the topic, see:

• Automatic summarization on Wikipedia.
• Has Deep Learning been applied to automatic text summarization (successfully)?

Below are 3 examples of deep learning for document summarization:

• Summarization of sentences in news articles.
  • A Neural Attention Model for Abstractive Summarization, 2015.
• Summarization of sentences in news articles.
  • Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond, 2016.
• Summarization of sentences in news articles.
  • Neural Summarization by Extracting Sentences and Word, 2016.

7. Question Answering

Question answering is the problem where given a subject, such as a document of text, answer a specific question about the subject.

… question answering systems which try to answer a user query that is formulated in the form of a question by return the appropriate none phrase such as a location, a person, or a date. For example, the question Why killed President Kennedy? might be answered with the noun phrase Oswald

— Page 377, Foundations of Statistical Natural Language Processing, 1999.

Some examples include:

• Question answering on Wikipedia

For more information on question answering, see:

• Answering questions about Wikipedia articles.
• Answering questions about news articles.
• Answering questions about medical records.

Below are 3 examples of deep learning for question answering:

• Answering questions about news articles.
  • Teaching Machines to Read and Comprehend, 2015.
• Answering general knowledge questions over freebase articles.
  • Question Answering over Freebase with Multi-Column Convolutional Neural Networks, 2015.
• Answers factoid questions given specific documents.
  • Deep Learning for Answer Sentence Selection, 2015.

Further Reading

This section provides more resources on deep learning applications for NLP if you are looking go deeper.

• A Primer on Neural Network Models for Natural Language Processing, 2015.
• Natural Language Processing (almost) from Scratch, 2011.
• Deep Learning for Natural Language Processing, Practicals Overview, Oxford, 2017
• What NLP problems has deep learning or neural networks been applied to successfully?
• Can deep learning make similar breakthroughs in natural language processing as it did in vision & speech?

Summary

In this post, you discovered 7 applications of deep learning to natural language processing tasks.

Was your favorite example of deep learning for NLP missed?
Let me know in the comments.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Develop Deep Learning models for Text Data Today!

Develop Your Own Text models in Minutes

...with just a few lines of python code

Discover how in my new Ebook:
Deep Learning for Natural Language Processing

It provides self-study tutorials on topics like:
Bag-of-Words, Word Embedding, Language Models, Caption Generation, Text Translation and much more...

Finally Bring Deep Learning to your Natural Language Processing Projects

Skip the Academics. Just Results.

See What's Inside