How to Prepare News Articles for Text Summarization

Text summarization is the task of creating a short, accurate, and fluent summary of an article.

A popular and free dataset for use in text summarization experiments with deep learning methods is the CNN News story dataset.

In this tutorial, you will discover how to prepare the CNN News Dataset for text summarization.

After completing this tutorial, you will know:

  • About the CNN News dataset and how to download the story data to your workstation.
  • How to load the dataset and split each article into story text and highlights.
  • How to clean the dataset ready for modeling and save the cleaned data to file for later use.

Let’s get started.

How to Prepare News Articles for Text Summarization

How to Prepare News Articles for Text Summarization
Photo by DieselDemon, some rights reserved.

Tutorial Overview

This tutorial is divided into 5 parts; they are:

  1. CNN News Story Dataset
  2. Inspect the Dataset
  3. Load Data
  4. Data Cleaning
  5. Save Clean Data

Need help with Deep Learning for Text Data?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

Start Your FREE Crash-Course Now

CNN News Story Dataset

The DeepMind Q&A Dataset is a large collection of news articles from CNN and the Daily Mail with associated questions.

The dataset was developed as a question and answering task for deep learning and was presented in the 2015 paper “Teaching Machines to Read and Comprehend.”

This dataset has been used in text summarization where sentences from the news articles are summarized. Notable examples are the papers:

Kyunghyun Cho is an academic at New York University and has made the dataset available for download:

In this tutorial, we will work with the CNN dataset, specifically the download of the ASCII text of the news stories available here:

This dataset contains more than 93,000 news articles where each article is stored in a single “.story” file.

Download this dataset to your workstation and unzip it. Once downloaded, you can unzip the archive on your command line as follows:

This will create a cnn/stories/ directory filled with .story files.

For example, we can count the number of story files on the command line as follows:

Which shows us that we have a total of 92,580 stores.

Inspect the Dataset

Using a text editor, review some of the stories and note down some ideas for preparing this data.

For example, below is an example of a story, with the body truncated for brevity.

I note that the general structure of the dataset is to have the story text followed by a number of “highlight” points.

Reviewing articles on the CNN website, I can see that this pattern is still common.

Example of a CNN News Article With Highlights from cnn.com

Example of a CNN News Article With Highlights from cnn.com

The ASCII text does not include the article titles, but we can use these human-written “highlights” as multiple reference summaries for each news article.

I can also see that many articles start with source information, presumably the CNN office that produced the story; for example:

These can be removed completely.

Data cleaning is a challenging problem and must be tailored for the specific application of the system.

If we are generally interested in developing a news article summarization system, then we may clean the text in order to simplify the learning problem by reducing the size of the vocabulary.

Some data cleaning ideas for this data include.

  • Normalize case to lowercase (e.g. “An Italian”).
  • Remove punctuation (e.g. “on-time”).

We could also further reduce the vocabulary to speed up testing models, such as:

  • Remove numbers (e.g. “93.4%”).
  • Remove low-frequency words like names (e.g. “Tom Watkins”).
  • Truncating stories to the first 5 or 10 sentences.

Load Data

The first step is to load the data.

We can start by writing a function to load a single document given a filename. The data has some unicode characters, so we will load the dataset by forcing the encoding to be UTF-8.

The function below named load_doc() will load a single document as text given a filename.

Next, we need to step over each filename in the stories directory and load them.

We can use the listdir() function to load all filenames in the directory, then load each one in turn. The function below named load_stories() implements this behavior and provides a starting point for preparing the loaded documents.

Each document can be separated into the news story text and the highlights or summary text.

The split for these two points is the first occurrence of the ‘@highlight‘ token. Once split, we can organize the highlights into a list.

The function below named split_story() implements this behavior and splits a given loaded document text into a story and list of highlights.

We can now update the load_stories() function to call the split_story() function for each loaded document and then store the results in a list.

Tying all of this together, the complete example of loading the entire dataset is listed below.

Running the example prints the number of loaded stories.

We can now access the loaded story and highlight data, for example:

Data Cleaning

Now that we can load the story data, we can pre-process the text by cleaning it.

We can process the stories line-by line and use the same cleaning operations on each highlight line.

For a given line, we will perform the following operations:

Remove the CNN office information.

Split the line using white space tokens:

Normalize the case to lowercase.

Remove all punctuation characters from each token (Python 3 specific).

Remove any words that have non-alphabetic characters.

Putting this all together, below is a new function named clean_lines() that takes a list of lines of text and returns a list of clean lines of text.

We can call this for a story, by first converting it to a line of text. The function can be called directly on the list of highlights.

The complete example of loading and cleaning the dataset is listed below.

Note that the story is now stored as a list of clean lines, nominally, separated by sentences.

Save Clean Data

Finally, now that the data has been cleaned, we can save it to file.

An easy way to save the cleaned data is to Pickle the list of stories and highlights.

For example:

This will create a new file named cnn_dataset.pkl with all of the cleaned data. This file will be about 374 Megabytes in size.

We can then load it later and use it with a text summarization model as follows:

Further Reading

This section provides more resources on the topic if you are looking go deeper.

Summary

In this tutorial, you discovered how to prepare the CNN News Dataset for text summarization.

Specifically, you learned:

  • About the CNN News dataset and how to download the story data to your workstation.
  • How to load the dataset and split each article into story text and highlights.
  • How to clean the dataset ready for modeling and save the cleaned data to file for later use.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.


Develop Deep Learning models for Text Data Today!

Deep Learning for Natural Language Processing

Develop Your Own Text models in Minutes

…with just a few lines of python code

Discover how in my new Ebook:
Deep Learning for Natural Language Processing

It provides self-study tutorials on topics like:
Bag-of-Words, Word Embedding, Language Models, Caption Generation, Text Translation and much more…

Finally Bring Deep Learning to your Natural Language Processing Projects

Skip the Academics. Just Results.

Click to learn more.


2 Responses to How to Prepare News Articles for Text Summarization

  1. kiran December 8, 2017 at 4:38 pm #

    while executing on a windows system “load data” code fragment getting the following error: “‘utf-8’ codec can’t decode byte 0xc0 in position 12: invalid start byte”.

    • Jason Brownlee December 9, 2017 at 5:37 am #

      Interesting, perhaps try changing the encoding, try ‘ascii’?

      Or perhaps try loading the final as binary, then converting the text to ascii?

Leave a Reply