├── README.md
├── datasets
├── README.md
└── make_datafiles.py
└── results
├── test-BART-model.pt-2f41e316d-partial1000.hypo
├── test-BART-model.pt-2f41e316d.hypo
├── test-semsim-a2d63af3d-partial1000.hypo
├── test-semsim-a2d63af3d.hypo
├── test.target
└── test.target.sample-1000
/README.md:
--------------------------------------------------------------------------------
1 | # Learning by Semantic Similarity Makes Abstractive Summarization Better
2 |
3 | ### Author's note
4 | The initial version of the manuscript includes a model design (semsim), experimental results on our model, BART model, and the reference summaries (both automatic evaluation metric and human evaluation metric), and discussions on the results. After we archived the manuscript, we found that our model has flaws in its implementation and design.
5 |
6 | [The final version of the manuscript](https://arxiv.org/pdf/2002.07767v2.pdf) is from the rest of the initial paper; we included our findings on the benchmark dataset, BART generated results and human evaluations, and we excluded our model semsim.
7 |
8 |
9 | ## Folder description
10 | ```
11 | /--|datasets/
12 | |results/
13 | |README.md
14 | |model.jpg
15 | ```
16 |
17 | * `/datasets` : Our version of the pre-processed CNN/DM dataset and the pre-processing code. Modified from [PGN by See et al.](https://github.com/abisee/cnn-dailymail) following instructions of [BART (issue #1391)](https://github.com/pytorch/fairseq/issues/1391)
18 | * `/results` : We provide summarization results for the CNN/DM dataset and the reduced dataset (n=1000). Folder contains generated summaries of BART and SemSim and reference summaries (not tokenized).
19 |
--------------------------------------------------------------------------------
/datasets/README.md:
--------------------------------------------------------------------------------
1 |
2 | This folder provides the non-anonymized version of the CNN / Daily Mail summarization dataset for BART and the code to produce the dataset.
3 | The dataset and codes are modified from [here](https://github.com/abisee/cnn-dailymail).
4 |
5 | **Python 3 version**: This code is in Python 2. If you want a Python 3 version, see [@artmatsak's fork](https://github.com/artmatsak/cnn-dailymail). However, this will not produce the exact dataset we used for the experiment. In this case, we recommend downloading the dataset (Option 1).
6 |
7 | # Option 1: download the processed data
8 |
9 | Download `cnn_dm.tar.gz`(493MB) from [here](https://drive.google.com/open?id=1goxgX-0_2Jo7cNAFrsb9BJTsrH7re6by)
10 |
11 | | Name | SHA1SUM |
12 | |:------------:|:----------------------------------------:|
13 | | test.source | 8f20fcbaa1f1705d418470ffa34abc2a940d3644 |
14 | | test.target | b2283a8bd0378a0e8b18aba0b925c12f25e73d56 |
15 | | train.source | 91054e4258e5d2bf82414ae39da9ab43d379afdc |
16 | | train.target | 35e808c302a31ece0c4b962aa8e174b8f48ac2c2 |
17 | | val.source | 5583ca6c9c81da2904c4923c0a50fa2f554e588f |
18 | | val.target | c8812748baf8dcbbd71af1479efb0cf9afcfd3ba |
19 |
20 | # Option 2: process the data yourself
21 |
22 | ## 1. Replace make_datafiles.py
23 | Clone git repository by [See](https://github.com/abisee/cnn-dailymail).
24 | Replace `make_datafiles.py` with `make_datafiles.py` from this folder.
25 |
26 | ```
27 | git clone https://github.com/abisee/cnn-dailymail.git
28 | mv make_datafiles.py cnn-dailymail/make_datafiles.py
29 | cd cnn-dailymail
30 | ```
31 |
32 | ## 2. Download data
33 | Download and unzip the `stories` directories from [here](http://cs.nyu.edu/~kcho/DMQA/) for both CNN and Daily Mail.
34 |
35 | **Warning:** These files contain a few (114, in a dataset of over 300,000) examples for which the article text is missing - see for example `cnn/stories/72aba2f58178f2d19d3fae89d5f3e9a4686bc4bb.story`. The [Tensorflow code](https://github.com/abisee/pointer-generator) has been updated to discard these examples.
36 |
37 | ## 3. Process .source and .target files
38 | Execute the following command.
39 | ```python make_datafiles.py /path/to/cnn/stories /path/to/dailymail/stories```
40 |
41 | Replace `/path/to/cnn/stories` with the path to where you saved the `cnn/stories` directory that you downloaded; similarly for `dailymail/stories`.
42 |
43 |
44 |
45 |
--------------------------------------------------------------------------------
/datasets/make_datafiles.py:
--------------------------------------------------------------------------------
1 | import sys
2 | import os
3 | import hashlib
4 | import struct
5 | import subprocess
6 | import collections
7 | import tensorflow as tf
8 | from tensorflow.core.example import example_pb2
9 |
10 | # Modified for BART
11 | # Please see both https://github.com/pytorch/fairseq/issues/1391 and https://github.com/pytorch/fairseq/issues/1401 for details
12 |
13 | dm_single_close_quote = u'\u2019' # unicode
14 | dm_double_close_quote = u'\u201d'
15 | END_TOKENS = ['.', '!', '?', '...', "'", "`", '"', dm_single_close_quote, dm_double_close_quote, ")"] # acceptable ways to end a sentence
16 |
17 | # We use these to separate the summary sentences in the .bin datafiles
18 | SENTENCE_START = ''
19 | SENTENCE_END = ''
20 |
21 | all_train_urls = "url_lists/all_train.txt"
22 | all_val_urls = "url_lists/all_val.txt"
23 | all_test_urls = "url_lists/all_test.txt"
24 |
25 | cnn_tokenized_stories_dir = "cnn_stories_tokenized"
26 | dm_tokenized_stories_dir = "dm_stories_tokenized"
27 | finished_files_dir = "finished_files"
28 | chunks_dir = os.path.join(finished_files_dir, "chunked")
29 |
30 | # These are the number of .story files we expect there to be in cnn_stories_dir and dm_stories_dir
31 | num_expected_cnn_stories = 92579
32 | num_expected_dm_stories = 219506
33 |
34 | VOCAB_SIZE = 200000
35 | CHUNK_SIZE = 1000 # num examples per chunk, for the chunked data
36 |
37 |
38 | def chunk_file(set_name):
39 | in_file = 'finished_files/%s.bin' % set_name
40 | reader = open(in_file, "rb")
41 | chunk = 0
42 | finished = False
43 | while not finished:
44 | chunk_fname = os.path.join(chunks_dir, '%s_%03d.bin' % (set_name, chunk)) # new chunk
45 | with open(chunk_fname, 'wb') as writer:
46 | for _ in range(CHUNK_SIZE):
47 | len_bytes = reader.read(8)
48 | if not len_bytes:
49 | finished = True
50 | break
51 | str_len = struct.unpack('q', len_bytes)[0]
52 | example_str = struct.unpack('%ds' % str_len, reader.read(str_len))[0]
53 | writer.write(struct.pack('q', str_len))
54 | writer.write(struct.pack('%ds' % str_len, example_str))
55 | chunk += 1
56 |
57 |
58 | def chunk_all():
59 | # Make a dir to hold the chunks
60 | if not os.path.isdir(chunks_dir):
61 | os.mkdir(chunks_dir)
62 | # Chunk the data
63 | for set_name in ['train', 'val', 'test']:
64 | print "Splitting %s data into chunks..." % set_name
65 | chunk_file(set_name)
66 | print "Saved chunked data in %s" % chunks_dir
67 |
68 |
69 | def tokenize_stories(stories_dir, tokenized_stories_dir):
70 | """Maps a whole directory of .story files to a tokenized version using Stanford CoreNLP Tokenizer"""
71 | print "Preparing to tokenize %s to %s..." % (stories_dir, tokenized_stories_dir)
72 | stories = os.listdir(stories_dir)
73 | # make IO list file
74 | print "Making list of files to tokenize..."
75 | with open("mapping.txt", "w") as f:
76 | for s in stories:
77 | f.write("%s \t %s\n" % (os.path.join(stories_dir, s), os.path.join(tokenized_stories_dir, s)))
78 | command = ['java', 'edu.stanford.nlp.process.PTBTokenizer', '-ioFileList', '-preserveLines', 'mapping.txt']
79 | print "Tokenizing %i files in %s and saving in %s..." % (len(stories), stories_dir, tokenized_stories_dir)
80 | subprocess.call(command)
81 | print "Stanford CoreNLP Tokenizer has finished."
82 | os.remove("mapping.txt")
83 |
84 | # Check that the tokenized stories directory contains the same number of files as the original directory
85 | num_orig = len(os.listdir(stories_dir))
86 | num_tokenized = len(os.listdir(tokenized_stories_dir))
87 | if num_orig != num_tokenized:
88 | raise Exception("The tokenized stories directory %s contains %i files, but it should contain the same number as %s (which has %i files). Was there an error during tokenization?" % (tokenized_stories_dir, num_tokenized, stories_dir, num_orig))
89 | print "Successfully finished tokenizing %s to %s.\n" % (stories_dir, tokenized_stories_dir)
90 |
91 |
92 | def read_text_file(text_file):
93 | lines = []
94 | with open(text_file, "r") as f:
95 | for line in f:
96 | if chr(13) in line:
97 | lines.append(line.replace(chr(13), " ").strip())
98 | else:
99 | lines.append(line.strip())
100 | return lines
101 |
102 |
103 | def hashhex(s):
104 | """Returns a heximal formated SHA1 hash of the input string."""
105 | h = hashlib.sha1()
106 | h.update(s)
107 | return h.hexdigest()
108 |
109 |
110 | def get_url_hashes(url_list):
111 | return [hashhex(url) for url in url_list]
112 |
113 |
114 | def fix_missing_period(line):
115 | """Adds a period to a line that is missing a period"""
116 | if "@highlight" in line: return line
117 | if line=="": return line
118 | if line[-1] in END_TOKENS: return line
119 | # print line[-1]
120 | return line + "."
121 |
122 |
123 | def get_art_abs(story_file):
124 | lines = read_text_file(story_file)
125 |
126 | # Lowercase everything
127 | #lines = [line.lower() for line in lines]
128 |
129 | # Put periods on the ends of lines that are missing them (this is a problem in the dataset because many image captions don't end in periods; consequently they end up in the body of the article as run-on sentences)
130 | lines = [fix_missing_period(line) for line in lines]
131 |
132 | # Separate out article and abstract sentences
133 | article_lines = []
134 | highlights = []
135 | next_is_highlight = False
136 | for idx,line in enumerate(lines):
137 | if line == "":
138 | continue # empty line
139 | elif line.startswith("@highlight"):
140 | next_is_highlight = True
141 | elif next_is_highlight:
142 | highlights.append(line)
143 | else:
144 | article_lines.append(line)
145 |
146 | # Make article into a single string
147 | article = ' '.join(article_lines)
148 |
149 | # Make abstract into a signle string, putting and tags around the sentences
150 | abstract = ' '.join(highlights) # modified
151 |
152 | return article, abstract
153 |
154 |
155 | def write_to_file(url_file, out_file_source, out_file_target, makevocab=False):
156 | """Reads the tokenized .story files corresponding to the urls listed in the url_file and writes them to a out_file."""
157 | print "Making bin file for URLs listed in %s..." % url_file
158 | url_list = read_text_file(url_file)
159 | url_hashes = get_url_hashes(url_list)
160 | story_fnames = [s+".story" for s in url_hashes]
161 | num_stories = len(story_fnames)
162 |
163 | if makevocab:
164 | vocab_counter = collections.Counter()
165 |
166 | write_target = open(out_file_target, "wb")
167 | with open(out_file_source, 'wb') as writer:
168 | for idx,s in enumerate(story_fnames):
169 | if idx % 1000 == 0:
170 | print "Writing story %i of %i; %.2f percent done" % (idx, num_stories, float(idx)*100.0/float(num_stories))
171 | write_target.flush()
172 | writer.flush()
173 |
174 | # Look in the tokenized story dirs to find the .story file corresponding to this url
175 | if os.path.isfile(os.path.join(cnn_stories_dir, s)):
176 | story_file = os.path.join(cnn_stories_dir, s)
177 | elif os.path.isfile(os.path.join(dm_stories_dir, s)):
178 | story_file = os.path.join(dm_stories_dir, s)
179 | else:
180 | pass
181 | print "Error: Couldn't find tokenized story file %s in either tokenized story directories %s and %s. Was there an error during tokenization?" % (s, cnn_tokenized_stories_dir, dm_tokenized_stories_dir)
182 | # Check again if tokenized stories directories contain correct number of files
183 | print "Checking that the tokenized stories directories %s and %s contain correct number of files..." % (cnn_tokenized_stories_dir, dm_tokenized_stories_dir)
184 | check_num_stories(cnn_tokenized_stories_dir, num_expected_cnn_stories)
185 | check_num_stories(dm_tokenized_stories_dir, num_expected_dm_stories)
186 | raise Exception("Tokenized stories directories %s and %s contain correct number of files but story file %s found in neither." % (cnn_tokenized_stories_dir, dm_tokenized_stories_dir, s))
187 |
188 | # Get the strings to write to .bin file
189 | article, abstract = get_art_abs(story_file)
190 | if article[:5] == '(CNN)':
191 | article = article[5:]
192 |
193 | if idx % 20000 == 0:
194 | print "======= Showing examples... ======= "
195 | print "### idx : %d, article[:500] : %s"%(idx, article[:500])
196 | print "### idx : %d, abstract : %s"%(idx, abstract)
197 |
198 | write_target.write(abstract + '\n')
199 | writer.write(article + '\n')
200 |
201 | """# Write to tf.Example
202 | tf_example = example_pb2.Example()
203 | tf_example.features.feature['article'].bytes_list.value.extend([article])
204 | tf_example.features.feature['abstract'].bytes_list.value.extend([abstract])
205 | tf_example_str = tf_example.SerializeToString()
206 | str_len = len(tf_example_str)
207 | writer.write(struct.pack('q', str_len))
208 | writer.write(struct.pack('%ds' % str_len, tf_example_str))
209 | """
210 |
211 | # Write the vocab to file, if applicable
212 | if makevocab:
213 | art_tokens = article.split(' ')
214 | abs_tokens = abstract.split(' ')
215 | abs_tokens = [t for t in abs_tokens if t not in [SENTENCE_START, SENTENCE_END]] # remove these tags from vocab
216 | tokens = art_tokens + abs_tokens
217 | tokens = [t.strip() for t in tokens] # strip
218 | tokens = [t for t in tokens if t!=""] # remove empty
219 | vocab_counter.update(tokens)
220 |
221 | write_target.flush()
222 | write_target.close()
223 | with open(out_file_target, "r") as validatingF:
224 | for i, l in enumerate(validatingF):
225 | pass
226 | assert i == idx, "target file line No. : %d, source file line No. : %d"%(i, idx)
227 | print "Finished writing file %s\n" % out_file_source
228 |
229 | # write vocab to file
230 | if makevocab:
231 | print "Writing vocab file..."
232 | with open(os.path.join(finished_files_dir, "vocab"), 'w') as writer:
233 | for word, count in vocab_counter.most_common(VOCAB_SIZE):
234 | writer.write(word + ' ' + str(count) + '\n')
235 | print "Finished writing vocab file"
236 |
237 |
238 | def check_num_stories(stories_dir, num_expected):
239 | num_stories = len(os.listdir(stories_dir))
240 | if num_stories != num_expected:
241 | raise Exception("stories directory %s contains %i files but should contain %i" % (stories_dir, num_stories, num_expected))
242 |
243 |
244 | if __name__ == '__main__':
245 | if len(sys.argv) != 3:
246 | print "USAGE: python make_datafiles.py "
247 | sys.exit()
248 | cnn_stories_dir = sys.argv[1]
249 | dm_stories_dir = sys.argv[2]
250 |
251 | # Check the stories directories contain the correct number of .story files
252 | check_num_stories(cnn_stories_dir, num_expected_cnn_stories)
253 | check_num_stories(dm_stories_dir, num_expected_dm_stories)
254 |
255 | # Create some new directories
256 | #if not os.path.exists(cnn_tokenized_stories_dir): os.makedirs(cnn_tokenized_stories_dir)
257 | #if not os.path.exists(dm_tokenized_stories_dir): os.makedirs(dm_tokenized_stories_dir)
258 | if not os.path.exists(finished_files_dir): os.makedirs(finished_files_dir)
259 |
260 | # Run stanford tokenizer on both stories dirs, outputting to tokenized stories directories
261 | #tokenize_stories(cnn_stories_dir, cnn_tokenized_stories_dir)
262 | #tokenize_stories(dm_stories_dir, dm_tokenized_stories_dir)
263 |
264 | # Read the tokenized stories, do a little postprocessing then write to bin files
265 | write_to_file(all_test_urls, out_file_source = os.path.join(finished_files_dir, "test.source"), out_file_target = os.path.join(finished_files_dir, "test.target"))
266 | write_to_file(all_val_urls, out_file_source = os.path.join(finished_files_dir, "val.source"), out_file_target = os.path.join(finished_files_dir, "val.target"))
267 | write_to_file(all_train_urls, out_file_source = os.path.join(finished_files_dir, "train.source"), out_file_target = os.path.join(finished_files_dir, "train.target"), makevocab=True)
268 |
269 | # Chunk the data. This splits each of train.bin, val.bin and test.bin into smaller chunks, each containing e.g. 1000 examples, and saves them in finished_files/chunks
270 | #chunk_all()
271 |
272 |
--------------------------------------------------------------------------------