├── README.md ├── datasets ├── README.md └── make_datafiles.py └── results ├── test-BART-model.pt-2f41e316d-partial1000.hypo ├── test-BART-model.pt-2f41e316d.hypo ├── test-semsim-a2d63af3d-partial1000.hypo ├── test-semsim-a2d63af3d.hypo ├── test.target └── test.target.sample-1000 /README.md: -------------------------------------------------------------------------------- 1 | # Learning by Semantic Similarity Makes Abstractive Summarization Better 2 | 3 | ### Author's note 4 | The initial version of the manuscript includes a model design (semsim), experimental results on our model, BART model, and the reference summaries (both automatic evaluation metric and human evaluation metric), and discussions on the results. After we archived the manuscript, we found that our model has flaws in its implementation and design. 5 | 6 | [The final version of the manuscript](https://arxiv.org/pdf/2002.07767v2.pdf) is from the rest of the initial paper; we included our findings on the benchmark dataset, BART generated results and human evaluations, and we excluded our model semsim. 7 | 8 | 9 | ## Folder description 10 | ``` 11 | /--|datasets/ 12 | |results/ 13 | |README.md 14 | |model.jpg 15 | ``` 16 | 17 | * `/datasets` : Our version of the pre-processed CNN/DM dataset and the pre-processing code. Modified from [PGN by See et al.](https://github.com/abisee/cnn-dailymail) following instructions of [BART (issue #1391)](https://github.com/pytorch/fairseq/issues/1391) 18 | * `/results` : We provide summarization results for the CNN/DM dataset and the reduced dataset (n=1000). Folder contains generated summaries of BART and SemSim and reference summaries (not tokenized). 19 | -------------------------------------------------------------------------------- /datasets/README.md: -------------------------------------------------------------------------------- 1 | 2 | This folder provides the non-anonymized version of the CNN / Daily Mail summarization dataset for BART and the code to produce the dataset. 3 | The dataset and codes are modified from [here](https://github.com/abisee/cnn-dailymail). 4 | 5 | **Python 3 version**: This code is in Python 2. If you want a Python 3 version, see [@artmatsak's fork](https://github.com/artmatsak/cnn-dailymail). However, this will not produce the exact dataset we used for the experiment. In this case, we recommend downloading the dataset (Option 1). 6 | 7 | # Option 1: download the processed data 8 | 9 | Download `cnn_dm.tar.gz`(493MB) from [here](https://drive.google.com/open?id=1goxgX-0_2Jo7cNAFrsb9BJTsrH7re6by) 10 | 11 | | Name | SHA1SUM | 12 | |:------------:|:----------------------------------------:| 13 | | test.source | 8f20fcbaa1f1705d418470ffa34abc2a940d3644 | 14 | | test.target | b2283a8bd0378a0e8b18aba0b925c12f25e73d56 | 15 | | train.source | 91054e4258e5d2bf82414ae39da9ab43d379afdc | 16 | | train.target | 35e808c302a31ece0c4b962aa8e174b8f48ac2c2 | 17 | | val.source | 5583ca6c9c81da2904c4923c0a50fa2f554e588f | 18 | | val.target | c8812748baf8dcbbd71af1479efb0cf9afcfd3ba | 19 | 20 | # Option 2: process the data yourself 21 | 22 | ## 1. Replace make_datafiles.py 23 | Clone git repository by [See](https://github.com/abisee/cnn-dailymail). 24 | Replace `make_datafiles.py` with `make_datafiles.py` from this folder. 25 | 26 | ``` 27 | git clone https://github.com/abisee/cnn-dailymail.git 28 | mv make_datafiles.py cnn-dailymail/make_datafiles.py 29 | cd cnn-dailymail 30 | ``` 31 | 32 | ## 2. Download data 33 | Download and unzip the `stories` directories from [here](http://cs.nyu.edu/~kcho/DMQA/) for both CNN and Daily Mail. 34 | 35 | **Warning:** These files contain a few (114, in a dataset of over 300,000) examples for which the article text is missing - see for example `cnn/stories/72aba2f58178f2d19d3fae89d5f3e9a4686bc4bb.story`. The [Tensorflow code](https://github.com/abisee/pointer-generator) has been updated to discard these examples. 36 | 37 | ## 3. Process .source and .target files 38 | Execute the following command. 39 | ```python make_datafiles.py /path/to/cnn/stories /path/to/dailymail/stories``` 40 | 41 | Replace `/path/to/cnn/stories` with the path to where you saved the `cnn/stories` directory that you downloaded; similarly for `dailymail/stories`. 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /datasets/make_datafiles.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import hashlib 4 | import struct 5 | import subprocess 6 | import collections 7 | import tensorflow as tf 8 | from tensorflow.core.example import example_pb2 9 | 10 | # Modified for BART 11 | # Please see both https://github.com/pytorch/fairseq/issues/1391 and https://github.com/pytorch/fairseq/issues/1401 for details 12 | 13 | dm_single_close_quote = u'\u2019' # unicode 14 | dm_double_close_quote = u'\u201d' 15 | END_TOKENS = ['.', '!', '?', '...', "'", "`", '"', dm_single_close_quote, dm_double_close_quote, ")"] # acceptable ways to end a sentence 16 | 17 | # We use these to separate the summary sentences in the .bin datafiles 18 | SENTENCE_START = '' 19 | SENTENCE_END = '' 20 | 21 | all_train_urls = "url_lists/all_train.txt" 22 | all_val_urls = "url_lists/all_val.txt" 23 | all_test_urls = "url_lists/all_test.txt" 24 | 25 | cnn_tokenized_stories_dir = "cnn_stories_tokenized" 26 | dm_tokenized_stories_dir = "dm_stories_tokenized" 27 | finished_files_dir = "finished_files" 28 | chunks_dir = os.path.join(finished_files_dir, "chunked") 29 | 30 | # These are the number of .story files we expect there to be in cnn_stories_dir and dm_stories_dir 31 | num_expected_cnn_stories = 92579 32 | num_expected_dm_stories = 219506 33 | 34 | VOCAB_SIZE = 200000 35 | CHUNK_SIZE = 1000 # num examples per chunk, for the chunked data 36 | 37 | 38 | def chunk_file(set_name): 39 | in_file = 'finished_files/%s.bin' % set_name 40 | reader = open(in_file, "rb") 41 | chunk = 0 42 | finished = False 43 | while not finished: 44 | chunk_fname = os.path.join(chunks_dir, '%s_%03d.bin' % (set_name, chunk)) # new chunk 45 | with open(chunk_fname, 'wb') as writer: 46 | for _ in range(CHUNK_SIZE): 47 | len_bytes = reader.read(8) 48 | if not len_bytes: 49 | finished = True 50 | break 51 | str_len = struct.unpack('q', len_bytes)[0] 52 | example_str = struct.unpack('%ds' % str_len, reader.read(str_len))[0] 53 | writer.write(struct.pack('q', str_len)) 54 | writer.write(struct.pack('%ds' % str_len, example_str)) 55 | chunk += 1 56 | 57 | 58 | def chunk_all(): 59 | # Make a dir to hold the chunks 60 | if not os.path.isdir(chunks_dir): 61 | os.mkdir(chunks_dir) 62 | # Chunk the data 63 | for set_name in ['train', 'val', 'test']: 64 | print "Splitting %s data into chunks..." % set_name 65 | chunk_file(set_name) 66 | print "Saved chunked data in %s" % chunks_dir 67 | 68 | 69 | def tokenize_stories(stories_dir, tokenized_stories_dir): 70 | """Maps a whole directory of .story files to a tokenized version using Stanford CoreNLP Tokenizer""" 71 | print "Preparing to tokenize %s to %s..." % (stories_dir, tokenized_stories_dir) 72 | stories = os.listdir(stories_dir) 73 | # make IO list file 74 | print "Making list of files to tokenize..." 75 | with open("mapping.txt", "w") as f: 76 | for s in stories: 77 | f.write("%s \t %s\n" % (os.path.join(stories_dir, s), os.path.join(tokenized_stories_dir, s))) 78 | command = ['java', 'edu.stanford.nlp.process.PTBTokenizer', '-ioFileList', '-preserveLines', 'mapping.txt'] 79 | print "Tokenizing %i files in %s and saving in %s..." % (len(stories), stories_dir, tokenized_stories_dir) 80 | subprocess.call(command) 81 | print "Stanford CoreNLP Tokenizer has finished." 82 | os.remove("mapping.txt") 83 | 84 | # Check that the tokenized stories directory contains the same number of files as the original directory 85 | num_orig = len(os.listdir(stories_dir)) 86 | num_tokenized = len(os.listdir(tokenized_stories_dir)) 87 | if num_orig != num_tokenized: 88 | raise Exception("The tokenized stories directory %s contains %i files, but it should contain the same number as %s (which has %i files). Was there an error during tokenization?" % (tokenized_stories_dir, num_tokenized, stories_dir, num_orig)) 89 | print "Successfully finished tokenizing %s to %s.\n" % (stories_dir, tokenized_stories_dir) 90 | 91 | 92 | def read_text_file(text_file): 93 | lines = [] 94 | with open(text_file, "r") as f: 95 | for line in f: 96 | if chr(13) in line: 97 | lines.append(line.replace(chr(13), " ").strip()) 98 | else: 99 | lines.append(line.strip()) 100 | return lines 101 | 102 | 103 | def hashhex(s): 104 | """Returns a heximal formated SHA1 hash of the input string.""" 105 | h = hashlib.sha1() 106 | h.update(s) 107 | return h.hexdigest() 108 | 109 | 110 | def get_url_hashes(url_list): 111 | return [hashhex(url) for url in url_list] 112 | 113 | 114 | def fix_missing_period(line): 115 | """Adds a period to a line that is missing a period""" 116 | if "@highlight" in line: return line 117 | if line=="": return line 118 | if line[-1] in END_TOKENS: return line 119 | # print line[-1] 120 | return line + "." 121 | 122 | 123 | def get_art_abs(story_file): 124 | lines = read_text_file(story_file) 125 | 126 | # Lowercase everything 127 | #lines = [line.lower() for line in lines] 128 | 129 | # Put periods on the ends of lines that are missing them (this is a problem in the dataset because many image captions don't end in periods; consequently they end up in the body of the article as run-on sentences) 130 | lines = [fix_missing_period(line) for line in lines] 131 | 132 | # Separate out article and abstract sentences 133 | article_lines = [] 134 | highlights = [] 135 | next_is_highlight = False 136 | for idx,line in enumerate(lines): 137 | if line == "": 138 | continue # empty line 139 | elif line.startswith("@highlight"): 140 | next_is_highlight = True 141 | elif next_is_highlight: 142 | highlights.append(line) 143 | else: 144 | article_lines.append(line) 145 | 146 | # Make article into a single string 147 | article = ' '.join(article_lines) 148 | 149 | # Make abstract into a signle string, putting and tags around the sentences 150 | abstract = ' '.join(highlights) # modified 151 | 152 | return article, abstract 153 | 154 | 155 | def write_to_file(url_file, out_file_source, out_file_target, makevocab=False): 156 | """Reads the tokenized .story files corresponding to the urls listed in the url_file and writes them to a out_file.""" 157 | print "Making bin file for URLs listed in %s..." % url_file 158 | url_list = read_text_file(url_file) 159 | url_hashes = get_url_hashes(url_list) 160 | story_fnames = [s+".story" for s in url_hashes] 161 | num_stories = len(story_fnames) 162 | 163 | if makevocab: 164 | vocab_counter = collections.Counter() 165 | 166 | write_target = open(out_file_target, "wb") 167 | with open(out_file_source, 'wb') as writer: 168 | for idx,s in enumerate(story_fnames): 169 | if idx % 1000 == 0: 170 | print "Writing story %i of %i; %.2f percent done" % (idx, num_stories, float(idx)*100.0/float(num_stories)) 171 | write_target.flush() 172 | writer.flush() 173 | 174 | # Look in the tokenized story dirs to find the .story file corresponding to this url 175 | if os.path.isfile(os.path.join(cnn_stories_dir, s)): 176 | story_file = os.path.join(cnn_stories_dir, s) 177 | elif os.path.isfile(os.path.join(dm_stories_dir, s)): 178 | story_file = os.path.join(dm_stories_dir, s) 179 | else: 180 | pass 181 | print "Error: Couldn't find tokenized story file %s in either tokenized story directories %s and %s. Was there an error during tokenization?" % (s, cnn_tokenized_stories_dir, dm_tokenized_stories_dir) 182 | # Check again if tokenized stories directories contain correct number of files 183 | print "Checking that the tokenized stories directories %s and %s contain correct number of files..." % (cnn_tokenized_stories_dir, dm_tokenized_stories_dir) 184 | check_num_stories(cnn_tokenized_stories_dir, num_expected_cnn_stories) 185 | check_num_stories(dm_tokenized_stories_dir, num_expected_dm_stories) 186 | raise Exception("Tokenized stories directories %s and %s contain correct number of files but story file %s found in neither." % (cnn_tokenized_stories_dir, dm_tokenized_stories_dir, s)) 187 | 188 | # Get the strings to write to .bin file 189 | article, abstract = get_art_abs(story_file) 190 | if article[:5] == '(CNN)': 191 | article = article[5:] 192 | 193 | if idx % 20000 == 0: 194 | print "======= Showing examples... ======= " 195 | print "### idx : %d, article[:500] : %s"%(idx, article[:500]) 196 | print "### idx : %d, abstract : %s"%(idx, abstract) 197 | 198 | write_target.write(abstract + '\n') 199 | writer.write(article + '\n') 200 | 201 | """# Write to tf.Example 202 | tf_example = example_pb2.Example() 203 | tf_example.features.feature['article'].bytes_list.value.extend([article]) 204 | tf_example.features.feature['abstract'].bytes_list.value.extend([abstract]) 205 | tf_example_str = tf_example.SerializeToString() 206 | str_len = len(tf_example_str) 207 | writer.write(struct.pack('q', str_len)) 208 | writer.write(struct.pack('%ds' % str_len, tf_example_str)) 209 | """ 210 | 211 | # Write the vocab to file, if applicable 212 | if makevocab: 213 | art_tokens = article.split(' ') 214 | abs_tokens = abstract.split(' ') 215 | abs_tokens = [t for t in abs_tokens if t not in [SENTENCE_START, SENTENCE_END]] # remove these tags from vocab 216 | tokens = art_tokens + abs_tokens 217 | tokens = [t.strip() for t in tokens] # strip 218 | tokens = [t for t in tokens if t!=""] # remove empty 219 | vocab_counter.update(tokens) 220 | 221 | write_target.flush() 222 | write_target.close() 223 | with open(out_file_target, "r") as validatingF: 224 | for i, l in enumerate(validatingF): 225 | pass 226 | assert i == idx, "target file line No. : %d, source file line No. : %d"%(i, idx) 227 | print "Finished writing file %s\n" % out_file_source 228 | 229 | # write vocab to file 230 | if makevocab: 231 | print "Writing vocab file..." 232 | with open(os.path.join(finished_files_dir, "vocab"), 'w') as writer: 233 | for word, count in vocab_counter.most_common(VOCAB_SIZE): 234 | writer.write(word + ' ' + str(count) + '\n') 235 | print "Finished writing vocab file" 236 | 237 | 238 | def check_num_stories(stories_dir, num_expected): 239 | num_stories = len(os.listdir(stories_dir)) 240 | if num_stories != num_expected: 241 | raise Exception("stories directory %s contains %i files but should contain %i" % (stories_dir, num_stories, num_expected)) 242 | 243 | 244 | if __name__ == '__main__': 245 | if len(sys.argv) != 3: 246 | print "USAGE: python make_datafiles.py " 247 | sys.exit() 248 | cnn_stories_dir = sys.argv[1] 249 | dm_stories_dir = sys.argv[2] 250 | 251 | # Check the stories directories contain the correct number of .story files 252 | check_num_stories(cnn_stories_dir, num_expected_cnn_stories) 253 | check_num_stories(dm_stories_dir, num_expected_dm_stories) 254 | 255 | # Create some new directories 256 | #if not os.path.exists(cnn_tokenized_stories_dir): os.makedirs(cnn_tokenized_stories_dir) 257 | #if not os.path.exists(dm_tokenized_stories_dir): os.makedirs(dm_tokenized_stories_dir) 258 | if not os.path.exists(finished_files_dir): os.makedirs(finished_files_dir) 259 | 260 | # Run stanford tokenizer on both stories dirs, outputting to tokenized stories directories 261 | #tokenize_stories(cnn_stories_dir, cnn_tokenized_stories_dir) 262 | #tokenize_stories(dm_stories_dir, dm_tokenized_stories_dir) 263 | 264 | # Read the tokenized stories, do a little postprocessing then write to bin files 265 | write_to_file(all_test_urls, out_file_source = os.path.join(finished_files_dir, "test.source"), out_file_target = os.path.join(finished_files_dir, "test.target")) 266 | write_to_file(all_val_urls, out_file_source = os.path.join(finished_files_dir, "val.source"), out_file_target = os.path.join(finished_files_dir, "val.target")) 267 | write_to_file(all_train_urls, out_file_source = os.path.join(finished_files_dir, "train.source"), out_file_target = os.path.join(finished_files_dir, "train.target"), makevocab=True) 268 | 269 | # Chunk the data. This splits each of train.bin, val.bin and test.bin into smaller chunks, each containing e.g. 1000 examples, and saves them in finished_files/chunks 270 | #chunk_all() 271 | 272 | --------------------------------------------------------------------------------