├── .gitignore ├── LICENSE ├── README.md ├── _config.yml ├── coherence_from_paper_script ├── AVTIM_200 ├── AVTIM_50 └── Readme.md ├── data ├── 20news_clean │ ├── test.txt.npy │ ├── train.txt.npy │ ├── valid.txt.npy │ └── vocab.pkl └── Readme.md ├── models ├── Readme.md ├── __init__.py ├── nvlda.py └── prodlda.py └── run.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | 27 | # PyInstaller 28 | # Usually these files are written by a python script from a template 29 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 30 | *.manifest 31 | *.spec 32 | 33 | # Installer logs 34 | pip-log.txt 35 | pip-delete-this-directory.txt 36 | 37 | # Unit test / coverage reports 38 | htmlcov/ 39 | .tox/ 40 | .coverage 41 | .coverage.* 42 | .cache 43 | nosetests.xml 44 | coverage.xml 45 | *,cover 46 | .hypothesis/ 47 | 48 | # Translations 49 | *.mo 50 | *.pot 51 | 52 | # Django stuff: 53 | *.log 54 | local_settings.py 55 | 56 | # Flask stuff: 57 | instance/ 58 | .webassets-cache 59 | 60 | # Scrapy stuff: 61 | .scrapy 62 | 63 | # Sphinx documentation 64 | docs/_build/ 65 | 66 | # PyBuilder 67 | target/ 68 | 69 | # IPython Notebook 70 | .ipynb_checkpoints 71 | 72 | # pyenv 73 | .python-version 74 | 75 | # celery beat schedule file 76 | celerybeat-schedule 77 | 78 | # dotenv 79 | .env 80 | 81 | # virtualenv 82 | venv/ 83 | ENV/ 84 | 85 | # Spyder project settings 86 | .spyderproject 87 | 88 | # Rope project settings 89 | .ropeproject 90 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Akash Srivastava 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Autoencoding Variational Inference for Topic Models 2 | 3 | __UPDATE__ 4 | 5 | >> Pyro added a prodlDA tutorial: https://pyro.ai/examples/prodlda.html 6 | 7 | >> AVITM is now available in OCTIS at https://github.com/MIND-Lab/OCTIS 8 | 9 | Please consider using OCTIS and Pyro versions as they are more upto date. 10 | 11 | 1. As pointed out by [@govg](https://github.com/govg), this code depends on a slightly older version of TF. I will try to update it soon, in the meantime you can look up a quick fix [here](https://github.com/akashgit/autoencoding_vi_for_topic_models/issues/5) for working with newer version of TF or (3) and (2) below if you'd rather prefer Keras or PyTorch. 12 | 13 | 2. [@nzw0301](https://github.com/nzw0301) has implemented a [Keras](https://github.com/nzw0301/keras-examples/blob/master/prodLDA.ipynb) version of prodLDA. 14 | 15 | 3. [@hyqneuron](https://github.com/hyqneuron) recently implemented a [PyTorch](https://github.com/hyqneuron/pytorch-avitm) version of AVITM. So check out his repo. 16 | 17 | 4. Added `topic_prop` method to both the models. Softmax the output of this method to get the topic proportions. 18 | 19 | --- 20 | #### Code for the ICLR 2017 paper: Autoencoding Variational Inference for Topic Models 21 | --- 22 | 23 | #### > [Arxiv](https://arxiv.org/abs/1703.01488) 24 | 25 | #### > [OpenReview](http://openreview.net/forum?id=BybtVK9lg) 26 | 27 | --- 28 | ###### This is a tensorflow implementation for both of the Autoencoded Topic Models mentioned in the paper. 29 | --- 30 | To run the `prodLDA` model in the `20Newgroup` dataset: 31 | 32 | > `CUDA_VISIBLE_DEVICES=0 python run.py -m prodlda -f 100 -s 100 -t 50 -b 200 -r 0.002 -e 200` 33 | 34 | Similarly for `NVLDA`: 35 | 36 | > `CUDA_VISIBLE_DEVICES=0 python run.py -m nvlda -f 100 -s 100 -t 50 -b 200 -r 0.005 -e 300` 37 | 38 | Check `run.py` for other options. 39 | 40 | 41 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-cayman -------------------------------------------------------------------------------- /coherence_from_paper_script/AVTIM_200: -------------------------------------------------------------------------------- 1 | (IPC_MAP)[haiti]s1362249: bash 20news.sh 2 | Computing word occurrence... 3 | Computing the observed coherence... 4 | [0.08] food microsoft detector menu msg app engine amp windows rf 5 | [0.15] bike helmet hitter ride honda rider bmw cop baseball sport 6 | [0.46] output buf io buffer oname input char printf int stream 7 | [0.23] bike honda bmw helmet engine rear ride mile kit rider 8 | [0.24] eternal heaven hell sin god bible resurrection koresh christ christian 9 | [0.16] virtual spacecraft km detector reality scientific nasa shuttle flight lunar 10 | [0.12] motherboard hus gateway mhz thanks monitor meg boot advance audio 11 | [0.21] monitor motherboard shipping vga card slot meg thanks upgrade video 12 | [0.12] annual module flight drink shuttle rocket ride nasa mile excellent 13 | [0.15] shipping rider annual art app appear bike motorcycle origin sale 14 | [0.16] baseball braves fan hitter coach hockey pitch defensive team bike 15 | [0.20] serial escrow agent enforcement staff chip wiretap vendor security agency 16 | [0.16] graphics font xterm toolkit window printer postscript documentation graphic compile 17 | [0.14] bike honda thanks rider hus ride bmw simm dod sale 18 | [0.26] motherboard adapter simm vga ide connector quadra meg slot pin 19 | [0.16] bike bmw honda ride motorcycle baseball ball dod helmet fan 20 | [0.36] eus rg bhj pl pd ax oo outlet mr wiring 21 | [0.28] annual det tor min van shipping que cd pit art 22 | [0.15] community role education secretary young college kid economy training sector 23 | [0.25] simm ram vga cpu mhz cache mac ibm video os 24 | [0.30] israel arab palestinian lebanese islamic israeli arabs marriage lebanon jew 25 | [0.24] nation prophecy prophet jesus verse passage matthew lord king mouth 26 | [0.31] ide simm isa scsi motherboard meg scsus quadra slot bus 27 | [0.35] lebanese lebanon civilian soldier peace israel village attack troops terrorist 28 | [0.22] dos pixel format gif color jpeg setting compression image shipping 29 | [0.11] insurance patient economy billion health batf fund drug violent hospital 30 | [0.23] converter context xt colormap destroy application resource widget visual variable 31 | [0.11] homosexual sexual marriage compound morality cult koresh gay islamic church 32 | [0.14] ripem cpu ram rsa motorola mac mhz des pgp cipher 33 | [0.24] nhl coach league pen hitter baseball montreal playoff cup team 34 | [0.12] hitler package koresh russia administration congress senior guilty official innocent 35 | [0.26] ide dos meg scsi scsus isa pc simm quadra hd 36 | [0.16] car radar honda detector vehicle engine mile dealer amp wheel 37 | [0.17] hitter baseball cop pitch player pitcher hit accident ball team 38 | [0.17] binary macintosh dec kit xterm patch configuration hp mit font 39 | [0.09] trial cancer child medical mouse center north patient laboratory education 40 | [0.20] puck slot flyers defensive defense coach playoff leafs season deserve 41 | [0.20] armenian rsa ripem armenians armenia pgp turkish massacre russian cryptography 42 | [0.10] module rocket excellent coach player houston nhl mission km orbit 43 | [0.20] victim greece greek shoot turks turkish defend community terrorist massacre 44 | [0.47] eus rg pd pl mg mw hd hus bhj oo 45 | [0.28] rangers coach montreal leafs stanley playoff pittsburgh braves detroit wings 46 | [0.29] christ jesus sin heaven god eternal hell lord mary satan 47 | [0.24] floppy windows card rom upgrade bio modem connector disk port 48 | [0.12] thanks vga hus advance printer video hello graphic port modem 49 | [0.26] escrow wiretap encryption nsa criminal agency government chip cipher crypto 50 | [0.13] cop escrow batf nsa wiretap crypto gun criminal gang government 51 | [0.11] amiga graphics pc saturn ftp graphic turbo processing lunar sky 52 | [0.18] colormap windows screen window printer font setup menu microsoft expose 53 | [0.23] hockey fan pitcher nhl player braves defensive deserve team baseball 54 | [0.14] motherboard honda bike connector ac simm install monitor switch battery 55 | [0.23] christ heaven eternal sin jesus resurrection mary pray bible koresh 56 | [0.28] motherboard quadra hd simm ide isa ram mhz connector slot 57 | [0.20] audio war ship secret nuclear installation island military georgia sea 58 | [0.29] christ eternal son jesus shall lord sin scripture holy god 59 | [0.06] thanks hus bike shipping advance email hello bmw baseball honda 60 | [0.16] windows swap processor quadra disk screen setup upgrade menu ram 61 | [0.10] compound weapon accident waco rider gun fire clinton homosexual criminal 62 | [0.15] neighbor apartment woman girl bus club afraid rape burn beat 63 | [0.21] law jesus scripture worship prophet sin prophecy christ truth resurrection 64 | [0.09] hell satan evil koresh bike eternal witness heaven dog prayer 65 | [0.16] decision gay meeting stephanopoulos president schedule resolution package compromise tax 66 | [0.15] americans device commit encryption policy manufacturer establish access industry authority 67 | [0.13] universe nec cable voltage connect panel install theory wiring plug 68 | [0.33] floppy controller bio mb disk card connector rom ram adapter 69 | [0.23] simm meg motherboard slot mouse vga video card modem gateway 70 | [0.33] eus rg tor det bhj pl pd nj min que 71 | [0.18] militia minority crime homicide vancouver gun constitution seattle hitler violent 72 | [0.15] professor history muslims murder genocide massacre isa armenian jews university 73 | [0.15] patch amiga draft macintosh export binary vendor server toolkit dec 74 | [0.13] bike motherboard hd honda saturn turbo boot rear dealer gear 75 | [0.19] compound gun batf accident knife criminal death crime shoot violent 76 | [0.21] israel turkey minority jew lebanon turkish lebanese greece jews israeli 77 | [0.24] motherboard slot simm vga modem connector gateway adapter pin meg 78 | [0.19] bike thanks motorcycle ride dod surrender honda rider helmet wheel 79 | [0.41] pd mw pl mb mg oo hus mr sl wm 80 | [0.18] flyers slot puck defense deserve pitcher season leafs fan baseball 81 | [0.16] screen microsoft colormap mouse window windows swap gateway xterm boot 82 | [0.07] age consistent francisco child among staff male intelligence atlanta police 83 | [0.17] americans device manufacturer motherboard enforcement industry telephone encryption dos technology 84 | [0.11] app rider vga cable annual art vs justice weapon adams 85 | [0.22] baseball player fan reaction pitch defensive game pitcher season team 86 | [0.23] muslim genocide muslims professor armenian history population political participate armenia 87 | [0.10] electronic digital mg privacy cd internet social fbi foundation email 88 | [0.09] sale electronics thanks shipping email hus speaker audio advance monitor 89 | [0.34] ide connector floppy bio scsi scsus isa controller mb disk 90 | [0.20] militia constitution regulation libertarian government political civil rights sentence liberty 91 | [0.34] ide isa meg simm motherboard scsi controller scsus ram mhz 92 | [0.26] serial americans device enforcement wiretap escrow encryption agency des conversation 93 | [0.32] pp philadelphia leafs calgary detroit rangers wings winnipeg gm angeles 94 | [0.14] detector radar engine gear amp insurance rear ford corner car 95 | [0.17] christianity god morality faith resurrection christ sex sexual bible revelation 96 | [0.19] atheist atheism religious belief religion islam weak justify encourage prayer 97 | [0.15] bike ride motorcycle dod rider sin dog witness eternal rear 98 | [0.16] islamic islam morality faith absolute scripture marriage god teaching truth 99 | [0.13] windows window screen menu app modem microsoft encrypt nsa nt 100 | [0.08] pm amiga workstation crypto intelligence sequence prediction conference pc francisco 101 | [0.17] absolute truth moral morality christianity scripture militia objective marriage principle 102 | [0.22] flyers slot puck leafs season score nhl penalty coach canadian 103 | [0.14] shuttle mission detector rocket radar energy patient schedule foundation spacecraft 104 | [0.31] lebanese lebanon buffer israel civilian village peace innocent arab israeli 105 | [0.12] contract satellite predict shipping rocket launch demand establish remote iii 106 | [0.27] escrow wiretap cryptography nsa rsa clipper chip crypto government proposal 107 | [0.27] bike helmet ride rear honda gear seat mile bmw rider 108 | [0.27] nhl hockey coach hitter draft ice defensive player season league 109 | [0.13] atheist bush existence argument universe religious typical biblical religion doctrine 110 | [0.39] pp philadelphia winnipeg detroit calgary angeles minnesota rangers louis montreal 111 | [0.25] hitter nhl hockey stanley team player cup coach fan braves 112 | [0.57] det tor pit que pt min nj van cal vs 113 | [0.17] probe saturn rocket lunar predict remote universe isa satellite launch 114 | [0.22] holy catholic pope spirit passage teaching church bible mr tradition 115 | [0.17] nhl hockey shipping annual coach league draft stats leafs cup 116 | [0.13] modem menu swap cross windows mouse upgrade screen font port 117 | [0.13] holy professor muslims history spirit son century historical act explanation 118 | [0.07] microsoft msg window windows warrant nt patient food manager clinton 119 | [0.08] gm dos professor history muslims st bike minority wings muslim 120 | [0.19] cop gun accident criminal fire batf gang shoot kid police 121 | [0.25] pt vs calgary winnipeg la mw tor mg det dos 122 | [0.15] keyboard pc ripem shipping ibm workstation mac rsa windows button 123 | [0.15] cache cpu mhz ram vga simm colormap os xt visual 124 | [0.21] window xterm font windows screen printer app colormap menu manager 125 | [0.18] graphics keyboard amiga workstation graphic pc toolkit shipping hp render 126 | [0.20] morality moral belief god christianity bible christ faith objective existence 127 | [0.16] thanks hus appreciate db hello bh mov advance cs byte 128 | [0.19] motherboard adapter boot hd modem card setup connector hus gateway 129 | [0.12] political constitution party minority francisco civil council crime gun social 130 | [0.13] pt dos la vga hus simm vs mw mg thanks 131 | [0.15] vancouver seattle homicide victim age committee annual minority injury among 132 | [0.15] honda bike ford ac apple car dealer battery turbo kit 133 | [0.14] shall military authority division weapon respect firearm acquire violation knife 134 | [0.20] tire brake bike manual ab rear honda wheel helmet rider 135 | [0.11] shipping audio tube probe motorola remote launch microsoft contract satellite 136 | [0.16] conclusion truth argument islam evidence moral absolute murder morality objective 137 | [0.15] ab manual brake tire driver wheel manufacturer oil performance swap 138 | [0.08] shipping map jet audio shuttle schedule spacecraft lunar plant join 139 | [0.14] gun criminal crime weapon insurance batf waco homosexual economy violent 140 | [0.23] db mov bh cs byte pop americans al encryption device 141 | [0.38] ide scsi mb isa controller scsus floppy bio bus interface 142 | [0.29] setting gif pixel color jpeg compression format image display mirror 143 | [0.14] amiga menu configuration hp interface macintosh extension platform patch graphic 144 | [0.24] motherboard mhz vga hd simm isa gateway connector slot ide 145 | [0.29] gif color pixel setting jpeg compression image display mirror format 146 | [0.11] printer processing encrypt workstation key processor windows crypto microsoft serial 147 | [0.70] det tor pit que nj min van mon buf cal 148 | [0.07] principle anonymous ban human rights server legitimate user sentence trial 149 | [0.22] bike honda helmet dog gear ride rear mile turbo seat 150 | [0.19] gm bio jumper controller floppy connector st mb rom minority 151 | [0.09] printer hus thanks vga video colormap advance windows shipping hello 152 | [0.29] holy spirit son father doctrine church catholic prophet scripture pope 153 | [0.14] cipher des encrypt mode serial quadra cpu swap mhz session 154 | [0.27] dos pt pp philadelphia calgary pittsburgh louis winnipeg rangers gm 155 | [0.30] jumper mb slave connector pin disk floppy master controller bio 156 | [0.10] anonymous abuse thread user responsibility server internet legitimate unix cryptography 157 | [0.20] serial ripem enforcement escrow security rsa agent agency wiretap americans 158 | [0.15] jews islam teaching belief scripture nazi existence faith doctrine islamic 159 | [0.22] resurrection christ jesus faith satan god cult church catholic eternal 160 | [0.10] terrorist muslim troops nation population genocide warning arabs child witness 161 | [0.28] arabs israeli arab islamic islam jews jew israel muslims muslim 162 | [0.10] cop bike rock koresh compound pray satan batf rider guilty 163 | [0.12] spacecraft database map sequence distribute flight data nasa virtual papers 164 | [0.18] father armenians woman neighbor apartment rape soldier victim afraid walk 165 | [0.11] militia jews jesus jew israel prophet prophecy sentence worship constitution 166 | [0.19] revelation morality catholic belief truth absolute moral faith christianity sex 167 | [0.29] revelation jesus christian bible christ heaven faith teaching god satan 168 | [0.23] xlib window interface xt patch widget manual thanks toolkit motif 169 | [0.22] muslims israeli islam innocent jew civilian lebanese islamic jews israel 170 | [0.14] bike honda rear amp ride engine car helmet tube speaker 171 | [0.14] islamic resurrection morality christian christ muslim christianity belief god arabs 172 | [0.14] social privacy electronic internet email responsibility unix digital network protocol 173 | [0.19] bike insurance helmet honda turbo gear seat rear saturn ride 174 | [0.34] gm hd pd mb mg mw st rg eus pl 175 | [0.14] motherboard thanks hus shipping gateway monitor modem slot video brand 176 | [0.40] calgary pp detroit philadelphia pittsburgh angeles winnipeg montreal louis rangers 177 | [0.20] converter visual xt widget string vendor default cache colormap export 178 | [0.14] moral islamic morality militia murder amendment punishment islam sentence innocent 179 | [0.17] flyers puck hitter career pitcher defense season slot deserve pitch 180 | [0.15] bike honda baseball helmet rider ride hitter sport fan dog 181 | [0.70] det tor que pit min nj van cal buf mon 182 | [0.20] simm meg motherboard card modem pin hus ram monitor window 183 | [0.08] bike saturn battery baseball coach player transmission rear honda helmet 184 | [0.22] woman rape neighbor kill daughter apartment afraid burn alive tear 185 | [0.15] fuel detector orbit flight shuttle engine voltage radar cell launch 186 | [0.19] vga motherboard mouse microsoft windows gateway cpu swap ram adapter 187 | [0.21] israeli lebanese israel jews jew turkey turkish nazi arabs armenia 188 | [0.30] nec wiring wire cable connect circuit outlet voltage install panel 189 | [0.37] db byte mov bh cs push al pop ax gas 190 | [0.47] van det pit tor que nj min shipping de cal 191 | [0.22] motherboard adapter audio connector ram monitor gateway hd cable meg 192 | [0.29] rom pin bio connector controller jumper floppy feature card monitor 193 | [0.27] jesus christianity doctrine bible satan god belief faith scripture christ 194 | [0.36] ide scsi scsus isa controller meg floppy mb disk interface 195 | [0.20] audio nuclear ship secret installation georgia south war azerbaijan southern 196 | [0.24] arab arabs peace jews israel country jew buffer lebanon adam 197 | [0.15] bike honda rear seat ford dealer saturn helmet pitch sport 198 | [0.16] variable winner judge abuse entry award submit compile execute rule 199 | [0.21] io input buffer int output buf winner judge variable null 200 | [0.23] puck defensive score flyers slot player nhl roger season hockey 201 | [0.35] controller ide meg floppy bio scsi scsus interface disk isa 202 | [0.15] woman girl massacre greek muslim hide turks witness turkish lay 203 | [0.21] administration russia senior russian official fund food sector stephanopoulos package 204 | ========================================================================== 205 | Average Topic Coherence = 0.204785720489 206 | (IPC_MAP)[haiti]s1362249: bash 20news.sh 207 | Computing word occurrence... 208 | Computing the observed coherence... 209 | [0.38] probe saturn orbit mission spacecraft lunar moon solar rocket shuttle 210 | [0.20] window button xterm colormap expose font manager map app motif 211 | [0.19] christianity scripture belief absolute islamic christian church teaching morality god 212 | [0.26] bike rear honda helmet gear mile brake seat ride engine 213 | [0.25] simm modem bio card floppy vga rom controller port disk 214 | [0.18] audio connector amp voltage switch battery motherboard signal noise mouse 215 | [0.20] printer slot windows port ram vga card simm mhz cpu 216 | [0.18] bike helmet ride hitter dog rider kit rear motorcycle ryan 217 | [0.17] morality moral objective nsa crypto escrow scheme des rsa pgp 218 | [0.19] batf waco warrant compound fbi regulation assault detector weapon radar 219 | [0.15] crypto nsa wiretap pgp detector clipper phone warrant radar rf 220 | [0.27] serial session escrow agent encrypt crypto nsa key chip random 221 | [0.13] battery amp car dealer voltage honda detector radar electronics engine 222 | [0.11] kit configuration xterm binary server backup keyboard voltage tape nec 223 | [0.22] existence faith belief atheism god atheist evidence observation christianity moral 224 | [0.25] cult batf koresh fbi tank compound waco guilty detroit fire 225 | [0.26] mhz quadra motherboard ram meg simm processor card video modem 226 | [0.33] disk controller floppy bio rom feature mb interface virtual card 227 | [0.58] det tor pit que cal dos van buf nj min 228 | [0.16] morality moral objective scientific murder existence science observation belief definition 229 | [0.20] gateway windows mouse screen window port driver meg monitor card 230 | [0.11] predict capability module universe contract remote satellite vehicle star physical 231 | [0.16] constitution minority libertarian government majority senate regulation federal abuse principle 232 | [0.21] windows tape backup kit driver floppy upgrade scsi disk configuration 233 | [0.26] prophecy nation prophet son jesus spirit mouth holy passage father 234 | [0.14] punishment innocent murder sex moral batf execute homosexual society blame 235 | [0.17] player braves coach roger team baseball defensive playoff scientific stanley 236 | [0.25] bike helmet honda ride rear turbo rider gear motorcycle bmw 237 | [0.16] azerbaijan walk neighbor truck car father fight town apartment armenians 238 | [0.12] anonymous responsibility user abuse threat digital social privacy server unix 239 | [0.35] arab arabs israel lebanese lebanon civilian israeli adam palestinian jew 240 | [0.07] thanks hus advance sale modem mw brand dos hello fax 241 | [0.08] constitution ram card simm vga meg percent court council muslim 242 | [0.14] privacy cryptography graphics electronic social network digital computing unix cipher 243 | [0.21] shipping motherboard meg printer sale floppy slot ram monitor upgrade 244 | [0.20] muslims jews jewish prophet greece arab greek minority israel arabs 245 | [0.09] honda oo sexual homosexual bike knife uk commit male rider 246 | [0.17] cipher des encrypt serial attack random key gif pair jpeg 247 | [0.09] cancer child patient impact laboratory violate age medical education safety 248 | [0.29] season nhl draft playoff hockey cup conference ice flyers team 249 | [0.17] cult koresh batf islamic fbi compound christ islam muslims marry 250 | [0.11] passage death act hang mr conclude fall greek argument die 251 | [0.26] crime homicide vancouver minority gun commit violent seattle death victim 252 | [0.29] meg ide simm scsus scsi isa pc bus interface quadra 253 | [0.16] islam morality moral islamic marriage muslim sex sexual homosexual objective 254 | [0.18] atheist religious atheism argument conclusion religion false existence evidence argue 255 | [0.13] hitter food pitch baseball reaction msg patient ball fan eat 256 | [0.09] hus thanks advance hello email fax appreciate greatly kent motorcycle 257 | [0.14] insurance fan coverage canadian health hospital doctor nhl stanley private 258 | [0.33] motherboard hd connector ide floppy boot meg controller scsi scsus 259 | [0.19] vga printer adapter motherboard shipping monitor upgrade video macintosh meg 260 | [0.20] serial americans escrow enforcement chip agency agent commit algorithm security 261 | [0.29] sexual homosexual sex percent marriage gay male marry man population 262 | [0.24] motherboard meg hd slot floppy boot mb monitor pin simm 263 | [0.25] nec voltage panel connect circuit wire cable install wiring permit 264 | [0.22] encryption device commit americans privacy electronic policy establish protect access 265 | [0.09] sale shipping bike manual thanks hus brand music price ride 266 | [0.16] graphics windows utility database processing fax graphic ftp manual virtual 267 | [0.27] holy prophet spirit father son prophecy jesus punishment verse passage 268 | [0.17] compatible ripem rsa pgp port macintosh data contrary simm license 269 | [0.15] cache converter visual widget application io xt resource rsa ripem 270 | [0.20] civilian innocent human territory rights israeli israel village border living 271 | [0.27] christianity god bible eternal christian belief faith heaven sin jesus 272 | [0.09] gay atheism atheist president stephanopoulos symptom universe christianity meeting compromise 273 | [0.18] gif jpeg setting compression format xterm font image faster mirror 274 | [0.12] input output judge file winner io program variable abuse format 275 | [0.25] car engine transmission gear dealer honda brake ford mile tire 276 | [0.26] jumper slave mb setting pin connector mode install master hd 277 | [0.11] insurance water plant fund engine billion private hospital fuel shuttle 278 | [0.25] hitter defensive player braves roger team baseball pitch coach score 279 | [0.20] serial enforcement cipher encrypt mode des agent block session protocol 280 | [0.13] community young opportunity officer country challenge sector american education russian 281 | [0.22] converter widget mit variable resource null application toolkit cache visual 282 | [0.23] radar tire detector car brake manual rear transmission gear speed 283 | [0.19] absolute scripture morality moral murder truth principle culture interpretation christianity 284 | [0.70] tor det pit que min mon cal buf nj van 285 | [0.34] coach team stanley leafs cup nhl wings playoff fan hockey 286 | [0.19] fbi compound witness woman koresh waco shoot walk batf kill 287 | [0.44] gm pd mg mw pl eus rg hd mr bhj 288 | [0.17] law islam muslims worship muslim islamic cite weak teaching jewish 289 | [0.14] mouse bike audio amp connect connector switch noise inch battery 290 | [0.17] palestinian arabs arab waco adam warrant israeli israel islamic violate 291 | [0.23] rocket billion flight shuttle orbit plant fuel nasa fund lunar 292 | [0.20] colormap expose window app default null visual button screen swap 293 | [0.12] gm witness st wings bike trial judge oo ride leafs 294 | [0.22] doctrine revelation bible scripture jesus reject law truth lord god 295 | [0.12] professor history soul muslims nazi muslim homosexual sexual soviet mountain 296 | [0.22] mac mhz cpu os pc ibm ide printer ram hardware 297 | [0.57] det tor pit que cal min buf nj mon bike 298 | [0.15] app rider cable annual justice weapon art origin frank vs 299 | [0.21] shipping motherboard vga connector sale adapter meg slot manual monitor 300 | [0.20] audio secret ship nuclear installation dos island war south georgia 301 | [0.27] militia sentence constitution amendment bear arm shall army organize interpretation 302 | [0.08] child safety seat consistent injury atlanta patient season age gang 303 | [0.26] militia sentence amendment arm bear constitution regulation organize army shall 304 | [0.14] fbi eternal kid koresh compound hell resurrection sin jesus fire 305 | [0.36] pp detroit calgary philadelphia angeles rangers pittsburgh louis winnipeg shot 306 | [0.10] honda bike hus modem thanks shipping motherboard sale meg monitor 307 | [0.22] resurrection marriage church tradition christ catholic doctrine marry prophet jesus 308 | [0.22] deserve fbi tank tear fire compound hall sick batf koresh 309 | [0.13] hus thanks hello advance graphic windows window appreciate fax greatly 310 | [0.13] beat woman apartment neighbor prophecy rape prophet afraid steal verse 311 | [0.20] francisco league police angeles deny committee times san los pen 312 | [0.35] pt calgary winnipeg pp vs philadelphia detroit montreal pit pittsburgh 313 | [0.26] leafs stanley rangers fan baseball braves wings cup atlanta montreal 314 | [0.17] neighbor troops apartment father woman town fight massacre defend february 315 | [0.17] swap windows screen vga ram nt card modem gateway window 316 | [0.33] io input output buffer buf postscript char file null int 317 | [0.10] sexual judge homosexual kent rider bike ryan sin marry gay 318 | [0.26] knife gun crime criminal weapon violent death homicide victim gang 319 | [0.22] gm stanley cup coach leafs wings braves rangers witness fan 320 | [0.15] mouse gateway microsoft screen swap jump resolution modem window menu 321 | [0.17] shipping motherboard sale brand slot connector pin speaker floppy remote 322 | [0.26] resurrection christ heaven sin christian bible god satan scripture biblical 323 | [0.16] ide link scsus cross scsi isa simm unit cache byte 324 | [0.13] economy russia government fund community officer sector country russian program 325 | [0.07] bike hus honda thanks simm advance rear modem hello motorcycle 326 | [0.19] ftp graphics graphic database amiga workstation render map processing macintosh 327 | [0.10] hus child health trial patient disease mouse among culture age 328 | [0.10] police francisco file privacy electronic league violate digital rsa network 329 | [0.34] byte db mov bh cs push al gas pop east 330 | [0.12] window button colormap hus expose font modem vga windows map 331 | [0.31] nhl league hockey draft team cup atlanta montreal minnesota rangers 332 | [0.12] hell lord fear bike pray prophet prophecy nation prayer jesus 333 | [0.19] export postscript implementation font kit binary dec vendor motif patch 334 | [0.41] armenia turks armenian armenians turkey genocide turkish azerbaijan massacre nazi 335 | [0.15] manual brake tire ab fuel car specify wheel oil performance 336 | [0.22] hitter pitcher braves pitch deserve hit baseball ryan defense average 337 | [0.15] radar rocket configuration nasa lunar japanese signal satellite international spacecraft 338 | [0.32] civilian lebanese attack israel peace buffer lebanon innocent village israeli 339 | [0.44] mg mw hd wm sl mr mi pl pd ps 340 | [0.17] gateway screen mouse hd swap motherboard quadra meg card boot 341 | [0.12] muslims homosexual century history jews genocide tradition page pope professor 342 | [0.17] bike honda bmw helmet shipping kit rear ride ac sport 343 | [0.12] wiretap clinton criminal steal escrow gun government libertarian proposal crypto 344 | [0.56] pd pl rg eus bhj oo mg ax sl mr 345 | [0.38] probe mission orbit lunar saturn rocket satellite moon launch spacecraft 346 | [0.24] americans commit encryption criminal enforcement government device protect administration policy 347 | [0.18] pp puck module printer score calgary slot penalty shot period 348 | [0.19] mac hardware pc ibm os keyboard mhz apple vga microsoft 349 | [0.19] minority greek hitler greece jews liberal islam islamic turkey turkish 350 | [0.16] printer windows compatible port motherboard utility shipping manual vga video 351 | [0.17] koresh hell sin punishment fbi god evil heaven fear eternal 352 | [0.12] ac defend plane civilian search shoot armenia pl soldier content 353 | [0.25] converter widget application colormap cache visual xt manual xlib resource 354 | [0.12] honda audio amp manual shipping ac kit motorola turbo detector 355 | [0.17] msg food reaction compound sick patient blood symptom human study 356 | [0.12] ship installation jew israeli audio arab nuclear military arabs sentence 357 | [0.09] defense society international staff coverage foundation chapter committee cancer conference 358 | [0.26] slot puck flyers penalty shoot defense playoff goal shot score 359 | [0.15] extension application motif keyboard widget windows toolkit spacecraft menu virtual 360 | [0.17] muslim history marry muslims professor genocide witness criminal turks historical 361 | [0.15] decision president meeting resolution deny package stephanopoulos schedule vote consideration 362 | [0.14] probe playoff rangers coach lunar puck penalty saturn wings orbit 363 | [0.12] file directory assault sale batf senate prohibit gun february journal 364 | [0.09] mw hus mg thanks oo fax sale modem advance mailing 365 | [0.20] bike dog ride helmet cop honda rider ryan bmw sport 366 | [0.21] workstation ftp hp dec unix pc graphics turbo graphic database 367 | [0.25] jews turkish turkey greek minority citizen jewish nazi french greece 368 | [0.27] crypto cryptography security algorithm nsa pgp escrow rsa clipper data 369 | [0.33] controller floppy rom disk bio mb virtual interface feature card 370 | [0.18] batf fbi bike compound waco koresh tear dog helmet ride 371 | [0.19] graphic amiga processing utility graphics image database printer ftp analysis 372 | [0.21] font windows xterm menu printer screen microsoft window postscript character 373 | [0.18] cop gang police training officer fire knife criminal merely shoot 374 | [0.10] dos cd library mit det cambridge animal van civil tor 375 | [0.07] jew islamic islam deny arab marriage civil proposal compatible attitude 376 | [0.16] rider vs pt print annual copy justice calgary tie philadelphia 377 | [0.21] truth absolute belief scripture faith interpretation revelation culture god pope 378 | [0.08] sexual homosexual prohibit journal handgun islam gun states islamic file 379 | [0.10] link cross unit windows cache simm hus disk gateway modem 380 | [0.13] escrow crypto detector nsa serial radar database wiretap rf government 381 | [0.16] msg food scientific reaction study pain patient treatment eat human 382 | [0.26] rsa ripem cryptography pgp strong cipher encrypt ftp compression des 383 | [0.31] meg ide scsus pc scsi isa workstation spec interface bus 384 | [0.17] baseball fan hitter pitch msg coach player leafs team ball 385 | [0.18] install connect nec voltage cable panel plug trip wire equipment 386 | [0.22] atheist religious religion atheism god belief islam nation typical assumption 387 | [0.31] mw mb hd mg wm bh db ps nt pd 388 | [0.23] americans commit encryption government device administration criminal protect enforcement economic 389 | [0.19] cop flyers puck penalty defense slot police goal shot ice 390 | [0.24] coach cup stanley gm player wings team baseball braves roger 391 | [0.25] leafs wings stanley team playoff defensive coach cup braves player 392 | [0.21] ride dod motorcycle bmw club patch bike helmet pin flame 393 | [0.13] fbi gay koresh compound tear laugh deserve compromise gas shoot 394 | [0.15] bike helmet cop ride gear mile accident engine car dog 395 | [0.30] nhl hockey league season draft team bay minnesota jet cup 396 | [0.26] pope catholic church teaching holy biblical doctrine resurrection tradition council 397 | [0.16] colormap window xterm visual echo expose item map int manager 398 | [0.20] soldier civilian lebanese village cop gang lebanon justify terrorist batf 399 | [0.13] muslim greek lebanese terrorist arab minority population armenia greece soul 400 | [0.10] argument conclusion false valid argue islam child pro commit therefore 401 | [0.12] shipping cd sale speaker bike motorola amp dealer honda ride 402 | [0.18] font xterm postscript server directory binary mit format mailing amiga 403 | [0.25] wiretap crypto clipper privacy encryption conference escrow government cryptography chip 404 | [0.22] braves pitcher hitter player defensive pitch baseball deserve ab solid 405 | [0.11] bike rear rider ride register convince brake motorcycle life eternal 406 | [0.28] lebanese peace lebanon israel stephanopoulos soldier attack buffer troops israeli 407 | [0.04] anonymous hus user gateway hello abuse windows thanks ban marriage 408 | [0.14] hitter season cop defensive baseball player puck pitch braves car 409 | ========================================================================== 410 | Average Topic Coherence = 0.201569879576 411 | (IPC_MAP)[haiti]s1362249: bash 20news.sh 412 | Computing word occurrence... 413 | Computing the observed coherence... 414 | [0.14] motherboard bike gear quadra turbo manual hd simm mhz engine 415 | [0.58] tor det que cal pit min buf nj batf mon 416 | [0.69] det tor pit que cal nj la van min buf 417 | [0.13] saturn cop crime gun turbo criminal accident insurance violent dealer 418 | [0.11] simm jew motherboard slot meg soldier islamic islam lebanese civilian 419 | [0.17] slot puck playoff defense pitch penalty season cal wings game 420 | [0.17] pope act church catholic legitimate passage command prophet tradition worship 421 | [0.22] mr scripture greek passage biblical doctrine tradition church bible revelation 422 | [0.27] rsa pgp nsa escrow encrypt serial crypto key clipper cryptography 423 | [0.07] hus hello thanks fax mailing advance virtual email conference ftp 424 | [0.26] install panel connect voltage nec audio circuit cable wire wiring 425 | [0.19] cache converter gif setting compression jpeg visual pixel format mirror 426 | [0.25] baseball defensive pitch hitter coach player braves pitcher roger team 427 | [0.21] jews minority greek greece turkish page jew turks historical reject 428 | [0.20] existence atheist religious god atheism belief religion scientific conclusion argue 429 | [0.19] window button expose colormap manager xterm map item echo app 430 | [0.18] jpeg gif menu setting windows microsoft format injury image pixel 431 | [0.20] mouse screen window microsoft windows vga font printer menu colormap 432 | [0.19] serial bio controller rom disk floppy cipher session des encrypt 433 | [0.33] io input output buf char file buffer postscript int printf 434 | [0.33] lebanese buffer soldier israel peace lebanon civilian attack village terrorist 435 | [0.17] msg food reaction blood patient compound human sick symptom treatment 436 | [0.57] pt tor det pit que vs la cal min mon 437 | [0.22] app hockey nhl cup season team rider player playoff montreal 438 | [0.28] leafs rangers stanley wings fan pittsburgh playoff cup coach baseball 439 | [0.10] civilian nasa houston degree training participate space pm remote flight 440 | [0.12] fan baseball playoff hitter leafs team pitch ride rangers eat 441 | [0.14] electronic unix privacy cryptography social ftp digital directory windows network 442 | [0.17] son daughter wife muslims murder massacre arabs soldier genocide muslim 443 | [0.32] bio floppy disk controller rom connector feature interface mb digital 444 | [0.16] sexual homosexual sex morality msg gay food moral christianity marriage 445 | [0.21] crypto wiretap escrow nsa proposal phone government clipper police brad 446 | [0.45] dos tor det cal pit gm que van min mon 447 | [0.11] marriage pope islamic israeli catholic rights jew human muslims nazi 448 | [0.42] pp calgary philadelphia angeles louis pt detroit pittsburgh period los 449 | [0.25] serial protocol agent session enforcement escrow chip wiretap fbi encrypt 450 | [0.19] configuration xterm font vendor patch motif mit interface toolkit implementation 451 | [0.12] neighbor apartment beat woman rape ride helmet afraid tear father 452 | [0.17] windows swap colormap disk floppy gif controller ram bio jpeg 453 | [0.13] military license violation division weapon application firearm shall inch acquire 454 | [0.26] sentence militia amendment bear constitution arm organize shall rights cite 455 | [0.07] rights living human serial murder solve arabs territory system transfer 456 | [0.30] gm mw det tor mg pit vs cal que wm 457 | [0.10] msg food reaction eternal sin satan sexual evil fear koresh 458 | [0.08] koresh fbi cult heaven fan sin hell baseball prayer tear 459 | [0.21] motherboard connector shipping vga meg monitor adapter sale floppy upgrade 460 | [0.01] dos cd civil shipping library baseball war bible animal wing 461 | [0.20] slot leafs playoff puck seconds penalty goal montreal game draft 462 | [0.29] pd pl eus rg bhj audio ship oo mg installation 463 | [0.19] bike helmet ride honda kit turbo rear oil rider motorcycle 464 | [0.18] insurance canadian nhl rangers player team fee pp wings pittsburgh 465 | [0.24] francisco police league angeles deny political san los jewish intelligence 466 | [0.09] child consistent safety injury patient age laboratory increase procedure cycle 467 | [0.26] simm isa pd ide scsi mac pc cpu scsus meg 468 | [0.10] saturn defensive ford dealer hitter car player rear amp turbo 469 | [0.15] shipping honda bike sale car price dealer bmw turbo thanks 470 | [0.30] ide scsus scsi meg pc isa interface quadra mhz mb 471 | [0.13] shipping sale manual windows printer microsoft hus thanks graphics vga 472 | [0.25] sentence militia amendment constitution arm bear army organize shall cite 473 | [0.14] young community opportunity country economy congress sector challenge education percent 474 | [0.13] encrypt windows port serial modem hus nt window processor swap 475 | [0.14] swap font xterm export xt vendor path patch mit manager 476 | [0.14] church bible hus christ christian biblical thanks catholic heaven sin 477 | [0.13] player baseball deserve pitcher team career gun gang knife stats 478 | [0.14] sexual homosexual sex gay male ryan mary sin christ file 479 | [0.30] bike ride bmw honda helmet dod motorcycle rider rear dog 480 | [0.22] puck flyers slot shot penalty goal board zone nec playoff 481 | [0.26] attack civilian innocent lebanese harm village lebanon buffer israel prepare 482 | [0.15] stephanopoulos decision meeting resolution president consideration deny gay senate context 483 | [0.13] audio printer keyboard windows amp pc switch microsoft vga noise 484 | [0.24] armenia plane defend turkey nazi conflict shoot armenians azerbaijan search 485 | [0.29] batf fbi compound koresh waco tear cult agent deserve gas 486 | [0.27] mhz pc ibm mac ram cpu os ide scsus cache 487 | [0.23] os cpu mac windows ram ibm vga hardware mhz pc 488 | [0.22] americans criminal commit enforcement device encryption protect justice government administration 489 | [0.24] floppy disk windows controller bio connector rom adapter backup tape 490 | [0.16] probe saturn pitch season fan hitter mission lunar soviet moon 491 | [0.12] windows link ram vga unit cross port cache swap video 492 | [0.16] koresh fbi compound batf deserve satan bike cult dog tear 493 | [0.05] msg food voltage amp screen processor surrender noise fuel battery 494 | [0.30] revelation faith truth scripture bible interpretation belief doctrine absolute god 495 | [0.17] professor jews history jewish marriage jew century sea islamic nazi 496 | [0.20] menu button window colormap expose screen visual event id app 497 | [0.24] oo gm pp mw mg calgary cd wm pd rangers 498 | [0.12] abuse anonymous user ripem server harm responsibility rsa legitimate posting 499 | [0.24] motherboard meg slot connector vga adapter video floppy simm amiga 500 | [0.19] nec install connect cable voltage wiring circuit wire button insurance 501 | [0.17] islam muslims murder muslim blame moral morality innocent islamic society 502 | [0.25] player defensive braves pitcher team hitter baseball hr deserve season 503 | [0.21] prophecy nation prophet verse jesus mouth execute king matthew passage 504 | [0.17] moral morality murder objective animal natural libertarian regulation punishment goal 505 | [0.15] marriage arab deny jew justice compatible policy israeli arabs civil 506 | [0.22] vga port motherboard modem adapter connector shipping video monitor connect 507 | [0.13] judge winner compile entry variable abuse rule input award io 508 | [0.10] mouse child hus trial patient disease laboratory health age cancer 509 | [0.19] law worship jesus jewish cite christian sunday passage weak paul 510 | [0.20] americans palestinian commit device arab policy encryption enforcement administration agency 511 | [0.24] homicide assault vancouver handgun victim death seattle gun file crime 512 | [0.22] motherboard simm slot port ide modem isa quadra adapter hd 513 | [0.22] baseball rangers fan coach leafs stanley team cup braves hitter 514 | [0.27] atheist belief atheism scripture religious absolute god religion truth faith 515 | [0.30] widget toolkit button window motif application xt xlib colormap xterm 516 | [0.13] libertarian proposal crypto nsa escrow civil government arab wiretap regulation 517 | [0.09] shipping annual june sale vs print cambridge ma manual april 518 | [0.12] conclusion argument homosexual proof false scientific conclude valid item evidence 519 | [0.10] window xterm link colormap echo unit app null expose font 520 | [0.14] windows graphics xterm graphic printer postscript hus ftp unix toolkit 521 | [0.20] bike helmet honda ride rear bmw battery seat hitter engine 522 | [0.13] bike motherboard simm monitor turbo honda quadra apple rear slot 523 | [0.09] minority principle crime bias constitution knife death majority congress criminal 524 | [0.16] muslims satan muslim koresh kent islam christian islamic cult marriage 525 | [0.38] pp detroit winnipeg philadelphia leafs calgary pt montreal toronto minnesota 526 | [0.18] establish jew arab adam jews war guilty palestinian cop jewish 527 | [0.06] isa water remote plant predict compression contract market saturn motorola 528 | [0.27] jumper slave pin connector install hd setting mb sl configuration 529 | [0.23] militia minority constitution government greece amendment greek organize bear rights 530 | [0.21] americans commit device policy enforcement administration encryption economic propose criminal 531 | [0.18] flight fuel water plant orbit stage msg moon solar rocket 532 | [0.15] civilian defend compromise arabs lebanese lebanon conflict turkey turks arab 533 | [0.25] wings leafs playoff rangers pittsburgh philadelphia pp montreal shipping stanley 534 | [0.26] holy christ son spirit marriage father marry church sin mary 535 | [0.24] doctrine resurrection revelation church marriage sin scripture catholic christianity christ 536 | [0.17] woman witness massacre batf girl soldier shoot child fbi compound 537 | [0.15] saturn dealer radar honda insurance detector car mhz turbo price 538 | [0.15] jumper slave master configuration xt link xlib cross interface protocol 539 | [0.22] resurrection prayer pray jesus christ hell fear rise sin heaven 540 | [0.21] connector motherboard pin vga signal meg cable hd macintosh boot 541 | [0.26] pitcher hitter braves pitch baseball player deserve team season career 542 | [0.23] font xterm client patch postscript server compile xt mit export 543 | [0.12] homosexual sex sexual gay deserve msg player gang male christianity 544 | [0.10] woman walk committee beat dog club rape neighbor afraid toward 545 | [0.11] mouse bike screen battery amp inch microsoft button rear audio 546 | [0.14] printer shipping utility xterm microsoft manual windows sale postscript workstation 547 | [0.11] cipher des americans product block toolkit processing motif device round 548 | [0.21] cop gang batf criminal fire koresh waco gun police fbi 549 | [0.15] keyboard exercise pc workstation shipping platform warn processing macintosh compatible 550 | [0.12] turkey turkish nazi hitler jews islamic islam sex homosexual militia 551 | [0.14] pd eus pl rg bhj jews installation ship audio lebanese 552 | [0.26] graphics hp workstation amiga macintosh ibm platform dec pc graphic 553 | [0.23] population azerbaijan muslim region armenian foreign percent armenia marry army 554 | [0.10] bike honda msg dealer detector car food bmw radar gear 555 | [0.06] amp bike engine helmet battery plant food pain pressure fuel 556 | [0.11] modem windows port hus pgp microsoft thanks serial fax hello 557 | [0.12] century professor history muslims jewish jews village rise gay despite 558 | [0.17] graphics ftp crypto amiga macintosh workstation data ibm hp database 559 | [0.32] stanley leafs rangers wings coach cup playoff team gm fan 560 | [0.10] secretary community jews role young component education liberal challenge opportunity 561 | [0.12] morality moral objective simm crypto quadra processor natural animal modem 562 | [0.19] screen mouse window windows colormap gateway microsoft menu swap monitor 563 | [0.17] bike honda dog bmw ride waco sport rear car kit 564 | [0.36] gm rangers hockey wings cup leafs playoff stanley nhl detroit 565 | [0.25] xterm colormap client mouse font window resource widget server application 566 | [0.16] batf waco administration russia official plane senior fbi compound gas 567 | [0.12] bomb civilian muslims prophet innocent islam prophecy guilty atheist nation 568 | [0.10] signal map frequency wm attitude processing radar lunar society spacecraft 569 | [0.20] islam sexual islamic homosexual sex gay muslims muslim marriage arab 570 | [0.33] ide meg isa scsus scsi mb pc bus spec hd 571 | [0.21] pitch defense braves hitter pitcher hit season ball flyers coach 572 | [0.37] bh byte db cs mov push gas ax al ah 573 | [0.29] christ doctrine jesus spirit god holy son bible biblical revelation 574 | [0.08] bike quadra mouse helmet gateway apple simm amp honda gear 575 | [0.13] federal government waco clinton libertarian batf regulation islamic civil amendment 576 | [0.29] spacecraft lunar map rocket nasa shuttle orbit satellite flight saturn 577 | [0.12] crime msg insurance violent food cite vancouver drug gun homicide 578 | [0.12] mr shoot walk wife witness february body massacre truck son 579 | [0.16] woman neighbor beat rape apartment girl kill afraid bus walk 580 | [0.22] ripem rsa compatible des port pgp cipher cryptography macintosh implementation 581 | [0.20] bike helmet ride rear mouse rider dog dod honda turbo 582 | [0.21] graphics amiga graphic image module processing object render database ray 583 | [0.27] orbit solar spacecraft flight fuel plant module lunar km moon 584 | [0.23] package myers stephanopoulos president decision meeting gay yesterday february senate 585 | [0.19] christ sin doctrine god heaven shall law council holy eternal 586 | [0.26] scripture truth god absolute belief faith interpretation revelation atheism christianity 587 | [0.18] homicide vancouver hockey handgun seattle city percent nhl scientific population 588 | [0.22] god fear faith doctrine truth pray christ bible absolute revelation 589 | [0.10] brake tire ab wheel plant detector symptom hitler manual driver 590 | [0.10] baseball fan ride rangers leafs coach dog bike stanley pitcher 591 | [0.15] attitude processing saturn shuttle achieve orbit spacecraft frequency analysis conduct 592 | [0.12] islamic morality islam objective kent muslim christianity marriage moral nsa 593 | [0.09] universe star cipher frequency physical theory relation predict seconds des 594 | [0.07] gun constitution amendment digital gateway file prohibit protocol directory february 595 | [0.12] serial modem escrow amp audio voltage processor chip clipper encrypt 596 | [0.31] ide scsus isa mhz cpu pc meg scsi mb mac 597 | [0.09] simm bike honda thanks slot motherboard bmw det ride hus 598 | [0.16] app rider annual cable origin justice art appear miller league 599 | [0.23] brake rear tire ab seat manual gear car mile bike 600 | [0.25] motherboard simm meg slot quadra ram gateway hd modem card 601 | [0.14] islam arab conclusion israeli islamic jew commit arabs murder conclude 602 | [0.21] police waco angeles fbi francisco los times san armenia warrant 603 | [0.11] mw shipping sale hus printer manual thanks card meg motherboard 604 | [0.13] violation shall violate weapon division military respect device firearm dangerous 605 | [0.10] conference foundation professor mit pm sequence washington committee rider session 606 | [0.31] pd mw det tor pit mg pl que van hd 607 | [0.07] shipping sale thanks hus manual fax june modem annual vs 608 | [0.15] manual brake driver performance speed ab ide automatic scsi tire 609 | [0.21] batf waco clinton compound fbi koresh kid drug assault criminal 610 | [0.16] scientific science objective existence morality observation evidence truth belief faith 611 | [0.14] launch installation audio nasa satellite island war ship shuttle south 612 | [0.15] windows button printer microsoft window menu app graphic hus thanks 613 | [0.19] vga ram gateway monitor windows mouse screen nt video window 614 | ========================================================================== 615 | Average Topic Coherence = 0.192214898517 616 | 617 | -------------------------------------------------------------------------------- /coherence_from_paper_script/AVTIM_50: -------------------------------------------------------------------------------- 1 | (IPC_MAP)[haiti]s1362249: bash 20news.sh 2 | Computing word occurrence... 3 | Computing the observed coherence... 4 | [0.24] entry winner variable buf int remark char judge oname abuse 5 | [0.27] jesus christ verse scripture god resurrection prophecy christian prophet heaven 6 | [0.25] braves pitcher score team player hitter pitch defensive coach playoff 7 | [0.33] controller floppy bio disk interface rom mb connector jumper scsus 8 | [0.36] pt det calgary tor vs pit winnipeg detroit que pp 9 | [0.31] ide scsi motherboard scsus isa quadra meg hd floppy mb 10 | [0.21] militia israeli israel sentence homicide arabs arab lebanese lebanon amendment 11 | [0.21] privacy anonymous establish americans policy security enforcement protocol encryption administration 12 | [0.38] eus rg det pd tor mr bhj mg pl mw 13 | [0.27] mb jumper meg ide simm port scsus connector isa controller 14 | [0.08] apartment hitter woman neighbor father fan dog deserve soldier braves 15 | [0.24] hitter braves pitch baseball pitcher deserve player team defensive coach 16 | [0.27] turkish armenian israel genocide arab armenians jews turks muslim armenia 17 | [0.21] secretary stephanopoulos president senate congress meeting myers education decision community 18 | [0.25] armenians soldier woman apartment armenian child neighbor azerbaijan gang troops 19 | [0.19] motherboard quadra shipping processor gateway meg ide simm apple mhz 20 | [0.17] amp voltage wiring brake outlet circuit cable wire printer radar 21 | [0.16] islam israeli morality islamic israel arabs moral arab marriage human 22 | [0.19] motherboard simm dos windows mhz adapter quadra meg setup video 23 | [0.16] widget unix colormap graphics toolkit visual macintosh directory data motif 24 | [0.25] armenian armenians azerbaijan woman apartment neighbor armenia soldier kill hide 25 | [0.32] nhl hockey league draft season winnipeg team minnesota rangers cup 26 | [0.21] crime gun batf criminal constitution assault violent violate government weapon 27 | [0.26] israel islamic israeli arab islam palestinian militia lebanese lebanon muslims 28 | [0.22] toolkit xterm font export xt platform server widget binary workstation 29 | [0.22] motherboard meg simm vga modem printer slot port adapter ram 30 | [0.26] christ god heaven faith christian scripture christianity jesus revelation marriage 31 | [0.27] serial enforcement encrypt agency escrow encryption administration chip security session 32 | [0.27] shuttle nasa probe satellite lunar mission scientific module conduct spacecraft 33 | [0.30] pp philadelphia tor calgary det detroit winnipeg pt pit st 34 | [0.20] firearm weapon militia shall homicide constitution acquire violation handgun license 35 | [0.21] brake bike tire helmet manual speed gear rear engine honda 36 | [0.23] xterm xt client motif xlib widget visual swap toolkit font 37 | [0.25] god belief faith truth jesus christian church conclusion doctrine scripture 38 | [0.25] conclusion christian god jesus christianity faith argument christ doctrine belief 39 | [0.33] wiring nec wire outlet circuit panel cable permit connect neutral 40 | [0.41] armenian turks muslim genocide muslims armenia turkish armenians population massacre 41 | [0.23] passage jesus god faith belief christian christ prophecy atheist holy 42 | [0.32] ide scsus scsi mb connector controller meg jumper bio isa 43 | [0.19] stephanopoulos president turkish greek myers turkey turks decision armenian armenians 44 | [0.34] controller cache floppy bio disk rom scsus scsi interface mb 45 | [0.21] ftp workstation dec anonymous format directory graphics platform macintosh hp 46 | [0.27] hitter braves player defensive score coach season team pitcher career 47 | [0.19] widget xt font server pc visual toolkit xterm converter xlib 48 | [0.10] turbo manual audio bike kit shipping quadra amp brake nec 49 | [0.32] detroit nhl montreal rangers hockey league wings playoff season leafs 50 | [0.20] db byte mov bh cs stephanopoulos president myers decision meeting 51 | [0.43] buf output io char oname printf int stream input file 52 | [0.22] enforcement serial americans encryption ripem privacy protocol encrypt law device 53 | [0.30] nhl hockey league playoff season team puck cup coach score 54 | ========================================================================== 55 | Average Topic Coherence = 0.250585946611 56 | (IPC_MAP)[haiti]s1362249: bash 20news.sh 57 | Computing word occurrence... 58 | Computing the observed coherence... 59 | [0.16] apartment woman neighbor jesus armenians tear daughter soldier hide afraid 60 | [0.32] nhl hockey wings rangers montreal calgary leafs winnipeg angeles detroit 61 | [0.16] bike amp helmet kit honda turbo gear brake rear engine 62 | [0.23] privacy conduct security enforcement electronic policy encryption states americans agency 63 | [0.28] gun crime criminal assault weapon handgun batf violent abuse firearm 64 | [0.21] myers president stephanopoulos decision meeting congress february package secretary community 65 | [0.21] hitter pitcher baseball defensive braves player bike pitch fan career 66 | [0.28] season nhl hockey league player coach puck pitcher team playoff 67 | [0.15] bike honda turbo helmet gear dealer saturn rear engine amp 68 | [0.23] homicide weapon firearm handgun vancouver knife minority shall gun militia 69 | [0.15] cancer firearm committee volume amendment handgun health states united patient 70 | [0.18] windows gateway swap port window printer modem setup hd mouse 71 | [0.24] muslims islam jews genocide israel jewish muslim islamic jew turks 72 | [0.29] armenian armenia armenians village muslim turks israel lebanon azerbaijan turkish 73 | [0.24] entry output file winner variable oname buf abuse char io 74 | [0.21] god jesus christian revelation moral doctrine resurrection principle faith christianity 75 | [0.24] motherboard meg hd mb mw slot simm ram adapter port 76 | [0.25] azerbaijan armenian armenians apartment neighbor troops father soviet town hide 77 | [0.18] xterm menu client font swap workstation resource server unix directory 78 | [0.23] db mov bh cs dos byte ax connector hd adapter 79 | [0.23] db det mov tor bh que wm mw pit byte 80 | [0.33] entry buf output variable io oname winner char stream printf 81 | [0.17] anonymous probe privacy mission lunar electronic cipher satellite solar launch 82 | [0.16] motherboard mhz shipping mw printer simm slot adapter meg quadra 83 | [0.33] encryption enforcement serial chip encrypt wiretap escrow clipper agency device 84 | [0.23] jesus marriage prophet christ islam god scripture marry verse prophecy 85 | [0.20] bike manual helmet brake honda turbo gear engine rear ford 86 | [0.26] belief god existence faith christ christian doctrine truth teaching christianity 87 | [0.22] vga dos printer isa adapter pc windows ram scsus scsi 88 | [0.35] scsus scsi ide controller bio mb interface floppy ram rom 89 | [0.24] braves playoff season nhl puck hockey leafs wings cup coach 90 | [0.12] administration president stephanopoulos secret island congress russia government libertarian escrow 91 | [0.19] font xt toolkit colormap vendor export server xterm widget directory 92 | [0.22] turks israel israeli armenia turkish village arab armenian genocide greek 93 | [0.15] workstation wiring anonymous null server xterm directory platform ftp toolkit 94 | [0.28] tor det nhl hockey pit que winnipeg league leafs detroit 95 | [0.23] doctrine jesus god christ revelation scripture heaven atheist christian religious 96 | [0.24] season defensive playoff puck flyers coach league braves team penalty 97 | [0.30] pitch hitter player pitcher score baseball season defensive team braves 98 | [0.30] pt rg eus pd pp calgary philadelphia winnipeg detroit bhj 99 | [0.25] turkish armenian turks village genocide jews muslim murder israel greek 100 | [0.18] gun criminal knife crime gang cop batf violent insurance weapon 101 | [0.28] privacy enforcement encryption security americans device secure escrow rsa conversation 102 | [0.16] windows font binary os pc rom vga cache server hardware 103 | [0.30] connector ide scsi scsus quadra motherboard hd mb meg isa 104 | [0.24] motherboard quadra slot mhz hd meg adapter processor simm ide 105 | [0.34] ide controller scsi scsus floppy isa mb meg motherboard connector 106 | [0.23] jesus faith belief verse god scripture passage satan eternal interpretation 107 | [0.22] ram cache swap mb pc mac windows vga os scsi 108 | [0.13] anonymous wiring privacy wire outlet protocol unix nec ripem ftp 109 | ========================================================================== 110 | Average Topic Coherence = 0.231101832556 111 | (IPC_MAP)[haiti]s1362249: bash 20news.sh 112 | Computing word occurrence... 113 | Computing the observed coherence... 114 | [0.37] controller scsi scsus bio ide floppy rom interface disk mb 115 | [0.17] file states militia enforcement amendment law justice constitution united device 116 | [0.31] satellite launch mission probe lunar spacecraft shuttle module remote nasa 117 | [0.39] armenian armenians turkish turks armenia genocide turkey azerbaijan village territory 118 | [0.21] congress stephanopoulos administration president secretary package meeting sector russia russian 119 | [0.24] scripture belief christian god jesus faith passage moral christianity teaching 120 | [0.26] motherboard meg ide simm adapter mhz port scsi quadra isa 121 | [0.18] widget xt cache interface jpeg toolkit converter os xlib visual 122 | [0.27] os mac pc scsi mb cpu cache interface ibm mhz 123 | [0.14] connector kit ide shipping amp motherboard audio simm adapter voltage 124 | [0.19] image platform pixel graphics processing ftp format workstation widget toolkit 125 | [0.40] armenian muslim armenia armenians turkish turks genocide massacre greek azerbaijan 126 | [0.08] batf fbi congress child stephanopoulos economy myers secretary woman gang 127 | [0.30] voltage install panel circuit cable wire connect nec wiring outlet 128 | [0.15] menu ftp unix workstation directory anonymous digital amiga database privacy 129 | [0.19] batf fbi koresh homosexual crime gun cult violent waco criminal 130 | [0.29] jesus god christ faith bible doctrine christian holy belief revelation 131 | [0.17] windows colormap printer workstation window microsoft swap dos graphic vga 132 | [0.33] db byte mov bh cs ax push al shipping pop 133 | [0.16] encrypt escrow crypto wiretap nasa spacecraft solar serial secure clipper 134 | [0.21] belief reject truth bible god morality evidence atheist moral atheism 135 | [0.28] defensive braves season player league team coach hitter score pitcher 136 | [0.33] nhl hockey coach team league player season playoff penalty cup 137 | [0.23] season pitcher team player braves score puck playoff coach career 138 | [0.24] jesus scripture christian faith belief truth christianity revelation doctrine moral 139 | [0.26] braves season league team score hitter hockey leafs defensive player 140 | [0.40] armenian armenians turkish turks armenia massacre genocide muslim azerbaijan greek 141 | [0.32] nhl hockey detroit rangers league winnipeg calgary cup montreal wings 142 | [0.28] enforcement privacy law americans encryption device administration escrow communication government 143 | [0.15] stephanopoulos president senate criminal assault meeting decision gun crime ban 144 | [0.27] jesus god christ passage christian christianity prophecy faith prophet sin 145 | [0.19] father son holy woman apartment marry jesus neighbor christ rape 146 | [0.26] tor det pit db que bh mov byte pd mon 147 | [0.43] output buf oname char io printf int stream input file 148 | [0.30] entry variable winner int char remark buf widget oname output 149 | [0.14] motherboard turbo manual hd quadra pin amp ide mhz engine 150 | [0.30] meg ide motherboard scsi scsus ram hd floppy isa vga 151 | [0.31] israel israeli jews arab jew islamic muslims arabs lebanon lebanese 152 | [0.28] pt pp calgary det tor rangers mw detroit mg vs 153 | [0.25] wiring install cable circuit nec panel outlet wire manual voltage 154 | [0.27] calgary pp pt tor angeles philadelphia det app winnipeg vancouver 155 | [0.31] israel israeli arab militia lebanese islamic peace lebanon arabs territory 156 | [0.28] encrypt cipher ripem rsa des encryption privacy security serial enforcement 157 | [0.25] font xterm export colormap motif server toolkit widget client xt 158 | [0.11] francisco vancouver disease health study homicide patient investigation identify scientist 159 | [0.28] christ christian jesus god eternal christianity holy sin faith heaven 160 | [0.34] ide mb jumper controller interface scsus scsi connector bio isa 161 | [0.11] anonymous satellite privacy lunar spacecraft foundation rocket sequence cryptography responsibility 162 | [0.21] windows printer microsoft mb ram swap dos mouse cpu vga 163 | [0.14] christ woman jesus apartment neighbor father resurrection rape pray walk 164 | ========================================================================== 165 | Average Topic Coherence = 0.250596966953 166 | -------------------------------------------------------------------------------- /coherence_from_paper_script/Readme.md: -------------------------------------------------------------------------------- 1 | ### Coherence Scores for 20NewsGroup 2 | 3 | Since the Github code is slightly different in terms of some parameter settings used in the original paper. So here we 4 | provide two sets of results for `t=50` and `t=200` topics for `prodLDA` on the 20NewsGroup dataset from the code used 5 | in the original paper for reference. 6 | 7 | > It is recommended that if you use AVTIM for 20news or RCV1 datasets then use to the settings defined in the paper for 8 | the inference network. Our learning method is closely tied to the architectural settings and operations such as 9 | `BN` and `dropout` have very specific roles in the optimization. Therefore, for fair comparison it is strongly advised 10 | that to keep these settings unchanged. 11 | -------------------------------------------------------------------------------- /data/20news_clean/test.txt.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/akashgit/autoencoding_vi_for_topic_models/6066f5581c236b84ff0b7b56496d8823ebff752a/data/20news_clean/test.txt.npy -------------------------------------------------------------------------------- /data/20news_clean/train.txt.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/akashgit/autoencoding_vi_for_topic_models/6066f5581c236b84ff0b7b56496d8823ebff752a/data/20news_clean/train.txt.npy -------------------------------------------------------------------------------- /data/20news_clean/valid.txt.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/akashgit/autoencoding_vi_for_topic_models/6066f5581c236b84ff0b7b56496d8823ebff752a/data/20news_clean/valid.txt.npy -------------------------------------------------------------------------------- /data/20news_clean/vocab.pkl: -------------------------------------------------------------------------------- 1 | (dp1 2 | S'limited' 3 | p2 4 | I1481 5 | sS'child' 6 | p3 7 | I181 8 | sS'four' 9 | p4 10 | I586 11 | sS'sleep' 12 | p5 13 | I1714 14 | sS'hate' 15 | p6 16 | I1141 17 | sS'forget' 18 | p7 19 | I607 20 | sS'whose' 21 | p8 22 | I902 23 | sS'violate' 24 | p9 25 | I1945 26 | sS'bike' 27 | p10 28 | I370 29 | sS'swap' 30 | p11 31 | I1616 32 | sS'lord' 33 | p12 34 | I889 35 | sS'sorry' 36 | p13 37 | I480 38 | sS'worth' 39 | p14 40 | I625 41 | sS'risk' 42 | p15 43 | I911 44 | sS'rise' 45 | p16 46 | I1474 47 | sS'every' 48 | p17 49 | I119 50 | sS'jack' 51 | p18 52 | I1180 53 | sS'affect' 54 | p19 55 | I1291 56 | sS'school' 57 | p20 58 | I475 59 | sS'solution' 60 | p21 61 | I672 62 | sS'reliable' 63 | p22 64 | I1881 65 | sS'enjoy' 66 | p23 67 | I1346 68 | sS'force' 69 | p24 70 | I269 71 | sS'miller' 72 | p25 73 | I1896 74 | sS'consistent' 75 | p26 76 | I1904 77 | sS'direct' 78 | p27 79 | I835 80 | sS'second' 81 | p28 82 | I163 83 | sS'street' 84 | p29 85 | I741 86 | sS'even' 87 | p30 88 | I24 89 | sS'hide' 90 | p31 91 | I1464 92 | sS'christ' 93 | p32 94 | I467 95 | sS'nec' 96 | p33 97 | I1525 98 | sS'liberty' 99 | p34 100 | I1662 101 | sS'usenet' 102 | p35 103 | I993 104 | sS'conduct' 105 | p36 106 | I1534 107 | sS'new' 108 | p37 109 | I19 110 | sS'net' 111 | p38 112 | I435 113 | sS'ever' 114 | p39 115 | I217 116 | sS'widget' 117 | p40 118 | I517 119 | sS'never' 120 | p41 121 | I79 122 | sS'protection' 123 | p42 124 | I1078 125 | sS'active' 126 | p43 127 | I1752 128 | sS'path' 129 | p44 130 | I1402 131 | sS'interpret' 132 | p45 133 | I1536 134 | sS'voice' 135 | p46 136 | I606 137 | sS'daughter' 138 | p47 139 | I1890 140 | sS'study' 141 | p48 142 | I301 143 | sS'credit' 144 | p49 145 | I1584 146 | sS'permit' 147 | p50 148 | I1422 149 | sS'military' 150 | p51 151 | I580 152 | sS'punishment' 153 | p52 154 | I1625 155 | sS'secure' 156 | p53 157 | I780 158 | sS'replace' 159 | p54 160 | I643 161 | sS'visible' 162 | p55 163 | I1792 164 | sS'unix' 165 | p56 166 | I753 167 | sS'moral' 168 | p57 169 | I563 170 | sS'printf' 171 | p58 172 | I1673 173 | sS'total' 174 | p59 175 | I592 176 | sS'unit' 177 | p60 178 | I588 179 | sS'plot' 180 | p61 181 | I1900 182 | sS'army' 183 | p62 184 | I1036 185 | sS'hospital' 186 | p63 187 | I1435 188 | sS'tm' 189 | p64 190 | I1978 191 | sS'call' 192 | p65 193 | I42 194 | sS'therefore' 195 | p66 196 | I540 197 | sS'recommend' 198 | p67 199 | I980 200 | sS'strike' 201 | p68 202 | I1460 203 | sS'survive' 204 | p69 205 | I1869 206 | sS'type' 207 | p70 208 | I229 209 | sS'tell' 210 | p71 211 | I47 212 | sS'posting' 213 | p72 214 | I1263 215 | sS'holy' 216 | p73 217 | I1095 218 | sS'expose' 219 | p74 220 | I1755 221 | sS'award' 222 | p75 223 | I1924 224 | sS'aware' 225 | p76 226 | I1248 227 | sS'warn' 228 | p77 229 | I1872 230 | sS'phone' 231 | p78 232 | I192 233 | sS'excellent' 234 | p79 235 | I882 236 | sS'hole' 237 | p80 238 | I917 239 | sS'hold' 240 | p81 241 | I245 242 | sS'must' 243 | p82 244 | I75 245 | sS'shoot' 246 | p83 247 | I880 248 | sS'md' 249 | p84 250 | I1916 251 | sS'mg' 252 | p85 253 | I1986 254 | sS'ma' 255 | p86 256 | I1010 257 | sS'room' 258 | p87 259 | I821 260 | sS'mb' 261 | p88 262 | I866 263 | sS'setup' 264 | p89 265 | I1540 266 | sS'work' 267 | p90 268 | I16 269 | sS'mi' 270 | p91 271 | I1982 272 | sS'spec' 273 | p92 274 | I1397 275 | sS'mw' 276 | p93 277 | I1989 278 | sS'install' 279 | p94 280 | I468 281 | sS'mr' 282 | p95 283 | I1603 284 | sS'root' 285 | p96 286 | I1539 287 | sS'example' 288 | p97 289 | I150 290 | sS'give' 291 | p98 292 | I28 293 | sS'involve' 294 | p99 295 | I400 296 | sS'want' 297 | p100 298 | I21 299 | sS'keep' 300 | p101 301 | I90 302 | sS'absolute' 303 | p102 304 | I960 305 | sS'end' 306 | p103 307 | I133 308 | sS'thing' 309 | p104 310 | I23 311 | sS'provide' 312 | p105 313 | I140 314 | sS'ie' 315 | p106 316 | I1228 317 | sS'travel' 318 | p107 319 | I1871 320 | sS'feature' 321 | p108 322 | I637 323 | sS'machine' 324 | p109 325 | I202 326 | sS'hot' 327 | p110 328 | I1083 329 | sS'answer' 330 | p111 331 | I165 332 | sS'description' 333 | p112 334 | I1124 335 | sS'earlier' 336 | p113 337 | I1076 338 | sS'lab' 339 | p114 340 | I1249 341 | sS'wrong' 342 | p115 343 | I184 344 | sS'lay' 345 | p116 346 | I1361 347 | sS'president' 348 | p117 349 | I255 350 | sS'law' 351 | p118 352 | I70 353 | sS'parallel' 354 | p119 355 | I1666 356 | sS'purchase' 357 | p120 358 | I744 359 | sS'attempt' 360 | p121 361 | I484 362 | sS'third' 363 | p122 364 | I553 365 | sS'appreciate' 366 | p123 367 | I348 368 | sS'greek' 369 | p124 370 | I803 371 | sS'maintain' 372 | p125 373 | I908 374 | sS'green' 375 | p126 376 | I1331 377 | sS'enter' 378 | p127 379 | I969 380 | sS'worst' 381 | p128 382 | I1677 383 | sS'operate' 384 | p129 385 | I1261 386 | sS'order' 387 | p130 388 | I144 389 | sS'oname' 390 | p131 391 | I1879 392 | sS'restriction' 393 | p132 394 | I1789 395 | sS'interpretation' 396 | p133 397 | I1167 398 | sS'office' 399 | p134 400 | I520 401 | sS'xlib' 402 | p135 403 | I1781 404 | sS'vary' 405 | p136 406 | I1733 407 | sS'jpeg' 408 | p137 409 | I1088 410 | sS'keyboard' 411 | p138 412 | I819 413 | sS'que' 414 | p139 415 | I1906 416 | sS'fit' 417 | p140 418 | I760 419 | sS'personal' 420 | p141 421 | I593 422 | sS'fix' 423 | p142 424 | I682 425 | sS'better' 426 | p143 427 | I78 428 | sS'easier' 429 | p144 430 | I1282 431 | sS'eventually' 432 | p145 433 | I1434 434 | sS'powerful' 435 | p146 436 | I1769 437 | sS'safe' 438 | p147 439 | I1092 440 | sS'break' 441 | p148 442 | I383 443 | sS'band' 444 | p149 445 | I1709 446 | sS'echo' 447 | p150 448 | I1772 449 | sS'bank' 450 | p151 451 | I785 452 | sS'detroit' 453 | p152 454 | I1297 455 | sS'reasonable' 456 | p153 457 | I833 458 | sS'flyers' 459 | p154 460 | I1842 461 | sS'side' 462 | p155 463 | I249 464 | sS'mean' 465 | p156 466 | I58 467 | sS'prohibit' 468 | p157 469 | I1739 470 | sS'fairly' 471 | p158 472 | I1051 473 | sS'series' 474 | p159 475 | I669 476 | sS'carry' 477 | p160 478 | I311 479 | sS'doug' 480 | p161 481 | I1101 482 | sS'laboratory' 483 | p162 484 | I1766 485 | sS'ring' 486 | p163 487 | I1867 488 | sS'network' 489 | p164 490 | I466 491 | sS'god' 492 | p165 493 | I34 494 | sS'content' 495 | p166 496 | I1811 497 | sS're' 498 | p167 499 | I1581 500 | sS'encourage' 501 | p168 502 | I1679 503 | sS'daniel' 504 | p169 505 | I1856 506 | sS'reader' 507 | p170 508 | I1215 509 | sS'medicine' 510 | p171 511 | I1299 512 | sS'surprise' 513 | p172 514 | I1129 515 | sS'size' 516 | p173 517 | I399 518 | sS'foundation' 519 | p174 520 | I1571 521 | sS'associate' 522 | p175 523 | I1110 524 | sS'eternal' 525 | p176 526 | I1236 527 | sS'free' 528 | p177 529 | I200 530 | sS'standard' 531 | p178 532 | I193 533 | sS'estimate' 534 | p179 535 | I1831 536 | sS'kit' 537 | p180 538 | I1894 539 | sS'filter' 540 | p181 541 | I1858 542 | sS'oppose' 543 | p182 544 | I1197 545 | sS'signature' 546 | p183 547 | I1807 548 | sS'isa' 549 | p184 550 | I1378 551 | sS'onto' 552 | p185 553 | I1590 554 | sS'user' 555 | p186 556 | I246 557 | sS'already' 558 | p187 559 | I250 560 | sS'backup' 561 | p188 562 | I1650 563 | sS'render' 564 | p189 565 | I1822 566 | sS'nhl' 567 | p190 568 | I857 569 | sS'puck' 570 | p191 571 | I1805 572 | sS'primary' 573 | p192 574 | I1864 575 | sS'another' 576 | p193 577 | I84 578 | sS'electronic' 579 | p194 580 | I961 581 | sS'service' 582 | p195 583 | I324 584 | sS'tor' 585 | p196 586 | I1972 587 | sS'top' 588 | p197 589 | I462 590 | sS'master' 591 | p198 592 | I989 593 | sS'tom' 594 | p199 595 | I850 596 | sS'john' 597 | p200 598 | I174 599 | sS'listen' 600 | p201 601 | I883 602 | sS'murder' 603 | p202 604 | I650 605 | sS'tool' 606 | p203 607 | I626 608 | sS'serve' 609 | p204 610 | I679 611 | sS'japanese' 612 | p205 613 | I1556 614 | sS'western' 615 | p206 616 | I965 617 | sS'somewhat' 618 | p207 619 | I1425 620 | sS'stephanopoulos' 621 | p208 622 | I797 623 | sS'symptom' 624 | p209 625 | I1965 626 | sS'toronto' 627 | p210 628 | I820 629 | sS'distance' 630 | p211 631 | I1747 632 | sS'target' 633 | p212 634 | I1294 635 | sS'older' 636 | p213 637 | I1512 638 | sS'likely' 639 | p214 640 | I431 641 | sS'cite' 642 | p215 643 | I1746 644 | sS'project' 645 | p216 646 | I507 647 | sS'matter' 648 | p217 649 | I263 650 | sS'flame' 651 | p218 652 | I748 653 | sS'historical' 654 | p219 655 | I1680 656 | sS'feeling' 657 | p220 658 | I1428 659 | sS'boston' 660 | p221 661 | I1019 662 | sS'ram' 663 | p222 664 | I656 665 | sS'mind' 666 | p223 667 | I310 668 | sS'mine' 669 | p224 670 | I489 671 | sS'manner' 672 | p225 673 | I1861 674 | sS'leafs' 675 | p226 676 | I1411 677 | sS'seem' 678 | p227 679 | I51 680 | sS'seek' 681 | p228 682 | I1187 683 | sS'alive' 684 | p229 685 | I1369 686 | sS'ray' 687 | p230 688 | I1438 689 | sS'responsible' 690 | p231 691 | I1017 692 | sS'vga' 693 | p232 694 | I1174 695 | sS'pittsburgh' 696 | p233 697 | I909 698 | sS'contact' 699 | p234 700 | I330 701 | sS'rider' 702 | p235 703 | I1480 704 | sS'blue' 705 | p236 706 | I1075 707 | sS'nobody' 708 | p237 709 | I1072 710 | sS'philadelphia' 711 | p238 712 | I1458 713 | sS'though' 714 | p239 715 | I157 716 | sS'bruce' 717 | p240 718 | I1744 719 | sS'object' 720 | p241 721 | I601 722 | sS'regular' 723 | p242 724 | I1016 725 | sS'germany' 726 | p243 727 | I914 728 | sS'ensure' 729 | p244 730 | I1688 731 | sS'letter' 732 | p245 733 | I685 734 | sS'entry' 735 | p246 736 | I405 737 | sS'morality' 738 | p247 739 | I991 740 | sS'observation' 741 | p248 742 | I1484 743 | sS'medical' 744 | p249 745 | I723 746 | sS'dog' 747 | p250 748 | I978 749 | sS'dod' 750 | p251 751 | I483 752 | sS'definitely' 753 | p252 754 | I1427 755 | sS'principle' 756 | p253 757 | I1316 758 | sS'dos' 759 | p254 760 | I414 761 | sS'mailing' 762 | p255 763 | I1193 764 | sS'reserve' 765 | p256 766 | I1758 767 | sS'bomb' 768 | p257 769 | I1824 770 | sS'pope' 771 | p258 772 | I1750 773 | sS'lewis' 774 | p259 775 | I1952 776 | sS'random' 777 | p260 778 | I1395 779 | sS'radio' 780 | p261 781 | I568 782 | sS'participate' 783 | p262 784 | I1802 785 | sS'earth' 786 | p263 787 | I448 788 | sS'finally' 789 | p264 790 | I811 791 | sS'implementation' 792 | p265 793 | I1555 794 | sS'menu' 795 | p266 796 | I1599 797 | sS'explain' 798 | p267 799 | I386 800 | sS'minnesota' 801 | p268 802 | I1767 803 | sS'bush' 804 | p269 805 | I1630 806 | sS'rich' 807 | p270 808 | I1615 809 | sS'announce' 810 | p271 811 | I1104 812 | sS'scripture' 813 | p272 814 | I1128 815 | sS'cryptography' 816 | p273 817 | I1554 818 | sS'de' 819 | p274 820 | I903 821 | sS'db' 822 | p275 823 | I393 824 | sS'dc' 825 | p276 826 | I1334 827 | sS'despite' 828 | p277 829 | I1489 830 | sS'report' 831 | p278 832 | I214 833 | sS'saturn' 834 | p279 835 | I1937 836 | sS'hall' 837 | p280 838 | I1382 839 | sS'bar' 840 | p281 841 | I1477 842 | sS'method' 843 | p282 844 | I581 845 | sS'twice' 846 | p283 847 | I1607 848 | sS'bad' 849 | p284 850 | I194 851 | sS'release' 852 | p285 853 | I422 854 | sS'steal' 855 | p286 856 | I1109 857 | sS'secretary' 858 | p287 859 | I1530 860 | sS'respond' 861 | p288 862 | I735 863 | sS'victim' 864 | p289 865 | I1510 866 | sS'fair' 867 | p290 868 | I1227 869 | sS'motherboard' 870 | p291 871 | I1195 872 | sS'meg' 873 | p292 874 | I1098 875 | sS'result' 876 | p293 877 | I227 878 | sS'fail' 879 | p294 880 | I501 881 | sS'best' 882 | p295 883 | I131 884 | sS'subject' 885 | p296 886 | I185 887 | sS'away' 888 | p297 889 | I225 890 | sS'score' 891 | p298 892 | I541 893 | sS'finger' 894 | p299 895 | I1507 896 | sS'approach' 897 | p300 898 | I644 899 | sS'accord' 900 | p301 901 | I554 902 | sS'extend' 903 | p302 904 | I1929 905 | sS'nature' 906 | p303 907 | I771 908 | sS'modify' 909 | p304 910 | I1524 911 | sS'wm' 912 | p305 913 | I1993 914 | sS'weak' 915 | p306 916 | I1708 917 | sS'however' 918 | p307 919 | I112 920 | sS'wear' 921 | p308 922 | I1290 923 | sS'news' 924 | p309 925 | I336 926 | sS'improve' 927 | p310 928 | I1118 929 | sS'cop' 930 | p311 931 | I1139 932 | sS'protect' 933 | p312 934 | I460 935 | sS'accident' 936 | p313 937 | I1386 938 | sS'country' 939 | p314 940 | I243 941 | sS'adapter' 942 | p315 943 | I1720 944 | sS'cup' 945 | p316 946 | I1001 947 | sS'logic' 948 | p317 949 | I1094 950 | sS'argue' 951 | p318 952 | I762 953 | sS'cult' 954 | p319 955 | I1812 956 | sS'tony' 957 | p320 958 | I1601 959 | sS'trust' 960 | p321 961 | I814 962 | sS'hitler' 963 | p322 964 | I1738 965 | sS'speak' 966 | p323 967 | I332 968 | sS'conference' 969 | p324 970 | I913 971 | sS'basis' 972 | p325 973 | I678 974 | sS'three' 975 | p326 976 | I218 977 | sS'quickly' 978 | p327 979 | I1339 980 | sS'much' 981 | p328 982 | I38 983 | sS'west' 984 | p329 985 | I940 986 | sS'interest' 987 | p330 988 | I525 989 | sS'basic' 990 | p331 991 | I782 992 | sS'life' 993 | p332 994 | I122 995 | sS'easy' 996 | p333 997 | I536 998 | sS'wish' 999 | p334 1000 | I624 1001 | sS'dave' 1002 | p335 1003 | I657 1004 | sS'personally' 1005 | p336 1006 | I1089 1007 | sS'catch' 1008 | p337 1009 | I1012 1010 | sS'exception' 1011 | p338 1012 | I1940 1013 | sS'east' 1014 | p339 1015 | I966 1016 | sS'tank' 1017 | p340 1018 | I1035 1019 | sS'air' 1020 | p341 1021 | I764 1022 | sS'near' 1023 | p342 1024 | I676 1025 | sS'suppose' 1026 | p343 1027 | I432 1028 | sS'aid' 1029 | p344 1030 | I1226 1031 | sS'property' 1032 | p345 1033 | I1015 1034 | sS'procedure' 1035 | p346 1036 | I1668 1037 | sS'mistake' 1038 | p347 1039 | I1103 1040 | sS'toolkit' 1041 | p348 1042 | I1702 1043 | sS'ii' 1044 | p349 1045 | I777 1046 | sS'player' 1047 | p350 1048 | I212 1049 | sS'brake' 1050 | p351 1051 | I1354 1052 | sS'io' 1053 | p352 1054 | I1975 1055 | sS'vendor' 1056 | p353 1057 | I1791 1058 | sS'seattle' 1059 | p354 1060 | I1567 1061 | sS'mouse' 1062 | p355 1063 | I641 1064 | sS'id' 1065 | p356 1066 | I1349 1067 | sS'prior' 1068 | p357 1069 | I1727 1070 | sS'perform' 1071 | p358 1072 | I919 1073 | sS'suggest' 1074 | p359 1075 | I376 1076 | sS'make' 1077 | p360 1078 | I9 1079 | sS'complex' 1080 | p361 1081 | I1594 1082 | sS'several' 1083 | p362 1084 | I208 1085 | sS'european' 1086 | p363 1087 | I1330 1088 | sS'wheel' 1089 | p364 1090 | I1602 1091 | sS'independent' 1092 | p365 1093 | I1152 1094 | sS'satellite' 1095 | p366 1096 | I796 1097 | sS'catholic' 1098 | p367 1099 | I1400 1100 | sS'social' 1101 | p368 1102 | I1120 1103 | sS'hang' 1104 | p369 1105 | I1365 1106 | sS'evil' 1107 | p370 1108 | I899 1109 | sS'hand' 1110 | p371 1111 | I205 1112 | sS'opportunity' 1113 | p372 1114 | I1502 1115 | sS'kid' 1116 | p373 1117 | I767 1118 | sS'client' 1119 | p374 1120 | I923 1121 | sS'mother' 1122 | p375 1123 | I949 1124 | sS'left' 1125 | p376 1126 | I972 1127 | sS'protocol' 1128 | p377 1129 | I1718 1130 | sS'sentence' 1131 | p378 1132 | I1604 1133 | sS'distribute' 1134 | p379 1135 | I1357 1136 | sS'identify' 1137 | p380 1138 | I1479 1139 | sS'thanks' 1140 | p381 1141 | I73 1142 | sS'human' 1143 | p382 1144 | I186 1145 | sS'yes' 1146 | p383 1147 | I188 1148 | sS'yet' 1149 | p384 1150 | I224 1151 | sS'previous' 1152 | p385 1153 | I647 1154 | sS'jose' 1155 | p386 1156 | I1436 1157 | sS'mix' 1158 | p387 1159 | I1897 1160 | sS'character' 1161 | p388 1162 | I859 1163 | sS'spread' 1164 | p389 1165 | I1868 1166 | sS'board' 1167 | p390 1168 | I398 1169 | sS'innocent' 1170 | p391 1171 | I1119 1172 | sS'save' 1173 | p392 1174 | I335 1175 | sS'possible' 1176 | p393 1177 | I156 1178 | sS'arab' 1179 | p394 1180 | I834 1181 | sS'possibly' 1182 | p395 1183 | I973 1184 | sS'background' 1185 | p396 1186 | I1269 1187 | sS'destroy' 1188 | p397 1189 | I1050 1190 | sS'judge' 1191 | p398 1192 | I646 1193 | sS'highly' 1194 | p399 1195 | I1151 1196 | sS'desire' 1197 | p400 1198 | I1271 1199 | sS'manual' 1200 | p401 1201 | I696 1202 | sS'mit' 1203 | p402 1204 | I1482 1205 | sS'specific' 1206 | p403 1207 | I677 1208 | sS'remind' 1209 | p404 1210 | I1959 1211 | sS'officer' 1212 | p405 1213 | I1065 1214 | sS'night' 1215 | p406 1216 | I522 1217 | sS'security' 1218 | p407 1219 | I444 1220 | sS'right' 1221 | p408 1222 | I41 1223 | sS'old' 1224 | p409 1225 | I180 1226 | sS'deal' 1227 | p410 1228 | I313 1229 | sS'people' 1230 | p411 1231 | I8 1232 | sS'somehow' 1233 | p412 1234 | I1440 1235 | sS'dead' 1236 | p413 1237 | I529 1238 | sS'printer' 1239 | p414 1240 | I768 1241 | sS'escape' 1242 | p415 1243 | I1080 1244 | sS'pain' 1245 | p416 1246 | I996 1247 | sS'bottom' 1248 | p417 1249 | I1533 1250 | sS'ice' 1251 | p418 1252 | I1348 1253 | sS'everything' 1254 | p419 1255 | I327 1256 | sS'colormap' 1257 | p420 1258 | I1855 1259 | sS'burn' 1260 | p421 1261 | I943 1262 | sS'defensive' 1263 | p422 1264 | I1813 1265 | sS'post' 1266 | p423 1267 | I48 1268 | sS'plug' 1269 | p424 1270 | I1245 1271 | sS'acquire' 1272 | p425 1273 | I1743 1274 | sS'palestinian' 1275 | p426 1276 | I1706 1277 | sS'magazine' 1278 | p427 1279 | I887 1280 | sS'plus' 1281 | p428 1282 | I707 1283 | sS'commit' 1284 | p429 1285 | I878 1286 | sS'slightly' 1287 | p430 1288 | I1486 1289 | sS'batf' 1290 | p431 1291 | I1399 1292 | sS'facility' 1293 | p432 1294 | I1337 1295 | sS'civil' 1296 | p433 1297 | I1026 1298 | sS'son' 1299 | p434 1300 | I718 1301 | sS'stop' 1302 | p435 1303 | I277 1304 | sS'doctrine' 1305 | p436 1306 | I1577 1307 | sS'rely' 1308 | p437 1309 | I1778 1310 | sS'compression' 1311 | p438 1312 | I1783 1313 | sS'support' 1314 | p439 1315 | I83 1316 | sS'initial' 1317 | p440 1318 | I1893 1319 | sS'fight' 1320 | p441 1321 | I765 1322 | sS'gif' 1323 | p442 1324 | I1082 1325 | sS'joseph' 1326 | p443 1327 | I1391 1328 | sS'editor' 1329 | p444 1330 | I1420 1331 | sS'way' 1332 | p445 1333 | I25 1334 | sS'music' 1335 | p446 1336 | I1775 1337 | sS'war' 1338 | p447 1339 | I265 1340 | sS'happy' 1341 | p448 1342 | I907 1343 | sS'head' 1344 | p449 1345 | I322 1346 | sS'form' 1347 | p450 1348 | I360 1349 | sS'offer' 1350 | p451 1351 | I270 1352 | sS'ford' 1353 | p452 1354 | I1951 1355 | sS'failure' 1356 | p453 1357 | I1585 1358 | sS'heat' 1359 | p454 1360 | I1634 1361 | sS'hear' 1362 | p455 1363 | I102 1364 | sS'solar' 1365 | p456 1366 | I1699 1367 | sS'surrender' 1368 | p457 1369 | I1882 1370 | sS'true' 1371 | p458 1372 | I123 1373 | sS'braves' 1374 | p459 1375 | I1753 1376 | sS'inside' 1377 | p460 1378 | I754 1379 | sS'delete' 1380 | p461 1381 | I618 1382 | sS'phil' 1383 | p462 1384 | I1636 1385 | sS'liberal' 1386 | p463 1387 | I1834 1388 | sS'computing' 1389 | p464 1390 | I1686 1391 | sS'drive' 1392 | p465 1393 | I50 1394 | sS'postscript' 1395 | p466 1396 | I1407 1397 | sS'exist' 1398 | p467 1399 | I167 1400 | sS'prayer' 1401 | p468 1402 | I1918 1403 | sS'ship' 1404 | p469 1405 | I1090 1406 | sS'archive' 1407 | p470 1408 | I895 1409 | sS'physical' 1410 | p471 1411 | I1033 1412 | sS'nj' 1413 | p472 1414 | I1589 1415 | sS'encrypt' 1416 | p473 1417 | I1620 1418 | sS'floor' 1419 | p474 1420 | I1653 1421 | sS'na' 1422 | p475 1423 | I1326 1424 | sS'generally' 1425 | p476 1426 | I849 1427 | sS'reality' 1428 | p477 1429 | I998 1430 | sS'ny' 1431 | p478 1432 | I1047 1433 | sS'interested' 1434 | p479 1435 | I307 1436 | sS'role' 1437 | p480 1438 | I1284 1439 | sS'digital' 1440 | p481 1441 | I839 1442 | sS'test' 1443 | p482 1444 | I306 1445 | sS'tie' 1446 | p483 1447 | I1225 1448 | sS'nt' 1449 | p484 1450 | I1196 1451 | sS'roll' 1452 | p485 1453 | I1960 1454 | sS'picture' 1455 | p486 1456 | I692 1457 | sS'intend' 1458 | p487 1459 | I874 1460 | sS'welcome' 1461 | p488 1462 | I1345 1463 | sS'update' 1464 | p489 1465 | I1165 1466 | sS'muslims' 1467 | p490 1468 | I1350 1469 | sS'variable' 1470 | p491 1471 | I1883 1472 | sS'hurt' 1473 | p492 1474 | I1240 1475 | sS'careful' 1476 | p493 1477 | I1876 1478 | sS'billion' 1479 | p494 1480 | I1157 1481 | sS'faster' 1482 | p495 1483 | I848 1484 | sS'assume' 1485 | p496 1486 | I254 1487 | sS'together' 1488 | p497 1489 | I628 1490 | sS'time' 1491 | p498 1492 | I14 1493 | sS'push' 1494 | p499 1495 | I997 1496 | sS'serious' 1497 | p500 1498 | I717 1499 | sS'basically' 1500 | p501 1501 | I995 1502 | sS'concept' 1503 | p502 1504 | I1086 1505 | sS'rom' 1506 | p503 1507 | I1396 1508 | sS'ron' 1509 | p504 1510 | I1162 1511 | sS'rob' 1512 | p505 1513 | I1275 1514 | sS'datum' 1515 | p506 1516 | I129 1517 | sS'focus' 1518 | p507 1519 | I1816 1520 | sS'manager' 1521 | p508 1522 | I577 1523 | sS'mile' 1524 | p509 1525 | I831 1526 | sS'battle' 1527 | p510 1528 | I1854 1529 | sS'certainly' 1530 | p511 1531 | I378 1532 | sS'depend' 1533 | p512 1534 | I701 1535 | sS'zone' 1536 | p513 1537 | I1476 1538 | sS'technique' 1539 | p514 1540 | I1208 1541 | sS'father' 1542 | p515 1543 | I651 1544 | sS'passage' 1545 | p516 1546 | I1494 1547 | sS'environment' 1548 | p517 1549 | I927 1550 | sS'charge' 1551 | p518 1552 | I716 1553 | sS'division' 1554 | p519 1555 | I565 1556 | sS'string' 1557 | p520 1558 | I1207 1559 | sS'advantage' 1560 | p521 1561 | I1121 1562 | sS'choice' 1563 | p522 1564 | I673 1565 | sS'anonymous' 1566 | p523 1567 | I582 1568 | sS'join' 1569 | p524 1570 | I1324 1571 | sS'trouble' 1572 | p525 1573 | I931 1574 | sS'exact' 1575 | p526 1576 | I1375 1577 | sS'minute' 1578 | p527 1579 | I684 1580 | sS'cool' 1581 | p528 1582 | I1652 1583 | sS'level' 1584 | p529 1585 | I359 1586 | sS'tear' 1587 | p530 1588 | I1368 1589 | sS'die' 1590 | p531 1591 | I312 1592 | sS'rights' 1593 | p532 1594 | I295 1595 | sS'gun' 1596 | p533 1597 | I134 1598 | sS'leave' 1599 | p534 1600 | I142 1601 | sS'item' 1602 | p535 1603 | I896 1604 | sS'team' 1605 | p536 1606 | I106 1607 | sS'quick' 1608 | p537 1609 | I1218 1610 | sS'guy' 1611 | p538 1612 | I303 1613 | sS'round' 1614 | p539 1615 | I910 1616 | sS'prevent' 1617 | p540 1618 | I659 1619 | sS'discover' 1620 | p541 1621 | I1356 1622 | sS'sign' 1623 | p542 1624 | I671 1625 | sS'cost' 1626 | p543 1627 | I221 1628 | sS'run' 1629 | p544 1630 | I43 1631 | sS'port' 1632 | p545 1633 | I556 1634 | sS'appear' 1635 | p546 1636 | I314 1637 | sS'current' 1638 | p547 1639 | I248 1640 | sS'suspect' 1641 | p548 1642 | I900 1643 | sS'reply' 1644 | p549 1645 | I426 1646 | sS'muslim' 1647 | p550 1648 | I1002 1649 | sS'french' 1650 | p551 1651 | I1905 1652 | sS'understanding' 1653 | p552 1654 | I1295 1655 | sS'water' 1656 | p553 1657 | I610 1658 | sS'slow' 1659 | p554 1660 | I951 1661 | sS'baseball' 1662 | p555 1663 | I730 1664 | sS'address' 1665 | p556 1666 | I189 1667 | sS'alone' 1668 | p557 1669 | I915 1670 | sS'along' 1671 | p558 1672 | I623 1673 | sS'change' 1674 | p559 1675 | I110 1676 | sS'wait' 1677 | p560 1678 | I478 1679 | sS'box' 1680 | p561 1681 | I281 1682 | sS'boy' 1683 | p562 1684 | I1371 1685 | sS'dec' 1686 | p563 1687 | I1771 1688 | sS'canadian' 1689 | p564 1690 | I1659 1691 | sS'institute' 1692 | p565 1693 | I807 1694 | sS'shift' 1695 | p566 1696 | I1647 1697 | sS'teaching' 1698 | p567 1699 | I1541 1700 | sS'guilty' 1701 | p568 1702 | I1832 1703 | sS'trial' 1704 | p569 1705 | I901 1706 | sS'suggestion' 1707 | p570 1708 | I818 1709 | sS'usually' 1710 | p571 1711 | I452 1712 | sS'bob' 1713 | p572 1714 | I655 1715 | sS'myers' 1716 | p573 1717 | I1801 1718 | sS'montreal' 1719 | p574 1720 | I1394 1721 | sS'macintosh' 1722 | p575 1723 | I1827 1724 | sS'winner' 1725 | p576 1726 | I1542 1727 | sS'love' 1728 | p577 1729 | I293 1730 | sS'extra' 1731 | p578 1732 | I941 1733 | sS'merely' 1734 | p579 1735 | I1288 1736 | sS'prefer' 1737 | p580 1738 | I979 1739 | sS'logical' 1740 | p581 1741 | I1774 1742 | sS'rg' 1743 | p582 1744 | I1985 1745 | sS'fbi' 1746 | p583 1747 | I743 1748 | sS'employer' 1749 | p584 1750 | I1500 1751 | sS'firearm' 1752 | p585 1753 | I691 1754 | sS'market' 1755 | p586 1756 | I687 1757 | sS'everybody' 1758 | p587 1759 | I1206 1760 | sS'troops' 1761 | p588 1762 | I1487 1763 | sS'prove' 1764 | p589 1765 | I504 1766 | sS'positive' 1767 | p590 1768 | I1503 1769 | sS'visit' 1770 | p591 1771 | I1516 1772 | sS'connector' 1773 | p592 1774 | I1568 1775 | sS'live' 1776 | p593 1777 | I158 1778 | sS'andy' 1779 | p594 1780 | I1721 1781 | sS'memory' 1782 | p595 1783 | I299 1784 | sS'msg' 1785 | p596 1786 | I985 1787 | sS'satan' 1788 | p597 1789 | I1936 1790 | sS'today' 1791 | p598 1792 | I219 1793 | sS'chapter' 1794 | p599 1795 | I1762 1796 | sS'club' 1797 | p600 1798 | I1169 1799 | sS'acceptable' 1800 | p601 1801 | I1809 1802 | sS'visual' 1803 | p602 1804 | I1414 1805 | sS'prophecy' 1806 | p603 1807 | I1901 1808 | sS'effort' 1809 | p604 1810 | I695 1811 | sS'fly' 1812 | p605 1813 | I1223 1814 | sS'graphic' 1815 | p606 1816 | I567 1817 | sS'ibm' 1818 | p607 1819 | I732 1820 | sS'car' 1821 | p608 1822 | I86 1823 | sS'originally' 1824 | p609 1825 | I1899 1826 | sS'cap' 1827 | p610 1828 | I1723 1829 | sS'soul' 1830 | p611 1831 | I1145 1832 | sS'achieve' 1833 | p612 1834 | I1793 1835 | sS'cal' 1836 | p613 1837 | I1455 1838 | sS'following' 1839 | p614 1840 | I1654 1841 | sS'arrive' 1842 | p615 1843 | I1795 1844 | sS'claim' 1845 | p616 1846 | I136 1847 | sS'figure' 1848 | p617 1849 | I434 1850 | sS'predict' 1851 | p618 1852 | I1956 1853 | sS'chip' 1854 | p619 1855 | I143 1856 | sS'agent' 1857 | p620 1858 | I1057 1859 | sS'sample' 1860 | p621 1861 | I1258 1862 | sS'council' 1863 | p622 1864 | I1949 1865 | sS'offense' 1866 | p623 1867 | I1917 1868 | sS'occur' 1869 | p624 1870 | I604 1871 | sS'discussion' 1872 | p625 1873 | I394 1874 | sS'write' 1875 | p626 1876 | I1 1877 | sS'sunday' 1878 | p627 1879 | I1251 1880 | sS'vancouver' 1881 | p628 1882 | I1148 1883 | sS'familiar' 1884 | p629 1885 | I1950 1886 | sS'genocide' 1887 | p630 1888 | I1179 1889 | sS'guarantee' 1890 | p631 1891 | I1114 1892 | sS'economy' 1893 | p632 1894 | I1785 1895 | sS'map' 1896 | p633 1897 | I1287 1898 | sS'product' 1899 | p634 1900 | I419 1901 | sS'huge' 1902 | p635 1903 | I1469 1904 | sS'may' 1905 | p636 1906 | I30 1907 | sS'max' 1908 | p637 1909 | I1698 1910 | sS'spot' 1911 | p638 1912 | I1447 1913 | sS'islamic' 1914 | p639 1915 | I1360 1916 | sS'mac' 1917 | p640 1918 | I291 1919 | sS'produce' 1920 | p641 1921 | I508 1922 | sS'date' 1923 | p642 1924 | I474 1925 | sS'truly' 1926 | p643 1927 | I1550 1928 | sS'data' 1929 | p644 1930 | I1612 1931 | sS'grow' 1932 | p645 1933 | I1184 1934 | sS'man' 1935 | p646 1936 | I93 1937 | sS'natural' 1938 | p647 1939 | I947 1940 | sS'remember' 1941 | p648 1942 | I234 1943 | sS'johnson' 1944 | p649 1945 | I1860 1946 | sS'maybe' 1947 | p650 1948 | I187 1949 | sS'sp' 1950 | p651 1951 | I1981 1952 | sS'st' 1953 | p652 1954 | I1979 1955 | sS'switch' 1956 | p653 1957 | I621 1958 | sS'truck' 1959 | p654 1960 | I1670 1961 | sS'april' 1962 | p655 1963 | I331 1964 | sS'talk' 1965 | p656 1966 | I138 1967 | sS'typical' 1968 | p657 1969 | I1328 1970 | sS'indeed' 1971 | p658 1972 | I804 1973 | sS'damage' 1974 | p659 1975 | I1244 1976 | sS'course' 1977 | p660 1978 | I120 1979 | sS'pitch' 1980 | p661 1981 | I1186 1982 | sS'cold' 1983 | p662 1984 | I1418 1985 | sS'still' 1986 | p663 1987 | I62 1988 | sS'worship' 1989 | p664 1990 | I1415 1991 | sS'group' 1992 | p665 1993 | I85 1994 | sS'monitor' 1995 | p666 1996 | I413 1997 | sS'interesting' 1998 | p667 1999 | I545 2000 | sS'jim' 2001 | p668 2002 | I506 2003 | sS'platform' 2004 | p669 2005 | I1692 2006 | sS'window' 2007 | p670 2008 | I101 2009 | sS'winnipeg' 2010 | p671 2011 | I1911 2012 | sS'policy' 2013 | p672 2014 | I485 2015 | sS'mail' 2016 | p673 2017 | I228 2018 | sS'main' 2019 | p674 2020 | I719 2021 | sS'texas' 2022 | p675 2023 | I986 2024 | sS'civilian' 2025 | p676 2026 | I1007 2027 | sS'nation' 2028 | p677 2029 | I944 2030 | sS'half' 2031 | p678 2032 | I756 2033 | sS'discuss' 2034 | p679 2035 | I639 2036 | sS'term' 2037 | p680 2038 | I433 2039 | sS'name' 2040 | p681 2041 | I88 2042 | sS'james' 2043 | p682 2044 | I660 2045 | sS'times' 2046 | p683 2047 | I1658 2048 | sS'drop' 2049 | p684 2050 | I755 2051 | sS'crypto' 2052 | p685 2053 | I1364 2054 | sS'rangers' 2055 | p686 2056 | I1429 2057 | sS'rock' 2058 | p687 2059 | I1317 2060 | sS'entirely' 2061 | p688 2062 | I1190 2063 | sS'domain' 2064 | p689 2065 | I1374 2066 | sS'ed' 2067 | p690 2068 | I876 2069 | sS'yeah' 2070 | p691 2071 | I981 2072 | sS'year' 2073 | p692 2074 | I18 2075 | sS'assumption' 2076 | p693 2077 | I1370 2078 | sS'happen' 2079 | p694 2080 | I116 2081 | sS'morning' 2082 | p695 2083 | I1250 2084 | sS'minority' 2085 | p696 2086 | I1170 2087 | sS'canada' 2088 | p697 2089 | I547 2090 | sS'living' 2091 | p698 2092 | I1818 2093 | sS'space' 2094 | p699 2095 | I111 2096 | sS'increase' 2097 | p700 2098 | I497 2099 | sS'seriously' 2100 | p701 2101 | I1056 2102 | sS'investigation' 2103 | p702 2104 | I1942 2105 | sS'kent' 2106 | p703 2107 | I1705 2108 | sS'internet' 2109 | p704 2110 | I201 2111 | sS'correct' 2112 | p705 2113 | I391 2114 | sS'theory' 2115 | p706 2116 | I583 2117 | sS'million' 2118 | p707 2119 | I498 2120 | sS'possibility' 2121 | p708 2122 | I1142 2123 | sS'quite' 2124 | p709 2125 | I216 2126 | sS'california' 2127 | p710 2128 | I946 2129 | sS'besides' 2130 | p711 2131 | I1211 2132 | sS'byte' 2133 | p712 2134 | I885 2135 | sS'keith' 2136 | p713 2137 | I962 2138 | sS'card' 2139 | p714 2140 | I126 2141 | sS'care' 2142 | p715 2143 | I304 2144 | sS'advance' 2145 | p716 2146 | I423 2147 | sS'training' 2148 | p717 2149 | I1318 2150 | sS'language' 2151 | p718 2152 | I704 2153 | sS'programming' 2154 | p719 2155 | I1274 2156 | sS'british' 2157 | p720 2158 | I1319 2159 | sS'turn' 2160 | p721 2161 | I190 2162 | sS'place' 2163 | p722 2164 | I109 2165 | sS'gordon' 2166 | p723 2167 | I1377 2168 | sS'think' 2169 | p724 2170 | I12 2171 | sS'first' 2172 | p725 2173 | I39 2174 | sS'origin' 2175 | p726 2176 | I1863 2177 | sS'btw' 2178 | p727 2179 | I906 2180 | sS'one' 2181 | p728 2182 | I2 2183 | sS'insist' 2184 | p729 2185 | I1756 2186 | sS'americans' 2187 | p730 2188 | I1046 2189 | sS'specifically' 2190 | p731 2191 | I1182 2192 | sS'directly' 2193 | p732 2194 | I840 2195 | sS'vote' 2196 | p733 2197 | I749 2198 | sS'impossible' 2199 | p734 2200 | I1166 2201 | sS'message' 2202 | p735 2203 | I237 2204 | sS'open' 2205 | p736 2206 | I253 2207 | sS'george' 2208 | p737 2209 | I928 2210 | sS'xt' 2211 | p738 2212 | I1380 2213 | sS'engineer' 2214 | p739 2215 | I1204 2216 | sS'city' 2217 | p740 2218 | I339 2219 | sS'little' 2220 | p741 2221 | I108 2222 | sS'necessarily' 2223 | p742 2224 | I1060 2225 | sS'anyone' 2226 | p743 2227 | I46 2228 | sS'indicate' 2229 | p744 2230 | I948 2231 | sS'draft' 2232 | p745 2233 | I1214 2234 | sS'white' 2235 | p746 2236 | I443 2237 | sS'national' 2238 | p747 2239 | I260 2240 | sS'friend' 2241 | p748 2242 | I329 2243 | sS'mostly' 2244 | p749 2245 | I1185 2246 | sS'floppy' 2247 | p750 2248 | I873 2249 | sS'season' 2250 | p751 2251 | I382 2252 | sS'workstation' 2253 | p752 2254 | I1873 2255 | sS'alan' 2256 | p753 2257 | I1574 2258 | sS'hus' 2259 | p754 2260 | I578 2261 | sS'copy' 2262 | p755 2263 | I290 2264 | sS'specify' 2265 | p756 2266 | I1277 2267 | sS'population' 2268 | p757 2269 | I693 2270 | sS'wide' 2271 | p758 2272 | I1175 2273 | sS'unfortunately' 2274 | p759 2275 | I858 2276 | sS'require' 2277 | p760 2278 | I203 2279 | sS'future' 2280 | p761 2281 | I573 2282 | sS'russia' 2283 | p762 2284 | I1312 2285 | sS'documentation' 2286 | p763 2287 | I1490 2288 | sS'san' 2289 | p764 2290 | I609 2291 | sS'craig' 2292 | p765 2293 | I1632 2294 | sS'pro' 2295 | p766 2296 | I1559 2297 | sS'argument' 2298 | p767 2299 | I262 2300 | sS'say' 2301 | p768 2302 | I5 2303 | sS'allen' 2304 | p769 2305 | I1884 2306 | sS'nazi' 2307 | p770 2308 | I1401 2309 | sS'kevin' 2310 | p771 2311 | I1547 2312 | sS'aside' 2313 | p772 2314 | I1853 2315 | sS'note' 2316 | p773 2317 | I148 2318 | sS'equipment' 2319 | p774 2320 | I1064 2321 | sS'potential' 2322 | p775 2323 | I1311 2324 | sS'take' 2325 | p776 2326 | I17 2327 | sS'objective' 2328 | p777 2329 | I720 2330 | sS'performance' 2331 | p778 2332 | I713 2333 | sS'wonder' 2334 | p779 2335 | I362 2336 | sS'channel' 2337 | p780 2338 | I1280 2339 | sS'begin' 2340 | p781 2341 | I397 2342 | sS'sure' 2343 | p782 2344 | I82 2345 | sS'multiple' 2346 | p783 2347 | I1052 2348 | sS'normal' 2349 | p784 2350 | I790 2351 | sS'track' 2352 | p785 2353 | I1413 2354 | sS'price' 2355 | p786 2356 | I211 2357 | sS'assault' 2358 | p787 2359 | I1511 2360 | sS'pair' 2361 | p788 2362 | I1836 2363 | sS'america' 2364 | p789 2365 | I801 2366 | sS'especially' 2367 | p790 2368 | I463 2369 | sS'average' 2370 | p791 2371 | I688 2372 | sS'later' 2373 | p792 2374 | I364 2375 | sS'sale' 2376 | p793 2377 | I465 2378 | sS'wing' 2379 | p794 2380 | I1693 2381 | sS'larry' 2382 | p795 2383 | I1470 2384 | sS'pattern' 2385 | p796 2386 | I1847 2387 | sS'professional' 2388 | p797 2389 | I1614 2390 | sS'senior' 2391 | p798 2392 | I1961 2393 | sS'shop' 2394 | p799 2395 | I1837 2396 | sS'shot' 2397 | p800 2398 | I758 2399 | sS'show' 2400 | p801 2401 | I103 2402 | sS'german' 2403 | p802 2404 | I1150 2405 | sS'cheap' 2406 | p803 2407 | I1054 2408 | sS'corner' 2409 | p804 2410 | I1798 2411 | sS'azerbaijan' 2412 | p805 2413 | I1563 2414 | sS'ground' 2415 | p806 2416 | I492 2417 | sS'slot' 2418 | p807 2419 | I1301 2420 | sS'upgrade' 2421 | p808 2422 | I898 2423 | sS'title' 2424 | p809 2425 | I709 2426 | sS'daily' 2427 | p810 2428 | I1912 2429 | sS'behind' 2430 | p811 2431 | I617 2432 | sS'crime' 2433 | p812 2434 | I403 2435 | sS'black' 2436 | p813 2437 | I439 2438 | sS'hockey' 2439 | p814 2440 | I464 2441 | sS'get' 2442 | p815 2443 | I4 2444 | sS'mission' 2445 | p816 2446 | I884 2447 | sS'nearly' 2448 | p817 2449 | I1493 2450 | sS'reveal' 2451 | p818 2452 | I1424 2453 | sS'resource' 2454 | p819 2455 | I700 2456 | sS'girl' 2457 | p820 2458 | I1878 2459 | sS'summary' 2460 | p821 2461 | I1340 2462 | sS'stupid' 2463 | p822 2464 | I869 2465 | sS'roger' 2466 | p823 2467 | I1219 2468 | sS'seat' 2469 | p824 2470 | I1529 2471 | sS'relative' 2472 | p825 2473 | I1683 2474 | sS'college' 2475 | p826 2476 | I1149 2477 | sS'existence' 2478 | p827 2479 | I766 2480 | sS'stanley' 2481 | p828 2482 | I1715 2483 | sS'sport' 2484 | p829 2485 | I1164 2486 | sS'concern' 2487 | p830 2488 | I502 2489 | sS'israeli' 2490 | p831 2491 | I410 2492 | sS'detect' 2493 | p832 2494 | I1815 2495 | sS'federal' 2496 | p833 2497 | I648 2498 | sS'review' 2499 | p834 2500 | I977 2501 | sS'outside' 2502 | p835 2503 | I697 2504 | sS'reading' 2505 | p836 2506 | I1322 2507 | sS'across' 2508 | p837 2509 | I968 2510 | sS'jumper' 2511 | p838 2512 | I1557 2513 | sS'notice' 2514 | p839 2515 | I600 2516 | sS'parent' 2517 | p840 2518 | I1063 2519 | sS'quadra' 2520 | p841 2521 | I1777 2522 | sS'screen' 2523 | p842 2524 | I493 2525 | sS'blame' 2526 | p843 2527 | I1259 2528 | sS'armenian' 2529 | p844 2530 | I374 2531 | sS'article' 2532 | p845 2533 | I6 2534 | sS'come' 2535 | p846 2536 | I22 2537 | sS'recall' 2538 | p847 2539 | I999 2540 | sS'reaction' 2541 | p848 2542 | I1504 2543 | sS'installation' 2544 | p849 2545 | I1655 2546 | sS'many' 2547 | p850 2548 | I37 2549 | sS'region' 2550 | p851 2551 | I1682 2552 | sS'contract' 2553 | p852 2554 | I1520 2555 | sS'somewhere' 2556 | p853 2557 | I1003 2558 | sS'duty' 2559 | p854 2560 | I1799 2561 | sS'among' 2562 | p855 2563 | I514 2564 | sS'armenia' 2565 | p856 2566 | I816 2567 | sS'cancer' 2568 | p857 2569 | I1610 2570 | sS'color' 2571 | p858 2572 | I272 2573 | sS'effective' 2574 | p859 2575 | I1027 2576 | sS'period' 2577 | p860 2578 | I425 2579 | sS'pop' 2580 | p861 2581 | I1646 2582 | sS'turkey' 2583 | p862 2584 | I772 2585 | sS'capable' 2586 | p863 2587 | I1488 2588 | sS'encryption' 2589 | p864 2590 | I300 2591 | sS'mark' 2592 | p865 2593 | I258 2594 | sS'hardly' 2595 | p866 2596 | I1726 2597 | sS'mary' 2598 | p867 2599 | I1465 2600 | sS'engine' 2601 | p868 2602 | I598 2603 | sS'direction' 2604 | p869 2605 | I1304 2606 | sS'thousand' 2607 | p870 2608 | I822 2609 | sS'andrew' 2610 | p871 2611 | I1018 2612 | sS'observe' 2613 | p872 2614 | I1828 2615 | sS'external' 2616 | p873 2617 | I1358 2618 | sS'evidence' 2619 | p874 2620 | I259 2621 | sS'spirit' 2622 | p875 2623 | I1132 2624 | sS'case' 2625 | p876 2626 | I71 2627 | sS'mount' 2628 | p877 2629 | I1644 2630 | sS'single' 2631 | p878 2632 | I392 2633 | sS'compile' 2634 | p879 2635 | I1303 2636 | sS'newspaper' 2637 | p880 2638 | I1580 2639 | sS'situation' 2640 | p881 2641 | I407 2642 | sS'telephone' 2643 | p882 2644 | I1232 2645 | sS'middle' 2646 | p883 2647 | I1084 2648 | sS'someone' 2649 | p884 2650 | I94 2651 | sS'somebody' 2652 | p885 2653 | I881 2654 | sS'technology' 2655 | p886 2656 | I271 2657 | sS'worry' 2658 | p887 2659 | I1221 2660 | sS'helmet' 2661 | p888 2662 | I1597 2663 | sS'different' 2664 | p889 2665 | I135 2666 | sS'pat' 2667 | p890 2668 | I1177 2669 | sS'doctor' 2670 | p891 2671 | I731 2672 | sS'pay' 2673 | p892 2674 | I215 2675 | sS'wiring' 2676 | p893 2677 | I1761 2678 | sS'dealer' 2679 | p894 2680 | I844 2681 | sS'speech' 2682 | p895 2683 | I1560 2684 | sS'document' 2685 | p896 2686 | I714 2687 | sS'status' 2688 | p897 2689 | I1293 2690 | sS'oil' 2691 | p898 2692 | I1024 2693 | sS'driver' 2694 | p899 2695 | I195 2696 | sS'scientist' 2697 | p900 2698 | I1216 2699 | sS'totally' 2700 | p901 2701 | I1220 2702 | sS'tradition' 2703 | p902 2704 | I1617 2705 | sS'without' 2706 | p903 2707 | I87 2708 | sS'solve' 2709 | p904 2710 | I1353 2711 | sS'model' 2712 | p905 2713 | I353 2714 | sS'modem' 2715 | p906 2716 | I789 2717 | sS'justify' 2718 | p907 2719 | I1765 2720 | sS'summer' 2721 | p908 2722 | I1308 2723 | sS'money' 2724 | p909 2725 | I251 2726 | sS'rest' 2727 | p910 2728 | I518 2729 | sS'violent' 2730 | p911 2731 | I1921 2732 | sS'kill' 2733 | p912 2734 | I160 2735 | sS'aspect' 2736 | p913 2737 | I1451 2738 | sS'touch' 2739 | p914 2740 | I1770 2741 | sS'speed' 2742 | p915 2743 | I252 2744 | sS'blow' 2745 | p916 2746 | I987 2747 | sS'death' 2748 | p917 2749 | I343 2750 | sS'tim' 2751 | p918 2752 | I1067 2753 | sS'thinking' 2754 | p919 2755 | I1830 2756 | sS'except' 2757 | p920 2758 | I455 2759 | sS'verse' 2760 | p921 2761 | I1684 2762 | sS'setting' 2763 | p922 2764 | I1454 2765 | sS'treatment' 2766 | p923 2767 | I1071 2768 | sS'real' 2769 | p924 2770 | I168 2771 | sS'around' 2772 | p925 2773 | I117 2774 | sS'read' 2775 | p926 2776 | I65 2777 | sS'papers' 2778 | p927 2779 | I1681 2780 | sS'mov' 2781 | p928 2782 | I1819 2783 | sS'early' 2784 | p929 2785 | I584 2786 | sS'traffic' 2787 | p930 2788 | I1137 2789 | sS'mon' 2790 | p931 2791 | I1663 2792 | sS'world' 2793 | p932 2794 | I105 2795 | sS'cd' 2796 | p933 2797 | I974 2798 | sS'outlet' 2799 | p934 2800 | I1665 2801 | sS'adams' 2802 | p935 2803 | I1823 2804 | sS'patrick' 2805 | p936 2806 | I1749 2807 | sS'server' 2808 | p937 2809 | I296 2810 | sS'benefit' 2811 | p938 2812 | I1049 2813 | sS'either' 2814 | p939 2815 | I159 2816 | sS'fully' 2817 | p940 2818 | I1305 2819 | sS'output' 2820 | p941 2821 | I395 2822 | sS'jews' 2823 | p942 2824 | I319 2825 | sS'lebanon' 2826 | p943 2827 | I1657 2828 | sS'table' 2829 | p944 2830 | I1053 2831 | sS'tube' 2832 | p945 2833 | I1849 2834 | sS'legal' 2835 | p946 2836 | I663 2837 | sS'moon' 2838 | p947 2839 | I990 2840 | sS'exit' 2841 | p948 2842 | I1913 2843 | sS'lunar' 2844 | p949 2845 | I1260 2846 | sS'knife' 2847 | p950 2848 | I1895 2849 | sS'refer' 2850 | p951 2851 | I590 2852 | sS'scientific' 2853 | p952 2854 | I843 2855 | sS'business' 2856 | p953 2857 | I505 2858 | sS'seconds' 2859 | p954 2860 | I1938 2861 | sS'equivalent' 2862 | p955 2863 | I1689 2864 | sS'resurrection' 2865 | p956 2866 | I1919 2867 | sS'throw' 2868 | p957 2869 | I661 2870 | sS'manufacturer' 2871 | p958 2872 | I1194 2873 | sS'oo' 2874 | p959 2875 | I1820 2876 | sS'comparison' 2877 | p960 2878 | I1419 2879 | sS'stone' 2880 | p961 2881 | I1701 2882 | sS'ok' 2883 | p962 2884 | I555 2885 | sS'oh' 2886 | p963 2887 | I461 2888 | sS'island' 2889 | p964 2890 | I1565 2891 | sS'industry' 2892 | p965 2893 | I1028 2894 | sS'marc' 2895 | p966 2896 | I1622 2897 | sS'greatly' 2898 | p967 2899 | I1117 2900 | sS'neighbor' 2901 | p968 2902 | I1154 2903 | sS'act' 2904 | p969 2905 | I292 2906 | sS'luck' 2907 | p970 2908 | I1257 2909 | sS'os' 2910 | p971 2911 | I1160 2912 | sS'processor' 2913 | p972 2914 | I1526 2915 | sS'road' 2916 | p973 2917 | I442 2918 | sS'cambridge' 2919 | p974 2920 | I1685 2921 | sS'communication' 2922 | p975 2923 | I802 2924 | sS'image' 2925 | p976 2926 | I171 2927 | sS'determine' 2928 | p977 2929 | I702 2930 | sS'gary' 2931 | p978 2932 | I1183 2933 | sS'prepare' 2934 | p979 2935 | I1600 2936 | sS'area' 2937 | p980 2938 | I230 2939 | sS'hey' 2940 | p981 2941 | I1055 2942 | sS'christianity' 2943 | p982 2944 | I776 2945 | sS'los' 2946 | p983 2947 | I1107 2948 | sS'low' 2949 | p984 2950 | I515 2951 | sS'lot' 2952 | p985 2953 | I92 2954 | sS'complete' 2955 | p986 2956 | I633 2957 | sS'enough' 2958 | p987 2959 | I141 2960 | sS'regard' 2961 | p988 2962 | I509 2963 | sS'pitcher' 2964 | p989 2965 | I1880 2966 | sS'cycle' 2967 | p990 2968 | I1611 2969 | sS'longer' 2970 | p991 2971 | I579 2972 | sS'pull' 2973 | p992 2974 | I845 2975 | sS'default' 2976 | p993 2977 | I1254 2978 | sS'organize' 2979 | p994 2980 | I1566 2981 | sS'agree' 2982 | p995 2983 | I256 2984 | sS'connect' 2985 | p996 2986 | I750 2987 | sS'strongly' 2988 | p997 2989 | I1963 2990 | sS'ac' 2991 | p998 2992 | I1582 2993 | sS'ab' 2994 | p999 2995 | I1977 2996 | sS'ad' 2997 | p1000 2998 | I1004 2999 | sS'certain' 3000 | p1001 3001 | I415 3002 | sS'arabs' 3003 | p1002 3004 | I1517 3005 | sS'describe' 3006 | p1003 3007 | I530 3008 | sS'al' 3009 | p1004 3010 | I806 3011 | sS'deep' 3012 | p1005 3013 | I1735 3014 | sS'general' 3015 | p1006 3016 | I213 3017 | sS'examine' 3018 | p1007 3019 | I1857 3020 | sS'ottawa' 3021 | p1008 3022 | I1846 3023 | sS'file' 3024 | p1009 3025 | I35 3026 | sS'senate' 3027 | p1010 3028 | I1730 3029 | sS'ax' 3030 | p1011 3031 | I1991 3032 | sS'fill' 3033 | p1012 3034 | I1283 3035 | sS'uucp' 3036 | p1013 3037 | I1289 3038 | sS'deserve' 3039 | p1014 3040 | I1373 3041 | sS'insurance' 3042 | p1015 3043 | I784 3044 | sS'field' 3045 | p1016 3046 | I603 3047 | sS'valid' 3048 | p1017 3049 | I1198 3050 | sS'poor' 3051 | p1018 3052 | I1006 3053 | sS'trip' 3054 | p1019 3055 | I1732 3056 | sS'congress' 3057 | p1020 3058 | I824 3059 | sS'separate' 3060 | p1021 3061 | I1034 3062 | sS'important' 3063 | p1022 3064 | I334 3065 | sS'coverage' 3066 | p1023 3067 | I1631 3068 | sS'chris' 3069 | p1024 3070 | I982 3071 | sS'building' 3072 | p1025 3073 | I783 3074 | sS'remote' 3075 | p1026 3076 | I1344 3077 | sS'wife' 3078 | p1027 3079 | I877 3080 | sS'mask' 3081 | p1028 3082 | I1944 3083 | sS'mass' 3084 | p1029 3085 | I924 3086 | sS'adam' 3087 | p1030 3088 | I1852 3089 | sS'directory' 3090 | p1031 3091 | I576 3092 | sS'resolution' 3093 | p1032 3094 | I1224 3095 | sS'cpu' 3096 | p1033 3097 | I1313 3098 | sS'bmw' 3099 | p1034 3100 | I1416 3101 | sS'represent' 3102 | p1035 3103 | I1155 3104 | sS'consider' 3105 | p1036 3106 | I124 3107 | sS'lack' 3108 | p1037 3109 | I746 3110 | sS'dollar' 3111 | p1038 3112 | I1203 3113 | sS'month' 3114 | p1039 3115 | I338 3116 | sS'spacecraft' 3117 | p1040 3118 | I1707 3119 | sS'coach' 3120 | p1041 3121 | I1710 3122 | sS'douglas' 3123 | p1042 3124 | I1927 3125 | sS'follow' 3126 | p1043 3127 | I100 3128 | sS'disk' 3129 | p1044 3130 | I197 3131 | sS'religious' 3132 | p1045 3133 | I634 3134 | sS'reasoning' 3135 | p1046 3136 | I1728 3137 | sS'apartment' 3138 | p1047 3139 | I1461 3140 | sS'tv' 3141 | p1048 3142 | I740 3143 | sS'rf' 3144 | p1049 3145 | I1987 3146 | sS'former' 3147 | p1050 3148 | I1032 3149 | sS'program' 3150 | p1051 3151 | I52 3152 | sS'health' 3153 | p1052 3154 | I561 3155 | sS'sound' 3156 | p1053 3157 | I232 3158 | sS'fax' 3159 | p1054 3160 | I289 3161 | sS'woman' 3162 | p1055 3163 | I318 3164 | sS'worse' 3165 | p1056 3166 | I862 3167 | sS'far' 3168 | p1057 3169 | I154 3170 | sS'faq' 3171 | p1058 3172 | I569 3173 | sS'wiretap' 3174 | p1059 3175 | I1649 3176 | sS'fan' 3177 | p1060 3178 | I500 3179 | sS'decide' 3180 | p1061 3181 | I409 3182 | sS'fall' 3183 | p1062 3184 | I511 3185 | sS'ticket' 3186 | p1063 3187 | I1449 3188 | sS'difference' 3189 | p1064 3190 | I297 3191 | sS'condition' 3192 | p1065 3193 | I532 3194 | sS'heaven' 3195 | p1066 3196 | I1062 3197 | sS'cable' 3198 | p1067 3199 | I728 3200 | sS'louis' 3201 | p1068 3202 | I1538 3203 | sS'list' 3204 | p1069 3205 | I107 3206 | sS'large' 3207 | p1070 3208 | I288 3209 | sS'small' 3210 | p1071 3211 | I286 3212 | sS'tel' 3213 | p1072 3214 | I1398 3215 | sS'andor' 3216 | p1073 3217 | I1202 3218 | sS'ten' 3219 | p1074 3220 | I964 3221 | sS'ted' 3222 | p1075 3223 | I1573 3224 | sS'past' 3225 | p1076 3226 | I560 3227 | sS'rate' 3228 | p1077 3229 | I309 3230 | sS'design' 3231 | p1078 3232 | I267 3233 | sS'ripem' 3234 | p1079 3235 | I1729 3236 | sS'pass' 3237 | p1080 3238 | I346 3239 | sS'material' 3240 | p1081 3241 | I827 3242 | sS'richard' 3243 | p1082 3244 | I926 3245 | sS'clock' 3246 | p1083 3247 | I1070 3248 | sS'sun' 3249 | p1084 3250 | I450 3251 | sS'section' 3252 | p1085 3253 | I471 3254 | sS'version' 3255 | p1086 3256 | I152 3257 | sS'public' 3258 | p1087 3259 | I183 3260 | sS'movement' 3261 | p1088 3262 | I1421 3263 | sS'full' 3264 | p1089 3265 | I371 3266 | sS'christian' 3267 | p1090 3268 | I127 3269 | sS'component' 3270 | p1091 3271 | I1506 3272 | sS'supposedly' 3273 | p1092 3274 | I1964 3275 | sS'strong' 3276 | p1093 3277 | I559 3278 | sS'search' 3279 | p1094 3280 | I738 3281 | sS'bible' 3282 | p1095 3283 | I287 3284 | sS'ahead' 3285 | p1096 3286 | I1467 3287 | sS'_eos_' 3288 | p1097 3289 | I0 3290 | sS'experience' 3291 | p1098 3292 | I305 3293 | sS'soldier' 3294 | p1099 3295 | I956 3296 | sS'amount' 3297 | p1100 3298 | I632 3299 | sS'pick' 3300 | p1101 3301 | I499 3302 | sS'action' 3303 | p1102 3304 | I351 3305 | sS'via' 3306 | p1103 3307 | I487 3308 | sS'family' 3309 | p1104 3310 | I620 3311 | sS'ask' 3312 | p1105 3313 | I81 3314 | sS'establish' 3315 | p1106 3316 | I984 3317 | sS'select' 3318 | p1107 3319 | I970 3320 | sS'eye' 3321 | p1108 3322 | I774 3323 | sS'proceed' 3324 | p1109 3325 | I1624 3326 | sS'revelation' 3327 | p1110 3328 | I1806 3329 | sS'two' 3330 | p1111 3331 | I40 3332 | sS'injury' 3333 | p1112 3334 | I1453 3335 | sS'minor' 3336 | p1113 3337 | I1127 3338 | sS'turbo' 3339 | p1114 3340 | I1850 3341 | sS'israel' 3342 | p1115 3343 | I239 3344 | sS'door' 3345 | p1116 3346 | I830 3347 | sS'company' 3348 | p1117 3349 | I279 3350 | sS'excuse' 3351 | p1118 3352 | I1320 3353 | sS'stick' 3354 | p1119 3355 | I841 3356 | sS'particular' 3357 | p1120 3358 | I523 3359 | sS'town' 3360 | p1121 3361 | I1116 3362 | sS'none' 3363 | p1122 3364 | I779 3365 | sS'hour' 3366 | p1123 3367 | I703 3368 | sS'science' 3369 | p1124 3370 | I274 3371 | sS'des' 3372 | p1125 3373 | I988 3374 | sS'det' 3375 | p1126 3376 | I1814 3377 | sS'remain' 3378 | p1127 3379 | I699 3380 | sS'bias' 3381 | p1128 3382 | I1545 3383 | sS'learn' 3384 | p1129 3385 | I451 3386 | sS'male' 3387 | p1130 3388 | I1392 3389 | sS'history' 3390 | p1131 3391 | I388 3392 | sS'compare' 3393 | p1132 3394 | I574 3395 | sS'brown' 3396 | p1133 3397 | I1874 3398 | sS'share' 3399 | p1134 3400 | I757 3401 | sS'accept' 3402 | p1135 3403 | I333 3404 | sS'states' 3405 | p1136 3406 | I690 3407 | sS'sense' 3408 | p1137 3409 | I406 3410 | sS'libertarian' 3411 | p1138 3412 | I1933 3413 | sS'homicide' 3414 | p1139 3415 | I1817 3416 | sS'information' 3417 | p1140 3418 | I67 3419 | sS'court' 3420 | p1141 3421 | I546 3422 | sS'goal' 3423 | p1142 3424 | I446 3425 | sS'rather' 3426 | p1143 3427 | I166 3428 | sS'reject' 3429 | p1144 3430 | I1362 3431 | sS'plant' 3432 | p1145 3433 | I1645 3434 | sS'okay' 3435 | p1146 3436 | I1300 3437 | sS'threat' 3438 | p1147 3439 | I1448 3440 | sS'binary' 3441 | p1148 3442 | I1609 3443 | sS'advice' 3444 | p1149 3445 | I1048 3446 | sS'long' 3447 | p1150 3448 | I121 3449 | sS'reflect' 3450 | p1151 3451 | I1821 3452 | sS'plane' 3453 | p1152 3454 | I1262 3455 | sS'director' 3456 | p1153 3457 | I1736 3458 | sS'blood' 3459 | p1154 3460 | I888 3461 | sS'develop' 3462 | p1155 3463 | I535 3464 | sS'response' 3465 | p1156 3466 | I470 3467 | sS'refuse' 3468 | p1157 3469 | I1412 3470 | sS'short' 3471 | p1158 3472 | I558 3473 | sS'register' 3474 | p1159 3475 | I1323 3476 | sS'responsibility' 3477 | p1160 3478 | I1147 3479 | sS'media' 3480 | p1161 3481 | I788 3482 | sS'scsi' 3483 | p1162 3484 | I638 3485 | sS'dream' 3486 | p1163 3487 | I1499 3488 | sS'help' 3489 | p1164 3490 | I64 3491 | sS'soon' 3492 | p1165 3493 | I611 3494 | sS'trade' 3495 | p1166 3496 | I653 3497 | sS'radar' 3498 | p1167 3499 | I1352 3500 | sS'attitude' 3501 | p1168 3502 | I1363 3503 | sS'paper' 3504 | p1169 3505 | I867 3506 | sS'scott' 3507 | p1170 3508 | I938 3509 | sS'committee' 3510 | p1171 3511 | I1105 3512 | sS'hell' 3513 | p1172 3514 | I457 3515 | sS'suffer' 3516 | p1173 3517 | I952 3518 | sS'islam' 3519 | p1174 3520 | I1153 3521 | sS'style' 3522 | p1175 3523 | I1531 3524 | sS'pray' 3525 | p1176 3526 | I1613 3527 | sS'actually' 3528 | p1177 3529 | I153 3530 | sS'late' 3531 | p1178 3532 | I936 3533 | sS'systems' 3534 | p1179 3535 | I769 3536 | sS'stephen' 3537 | p1180 3538 | I1385 3539 | sS'might' 3540 | p1181 3541 | I80 3542 | sS'fool' 3543 | p1182 3544 | I1570 3545 | sS'good' 3546 | p1183 3547 | I20 3548 | sS'return' 3549 | p1184 3550 | I308 3551 | sS'food' 3552 | p1185 3553 | I551 3554 | sS'propose' 3555 | p1186 3556 | I950 3557 | sS'scsus' 3558 | p1187 3559 | I1626 3560 | sS'inch' 3561 | p1188 3562 | I1915 3563 | sS'compound' 3564 | p1189 3565 | I1333 3566 | sS'foot' 3567 | p1190 3568 | I942 3569 | sS'complain' 3570 | p1191 3571 | I1640 3572 | sS'association' 3573 | p1192 3574 | I1575 3575 | sS'easily' 3576 | p1193 3577 | I825 3578 | sS'always' 3579 | p1194 3580 | I199 3581 | sS'capability' 3582 | p1195 3583 | I1253 3584 | sS'marry' 3585 | p1196 3586 | I1943 3587 | sS'heavy' 3588 | p1197 3589 | I1366 3590 | sS'week' 3591 | p1198 3592 | I236 3593 | sS'harm' 3594 | p1199 3595 | I1914 3596 | sS'everyone' 3597 | p1200 3598 | I357 3599 | sS'weight' 3600 | p1201 3601 | I1188 3602 | sS'generation' 3603 | p1202 3604 | I1579 3605 | sS'house' 3606 | p1203 3607 | I344 3608 | sS'energy' 3609 | p1204 3610 | I1342 3611 | sS'hard' 3612 | p1205 3613 | I177 3614 | sS'reduce' 3615 | p1206 3616 | I815 3617 | sS'idea' 3618 | p1207 3619 | I145 3620 | sS'brother' 3621 | p1208 3622 | I1234 3623 | sS'finish' 3624 | p1209 3625 | I1265 3626 | sS'expect' 3627 | p1210 3628 | I317 3629 | sS'operation' 3630 | p1211 3631 | I808 3632 | sS'beyond' 3633 | p1212 3634 | I1037 3635 | sS'event' 3636 | p1213 3637 | I421 3638 | sS'really' 3639 | p1214 3640 | I56 3641 | sS'ftp' 3642 | p1215 3643 | I408 3644 | sS'robert' 3645 | p1216 3646 | I585 3647 | sS'since' 3648 | p1217 3649 | I57 3650 | sS'publish' 3651 | p1218 3652 | I721 3653 | sS'research' 3654 | p1219 3655 | I235 3656 | sS'safety' 3657 | p1220 3658 | I775 3659 | sS'print' 3660 | p1221 3661 | I799 3662 | sS'issue' 3663 | p1222 3664 | I155 3665 | sS'belief' 3666 | p1223 3667 | I453 3668 | sS'upon' 3669 | p1224 3670 | I602 3671 | sS'story' 3672 | p1225 3673 | I533 3674 | sS'reason' 3675 | p1226 3676 | I114 3677 | sS'base' 3678 | p1227 3679 | I162 3680 | sS'imagine' 3681 | p1228 3682 | I865 3683 | sS'put' 3684 | p1229 3685 | I97 3686 | sS'teach' 3687 | p1230 3688 | I912 3689 | sS'beginning' 3690 | p1231 3691 | I1796 3692 | sS'generate' 3693 | p1232 3694 | I955 3695 | sS'definition' 3696 | p1233 3697 | I853 3698 | sS'thread' 3699 | p1234 3700 | I1045 3701 | sS'launch' 3702 | p1235 3703 | I513 3704 | sS'prophet' 3705 | p1236 3706 | I1935 3707 | sS'american' 3708 | p1237 3709 | I282 3710 | sS'circuit' 3711 | p1238 3712 | I922 3713 | sS'feed' 3714 | p1239 3715 | I1491 3716 | sS'major' 3717 | p1240 3718 | I428 3719 | sS'feel' 3720 | p1241 3721 | I210 3722 | sS'relate' 3723 | p1242 3724 | I1423 3725 | sS'number' 3726 | p1243 3727 | I54 3728 | sS'miss' 3729 | p1244 3730 | I570 3731 | sS'guess' 3732 | p1245 3733 | I349 3734 | sS'jew' 3735 | p1246 3736 | I1443 3737 | sS'jet' 3738 | p1247 3739 | I1272 3740 | sS'bay' 3741 | p1248 3742 | I1651 3743 | sS'least' 3744 | p1249 3745 | I132 3746 | sS'paint' 3747 | p1250 3748 | I1955 3749 | sS'regulation' 3750 | p1251 3751 | I1866 3752 | sS'station' 3753 | p1252 3754 | I715 3755 | sS'statement' 3756 | p1253 3757 | I379 3758 | sS'compromise' 3759 | p1254 3760 | I1521 3761 | sS'scheme' 3762 | p1255 3763 | I1327 3764 | sS'southern' 3765 | p1256 3766 | I1962 3767 | sS'store' 3768 | p1257 3769 | I861 3770 | sS'option' 3771 | p1258 3772 | I552 3773 | sS'relationship' 3774 | p1259 3775 | I1549 3776 | sS'park' 3777 | p1260 3778 | I1351 3779 | sS'prediction' 3780 | p1261 3781 | I1804 3782 | sS'part' 3783 | p1262 3784 | I76 3785 | sS'translation' 3786 | p1263 3787 | I1379 3788 | sS'believe' 3789 | p1264 3790 | I49 3791 | sS'fred' 3792 | p1265 3793 | I1586 3794 | sS'king' 3795 | p1266 3796 | I904 3797 | sS'kind' 3798 | p1267 3799 | I206 3800 | sS'hundred' 3801 | p1268 3802 | I1041 3803 | sS'double' 3804 | p1269 3805 | I1235 3806 | sS'instruction' 3807 | p1270 3808 | I1256 3809 | sS'greg' 3810 | p1271 3811 | I1508 3812 | sS'marriage' 3813 | p1272 3814 | I1558 3815 | sS'toward' 3816 | p1273 3817 | I1450 3818 | sS'bhj' 3819 | p1274 3820 | I1994 3821 | sS'contrary' 3822 | p1275 3823 | I1948 3824 | sS'null' 3825 | p1276 3826 | I1635 3827 | sS'sell' 3828 | p1277 3829 | I222 3830 | sS'mountain' 3831 | p1278 3832 | I1335 3833 | sS'check' 3834 | p1279 3835 | I238 3836 | sS'self' 3837 | p1280 3838 | I1513 3839 | sS'majority' 3840 | p1281 3841 | I1023 3842 | sS'internal' 3843 | p1282 3844 | I855 3845 | sS'build' 3846 | p1283 3847 | I209 3848 | sS'play' 3849 | p1284 3850 | I113 3851 | sS'towards' 3852 | p1285 3853 | I1332 3854 | sS'quote' 3855 | p1286 3856 | I424 3857 | sS'reach' 3858 | p1287 3859 | I813 3860 | sS'plan' 3861 | p1288 3862 | I389 3863 | sS'significant' 3864 | p1289 3865 | I1246 3866 | sS'nothing' 3867 | p1290 3868 | I169 3869 | sS'extremely' 3870 | p1291 3871 | I1725 3872 | sS'constitution' 3873 | p1292 3874 | I1022 3875 | sS'windows' 3876 | p1293 3877 | I196 3878 | sS'clear' 3879 | p1294 3880 | I440 3881 | sS'sometimes' 3882 | p1295 3883 | I591 3884 | sS'cover' 3885 | p1296 3886 | I387 3887 | sS'km' 3888 | p1297 3889 | I1984 3890 | sS'clean' 3891 | p1298 3892 | I1496 3893 | sS'usual' 3894 | p1299 3895 | I1592 3896 | sS'physics' 3897 | p1300 3898 | I1667 3899 | sS'microsoft' 3900 | p1301 3901 | I1471 3902 | sS'sector' 3903 | p1302 3904 | I1741 3905 | sS'thomas' 3906 | p1303 3907 | I957 3908 | sS'particularly' 3909 | p1304 3910 | I1073 3911 | sS'session' 3912 | p1305 3913 | I1514 3914 | sS'relation' 3915 | p1306 3916 | I1740 3917 | sS'font' 3918 | p1307 3919 | I689 3920 | sS'fine' 3921 | p1308 3922 | I495 3923 | sS'find' 3924 | p1309 3925 | I33 3926 | sS'impact' 3927 | p1310 3928 | I1643 3929 | sS'cell' 3930 | p1311 3931 | I1782 3932 | sS'copyright' 3933 | p1312 3934 | I1888 3935 | sS'justice' 3936 | p1313 3937 | I1192 3938 | sS'writer' 3939 | p1314 3940 | I1946 3941 | sS'penalty' 3942 | p1315 3943 | I1069 3944 | sS'pretty' 3945 | p1316 3946 | I294 3947 | sS'factor' 3948 | p1317 3949 | I1146 3950 | sS'hit' 3951 | p1318 3952 | I345 3953 | sS'express' 3954 | p1319 3955 | I854 3956 | sS'cheaper' 3957 | p1320 3958 | I1519 3959 | sS'international' 3960 | p1321 3961 | I729 3962 | sS'remove' 3963 | p1322 3964 | I712 3965 | sS'common' 3966 | p1323 3967 | I496 3968 | sS'activity' 3969 | p1324 3970 | I747 3971 | sS'rear' 3972 | p1325 3973 | I1829 3974 | sS'set' 3975 | p1326 3976 | I98 3977 | sS'art' 3978 | p1327 3979 | I1210 3980 | sS'intelligence' 3981 | p1328 3982 | I1523 3983 | sS'sex' 3984 | p1329 3985 | I1079 3986 | sS'culture' 3987 | p1330 3988 | I1576 3989 | sS'see' 3990 | p1331 3991 | I13 3992 | sS'defense' 3993 | p1332 3994 | I722 3995 | sS'sea' 3996 | p1333 3997 | I1713 3998 | sS'close' 3999 | p1334 4000 | I363 4001 | sS'arm' 4002 | p1335 4003 | I510 4004 | sS'expert' 4005 | p1336 4006 | I1222 4007 | sS'movie' 4008 | p1337 4009 | I1800 4010 | sS'currently' 4011 | p1338 4012 | I665 4013 | sS'please' 4014 | p1339 4015 | I55 4016 | sS'various' 4017 | p1340 4018 | I664 4019 | sS'probably' 4020 | p1341 4021 | I118 4022 | sS'europe' 4023 | p1342 4024 | I1205 4025 | sS'recently' 4026 | p1343 4027 | I608 4028 | sS'attention' 4029 | p1344 4030 | I1408 4031 | sS'interface' 4032 | p1345 4033 | I622 4034 | sS'league' 4035 | p1346 4036 | I516 4037 | sS'last' 4038 | p1347 4039 | I74 4040 | sS'license' 4041 | p1348 4042 | I1159 4043 | sS'hint' 4044 | p1349 4045 | I1737 4046 | sS'foreign' 4047 | p1350 4048 | I1478 4049 | sS'connection' 4050 | p1351 4051 | I1008 4052 | sS'context' 4053 | p1352 4054 | I932 4055 | sS'whole' 4056 | p1353 4057 | I302 4058 | sS'load' 4059 | p1354 4060 | I778 4061 | sS'jeff' 4062 | p1355 4063 | I1298 4064 | sS'simple' 4065 | p1356 4066 | I486 4067 | sS'sky' 4068 | p1357 4069 | I1908 4070 | sS'community' 4071 | p1358 4072 | I794 4073 | sS'simply' 4074 | p1359 4075 | I358 4076 | sS'village' 4077 | p1360 4078 | I954 4079 | sS'expensive' 4080 | p1361 4081 | I1081 4082 | sS'ps' 4083 | p1362 4084 | I1973 4085 | sS'pp' 4086 | p1363 4087 | I1445 4088 | sS'raise' 4089 | p1364 4090 | I846 4091 | sS'pt' 4092 | p1365 4093 | I852 4094 | sS'create' 4095 | p1366 4096 | I233 4097 | sS'political' 4098 | p1367 4099 | I681 4100 | sS'due' 4101 | p1368 4102 | I566 4103 | sS'pc' 4104 | p1369 4105 | I354 4106 | sS'secret' 4107 | p1370 4108 | I727 4109 | sS'damn' 4110 | p1371 4111 | I1564 4112 | sS'pd' 4113 | p1372 4114 | I1974 4115 | sS'territory' 4116 | p1373 4117 | I1591 4118 | sS'meeting' 4119 | p1374 4120 | I1393 4121 | sS'pl' 4122 | p1375 4123 | I1990 4124 | sS'pm' 4125 | p1376 4126 | I1473 4127 | sS'modern' 4128 | p1377 4129 | I1043 4130 | sS'flight' 4131 | p1378 4132 | I1106 4133 | sS'gay' 4134 | p1379 4135 | I1321 4136 | sS'fire' 4137 | p1380 4138 | I366 4139 | sS'crack' 4140 | p1381 4141 | I1889 4142 | sS'gas' 4143 | p1382 4144 | I752 4145 | sS'remark' 4146 | p1383 4147 | I1292 4148 | sS'else' 4149 | p1384 4150 | I178 4151 | sS'fund' 4152 | p1385 4153 | I842 4154 | sS'understand' 4155 | p1386 4156 | I247 4157 | sS'representative' 4158 | p1387 4159 | I1902 4160 | sS'demand' 4161 | p1388 4162 | I1113 4163 | sS'georgia' 4164 | p1389 4165 | I1410 4166 | sS'look' 4167 | p1390 4168 | I27 4169 | sS'solid' 4170 | p1391 4171 | I1459 4172 | sS'straight' 4173 | p1392 4174 | I1111 4175 | sS'bill' 4176 | p1393 4177 | I257 4178 | sS'budget' 4179 | p1394 4180 | I1387 4181 | sS'receive' 4182 | p1395 4183 | I381 4184 | sS'behavior' 4185 | p1396 4186 | I1302 4187 | sS'error' 4188 | p1397 4189 | I356 4190 | sS'fun' 4191 | p1398 4192 | I1390 4193 | sS'guide' 4194 | p1399 4195 | I1239 4196 | sS'century' 4197 | p1400 4198 | I1161 4199 | sS'ready' 4200 | p1401 4201 | I1367 4202 | sS'funny' 4203 | p1402 4204 | I1843 4205 | sS'anymore' 4206 | p1403 4207 | I1794 4208 | sS'grant' 4209 | p1404 4210 | I826 4211 | sS'belong' 4212 | p1405 4213 | I1544 4214 | sS'relatively' 4215 | p1406 4216 | I1788 4217 | sS'conflict' 4218 | p1407 4219 | I1213 4220 | sS'higher' 4221 | p1408 4222 | I734 4223 | sS'development' 4224 | p1409 4225 | I733 4226 | sS'lie' 4227 | p1410 4228 | I627 4229 | sS'yesterday' 4230 | p1411 4231 | I1441 4232 | sS'moment' 4233 | p1412 4234 | I1171 4235 | sS'purpose' 4236 | p1413 4237 | I436 4238 | sS'recent' 4239 | p1414 4240 | I905 4241 | sS'lower' 4242 | p1415 4243 | I897 4244 | sS'task' 4245 | p1416 4246 | I1760 4247 | sS'database' 4248 | p1417 4249 | I1462 4250 | sS'anybody' 4251 | p1418 4252 | I385 4253 | sS'analysis' 4254 | p1419 4255 | I1000 4256 | sS'obviously' 4257 | p1420 4258 | I836 4259 | sS'person' 4260 | p1421 4261 | I164 4262 | sS'edge' 4263 | p1422 4264 | I1176 4265 | sS'organization' 4266 | p1423 4267 | I528 4268 | sS'spend' 4269 | p1424 4270 | I564 4271 | sS'know' 4272 | p1425 4273 | I7 4274 | sS'shape' 4275 | p1426 4276 | I1676 4277 | sS'amp' 4278 | p1427 4279 | I1678 4280 | sS'useful' 4281 | p1428 4282 | I828 4283 | sS'alternative' 4284 | p1429 4285 | I1156 4286 | sS'simm' 4287 | p1430 4288 | I1021 4289 | sS'cut' 4290 | p1431 4291 | I527 4292 | sS'also' 4293 | p1432 4294 | I15 4295 | sS'danger' 4296 | p1433 4297 | I1825 4298 | sS'source' 4299 | p1434 4300 | I161 4301 | sS'location' 4302 | p1435 4303 | I1444 4304 | sS'input' 4305 | p1436 4306 | I711 4307 | sS'bio' 4308 | p1437 4309 | I1703 4310 | sS'australia' 4311 | p1438 4312 | I1518 4313 | sS'march' 4314 | p1439 4315 | I959 4316 | sS'format' 4317 | p1440 4318 | I420 4319 | sS'big' 4320 | p1441 4321 | I231 4322 | sS'couple' 4323 | p1442 4324 | I384 4325 | sS'xterm' 4326 | p1443 4327 | I1310 4328 | sS'game' 4329 | p1444 4330 | I72 4331 | sS'bit' 4332 | p1445 4333 | I128 4334 | sS'wings' 4335 | p1446 4336 | I1922 4337 | sS'helpful' 4338 | p1447 4339 | I1969 4340 | sS'signal' 4341 | p1448 4342 | I929 4343 | sS'ignore' 4344 | p1449 4345 | I1009 4346 | sS'collect' 4347 | p1450 4348 | I1724 4349 | sS'continue' 4350 | p1451 4351 | I445 4352 | sS'popular' 4353 | p1452 4354 | I1267 4355 | sS'often' 4356 | p1453 4357 | I352 4358 | sS'absolutely' 4359 | p1454 4360 | I1201 4361 | sS'spring' 4362 | p1455 4363 | I1719 4364 | sS'creation' 4365 | p1456 4366 | I1389 4367 | sS'back' 4368 | p1457 4369 | I59 4370 | sS'economic' 4371 | p1458 4372 | I1381 4373 | sS'mirror' 4374 | p1459 4375 | I1862 4376 | sS'curious' 4377 | p1460 4378 | I1660 4379 | sS'scale' 4380 | p1461 4381 | I1527 4382 | sS'decision' 4383 | p1462 4384 | I795 4385 | sS'shall' 4386 | p1463 4387 | I706 4388 | sS'per' 4389 | p1464 4390 | I479 4391 | sS'religion' 4392 | p1465 4393 | I326 4394 | sS'pen' 4395 | p1466 4396 | I1934 4397 | sS'eliminate' 4398 | p1467 4399 | I1939 4400 | sS'patient' 4401 | p1468 4402 | I937 4403 | sS'bb' 4404 | p1469 4405 | I1619 4406 | sS'word' 4407 | p1470 4408 | I130 4409 | sS'processing' 4410 | p1471 4411 | I1495 4412 | sS'bh' 4413 | p1472 4414 | I1967 4415 | sS'step' 4416 | p1473 4417 | I890 4418 | sS'faith' 4419 | p1474 4420 | I412 4421 | sS'anything' 4422 | p1475 4423 | I115 4424 | sS'range' 4425 | p1476 4426 | I812 4427 | sS'militia' 4428 | p1477 4429 | I1059 4430 | sS'mouth' 4431 | p1478 4432 | I1954 4433 | sS'ide' 4434 | p1479 4435 | I856 4436 | sS'block' 4437 | p1480 4438 | I683 4439 | sS'clinton' 4440 | p1481 4441 | I542 4442 | sS'colorado' 4443 | p1482 4444 | I1639 4445 | sS'nsa' 4446 | p1483 4447 | I967 4448 | sS'within' 4449 | p1484 4450 | I437 4451 | sS'services' 4452 | p1485 4453 | I1627 4454 | sS'appropriate' 4455 | p1486 4456 | I893 4457 | sS'statistics' 4458 | p1487 4459 | I1759 4460 | sS'apparently' 4461 | p1488 4462 | I886 4463 | sS'question' 4464 | p1489 4465 | I45 4466 | sS'fast' 4467 | p1490 4468 | I488 4469 | sS'camera' 4470 | p1491 4471 | I1875 4472 | sS'forward' 4473 | p1492 4474 | I1125 4475 | sS'manufacture' 4476 | p1493 4477 | I1953 4478 | sS'jewish' 4479 | p1494 4480 | I512 4481 | sS'poster' 4482 | p1495 4483 | I1178 4484 | sS'calgary' 4485 | p1496 4486 | I1839 4487 | sS'properly' 4488 | p1497 4489 | I1338 4490 | sS'link' 4491 | p1498 4492 | I1029 4493 | sS'authority' 4494 | p1499 4495 | I597 4496 | sS'russian' 4497 | p1500 4498 | I805 4499 | sS'line' 4500 | p1501 4501 | I96 4502 | sS'info' 4503 | p1502 4504 | I278 4505 | sS'us' 4506 | p1503 4507 | I276 4508 | sS'planet' 4509 | p1504 4510 | I1623 4511 | sS'uk' 4512 | p1505 4513 | I1309 4514 | sS'similar' 4515 | p1506 4516 | I411 4517 | sS'professor' 4518 | p1507 4519 | I1712 4520 | sS'bell' 4521 | p1508 4522 | I1694 4523 | sS'char' 4524 | p1509 4525 | I1388 4526 | sS'engineering' 4527 | p1510 4528 | I879 4529 | sS'warning' 4530 | p1511 4531 | I1618 4532 | sS'buffer' 4533 | p1512 4534 | I1754 4535 | sS'peace' 4536 | p1513 4537 | I668 4538 | sS'armenians' 4539 | p1514 4540 | I438 4541 | sS'application' 4542 | p1515 4543 | I240 4544 | sS'electrical' 4545 | p1516 4546 | I1596 4547 | sS'department' 4548 | p1517 4549 | I562 4550 | sS'nice' 4551 | p1518 4552 | I449 4553 | sS'draw' 4554 | p1519 4555 | I589 4556 | sS'william' 4557 | p1520 4558 | I1593 4559 | sS'meaning' 4560 | p1521 4561 | I1030 4562 | sS'structure' 4563 | p1522 4564 | I1200 4565 | sS'ago' 4566 | p1523 4567 | I275 4568 | sS'land' 4569 | p1524 4570 | I539 4571 | sS'algorithm' 4572 | p1525 4573 | I613 4574 | sS'age' 4575 | p1526 4576 | I793 4577 | sS'orbit' 4578 | p1527 4579 | I860 4580 | sS'nasa' 4581 | p1528 4582 | I599 4583 | sS'hello' 4584 | p1529 4585 | I1158 4586 | sS'code' 4587 | p1530 4588 | I220 4589 | sS'jason' 4590 | p1531 4591 | I1456 4592 | sS'resistance' 4593 | p1532 4594 | I1637 4595 | sS'steve' 4596 | p1533 4597 | I369 4598 | sS'gang' 4599 | p1534 4600 | I1930 4601 | sS'go' 4602 | p1535 4603 | I10 4604 | sS'gm' 4605 | p1536 4606 | I933 4607 | sS'turkish' 4608 | p1537 4609 | I347 4610 | sS'young' 4611 | p1538 4612 | I616 4613 | sS'send' 4614 | p1539 4615 | I89 4616 | sS'include' 4617 | p1540 4618 | I60 4619 | sS'probe' 4620 | p1541 4621 | I1437 4622 | sS'wave' 4623 | p1542 4624 | I1426 4625 | sS'entire' 4626 | p1543 4627 | I872 4628 | sS'transmission' 4629 | p1544 4630 | I1848 4631 | sS'button' 4632 | p1545 4633 | I1372 4634 | sS'michael' 4635 | p1546 4636 | I469 4637 | sS'ryan' 4638 | p1547 4639 | I1638 4640 | sS'try' 4641 | p1548 4642 | I36 4643 | sS'race' 4644 | p1549 4645 | I1074 4646 | sS'challenge' 4647 | p1550 4648 | I1745 4649 | sS'disclaimer' 4650 | p1551 4651 | I787 4652 | sS'smaller' 4653 | p1552 4654 | I1751 4655 | sS'jump' 4656 | p1553 4657 | I1546 4658 | sS'imply' 4659 | p1554 4660 | I1058 4661 | sS'video' 4662 | p1555 4663 | I447 4664 | sS'folk' 4665 | p1556 4666 | I837 4667 | sS'index' 4668 | p1557 4669 | I1833 4670 | sS'power' 4671 | p1558 4672 | I95 4673 | sS'chicago' 4674 | p1559 4675 | I829 4676 | sS'access' 4677 | p1560 4678 | I298 4679 | sS'experiment' 4680 | p1561 4681 | I1468 4682 | sS'capital' 4683 | p1562 4684 | I1947 4685 | sS'exercise' 4686 | p1563 4687 | I1826 4688 | sS'body' 4689 | p1564 4690 | I368 4691 | sS'lee' 4692 | p1565 4693 | I1509 4694 | sS'degree' 4695 | p1566 4696 | I1123 4697 | sS'exchange' 4698 | p1567 4699 | I1877 4700 | sS'commercial' 4701 | p1568 4702 | I823 4703 | sS'let' 4704 | p1569 4705 | I77 4706 | sS'consideration' 4707 | p1570 4708 | I1763 4709 | sS'great' 4710 | p1571 4711 | I147 4712 | sS'technical' 4713 | p1572 4714 | I770 4715 | sS'larger' 4716 | p1573 4717 | I1383 4718 | sS'opinion' 4719 | p1574 4720 | I139 4721 | sS'implement' 4722 | p1575 4723 | I1229 4724 | sS'apple' 4725 | p1576 4726 | I519 4727 | sS'heart' 4728 | p1577 4729 | I975 4730 | sS'apr' 4731 | p1578 4732 | I773 4733 | sS'win' 4734 | p1579 4735 | I170 4736 | sS'app' 4737 | p1580 4738 | I1543 4739 | sS'manage' 4740 | p1581 4741 | I1097 4742 | sS'private' 4743 | p1582 4744 | I459 4745 | sS'motor' 4746 | p1583 4747 | I1716 4748 | sS'apply' 4749 | p1584 4750 | I473 4751 | sS'gateway' 4752 | p1585 4753 | I1691 4754 | sS'use' 4755 | p1586 4756 | I3 4757 | sS'fee' 4758 | p1587 4759 | I1690 4760 | sS'stream' 4761 | p1588 4762 | I1278 4763 | sS'usa' 4764 | p1589 4765 | I619 4766 | sS'illegal' 4767 | p1590 4768 | I1096 4769 | sS'next' 4770 | p1591 4771 | I182 4772 | sS'doubt' 4773 | p1592 4774 | I636 4775 | sS'vehicle' 4776 | p1593 4777 | I892 4778 | sS'stage' 4779 | p1594 4780 | I1515 4781 | sS'start' 4782 | p1595 4783 | I68 4784 | sS'sort' 4785 | p1596 4786 | I350 4787 | sS'biblical' 4788 | p1597 4789 | I1722 4790 | sS'train' 4791 | p1598 4792 | I1870 4793 | sS'baby' 4794 | p1599 4795 | I1810 4796 | sS'central' 4797 | p1600 4798 | I1840 4799 | sS'iii' 4800 | p1601 4801 | I1932 4802 | sS'customer' 4803 | p1602 4804 | I1926 4805 | sS'topic' 4806 | p1603 4807 | I1099 4808 | sS'account' 4809 | p1604 4810 | I710 4811 | sS'ride' 4812 | p1605 4813 | I544 4814 | sS'package' 4815 | p1606 4816 | I315 4817 | sS'anywhere' 4818 | p1607 4819 | I1442 4820 | sS'obvious' 4821 | p1608 4822 | I868 4823 | sS'freedom' 4824 | p1609 4825 | I724 4826 | sS'meet' 4827 | p1610 4828 | I680 4829 | sS'proof' 4830 | p1611 4831 | I1168 4832 | sS'control' 4833 | p1612 4834 | I172 4835 | sS'process' 4836 | p1613 4837 | I472 4838 | sS'lock' 4839 | p1614 4840 | I1131 4841 | sS'tax' 4842 | p1615 4843 | I503 4844 | sS'high' 4845 | p1616 4846 | I198 4847 | sS'something' 4848 | p1617 4849 | I53 4850 | sS'serial' 4851 | p1618 4852 | I791 4853 | sS'rape' 4854 | p1619 4855 | I1958 4856 | sS'compatible' 4857 | p1620 4858 | I1431 4859 | sS'united' 4860 | p1621 4861 | I629 4862 | sS'sit' 4863 | p1622 4864 | I838 4865 | sS'six' 4866 | p1623 4867 | I1144 4868 | sS'brian' 4869 | p1624 4870 | I763 4871 | sS'animal' 4872 | p1625 4873 | I1040 4874 | sS'instead' 4875 | p1626 4876 | I380 4877 | sS'sin' 4878 | p1627 4879 | I686 4880 | sS'stand' 4881 | p1628 4882 | I481 4883 | sS'honda' 4884 | p1629 4885 | I1773 4886 | sS'watch' 4887 | p1630 4888 | I476 4889 | sS'collection' 4890 | p1631 4891 | I1671 4892 | sS'abuse' 4893 | p1632 4894 | I1279 4895 | sS'light' 4896 | p1633 4897 | I367 4898 | sS'element' 4899 | p1634 4900 | I1463 4901 | sS'allow' 4902 | p1635 4903 | I179 4904 | sS'houston' 4905 | p1636 4906 | I1931 4907 | sS'move' 4908 | p1637 4909 | I241 4910 | sS'motorcycle' 4911 | p1638 4912 | I1230 4913 | sS'bunch' 4914 | p1639 4915 | I1212 4916 | sS'perfect' 4917 | p1640 4918 | I1011 4919 | sS'la' 4920 | p1641 4921 | I667 4922 | sS'newsgroup' 4923 | p1642 4924 | I482 4925 | sS'choose' 4926 | p1643 4927 | I524 4928 | sS'criminal' 4929 | p1644 4930 | I594 4931 | sS'pixel' 4932 | p1645 4933 | I1717 4934 | sS'eus' 4935 | p1646 4936 | I1988 4937 | sS'crash' 4938 | p1647 4939 | I1497 4940 | sS'greater' 4941 | p1648 4942 | I1306 4943 | sS'auto' 4944 | p1649 4945 | I1608 4946 | sS'dan' 4947 | p1650 4948 | I1068 4949 | sS'practice' 4950 | p1651 4951 | I726 4952 | sS'mention' 4953 | p1652 4954 | I273 4955 | sS'front' 4956 | p1653 4957 | I595 4958 | sS'day' 4959 | p1654 4960 | I61 4961 | sS'february' 4962 | p1655 4963 | I1941 4964 | sS'university' 4965 | p1656 4966 | I104 4967 | sS'mode' 4968 | p1657 4969 | I404 4970 | sS'truth' 4971 | p1658 4972 | I341 4973 | sS'strip' 4974 | p1659 4975 | I1731 4976 | sS'related' 4977 | p1660 4978 | I1143 4979 | sS'society' 4980 | p1661 4981 | I534 4982 | sS'frequency' 4983 | p1662 4984 | I1786 4985 | sS'measure' 4986 | p1663 4987 | I875 4988 | sS'programmer' 4989 | p1664 4990 | I1865 4991 | sS'patch' 4992 | p1665 4993 | I1472 4994 | sS'sexual' 4995 | p1666 4996 | I1307 4997 | sS'special' 4998 | p1667 4999 | I605 5000 | sS'category' 5001 | p1668 5002 | I1925 5003 | sS'defend' 5004 | p1669 5005 | I1286 5006 | sS'electronics' 5007 | p1670 5008 | I1672 5009 | sS'cause' 5010 | p1671 5011 | I173 5012 | sS'red' 5013 | p1672 5014 | I587 5015 | sS'shut' 5016 | p1673 5017 | I1907 5018 | sS'frank' 5019 | p1674 5020 | I864 5021 | sS'ban' 5022 | p1675 5023 | I1341 5024 | sS'completely' 5025 | p1676 5026 | I742 5027 | sS'surely' 5028 | p1677 5029 | I1605 5030 | sS'york' 5031 | p1678 5032 | I612 5033 | sS'automatic' 5034 | p1679 5035 | I1551 5036 | sS'david' 5037 | p1680 5038 | I175 5039 | sS'conversation' 5040 | p1681 5041 | I1403 5042 | sS'length' 5043 | p1682 5044 | I1572 5045 | sS'hence' 5046 | p1683 5047 | I1844 5048 | sS'south' 5049 | p1684 5050 | I920 5051 | sS'confuse' 5052 | p1685 5053 | I1687 5054 | sS'jesus' 5055 | p1686 5056 | I151 5057 | sS'koresh' 5058 | p1687 5059 | I1042 5060 | sS'submit' 5061 | p1688 5062 | I1742 5063 | sS'pgp' 5064 | p1689 5065 | I1273 5066 | sS'owner' 5067 | p1690 5068 | I761 5069 | sS'quality' 5070 | p1691 5071 | I705 5072 | sS'management' 5073 | p1692 5074 | I1315 5075 | sS'reference' 5076 | p1693 5077 | I365 5078 | sS'privacy' 5079 | p1694 5080 | I675 5081 | sS'system' 5082 | p1695 5083 | I29 5084 | sS'attach' 5085 | p1696 5086 | I1903 5087 | sS'attack' 5088 | p1697 5089 | I316 5090 | sS'perfectly' 5091 | p1698 5092 | I1776 5093 | sS'final' 5094 | p1699 5095 | I666 5096 | sS'enforcement' 5097 | p1700 5098 | I871 5099 | sS'shell' 5100 | p1701 5101 | I1163 5102 | sS'gear' 5103 | p1702 5104 | I1891 5105 | sS'explanation' 5106 | p1703 5107 | I1578 5108 | sS'exactly' 5109 | p1704 5110 | I454 5111 | sS'rsa' 5112 | p1705 5113 | I1675 5114 | sS'bother' 5115 | p1706 5116 | I1112 5117 | sS'individual' 5118 | p1707 5119 | I430 5120 | sS'bet' 5121 | p1708 5122 | I1552 5123 | sS'imho' 5124 | p1709 5125 | I1711 5126 | sS'false' 5127 | p1710 5128 | I1038 5129 | sS'unlikely' 5130 | p1711 5131 | I1910 5132 | sS'need' 5133 | p1712 5134 | I31 5135 | sS'border' 5136 | p1713 5137 | I1238 5138 | sS'clearly' 5139 | p1714 5140 | I694 5141 | sS'screw' 5142 | p1715 5143 | I1276 5144 | sS'afraid' 5145 | p1716 5146 | I1562 5147 | sS'min' 5148 | p1717 5149 | I1845 5150 | sS'agency' 5151 | p1718 5152 | I631 5153 | sS'able' 5154 | p1719 5155 | I191 5156 | sS'mechanism' 5157 | p1720 5158 | I1501 5159 | sS'instance' 5160 | p1721 5161 | I1199 5162 | sS'detector' 5163 | p1722 5164 | I1595 5165 | sS'brand' 5166 | p1723 5167 | I1417 5168 | sS'unless' 5169 | p1724 5170 | I401 5171 | sS'sl' 5172 | p1725 5173 | I1992 5174 | sS'cipher' 5175 | p1726 5176 | I1886 5177 | sS'device' 5178 | p1727 5179 | I342 5180 | sS'class' 5181 | p1728 5182 | I649 5183 | sS'deny' 5184 | p1729 5185 | I983 5186 | sS'request' 5187 | p1730 5188 | I491 5189 | sS'disease' 5190 | p1731 5191 | I759 5192 | sS'face' 5193 | p1732 5194 | I429 5195 | sS'normally' 5196 | p1733 5197 | I1483 5198 | sS'fact' 5199 | p1734 5200 | I91 5201 | sS'violation' 5202 | p1735 5203 | I1779 5204 | sS'text' 5205 | p1736 5206 | I531 5207 | sS'charles' 5208 | p1737 5209 | I1583 5210 | sS'anyway' 5211 | p1738 5212 | I372 5213 | sS'bring' 5214 | p1739 5215 | I321 5216 | sS'fear' 5217 | p1740 5218 | I1031 5219 | sS'debate' 5220 | p1741 5221 | I1522 5222 | sS'staff' 5223 | p1742 5224 | I1909 5225 | sS'knowledge' 5226 | p1743 5227 | I658 5228 | sS'tire' 5229 | p1744 5230 | I1243 5231 | sS'cache' 5232 | p1745 5233 | I1355 5234 | sS'eat' 5235 | p1746 5236 | I863 5237 | sS'terminal' 5238 | p1747 5239 | I1606 5240 | sS'tape' 5241 | p1748 5242 | I596 5243 | sS'fuel' 5244 | p1749 5245 | I1492 5246 | sS'local' 5247 | p1750 5248 | I283 5249 | sS'hope' 5250 | p1751 5251 | I223 5252 | sS'handle' 5253 | p1752 5254 | I670 5255 | sS'means' 5256 | p1753 5257 | I1005 5258 | sS'beat' 5259 | p1754 5260 | I810 5261 | sS'convert' 5262 | p1755 5263 | I737 5264 | sS'massacre' 5265 | p1756 5266 | I1406 5267 | sS'overall' 5268 | p1757 5269 | I1835 5270 | sS'bear' 5271 | p1758 5272 | I550 5273 | sS'legitimate' 5274 | p1759 5275 | I1537 5276 | sS'handgun' 5277 | p1760 5278 | I1087 5279 | sS'brain' 5280 | p1761 5281 | I1122 5282 | sS'stuff' 5283 | p1762 5284 | I320 5285 | sS'contain' 5286 | p1763 5287 | I390 5288 | sS'view' 5289 | p1764 5290 | I268 5291 | sS'available' 5292 | p1765 5293 | I99 5294 | sS'requirement' 5295 | p1766 5296 | I1241 5297 | sS'frame' 5298 | p1767 5299 | I1359 5300 | sS'module' 5301 | p1768 5302 | I1898 5303 | sS'edition' 5304 | p1769 5305 | I1704 5306 | sS'computer' 5307 | p1770 5308 | I125 5309 | sS'greece' 5310 | p1771 5311 | I1892 5312 | sS'closer' 5313 | p1772 5314 | I1887 5315 | sS'wire' 5316 | p1773 5317 | I745 5318 | sS'stats' 5319 | p1774 5320 | I1669 5321 | sS'turks' 5322 | p1775 5323 | I832 5324 | sS'nuclear' 5325 | p1776 5326 | I1136 5327 | sS'tend' 5328 | p1777 5329 | I891 5330 | sS'favor' 5331 | p1778 5332 | I1621 5333 | sS'state' 5334 | p1779 5335 | I63 5336 | sS'correctly' 5337 | p1780 5338 | I1569 5339 | sS'routine' 5340 | p1781 5341 | I1242 5342 | sS'neither' 5343 | p1782 5344 | I918 5345 | sS'email' 5346 | p1783 5347 | I176 5348 | sS'ken' 5349 | p1784 5350 | I1325 5351 | sS'ability' 5352 | p1785 5353 | I894 5354 | sS'ignorance' 5355 | p1786 5356 | I1859 5357 | sS'job' 5358 | p1787 5359 | I325 5360 | sS'joe' 5361 | p1788 5362 | I1135 5363 | sS'key' 5364 | p1789 5365 | I69 5366 | sS'police' 5367 | p1790 5368 | I456 5369 | sS'configuration' 5370 | p1791 5371 | I1695 5372 | sS'problem' 5373 | p1792 5374 | I32 5375 | sS'jon' 5376 | p1793 5377 | I1100 5378 | sS'thank' 5379 | p1794 5380 | I739 5381 | sS'career' 5382 | p1795 5383 | I1734 5384 | sS'joke' 5385 | p1796 5386 | I1656 5387 | sS'equal' 5388 | p1797 5389 | I1404 5390 | sS'drug' 5391 | p1798 5392 | I427 5393 | sS'etc' 5394 | p1799 5395 | I817 5396 | sS'admit' 5397 | p1800 5398 | I945 5399 | sS'otherwise' 5400 | p1801 5401 | I674 5402 | sS'comment' 5403 | p1802 5404 | I340 5405 | sS'relevant' 5406 | p1803 5407 | I1674 5408 | sS'author' 5409 | p1804 5410 | I575 5411 | sS'conclude' 5412 | p1805 5413 | I1784 5414 | sS'wall' 5415 | p1806 5416 | I1252 5417 | sS'ca' 5418 | p1807 5419 | I537 5420 | sS'walk' 5421 | p1808 5422 | I958 5423 | sS'laugh' 5424 | p1809 5425 | I1548 5426 | sS'cs' 5427 | p1810 5428 | I1976 5429 | sS'respect' 5430 | p1811 5431 | I976 5432 | sS'addition' 5433 | p1812 5434 | I847 5435 | sS'decent' 5436 | p1813 5437 | I1661 5438 | sS'amiga' 5439 | p1814 5440 | I1966 5441 | sS'willing' 5442 | p1815 5443 | I736 5444 | sS'treat' 5445 | p1816 5446 | I930 5447 | sS'ah' 5448 | p1817 5449 | I1983 5450 | sS'proposal' 5451 | p1818 5452 | I1039 5453 | sS'waste' 5454 | p1819 5455 | I1093 5456 | sS'bbs' 5457 | p1820 5458 | I1532 5459 | sS'define' 5460 | p1821 5461 | I615 5462 | sS'mike' 5463 | p1822 5464 | I458 5465 | sS'homosexual' 5466 | p1823 5467 | I1126 5468 | sS'sufficient' 5469 | p1824 5470 | I1648 5471 | sS'essentially' 5472 | p1825 5473 | I1696 5474 | sS'controller' 5475 | p1826 5476 | I614 5477 | sS'graphics' 5478 | p1827 5479 | I1255 5480 | sS'atlanta' 5481 | p1828 5482 | I1923 5483 | sS'present' 5484 | p1829 5485 | I396 5486 | sS'lebanese' 5487 | p1830 5488 | I1588 5489 | sS'plain' 5490 | p1831 5491 | I1787 5492 | sS'value' 5493 | p1832 5494 | I261 5495 | sS'fault' 5496 | p1833 5497 | I1376 5498 | sS'supply' 5499 | p1834 5500 | I781 5501 | sS'annual' 5502 | p1835 5503 | I1885 5504 | sS'motif' 5505 | p1836 5506 | I725 5507 | sS'almost' 5508 | p1837 5509 | I375 5510 | sS'thus' 5511 | p1838 5512 | I494 5513 | sS'site' 5514 | p1839 5515 | I417 5516 | sS'surface' 5517 | p1840 5518 | I1085 5519 | sS'hardware' 5520 | p1841 5521 | I477 5522 | sS'vs' 5523 | p1842 5524 | I921 5525 | sS'terrorist' 5526 | p1843 5527 | I1475 5528 | sS'perhaps' 5529 | p1844 5530 | I280 5531 | sS'administration' 5532 | p1845 5533 | I654 5534 | sS'cross' 5535 | p1846 5536 | I1133 5537 | sS'member' 5538 | p1847 5539 | I337 5540 | sS'matthew' 5541 | p1848 5542 | I1077 5543 | sS'strange' 5544 | p1849 5545 | I1920 5546 | sS'speaker' 5547 | p1850 5548 | I1700 5549 | sS'shuttle' 5550 | p1851 5551 | I1237 5552 | sS'party' 5553 | p1852 5554 | I635 5555 | sS'difficult' 5556 | p1853 5557 | I851 5558 | sS'ball' 5559 | p1854 5560 | I1138 5561 | sS'slave' 5562 | p1855 5563 | I1505 5564 | sS'drink' 5565 | p1856 5566 | I1432 5567 | sS'disagree' 5568 | p1857 5569 | I1281 5570 | sS'effect' 5571 | p1858 5572 | I361 5573 | sS'student' 5574 | p1859 5575 | I800 5576 | sS'frequently' 5577 | p1860 5578 | I1633 5579 | sS'expand' 5580 | p1861 5581 | I1928 5582 | sS'center' 5583 | p1862 5584 | I244 5585 | sS'weapon' 5586 | p1863 5587 | I377 5588 | sS'variety' 5589 | p1864 5590 | I1790 5591 | sS'well' 5592 | p1865 5593 | I26 5594 | sS'thought' 5595 | p1866 5596 | I939 5597 | sS'command' 5598 | p1867 5599 | I548 5600 | sS'english' 5601 | p1868 5602 | I1433 5603 | sS'position' 5604 | p1869 5605 | I402 5606 | sS'audio' 5607 | p1870 5608 | I1452 5609 | sS'soviet' 5610 | p1871 5611 | I1231 5612 | sS'latest' 5613 | p1872 5614 | I1130 5615 | sS'rocket' 5616 | p1873 5617 | I1066 5618 | sS'less' 5619 | p1874 5620 | I207 5621 | sS'percent' 5622 | p1875 5623 | I1485 5624 | sS'accurate' 5625 | p1876 5626 | I1628 5627 | sS'obtain' 5628 | p1877 5629 | I798 5630 | sS'voltage' 5631 | p1878 5632 | I1851 5633 | sS'paul' 5634 | p1879 5635 | I441 5636 | sS'brad' 5637 | p1880 5638 | I1629 5639 | sS'clipper' 5640 | p1881 5641 | I521 5642 | sS'point' 5643 | p1882 5644 | I44 5645 | sS'smith' 5646 | p1883 5647 | I792 5648 | sS'realize' 5649 | p1884 5650 | I698 5651 | sS'battery' 5652 | p1885 5653 | I1270 5654 | sS'add' 5655 | p1886 5656 | I264 5657 | sS'citizen' 5658 | p1887 5659 | I630 5660 | sS'warrant' 5661 | p1888 5662 | I1430 5663 | sS'match' 5664 | p1889 5665 | I1457 5666 | sS'gmt' 5667 | p1890 5668 | I1535 5669 | sS'branch' 5670 | p1891 5671 | I1838 5672 | sS'government' 5673 | p1892 5674 | I66 5675 | sS'execute' 5676 | p1893 5677 | I1841 5678 | sS'five' 5679 | p1894 5680 | I953 5681 | sS'whatever' 5682 | p1895 5683 | I557 5684 | sS'press' 5685 | p1896 5686 | I418 5687 | sS'immediately' 5688 | p1897 5689 | I1405 5690 | sS'loss' 5691 | p1898 5692 | I1108 5693 | sS'motorola' 5694 | p1899 5695 | I1768 5696 | sS'necessary' 5697 | p1900 5698 | I652 5699 | sS'like' 5700 | p1901 5701 | I11 5702 | sS'success' 5703 | p1902 5704 | I1561 5705 | sS'mhz' 5706 | p1903 5707 | I1266 5708 | sS'lose' 5709 | p1904 5710 | I323 5711 | sS'become' 5712 | p1905 5713 | I204 5714 | sS'page' 5715 | p1906 5716 | I543 5717 | sS'regardless' 5718 | p1907 5719 | I1757 5720 | sS'amendment' 5721 | p1908 5722 | I916 5723 | sS'sequence' 5724 | p1909 5725 | I1439 5726 | sS'church' 5727 | p1910 5728 | I328 5729 | sS'library' 5730 | p1911 5731 | I640 5732 | sS'francisco' 5733 | p1912 5734 | I1697 5735 | sS'export' 5736 | p1913 5737 | I1314 5738 | sS'proper' 5739 | p1914 5740 | I1181 5741 | sS'home' 5742 | p1915 5743 | I226 5744 | sS'peter' 5745 | p1916 5746 | I786 5747 | sS'lead' 5748 | p1917 5749 | I285 5750 | sS'avoid' 5751 | p1918 5752 | I645 5753 | sS'leader' 5754 | p1919 5755 | I971 5756 | sS'noise' 5757 | p1920 5758 | I1664 5759 | sS'schedule' 5760 | p1921 5761 | I1528 5762 | sS'journal' 5763 | p1922 5764 | I1587 5765 | sS'expansion' 5766 | p1923 5767 | I1642 5768 | sS'pressure' 5769 | p1924 5770 | I1172 5771 | sS'host' 5772 | p1925 5773 | I1102 5774 | sS'although' 5775 | p1926 5776 | I373 5777 | sS'panel' 5778 | p1927 5779 | I1384 5780 | sS'actual' 5781 | p1928 5782 | I1209 5783 | sS'extension' 5784 | p1929 5785 | I1748 5786 | sS'universe' 5787 | p1930 5788 | I992 5789 | sS'shipping' 5790 | p1931 5791 | I1217 5792 | sS'angeles' 5793 | p1932 5794 | I1268 5795 | sS'software' 5796 | p1933 5797 | I149 5798 | sS'previously' 5799 | p1934 5800 | I1957 5801 | sS'washington' 5802 | p1935 5803 | I662 5804 | sS'playoff' 5805 | p1936 5806 | I1061 5807 | sS'promise' 5808 | p1937 5809 | I1343 5810 | sS'utility' 5811 | p1938 5812 | I1233 5813 | sS'van' 5814 | p1939 5815 | I1025 5816 | sS'additional' 5817 | p1940 5818 | I1247 5819 | sS'waco' 5820 | p1941 5821 | I1780 5822 | sS'transfer' 5823 | p1942 5824 | I809 5825 | sS'biggest' 5826 | p1943 5827 | I1968 5828 | sS'function' 5829 | p1944 5830 | I416 5831 | sS'buy' 5832 | p1945 5833 | I137 5834 | sS'north' 5835 | p1946 5836 | I925 5837 | sS'hr' 5838 | p1947 5839 | I1980 5840 | sS'bus' 5841 | p1948 5842 | I526 5843 | sS'hp' 5844 | p1949 5845 | I1014 5846 | sS'escrow' 5847 | p1950 5848 | I935 5849 | sS'volume' 5850 | p1951 5851 | I994 5852 | sS'neutral' 5853 | p1952 5854 | I1466 5855 | sS'gain' 5856 | p1953 5857 | I1134 5858 | sS'highest' 5859 | p1954 5860 | I1970 5861 | sS'buf' 5862 | p1955 5863 | I1409 5864 | sS'bug' 5865 | p1956 5866 | I1285 5867 | sS'hd' 5868 | p1957 5869 | I1347 5870 | sS'count' 5871 | p1958 5872 | I751 5873 | sS'converter' 5874 | p1959 5875 | I1764 5876 | sS'hitter' 5877 | p1960 5878 | I1971 5879 | sS'whether' 5880 | p1961 5881 | I242 5882 | sS'dangerous' 5883 | p1962 5884 | I1140 5885 | sS'official' 5886 | p1963 5887 | I642 5888 | sS'record' 5889 | p1964 5890 | I490 5891 | sS'convince' 5892 | p1965 5893 | I1296 5894 | sS'limit' 5895 | p1966 5896 | I571 5897 | sS'demonstrate' 5898 | p1967 5899 | I1498 5900 | sS'distribution' 5901 | p1968 5902 | I1091 5903 | sS'piece' 5904 | p1969 5905 | I1044 5906 | sS'display' 5907 | p1970 5908 | I266 5909 | sS'pin' 5910 | p1971 5911 | I963 5912 | sS'int' 5913 | p1972 5914 | I1191 5915 | sS'recognize' 5916 | p1973 5917 | I934 5918 | sS'contribute' 5919 | p1974 5920 | I1797 5921 | sS'et' 5922 | p1975 5923 | I1641 5924 | sS'education' 5925 | p1976 5926 | I1013 5927 | sS'pit' 5928 | p1977 5929 | I1803 5930 | sS'eric' 5931 | p1978 5932 | I1020 5933 | sS'contest' 5934 | p1979 5935 | I1553 5936 | sS'corporation' 5937 | p1980 5938 | I1336 5939 | sS'boot' 5940 | p1981 5941 | I1329 5942 | sS'detail' 5943 | p1982 5944 | I538 5945 | sS'virtual' 5946 | p1983 5947 | I1808 5948 | sS'book' 5949 | p1984 5950 | I146 5951 | sS'witness' 5952 | p1985 5953 | I1189 5954 | sS'sick' 5955 | p1986 5956 | I1446 5957 | sS'conclusion' 5958 | p1987 5959 | I870 5960 | sS'repeat' 5961 | p1988 5962 | I1173 5963 | sS'star' 5964 | p1989 5965 | I1115 5966 | sS'june' 5967 | p1990 5968 | I1598 5969 | sS'stay' 5970 | p1991 5971 | I708 5972 | sS'chance' 5973 | p1992 5974 | I549 5975 | sS'atheist' 5976 | p1993 5977 | I572 5978 | sS'rule' 5979 | p1994 5980 | I284 5981 | sS'original' 5982 | p1995 5983 | I355 5984 | sS'atheism' 5985 | p1996 5986 | I1264 5987 | s. -------------------------------------------------------------------------------- /data/Readme.md: -------------------------------------------------------------------------------- 1 | This is an example dataset. Its basically a version of the `20Newsgroup` dataset 2 | that was used in the paper. 3 | -------------------------------------------------------------------------------- /models/Readme.md: -------------------------------------------------------------------------------- 1 | Tensorflow implementation of the topic models mentioned in the paper. 2 | 3 | #### Our implementation of VAEs follows the scikit style. You should check out [Jan Hendrik Metzen](https://jmetzen.github.io/2015-11-27/vae.html) blog entry about this. A very clean and straighforward way to implement VAEs in scikit-style. 4 | -------------------------------------------------------------------------------- /models/__init__.py: -------------------------------------------------------------------------------- 1 | from prodlda import VAE 2 | from nvlda import VAE 3 | -------------------------------------------------------------------------------- /models/nvlda.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | import itertools,time 4 | import sys, os 5 | from collections import OrderedDict 6 | from copy import deepcopy 7 | from time import time 8 | import matplotlib.pyplot as plt 9 | import pickle 10 | def xavier_init(fan_in, fan_out, constant=1): 11 | low = -constant*np.sqrt(6.0/(fan_in + fan_out)) 12 | high = constant*np.sqrt(6.0/(fan_in + fan_out)) 13 | return tf.random_uniform((fan_in, fan_out), 14 | minval=low, maxval=high, 15 | dtype=tf.float32) 16 | 17 | def log_dir_init(fan_in, fan_out,topics=50): 18 | return tf.log((1.0/topics)*tf.ones([fan_in, fan_out])) 19 | 20 | tf.reset_default_graph() 21 | class VAE(object): 22 | """ 23 | See "Auto-Encoding Variational Bayes" by Kingma and Welling for more details. 24 | """ 25 | def __init__(self, network_architecture, transfer_fct=tf.nn.softplus, 26 | learning_rate=0.002, batch_size=200): 27 | self.network_architecture = network_architecture 28 | self.transfer_fct = transfer_fct 29 | self.learning_rate = learning_rate 30 | self.batch_size = batch_size 31 | '''----------------Inputs----------------''' 32 | self.x = tf.placeholder(tf.float32, [None, network_architecture["n_input"]]) 33 | self.keep_prob = tf.placeholder(tf.float32) 34 | 35 | '''-------Constructing Laplace Approximation to Dirichlet Prior--------------''' 36 | self.h_dim = float(network_architecture["n_z"]) 37 | self.a = 1*np.ones((1 , self.h_dim)).astype(np.float32) 38 | self.mu2 = tf.constant((np.log(self.a).T-np.mean(np.log(self.a),1)).T) 39 | self.var2 = tf.constant( ( ( (1.0/self.a)*( 1 - (2.0/self.h_dim) ) ).T + 40 | ( 1.0/(self.h_dim*self.h_dim) )*np.sum(1.0/self.a,1) ).T ) 41 | 42 | # Create autoencoder network 43 | self._create_network() 44 | self._create_loss_optimizer() 45 | init = tf.initialize_all_variables() 46 | self.sess = tf.InteractiveSession() 47 | self.sess.run(init) 48 | 49 | def _create_network(self): 50 | self.network_weights = self._initialize_weights(**self.network_architecture) 51 | 52 | self.z_mean,self.z_log_sigma_sq = \ 53 | self._recognition_network(self.network_weights["weights_recog"], 54 | self.network_weights["biases_recog"]) 55 | 56 | n_z = self.network_architecture["n_z"] 57 | eps = tf.random_normal((1, n_z), 0, 1, 58 | dtype=tf.float32) 59 | self.z = tf.add(self.z_mean, 60 | tf.mul(tf.sqrt(tf.exp(self.z_log_sigma_sq)), eps)) 61 | self.sigma = tf.exp(self.z_log_sigma_sq) 62 | self.x_reconstr_mean = \ 63 | self._generator_network(self.z,self.network_weights["weights_gener"]) 64 | 65 | def _initialize_weights(self, n_hidden_recog_1, n_hidden_recog_2, 66 | n_hidden_gener_1, 67 | n_input, n_z): 68 | all_weights = dict() 69 | all_weights['weights_recog'] = { 70 | 'h1': tf.get_variable('h1',[n_input, n_hidden_recog_1]), 71 | 'h2': tf.get_variable('h2',[n_hidden_recog_1, n_hidden_recog_2]), 72 | 'out_mean': tf.get_variable('out_mean',[n_hidden_recog_2, n_z]), 73 | 'out_log_sigma': tf.get_variable('out_log_sigma',[n_hidden_recog_2, n_z])} 74 | all_weights['biases_recog'] = { 75 | 'b1': tf.Variable(tf.zeros([n_hidden_recog_1], dtype=tf.float32)), 76 | 'b2': tf.Variable(tf.zeros([n_hidden_recog_2], dtype=tf.float32)), 77 | 'out_mean': tf.Variable(tf.zeros([n_z], dtype=tf.float32)), 78 | 'out_log_sigma': tf.Variable(tf.zeros([n_z], dtype=tf.float32))} 79 | all_weights['weights_gener'] = { 80 | 'h2': tf.Variable(xavier_init(n_z, n_hidden_gener_1))} 81 | return all_weights 82 | 83 | def _recognition_network(self, weights, biases): 84 | layer_1 = self.transfer_fct(tf.add(tf.matmul(self.x, weights['h1']), 85 | biases['b1'])) 86 | layer_2 = self.transfer_fct(tf.add(tf.matmul(layer_1, weights['h2']), 87 | biases['b2'])) 88 | layer_do = tf.nn.dropout(layer_2, self.keep_prob) 89 | 90 | z_mean = tf.contrib.layers.batch_norm(tf.add(tf.matmul(layer_do, weights['out_mean']), 91 | biases['out_mean'])) 92 | z_log_sigma_sq = \ 93 | tf.contrib.layers.batch_norm(tf.add(tf.matmul(layer_do, weights['out_log_sigma']), 94 | biases['out_log_sigma'])) 95 | 96 | return (z_mean, z_log_sigma_sq) 97 | 98 | def _generator_network(self,z, weights): 99 | self.layer_do_0 = tf.nn.dropout(tf.nn.softmax(z), self.keep_prob) 100 | x_reconstr_mean = tf.add(tf.matmul(self.layer_do_0, tf.nn.softmax(tf.contrib.layers.batch_norm(weights['h2']))),0.0) 101 | return x_reconstr_mean 102 | 103 | def _create_loss_optimizer(self): 104 | self.x_reconstr_mean+=1e-10 105 | reconstr_loss = \ 106 | -tf.reduce_sum(self.x * tf.log(self.x_reconstr_mean),1)#/tf.reduce_sum(self.x,1) 107 | 108 | latent_loss = 0.5*( tf.reduce_sum(tf.div(self.sigma,self.var2),1)+\ 109 | tf.reduce_sum( tf.mul(tf.div((self.mu2 - self.z_mean),self.var2), 110 | (self.mu2 - self.z_mean)),1) - self.h_dim +\ 111 | tf.reduce_sum(tf.log(self.var2),1) - tf.reduce_sum(self.z_log_sigma_sq ,1) ) 112 | 113 | self.cost = tf.reduce_mean(reconstr_loss) + tf.reduce_mean(latent_loss) # average over batch 114 | 115 | 116 | self.optimizer = \ 117 | tf.train.AdamOptimizer(learning_rate=self.learning_rate,beta1=0.99).minimize(self.cost) 118 | 119 | def partial_fit(self, X): 120 | opt, cost,emb = self.sess.run((self.optimizer, self.cost,self.network_weights['weights_gener']['h2']),feed_dict={self.x: X,self.keep_prob: .75}) 121 | return cost,emb 122 | 123 | def test(self, X): 124 | cost = self.sess.run((self.cost), 125 | feed_dict={self.x: np.expand_dims(X, axis=0),self.keep_prob: 1.0}) 126 | return cost 127 | def topic_prop(self, X): 128 | """heta_ is the topic proportion vector. Apply softmax transformation to it before use. 129 | """ 130 | theta_ = self.sess.run((self.z),feed_dict={self.x: np.expand_dims(X, axis=0),self.keep_prob: 1.0}) 131 | return theta_ 132 | -------------------------------------------------------------------------------- /models/prodlda.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | import itertools,time 4 | import sys, os 5 | from collections import OrderedDict 6 | from copy import deepcopy 7 | from time import time 8 | import matplotlib.pyplot as plt 9 | import pickle 10 | def xavier_init(fan_in, fan_out, constant=1): 11 | low = -constant*np.sqrt(6.0/(fan_in + fan_out)) 12 | high = constant*np.sqrt(6.0/(fan_in + fan_out)) 13 | return tf.random_uniform((fan_in, fan_out), 14 | minval=low, maxval=high, 15 | dtype=tf.float32) 16 | def log_dir_init(fan_in, fan_out,topics=50): 17 | return tf.log((1.0/topics)*tf.ones([fan_in, fan_out])) 18 | 19 | tf.reset_default_graph() 20 | class VAE(object): 21 | """ 22 | See "Auto-Encoding Variational Bayes" by Kingma and Welling for more details. 23 | """ 24 | 25 | 26 | def __init__(self, network_architecture, transfer_fct=tf.nn.softplus, 27 | learning_rate=0.001, batch_size=100): 28 | self.network_architecture = network_architecture 29 | self.transfer_fct = transfer_fct 30 | self.learning_rate = learning_rate 31 | self.batch_size = batch_size 32 | print 'Learning Rate:', self.learning_rate 33 | 34 | # tf Graph input 35 | self.x = tf.placeholder(tf.float32, [None, network_architecture["n_input"]]) 36 | self.keep_prob = tf.placeholder(tf.float32) 37 | 38 | self.h_dim = float(network_architecture["n_z"]) 39 | self.a = 1*np.ones((1 , self.h_dim)).astype(np.float32) 40 | self.mu2 = tf.constant((np.log(self.a).T-np.mean(np.log(self.a),1)).T) 41 | self.var2 = tf.constant( ( ( (1.0/self.a)*( 1 - (2.0/self.h_dim) ) ).T + 42 | ( 1.0/(self.h_dim*self.h_dim) )*np.sum(1.0/self.a,1) ).T ) 43 | 44 | self._create_network() 45 | self._create_loss_optimizer() 46 | 47 | init = tf.initialize_all_variables() 48 | 49 | self.sess = tf.InteractiveSession() 50 | self.sess.run(init) 51 | 52 | def _create_network(self): 53 | self.network_weights = self._initialize_weights(**self.network_architecture) 54 | self.z_mean,self.z_log_sigma_sq = \ 55 | self._recognition_network(self.network_weights["weights_recog"], 56 | self.network_weights["biases_recog"]) 57 | 58 | n_z = self.network_architecture["n_z"] 59 | eps = tf.random_normal((self.batch_size, n_z), 0, 1, 60 | dtype=tf.float32) 61 | self.z = tf.add(self.z_mean, 62 | tf.mul(tf.sqrt(tf.exp(self.z_log_sigma_sq)), eps)) 63 | self.sigma = tf.exp(self.z_log_sigma_sq) 64 | 65 | self.x_reconstr_mean = \ 66 | self._generator_network(self.z,self.network_weights["weights_gener"]) 67 | 68 | print self.x_reconstr_mean 69 | 70 | def _initialize_weights(self, n_hidden_recog_1, n_hidden_recog_2, 71 | n_hidden_gener_1, 72 | n_input, n_z): 73 | all_weights = dict() 74 | all_weights['weights_recog'] = { 75 | 'h1': tf.get_variable('h1',[n_input, n_hidden_recog_1]), 76 | 'h2': tf.get_variable('h2',[n_hidden_recog_1, n_hidden_recog_2]), 77 | 'out_mean': tf.get_variable('out_mean',[n_hidden_recog_2, n_z]), 78 | 'out_log_sigma': tf.get_variable('out_log_sigma',[n_hidden_recog_2, n_z])} 79 | all_weights['biases_recog'] = { 80 | 'b1': tf.Variable(tf.zeros([n_hidden_recog_1], dtype=tf.float32)), 81 | 'b2': tf.Variable(tf.zeros([n_hidden_recog_2], dtype=tf.float32)), 82 | 'out_mean': tf.Variable(tf.zeros([n_z], dtype=tf.float32)), 83 | 'out_log_sigma': tf.Variable(tf.zeros([n_z], dtype=tf.float32))} 84 | all_weights['weights_gener'] = { 85 | 'h2': tf.Variable(xavier_init(n_z, n_hidden_gener_1))} 86 | 87 | return all_weights 88 | 89 | def _recognition_network(self, weights, biases): 90 | # Generate probabilistic encoder (recognition network) 91 | layer_1 = self.transfer_fct(tf.add(tf.matmul(self.x, weights['h1']), 92 | biases['b1'])) 93 | layer_2 = self.transfer_fct(tf.add(tf.matmul(layer_1, weights['h2']), 94 | biases['b2'])) 95 | layer_do = tf.nn.dropout(layer_2, self.keep_prob) 96 | 97 | z_mean = tf.contrib.layers.batch_norm(tf.add(tf.matmul(layer_do, weights['out_mean']), 98 | biases['out_mean'])) 99 | z_log_sigma_sq = \ 100 | tf.contrib.layers.batch_norm(tf.add(tf.matmul(layer_do, weights['out_log_sigma']), 101 | biases['out_log_sigma'])) 102 | 103 | 104 | return (z_mean, z_log_sigma_sq) 105 | 106 | def _generator_network(self,z, weights): 107 | self.layer_do_0 = tf.nn.dropout(tf.nn.softmax(z), self.keep_prob) 108 | x_reconstr_mean = tf.nn.softmax(tf.contrib.layers.batch_norm(tf.add( 109 | tf.matmul(self.layer_do_0, weights['h2']),0.0))) 110 | return x_reconstr_mean 111 | 112 | 113 | def _create_loss_optimizer(self): 114 | 115 | self.x_reconstr_mean+=1e-10 116 | 117 | reconstr_loss = \ 118 | -tf.reduce_sum(self.x * tf.log(self.x_reconstr_mean),1)#/tf.reduce_sum(self.x,1) 119 | 120 | latent_loss = 0.5*( tf.reduce_sum(tf.div(self.sigma,self.var2),1)+\ 121 | tf.reduce_sum( tf.mul(tf.div((self.mu2 - self.z_mean),self.var2), 122 | (self.mu2 - self.z_mean)),1) - self.h_dim +\ 123 | tf.reduce_sum(tf.log(self.var2),1) - tf.reduce_sum(self.z_log_sigma_sq ,1) ) 124 | 125 | self.cost = tf.reduce_mean(reconstr_loss) + tf.reduce_mean(latent_loss) # average over batch 126 | 127 | 128 | self.optimizer = \ 129 | tf.train.AdamOptimizer(learning_rate=self.learning_rate,beta1=0.99).minimize(self.cost) 130 | 131 | def partial_fit(self, X): 132 | 133 | opt, cost,emb = self.sess.run((self.optimizer, self.cost,self.network_weights['weights_gener']['h2']),feed_dict={self.x: X,self.keep_prob: .4}) 134 | return cost,emb 135 | 136 | def test(self, X): 137 | """Test the model and return the lowerbound on the log-likelihood. 138 | """ 139 | cost = self.sess.run((self.cost),feed_dict={self.x: np.expand_dims(X, axis=0),self.keep_prob: 1.0}) 140 | return cost 141 | def topic_prop(self, X): 142 | """heta_ is the topic proportion vector. Apply softmax transformation to it before use. 143 | """ 144 | theta_ = self.sess.run((self.z),feed_dict={self.x: np.expand_dims(X, axis=0),self.keep_prob: 1.0}) 145 | return theta_ 146 | -------------------------------------------------------------------------------- /run.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import numpy as np 4 | import tensorflow as tf 5 | import itertools,time 6 | import sys, os 7 | from collections import OrderedDict 8 | from copy import deepcopy 9 | from time import time 10 | import matplotlib.pyplot as plt 11 | import pickle 12 | import sys, getopt 13 | from models import prodlda, nvlda 14 | '''-----------Data--------------''' 15 | def onehot(data, min_length): 16 | return np.bincount(data, minlength=min_length) 17 | 18 | dataset_tr = 'data/20news_clean/train.txt.npy' 19 | data_tr = np.load(dataset_tr) 20 | dataset_te = 'data/20news_clean/test.txt.npy' 21 | data_te = np.load(dataset_te) 22 | vocab = 'data/20news_clean/vocab.pkl' 23 | vocab = pickle.load(open(vocab,'r')) 24 | vocab_size=len(vocab) 25 | #--------------convert to one-hot representation------------------ 26 | print 'Converting data to one-hot representation' 27 | data_tr = np.array([onehot(doc.astype('int'),vocab_size) for doc in data_tr if np.sum(doc)!=0]) 28 | data_te = np.array([onehot(doc.astype('int'),vocab_size) for doc in data_te if np.sum(doc)!=0]) 29 | #--------------print the data dimentions-------------------------- 30 | print 'Data Loaded' 31 | print 'Dim Training Data',data_tr.shape 32 | print 'Dim Test Data',data_te.shape 33 | '''-----------------------------''' 34 | 35 | '''--------------Global Params---------------''' 36 | n_samples_tr = data_tr.shape[0] 37 | n_samples_te = data_te.shape[0] 38 | docs_tr = data_tr 39 | docs_te = data_te 40 | batch_size=200 41 | learning_rate=0.002 42 | network_architecture = \ 43 | dict(n_hidden_recog_1=100, # 1st layer encoder neurons 44 | n_hidden_recog_2=100, # 2nd layer encoder neurons 45 | n_hidden_gener_1=data_tr.shape[1], # 1st layer decoder neurons 46 | n_input=data_tr.shape[1], # MNIST data input (img shape: 28*28) 47 | n_z=50) # dimensionality of latent space 48 | 49 | '''-----------------------------''' 50 | 51 | '''--------------Netowrk Architecture and settings---------------''' 52 | 53 | def make_network(layer1=100,layer2=100,num_topics=50,bs=200,eta=0.002): 54 | tf.reset_default_graph() 55 | network_architecture = \ 56 | dict(n_hidden_recog_1=layer1, # 1st layer encoder neurons 57 | n_hidden_recog_2=layer2, # 2nd layer encoder neurons 58 | n_hidden_gener_1=data_tr.shape[1], # 1st layer decoder neurons 59 | n_input=data_tr.shape[1], # MNIST data input (img shape: 28*28) 60 | n_z=num_topics) # dimensionality of latent space 61 | batch_size=bs 62 | learning_rate=eta 63 | return network_architecture,batch_size,learning_rate 64 | 65 | 66 | 67 | '''--------------Methods--------------''' 68 | def create_minibatch(data): 69 | rng = np.random.RandomState(10) 70 | 71 | while True: 72 | # Return random data samples of a size 'minibatch_size' at each iteration 73 | ixs = rng.randint(data.shape[0], size=batch_size) 74 | yield data[ixs] 75 | 76 | 77 | def train(network_architecture, minibatches, type='prodlda',learning_rate=0.001, 78 | batch_size=200, training_epochs=100, display_step=5): 79 | tf.reset_default_graph() 80 | vae='' 81 | if type=='prodlda': 82 | vae = prodlda.VAE(network_architecture, 83 | learning_rate=learning_rate, 84 | batch_size=batch_size) 85 | elif type=='nvlda': 86 | vae = nvlda.VAE(network_architecture, 87 | learning_rate=learning_rate, 88 | batch_size=batch_size) 89 | emb=0 90 | # Training cycle 91 | for epoch in range(training_epochs): 92 | avg_cost = 0. 93 | total_batch = int(n_samples_tr / batch_size) 94 | # Loop over all batches 95 | for i in range(total_batch): 96 | batch_xs = minibatches.next() 97 | # Fit training using batch data 98 | cost,emb = vae.partial_fit(batch_xs) 99 | # Compute average loss 100 | avg_cost += cost / n_samples_tr * batch_size 101 | 102 | if np.isnan(avg_cost): 103 | print epoch,i,np.sum(batch_xs,1).astype(np.int),batch_xs.shape 104 | print 'Encountered NaN, stopping training. Please check the learning_rate settings and the momentum.' 105 | # return vae,emb 106 | sys.exit() 107 | 108 | # Display logs per epoch step 109 | if epoch % display_step == 0: 110 | print "Epoch:", '%04d' % (epoch+1), \ 111 | "cost=", "{:.9f}".format(avg_cost) 112 | return vae,emb 113 | 114 | def print_top_words(beta, feature_names, n_top_words=10): 115 | print '---------------Printing the Topics------------------' 116 | for i in range(len(beta)): 117 | print(" ".join([feature_names[j] 118 | for j in beta[i].argsort()[:-n_top_words - 1:-1]])) 119 | print '---------------End of Topics------------------' 120 | 121 | def calcPerp(model): 122 | cost=[] 123 | for doc in docs_te: 124 | doc = doc.astype('float32') 125 | n_d = np.sum(doc) 126 | c=model.test(doc) 127 | cost.append(c/n_d) 128 | print 'The approximated perplexity is: ',(np.exp(np.mean(np.array(cost)))) 129 | 130 | def main(argv): 131 | m = '' 132 | f = '' 133 | s = '' 134 | t = '' 135 | b = '' 136 | r = '' 137 | e = '' 138 | try: 139 | opts, args = getopt.getopt(argv,"hpnm:f:s:t:b:r:,e:",["default=","model=","layer1=","layer2=","num_topics=","batch_size=","learning_rate=","training_epochs"]) 140 | except getopt.GetoptError: 141 | print 'CUDA_VISIBLE_DEVICES=0 python run.py -m -f <#units> -s <#units> -t <#topics> -b -r ' 142 | sys.exit(2) 143 | for opt, arg in opts: 144 | if opt == '-h': 145 | print 'CUDA_VISIBLE_DEVICES=0 python run.py -m -f <#units> -s <#units> -t <#topics> -b -r -e ' 146 | sys.exit() 147 | elif opt == '-p': 148 | print 'Running with the Default settings for prodLDA...' 149 | print 'CUDA_VISIBLE_DEVICES=0 python run.py -m prodlda -f 100 -s 100 -t 50 -b 200 -r 0.002 -e 100' 150 | m='prodlda' 151 | f=100 152 | s=100 153 | t=50 154 | b=200 155 | r=0.002 156 | e=100 157 | elif opt == '-n': 158 | print 'Running with the Default settings for NVLDA...' 159 | print 'CUDA_VISIBLE_DEVICES=0 python run.py -m nvlda -f 100 -s 100 -t 50 -b 200 -r 0.005 -e 300' 160 | m='nvlda' 161 | f=100 162 | s=100 163 | t=50 164 | b=200 165 | r=0.01 166 | e=300 167 | elif opt == "-m": 168 | m=arg 169 | elif opt == "-f": 170 | f=int(arg) 171 | elif opt == "-s": 172 | s=int(arg) 173 | elif opt == "-t": 174 | t=int(arg) 175 | elif opt == "-b": 176 | b=int(arg) 177 | elif opt == "-r": 178 | r=float(arg) 179 | elif opt == "-e": 180 | e=int(arg) 181 | 182 | minibatches = create_minibatch(docs_tr.astype('float32')) 183 | network_architecture,batch_size,learning_rate=make_network(f,s,t,b,r) 184 | print network_architecture 185 | print opts 186 | vae,emb = train(network_architecture, minibatches,m, training_epochs=e,batch_size=batch_size,learning_rate=learning_rate) 187 | print_top_words(emb, zip(*sorted(vocab.items(), key=lambda x: x[1]))[0]) 188 | calcPerp(vae) 189 | 190 | if __name__ == "__main__": 191 | main(sys.argv[1:]) 192 | --------------------------------------------------------------------------------