├── .gitignore ├── Fine-tuning BERT.ipynb ├── Introduction to BERT.ipynb ├── README.md ├── data └── disaster_sample.csv ├── images ├── bert_qa.png ├── bert_seq_class.png ├── hf_models.png ├── oreilly.png ├── python_embedding.png └── transformers.png └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | logs/ 2 | .ipynb_checkpoints/ 3 | .DS_Store 4 | Results/ -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![oreilly-logo](images/oreilly.png) 2 | 3 | # BERT Transformer Architecture for NLP 4 | 5 | This repository contains code for the [O'Reilly Live Online Training for BERT](https://www.oreilly.com/live-events/bert-transformer-architecture-for-nlp/0636920054919/0636920068720/) 6 | 7 | This training will focus on how BERT is used for a wide variety of NLP tasks including text classification, question answering, and machine translation. The training will begin with an introduction to necessary concepts including language models and transformers and then build on those concepts to introduce the BERT architecture. We will then move into how BERT is architected specifically as a language model that can be used for multiple natural language processing tasks with hands-on examples of fine-tuning pre-trained BERT models. 8 | 9 | BERT is one of the most relevant NLP architectures today and it is closely related to other important NLP deep learning models like GPT-3. Both of these models are derived from the newly invented transformer architecture and represent an inflection point in how machines process language and context. 10 | 11 | Notebook 1: [Introduction to BERT](Introduction to BERT.ipynb) 12 | 13 | Notebook 2: [Pre-training and fine-tuning BERT](Fine-tuning BERT.ipynb) 14 | 15 | ## Instructor 16 | 17 | *Sinan Ozdemir* is currently the Director of Data Science at Directly, managing the AI and machine learning models that power the company’s intelligent customer support platform. Sinan is a former lecturer of Data Science at Johns Hopkins University and the author of multiple textbooks on data science and machine learning. Additionally, he is the founder of the recently acquired Kylie.ai, an enterprise-grade conversational AI platform with RPA capabilities. He holds a Master’s Degree in Pure Mathematics from Johns Hopkins University and is based in San Francisco, CA. 18 | -------------------------------------------------------------------------------- /data/disaster_sample.csv: -------------------------------------------------------------------------------- 1 | index,id,keyword,location,text,target,label 2 | 7138,10224,volcano,,@MrMikeEaton @Muazimus_Prime hill hill mountain volcano of hell mountain hill hil.,1,1 3 | 2151,3086,deaths,Blackpool,"Cancers equate for around 25% of all deaths in #Blackpool. 4 | 5 | Kowing the signs could save your life: http://t.co/5lNIdvoBff 6 | #BeClearOnCancer",1,1 7 | 4395,6247,hijacking,World,The Murderous Story Of America‰Ûªs First Hijacking: Earnest Pletch‰Ûªs cold-blooded killing of‰Û_ http://t.co/B9JAxx0vCf,1,1 8 | 2508,3602,desolation,"Birmingham, UK",The date for the release of EP03 DESOLATION is set. Stay tuned for more info while we finalise the schedule. #alt #electro #rock #comingsoon,1,1 9 | 1378,1987,bush%20fires,London/Bristol/Guildford,On holiday to relax sunbathe and drink ... Putting out bush fires? Not so much ?? #spain https://t.co/dRno7OKM21,0,0 10 | 6825,9775,trapped,????s ?? ????Ìø????Ì¡a,(?EudryLantiqua?) Hollywood Movie About Trapped Miners Released in Chile: 'The 33' Holly... http://t.co/us1DMdXZVb (?EudryLantiqua?),1,1 11 | 3877,5514,flattened,Some other mansion,"Flattened all cartoony-like. 12 | 'Whoa there Papa!' https://t.co/4zmcqRMOIs",0,0 13 | 3465,4957,exploded,,#news #science London warship exploded in 1665 because sailors were recycling artillery cartridges... http://t.co/r4WGXrA59M #life #tech,1,1 14 | 6245,8921,snowstorm,"Brooklyn, NY",'Cooler than Freddie Jackson sippin' a milkshake in a snowstorm',0,0 15 | 5664,8083,rescue,Wanderlust,Mary coming to Troy rescue. ?????? https://t.co/rosVXQeLQj,0,0 16 | 3938,5599,flood,New York,Spot Flood Combo 53inch 300W Curved Cree LED Work Light Bar 4X4 Offroad Fog Lamp - Full re‰Û_ http://t.co/fDSaoOiskJ http://t.co/2uVmq4vAfQ,0,0 17 | 3059,4388,earthquake,in the Word of God,@DArchambau THX for your great encouragement and for RT of NEW VIDEO http://t.co/cybKsXHF7d The Coming Apocalyptic US Earthquake & Tsunami,1,1 18 | 5985,8546,screams,,When you on the phone and @Worstoverdose screams 'jaileens caked up on the phone' so everyone looks at you ????????????,0,0 19 | 7186,10296,weapon,,Cat Of Nine Irons XII: This nightmarishly brutal weapon is used in ritualistic country club de http://t.co/xpFmR368uF http://t.co/nmAUMYdKe1,1,1 20 | 6493,9283,sunk,"San Jose, CA",Its like I never left. I just sunk to the background,0,0 21 | 816,1185,blizzard,columbus ohio,@StevenOnTwatter @PussyxDestroyer just order a blizzard pay then put your nuts in it say they have you ball flavored. Boom free ice cream,0,0 22 | 4127,5868,hailstorm,facebook.com/tradcatknights,"Canada: Hailstorm flash flooding slam Calgary knocks out power to 20k customers 23 | http://t.co/SkY9EokgGB http://t.co/5IyZsDA6xB",1,1 24 | 4623,6571,injury,,incident with injury:I-495 inner loop Exit 31 - MD 97/Georgia Ave Silver Spring,1,1 25 | 895,1296,bloody,Leicester,Bloody insomnia again! Grrrr!! #Insomnia,1,1 26 | 7450,10662,wounds,The American Wasteland (MV),@FEVWarrior -in the Vault that could take a look at those wounds of yours if you'd like to go to one of these places first.' Zarry has had-,0,0 27 | 3043,4366,earthquake,a box,"@AGeekyFangirl14 's things she looks in a significant other: 28 | 1. Beautiful eyes. 29 | 2. Humor. 30 | 3. Farts that creates an earthquake. 31 | 32 | ????????",0,0 33 | 422,612,arsonist,America,If you don't have anything nice to say you can come sit with me.,0,0 34 | 194,273,ambulance,Loveland Colorado,@Kiwi_Karyn Check out what's in my parking lot!! He said that until last year it was an ambulance in St Johns. http://t.co/hPvOdUD7iP,0,0 35 | 542,788,avalanche,Freeport il ,The possible new jerseys for the Avalanche next year. ???? http://t.co/nruzhR5XQu,0,0 36 | 6950,9972,tsunami,,"Crptotech tsunami and banks. 37 | http://t.co/KHzTeVeDja #Banking #tech #bitcoing #blockchain",1,1 38 | 4860,6919,mass%20murderer,,Julian Knight - @SCVSupremeCourt dismisses mass murderer's attempt to increase prisoner pay. Challenged quantum of 5% increase 2013.,1,1 39 | 3297,4725,evacuate,,Disregard my snap story there is an angry white girl riot happening as we speak. #evacuate,1,1 40 | 4977,7097,military," Quantico Marine Base, VA.",@FurTrix then find cougars who look like her even better if they're in military uniform!,0,0 41 | 411,594,arson,,Arson suspect linked to 30 fires caught in Northern California http://t.co/u1fuWrGK5U,1,1 42 | 3817,5425,first%20responders,,Carmike Cinemas on Antioch Shooting: 'We Are Grateful' for Staff and First Responders Safety Is 'Highest Priority' http://t.co/BehfHspPud,1,1 43 | 1121,1617,bombed,"Screwston, TX",'Redskins WR Roberts Belly-Bombed ' via @TeamStream http://t.co/GbcvVEvDTY,1,1 44 | 4233,6013,hazardous,"Oregon, USA",Is it possible to sneak into a hospital so I can stab myself with a hazardous needle and inject some crazy disease into my veins until I die,0,0 45 | 6397,9143,suicide%20bomber,,Suicide Bomber Kills 13 At Saudi Mosque http://t.co/oZ1DS3Xu0D #Saudi #Tripoli #Libya,1,1 46 | 4622,6570,injury,,@Sport_EN Just being linked to Arsenal causes injury.,1,1 47 | 4149,5898,harm,Kansas City,@dinallyhot Love what you picked! We're playing WORTH IT by FIFTH HARM/KID INK because of you! Listen & Vote: http://t.co/0wrATkA2jL,0,0 48 | 2927,4206,drowned,"United Kingdom,Fraserburgh","@Stephen_Georg Hey Stephen Remember that time you drowned all the yellows 49 | 50 | Read: http://t.co/0sa6Xx1oQ7",0,0 51 | 4895,6970,massacre,,"Headed to the massacre 52 | Bodies arriving everyday 53 | What were those shells you heard 54 | Picking the bones up along the way",1,1 55 | 2267,3249,deluged,balvanera,'Afterwards I had to be alone for an hour to savour and prolong the almost physical intensity of the feelings that deluged me'. Y sigue:,0,0 56 | 6027,8613,seismic,,SEISMIC AUDIO SA-15T SA15T Padded Black Speaker COVERS (2) - Qty of 1 = 1 Pair! http://t.co/2jbIbeib9G http://t.co/p5KtaqW5QG,0,0 57 | 7354,10527,wildfire,"Eddyville, Oregon 97343",Oregon's biggest wildfire slows growth http://t.co/P0GoS5URXG via @katunews,1,1 58 | 1935,2783,curfew,Uganda,@DavisKawalya I know @Mauryn143 will be saying her final goodbyes to grandpa as seen on news RiP Me? always open to ideas but may ve curfew,0,0 59 | 932,1349,blown%20up,Cobblestone,"'If a truckload of soldiers will be blown up nobody panics but when one little lion dies everyone loses their mind' 60 | http://t.co/wjNTaOkdHf",1,1 61 | 3963,5632,flooding,"Honolulu, Hawaii",If Flooding occurs in your home - How to stop mold growth via @ProudGreenHome http://t.co/KAVAovJz2V,1,1 62 | 2805,4033,disaster,,@cncpts @SOLELINKS what a disaster - can't say I'm surprised,0,0 63 | 1683,2428,collide,The Forever Girl,The Witches of the Glass Castle. Supernatural YA where sibling rivalry magic and love collide #wogc #kindle http://t.co/IzakNpJeQW,0,0 64 | 5754,8214,riot,,"@abran_caballero Discovered by @NickCannon 65 | Listen/Buy @realmandyrain #RIOT on @iTunesMusic @iTunes https://t.co/dehMym5lpk ‰Û_ #BlowMandyUp",0,0 66 | 3907,5557,flattened,,100 1' MIX NEW FLAT DOUBLE SIDED LINERLESS BOTTLE CAPS YOU CHOOSE MIX FLATTENED - Full re‰Û_ http://t.co/w00kjPrfdR http://t.co/mIXl1pFRJe,0,0 67 | 4123,5861,hailstorm,,Summer heat drives bobcats to Calgary backyards ~ 3 http://t.co/kEstuUYc4t http://t.co/PzFBb1P1mj,1,1 68 | 4842,6896,mass%20murder,"Huntsville, AL",Okay not sure the word 'mass murder' applies during this war but it was horrendous none the less. https://t.co/Sb3rjQqzIX,1,1 69 | 6892,9880,traumatised,,@wrongdejavu I'm traumatised,0,0 70 | 4122,5858,hailstorm,,IG: http://t.co/2WBiVKzJIP 'It's hailing again! #abstorm #yyc #hail #hailstorm #haildamage #yycweather #calgary #captureyyc #alberta #sto‰Û_,1,1 71 | 5093,7265,nuclear%20disaster,,3 former executives to be prosecuted in Fukushima nuclear disaster http://t.co/Gvj7slbELP,1,1 72 | 5327,7605,pandemonium,"Mumbai, India",@minhazmerchant Govt should pass the bills in the Pandemonium. UPA used to do it why cant NDA?,0,0 73 | 2348,3379,demolition,,Israel continues its demolition of Palestinian homes #gop #potus #irandeal #isis https://t.co/NMgp7iMEIi,1,1 74 | 1123,1619,bombed,My old New England home,I liked a @YouTube video http://t.co/FX7uZZXtE4 Benedict Cumberbatch Gets Video Bombed,0,0 75 | 2645,3797,destruction,denver colorado,it sure made an impact on me http://t.co/GS50DdG1JY,0,0 76 | 3220,4620,emergency%20services,"Vancouver, British Columbia",Removing tsunami debris from the West Coast: Karen Robinson Enviromental and Emergency services manager of the‰Û_ http://t.co/1MeEo3WJcO,1,1 77 | 4117,5850,hailstorm,Washington State,We The Free Hailstorm Maxi http://t.co/ERWs6IELdG,1,1 78 | 1913,2753,curfew,,@TheComedyQuote @50ShadezOfGrey the thirst has no curfew ???????????? @P45Perez,0,0 79 | 3372,4830,evacuation,,Run out evacuation hospital indexing remedial of angioplasty dissertation at power elite hospitals dismayed:‰Û_ http://t.co/VGvJGr8zoO,1,1 80 | 6382,9120,suicide%20bomb,lagos. Unilag,16yr old PKK suicide bomber who detonated bomb in Turkey Army trench released http://t.co/mGZslZz1wF,1,1 81 | 1966,2830,cyclone,"Sioux Falls, SD",Excited for Cyclone football https://t.co/Xqv6gzZMmN,0,0 82 | 150,215,airplane%20accident,"New Mexico, USA",@mickinyman @TheAtlantic That or they might be killed in an airplane accident in the night a car wreck! Politics at it's best.,0,0 83 | 1938,2786,curfew,"Elkhart, IN",Had to cancel my cats doctor appointment because she decided to go out and play and not come home by curfew ...,0,0 84 | 4735,6734,lava,Venezuela,I LAVA YOU.,0,0 85 | 4111,5842,hailstorm,,Hail:The user summons a hailstorm lasting five turns. It damages all Pokemon except the Ice type.,0,0 86 | 498,721,attacked,Peshawar,IK Only Troll His Pol Rivals Never Literally Abused Them Or Attacked Their Families. While All Of Them Literally Abuse IK. Loosers,0,0 87 | 4246,6033,hazardous,United States,DLH issues Hazardous Weather Outlook (HWO) http://t.co/a0Ad8z5Vsr #WX,1,1 88 | 4578,6511,injuries,"Sutton, London UK",@fkhanage look what Shad Forsythe has done in 1 year we won't have as many injuries as before we will inevitably have injuries like others,0,0 89 | 3532,5049,eyewitness,Pennsylvania,A true #TBT Eyewitness News WBRE WYOU http://t.co/JHVigsX5Jg,0,0 90 | 3070,4405,electrocute,,when you got an extension cord that extends from your bed to your bath tub ?? lets pray I don't electrocute myself,0,0 91 | 1829,2629,crashed,,Pakistan says army helicopter has crashed in country's restive northwest killing at least 8 http://t.co/QV1RMZI3J1,1,1 92 | 6818,9765,trapped,10 Steps Ahead. Cloud 9,Bomb head? Explosive decisions dat produced more dead children than dead bodies trapped tween buildings on that day in September there,1,1 93 | 4355,6187,hijacker,San Francisco,Governor allows parole for California school bus hijacker who kidnapped 26 children in 1976. http://t.co/hdAhLgrprl http://t.co/Z1s3T77P3L,1,1 94 | 2961,4253,drowning,Numa casa de old yellow bricks,LONDON IS DROWNING AND IIII LIVE BY THE RIVEEEEEER,1,1 95 | 4285,6088,hellfire,,Hellfire is surrounded by desires so be careful and don‰Ûªt let your desires control you! #Afterlife,0,0 96 | 6572,9405,survivors,"Dallas, TX ",Haunting memories drawn by survivors http://t.co/pRAro2OWia,1,1 97 | 5047,7195,natural%20disaster,home ,she's a natural disaster she's the last of the American girls ??,0,0 98 | 249,354,annihilation,Spain,:StarMade: :Stardate 3: :Planetary Annihilation:: http://t.co/I2hHvIUmTm via @YouTube,1,1 99 | 5382,7680,panic,,@panic awesome thanks.,0,0 100 | 2898,4163,drown,,@cameronhigdon34 I can't drown my demons they know how to swim.,0,0 101 | 2771,3983,devastation,,70 Years After Atomic Bombs Japan Still Struggles With War Past: The anniversary of the devastation wrought b... http://t.co/Targ56iGBZ,1,1 102 | 6043,8638,seismic,,A subcontractor working for French seismic survey group CGG has been kidnapped in Cairo and is held by Islamic State the company said on W‰Û_,1,1 103 | 3868,5499,flames,,If you have an opinion and you don't put it on thh internet you will furst into flames.,0,0 104 | 7452,10665,wounds,"Cleveland, OH",If time heals all wounds how come the belly button stays the same?,0,0 105 | 5434,7753,police,,Police unions retard justice & drain gov $ but cops vote 4 Rs so Rs go after teachers unions instead! @DCCC @VJ44 @Lawrence @JBouie @mmfa,0,0 106 | 6757,9681,tornado,Toronto,Environment Canada confirms 2nd tornado touched down last weekend åÈ http://t.co/x8zqbwNfO1,1,1 107 | 5874,8391,ruin,"MÌ©rida, YucatÌÁn",babe I'm gonna ruin you if you let me stay,0,0 108 | 6036,8627,seismic,"Hatteras, North Carolina",Agency seeks comments on seismic permits http://t.co/9Vd6x4WDOY,0,0 109 | 7589,10843,,,Omg earthquake,1,1 110 | 5386,7686,panic,Milwaukee WI,Someone asked me about a monkey fist about 2 feet long with a panic snap like the one pictured to be used as a... http://t.co/Yi9BBbx3FE,0,0 111 | 4604,6547,injury,"Los Angeles, California",California Law‰ÛÓNegligence and Fireworks Explosion Incidents http://t.co/d5w2zynP7b,1,1 112 | 5743,8201,riot,"Cardiff, UK",@PipRhys I predict a riot.,1,1 113 | 1015,1474,body%20bagging,401 livin',Aubrey really out here body-bagging Meek.,1,1 114 | 3425,4898,explode,they/them,"tagged by @attackonstiles 115 | 116 | millions 117 | a-punk 118 | hang em high 119 | alpha dog 120 | yeah boy and doll face 121 | little white lies 122 | explode http://t.co/lAtsSUo4wS",0,0 123 | 102,146,aftershock,Instagram - @heyimginog ,@afterShock_DeLo scuf ps live and the game... cya,0,0 124 | 5373,7669,panic,"Philadelphia, PA",Despite the crippling anxiety and overwhelming panic attacks I'd say I'm fairly well-adjusted.,0,0 125 | 3850,5479,flames,hell,@gilderoy i wish i was good enough to add flames to my nails im on fire,1,1 126 | 5945,8491,screamed,,i dont even remember slsp happening i just remember being like wtf and then the lights turned off and everyone screamed for the encore,0,0 127 | 5920,8452,screamed,,I heard the steven universe theme song from upstairs and screamed his name at the part of the song and scared my cousin,0,0 128 | 5310,7583,outbreak,Finland,Families to sue over Legionnaires: More than 40 families affected by the fatal outbreak of Legionnaires' disea... http://t.co/ZA4AXFJSVB,1,1 129 | 5955,8506,screaming,rio de janeiro | brazil,SCREAMING @MariahCarey @ArianaGrande http://t.co/xxZD1nmb1i,0,0 130 | 5454,7780,police,"Mesa, AZ",@ArizonaDOT Price Rd North bound closed from University to Rio Salado.. Lots of police.. What's crackin?,1,1 131 | 365,523,army,,One Direction Is my pick for http://t.co/q2eBlOKeVE Fan Army #Directioners http://t.co/eNCmhz6y34 x1386,0,0 132 | 5598,7988,razed,,The Latest: More Homes Razed by Northern California Wildfire - ABC News http://t.co/2872J5d4HB,1,1 133 | 5152,7348,obliterate,"Cavite, Philippines",The People's Republic Of China ( PROC ): Abandon the West Philippine Sea and all the ... https://t.co/pD14GsrfSC via @ChangePilipinas,0,0 134 | 128,184,aftershock,304,'Remembering that you are going to die is the best way I know to avoid the trap of thinking you have something to lose.' ‰ÛÒ Steve Jobs,0,0 135 | 4635,6588,inundated,"Zeerust, South Africa",Most of us ddnt get this English RT @ReIgN_CoCo: The World Is Inundated With Ostentatious People Stay Woke!,0,0 136 | 2519,3619,desolation,Arizona,Did Josephus get it wrong about Antiochus Epiphanes and the Abomination of Desolation? Read more: http://t.co/FWj9CcYw6k,0,0 137 | 2684,3850,detonation,,Ignition Knock (Detonation) Sensor-Senso Standard KS161 http://t.co/WadPP69LwJ http://t.co/yjTh2nABv5,0,0 138 | 4250,6038,heat%20wave,USA,Heat Advisory is In Effect From 1 PM Through 7 PM Thursday. A Building Heat Wave and Increasing Humidity... #lawx http://t.co/u0SYkowVWV,1,1 139 | 6610,9466,terrorism,,DHS Refuses to Call Chattanooga ‰Û÷Islamic Terrorism‰Ûª out of respect for MUSLIMS ... http://t.co/u8RGB51d22 via @po_st http://t.co/2tnu95VGFE,1,1 140 | 7386,10570,windstorm,Houston,New roof and hardy up..Windstorm inspection tomorrow http://t.co/kKeH8qCgc3,0,0 141 | 3921,5577,flood,New York,Spot Flood Combo 53inch 300W Curved Cree LED Work Light Bar 4X4 Offroad Fog Lamp - Full re‰Û_ http://t.co/mTmoIa0Oo0 http://t.co/Nn4ZtCmSRU,0,0 142 | 629,906,bioterrorism,,70 won 70...& some think possibility of my full transformation is impossible. I don't quite like medical mysteries. BIOTERRORISM sucks.,0,0 143 | 6986,10018,twister,,Twister was fun https://t.co/qCT6fb8wOn,0,0 144 | 7173,10278,war%20zone,,@RobertONeill31 Getting hit by a foul ball while sitting there is hardly a freak accident. It's a war zone.,0,0 145 | 2493,3582,desolate,Macclesfield,@binellithresa TY for the follow Go To http://t.co/UAN05TNkSW BRUTALLY ABUSED+DESOLATE&LOST + HER LOVELY MUM DIES..Is it Murder?,1,1 146 | 7467,10684,wounds,Charlotte,ME says many of these wounds could be fatal some rather quickly others slower and a couple not lethal at all. #KerrickTrial,0,0 147 | 1657,2394,collapsed,#ForeverWithBAP 8 ,and he almost collapsed bc he said his wish came true moderately FUCK,0,0 148 | 4737,6736,lava,Indonesia,@YoungHeroesID 4. Lava Blast Power Red #PantherAttack,0,0 149 | 5690,8120,rescued,"Winston-Salem, NC",'You can only be rescued from where you actually are and not from where you pretend you are.' Giorgio Hiatt,0,0 150 | 807,1173,blight,Kama | 18 | France ,@anellatulip there is a theory that makes way too much sense that says that the dwarves may be the actual origin of the blight,0,0 151 | 3149,4523,emergency,We are global!,SF Asian Women's Shelter crisis line (415) 751-0880. Emergency shelter/support services 4 non-English speaking Asian women & children.,1,1 152 | 3194,4585,emergency%20plan,Los Angeles,Senators calling for emergency housing: Boxer Feinstein back plan to move #homeless vets to VA campus http://t.co/Gm80X3vutf,1,1 153 | 3583,5119,fatal,Nakhon Si Thammarat,Investigators shift focus to cause of fatal Waimate fire http://t.co/c9dVDsSoFn,1,1 154 | 2380,3421,derail, Road to the Billionaires Club,@TheJenMorillo GM! I pray any attack of the enemy 2 derail ur destiny is blocked by the Lord & that He floods ur life w/heavenly Blessings,0,0 155 | 4456,6339,hostages,"Cumming, GA",C-130 specially modified to land in a stadium and rescue hostages in Iran in 1980 http://t.co/jkD7CTi2iW http://t.co/LAjN2n5e2d,1,1 156 | 2669,3831,detonate,,@WoundedPigeon http://t.co/s9soAeVcVo Detonate by @ApolloBrown ft. M.O.P.,0,0 157 | 3393,4857,evacuation,The Empire/First Order,@ariabrisard @leiaorganasolo Good. Play along with her. You may begin your operation with the death star. The evacuation is nearly complete.,0,0 158 | 3134,4503,emergency,,Emergency Flow http://t.co/lH9mrYpDrJ mp3 http://t.co/PqhuthSS3i rar http://t.co/0iW6dRf5X9,1,1 159 | 1799,2585,crash,Galatians 2:20 ,Please keep Josh the Salyers/Blair/Hall families & Jenna's friends in your prayers. She was taken far too soon. RIP http://t.co/bDN2FDPdAz,0,0 160 | 6339,9061,structural%20failure,,Investigators say a fatal Virgin Galactic spaceship crash last year was caused by structural failure after the co‰Û_,1,1 161 | 5162,7361,obliterate,Cymru araul,@McCaineNL Think how spectacular it will look when the Stonewall riots obliterate the white house.,1,1 162 | 4788,6811,loud%20bang,,Who the fuck plays music extremely loud at 9am on a Tuesday morning?? Bruv do they want me to come bang them??,0,0 163 | 2871,4127,drought,Meereen ,Pizza drought is over I just couldn't anymore...,0,0 164 | 3020,4335,dust%20storm,"Atlanta, GA",DUST IN THE WIND: @82ndABNDIV paratroopers move to a loading zone during a dust storm in support of Operation Fury: http://t.co/uGesKLCn8M,1,1 165 | 880,1275,blood,,Shed innocent blood of their sons and daughters and the land was polluted Psalms 106:38 Help stop the sin of abortion.,0,0 166 | 5138,7328,nuclear%20reactor,"Den Helder, Rijkswerf",US Navy Sidelines 3 Newest #Subs http://t.co/9WQixGMHfh,0,0 167 | 4216,5989,hazardous,,DLH issues Hazardous Weather Outlook (HWO) http://t.co/WOzuBXRi2p,1,1 168 | 1310,1892,burning,"Caracas, Venezuela.",We will be burning up like neon lights??????,1,1 169 | 3939,5600,flood,United States,Flood Advisory issued August 05 at 4:28PM EDT by NWS http://t.co/D02sbM0ojs #WxKY,1,1 170 | 6017,8595,seismic,garowe puntland somalia,Oil and Gas Exploration http://t.co/PckF0nl2yN,1,1 171 | 6105,8717,sinking,"Michigan, USA",‰Û¢‰Û¢If your lost & alone or your sinking like a stone carry onå¡å¡,0,0 172 | 937,1354,blown%20up,Nowhere Islands/Smash Manor,@TheBoyOfMasks 'Thanks again for letting me stay here since the manor was blown up..... Anyways how are you doing buddy?',0,0 173 | 314,457,armageddon,Canada,@ENews Ben Affleck......I know there's a wife/kids and other girls but I can't help it. I've loved him since Armageddon #eonlinechat,0,0 174 | 6275,8966,storm,,What tropical storm? #guillermo by hawaiianpaddlesports http://t.co/LgPgAjgomY http://t.co/FKd1mBTB68,1,1 175 | 1889,2715,crushed,#HAMont,Edwin wow. Crushed.,0,0 176 | 5550,7916,rainstorm,,#DnB #NewRelease EDGE Jimmy - Summer Rainstorm (Lapaka Sounds) http://t.co/4L8h2FKlNO via http://t.co/ZITQKDFXJY,0,0 177 | 6415,9173,suicide%20bomber,,http://t.co/9k1tqsAarM Suicide bomber kills 15 in Saudi security site mosque - Reuters http://t.co/Ev3nX9scx3,1,1 178 | 5722,8165,rescuers,,Last Second Ebay Bid RT? http://t.co/oEKUcq4ZL0 Shaolin Rescuers (dvd 2010) Shen Chan Nan Chiang Five Venoms Kung Fu ?Please Favori,0,0 179 | 5071,7229,natural%20disaster,"Suburban Detroit, Michigan",We have overrun a Natural Disaster Survival server!,1,1 180 | 4274,6072,heat%20wave,"Oklahoma City, OK",Longest Streak of Triple-Digit Heat Since 2013 Forecast in Dallas: An unrelenting and dangerous heat wave will... http://t.co/s4Srgrmqcz,1,1 181 | 1035,1501,body%20bags,Swaning Around,US ‰Û÷Institute Of Peace‰Ûª Chairman Wants Russian Body Bags http://t.co/owbUjez3q4,0,0 182 | 4316,6128,hellfire,"Denver, Colorado",@MechaMacGyver Wow bet you got blamed for that too huh?,0,0 183 | 2077,2983,dead,"Sochi, KDA, RU",@hlportal Hello! I'm looking for mod Cold Ice. I saw it on your site but link to download dead. Maybe you have it and share with me? Thanks.,0,0 184 | 3899,5545,flattened,Le Memenet,@KainYusanagi @Grummz @PixelCanuck and flattened raynor. Raynor was a balding imperfect biker marine not a emo generic western hero.,0,0 185 | 3974,5650,flooding,Global-NoLocation,#flood #disaster Burst Water Pipe Floods Apartments at NYCHA Senior Center - NY1: NY1Burst Water Pipe Floods A... http://t.co/w7SIIdujOH,1,1 186 | 7050,10101,typhoon,The Peach State,I think a Typhoon just passed through here lol,1,1 187 | 3383,4845,evacuation,"sydney, australia",my school just put the evacuation alarms on accidently with 2 different trial exams happening are you kidding me,0,0 188 | 7562,10811,wrecked,,you wrecked my whole world,0,0 189 | 2088,3001,dead,Milton Keynes ,Can't believe Ross is dead???????? @emmerdale @MikeParrActor #Emmerdale #summerfate,0,0 190 | 6011,8584,screams,5-Feb,When you go to a concert and someone screams in your ear... Does it look like I wanna loose my hearing anytime soon???,1,1 191 | 2295,3292,demolish,,I have completed the quest 'Demolish 5 Murlo...' in the #Android game The Tribez. http://t.co/pBclFsXRld #androidgames #gameinsight,0,0 192 | 4745,6749,lightning,,Thunder and lightning possible in the Pinpoint Foothill Forecast. http://t.co/CtIjdPXABk,1,1 193 | 16,24,,,I love fruits,0,0 194 | 6596,9448,terrorism,,@RobPulseNews @huyovoeTripolye Phillips should be charged for assisting terrorism. LDNR-terrorists' organizations. http://t.co/XwnJYsV9V9,1,1 195 | 514,740,attacked,"SÌ£o Paulo SP, Brasil",Christian Attacked by Muslims at the Temple Mount after Waving Israeli Flag via Pamela Geller - ... http://t.co/e4wK8Uri8A,1,1 196 | 2551,3659,destroy,,"ng2x5 mhtw4fnet 197 | 198 | Watch Michael Jordan absolutely destroy this meme-baiting camper - FOXSportscom",0,0 199 | 6184,8829,sirens,,@iK4LEN Sirens was cancelled.,0,0 200 | 7359,10536,windstorm,,Windstorm lastingness perquisite - acquiesce in a twister retreat: ZiUW http://t.co/iRt4kkgsJx,1,1 201 | 927,1343,blown%20up,Georgia,Man why hasn't @machinegunkelly blown up? He's still underground.,0,0 202 | 7238,10366,weapons,??????,#Battlefield 1942 forgotten hope secret weapons,1,1 203 | 88,130,accident,"Manchester, NH",Accident left lane blocked in #Manchester on Rt 293 NB before Eddy Rd stop and go traffic back to NH-3A delay of 4 mins #traffic,1,1 204 | 4374,6215,hijacker,,Governor allows parole for California school bus hijacker: Local... http://t.co/tAM6aoskoJ http://t.co/eL24mnFcHw,1,1 205 | 5922,8454,screamed,ljp/4,THE GIRLS NEXT TO ME SCREAMED WHAT THE FUCK IS A CHONCE I'm CRYIBG,0,0 206 | 6834,9788,trapped,å_å_Los Mina City‰ã¢,Hollywood Movie About Trapped Miners Released in Chile: 'The 33' Hollywood movie about trapped miners starring... http://t.co/x8moYeVjsJ,0,0 207 | 5842,8348,ruin,London / Birmingham,Im so anxious though because so many ppl will me watching me meet them and that makes me uncomfortable BUT I CANT LET THAT RUIN THE MOMENT,0,0 208 | 2039,2927,danger,Spinning through time.,@riverroaming 'And not too much danger please.',0,0 209 | 2942,4230,drowned,,Thank you Uplifting spirit. When Im drowned you've been an anchor,0,0 210 | 523,755,avalanche,Ireland,A little piece I wrote for the Avalanche Designs blog! I'd appreciate it greatly if you checked it out :-) https://t.co/rfvjh58eF2,0,0 211 | 6432,9203,suicide%20bombing,,< 25 Dead In Kuwait Mosque Suicide Bombing Claimed By ISIS Offshoot on http://t.co/eTITgPSrUN,1,1 212 | 925,1340,blown%20up,USA,@troylercraft YEAH ITS NOT WORTH IT BC HE ALREADY HAS SO MANY SPAMMERS & HIS TWITTER IS PROBABLY BLOWN UP EVERY SECOND,0,0 213 | 587,848,bioterror,iTunes,#world FedEx no longer to transport bioterror germs in wake of anthrax lab mishaps http://t.co/wvExJjRG6E,1,1 214 | 3661,5211,fatality,"Lowell, MA",@EBROINTHEAM jay....big L....pun....biggie...wrap over...zero question....fatality...flawless victory http://t.co/Y33QcKq7qD,0,0 215 | 6503,9297,survive,United States,AHH forgot my headphones how am I supposed to survive a day without music AYHHHHHDJJFJRJJRDJJEKS,0,0 216 | 5120,7301,nuclear%20reactor,"USA, North Dakota",Salem 2 nuclear reactor shut down over electrical circuit failure on pump: The Salem 2 nuclear reactor had bee... http://t.co/5hkGXzJLmX,1,1 217 | 7541,10782,wreckage,"New Delhi,India",Wreckage 'Conclusively Confirmed' as From MH370: Malaysia PM: Investigators and the families of those who were... http://t.co/1YIxFG1Hdy,1,1 218 | 4779,6800,loud%20bang,Kenya,tkyonly1fmk: Breaking news! Unconfirmed! I just heard a loud bang nearby. in what appears to be a blast of wind from my neighbour's ass.,0,0 219 | 2342,3369,demolition,,#download & #watch Demolition Frog (2002) http://t.co/81nEizeknm #movie,1,1 220 | 7526,10764,wreckage,"Dublin City, Ireland",Wreckage 'Conclusively Confirmed' as From MH370: Malaysia PM: Investigators and the families of those who were... http://t.co/VAZpG0ftmU,1,1 221 | 2614,3752,destruction,,Crackdown 3 Destruction Restricted to Multiplayer: Crackdown 3 impressed earlier this week with a demonstratio... http://t.co/LMWKjsYCgj,0,0 222 | 2638,3786,destruction,Jersey,That's the ultimate road to destruction,0,0 223 | 963,1393,body%20bag,,new summer long thin body bag hip A word skirt Blue http://t.co/lvKoEMsq8m http://t.co/CjiRhHh4vj,0,0 224 | 383,552,arson,"Charlotte, NC",Add Familia to the arson squad.,0,0 225 | 4522,6425,hurricane,,Stream HYPE HURRICANE,0,0 226 | 2049,2941,danger,Instagram: trillrebel_,"Guns are for protection.. 227 | That shit really shouldn't be used unless your life in danger",0,0 228 | 7532,10770,wreckage,iTunes,#science Now that a piece of wreckage from flight MH370 has been confirmed on RÌ©union Island is it possible t... http://t.co/Z2vDGIyOwf,1,1 229 | 5345,7630,pandemonium,illinois. united state ,Pandemonium In Aba As Woman Delivers Baby Without Face (Photos) @... http://t.co/JbxBi93CLu,0,0 230 | 1342,1940,burning%20buildings,,Hero's fight wars and save ppl from burning buildings etc I'm sorry but u gotta do more than pay 4 a sex change be4 I call u a hero,0,0 231 | -------------------------------------------------------------------------------- /images/bert_qa.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sinanuozdemir/oreilly-bert-nlp/a9ad980d801b373eb8faf15ed23e411af2d814b0/images/bert_qa.png -------------------------------------------------------------------------------- /images/bert_seq_class.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sinanuozdemir/oreilly-bert-nlp/a9ad980d801b373eb8faf15ed23e411af2d814b0/images/bert_seq_class.png -------------------------------------------------------------------------------- /images/hf_models.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sinanuozdemir/oreilly-bert-nlp/a9ad980d801b373eb8faf15ed23e411af2d814b0/images/hf_models.png -------------------------------------------------------------------------------- /images/oreilly.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sinanuozdemir/oreilly-bert-nlp/a9ad980d801b373eb8faf15ed23e411af2d814b0/images/oreilly.png -------------------------------------------------------------------------------- /images/python_embedding.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sinanuozdemir/oreilly-bert-nlp/a9ad980d801b373eb8faf15ed23e411af2d814b0/images/python_embedding.png -------------------------------------------------------------------------------- /images/transformers.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sinanuozdemir/oreilly-bert-nlp/a9ad980d801b373eb8faf15ed23e411af2d814b0/images/transformers.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | nlp==0.4.0 2 | transformers==4.16.2 3 | seaborn==0.11.2 4 | pandas==1.3.4 5 | bertviz==1.3.0 6 | datasets==1.16.1 7 | scikit-learn==1.0.1 8 | --------------------------------------------------------------------------------