├── MCQ_Question_Generator_BERT_Wordnet_Conceptnet.ipynb ├── README.md └── egypt.txt /MCQ_Question_Generator_BERT_Wordnet_Conceptnet.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "\n", 10 | "!pip install gensim\n", 11 | "!pip install git+https://github.com/boudinfl/pke.git\n", 12 | "!python -m spacy download en\n", 13 | "!pip install bert-extractive-summarizer --upgrade --force-reinstall\n", 14 | "!pip install spacy==2.1.3 --upgrade --force-reinstall\n", 15 | "!pip install -U nltk\n", 16 | "!pip install -U pywsd" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": null, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "import nltk\n", 26 | "nltk.download('stopwords')\n", 27 | "nltk.download('popular')" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "\n", 35 | "\n", 36 | "\n", 37 | "\n", 38 | "## BERT Extractive Summarizer\n", 39 | "Summarize the text using BERT extractive summarizer. This is used to find important sentences and useful sentences from the complete text." 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 21, 45 | "metadata": {}, 46 | "outputs": [ 47 | { 48 | "name": "stdout", 49 | "output_type": "stream", 50 | "text": [ 51 | "The Nile River fed Egyptian civilization for hundreds of years. It begins near the equator in Africa and flows north to the Mediterranean Sea. A delta is an area near a river’s mouth where the water deposits fine soil called silt. This soil was fertile, which means it was good for growing crops. The red land was the barren desert beyond the fertile region. When the birds arrived, the annual flood waters would soon follow. Then they used a tool called a shaduf to spread the water across the fields. These innovative, or new, techniques gave them more farmland. They were the first to grind wheat into flour and to mix the flour with yeast and water to make dough rise into bread. Egyptians often painted walls white to reflect the blazing heat. Poorer Egyptians simply went to the roof to cool off after sunset. Even during the cool season, chipping minerals out of the rock was miserable work. One ancient painting even shows a man ready to hit a catfish with a wooden hammer. A boomerang is a curved stick that returns to the person who threw it.) The river’s current was slow, so boaters used paddles to go faster when they traveled north with the current. Going south, they raised a sail and let the winds that blew in that direction push them. The Nile provided so well for Egyptians that sometimes they had surpluses, or more goods than they needed. Ancient Egypt had no money, so people exchanged goods that they grew or made. This prosperity made life easier and provided greater opportunities for many Egyptians. For example, some ancient Egyptians learned to be scribes, people whose job was to write and keep records. Some skilled artisans erected stone or brick houses and temples. A few Egyptians traveled to the upper Nile to trade with other Africans. They brought back exotic woods, animal skins, and live beasts. Egyptians created a government that divided the empire into 42 provinces. Many officials worked to keep the provinces running smoothly. Priests followed formal rituals and took care of the temples. Before entering a temple, a priest bathed and put on special linen garments and white sandals. Together, the priests and the ruler held ceremonies to please the gods. By doing so, they hoped to maintain the social and political order. In Egypt, people became slaves if they owed a debt, committed a crime, or were captured in war. Unlike other ancient African cultures, in Egyptian society men and women had fairly equal rights. For example, they could both own and manage their own property. Children in Egypt played with toys such as dolls, animal figures, board games, and marbles. Almost all Egyptians married when they were in their early teens. As in many ancient societies, much of the knowledge of Egypt came about as priests studied the world to find ways to please the gods. Doctors believed that the heart controlled thought and the brain circulated blood, which is the opposite of what is known now. Early Egyptians created a hieroglyphic system with about 700 characters. Legend says a king named Narmer united Upper and Lower Egypt. Some historians think Narmer actually represents several kings who gradually joined the two lands. It combined the red Crown of Lower Egypt with the white Crown of Upper Egypt. When a king died, one of his children usually took his place as ruler. Historians divide ancient Egyptian dynasties into the Old Kingdom, the Middle Kingdom, and the New Kingdom. The Old Kingdom started about 2575 B.C., when the Egyptian empire was gaining strength. In such a case, a rival might drive him from power and start a new dynasty. The first rulers of Egypt were often buried in an underground tomb topped by mud brick. They replaced the mud brick with a small pyramid of brick or stone. It is called a step pyramid because its sides rise in a series of giant steps. He ordered the construction of the largest pyramid ever built. One reason is that the pyramids drew attention to the tombs inside them. Grave robbers broke into the tombs to steal the treasure buried with the pharaohs. Egyptians believed that if a tomb was robbed, the person buried there could not have a happy afterlife. This way, the pharaohs hoped to protect their bodies and treasures from robbers. This was to confuse grave robbers about which passage to take. Tombs were supposed to be the palaces of pharaohs in the afterlife. Mourners filled the tomb with objects ranging from food to furniture that the mummified pharaoh would need. Such activities included growing and preparing food, caring for animals, and building boats. Only a secret tomb built for a New Kingdom pharaoh was ever found with much of its treasure untouched. The dazzling riches found in this tomb show how much wealth the pharaohs spent preparing for the afterlife. This period of Egyptian history is called the Middle Kingdom.\n" 52 | ] 53 | } 54 | ], 55 | "source": [ 56 | "from summarizer import Summarizer\n", 57 | "\n", 58 | "f = open(\"egypt.txt\",\"r\")\n", 59 | "full_text = f.read()\n", 60 | "\n", 61 | "model = Summarizer()\n", 62 | "result = model(full_text, min_length=60, max_length = 500 , ratio = 0.4)\n", 63 | "\n", 64 | "summarized_text = ''.join(result)\n", 65 | "print (summarized_text)" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "## Keyword Extraction\n", 73 | "Get important keywords from the text and filter those keywords that are present in the summarized text." 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 22, 79 | "metadata": {}, 80 | "outputs": [ 81 | { 82 | "name": "stdout", 83 | "output_type": "stream", 84 | "text": [ 85 | "['egyptians', 'nile river', 'egypt', 'nile', 'euphrates', 'tigris', 'old kingdom', 'red land', 'crown', 'upper', 'lower egypt', 'narmer', 'longest river', 'africa', 'mediterranean sea', 'hyksos', 'new kingdom', 'black land', 'ethiopia', 'middle kingdom']\n", 86 | "['egyptians', 'nile river', 'egypt', 'nile', 'old kingdom', 'red land', 'crown', 'upper', 'lower egypt', 'narmer', 'africa', 'mediterranean sea', 'new kingdom', 'middle kingdom']\n" 87 | ] 88 | } 89 | ], 90 | "source": [ 91 | "\n", 92 | "\n", 93 | "import pprint\n", 94 | "import itertools\n", 95 | "import re\n", 96 | "import pke\n", 97 | "import string\n", 98 | "from nltk.corpus import stopwords\n", 99 | "\n", 100 | "def get_nouns_multipartite(text):\n", 101 | " out=[]\n", 102 | "\n", 103 | " extractor = pke.unsupervised.MultipartiteRank()\n", 104 | " extractor.load_document(input=text)\n", 105 | " # not contain punctuation marks or stopwords as candidates.\n", 106 | " pos = {'PROPN'}\n", 107 | " #pos = {'VERB', 'ADJ', 'NOUN'}\n", 108 | " stoplist = list(string.punctuation)\n", 109 | " stoplist += ['-lrb-', '-rrb-', '-lcb-', '-rcb-', '-lsb-', '-rsb-']\n", 110 | " stoplist += stopwords.words('english')\n", 111 | " extractor.candidate_selection(pos=pos, stoplist=stoplist)\n", 112 | " # 4. build the Multipartite graph and rank candidates using random walk,\n", 113 | " # alpha controls the weight adjustment mechanism, see TopicRank for\n", 114 | " # threshold/method parameters.\n", 115 | " extractor.candidate_weighting(alpha=1.1,\n", 116 | " threshold=0.75,\n", 117 | " method='average')\n", 118 | " keyphrases = extractor.get_n_best(n=20)\n", 119 | "\n", 120 | " for key in keyphrases:\n", 121 | " out.append(key[0])\n", 122 | "\n", 123 | " return out\n", 124 | "\n", 125 | "keywords = get_nouns_multipartite(full_text) \n", 126 | "print (keywords)\n", 127 | "filtered_keys=[]\n", 128 | "for keyword in keywords:\n", 129 | " if keyword.lower() in summarized_text.lower():\n", 130 | " filtered_keys.append(keyword)\n", 131 | " \n", 132 | "print (filtered_keys)" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "## Sentence Mapping\n", 140 | "For each keyword get the sentences from the summarized text containing that keyword. " 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 23, 146 | "metadata": {}, 147 | "outputs": [ 148 | { 149 | "name": "stdout", 150 | "output_type": "stream", 151 | "text": [ 152 | "{'egyptians': ['The Nile provided so well for Egyptians that sometimes they had surpluses, or more goods than they needed.', 'For example, some ancient Egyptians learned to be scribes, people whose job was to write and keep records.', 'Egyptians believed that if a tomb was robbed, the person buried there could not have a happy afterlife.', 'This prosperity made life easier and provided greater opportunities for many Egyptians.', 'Egyptians created a government that divided the empire into 42 provinces.', 'A few Egyptians traveled to the upper Nile to trade with other Africans.', 'Early Egyptians created a hieroglyphic system with about 700 characters.', 'Poorer Egyptians simply went to the roof to cool off after sunset.', 'Almost all Egyptians married when they were in their early teens.', 'Egyptians often painted walls white to reflect the blazing heat.'], 'nile river': ['The Nile River fed Egyptian civilization for hundreds of years.'], 'egypt': ['As in many ancient societies, much of the knowledge of Egypt came about as priests studied the world to find ways to please the gods.', 'In Egypt, people became slaves if they owed a debt, committed a crime, or were captured in war.', 'Children in Egypt played with toys such as dolls, animal figures, board games, and marbles.', 'The first rulers of Egypt were often buried in an underground tomb topped by mud brick.', 'Ancient Egypt had no money, so people exchanged goods that they grew or made.', 'It combined the red Crown of Lower Egypt with the white Crown of Upper Egypt.'], 'nile': ['The Nile provided so well for Egyptians that sometimes they had surpluses, or more goods than they needed.', 'A few Egyptians traveled to the upper Nile to trade with other Africans.'], 'old kingdom': ['Historians divide ancient Egyptian dynasties into the Old Kingdom, the Middle Kingdom, and the New Kingdom.', 'The Old Kingdom started about 2575 B.C., when the Egyptian empire was gaining strength.'], 'red land': ['The red land was the barren desert beyond the fertile region.'], 'crown': ['It combined the red Crown of Lower Egypt with the white Crown of Upper Egypt.', 'It combined the red Crown of Lower Egypt with the white Crown of Upper Egypt.'], 'upper': ['It combined the red Crown of Lower Egypt with the white Crown of Upper Egypt.', 'A few Egyptians traveled to the upper Nile to trade with other Africans.', 'Legend says a king named Narmer united Upper and Lower Egypt.'], 'lower egypt': ['It combined the red Crown of Lower Egypt with the white Crown of Upper Egypt.', 'Legend says a king named Narmer united Upper and Lower Egypt.'], 'narmer': ['Some historians think Narmer actually represents several kings who gradually joined the two lands.', 'Legend says a king named Narmer united Upper and Lower Egypt.'], 'africa': ['It begins near the equator in Africa and flows north to the Mediterranean Sea.'], 'mediterranean sea': ['It begins near the equator in Africa and flows north to the Mediterranean Sea.'], 'new kingdom': ['Historians divide ancient Egyptian dynasties into the Old Kingdom, the Middle Kingdom, and the New Kingdom.', 'Only a secret tomb built for a New Kingdom pharaoh was ever found with much of its treasure untouched.'], 'middle kingdom': ['Historians divide ancient Egyptian dynasties into the Old Kingdom, the Middle Kingdom, and the New Kingdom.', 'This period of Egyptian history is called the Middle Kingdom.']}\n" 153 | ] 154 | } 155 | ], 156 | "source": [ 157 | "from nltk.tokenize import sent_tokenize\n", 158 | "from flashtext import KeywordProcessor\n", 159 | "\n", 160 | "def tokenize_sentences(text):\n", 161 | " sentences = [sent_tokenize(text)]\n", 162 | " sentences = [y for x in sentences for y in x]\n", 163 | " # Remove any short sentences less than 20 letters.\n", 164 | " sentences = [sentence.strip() for sentence in sentences if len(sentence) > 20]\n", 165 | " return sentences\n", 166 | "\n", 167 | "def get_sentences_for_keyword(keywords, sentences):\n", 168 | " keyword_processor = KeywordProcessor()\n", 169 | " keyword_sentences = {}\n", 170 | " for word in keywords:\n", 171 | " keyword_sentences[word] = []\n", 172 | " keyword_processor.add_keyword(word)\n", 173 | " for sentence in sentences:\n", 174 | " keywords_found = keyword_processor.extract_keywords(sentence)\n", 175 | " for key in keywords_found:\n", 176 | " keyword_sentences[key].append(sentence)\n", 177 | "\n", 178 | " for key in keyword_sentences.keys():\n", 179 | " values = keyword_sentences[key]\n", 180 | " values = sorted(values, key=len, reverse=True)\n", 181 | " keyword_sentences[key] = values\n", 182 | " return keyword_sentences\n", 183 | "\n", 184 | "sentences = tokenize_sentences(summarized_text)\n", 185 | "keyword_sentence_mapping = get_sentences_for_keyword(filtered_keys, sentences)\n", 186 | " \n", 187 | "print (keyword_sentence_mapping)" 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "metadata": {}, 193 | "source": [ 194 | "## Generate MCQ\n", 195 | "Get distractors (wrong answer choices) from Wordnet/Conceptnet and generate MCQ Questions." 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": 27, 201 | "metadata": {}, 202 | "outputs": [ 203 | { 204 | "name": "stdout", 205 | "output_type": "stream", 206 | "text": [ 207 | "#############################################################################\n", 208 | "NOTE:::::::: Since the algorithm might have errors along the way, wrong answer choices generated might not be correct for some questions. \n", 209 | "#############################################################################\n", 210 | "\n", 211 | "\n", 212 | "1) The Nile provided so well for _______ that sometimes they had surpluses, or more goods than they needed.\n", 213 | "\t a ) Egyptians\n", 214 | "\t b ) Bantu\n", 215 | "\t c ) Algerian\n", 216 | "\t d ) Angolan\n", 217 | "\n", 218 | "More options: ['Basotho', 'Beninese', 'Berber', 'Black African', 'Burundian', 'Cameroonian', 'Carthaginian', 'Chadian', 'Chewa', 'Congolese', 'Djiboutian', 'Egyptian', 'Ethiopian', 'Eurafrican', 'Ewe', 'Fulani'] \n", 219 | "\n", 220 | "\n", 221 | "2) As in many ancient societies, much of the knowledge of _______ came about as priests studied the world to find ways to please the gods.\n", 222 | "\t a ) Egypt\n", 223 | "\t b ) Malawi\n", 224 | "\t c ) East Africa\n", 225 | "\t d ) Somalia\n", 226 | "\n", 227 | "More options: ['Togo', 'Zimbabwe', 'Gabon', 'Ghana', 'Lake Tanganyika', 'Ottoman Empire', 'Mozambique', 'Iran', 'Israel', 'Saudi Arabia', 'Lebanon', 'Turkey', 'Iraq', 'Levant', 'Syria', 'Jordan'] \n", 228 | "\n", 229 | "\n", 230 | "3) The _______ provided so well for Egyptians that sometimes they had surpluses, or more goods than they needed.\n", 231 | "\t a ) Port Sudan\n", 232 | "\t b ) Omdurman\n", 233 | "\t c ) Nile\n", 234 | "\t d ) Nyala\n", 235 | "\n", 236 | "More options: ['Khartoum', 'Nubian Desert', 'Darfur', 'Libyan Desert', 'Kordofan', 'Gulu', 'Buganda', 'Entebbe', 'Jinja', 'Lake Edward', 'entebbe', 'gulu', 'kayunga', 'Upper Egypt', 'Suez Canal', 'Aswan High Dam'] \n", 237 | "\n", 238 | "\n", 239 | "4) It combined the red _______ of Lower Egypt with the white _______ of Upper Egypt.\n", 240 | "\t a ) Head\n", 241 | "\t b ) Capital\n", 242 | "\t c ) Masthead\n", 243 | "\t d ) Crown\n", 244 | "\n", 245 | "More options: [] \n", 246 | "\n", 247 | "\n", 248 | "5) It combined the red Crown of Lower Egypt with the white Crown of _______ Egypt.\n", 249 | "\t a ) Upper Berth\n", 250 | "\t b ) Lower Berth\n", 251 | "\t c ) Upper\n", 252 | "\n", 253 | "More options: [] \n", 254 | "\n", 255 | "\n", 256 | "6) It combined the red Crown of _______ with the white Crown of Upper Egypt.\n", 257 | "\t a ) Upper Egypt\n", 258 | "\t b ) Suez Canal\n", 259 | "\t c ) Lower egypt\n", 260 | "\t d ) Libyan Desert\n", 261 | "\n", 262 | "More options: ['Aswan High Dam', 'Eastern Desert', 'Aswan', 'Lake Nasser', 'Suez', 'Thebes', 'Sinai'] \n", 263 | "\n", 264 | "\n", 265 | "7) It begins near the equator in _______ and flows north to the Mediterranean Sea.\n", 266 | "\t a ) Eurasia\n", 267 | "\t b ) Old World\n", 268 | "\t c ) Australia\n", 269 | "\t d ) Africa\n", 270 | "\n", 271 | "More options: [] \n", 272 | "\n", 273 | "\n" 274 | ] 275 | } 276 | ], 277 | "source": [ 278 | "import requests\n", 279 | "import json\n", 280 | "import re\n", 281 | "import random\n", 282 | "from pywsd.similarity import max_similarity\n", 283 | "from pywsd.lesk import adapted_lesk\n", 284 | "from pywsd.lesk import simple_lesk\n", 285 | "from pywsd.lesk import cosine_lesk\n", 286 | "from nltk.corpus import wordnet as wn\n", 287 | "\n", 288 | "# Distractors from Wordnet\n", 289 | "def get_distractors_wordnet(syn,word):\n", 290 | " distractors=[]\n", 291 | " word= word.lower()\n", 292 | " orig_word = word\n", 293 | " if len(word.split())>0:\n", 294 | " word = word.replace(\" \",\"_\")\n", 295 | " hypernym = syn.hypernyms()\n", 296 | " if len(hypernym) == 0: \n", 297 | " return distractors\n", 298 | " for item in hypernym[0].hyponyms():\n", 299 | " name = item.lemmas()[0].name()\n", 300 | " #print (\"name \",name, \" word\",orig_word)\n", 301 | " if name == orig_word:\n", 302 | " continue\n", 303 | " name = name.replace(\"_\",\" \")\n", 304 | " name = \" \".join(w.capitalize() for w in name.split())\n", 305 | " if name is not None and name not in distractors:\n", 306 | " distractors.append(name)\n", 307 | " return distractors\n", 308 | "\n", 309 | "def get_wordsense(sent,word):\n", 310 | " word= word.lower()\n", 311 | " \n", 312 | " if len(word.split())>0:\n", 313 | " word = word.replace(\" \",\"_\")\n", 314 | " \n", 315 | " \n", 316 | " synsets = wn.synsets(word,'n')\n", 317 | " if synsets:\n", 318 | " wup = max_similarity(sent, word, 'wup', pos='n')\n", 319 | " adapted_lesk_output = adapted_lesk(sent, word, pos='n')\n", 320 | " lowest_index = min (synsets.index(wup),synsets.index(adapted_lesk_output))\n", 321 | " return synsets[lowest_index]\n", 322 | " else:\n", 323 | " return None\n", 324 | "\n", 325 | "# Distractors from http://conceptnet.io/\n", 326 | "def get_distractors_conceptnet(word):\n", 327 | " word = word.lower()\n", 328 | " original_word= word\n", 329 | " if (len(word.split())>0):\n", 330 | " word = word.replace(\" \",\"_\")\n", 331 | " distractor_list = [] \n", 332 | " url = \"http://api.conceptnet.io/query?node=/c/en/%s/n&rel=/r/PartOf&start=/c/en/%s&limit=5\"%(word,word)\n", 333 | " obj = requests.get(url).json()\n", 334 | "\n", 335 | " for edge in obj['edges']:\n", 336 | " link = edge['end']['term'] \n", 337 | "\n", 338 | " url2 = \"http://api.conceptnet.io/query?node=%s&rel=/r/PartOf&end=%s&limit=10\"%(link,link)\n", 339 | " obj2 = requests.get(url2).json()\n", 340 | " for edge in obj2['edges']:\n", 341 | " word2 = edge['start']['label']\n", 342 | " if word2 not in distractor_list and original_word.lower() not in word2.lower():\n", 343 | " distractor_list.append(word2)\n", 344 | " \n", 345 | " return distractor_list\n", 346 | "\n", 347 | "key_distractor_list = {}\n", 348 | "\n", 349 | "for keyword in keyword_sentence_mapping:\n", 350 | " wordsense = get_wordsense(keyword_sentence_mapping[keyword][0],keyword)\n", 351 | " if wordsense:\n", 352 | " distractors = get_distractors_wordnet(wordsense,keyword)\n", 353 | " if len(distractors) ==0:\n", 354 | " distractors = get_distractors_conceptnet(keyword)\n", 355 | " if len(distractors) != 0:\n", 356 | " key_distractor_list[keyword] = distractors\n", 357 | " else:\n", 358 | " \n", 359 | " distractors = get_distractors_conceptnet(keyword)\n", 360 | " if len(distractors) != 0:\n", 361 | " key_distractor_list[keyword] = distractors\n", 362 | "\n", 363 | "index = 1\n", 364 | "print (\"#############################################################################\")\n", 365 | "print (\"NOTE:::::::: Since the algorithm might have errors along the way, wrong answer choices generated might not be correct for some questions. \")\n", 366 | "print (\"#############################################################################\\n\\n\")\n", 367 | "for each in key_distractor_list:\n", 368 | " sentence = keyword_sentence_mapping[each][0]\n", 369 | " pattern = re.compile(each, re.IGNORECASE)\n", 370 | " output = pattern.sub( \" _______ \", sentence)\n", 371 | " print (\"%s)\"%(index),output)\n", 372 | " choices = [each.capitalize()] + key_distractor_list[each]\n", 373 | " top4choices = choices[:4]\n", 374 | " random.shuffle(top4choices)\n", 375 | " optionchoices = ['a','b','c','d']\n", 376 | " for idx,choice in enumerate(top4choices):\n", 377 | " print (\"\\t\",optionchoices[idx],\")\",\" \",choice)\n", 378 | " print (\"\\nMore options: \", choices[4:20],\"\\n\\n\")\n", 379 | " index = index + 1\n", 380 | " \n" 381 | ] 382 | }, 383 | { 384 | "cell_type": "code", 385 | "execution_count": null, 386 | "metadata": {}, 387 | "outputs": [], 388 | "source": [] 389 | } 390 | ], 391 | "metadata": { 392 | "kernelspec": { 393 | "display_name": "conda_pytorch_p36", 394 | "language": "python", 395 | "name": "conda_pytorch_p36" 396 | }, 397 | "language_info": { 398 | "codemirror_mode": { 399 | "name": "ipython", 400 | "version": 3 401 | }, 402 | "file_extension": ".py", 403 | "mimetype": "text/x-python", 404 | "name": "python", 405 | "nbconvert_exporter": "python", 406 | "pygments_lexer": "ipython3", 407 | "version": "3.6.5" 408 | } 409 | }, 410 | "nbformat": 4, 411 | "nbformat_minor": 2 412 | } 413 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Generate_MCQ_BERT_Wordnet_Conceptnet 2 | Generate Multiple choice Questions from any content or news article using BERT Extractive Summarization, Wordnet and Conceptnet 3 | 4 | Medium blog post explaining the algorithm : 5 | https://towardsdatascience.com/practical-ai-automatically-generate-multiple-choice-questions-mcqs-from-any-content-with-bert-2140d53a9bf5 6 | -------------------------------------------------------------------------------- /egypt.txt: -------------------------------------------------------------------------------- 1 | The Greek historian knew what he was talking about. The Nile River fed Egyptian civilization for hundreds of years. The Longest River the Nile is 4,160 miles long—the world’s longest river. It begins near the equator in Africa and flows north to the Mediterranean Sea. In the south the Nile churns with cataracts. A cataract is a waterfall. Near the sea the Nile branches into a delta. A delta is an area near a river’s mouth where the water deposits fine soil called silt. In the delta, the Nile divides into many streams. The river is called the upper Nile in the south and the lower Nile in the north. For centuries, heavy rains in Ethiopia caused the Nile to flood every summer. The floods deposited rich soil along the Nile’s shores. This soil was fertile, which means it was good for growing crops. Unlike the Tigris and Euphrates, the Nile River flooded at the same time every year, so farmers could predict when to plant their crops. Red Land, Black Land The ancient Egyptians lived in narrow bands of land on each side of the Nile. They called this region the black land because of the fertile soil that the floods deposited. The red land was the barren desert beyond the fertile region. Weather in Egypt was almost always the same. Eight months of the year were sunny and hot. The four months of winter were sunny but cooler. Most of the region received only an inch of rain a year. The parts of Egypt not near the Nile were a desert. Isolation The harsh desert acted as a barrier to keep out enemies. The Mediterranean coast was swampy and lacked good harbors. For these reasons, early Egyptians stayed close to home. Each year, Egyptian farmers watched for white birds called ibises, which flew up from the south. When the birds arrived, the annual flood waters would soon follow. After the waters drained away, farmers could plant seeds in the fertile soil. Agricultural Techniques By about 2400 B.C., farmers used technology to expand their farmland. Working together, they dug irrigation canals that carried river water to dry areas. Then they used a tool called a shaduf to spread the water across the fields. These innovative, or new, techniques gave them more farmland. Egyptian Crops Ancient Egyptians grew a large variety of foods. They were the first to grind wheat into flour and to mix the flour with yeast and water to make dough rise into bread. They grew vegetables such as lettuce, radishes, asparagus, and cucumbers. Fruits included dates, figs, grapes, and watermelons. Egyptians also grew the materials for their clothes. They were the first to weave fibers from flax plants into a fabric called linen. Lightweight linen cloth was perfect for hot Egyptian days. Men wore linen wraps around their waists. Women wore loose, sleeveless dresses. Egyptians also wove marsh grasses into sandals. Egyptian Houses Egyptians built houses using bricks made of mud from the Nile mixed with chopped straw. They placed narrow windows high in the walls to reduce bright sunlight. Egyptians often painted walls white to reflect the blazing heat. They wove sticks and palm trees to make roofs. Inside, woven reed mats covered the dirt floor. Most Egyptians slept on mats covered with linen sheets. Wealthy citizens enjoyed bed frames and cushions. Egyptian nobles had fancier homes with tree-lined courtyards for shade. Some had a pool filled with lotus blossoms and fish. Poorer Egyptians simply went to the roof to cool off after sunset. They often cooked, ate, and even slept outside. Egypt’s economy depended on farming. However, the natural resources of the area allowed other economic activities to develop too. The Egyptians wanted valuable metals that were not found in the black land. For example, they wanted copper to make tools and weapons. Egyptians looked for copper as early as 6000 B.C. Later they learned that iron was stronger, and they sought it as well. Ancient Egyptians also desired gold for its bright beauty. The Egyptian word for gold was nub. Nubia was the Egyptian name for the area of the upper Nile that had the richest gold mines in Africa. Mining minerals was difficult. Veins (long streaks) of copper, iron, and bronze were hidden inside desert mountains in the hot Sinai Peninsula, east of Egypt. Even during the cool season, chipping minerals out of the rock was miserable work. Egyptians mined precious stones too. They were probably the first people in the world to mine turquoise. The Egyptians also mined lapis lazuli. These beautiful blue stones were used in jewelry.The Nile had fish and other wildlife that Egyptians wanted. To go on the river, Egyptians made lightweight rafts by binding together reeds. They used everything from nets to harpoons to catch fish. One ancient painting even shows a man ready to hit a catfish with a wooden hammer. More adventurous hunters speared hippopotamuses and crocodiles along the Nile. Egyptians also captured quail with nets. They used boomerangs to knock down flying ducks and geese. (A boomerang is a curved stick that returns to the person who threw it.) Eventually, Egyptians equipped their reed boats with sails and oars. The Nile then became a highway. The river’s current was slow, so boaters used paddles to go faster when they traveled north with the current. Going south, they raised a sail and let the winds that blew in that direction push them. The Nile provided so well for Egyptians that sometimes they had surpluses, or more goods than they needed. They began to trade with each other. Ancient Egypt had no money, so people exchanged goods that they grew or made. This method of trade is called bartering. Egypt prospered along the Nile. This prosperity made life easier and provided greater opportunities for many Egyptians. When farmers produce food surpluses, the society’s economy begins to expand. Cities emerge as centers of culture and power, and people learn to do jobs that do not involve agriculture. For example, some ancient Egyptians learned to be scribes, people whose job was to write and keep records. As Egyptian civilization grew more complex, people took on jobs other than that of a farmer or scribe. Some skilled artisans erected stone or brick houses and temples. Other artisans made pottery, incense, mats, furniture, linen clothing, sandals, or jewelry. A few Egyptians traveled to the upper Nile to trade with other Africans. These traders took Egyptian products such as scrolls, linen, gold, and jewelry. They brought back exotic woods, animal skins, and live beasts. As Egypt grew, so did its need to organize. Egyptians created a government that divided the empire into 42 provinces. Many officials worked to keep the provinces running smoothly. Egypt also created an army to defend itself. One of the highest jobs in Egypt was to be a priest. Priests followed formal rituals and took care of the temples. Before entering a temple, a priest bathed and put on special linen garments and white sandals. Priests cleaned the sacred statues in temples, changed their clothes, and even fed them meals. Together, the priests and the ruler held ceremonies to please the gods. Egyptians believed that if the gods were angry, the Nile would not flood. As a result, crops would not grow, and people would die. So the ruler and the priests tried hard to keep the gods happy. By doing so, they hoped to maintain the social and political order. Slaves were at the bottom of society. In Egypt, people became slaves if they owed a debt, committed a crime, or were captured in war. Egyptian slaves were usually freed after a period of time. One exception was the slaves who had to work in the mines. Many died from the exhausting labor. Egypt was one of the best places in the ancient world to be a woman. Unlike other ancient African cultures, in Egyptian society men and women had fairly equal rights. For example, they could both own and manage their own property. The main job of most women was to care for their children and home, but some did other jobs too. Some women wove cloth. Others worked with their husbands in fields or workshops. Some women, such as Queen Tiy, even rose to important positions in the government. Children in Egypt played with toys such as dolls, animal figures, board games, and marbles. Their parents made the toys from wood or clay. Boys and girls also played rough physical games with balls made of leather or reeds. Boys and some girls from wealthy families went to schools run by scribes or priests. Most other children learned their parents’ jobs. Almost all Egyptians married when they were in their early teens. As in many ancient societies, much of the knowledge of Egypt came about as priests studied the world to find ways to please the gods. Other advances came about because of practical discoveries. Egyptian priests studied the sky as part of their religion. About 5,000 years ago, they noticed that a star now called Sirius appeared shortly before the Nile began to flood. The star returned to the same position in 365 days. Based on that, Egyptians developed the world’s first practical calendar. The Egyptians developed some of the first geometry. Each year the Nile’s floods washed away land boundaries. To restore property lines, surveyors measured the land by using ropes that were knotted at regular intervals. Geometric shapes such as squares and triangles were sacred to Egyptians. Architects used them in the design of royal temples and monuments. Egyptian doctors often prepared dead bodies for burial, so they knew the parts of the body. That knowledge helped them perform some of the world’s first surgery. Some doctors specialized in using medicines made of herbs. Egyptian medicine was far from perfect. Doctors believed that the heart controlled thought and the brain circulated blood, which is the opposite of what is known now. Some Egyptian treatments would raise eyebrows today. One “cure” for an upset stomach was to eat a hog’s tooth crushed inside sugar cakes! Beginning about 3000 B.C., Egyptians developed a writing system using hieroglyphs. Hieroglyphs Hieroglyphs are pictures that stand for different words or sounds. Early Egyptians created a hieroglyphic system with about 700 characters. Over time the system grew to include more than 6,000 symbols. The Egyptians also developed a paperlike material called papyrus papyrus from a reed of the same name. Egyptians cut the stems into strips, pressed them, and dried them into sheets that could be rolled into scrolls. Papyrus scrolls were light and easy to carry. With them, Egyptians created some of the first books. Legend says a king named Narmer united Upper and Lower Egypt. Some historians think Narmer actually represents several kings who gradually joined the two lands. After Egypt was united, its ruler wore the Double Crown. It combined the red Crown of Lower Egypt with the white Crown of Upper Egypt. The first dynasty of the Egyptian empire began about 2925 B.C. A dynasty is a line of rulers from the same family. When a king died, one of his children usually took his place as ruler. The order in which members of a royal family inherit a throne is called the succession. More than 30 dynasties ruled ancient Egypt. Historians divide ancient Egyptian dynasties into the Old Kingdom, the Middle Kingdom, and the New Kingdom. The Old Kingdom started about 2575 B.C., when the Egyptian empire was gaining strength. The king of Egypt became known as the pharaoh pharaoh. The word pharaoh meant “great house,” and it was originally used to describe the king’s palace. Later it became the title of the king himself. The pharaoh ruled from the capital city of Memphis. The ancient Egyptians thought the pharaoh was a child of the gods and a god himself. Egyptians believed that if the pharaoh and his subjects honored the gods, their lives would be happy. If Egypt suffered hard times for a long period, the people blamed the pharaoh for angering the gods. In such a case, a rival might drive him from power and start a new dynasty. Because the pharaoh was thought to be a god, government and religion were not separate in ancient Egypt. Priests had much power in the government. Many high officials were priests. The first rulers of Egypt were often buried in an underground tomb topped by mud brick. Soon, kings wanted more permanent monuments. They replaced the mud brick with a small pyramid of brick or stone. A pyramid is a structure shaped like a triangle, with four sides that meet at a point. About 2630 B.C., King Djoser built a much larger pyramid over his tomb. It is called a step pyramid because its sides rise in a series of giant steps. It is the oldest-known large stone structure in the world. About 80 years later, a pharaoh named Khufu decided he wanted a monument that would show the world how great he was. He ordered the construction of the largest pyramid ever built. Along its base, each side was about 760 feet long. The core was built from 2.3 million blocks of stone. Building the Great Pyramid was hard work. Miners cut the huge blocks of stone using copper saws and chisels. These tools were much softer than the iron tools developed later. Other teams of workers pulled the stone slabs up long, sloping ramps to their place on the pyramid. Near the top of the pyramid, the ramps ended. Workers dragged each heavy block hundreds of feet and then set it in place. Farmers did the heavy labor of hauling stone during the season when the Nile flooded their fields. Skilled stonecutters and overseers worked year-round. The Great Pyramid took nearly 20 years to build. An estimated 20,000 Egyptians worked on it. A city called Giza was built for the pyramid workers and the people who fed, clothed, and housed them. Eventually, Egyptians stopped building pyramids. One reason is that the pyramids drew attention to the tombs inside them. Grave robbers broke into the tombs to steal the treasure buried with the pharaohs. Sometimes they also stole the mummies. Egyptians believed that if a tomb was robbed, the person buried there could not have a happy afterlife. During the New Kingdom, pharaohs began building more secret tombs in an area called the Valley of the Kings. The burial chambers were hidden in mountains near the Nile. This way, the pharaohs hoped to protect their bodies and treasures from robbers. Both the pyramids and later tombs had several passageways leading to different rooms. This was to confuse grave robbers about which passage to take. Sometimes relatives, such as the queen, were buried in the extra rooms. Tombs were supposed to be the palaces of pharaohs in the afterlife. Mourners filled the tomb with objects ranging from food to furniture that the mummified pharaoh would need. Some tombs contained small statues that were supposed to be servants for the dead person. Egyptian artists decorated royal tombs with wall paintings and sculptures carved into the walls. Art was meant to glorify both the gods and the dead person. A sculpture of a dead pharaoh had “perfect” features, no matter how he really looked. Artists also followed strict rules about how to portray humans. Paintings showed a person’s head, arms, and legs from the side. They showed the front of the body from the neck down to the waist. Wall paintings showed pharaohs enjoying themselves so they could have a happy afterlife. One favorite scene was of the pharaoh fishing in a papyrus marsh. Warlike kings were often portrayed in battle. Scenes might also show people providing for the needs of the dead person. Such activities included growing and preparing food, caring for animals, and building boats. As hard as the pharaohs tried to hide themselves, robbers stole the treasures from almost every tomb. Only a secret tomb built for a New Kingdom pharaoh was ever found with much of its treasure untouched. The dazzling riches found in this tomb show how much wealth the pharaohs spent preparing for the afterlife. By about 2130 B.C., Egyptian kings began to lose their power to local rulers of the provinces. For about 500 more years, the kings held Egypt together, but with a much weaker central government. This period of Egyptian history is called the Middle Kingdom. Rulers during the Middle Kingdom also faced challenges from outside Egypt. A nomadic people called the Hyksos invaded Egypt from the northeast. Their army conquered by using better weapons and horse-drawn chariots, which were new to Egyptians. After about 100 years, the Egyptians drove out the Hyksos and began the New Kingdom. --------------------------------------------------------------------------------