├── README.md ├── Week 1 ├── Upload_Pic_1.PNG └── Upload_Pic_2.PNG ├── Week 2 ├── Extracting Numbers using RegEx.py ├── Quiz 1 - Regular Expressions.docx └── numbers.txt ├── Week 3 ├── Quiz 2 Networks and Sockets.docx └── socket1.py ├── Week 4 ├── Quiz 3 Reading Web Data From Python.docx └── web_parse.py ├── Week 5 ├── Quiz 4 eXtensible Markup Language.docx └── xml_parse.py └── Week 6 ├── Quiz 5 REST.docx ├── geo_json.py └── json_parse.py /README.md: -------------------------------------------------------------------------------- 1 | # This is my work for the Coursera Course - Using Python to Access Web Data 2 | 3 | ## Coursera Link - https://www.coursera.org/learn/python-network-data 4 | 5 | ### Instructor - Charles Severence 6 | 7 | ### University - University of Michigan 8 | 9 | 10 | ## Course Strucutre - 11 | 12 | - 6 weeks of Course 13 | 14 | - 1 peer reviewed assignment 15 | 16 | - 5 Quiz 17 | 18 | - 7 Python programming Assignment on external tool 19 | 20 | The answers to each programming assignment and Quiz is uploaded in each week. 21 | 22 | Instruction to use programming codes - 23 | 24 | 1. You need to install Python2 or Python3 in order to run the codes on your own machines. 25 | 26 | 2. You also require to download and install the BeautifulSoup library. 27 | 28 | For linux - 29 | ``` sudo apt-get install python-bs4 ``` (for Python 2) 30 | 31 | sudo apt-get install python3-bs4 (for Python 3) 32 | 33 | For more refer to - https://www.crummy.com/software/BeautifulSoup/bs4/doc/ 34 | 35 | 3. For windows you need to install pip and then use it to install BS 36 | 37 | 4. Some code have comments, Please read them to get your answers 38 | 39 | 5. All the codes follow the standard pattern prescribed by the Course instrucutor. 40 | 41 | 42 | Instruction for Quiz document - 43 | 44 | 1. The quiz document are in the form of Word Document. 45 | 46 | 2. Each Document has Screenshot of the right answer along with the questions. 47 | 48 | 3. There may be a few exception where the answer is wrong, In those cases the answer is explicitly mentioned below the screenshot. 49 | 50 | ### NOTE - Some of the theoretical question may differ due to being shuffled and the order of the option may also differ. 51 | 52 | ### NOTE - In case of any confusion regarding how the code works and other issues, you can refer to the worked examples in the course. They will explain each and every line. 53 | -------------------------------------------------------------------------------- /Week 1/Upload_Pic_1.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/I-am-Harsh/Using-Python-to-Access-Web-Data/8064c7c5ea52c216900dc1f6412f7dcf86ff023c/Week 1/Upload_Pic_1.PNG -------------------------------------------------------------------------------- /Week 1/Upload_Pic_2.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/I-am-Harsh/Using-Python-to-Access-Web-Data/8064c7c5ea52c216900dc1f6412f7dcf86ff023c/Week 1/Upload_Pic_2.PNG -------------------------------------------------------------------------------- /Week 2/Extracting Numbers using RegEx.py: -------------------------------------------------------------------------------- 1 | import re 2 | 3 | #Create the file numbers.txt or anything you like and replace it 4 | #The file is already included in the repo 5 | #However your file may be different 6 | 7 | file = open("numbers.txt") 8 | 9 | text = file.read() 10 | 11 | number_final = re.findall('[0-9]+',text) 12 | 13 | total = 0 14 | for i in number_final: 15 | i = int(i) 16 | total = total + i; 17 | 18 | print(total) 19 | 20 | file.close() 21 | -------------------------------------------------------------------------------- /Week 2/Quiz 1 - Regular Expressions.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/I-am-Harsh/Using-Python-to-Access-Web-Data/8064c7c5ea52c216900dc1f6412f7dcf86ff023c/Week 2/Quiz 1 - Regular Expressions.docx -------------------------------------------------------------------------------- /Week 2/numbers.txt: -------------------------------------------------------------------------------- 1 | This file contains the actual data for your assignment - good luck! 2 | 3 | 4 | 2221 Why should you learn to write programs? 5 | 6 | Writing programs (or programming) is a very creative 7 | 981 and rewarding activity. You can write programs for 7804 8 | many reasons, ranging from making your living to solving 9 | a difficult data analysis problem to having fun to helping 10 | 8361 someone else solve a problem. This book assumes that 11 | everyone needs to know how to program, and that once 12 | you know how to program you will figure out what you want 13 | to do with your newfound skills. 14 | 15 | We are surrounded in our daily lives with computers ranging 16 | 5069 from laptops to cell phones. We can think of these computers 17 | as 904 our 5341 personal 7892 assistants who can take care of many things 18 | on our behalf. The hardware in our current-day computers 19 | is essentially built to continuously ask us the question, 20 | What would you like me to do next? 21 | 22 | Programmers add an operating system and a set of applications 23 | to the hardware and we end up with a Personal Digital 24 | Assistant that is quite helpful and capable of helping 25 | us do many different things. 26 | 27 | Our computers are fast and have vast amounts of memory and 28 | could be very helpful to us if we only knew the language to 29 | speak to explain to the computer what we would like it to 30 | do next. If we knew this language, we could tell the 31 | computer 1133 to 281 do 7616 tasks on our behalf that were repetitive. 32 | Interestingly, the kinds of things computers can do best 33 | are often the kinds of things that we humans find boring 34 | and mind-numbing. 35 | 36 | For example, look at the first three paragraphs of this 37 | chapter and tell me the most commonly used word and how 38 | many times the word is used. While you were able to read 39 | 481 and understand the words in a few seconds, counting them 3741 40 | is almost painful because it is not the kind of problem 41 | that human minds are designed to solve. For a computer 42 | 1800 the opposite is true, reading and understanding text 559 43 | from a piece of paper is hard for a computer to do 44 | but counting the words and telling you how many times 45 | the most used word was used is very easy for the 46 | computer: 47 | 48 | Our personal information analysis assistant quickly 49 | told us that the word to was used sixteen times in the 50 | first three paragraphs of this chapter. 51 | 52 | 9226 This very fact that computers are good at things 53 | that humans are not is why you need to become 54 | skilled at talking computer language. Once you learn 55 | this new language, you can delegate mundane tasks 56 | to your partner (the computer), leaving more time 57 | for you to do the 58 | things that you are uniquely suited for. You bring 59 | creativity, intuition, and inventiveness to this 60 | partnership. 61 | 62 | Creativity and motivation 63 | 64 | While this book is not intended for professional programmers, professional 65 | programming can be a very rewarding job both financially and personally. 66 | Building useful, elegant, and clever programs for others to use is a very 67 | creative activity. Your computer or Personal Digital Assistant (PDA) 68 | usually contains many different programs from many different groups of 69 | programmers, each competing for your attention and interest. They try 70 | their best to meet your needs and give you a great user experience in the 71 | process. In some situations, when you choose a piece of software, the 72 | programmers are directly compensated because of your choice. 73 | 5784 74 | If we think of programs as the creative output of groups of programmers, 75 | perhaps the following figure is a more sensible version of our PDA: 76 | 599 5894 77 | For now, our primary motivation is not to make money or please end users, but 78 | instead for us to be more productive in handling the data and 79 | information that we will encounter in our lives. 80 | When you first start, you will be both the programmer and the end user of 81 | your programs. As you gain skill as a programmer and 82 | programming feels more creative to you, your thoughts may turn 83 | toward developing programs for others. 84 | 85 | Computer hardware architecture 86 | 87 | Before we start learning the language we 88 | speak to give instructions to computers to 89 | develop software, we need to learn a small amount about 90 | how computers are built. 91 | 92 | Central Processing Unit (or CPU) is 93 | the part of the computer that is built to be obsessed 94 | 5718 with what is next? If your computer is rated 5507 95 | 5591 at three Gigahertz, it means that the CPU will ask What next? 1029 96 | three billion times per second. You are going to have to 97 | 2726 learn how to talk fast to keep up with the CPU. 6526 98 | 4449 1823 99 | Main Memory is used to store information 100 | that the CPU needs in a hurry. The main memory is nearly as 101 | fast as the CPU. But the information stored in the main 102 | 7338 memory vanishes when the computer is turned off. 3115 103 | 104 | Secondary Memory is also used to store 105 | information, but it is much slower than the main memory. 106 | The advantage of the secondary memory is that it can 107 | store information even when there is no power to the 108 | computer. Examples of secondary memory are disk drives 109 | or flash memory (typically found in USB sticks and portable 110 | music players). 111 | 7832 2395 112 | Input and Output Devices are simply our 113 | screen, keyboard, mouse, microphone, speaker, touchpad, etc. 114 | They are all of the ways we interact with the computer. 115 | 8902 116 | These days, most computers also have a 117 | Network Connection to retrieve information over a network. 118 | We can think of the network as a very slow place to store and 119 | retrieve data that might not always be up. So in a sense, 120 | the network is a slower and at times unreliable form of 121 | Secondary Memory. 122 | 7904 123 | While most of the detail of how these components work is best left 124 | to computer builders, it helps to have some terminology 125 | so we can talk about these different parts as we write our programs. 126 | 127 | As a programmer, your job is to use and orchestrate 128 | each of these resources to solve the problem that you need to solve 129 | and analyze the data you get from the solution. As a programmer you will 130 | mostly be talking to the CPU and telling it what to 131 | do next. Sometimes you will tell the CPU to use the main memory, 132 | secondary memory, network, or the input/output devices. 133 | 6015 7180 134 | You need to be the person who answers the CPU's What next? 135 | question. But it would be very uncomfortable to shrink you 136 | down to five mm tall and insert you into the computer just so you 137 | could issue a command three billion times per second. So instead, 138 | you must write down your instructions in advance. 139 | We call these stored instructions a program and the act 140 | 4443 of writing these instructions down and getting the instructions to 141 | be correct programming. 142 | 143 | Understanding programming 144 | 1791 5962 145 | In the rest of this book, we will try to turn you into a person 146 | who 3669 is 9352 skilled 2675 in the art of programming. In the end you will be a 147 | programmer --- perhaps not a professional programmer, but 148 | at least you will have the skills to look at a data/information 149 | analysis problem and develop a program to solve the problem. 150 | 151 | problem solving 152 | 153 | In a sense, you need two skills to be a programmer: 154 | 155 | First, you need to know the programming language (Python) - 156 | you need to know the vocabulary and the grammar. You need to be able 157 | to spell the words in this new language properly and know how to construct 158 | well-formed sentences in this new language. 159 | 160 | Second, you need to tell a story. In writing a story, 161 | you combine words and sentences to convey an idea to the reader. 162 | There is a skill and art in constructing the story, and skill in 163 | 1824 story writing is improved by doing some writing and getting some 164 | feedback. In programming, our program is the story and the 165 | problem you are trying to solve is the idea. 166 | 167 | itemize 168 | 169 | Once you learn one programming language such as Python, you will 170 | find it much easier to learn a second programming language such 171 | as JavaScript or C++. The new programming language has very different 172 | vocabulary and grammar but the problem-solving skills 173 | will be the same across all programming languages. 174 | 2483 175 | You will learn the vocabulary and sentences of Python pretty quickly. 176 | It will take longer for you to be able to write a coherent program 177 | to solve a brand-new problem. We teach programming much like we teach 178 | writing. 1283 7015 We 4811 start reading and explaining programs, then we write 179 | simple programs, and then we write increasingly complex programs over time. 180 | At some point you get your muse and see the patterns on your own 181 | and can see more naturally how to take a problem and 182 | write a program that solves that problem. And once you get 183 | to that point, programming becomes a very pleasant and creative process. 184 | 185 | We start with the vocabulary and structure of Python programs. Be patient 186 | 633 as the simple examples remind you of when you started reading for the first 8425 187 | time. 188 | 189 | Words and sentences 190 | 191 | Unlike human languages, the Python vocabulary is actually pretty small. 192 | We call this vocabulary the reserved words. These are words that 193 | have very special meaning to Python. When Python sees these words in 194 | a Python program, they have one and only one meaning to Python. Later 195 | as you write programs you will make up your own words that have meaning to 196 | you called variables. You will have great latitude in choosing 197 | 5901 your names for your variables, but you cannot use any of Python's 198 | reserved words as a name for a variable. 199 | 1641 2484 5837 200 | When we train a dog, we use special words like 201 | sit, stay, and fetch. When you talk to a dog and 202 | don't use any of the reserved words, they just look at you with a 203 | 8408 quizzical look on their face until you say a reserved word. 6937 204 | For example, if you say, 205 | I wish more people would walk to improve their overall health, 206 | what most dogs likely hear is, 207 | blah blah blah walk blah blah blah blah. 208 | That is because walk is a reserved word in dog language. 209 | 210 | 2143 The reserved words in the language where humans talk to 211 | Python include the following: 212 | 213 | and del from not while 214 | as elif global or with 215 | assert else if pass yield 216 | 6540 break except import print 217 | class exec in raise 218 | continue finally is return 219 | def for lambda try 220 | 221 | 9456 That is it, and unlike a dog, Python is already completely trained. 1182 222 | When you say try, Python will try every time you say it without 223 | fail. 224 | 225 | We will learn these reserved words and how they are used in good time, 226 | but for now we will focus on the Python equivalent of speak (in 227 | human-to-dog language). The nice thing about telling Python to speak 228 | is that we can even tell it what to say by giving it a message in quotes: 229 | 230 | And we have even written our first syntactically correct Python sentence. 231 | Our sentence starts with the reserved word print followed 232 | by a string of text of our choosing enclosed in single quotes. 233 | 234 | Conversing with Python 235 | 4955 236 | Now that we have a word and a simple sentence that we know in Python, 237 | we need to know how to start a conversation with Python to test 238 | our new language skills. 239 | 240 | Before you can converse with Python, you must first install the Python 241 | software on your computer and learn how to start Python on your 242 | computer. That is too much detail for this chapter so I suggest 243 | that you consult www.py4e.com where I have detailed 244 | instructions and screencasts of setting up and starting Python 245 | on Macintosh and Windows systems. At some point, you will be in 246 | a terminal or command window and you will type python and 247 | the Python interpreter will start executing in interactive mode 248 | 2690 and appear somewhat as follows: 6228 249 | interactive mode 250 | 251 | The >>> prompt is the Python interpreter's way of asking you, What 252 | do you want me to do next? Python is ready to have a conversation with 253 | you. All you have to know is how to speak the Python language. 254 | 255 | Let's say for example that you did not know even the simplest Python language 256 | words or sentences. You might want to use the standard line that astronauts 257 | use when they land on a faraway planet and try to speak with the inhabitants 258 | of the planet: 259 | 260 | This is not going so well. Unless you think of something quickly, 261 | 787 the inhabitants of the planet are likely to stab you with their spears, 262 | put you on a spit, roast you over a fire, and eat you for dinner. 263 | 264 | At 171 this 5173 point, 5971 you should also realize that while Python 265 | is amazingly complex and powerful and very picky about 266 | the syntax you use to communicate with it, Python is 267 | not intelligent. You are really just having a conversation 268 | with yourself, but using proper syntax. 269 | 270 | In 8062 a 3290 sense, 5289 when you use a program written by someone else 271 | the conversation is between you and those other 272 | programmers with Python acting as an intermediary. Python 273 | is a way for the creators of programs to express how the 274 | conversation is supposed to proceed. And 275 | in just a few more chapters, you will be one of those 276 | programmers using Python to talk to the users of your program. 277 | 7302 4092 8443 278 | 9330 Before we leave our first conversation with the Python 279 | interpreter, you should probably know the proper way 280 | to say good-bye when interacting with the inhabitants 281 | of Planet Python: 282 | 6009 1409 6442 283 | You will notice that the error is different for the first two 284 | incorrect attempts. The second error is different because 285 | if is a reserved word and Python saw the reserved word 286 | and thought we were trying to say something but got the syntax 287 | of the sentence wrong. 288 | 289 | Terminology: interpreter and compiler 290 | 291 | Python is a high-level language intended to be relatively 292 | straightforward for humans to read and write and for computers 293 | to read and process. Other high-level languages include Java, C++, 294 | PHP, Ruby, Basic, Perl, JavaScript, and many more. The actual hardware 295 | inside the Central Processing Unit (CPU) does not understand any 296 | of these high-level languages. 297 | 298 | The CPU understands a language we call machine language. Machine 299 | language is very simple and frankly very tiresome to write because it 300 | is represented all in zeros and ones. 301 | 302 | Machine language seems quite simple on the surface, given that there 303 | are only zeros and ones, but its syntax is even more complex 304 | and far more intricate than Python. So very few programmers ever write 305 | machine language. Instead we build various translators to allow 306 | programmers to write in high-level languages like Python or JavaScript 307 | and these translators convert the programs to machine language for actual 308 | execution by the CPU. 309 | 310 | 6361 Since machine language is tied to the computer hardware, machine language 311 | is not portable across different types of hardware. Programs written in 312 | high-level languages can be moved between different computers by using a 313 | different interpreter on the new machine or recompiling the code to create 314 | a machine language version of the program for the new machine. 315 | 316 | These programming language translators fall into two general categories: 317 | (one) interpreters and (two) compilers. 318 | 319 | An interpreter reads the source code of the program as written by the 320 | programmer, parses the source code, and interprets the instructions on the fly. 321 | Python is an interpreter and when we are running Python interactively, 322 | we can type a line of Python (a sentence) and Python processes it immediately 323 | and is ready for us to type another line of Python. 324 | 325 | Some of the lines of Python tell Python that you want it to remember some 326 | value for later. We need to pick a name for that value to be remembered and 327 | 3622 we can use that symbolic name to retrieve the value later. We use the 2641 328 | term variable to refer to the labels we use to refer to this stored data. 329 | 330 | In this example, we ask Python to remember the value six and use the label x 331 | so we can retrieve the value later. We verify that Python has actually remembered 332 | the value using x and multiply 333 | it by seven and put the newly computed value in y. Then we ask Python to print out 334 | the value currently in y. 335 | 336 | Even though we are typing these commands into Python one line at a time, Python 337 | is treating them as an ordered sequence of statements with later statements able 338 | to retrieve data created in earlier statements. We are writing our first 339 | simple paragraph with four sentences in a logical and meaningful order. 340 | 341 | It is the nature of an interpreter to be able to have an interactive conversation 342 | as shown above. A compiler needs to be handed the entire program in a file, and then 343 | it runs a process to translate the high-level source code into machine language 344 | and then the compiler puts the resulting machine language into a file for later 345 | execution. 346 | 347 | If you have a Windows system, often these executable machine language programs have a 348 | suffix of .exe or .dll which stand for executable and dynamic link 349 | library respectively. In Linux and Macintosh, there is no suffix that uniquely marks 350 | a file as executable. 351 | 352 | If you were to open an executable file in a text editor, it would look 353 | completely crazy and be unreadable: 354 | 8116 355 | It is not easy to read or write machine language, so it is nice that we have 356 | compilers that allow us to write in high-level 357 | languages like Python or C. 358 | 359 | Now at this point in our discussion of compilers and interpreters, you should 360 | be wondering a bit about the Python interpreter itself. What language is 361 | it written in? Is it written in a compiled language? When we type 362 | python, what exactly is happening? 363 | 364 | The Python interpreter is written in a high-level language called C. 365 | You can look at the actual source code for the Python interpreter by 366 | going to www.python.org and working your way to their source code. 367 | 9705 So Python is a program itself and it is compiled into machine code. 6810 368 | When you installed Python on your computer (or the vendor installed it), 369 | you copied a machine-code copy of the translated Python program onto your 370 | system. In Windows, the executable machine code for Python itself is likely 371 | in a file. 372 | 373 | That is more than you really need to know to be a Python programmer, but 374 | sometimes it pays to answer those little nagging questions right at 375 | the beginning. 376 | 377 | Writing a program 378 | 6012 379 | Typing commands into the Python interpreter is a great way to experiment 380 | with Python's features, but it is not recommended for solving more complex problems. 381 | 1331 4656 382 | When we want to write a program, 383 | we use a text editor to write the Python instructions into a file, 384 | which is called a script. By 385 | convention, Python scripts have names that end with .py. 386 | 387 | script 388 | 389 | To execute the script, you have to tell the Python interpreter 390 | the name of the file. In a Unix or Windows command window, 391 | you would type python hello.py as follows: 392 | 393 | We call the Python interpreter and tell it to read its source code from 394 | the file hello.py instead of prompting us for lines of Python code 395 | interactively. 396 | 397 | You will notice that there was no need to have quit() at the end of 398 | the Python program in the file. When Python is reading your source code 399 | from a file, it knows to stop when it reaches the end of the file. 400 | 401 | What is a program? 402 | 403 | The definition of a program at its most basic is a sequence 404 | of Python statements that have been crafted to do something. 405 | Even our simple hello.py script is a program. It is a one-line 406 | program and is not particularly useful, but in the strictest definition, 407 | it is a Python program. 408 | 409 | It might be easiest to understand what a program is by thinking about a problem 410 | that a program might be built to solve, and then looking at a program 411 | that would solve that problem. 412 | 413 | Lets say you are doing Social Computing research on Facebook posts and 414 | you are interested in the most frequently used word in a series of posts. 415 | You could print out the stream of Facebook posts and pore over the text 416 | looking for the most common word, but that would take a long time and be very 417 | mistake prone. You would be smart to write a Python program to handle the 418 | task quickly and accurately so you can spend the weekend doing something 419 | fun. 420 | 421 | For example, look at the following text about a clown and a car. Look at the 422 | text and figure out the most common word and how many times it occurs. 423 | 424 | Then imagine that you are doing this task looking at millions of lines of 425 | text. Frankly it would be quicker for you to learn Python and write a 426 | Python program to count the words than it would be to manually 427 | scan the words. 428 | 429 | The even better news is that I already came up with a simple program to 430 | find the most common word in a text file. I wrote it, 431 | tested it, and now I am giving it to you to use so you can save some time. 432 | 433 | You don't even need to know Python to use this program. You will need to get through 434 | Chapter ten of this book to fully understand the awesome Python techniques that were 435 | used to make the program. You are the end user, you simply use the program and marvel 436 | at its cleverness and how it saved you so much manual effort. 437 | You simply type the code 438 | into a file called words.py and run it or you download the source 439 | code from http://www.py4e.com/code3/ and run it. 440 | 441 | This is a good example of how Python and the Python language are acting as an intermediary 442 | between you (the end user) and me (the programmer). Python is a way for us to exchange useful 443 | instruction sequences (i.e., programs) in a common language that can be used by anyone who 444 | installs Python on their computer. So neither of us are talking to Python, 445 | instead we are communicating with each other through Python. 446 | 447 | The building blocks of programs 448 | 449 | In the next few chapters, we will learn more about the vocabulary, sentence structure, 450 | paragraph structure, and story structure of Python. We will learn about the powerful 451 | capabilities of Python and how to compose those capabilities together to create useful 452 | programs. 453 | 454 | There are some low-level conceptual patterns that we use to construct programs. These 455 | constructs are not just for Python programs, they are part of every programming language 456 | from machine language up to the high-level languages. 457 | 458 | description 459 | 460 | Get data from the outside world. This might be 461 | reading data from a file, or even some kind of sensor like 462 | a microphone or GPS. In our initial programs, our input will come from the user 463 | typing data on the keyboard. 464 | 465 | Display the results of the program on a screen 466 | or store them in a file or perhaps write them to a device like a 467 | speaker to play music or speak text. 468 | 469 | Perform statements one after 470 | another in the order they are encountered in the script. 471 | 472 | Check for certain conditions and 473 | then execute or skip a sequence of statements. 474 | 475 | Perform some set of statements 476 | repeatedly, usually with 477 | some variation. 478 | 479 | Write a set of instructions once and give them a name 480 | and then reuse those instructions as needed throughout your program. 481 | 482 | description 483 | 484 | It sounds almost too simple to be true, and of course it is never 485 | so simple. It is like saying that walking is simply 486 | putting one foot in front of the other. The art 487 | of writing a program is composing and weaving these 488 | basic elements together many times over to produce something 489 | that is useful to its users. 490 | 491 | The word counting program above directly uses all of 492 | these patterns except for one. 493 | 494 | What could possibly go wrong? 495 | 496 | As we saw in our earliest conversations with Python, we must 497 | communicate very precisely when we write Python code. The smallest 498 | deviation or mistake will cause Python to give up looking at your 499 | program. 500 | 501 | Beginning programmers often take the fact that Python leaves no 502 | room for errors as evidence that Python is mean, hateful, and cruel. 503 | While Python seems to like everyone else, Python knows them 504 | personally and holds a grudge against them. Because of this grudge, 505 | Python takes our perfectly written programs and rejects them as 506 | unfit just to torment us. 507 | 508 | There is little to be gained by arguing with Python. It is just a tool. 509 | It has no emotions and it is happy and ready to serve you whenever you 510 | need it. Its error messages sound harsh, but they are just Python's 511 | call for help. It has looked at what you typed, and it simply cannot 512 | understand what you have entered. 513 | 514 | Python is much more like a dog, loving you unconditionally, having a few 515 | key words that it understands, looking you with a sweet look on its 516 | face (>>>), and waiting for you to say something it understands. 517 | When Python says SyntaxError: invalid syntax, it is simply wagging 518 | its tail and saying, You seemed to say something but I just don't 519 | understand what you meant, but please keep talking to me (>>>). 520 | 521 | As your programs become increasingly sophisticated, you will encounter three 522 | general types of errors: 523 | 524 | description 525 | 526 | These are the first errors you will make and the easiest 527 | to fix. A syntax error means that you have violated the grammar rules of Python. 528 | Python does its best to point right at the line and character where 529 | it noticed it was confused. The only tricky bit of syntax errors is that sometimes 530 | the mistake that needs fixing is actually earlier in the program than where Python 531 | noticed it was confused. So the line and character that Python indicates in 532 | a syntax error may just be a starting point for your investigation. 533 | 534 | A logic error is when your program has good syntax but there is a mistake 535 | in the order of the statements or perhaps a mistake in how the statements relate to one another. 536 | A good example of a logic error might be, take a drink from your water bottle, put it 537 | in your backpack, walk to the library, and then put the top back on the bottle. 538 | 539 | A semantic error is when your description of the steps to take 540 | is syntactically perfect and in the right order, but there is simply a mistake in 541 | the program. The program is perfectly correct but it does not do what 542 | you intended for it to do. A simple example would 543 | be if you were giving a person directions to a restaurant and said, ...when you reach 544 | the intersection with the gas station, turn left and go one mile and the restaurant 545 | is a red building on your left. Your friend is very late and calls you to tell you that 546 | they are on a farm and walking around behind a barn, with no sign of a restaurant. 547 | Then you say did you turn left or right at the gas station? and 548 | they say, I followed your directions perfectly, I have 549 | them written down, it says turn left and go one mile at the gas station. Then you say, 550 | I am very sorry, because while my instructions were syntactically correct, they 551 | sadly contained a small but undetected semantic error.. 552 | 553 | description 554 | 555 | Again in all three types of errors, Python is merely trying its hardest to 556 | do exactly what you have asked. 557 | 558 | The learning journey 559 | 560 | As you progress through the rest of the book, don't be afraid if the concepts 561 | don't seem to fit together well the first time. When you were learning to speak, 562 | it was not a problem for your first few years that you just made cute gurgling noises. 563 | And it was OK if it took six months for you to move from simple vocabulary to 564 | simple sentences and took five or six more years to move from sentences to paragraphs, and a 565 | few more years to be able to write an interesting complete short story on your own. 566 | 567 | We want you to learn Python much more rapidly, so we teach it all at the same time 568 | over the next few chapters. 569 | But it is like learning a new language that takes time to absorb and understand 570 | before it feels natural. 571 | That leads to some confusion as we visit and revisit 572 | topics to try to get you to see the big picture while we are defining the tiny 573 | fragments that make up that big picture. While the book is written linearly, and 574 | if you are taking a course it will progress in a linear fashion, don't hesitate 575 | to be very nonlinear in how you approach the material. Look forwards and backwards 576 | and read with a light touch. By skimming more advanced material without 577 | fully understanding the details, you can get a better understanding of the why? 578 | of programming. By reviewing previous material and even redoing earlier 579 | exercises, you will realize that you actually learned a lot of material even 580 | if the material you are currently staring at seems a bit impenetrable. 581 | 582 | Usually when you are learning your first programming language, there are a few 583 | wonderful Ah Hah! moments where you can look up from pounding away at some rock 584 | with a hammer and chisel and step away and see that you are indeed building 585 | a beautiful sculpture. 586 | 587 | If something seems particularly hard, there is usually no value in staying up all 588 | night and staring at it. Take a break, take a nap, have a snack, explain what you 589 | are having a problem with to someone (or perhaps your dog), and then come back to it with 590 | fresh eyes. I assure you that once you learn the programming concepts in the book 591 | you will look back and see that it was all really easy and elegant and it simply 592 | took you a bit of time to absorb it. 593 | 42 594 | The end -------------------------------------------------------------------------------- /Week 3/Quiz 2 Networks and Sockets.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/I-am-Harsh/Using-Python-to-Access-Web-Data/8064c7c5ea52c216900dc1f6412f7dcf86ff023c/Week 3/Quiz 2 Networks and Sockets.docx -------------------------------------------------------------------------------- /Week 3/socket1.py: -------------------------------------------------------------------------------- 1 | import socket 2 | 3 | mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 4 | 5 | #This is the URL which the program will connect to. 6 | #Change it in case you require it to 7 | mysock.connect(('data.pr4e.org', 80)) 8 | #This url is leads to the file. 9 | #Change it accordingly 10 | 11 | cmd = 'GET http://data.pr4e.org/intro-short.txt HTTP/1.0\r\n\r\n'.encode() 12 | mysock.send(cmd) 13 | 14 | while True: 15 | data = mysock.recv(512) 16 | if len(data) < 1: 17 | break 18 | print(data.decode(),end='') 19 | 20 | mysock.close() 21 | -------------------------------------------------------------------------------- /Week 4/Quiz 3 Reading Web Data From Python.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/I-am-Harsh/Using-Python-to-Access-Web-Data/8064c7c5ea52c216900dc1f6412f7dcf86ff023c/Week 4/Quiz 3 Reading Web Data From Python.docx -------------------------------------------------------------------------------- /Week 4/web_parse.py: -------------------------------------------------------------------------------- 1 | from bs4 import BeautifulSoup 2 | import urllib.request, urllib.parse, urllib.error 3 | import ssl 4 | 5 | #Insert the URL when prompted in terminal window 6 | 7 | #This code will serve as the answer for both the programming assignment. 8 | 9 | url = input("Enter URL - ") 10 | 11 | ############################################### Comment this for first question ############################## 12 | pos = int(input("Enter the position - ")) - 1 13 | count = int(input("Enter the count - ")) 14 | count1 = 0 15 | while(count1 < count): 16 | 17 | html = urllib.request.urlopen(url).read() 18 | soup = BeautifulSoup(html, 'html.parser') 19 | 20 | tags = soup('a') 21 | url = tags[pos].get('href') 22 | name = tags[pos].contents[0] 23 | count1 = count1 + 1 24 | 25 | print(name) 26 | 27 | ################################################################## 28 | 29 | 30 | 31 | 32 | 33 | 34 | ################################################ Comment this for second question ############################################### 35 | 36 | tags = soup('span') 37 | sum = 0 38 | for value in tags: 39 | temp = int(value.contents[0]) 40 | sum = sum + temp 41 | print(sum) 42 | 43 | ############################################################################################## -------------------------------------------------------------------------------- /Week 5/Quiz 4 eXtensible Markup Language.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/I-am-Harsh/Using-Python-to-Access-Web-Data/8064c7c5ea52c216900dc1f6412f7dcf86ff023c/Week 5/Quiz 4 eXtensible Markup Language.docx -------------------------------------------------------------------------------- /Week 5/xml_parse.py: -------------------------------------------------------------------------------- 1 | import urllib.request, urllib.response, urllib.error 2 | import xml.etree.ElementTree as ET 3 | 4 | 5 | url = input("Enter the URL - ") 6 | sum = 0 7 | 8 | html = urllib.request.urlopen(url).read() 9 | 10 | tree = ET.fromstring(html) 11 | 12 | count_list = tree.findall("comments/comment/count") 13 | 14 | for value in count_list: 15 | # count = value.find('count') 16 | sum = sum + int(value.text) 17 | print(sum) 18 | -------------------------------------------------------------------------------- /Week 6/Quiz 5 REST.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/I-am-Harsh/Using-Python-to-Access-Web-Data/8064c7c5ea52c216900dc1f6412f7dcf86ff023c/Week 6/Quiz 5 REST.docx -------------------------------------------------------------------------------- /Week 6/geo_json.py: -------------------------------------------------------------------------------- 1 | import urllib.request, urllib.parse, urllib.error 2 | import json 3 | import ssl 4 | 5 | api_key = False 6 | # If you have a Google Places API key, enter it here 7 | # api_key = 'AIzaSy___IDByT70' 8 | # https://developers.google.com/maps/documentation/geocoding/intro 9 | 10 | if api_key is False: 11 | api_key = 42 12 | serviceurl = 'http://py4e-data.dr-chuck.net/json?' 13 | else : 14 | serviceurl = 'https://maps.googleapis.com/maps/api/geocode/json?' 15 | 16 | # Ignore SSL certificate errors 17 | ctx = ssl.create_default_context() 18 | ctx.check_hostname = False 19 | ctx.verify_mode = ssl.CERT_NONE 20 | change = 1 21 | 22 | #Add the below code in a while loop if you wish to have a revurring code. 23 | address = input('Enter location: ') 24 | 25 | parms = dict() 26 | parms['address'] = address 27 | if api_key is not False: parms['key'] = api_key 28 | url = serviceurl + urllib.parse.urlencode(parms) 29 | 30 | uh = urllib.request.urlopen(url, context=ctx) 31 | data = uh.read().decode() 32 | js = json.loads(data) 33 | 34 | for place in js['results']: 35 | print("Place ID = ", place['place_id']) 36 | -------------------------------------------------------------------------------- /Week 6/json_parse.py: -------------------------------------------------------------------------------- 1 | import urllib.request, urllib.parse, urllib.error 2 | import json 3 | 4 | url = input("Enter the URL - ") 5 | 6 | data = urllib.request.urlopen(url).read() 7 | 8 | json_data = json.loads(data) 9 | 10 | 11 | sum = 0 12 | for user in json_data['comments']: 13 | sum = sum + int(user['count']) 14 | print(sum) --------------------------------------------------------------------------------