Here is some simple content for this page.
\\n \\n'" 124 | ] 125 | }, 126 | "execution_count": 21, 127 | "metadata": {}, 128 | "output_type": "execute_result" 129 | } 130 | ], 131 | "source": [ 132 | "page.content" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "#### You can see we extract the content but it's not look good." 140 | ] 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "metadata": {}, 145 | "source": [ 146 | "## We use BeautifulSoup library to parse the document and extract the text in beautiful manner" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": 22, 152 | "metadata": {}, 153 | "outputs": [], 154 | "source": [ 155 | "from bs4 import BeautifulSoup\n", 156 | "soup = BeautifulSoup(page.content,'html.parser')" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": 23, 162 | "metadata": {}, 163 | "outputs": [ 164 | { 165 | "data": { 166 | "text/plain": [ 167 | "\n", 168 | "\n", 169 | "\n", 170 | "\n", 171 | "Here is some simple content for this page.
\n", 175 | "\n", 176 | "" 177 | ] 178 | }, 179 | "execution_count": 23, 180 | "metadata": {}, 181 | "output_type": "execute_result" 182 | } 183 | ], 184 | "source": [ 185 | "soup" 186 | ] 187 | }, 188 | { 189 | "cell_type": "markdown", 190 | "metadata": {}, 191 | "source": [ 192 | "#### NOW IT'S LOOK BETTER!" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "#### To get even more beautiful we can use prettify method" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": 25, 205 | "metadata": {}, 206 | "outputs": [ 207 | { 208 | "name": "stdout", 209 | "output_type": "stream", 210 | "text": [ 211 | "\n", 212 | "\n", 213 | " \n", 214 | "\n", 220 | " Here is some simple content for this page.\n", 221 | "
\n", 222 | " \n", 223 | "\n" 224 | ] 225 | } 226 | ], 227 | "source": [ 228 | "print(soup.prettify())" 229 | ] 230 | }, 231 | { 232 | "cell_type": "markdown", 233 | "metadata": {}, 234 | "source": [ 235 | "#### Note - If you don't use print() and directly try to print soup.prettify() you will end up with messy text!" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": 26, 241 | "metadata": {}, 242 | "outputs": [ 243 | { 244 | "data": { 245 | "text/plain": [ 246 | "'\\n\\n \\n\\n Here is some simple content for this page.\\n
\\n \\n'" 247 | ] 248 | }, 249 | "execution_count": 26, 250 | "metadata": {}, 251 | "output_type": "execute_result" 252 | } 253 | ], 254 | "source": [ 255 | "soup.prettify()" 256 | ] 257 | }, 258 | { 259 | "cell_type": "markdown", 260 | "metadata": {}, 261 | "source": [ 262 | "#### LIKE THIS ..." 263 | ] 264 | }, 265 | { 266 | "cell_type": "markdown", 267 | "metadata": {}, 268 | "source": [ 269 | "### Now, if you want to select all the elements at the top level of the page using the children property of soup. Note that children returns a list generator, so we need to call the list function on it" 270 | ] 271 | }, 272 | { 273 | "cell_type": "code", 274 | "execution_count": 28, 275 | "metadata": {}, 276 | "outputs": [ 277 | { 278 | "data": { 279 | "text/plain": [ 280 | "Here is some simple content for this page.
\n", 306 | " \n", 307 | " ]" 308 | ] 309 | }, 310 | "execution_count": 29, 311 | "metadata": {}, 312 | "output_type": "execute_result" 313 | } 314 | ], 315 | "source": [ 316 | "list(soup.children)" 317 | ] 318 | }, 319 | { 320 | "cell_type": "code", 321 | "execution_count": 30, 322 | "metadata": {}, 323 | "outputs": [ 324 | { 325 | "data": { 326 | "text/plain": [ 327 | "'html'" 328 | ] 329 | }, 330 | "execution_count": 30, 331 | "metadata": {}, 332 | "output_type": "execute_result" 333 | } 334 | ], 335 | "source": [ 336 | "list(soup.children)[0]" 337 | ] 338 | }, 339 | { 340 | "cell_type": "code", 341 | "execution_count": 31, 342 | "metadata": {}, 343 | "outputs": [ 344 | { 345 | "data": { 346 | "text/plain": [ 347 | "'\\n'" 348 | ] 349 | }, 350 | "execution_count": 31, 351 | "metadata": {}, 352 | "output_type": "execute_result" 353 | } 354 | ], 355 | "source": [ 356 | "list(soup.children)[1]" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": 33, 362 | "metadata": {}, 363 | "outputs": [ 364 | { 365 | "data": { 366 | "text/plain": [ 367 | "\n", 368 | "\n", 369 | "Here is some simple content for this page.
\n", 373 | "\n", 374 | "" 375 | ] 376 | }, 377 | "execution_count": 33, 378 | "metadata": {}, 379 | "output_type": "execute_result" 380 | } 381 | ], 382 | "source": [ 383 | "list(soup.children)[2]" 384 | ] 385 | }, 386 | { 387 | "cell_type": "code", 388 | "execution_count": 34, 389 | "metadata": {}, 390 | "outputs": [ 391 | { 392 | "data": { 393 | "text/plain": [ 394 | "[bs4.element.Doctype, bs4.element.NavigableString, bs4.element.Tag]" 395 | ] 396 | }, 397 | "execution_count": 34, 398 | "metadata": {}, 399 | "output_type": "execute_result" 400 | } 401 | ], 402 | "source": [ 403 | "# To see the type\n", 404 | "[type(item) for item in list(soup.children)]" 405 | ] 406 | }, 407 | { 408 | "cell_type": "code", 409 | "execution_count": 35, 410 | "metadata": {}, 411 | "outputs": [], 412 | "source": [ 413 | "html = list(soup.children)[2]" 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "execution_count": 37, 419 | "metadata": {}, 420 | "outputs": [ 421 | { 422 | "data": { 423 | "text/plain": [ 424 | "['\\n', \n", 425 | "Here is some simple content for this page.
\n", 428 | " , '\\n']" 429 | ] 430 | }, 431 | "execution_count": 37, 432 | "metadata": {}, 433 | "output_type": "execute_result" 434 | } 435 | ], 436 | "source": [ 437 | "list(html.children)" 438 | ] 439 | }, 440 | { 441 | "cell_type": "markdown", 442 | "metadata": {}, 443 | "source": [ 444 | "#### As you can see above, there are two tags here, head, and body. We want to extract the text inside the p tag, so we'll dive into the body" 445 | ] 446 | }, 447 | { 448 | "cell_type": "code", 449 | "execution_count": 47, 450 | "metadata": {}, 451 | "outputs": [], 452 | "source": [ 453 | "body = body = list(html.children)[3]" 454 | ] 455 | }, 456 | { 457 | "cell_type": "code", 458 | "execution_count": 48, 459 | "metadata": {}, 460 | "outputs": [ 461 | { 462 | "data": { 463 | "text/plain": [ 464 | "\n", 465 | "Here is some simple content for this page.
\n", 466 | "" 467 | ] 468 | }, 469 | "execution_count": 48, 470 | "metadata": {}, 471 | "output_type": "execute_result" 472 | } 473 | ], 474 | "source": [ 475 | "body" 476 | ] 477 | }, 478 | { 479 | "cell_type": "code", 480 | "execution_count": 49, 481 | "metadata": {}, 482 | "outputs": [ 483 | { 484 | "data": { 485 | "text/plain": [ 486 | "['\\n',Here is some simple content for this page.
, '\\n']" 487 | ] 488 | }, 489 | "execution_count": 49, 490 | "metadata": {}, 491 | "output_type": "execute_result" 492 | } 493 | ], 494 | "source": [ 495 | "list(body.children)" 496 | ] 497 | }, 498 | { 499 | "cell_type": "code", 500 | "execution_count": 50, 501 | "metadata": {}, 502 | "outputs": [ 503 | { 504 | "data": { 505 | "text/plain": [ 506 | "Here is some simple content for this page.
" 507 | ] 508 | }, 509 | "execution_count": 50, 510 | "metadata": {}, 511 | "output_type": "execute_result" 512 | } 513 | ], 514 | "source": [ 515 | "list(body.children)[1]" 516 | ] 517 | }, 518 | { 519 | "cell_type": "code", 520 | "execution_count": 51, 521 | "metadata": {}, 522 | "outputs": [ 523 | { 524 | "data": { 525 | "text/plain": [ 526 | "'Here is some simple content for this page.'" 527 | ] 528 | }, 529 | "execution_count": 51, 530 | "metadata": {}, 531 | "output_type": "execute_result" 532 | } 533 | ], 534 | "source": [ 535 | "p = list(body.children)[1]\n", 536 | "p.get_text()" 537 | ] 538 | }, 539 | { 540 | "cell_type": "markdown", 541 | "metadata": {}, 542 | "source": [ 543 | "#### We can use the get_text method to extract all of the text inside the tag" 544 | ] 545 | }, 546 | { 547 | "cell_type": "markdown", 548 | "metadata": {}, 549 | "source": [ 550 | "### What we did above was useful for figuring out how to navigate a page, but it took a lot of commands to do something fairly simple. If we want to extract a single tag, we can instead use the find_all method, which will find all the instances of a tag on a page" 551 | ] 552 | }, 553 | { 554 | "cell_type": "code", 555 | "execution_count": 52, 556 | "metadata": {}, 557 | "outputs": [ 558 | { 559 | "data": { 560 | "text/plain": [ 561 | "[Here is some simple content for this page.
]" 562 | ] 563 | }, 564 | "execution_count": 52, 565 | "metadata": {}, 566 | "output_type": "execute_result" 567 | } 568 | ], 569 | "source": [ 570 | "soup = BeautifulSoup(page.content, 'html.parser')\n", 571 | "soup.find_all('p')" 572 | ] 573 | }, 574 | { 575 | "cell_type": "code", 576 | "execution_count": 53, 577 | "metadata": {}, 578 | "outputs": [], 579 | "source": [ 580 | "# Note it returns the list so we use list indexing" 581 | ] 582 | }, 583 | { 584 | "cell_type": "code", 585 | "execution_count": 54, 586 | "metadata": {}, 587 | "outputs": [ 588 | { 589 | "data": { 590 | "text/plain": [ 591 | "Here is some simple content for this page.
" 592 | ] 593 | }, 594 | "execution_count": 54, 595 | "metadata": {}, 596 | "output_type": "execute_result" 597 | } 598 | ], 599 | "source": [ 600 | "soup.find_all('p')[0]" 601 | ] 602 | }, 603 | { 604 | "cell_type": "code", 605 | "execution_count": 55, 606 | "metadata": {}, 607 | "outputs": [ 608 | { 609 | "data": { 610 | "text/plain": [ 611 | "'Here is some simple content for this page.'" 612 | ] 613 | }, 614 | "execution_count": 55, 615 | "metadata": {}, 616 | "output_type": "execute_result" 617 | } 618 | ], 619 | "source": [ 620 | "soup.find_all('p')[0].get_text()" 621 | ] 622 | }, 623 | { 624 | "cell_type": "markdown", 625 | "metadata": {}, 626 | "source": [ 627 | "#### You can also use find()" 628 | ] 629 | }, 630 | { 631 | "cell_type": "code", 632 | "execution_count": 56, 633 | "metadata": {}, 634 | "outputs": [ 635 | { 636 | "data": { 637 | "text/plain": [ 638 | "Here is some simple content for this page.
" 639 | ] 640 | }, 641 | "execution_count": 56, 642 | "metadata": {}, 643 | "output_type": "execute_result" 644 | } 645 | ], 646 | "source": [ 647 | "soup.find('p')" 648 | ] 649 | }, 650 | { 651 | "cell_type": "code", 652 | "execution_count": 57, 653 | "metadata": {}, 654 | "outputs": [ 655 | { 656 | "data": { 657 | "text/plain": [ 658 | "'Here is some simple content for this page.'" 659 | ] 660 | }, 661 | "execution_count": 57, 662 | "metadata": {}, 663 | "output_type": "execute_result" 664 | } 665 | ], 666 | "source": [ 667 | "soup.find('p').get_text()" 668 | ] 669 | }, 670 | { 671 | "cell_type": "markdown", 672 | "metadata": {}, 673 | "source": [ 674 | "#### Now, you have a good idea how to do web scraping. I will highly recommend you to check out how web scraping done in web pages using python. Refer to this - https://themenyouwanttobe.wordpress.com " 675 | ] 676 | }, 677 | { 678 | "cell_type": "code", 679 | "execution_count": null, 680 | "metadata": {}, 681 | "outputs": [], 682 | "source": [] 683 | } 684 | ], 685 | "metadata": { 686 | "kernelspec": { 687 | "display_name": "Python 3", 688 | "language": "python", 689 | "name": "python3" 690 | }, 691 | "language_info": { 692 | "codemirror_mode": { 693 | "name": "ipython", 694 | "version": 3 695 | }, 696 | "file_extension": ".py", 697 | "mimetype": "text/x-python", 698 | "name": "python", 699 | "nbconvert_exporter": "python", 700 | "pygments_lexer": "ipython3", 701 | "version": "3.6.5" 702 | } 703 | }, 704 | "nbformat": 4, 705 | "nbformat_minor": 2 706 | } 707 | -------------------------------------------------------------------------------- /Web Scraping.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Import the Library" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "from bs4 import BeautifulSoup\n", 17 | "import requests" 18 | ] 19 | }, 20 | { 21 | "cell_type": "markdown", 22 | "metadata": {}, 23 | "source": [ 24 | "#### Download the web page containing the forecast." 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 2, 30 | "metadata": {}, 31 | "outputs": [], 32 | "source": [ 33 | "page = requests.get(\"http://forecast.weather.gov/MapClick.php?lat=37.7772&lon=-122.4168\")" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "#### Create a BeautifulSoup class to parse the page." 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": 3, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "soup = BeautifulSoup(page.content, 'html.parser')" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "#### Find the div with id seven-day-forecast, and assign to seven_day" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": 4, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "seven_day = soup.find(id=\"seven-day-forecast\")" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "#### Inside seven_day, find each individual forecast item." 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 5, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "forecast_items = seven_day.find_all(class_=\"tombstone-container\")" 82 | ] 83 | }, 84 | { 85 | "cell_type": "markdown", 86 | "metadata": {}, 87 | "source": [ 88 | "#### Extract and print the first forecast item." 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 6, 94 | "metadata": {}, 95 | "outputs": [ 96 | { 97 | "name": "stdout", 98 | "output_type": "stream", 99 | "text": [ 100 | "\n",
101 | "
\n"
116 | ]
117 | }
118 | ],
119 | "source": [
120 | "tonight = forecast_items[0]\n",
121 | "print(tonight.prettify())"
122 | ]
123 | },
124 | {
125 | "cell_type": "markdown",
126 | "metadata": {},
127 | "source": [
128 | "#### We'll extract the name of the forecast item"
129 | ]
130 | },
131 | {
132 | "cell_type": "code",
133 | "execution_count": 7,
134 | "metadata": {},
135 | "outputs": [
136 | {
137 | "name": "stdout",
138 | "output_type": "stream",
139 | "text": [
140 | "Today\n",
141 | "Sunny\n",
142 | "High: 69 °F\n"
143 | ]
144 | }
145 | ],
146 | "source": [
147 | "period = tonight.find(class_ = \"period-name\").get_text()\n",
148 | "short_Desc = tonight.find(class_ = \"short-desc\").get_text()\n",
149 | "temp = tonight.find(class_ = \"temp temp-high\").get_text()\n",
150 | "\n",
151 | "print(period)\n",
152 | "print(short_Desc)\n",
153 | "print(temp)"
154 | ]
155 | },
156 | {
157 | "cell_type": "markdown",
158 | "metadata": {},
159 | "source": [
160 | "#### we can extract the title attribute from the img tag"
161 | ]
162 | },
163 | {
164 | "cell_type": "code",
165 | "execution_count": 8,
166 | "metadata": {},
167 | "outputs": [],
168 | "source": [
169 | "img = tonight.find('img')"
170 | ]
171 | },
172 | {
173 | "cell_type": "code",
174 | "execution_count": 9,
175 | "metadata": {},
176 | "outputs": [
177 | {
178 | "data": {
179 | "text/plain": [
180 | "'\n",
102 | " Today\n",
103 | "
\n",
104 | "
\n",
105 | "
\n",
107 | " ![\"Today:](\"newimages/medium/few.png\" "\\"Today:")
\n",
108 | "
\n", 110 | " Sunny\n", 111 | "
\n", 112 | "\n", 113 | " High: 69 °F\n", 114 | "
\n", 115 | "\\n'"
181 | ]
182 | },
183 | "execution_count": 9,
184 | "metadata": {},
185 | "output_type": "execute_result"
186 | }
187 | ],
188 | "source": [
189 | "img.prettify()"
190 | ]
191 | },
192 | {
193 | "cell_type": "code",
194 | "execution_count": 10,
195 | "metadata": {},
196 | "outputs": [
197 | {
198 | "name": "stdout",
199 | "output_type": "stream",
200 | "text": [
201 | "Today: Sunny, with a high near 69. West wind 11 to 18 mph, with gusts as high as 23 mph. \n"
202 | ]
203 | }
204 | ],
205 | "source": [
206 | "desc =img['title']\n",
207 | "print(desc)"
208 | ]
209 | },
210 | {
211 | "cell_type": "markdown",
212 | "metadata": {},
213 | "source": [
214 | "#### I use list comprehension to call the get_text method on each BeautifulSoup object"
215 | ]
216 | },
217 | {
218 | "cell_type": "code",
219 | "execution_count": 11,
220 | "metadata": {},
221 | "outputs": [
222 | {
223 | "data": {
224 | "text/plain": [
225 | "['Today',\n",
226 | " 'Tonight',\n",
227 | " 'Tuesday',\n",
228 | " 'TuesdayNight',\n",
229 | " 'Wednesday',\n",
230 | " 'WednesdayNight',\n",
231 | " 'Thursday',\n",
232 | " 'ThursdayNight',\n",
233 | " 'Friday']"
234 | ]
235 | },
236 | "execution_count": 11,
237 | "metadata": {},
238 | "output_type": "execute_result"
239 | }
240 | ],
241 | "source": [
242 | "period_tags = seven_day.select(\".tombstone-container .period-name\")\n",
243 | "periods = [pt.get_text() for pt in period_tags]\n",
244 | "periods"
245 | ]
246 | },
247 | {
248 | "cell_type": "code",
249 | "execution_count": 12,
250 | "metadata": {},
251 | "outputs": [
252 | {
253 | "name": "stdout",
254 | "output_type": "stream",
255 | "text": [
256 | "['Sunny', 'Partly Cloudy', 'Mostly Sunny', 'Partly Cloudy', 'Mostly Sunny', 'Partly Cloudy', 'Mostly Sunny', 'Partly Cloudy', 'Mostly Sunny']\n",
257 | "['High: 69 °F', 'Low: 55 °F', 'High: 67 °F', 'Low: 54 °F', 'High: 68 °F', 'Low: 55 °F', 'High: 68 °F', 'Low: 56 °F', 'High: 70 °F']\n",
258 | "['Today: Sunny, with a high near 69. West wind 11 to 18 mph, with gusts as high as 23 mph. ', 'Tonight: Partly cloudy, with a low around 55. West southwest wind 15 to 21 mph, with gusts as high as 25 mph. ', 'Tuesday: Mostly sunny, with a high near 67. West southwest wind 11 to 20 mph, with gusts as high as 24 mph. ', 'Tuesday Night: Partly cloudy, with a low around 54. West southwest wind 16 to 18 mph, with gusts as high as 23 mph. ', 'Wednesday: Mostly sunny, with a high near 68. West southwest wind 7 to 17 mph, with gusts as high as 21 mph. ', 'Wednesday Night: Partly cloudy, with a low around 55.', 'Thursday: Mostly sunny, with a high near 68.', 'Thursday Night: Partly cloudy, with a low around 56.', 'Friday: Mostly sunny, with a high near 70.']\n"
259 | ]
260 | }
261 | ],
262 | "source": [
263 | "short_descs = [sd.get_text() for sd in seven_day.select(\".tombstone-container .short-desc\")]\n",
264 | "temps = [t.get_text() for t in seven_day.select(\".tombstone-container .temp\")]\n",
265 | "descs = [d[\"title\"] for d in seven_day.select(\".tombstone-container img\")]\n",
266 | "\n",
267 | "print(short_descs)\n",
268 | "print(temps)\n",
269 | "print(descs)"
270 | ]
271 | },
272 | {
273 | "cell_type": "markdown",
274 | "metadata": {},
275 | "source": [
276 | "#### Combining our data into a Pandas Dataframe"
277 | ]
278 | },
279 | {
280 | "cell_type": "code",
281 | "execution_count": 13,
282 | "metadata": {},
283 | "outputs": [
284 | {
285 | "data": {
286 | "text/html": [
287 | "\n",
288 | "\n",
301 | "\n",
302 | " \n",
303 | "
\n",
377 | "
"
378 | ],
379 | "text/plain": [
380 | " period short_desc temp \\\n",
381 | "0 Today Sunny High: 69 °F \n",
382 | "1 Tonight Partly Cloudy Low: 55 °F \n",
383 | "2 Tuesday Mostly Sunny High: 67 °F \n",
384 | "3 TuesdayNight Partly Cloudy Low: 54 °F \n",
385 | "4 Wednesday Mostly Sunny High: 68 °F \n",
386 | "5 WednesdayNight Partly Cloudy Low: 55 °F \n",
387 | "6 Thursday Mostly Sunny High: 68 °F \n",
388 | "7 ThursdayNight Partly Cloudy Low: 56 °F \n",
389 | "8 Friday Mostly Sunny High: 70 °F \n",
390 | "\n",
391 | " desc \n",
392 | "0 Today: Sunny, with a high near 69. West wind 1... \n",
393 | "1 Tonight: Partly cloudy, with a low around 55. ... \n",
394 | "2 Tuesday: Mostly sunny, with a high near 67. We... \n",
395 | "3 Tuesday Night: Partly cloudy, with a low aroun... \n",
396 | "4 Wednesday: Mostly sunny, with a high near 68. ... \n",
397 | "5 Wednesday Night: Partly cloudy, with a low aro... \n",
398 | "6 Thursday: Mostly sunny, with a high near 68. \n",
399 | "7 Thursday Night: Partly cloudy, with a low arou... \n",
400 | "8 Friday: Mostly sunny, with a high near 70. "
401 | ]
402 | },
403 | "execution_count": 13,
404 | "metadata": {},
405 | "output_type": "execute_result"
406 | }
407 | ],
408 | "source": [
409 | "import pandas as pd\n",
410 | "weather = pd.DataFrame({\n",
411 | " \"period\": periods, \n",
412 | " \"short_desc\": short_descs, \n",
413 | " \"temp\": temps, \n",
414 | " \"desc\":descs\n",
415 | " })\n",
416 | "weather"
417 | ]
418 | },
419 | {
420 | "cell_type": "markdown",
421 | "metadata": {},
422 | "source": [
423 | "#### Use a regular expression and the Series.str.extract method to pull out the numeric temperature values:"
424 | ]
425 | },
426 | {
427 | "cell_type": "code",
428 | "execution_count": 14,
429 | "metadata": {},
430 | "outputs": [
431 | {
432 | "data": {
433 | "text/plain": [
434 | "0 69\n",
435 | "1 55\n",
436 | "2 67\n",
437 | "3 54\n",
438 | "4 68\n",
439 | "5 55\n",
440 | "6 68\n",
441 | "7 56\n",
442 | "8 70\n",
443 | "Name: temp_num, dtype: object"
444 | ]
445 | },
446 | "execution_count": 14,
447 | "metadata": {},
448 | "output_type": "execute_result"
449 | }
450 | ],
451 | "source": [
452 | "temp_nums = weather[\"temp\"].str.extract(\"(?P| \n", 305 | " | period | \n", 306 | "short_desc | \n", 307 | "temp | \n", 308 | "desc | \n", 309 | "
|---|---|---|---|---|
| 0 | \n", 314 | "Today | \n", 315 | "Sunny | \n", 316 | "High: 69 °F | \n", 317 | "Today: Sunny, with a high near 69. West wind 1... | \n", 318 | "
| 1 | \n", 321 | "Tonight | \n", 322 | "Partly Cloudy | \n", 323 | "Low: 55 °F | \n", 324 | "Tonight: Partly cloudy, with a low around 55. ... | \n", 325 | "
| 2 | \n", 328 | "Tuesday | \n", 329 | "Mostly Sunny | \n", 330 | "High: 67 °F | \n", 331 | "Tuesday: Mostly sunny, with a high near 67. We... | \n", 332 | "
| 3 | \n", 335 | "TuesdayNight | \n", 336 | "Partly Cloudy | \n", 337 | "Low: 54 °F | \n", 338 | "Tuesday Night: Partly cloudy, with a low aroun... | \n", 339 | "
| 4 | \n", 342 | "Wednesday | \n", 343 | "Mostly Sunny | \n", 344 | "High: 68 °F | \n", 345 | "Wednesday: Mostly sunny, with a high near 68. ... | \n", 346 | "
| 5 | \n", 349 | "WednesdayNight | \n", 350 | "Partly Cloudy | \n", 351 | "Low: 55 °F | \n", 352 | "Wednesday Night: Partly cloudy, with a low aro... | \n", 353 | "
| 6 | \n", 356 | "Thursday | \n", 357 | "Mostly Sunny | \n", 358 | "High: 68 °F | \n", 359 | "Thursday: Mostly sunny, with a high near 68. | \n", 360 | "
| 7 | \n", 363 | "ThursdayNight | \n", 364 | "Partly Cloudy | \n", 365 | "Low: 56 °F | \n", 366 | "Thursday Night: Partly cloudy, with a low arou... | \n", 367 | "
| 8 | \n", 370 | "Friday | \n", 371 | "Mostly Sunny | \n", 372 | "High: 70 °F | \n", 373 | "Friday: Mostly sunny, with a high near 70. | \n", 374 | "
\n",
517 | "\n",
530 | "\n",
531 | " \n",
532 | "
\n",
581 | "
"
582 | ],
583 | "text/plain": [
584 | " period short_desc temp \\\n",
585 | "1 Tonight Partly Cloudy Low: 55 °F \n",
586 | "3 TuesdayNight Partly Cloudy Low: 54 °F \n",
587 | "5 WednesdayNight Partly Cloudy Low: 55 °F \n",
588 | "7 ThursdayNight Partly Cloudy Low: 56 °F \n",
589 | "\n",
590 | " desc temp_num is_night \n",
591 | "1 Tonight: Partly cloudy, with a low around 55. ... 55 True \n",
592 | "3 Tuesday Night: Partly cloudy, with a low aroun... 54 True \n",
593 | "5 Wednesday Night: Partly cloudy, with a low aro... 55 True \n",
594 | "7 Thursday Night: Partly cloudy, with a low arou... 56 True "
595 | ]
596 | },
597 | "execution_count": 17,
598 | "metadata": {},
599 | "output_type": "execute_result"
600 | }
601 | ],
602 | "source": [
603 | "weather[is_night]"
604 | ]
605 | },
606 | {
607 | "cell_type": "markdown",
608 | "metadata": {},
609 | "source": [
610 | "## Thank You. To learn more about it Go to https://themenyouwanttobe.wordpress.com "
611 | ]
612 | }
613 | ],
614 | "metadata": {
615 | "kernelspec": {
616 | "display_name": "Python 3",
617 | "language": "python",
618 | "name": "python3"
619 | },
620 | "language_info": {
621 | "codemirror_mode": {
622 | "name": "ipython",
623 | "version": 3
624 | },
625 | "file_extension": ".py",
626 | "mimetype": "text/x-python",
627 | "name": "python",
628 | "nbconvert_exporter": "python",
629 | "pygments_lexer": "ipython3",
630 | "version": "3.6.5"
631 | }
632 | },
633 | "nbformat": 4,
634 | "nbformat_minor": 2
635 | }
636 |
--------------------------------------------------------------------------------
| \n", 534 | " | period | \n", 535 | "short_desc | \n", 536 | "temp | \n", 537 | "desc | \n", 538 | "temp_num | \n", 539 | "is_night | \n", 540 | "
|---|---|---|---|---|---|---|
| 1 | \n", 545 | "Tonight | \n", 546 | "Partly Cloudy | \n", 547 | "Low: 55 °F | \n", 548 | "Tonight: Partly cloudy, with a low around 55. ... | \n", 549 | "55 | \n", 550 | "True | \n", 551 | "
| 3 | \n", 554 | "TuesdayNight | \n", 555 | "Partly Cloudy | \n", 556 | "Low: 54 °F | \n", 557 | "Tuesday Night: Partly cloudy, with a low aroun... | \n", 558 | "54 | \n", 559 | "True | \n", 560 | "
| 5 | \n", 563 | "WednesdayNight | \n", 564 | "Partly Cloudy | \n", 565 | "Low: 55 °F | \n", 566 | "Wednesday Night: Partly cloudy, with a low aro... | \n", 567 | "55 | \n", 568 | "True | \n", 569 | "
| 7 | \n", 572 | "ThursdayNight | \n", 573 | "Partly Cloudy | \n", 574 | "Low: 56 °F | \n", 575 | "Thursday Night: Partly cloudy, with a low arou... | \n", 576 | "56 | \n", 577 | "True | \n", 578 | "