![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
31 | ]
32 | },
33 | {
34 | "cell_type": "markdown",
35 | "source": [
36 | "|[簡報](https://hackmd.io/@wiimax/ict_mqtt#/)|[Publisher](https://colab.research.google.com/drive/1DAYzTMHmv0X0FqIknvelkcKoVjZgCA3W)|[Subscriber](https://colab.research.google.com/drive/1UcUqnFHTx_DhmJd80WTjwcRql2KVt63W)\n",
37 | "|:-:|:-:|:-:\n",
38 | "|||"
39 | ],
40 | "metadata": {
41 | "id": "pw4jPKxPObAz"
42 | }
43 | },
44 | {
45 | "cell_type": "markdown",
46 | "source": [
47 | "## MQTT 實作\n",
48 | "\n",
49 | "- 需要 MQTT中間的**Broker**: 公用的\n",
50 | "- 需要發布訊息的**Publisher**\n",
51 | "- 需要接收訊息的**Subscribe**"
52 | ],
53 | "metadata": {
54 | "id": "OXqP3upSREed"
55 | }
56 | },
57 | {
58 | "cell_type": "markdown",
59 | "source": [
60 | "#### 熱門的公共 MQTT 伺服器：\n",
61 | "\n",
62 | ""
63 | ],
64 | "metadata": {
65 | "id": "cgWSnkP2P8CI"
66 | }
67 | },
68 | {
69 | "cell_type": "markdown",
70 | "source": [
71 | "## Publishe"
72 | ],
73 | "metadata": {
74 | "id": "1etdBS3dRSCz"
75 | }
76 | },
77 | {
78 | "cell_type": "code",
79 | "source": [
80 | "!pip install paho-mqtt"
81 | ],
82 | "metadata": {
83 | "id": "HuvDIuzz-tX2"
84 | },
85 | "execution_count": null,
86 | "outputs": []
87 | },
88 | {
89 | "cell_type": "code",
90 | "source": [
91 | "%%writefile mqtt_pub.py\n",
92 | "# Publishe\n",
93 | "\n",
94 | "import paho.mqtt.client as mqtt\n",
95 | "from random import randint\n",
96 | "import json\n",
97 | "import datetime\n",
98 | "import time\n",
99 | "\n",
100 | "client = mqtt.Client()\n",
101 | "\n",
102 | "# broker = 'broker.hivemq.com'\n",
103 | "# broker = '172.20.10.14'\n",
104 | "broker = 'broker.emqx.io'\n",
105 | "\n",
106 | "client.connect(broker, 1883, keepalive=60)\n",
107 | "\n",
108 | "client.loop_start()\n",
109 | "\n",
110 | "while True:\n",
111 | " topic = \"tcnr/class/n01/sensor01\"\n",
112 | " temp = randint(18,32)\n",
113 | " now_time = datetime.datetime.now().strftime('%m/%d %H:%M:%S')\n",
114 | " message = {'Temperature': temp, 'Time': now_time}\n",
115 | "\n",
116 | " client.publish(topic, json.dumps(message), qos=0)\n",
117 | " print(f'topic: {topic}, message: {message}')\n",
118 | " time.sleep(5)\n"
119 | ],
120 | "metadata": {
121 | "id": "8UOT-Ld1ROty",
122 | "colab": {
123 | "base_uri": "https://localhost:8080/"
124 | },
125 | "outputId": "0c6dcf98-cecb-446c-a561-356a1e3e5a8d"
126 | },
127 | "execution_count": null,
128 | "outputs": [
129 | {
130 | "output_type": "stream",
131 | "name": "stdout",
132 | "text": [
133 | "Writing mqtt_pub.py\n"
134 | ]
135 | }
136 | ]
137 | },
138 | {
139 | "cell_type": "code",
140 | "source": [
141 | "# !python mqtt_pub.py"
142 | ],
143 | "metadata": {
144 | "id": "K_26ARXMkyCo",
145 | "colab": {
146 | "base_uri": "https://localhost:8080/"
147 | },
148 | "outputId": "fa2c1d6c-f39d-4b88-af8d-b001e6bf957b"
149 | },
150 | "execution_count": null,
151 | "outputs": [
152 | {
153 | "output_type": "stream",
154 | "name": "stdout",
155 | "text": [
156 | "topic: tcnr/class/n01/sensor01, message: {'Temperature': 20, 'Time': '03/22 06:35:52'}\n",
157 | "topic: tcnr/class/n01/sensor01, message: {'Temperature': 23, 'Time': '03/22 06:35:57'}\n",
158 | "\n"
159 | ]
160 | }
161 | ]
162 | },
163 | {
164 | "cell_type": "markdown",
165 | "source": [
166 | "### 練習\n",
167 | "- 修改`topic`主題，改成`\"tcnr/class/你的座號/sensor01\"`\n",
168 | "- 修改Broker為hivemq，記得Publisher與Subscribe要同步修改。\n",
169 | "- 修改訊息的QoS為1，意思是Publisher送出會讓Broker確認是否收到1次。\n",
170 | "- 在此筆記本(`*.ipynb`)以`%%writefile mqtt_pub.py`存為`.py`檔案，並且用命令列執行`python`腳本。"
171 | ],
172 | "metadata": {
173 | "id": "U4KaMYREcWrX"
174 | }
175 | },
176 | {
177 | "cell_type": "markdown",
178 | "source": [
179 | "---"
180 | ],
181 | "metadata": {
182 | "id": "-W_109d3USPn"
183 | }
184 | },
185 | {
186 | "cell_type": "markdown",
187 | "source": [
188 | "## Subscribe"
189 | ],
190 | "metadata": {
191 | "id": "Xu73azdMRLz8"
192 | }
193 | },
194 | {
195 | "cell_type": "code",
196 | "execution_count": null,
197 | "metadata": {
198 | "id": "jQiBWVyi-dH9"
199 | },
200 | "outputs": [],
201 | "source": [
202 | "# Subscribe\n",
203 | "\n",
204 | "import paho.mqtt.client as mqtt\n",
205 | "\n",
206 | "def on_connect(client, userdata, flags, rc):\n",
207 | " print(f'connected with reqult code {rc}')\n",
208 | " client.subscribe(\"tcnr/#\")\n",
209 | "\n",
210 | "def on_message(client, userdata, msg):\n",
211 | " print(f\"{msg.topic} {msg.payload}\")\n",
212 | "\n",
213 | "client = mqtt.Client()\n",
214 | "client.on_connect = on_connect\n",
215 | "client.on_message = on_message\n",
216 | "\n",
217 | "broker = 'broker.emqx.io'\n",
218 | "client.connect(broker, 1883, keepalive=60)\n",
219 | "\n",
220 | "client.loop_forever()"
221 | ]
222 | }
223 | ]
224 | }
--------------------------------------------------------------------------------
/MQTT_Demo_Subscriber.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "nbformat": 4,
3 | "nbformat_minor": 0,
4 | "metadata": {
5 | "colab": {
6 | "name": "MQTT_Demo_Subscriber.ipynb",
7 | "provenance": [],
8 | "collapsed_sections": [
9 | "1etdBS3dRSCz"
10 | ],
11 | "authorship_tag": "ABX9TyO80twy5Doqx8BzJ8JzW86v",
12 | "include_colab_link": true
13 | },
14 | "kernelspec": {
15 | "name": "python3",
16 | "display_name": "Python 3"
17 | },
18 | "language_info": {
19 | "name": "python"
20 | }
21 | },
22 | "cells": [
23 | {
24 | "cell_type": "markdown",
25 | "metadata": {
26 | "id": "view-in-github",
27 | "colab_type": "text"
28 | },
29 | "source": [
30 | ""
31 | ]
32 | },
33 | {
34 | "cell_type": "markdown",
35 | "source": [
36 | "|[簡報](https://hackmd.io/@wiimax/ict_mqtt#/)|[Publisher](https://colab.research.google.com/drive/1DAYzTMHmv0X0FqIknvelkcKoVjZgCA3W)|[Subscriber](https://colab.research.google.com/drive/1UcUqnFHTx_DhmJd80WTjwcRql2KVt63W)\n",
37 | "|:-:|:-:|:-:\n",
38 | "|||"
39 | ],
40 | "metadata": {
41 | "id": "pw4jPKxPObAz"
42 | }
43 | },
44 | {
45 | "cell_type": "markdown",
46 | "source": [
47 | "## MQTT 實作\n",
48 | "\n",
49 | "- 需要 MQTT中間的**Broker**: 公用的\n",
50 | "- 需要發布訊息的**Publisher**\n",
51 | "- 需要接收訊息的**Subscribe**"
52 | ],
53 | "metadata": {
54 | "id": "OXqP3upSREed"
55 | }
56 | },
57 | {
58 | "cell_type": "markdown",
59 | "source": [
60 | "#### 熱門的公共 MQTT 伺服器：\n",
61 | "\n",
62 | ""
63 | ],
64 | "metadata": {
65 | "id": "cgWSnkP2P8CI"
66 | }
67 | },
68 | {
69 | "cell_type": "markdown",
70 | "source": [
71 | "## Subscribe"
72 | ],
73 | "metadata": {
74 | "id": "Xu73azdMRLz8"
75 | }
76 | },
77 | {
78 | "cell_type": "code",
79 | "source": [
80 | "!pip install paho-mqtt"
81 | ],
82 | "metadata": {
83 | "id": "HuvDIuzz-tX2"
84 | },
85 | "execution_count": null,
86 | "outputs": []
87 | },
88 | {
89 | "cell_type": "code",
90 | "execution_count": null,
91 | "metadata": {
92 | "id": "jQiBWVyi-dH9"
93 | },
94 | "outputs": [],
95 | "source": [
96 | "# %%writefile mqtt_sub.py\n",
97 | "# Subscribe\n",
98 | "\n",
99 | "import paho.mqtt.client as mqtt\n",
100 | "import json\n",
101 | "\n",
102 | "def on_connect(client, userdata, flags, rc):\n",
103 | " print(f'connected with reqult code {rc}')\n",
104 | " client.subscribe(\"tcnr/#\")\n",
105 | "\n",
106 | "def on_message(client, userdata, msg):\n",
107 | " print(f\"{msg.topic} {msg.payload.decode('utf-8')}\")\n",
108 | " temp = json.loads(msg.payload.decode('utf-8'))[\"Temperature\"]\n",
109 | " if temp >= 30:\n",
110 | " print(f\"溫度偵測{temp}大於30度，該開冷氣啦!\")\n",
111 | "\n",
112 | "client = mqtt.Client()\n",
113 | "client.on_connect = on_connect\n",
114 | "client.on_message = on_message\n",
115 | "\n",
116 | "# broker = 'broker.hivemq.com'\n",
117 | "# broker = '172.20.10.14'\n",
118 | "broker = 'broker.emqx.io'\n",
119 | "client.connect(broker, 1883, keepalive=60)\n",
120 | "\n",
121 | "client.loop_forever()\n"
122 | ]
123 | },
124 | {
125 | "cell_type": "code",
126 | "source": [
127 | "# !python mqtt_sub.py"
128 | ],
129 | "metadata": {
130 | "id": "aFfJ-KZQk888",
131 | "colab": {
132 | "base_uri": "https://localhost:8080/"
133 | },
134 | "outputId": "e50ed6a5-050e-4661-fa53-c1ed3413ad9c"
135 | },
136 | "execution_count": null,
137 | "outputs": [
138 | {
139 | "output_type": "stream",
140 | "name": "stdout",
141 | "text": [
142 | "connected with reqult code 0\n",
143 | "tcnr/class/n13/sensor01 {\"Temperature\": 28, \"Time\": \"03/22 06:35:05\"}\n",
144 | "tcnr/class/n01/sensor01 {\"Temperature\": 26, \"Time\": \"03/22 06:35:06\"}\n",
145 | "tcnr/class/n10/sensor01 {\"Temperature\": 32, \"Time\": \"03/22 06:35:08\"}\n",
146 | "溫度偵測32大於30度，該開冷氣啦!\n",
147 | "tcnr/class/n04/sensor01 {\"Temperature\": 16, \"Time\": \"03/22 06:35:08\"}\n",
148 | "tcnr/class/n13/sensor01 {\"Temperature\": 20, \"Time\": \"03/22 06:35:10\"}\n",
149 | "tcnr/class/n01/sensor01 {\"Temperature\": 31, \"Time\": \"03/22 06:35:11\"}\n",
150 | "溫度偵測31大於30度，該開冷氣啦!\n",
151 | "tcnr/class/n10/sensor01 {\"Temperature\": 30, \"Time\": \"03/22 06:35:13\"}\n",
152 | "溫度偵測30大於30度，該開冷氣啦!\n",
153 | "tcnr/class/n04/sensor01 {\"Temperature\": 25, \"Time\": \"03/22 06:35:13\"}\n",
154 | "tcnr/class/n13/sensor01 {\"Temperature\": 22, \"Time\": \"03/22 06:35:15\"}\n",
155 | "tcnr/class/n01/sensor01 {\"Temperature\": 26, \"Time\": \"03/22 06:35:16\"}\n",
156 | "tcnr/class/n10/sensor01 {\"Temperature\": 22, \"Time\": \"03/22 06:35:18\"}\n",
157 | "tcnr/class/n04/sensor01 {\"Temperature\": 27, \"Time\": \"03/22 06:35:18\"}\n",
158 | "tcnr/class/n13/sensor01 {\"Temperature\": 30, \"Time\": \"03/22 06:35:20\"}\n",
159 | "溫度偵測30大於30度，該開冷氣啦!\n",
160 | "tcnr/class/n01/sensor01 {\"Temperature\": 28, \"Time\": \"03/22 06:35:21\"}\n",
161 | "tcnr/class/n10/sensor01 {\"Temperature\": 19, \"Time\": \"03/22 06:35:23\"}\n",
162 | "tcnr/class/n04/sensor01 {\"Temperature\": 30, \"Time\": \"03/22 06:35:23\"}\n",
163 | "溫度偵測30大於30度，該開冷氣啦!\n",
164 | "tcnr/class/n13/sensor01 {\"Temperature\": 21, \"Time\": \"03/22 06:35:25\"}\n",
165 | "\n"
166 | ]
167 | }
168 | ]
169 | },
170 | {
171 | "cell_type": "markdown",
172 | "source": [
173 | "### 練習\n",
174 | "- 修改Broker為hivemq，記得Publisher與Subscribe要同步修改。\n",
175 | "- 修改`message`處理條件，如果溫度`temp`低於20，列印\"該蓋棉被啦\"。\n",
176 | "- 在此筆記本(`*.ipynb`)以`%%writefile mqtt_sub.py`存為`.py`檔案，並且用命令列執行`python`腳本。\n"
177 | ],
178 | "metadata": {
179 | "id": "6M7F18agd1KS"
180 | }
181 | },
182 | {
183 | "cell_type": "markdown",
184 | "source": [
185 | "## Publishe"
186 | ],
187 | "metadata": {
188 | "id": "1etdBS3dRSCz"
189 | }
190 | },
191 | {
192 | "cell_type": "code",
193 | "source": [
194 | "# Publishe\n",
195 | "\n",
196 | "import paho.mqtt.client as mqtt\n",
197 | "from random import randint\n",
198 | "import time\n",
199 | "\n",
200 | "client = mqtt.Client()\n",
201 | "\n",
202 | "broker = 'broker.emqx.io'\n",
203 | "client.connect(broker, 1883, keepalive=60)\n",
204 | "\n",
205 | "client.loop_start()\n",
206 | "\n",
207 | "while True:\n",
208 | " temp = randint(18,32)\n",
209 | " topic = \"tcnr/class/n01/temp\"\n",
210 | " client.publish(topic, temp)\n",
211 | " print(f'topic: {topic}, temperature: {temp}')\n",
212 | " time.sleep(5)\n"
213 | ],
214 | "metadata": {
215 | "id": "8UOT-Ld1ROty"
216 | },
217 | "execution_count": null,
218 | "outputs": []
219 | },
220 | {
221 | "cell_type": "code",
222 | "source": [],
223 | "metadata": {
224 | "id": "kYzjLOKX-ypj"
225 | },
226 | "execution_count": null,
227 | "outputs": []
228 | }
229 | ]
230 | }
--------------------------------------------------------------------------------
/04.Python資料分析應用-語意分析篇NLP.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "metadata": {},
5 | "cell_type": "markdown",
6 | "source": "# 04.Python資料分析入門-語意分析篇NLP"
7 | },
8 | {
9 | "metadata": {},
10 | "cell_type": "markdown",
11 | "source": "## 04-1 基本語意分析-以模組舉例\n- 以[手把手教你如何用 Python 做情感分析](https://www.itread01.com/articles/1498721884.html)為例"
12 | },
13 | {
14 | "metadata": {},
15 | "cell_type": "markdown",
16 | "source": "### 04-1-1 英文為例"
17 | },
18 | {
19 | "metadata": {
20 | "trusted": true
21 | },
22 | "cell_type": "code",
23 | "source": "#安裝相關套件\n!pip install snownlp\n!pip install -U textblob\n!python -m textblob.download_corpora",
24 | "execution_count": 1,
25 | "outputs": [
26 | {
27 | "output_type": "stream",
28 | "text": "Requirement already satisfied: snownlp in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (0.12.3)\nRequirement already up-to-date: textblob in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (0.15.3)\nRequirement already satisfied, skipping upgrade: nltk>=3.1 in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from textblob) (3.3)\nRequirement already satisfied, skipping upgrade: six in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (from nltk>=3.1->textblob) (1.11.0)\n[nltk_data] Downloading package brown to /home/nbuser/nltk_data...\n[nltk_data] Package brown is already up-to-date!\n[nltk_data] Downloading package punkt to /home/nbuser/nltk_data...\n[nltk_data] Package punkt is already up-to-date!\n[nltk_data] Downloading package wordnet to /home/nbuser/nltk_data...\n[nltk_data] Package wordnet is already up-to-date!\n[nltk_data] Downloading package averaged_perceptron_tagger to\n[nltk_data] /home/nbuser/nltk_data...\n[nltk_data] Package averaged_perceptron_tagger is already up-to-\n[nltk_data] date!\n[nltk_data] Downloading package conll2000 to /home/nbuser/nltk_data...\n[nltk_data] Package conll2000 is already up-to-date!\n[nltk_data] Downloading package movie_reviews to\n[nltk_data] /home/nbuser/nltk_data...\n[nltk_data] Package movie_reviews is already up-to-date!\nFinished.\n",
29 | "name": "stdout"
30 | }
31 | ]
32 | },
33 | {
34 | "metadata": {
35 | "trusted": true
36 | },
37 | "cell_type": "code",
38 | "source": "text = \"I am happy today. I feel sad today.\"",
39 | "execution_count": 2,
40 | "outputs": []
41 | },
42 | {
43 | "metadata": {
44 | "trusted": true
45 | },
46 | "cell_type": "code",
47 | "source": "from textblob import TextBlob\n\nblob = TextBlob(text)\nblob.sentences",
48 | "execution_count": 3,
49 | "outputs": [
50 | {
51 | "output_type": "execute_result",
52 | "execution_count": 3,
53 | "data": {
54 | "text/plain": "[Sentence(\"I am happy today.\"), Sentence(\"I feel sad today.\")]"
55 | },
56 | "metadata": {}
57 | }
58 | ]
59 | },
60 | {
61 | "metadata": {
62 | "trusted": true
63 | },
64 | "cell_type": "code",
65 | "source": "#情感極性的變化範圍是[-1, 1]，-1代表完全負面，1代表完全正面。\nprint(blob.sentences[0].sentiment)\nprint(blob.sentences[1].sentiment)\nprint(blob.sentiment)",
66 | "execution_count": 4,
67 | "outputs": [
68 | {
69 | "output_type": "stream",
70 | "text": "Sentiment(polarity=0.8, subjectivity=1.0)\nSentiment(polarity=-0.5, subjectivity=1.0)\nSentiment(polarity=0.15000000000000002, subjectivity=1.0)\n",
71 | "name": "stdout"
72 | }
73 | ]
74 | },
75 | {
76 | "metadata": {},
77 | "cell_type": "markdown",
78 | "source": "### 04-1-2 中文為例\n- 使用SnowNLP: http://t.cn/8kf1c3p"
79 | },
80 | {
81 | "metadata": {
82 | "trusted": true
83 | },
84 | "cell_type": "code",
85 | "source": "text = u\"我今天很快樂。我今天很憤怒。\" #u指文本的編碼是Unicode",
86 | "execution_count": 5,
87 | "outputs": []
88 | },
89 | {
90 | "metadata": {
91 | "trusted": true
92 | },
93 | "cell_type": "code",
94 | "source": "from snownlp import SnowNLP\n\ns = SnowNLP(text)\ns.sentences",
95 | "execution_count": 6,
96 | "outputs": [
97 | {
98 | "output_type": "execute_result",
99 | "execution_count": 6,
100 | "data": {
101 | "text/plain": "['我今天很快樂', '我今天很憤怒']"
102 | },
103 | "metadata": {}
104 | }
105 | ]
106 | },
107 | {
108 | "metadata": {
109 | "trusted": true
110 | },
111 | "cell_type": "code",
112 | "source": "#SnowNLP的情感分析取值，表達的是“這句話代表正面情感的概率”。\nprint(SnowNLP(s.sentences[0]).sentiments)\nprint(SnowNLP(s.sentences[1]).sentiments)\nprint(s.sentiments)",
113 | "execution_count": 7,
114 | "outputs": [
115 | {
116 | "output_type": "stream",
117 | "text": "0.9268071116367116\n0.1702660762575916\n0.7005413250638438\n",
118 | "name": "stdout"
119 | }
120 | ]
121 | },
122 | {
123 | "metadata": {
124 | "trusted": true
125 | },
126 | "cell_type": "code",
127 | "source": "text = u'''\n台中市雷姓男子去年10月間在西區公益路、大墩路的工地喝酒後，騎車上路，遭警方盤查後，酒測值超標為每公升0.70毫克，他遭警方依公共危險罪嫌送辦，檢方聲請簡易判決處刑，法官判他4月徒刑、得易科罰金，但他今年2月要報到執行時，檢方命他不得易科，需入監服刑，他向台中地院聲明異議，法官審理後認為，雷在2016至2018年間連續3度酒駕，前二次一次是緩起訴、另次則是易科罰金，但此次又被查獲，已是第三次，認為他漠視法律，不矯正難收矯正之效，駁回雷聲明異議，判雷要關。\n判決書指出，雷姓男子（41歲）去年10月間酒後騎車被查獲，酒測值為每公升0.70毫克，台中地院依公共危險罪，簡易判決處刑4月，得易科罰金12萬元，不過雷在今年2月到台中地檢署報到執行時，檢方諭令他不得易科，需入監服刑，他不服向台中地院聲明異議。\n\n雷辯稱，判決書上明寫可易科罰金，為何檢方堅持要讓他去服刑，而且他已離婚是單親家庭，有未成年子女要撫養，家境貧寒是中低收入戶，更是家中主要的經濟支柱，希望法官撤銷檢察官不得易科的執行指揮處分，准予讓他易科罰金。\n\n台中地院法官審理後認為，雖然法律有規定，本刑5年以下，宣告6月以下徒刑者， 得易科罰金，不過但書是「難收矯正之效或難以維持法秩序者，不在此限」，易科罰金的易刑處分，應否准許，依照《刑事訴訟法》第457條規定，由檢察官就是否准予受刑人易科罰金，有無但書情況，由檢方查明認定並指揮執行。\n\n法官指出，雷在2016年酒駕被查獲後，獲緩起訴處分，2017年又被查獲酒駕，當時被判3月徒刑，得易科9萬元，但他2018年又喝酒上路，顯示雷沒有因前兩次遭查獲的前例，而有所警惕，且去年被查獲時，是在喝完啤酒不久就騎車上路，顯然極度漠視法令，其行為對社會法秩序之危害重大，因此認定檢方的處分無不妥，駁回雷聲明異議。\n'''",
128 | "execution_count": 8,
129 | "outputs": []
130 | },
131 | {
132 | "metadata": {
133 | "trusted": true
134 | },
135 | "cell_type": "code",
136 | "source": "s = SnowNLP(text)\n#s.sentences[0]\ns.sentiments",
137 | "execution_count": 9,
138 | "outputs": [
139 | {
140 | "output_type": "execute_result",
141 | "execution_count": 9,
142 | "data": {
143 | "text/plain": "0.0"
144 | },
145 | "metadata": {}
146 | }
147 | ]
148 | },
149 | {
150 | "metadata": {},
151 | "cell_type": "markdown",
152 | "source": "## 04-2 基本語意分析-以PTT電影版透過機械學習方式舉例\n- 以下來自[Python 網路爬蟲與資料分析入門實戰 ](https://www.tenlong.com.tw/products/9789864343386)\n- 2019年10月出版的新書，裡面很多台灣在地的爬蟲應用教學，[github](https://github.com/willismax/py-scraping-analysis-book)也有該書的程式碼，可以先試試看。\n- 此例為以PTT電影版關鍵字輸入影片名稱做舉例，以「好雷、負雷」做分類，以機械學習方式，將各文章內文詞斷詞，並預測分類結果"
153 | },
154 | {
155 | "metadata": {
156 | "trusted": true
157 | },
158 | "cell_type": "code",
159 | "source": "import requests\nimport re\nimport csv\nfrom bs4 import BeautifulSoup\n\n\nPTT_URL = 'https://www.ptt.cc'\n\n\ndef get_articles(url):\n resp = requests.get(\n url=url,\n cookies={'over18': '1'} # 告知 Server 已回答過滿 18 歲的問題\n )\n soup = BeautifulSoup(resp.text, 'html5lib')\n prev_link = soup.find('div', 'btn-group-paging').find_all('a')[1]\n # 若 有 href 屬性, 代表有上一頁的超連結\n prev_link = prev_link['href'] if 'href' in prev_link.attrs else None\n\n # 巡覽每一篇文章所在區塊\n positive = []\n negative = []\n for div in soup.find_all('div', 'r-ent'):\n href = div.find('div', 'title').a['href']\n title = div.find('div', 'title').text.strip()\n # 若標題為 [] 開頭, e.g., [好雷] 星際大戰八-各種元素集於一身\n if re.match('\\[.*\\]', title):\n tag = re.match('\\[.*\\]', title).group(0)\n # 標籤內含'好'為好評; 含'負'或'爛'為負評\n if '好' in tag:\n positive.append([title, href])\n if '爛' in tag or '負' in tag:\n negative.append([title, href])\n return prev_link, positive, negative\n\n\nif __name__ == '__main__':\n start_url = PTT_URL + '/bbs/movie/search?q=黑豹'\n positive_posts, negative_posts = [], []\n prev_link, pos, neg = get_articles(start_url)\n positive_posts += pos\n negative_posts += neg\n while prev_link:\n url = PTT_URL + prev_link\n prev_link, pos, neg = get_articles(url)\n positive_posts += pos\n negative_posts += neg\n print(len(positive_posts), positive_posts[:3])\n print(len(negative_posts), negative_posts[:3])\n\n with open('./data/mov_pos.csv', 'w', encoding='utf-8', newline='') as f:\n writer = csv.writer(f)\n writer.writerow(['title', 'href'])\n writer.writerows(positive_posts)\n\n with open('./data/mov_neg.csv', 'w', encoding='utf-8', newline='') as f:\n writer = csv.writer(f)\n writer.writerow(['title', 'href'])\n writer.writerows(negative_posts)",
160 | "execution_count": 10,
161 | "outputs": [
162 | {
163 | "output_type": "stream",
164 | "text": "60 [['[好雷] 黑豹 --其實蠻好看的', '/bbs/movie/M.1552276420.A.F6F.html'], ['[好雷?] 黑豹 自慰片的新高度', '/bbs/movie/M.1545317279.A.EFC.html'], ['[二刷好雷] 水行俠真的不是黑豹', '/bbs/movie/M.1545065816.A.46A.html']]\n27 [['[負雷]黑豹-失衡的烏托邦', '/bbs/movie/M.1529245622.A.AF5.html'], ['[負雷] 四不像的黑豹', '/bbs/movie/M.1527918611.A.56C.html'], ['[微負雷] 黑豹有點讓人失望....', '/bbs/movie/M.1527839684.A.EF0.html']]\n",
165 | "name": "stdout"
166 | }
167 | ]
168 | },
169 | {
170 | "metadata": {
171 | "trusted": true
172 | },
173 | "cell_type": "code",
174 | "source": "import csv\nimport requests\nimport re\nimport json\nimport time\nfrom bs4 import BeautifulSoup\n\n\nPTT_URL = 'https://www.ptt.cc'\n\n\ndef sanitize(txt):\n # 保留英數字, 中文 (\\u4e00-\\u9fa5) 及中文標點, 部分特殊符號\n # http://ubuntu-rubyonrails.blogspot.com/2009/06/unicode.html\n expr = re.compile('[^\\u4e00-\\u9fa5。；，：“”（）、？「」『』【】\\s\\w:/\\-.()]') # ^ 表示\"非括號內指定的字元\"\n txt = re.sub(expr, '', txt)\n txt = re.sub('[。；，：“”（）、？「」『』【】:/\\-_.()]', ' ', txt) # 用空白取代中英文標點\n txt = re.sub('(\\s)+', ' ', txt) # 用單一空白取代多個換行或 tab 符號\n txt = txt.replace('--', '')\n txt = txt.lower() # 英文字轉為小寫\n return txt\n\n\ndef get_post(url):\n resp = requests.get(\n url=url,\n cookies={'over18': '1'} # 告知 Server 已回答過滿 18 歲的問題\n )\n soup = BeautifulSoup(resp.text, 'html5lib')\n main_content = soup.find('div', id='main-content')\n\n # 把非本文的部份 (標題區及推文區) 移除\n # 移除標題區塊\n for meta in main_content.find_all('div', 'article-metaline'):\n meta.extract()\n for meta in main_content.find_all('div', 'article-metaline-right'):\n meta.extract()\n # 移除推文區塊\n for push in main_content.find_all('div', 'push'):\n push.extract()\n\n parsed = []\n for txt in main_content.stripped_strings:\n # 移除 '※ 發信站:', '--' 開頭, 及本文區最後一行文章網址部份\n if txt[0] == '※' or txt[:2] == '--' or url in txt:\n continue\n txt = sanitize(txt)\n if txt:\n parsed.append(txt)\n return ' '.join(parsed)\n\n\ndef get_article_body(csv_file):\n id_to_body = {}\n with open(csv_file, 'r', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n print('處理', row['title'], row['href'])\n title = ' '.join(row['title'].split(']')[1:])\n title = sanitize(title)\n body = get_post(PTT_URL + row['href'])\n id_to_body[row['href']] = title + ' ' + body # 以文章超連結為 key, 標題 + 本文為 value\n time.sleep(1) # 放慢爬蟲速度\n return id_to_body\n\n\nif __name__ == '__main__':\n d1 = get_article_body('./data/mov_pos.csv')\n d2 = get_article_body('./data/mov_neg.csv')\n id_to_body = {**d1, **d2} # 將兩個 dict 合併為一個\n with open('./data/id_to_body.json', 'w', encoding='utf-8') as f:\n json.dump(id_to_body, f, indent=2, ensure_ascii=False)",
175 | "execution_count": 11,
176 | "outputs": [
177 | {
178 | "output_type": "stream",
179 | "text": "處理 [好雷] 黑豹 --其實蠻好看的 /bbs/movie/M.1552276420.A.F6F.html\n處理 [好雷?] 黑豹 自慰片的新高度 /bbs/movie/M.1545317279.A.EFC.html\n處理 [二刷好雷] 水行俠真的不是黑豹 /bbs/movie/M.1545065816.A.46A.html\n處理 [好雷] 盲點：關於《黑豹》的奧克蘭也關於你我的故事 /bbs/movie/M.1538060300.A.8FD.html\n處理 [好雷] 瘋狂亞洲富豪─絕不是新加坡黑豹 /bbs/movie/M.1536676591.A.C8F.html\n處理 [微好雷]《雷神索爾3諸神黃昏》＆《黑豹》 /bbs/movie/M.1536217925.A.B6E.html\n處理 [好雷] 比黑豹好看的蟻人與黃蜂女 /bbs/movie/M.1531217457.A.6C8.html\n處理 [好雷]黑豹 — 符合台灣政治的隱喻分析 /bbs/movie/M.1525369009.A.CFA.html\n處理 [好雷] 黑豹：一部政治寓言 /bbs/movie/M.1521128014.A.19B.html\n處理 [好雷]黑豹，好看但可惜的反派 /bbs/movie/M.1520467155.A.8E5.html\n處理 [好雷] 黑豹心得 /bbs/movie/M.1519920385.A.14A.html\n處理 [ 好雷] 唯一缺憾的黑豹 /bbs/movie/M.1519908058.A.B99.html\n處理 [好雷] 黑豹 感想 微負評 /bbs/movie/M.1519841715.A.3BF.html\n處理 [核心好雷] 黑豹-科技與部落的反差萌 人權與人道議題 /bbs/movie/M.1519812809.A.DC3.html\n處理 [普好雷] 立體的世界，被一掌拍平，淺談【黑豹】 /bbs/movie/M.1519787493.A.BFD.html\n處理 [有意見好雷] 比上沒得比，比下有餘的黑豹 /bbs/movie/M.1519734682.A.979.html\n處理 [好雷] 黑豹 ~ 屬於黑人的童話故事 /bbs/movie/M.1519618436.A.EA0.html\n處理 [ 好 雷] 黑豹 /bbs/movie/M.1519572758.A.D3C.html\n處理 [好雷]黑豹 王者的抉擇 /bbs/movie/M.1519544126.A.E20.html\n處理 [好雷] 黑豹就真的很好看咩~ /bbs/movie/M.1519356550.A.A17.html\n處理 [ 普好雷] 黑豹 /bbs/movie/M.1519142286.A.7CD.html\n處理 [好雷] 黑豹 /bbs/movie/M.1519070229.A.DE9.html\n處理 [好雷] 黑豹：王者路大不易 /bbs/movie/M.1519042966.A.803.html\n處理 [微好雷]《黑豹》 春秋五霸的基本套路 /bbs/movie/M.1518991149.A.BDD.html\n處理 [好雷]黑豹 不太一樣的超級英雄 /bbs/movie/M.1518963410.A.A2E.html\n處理 [好雷] 黑豹 其實我比較喜歡反派@@ /bbs/movie/M.1518939706.A.D48.html\n處理 [好雷] 黑豹 Black Panther，非洲未來主義 /bbs/movie/M.1518893529.A.3AD.html\n處理 [好雷] 黑豹： 智者造橋，愚者築牆 /bbs/movie/M.1518860486.A.B45.html\n處理 [微好雷] 黑豹的電影配樂 /bbs/movie/M.1518848376.A.BBB.html\n處理 [好無雷] 黑豹 /bbs/movie/M.1518802967.A.D32.html\n處理 [好雷] 黑豹 沒有大家說的那麼糟啦 /bbs/movie/M.1518793606.A.036.html\n處理 [好雷] 黑豹 /bbs/movie/M.1518778797.A.C52.html\n處理 [普好雷] 黑豹:Do you know da way? /bbs/movie/M.1518769142.A.229.html\n處理 [普好雷] 黑豹 來談談角色動機吧 /bbs/movie/M.1518703695.A.0DC.html\n處理 [好雷]YOU SHOW OFF?——《黑豹》無&有雷推薦 /bbs/movie/M.1518641125.A.57A.html\n處理 [普好雷] 覺得黑豹的編劇陷入一種兩難的情況 /bbs/movie/M.1518632731.A.55E.html\n處理 [好笑雷] 黑豹 /bbs/movie/M.1518627430.A.ADF.html\n處理 [好雷] 黑豹 耳目一新的英雄電影 /bbs/movie/M.1518617014.A.223.html\n處理 [好雷] 黑豹 最沒有在看爽的漫威片 /bbs/movie/M.1518584186.A.CA8.html\n處理 [偏好雷] 看完黑豹我再重看一次預告片發現... /bbs/movie/M.1518541335.A.C93.html\n處理 [好雷] 黑豹 國王成長日記 /bbs/movie/M.1518538495.A.43A.html\n處理 [好雷] 黑豹 世界觀科技感超讚 但故事有些許bug /bbs/movie/M.1518537228.A.F8C.html\n處理 [超爆幹好雷] 黑豹-名副其實漫威最佳 /bbs/movie/M.1518536561.A.C1C.html\n處理 [普好雷] 黑豹 /bbs/movie/M.1518535690.A.7CD.html\n處理 [好雷]【黑豹】非洲先進文明卓然而立 /bbs/movie/M.1518521887.A.244.html\n處理 [普好雷] 充滿新風格的黑豹 /bbs/movie/M.1518517571.A.DD1.html\n處理 [好雷] 不太懂有關黑豹反派的幾件事情 /bbs/movie/M.1518512928.A.747.html\n處理 [好雷] 小細節怪怪的黑豹 /bbs/movie/M.1518511893.A.636.html\n處理 [好雷] 黑豹 /bbs/movie/M.1518507928.A.ABD.html\n處理 [好無雷] 黑豹 /bbs/movie/M.1518505340.A.C9A.html\n處理 [好雷] 黑豹觀後討論 /bbs/movie/M.1518501262.A.9AC.html\n處理 [普好雷] 黑豹 優秀的漫威宇宙擴展片 /bbs/movie/M.1518500618.A.932.html\n處理 [極好無雷] 這就是我要的黑豹！ /bbs/movie/M.1518499054.A.561.html\n處理 [ 好無雷] 黑豹 絕對值得一看 /bbs/movie/M.1518496440.A.BE1.html\n處理 [好無雷] 大推推黑豹電影 /bbs/movie/M.1518495524.A.882.html\n處理 [好雷] 黑豹，劇情普通，美術太完美 /bbs/movie/M.1518495424.A.F0B.html\n處理 [好雷] 黑豹 /bbs/movie/M.1518494060.A.826.html\n處理 [ 好雷] 美國隊長3 黑豹疑問 /bbs/movie/M.1462012717.A.D23.html\n處理 [好雷]美隊3-關於黑豹以及各英雄的立場 /bbs/movie/M.1461921066.A.E77.html\n處理 [好雷] 美國隊長3 黑豹家世 /bbs/movie/M.1461748087.A.515.html\n處理 [負雷]黑豹-失衡的烏托邦 /bbs/movie/M.1529245622.A.AF5.html\n處理 [負雷] 四不像的黑豹 /bbs/movie/M.1527918611.A.56C.html\n處理 [微負雷] 黑豹有點讓人失望.... /bbs/movie/M.1527839684.A.EF0.html\n處理 [負雷] 是不是有精神分裂的黑豹 /bbs/movie/M.1526734538.A.5E2.html\n處理 [負雷] 黑豹 /bbs/movie/M.1520322497.A.E24.html\n處理 [負雷]黑豹 最差漫威電影 /bbs/movie/M.1519931938.A.207.html\n處理 [ 微負雷] 黑豹 /bbs/movie/M.1519721961.A.471.html\n處理 [負雷] 黑豹 Black Panther，暴力與分享 /bbs/movie/M.1518961786.A.79E.html\n處理 [ 負雷] 隨便拍拍隨便買單的「黑豹」 /bbs/movie/M.1518934190.A.F7B.html\n處理 [負雷] 只有政治正確的 黑豹 /bbs/movie/M.1518809750.A.4E0.html\n處理 [負雷]黑豹：除了美術視覺，其餘毫無魅力 /bbs/movie/M.1518808279.A.FC9.html\n處理 [睡負無雷] 看黑豹看到睡著 /bbs/movie/M.1518774686.A.889.html\n處理 [小負雷] 黑豹-請修邊幅可以嗎?(內有雷) /bbs/movie/M.1518761310.A.05C.html\n處理 [爛無雷] 千萬不要期待的黑豹 /bbs/movie/M.1518758578.A.8F0.html\n處理 [普負雷] 輸不起的黑豹 /bbs/movie/M.1518714311.A.957.html\n處理 [ 負雷] 黑豹-鋪陳復3的跳板 /bbs/movie/M.1518684255.A.489.html\n處理 [ 負雷] 黑豹 /bbs/movie/M.1518670825.A.A31.html\n處理 [普負雷] 黑豹出戲點 /bbs/movie/M.1518642702.A.497.html\n處理 [負雷] 黑豹特遣隊 /bbs/movie/M.1518609915.A.835.html\n處理 [負雷] 家天下黑豹 /bbs/movie/M.1518596627.A.3E0.html\n處理 [負雷] 失望的黑豹(補個優點) /bbs/movie/M.1518577527.A.4BB.html\n處理 [負雷] 真的過譽的黑豹......... /bbs/movie/M.1518536086.A.B1E.html\n處理 [負雷] 黑豹天下 /bbs/movie/M.1518534403.A.ECB.html\n處理 [ 負雷] 讓人失望的黑豹 /bbs/movie/M.1518529744.A.586.html\n處理 [普負雷] 一片歐罵罵的黑豹 /bbs/movie/M.1518513719.A.462.html\n處理 [微負雷] 黑豹 相較其他marvel片有點可惜 /bbs/movie/M.1518510319.A.1DC.html\n處理 [負雷]黑豹 /bbs/movie/M.1518495315.A.E27.html\n",
180 | "name": "stdout"
181 | }
182 | ]
183 | },
184 | {
185 | "metadata": {
186 | "trusted": true
187 | },
188 | "cell_type": "code",
189 | "source": "!pip3 install jieba",
190 | "execution_count": 12,
191 | "outputs": [
192 | {
193 | "output_type": "stream",
194 | "text": "Requirement already satisfied: jieba in /home/nbuser/anaconda3_501/lib/python3.6/site-packages (0.39)\r\n",
195 | "name": "stdout"
196 | }
197 | ]
198 | },
199 | {
200 | "metadata": {
201 | "trusted": true
202 | },
203 | "cell_type": "code",
204 | "source": "import json\nimport csv\nimport jieba\nimport random\nfrom sklearn.feature_extraction.text import CountVectorizer\nfrom sklearn.feature_extraction.text import TfidfTransformer\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.metrics import accuracy_score\n\njieba.set_dictionary('./data/dict.txt.big') # 對繁體中文斷詞較準確的字典檔\n\n\ndef load_data(a_csv, b_json, label):\n a_ids = []\n with open(a_csv, 'r', encoding='utf-8') as f:\n reader = csv.DictReader(f)\n for row in reader:\n a_ids.append(row['href'])\n with open(b_json, 'r', encoding='utf-8') as f:\n id_to_body = json.load(f)\n data = []\n for a_id in a_ids:\n tokenized_post = []\n txt = id_to_body[a_id]\n for sent in txt.split(): # 將文章以空白隔開\n # 斷詞後的結果, 若非空白且長度為 2 以上, 則列入詞庫\n filtered = [t for t in jieba.cut(sent) if t.split() and len(t) > 1]\n tokenized_post += filtered\n data.append((tokenized_post, label))\n return data\n\n\nif __name__ == '__main__':\n pos_data = load_data('./data/mov_pos.csv', './data/id_to_body.json', '正評')\n neg_data = load_data('./data/mov_neg.csv', './data/id_to_body.json', '負評')\n\n '''\n # 印出正評與負評文章的前幾個字, 確認資料無誤\n for post, label in pos_data[:3]:\n print(post[:5], label)\n for post, label in neg_data[:3]:\n print(post[:5], label)\n '''\n\n # 打亂資料順序\n random.seed(42)\n random.shuffle(pos_data)\n random.shuffle(neg_data)\n\n x_train, y_train, x_test, y_test = [], [], [], []\n # 前 22 筆資料 (及答案) 放入 training set\n # 建立資料時要用空白將斷好的詞重新連成一個字串, 以便之後使用 scikit-learn 建立字典並轉換文字資料為向量\n for i in range(10):\n x_train.append(' '.join(pos_data[i][0]))\n x_train.append(' '.join(neg_data[i][0]))\n y_train.append(pos_data[i][1])\n y_train.append(neg_data[i][1])\n # 最後 5 筆資料 (及答案) 放入 testing set\n# for i in range(5, len(pos_data)):\n for i in range(10, 27):\n x_test.append(' '.join(pos_data[i][0]))\n x_test.append(' '.join(neg_data[i][0]))\n y_test.append(pos_data[i][1])\n y_test.append(neg_data[i][1])\n\n vectorizer = CountVectorizer()\n x_train = vectorizer.fit_transform(x_train)\n transformer = TfidfTransformer()\n x_train = transformer.fit_transform(x_train)\n clf = SGDClassifier(random_state=42)\n clf.fit(x_train, y_train)\n x_test = vectorizer.transform(x_test)\n x_test = transformer.transform(x_test)\n y_pred = clf.predict(x_test)\n print('預測結果:', list(y_pred))\n print('正確答案:', y_test)\n print('正確率:', accuracy_score(y_test, y_pred))\n\n # 測試自己輸入的句子\n sentences = [\n '我 覺得 這部 電影 還 不錯',\n '這部 片 應該 可以 更好 才對'\n ]\n analyze = vectorizer.build_analyzer()\n print(analyze(sentences[0]))\n print(analyze(sentences[1]))\n\n custom_data = transformer.transform(vectorizer.transform(sentences))\n print(clf.predict(custom_data))",
205 | "execution_count": 14,
206 | "outputs": [
207 | {
208 | "output_type": "stream",
209 | "text": "Building prefix dict from /home/nbuser/library/lesson/data/dict.txt.big ...\nLoading model from cache /tmp/jieba.u863534f77a5b7aa5dc55e7aac03546ba.cache\nLoading model cost 7.137 seconds.\nPrefix dict has been built succesfully.\n",
210 | "name": "stderr"
211 | },
212 | {
213 | "output_type": "stream",
214 | "text": "預測結果: ['正評', '負評', '正評', '正評', '負評', '負評', '負評', '正評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '正評', '正評', '負評', '負評', '負評', '負評', '正評', '正評', '負評', '正評', '正評', '正評', '負評', '負評', '負評', '負評', '負評', '正評', '正評']\n正確答案: ['正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評', '正評', '負評']\n正確率: 0.6470588235294118\n['覺得', '這部', '電影', '不錯']\n['這部', '應該', '可以', '更好', '才對']\n['負評' '正評']\n",
215 | "name": "stdout"
216 | },
217 | {
218 | "output_type": "stream",
219 | "text": "/home/nbuser/anaconda3_501/lib/python3.6/site-packages/sklearn/linear_model/stochastic_gradient.py:128: FutureWarning: max_iter and tol parameters have been added in `|段落\n","``|超連結\n","\\ target \n"," This domain is for use in illustrative examples in documents. You may use this\n"," domain in literature without prior coordination or asking for permission.\n"," \n"," \n"," More information...\n"," \n"," This domain is for use in illustrative examples in documents. You may use this\n"," domain in literature without prior coordination or asking for permission. This domain is for use in illustrative examples in documents. You may use this\n"," domain in literature without prior coordination or asking for permission. This domain is for use in illustrative examples in documents. You may use this\n"," domain in literature without prior coordination or asking for permission. This domain is for use in illustrative examples in documents. You may use this\n"," domain in literature without prior coordination or asking for permission. This domain is for use in illustrative examples in documents. You may use this\n"," domain in literature without prior coordination or asking for permission.
|換行(無結束標籤)"]},{"cell_type":"markdown","metadata":{"id":"1MDh03CGhhdg","colab_type":"text"},"source":["#### 常用屬性(Attributes)\n","\n","|屬性名稱|用途|\n","-|-\n","class|標籤的類別(可重複)\n","id|標籤的id(不可重複)\n","title|標籤的顯示資訊\n","style|標籤的樣式\n","data-*|自行定義的屬性"]},{"cell_type":"markdown","metadata":{"id":"AiAe7hnEjftP","colab_type":"text"},"source":["### 擷取網頁必要知識"]},{"cell_type":"markdown","metadata":{"id":"G9fT8Fykjhi0","colab_type":"text"},"source":["- 在HTTP協定中，定義了多種不同的method做為服務的請求方法，近年來由於行動裝置的普及化，越來越多的產品及網站都提供了WebAPI服務，既然我們要擷取網頁內容，就必須知道對HTTP請求方式。\n","- 在HTTP 1.1的版本中定義了八種 Method (方法)，如下所示：\n"," - OPTIONS\n"," - **GET**\n"," - HEAD\n"," - **POST**\n"," - PUT\n"," - DELETE\n"," - TRACE\n"," - CONNECT"]},{"cell_type":"markdown","metadata":{"id":"ckztGOzfjlSG","colab_type":"text"},"source":["- 最常見的method為以下5種:\n","\n","|method|意義|\n","|-|- |\n","|GET|取得(想要的服務)的資料或是狀態。|\n","|POST|如同填表般的行為，以新增一項資料。\n","|PUT|利用更新的方式於\"指定位置\"新增一項資料。\n","|PATCH|在現有的資料欄位中，增加或部分更新一筆新的資料。\n","|DELETE|刪除指定資料。"]},{"cell_type":"markdown","metadata":{"id":"6S1aqUi_jquX","colab_type":"text"},"source":["- 更進一步了解請參閱W3C制定規範[RFC 5789](https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.5)\n","- [淺談 HTTP Method：表單中的 GET 與 POST 有什麼差別？](https://blog.toright.com/posts/1203/%E6%B7%BA%E8%AB%87-http-method%EF%BC%9A%E8%A1%A8%E5%96%AE%E4%B8%AD%E7%9A%84-get-%E8%88%87-post-%E6%9C%89%E4%BB%80%E9%BA%BC%E5%B7%AE%E5%88%A5%EF%BC%9F.html)"]},{"cell_type":"markdown","metadata":{"id":"mxDpTI-8RfSb","colab_type":"text"},"source":["- 另外在網頁與資料庫的操作過程中，也會經常聽到CRUD這個詞，CRUD是指 新增(Create)、讀取(Read)、更新(Update)、刪除(Delete)的主要4個操作資料庫(如MySQL等)常用的功能\n","\n","- 參閱[Day 15 - 實作第一個 CRUD 之 Create、Update、Delete](https://ithelp.ithome.com.tw/articles/10206716)"]},{"cell_type":"markdown","metadata":{"id":"yGUfAF1GjsBF","colab_type":"text"},"source":["### 淺談Restful API\n"]},{"cell_type":"markdown","metadata":{"id":"btWqwWF_juHm","colab_type":"text"},"source":["- REST全名 Resource Representational State Transfer，可譯為具象狀態傳輸，其核心精神在於借用 HTTP 協定做為基礎，讓API規格簡單一致:\n"," - Resource：資源。\n"," - Representational：表現形式，如JSON，XML．．．\n"," - State Transfer：狀態變化。即上述講到的可利用HTTP動詞們來做呼叫。"]},{"cell_type":"markdown","metadata":{"id":"PigVvHPgjwfe","colab_type":"text"},"source":["- REST指的是網路中Client端和Server端的一種呼叫服務形式，透過既定的規則，滿足約束條件和原則的應用程式設計，對資源的操作包括獲取、創建、修改和刪除資源，可對應資料庫基本操作：新增、讀取、更新、刪除(Create、Read、Update、Delete, **CRUD**)。"]},{"cell_type":"markdown","metadata":{"id":"z9v2bBUVjzsZ","colab_type":"text"},"source":["- 舉例商品WebAPI的interface：\n"," - 獲得商品資料 GET /getItem/9527\n"," - 新增商品資料 POST /createItem\n"," - 更新商品資料 POST /updateItem/\n"," - 刪除商品資料 POST /deleteItem/\n","\n","- 運用RESTful API 開發的WebAPI的interface:\n"," - 獲取商品資料 /GET/items/9527\n"," - 新增商品資料 /POST/items\n"," - 更新商品資料 /PATCH/items/9527\n"," - 刪除商品資料 /DELETE/items/9527\n","\n","- 即便有些離題，但增加網頁常識對蒐集真實世界資料總有助益。"]},{"cell_type":"markdown","metadata":{"id":"TBsilSkyj7t3","colab_type":"text"},"source":["- 延伸閱讀\n"," - [[不是工程師] 休息(REST)式架構- 寧靜式(RESTful)的Web API是現在的潮流？](https://progressbar.tw/posts/53)\n"," >當Web service使用Web API進行介面介接時，每一串我們設計的URL，就會是一個專屬的服務『窗口』。\n"," - [RESTful API 設計準則與實務經驗](https://blog.toright.com/posts/5523/restful-api-%E8%A8%AD%E8%A8%88%E6%BA%96%E5%89%87%E8%88%87%E5%AF%A6%E5%8B%99%E7%B6%93%E9%A9%97.html)"]},{"cell_type":"markdown","metadata":{"id":"uZP7CHTATafy","colab_type":"text"},"source":["### 什麼是網路爬蟲(Web Crawler)"]},{"cell_type":"markdown","metadata":{"id":"sOh4tWNniCUa","colab_type":"text"},"source":["\n","https://blog.apify.com/what-is-web-scraping-1b548f8d6ac1\n"]},{"cell_type":"markdown","metadata":{"id":"TTuweXJRT23m","colab_type":"text"},"source":["- 網路爬蟲像是機器人，自動化的幫你擷取目標資訊\n","- 爬蟲無所不在，谷哥(度娘?)都是"]},{"cell_type":"markdown","metadata":{"id":"gRUO9OXPVT0E","colab_type":"text"},"source":["- 爬蟲應用?\n"," - 熱門遊戲評論、輿論分析系統、銷售分析、旅遊訂票...\n"]},{"cell_type":"markdown","metadata":{"id":"K0NoiLeiiANu","colab_type":"text"},"source":["- 再看一次網頁元素結構\n"," - `\n"," \n"," \n","\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Bgtv3fFn6AD2","colab_type":"text"},"source":["#### BeautifulSoup的常用函數\n","- `soup.find()` 找一個標籤 tag\n","- 回傳第一個被tag包圍的區塊\n","- 傳入的引數第一個通常是 tag 名稱，第二個引數若未指明屬性就代表 class 名稱，也可以直接使用 id 等屬性去定位區塊。定位到區塊後，可以取出其屬性與包含的字串值"]},{"cell_type":"code","metadata":{"id":"-F3nFNZF6XUA","colab_type":"code","colab":{}},"source":["?soup.find()\n","#soup.find(name=None, attrs={}, recursive=True, text=None, **kwargs)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"9oNQZZcyknE0","colab_type":"code","outputId":"6dc630c9-5624-4433-daec-a082b7644c62","executionInfo":{"status":"ok","timestamp":1572073713068,"user_tz":-480,"elapsed":605,"user":{"displayName":"陳宇威","photoUrl":"","userId":"00090422027206355406"}},"colab":{"base_uri":"https://localhost:8080/","height":50}},"source":["soup.find('p')"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["\n"," Example Domain\n","
\n"," \n",", Example Domain
\n","\n","[無言] 誰會委託同事買餐巾紙?\n",", \n","Fw: [問卦] 鮭魚握壽司被蒸熟了該怎麼辦？\n",", \n","[無言] 牙線棒卡在牙套上\n",", \n","[公告] 笨板板規\n",", \n","[公告]本板即日起不可PO問卷文\n",", \n","[公告] 10月份置底閒聊文\n","]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"vRrphsEiq8YP","colab_type":"code","outputId":"2f7228db-538b-4511-e508-fc75d4d5ec4f","executionInfo":{"status":"ok","timestamp":1572075593395,"user_tz":-480,"elapsed":757,"user":{"displayName":"陳宇威","photoUrl":"","userId":"00090422027206355406"}},"colab":{"base_uri":"https://localhost:8080/","height":145}},"source":["article_href = soup.select(\"div.title a\")\n","article_href"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[[無言] 誰會委託同事買餐巾紙?,\n"," Fw: [問卦] 鮭魚握壽司被蒸熟了該怎麼辦？,\n"," [無言] 牙線棒卡在牙套上,\n"," [公告] 笨板板規,\n"," [公告]本板即日起不可PO問卷文,\n"," [公告] 10月份置底閒聊文]"]},"metadata":{"tags":[]},"execution_count":51}]},{"cell_type":"code","metadata":{"id":"zKnxpxaorHjr","colab_type":"code","outputId":"34acdd84-d21c-4a2e-cfe7-c8b4321d50d7","executionInfo":{"status":"ok","timestamp":1572075835147,"user_tz":-480,"elapsed":6039,"user":{"displayName":"陳宇威","photoUrl":"","userId":"00090422027206355406"}},"colab":{"base_uri":"https://localhost:8080/","height":319}},"source":["# 逐一取出標題、合併超連結\n","for a in article_href:\n"," print('title:', a.text)\n"," print('href:','https://www.ptt.cc'+a['href'])\n","\n"," #打開連結內的網頁並另存\n"," content_url = 'https://www.ptt.cc'+ a['href']\n"," r = requests.get(content_url)\n"," with open ( a.text + '.html', 'w+') as f:\n"," f.write(r.text)\n"," print('saved')"],"execution_count":0,"outputs":[{"output_type":"stream","text":["title: [無言] 誰會委託同事買餐巾紙?\n","href: https://www.ptt.cc/bbs/StupidClown/M.1572014689.A.F9B.html\n","saved\n","title: Fw: [問卦] 鮭魚握壽司被蒸熟了該怎麼辦？\n","href: https://www.ptt.cc/bbs/StupidClown/M.1572064384.A.96C.html\n","saved\n","title: [無言] 牙線棒卡在牙套上\n","href: https://www.ptt.cc/bbs/StupidClown/M.1572068103.A.43A.html\n","saved\n","title: [公告] 笨板板規\n","href: https://www.ptt.cc/bbs/StupidClown/M.1158735717.A.828.html\n","saved\n","title: [公告]本板即日起不可PO問卷文\n","href: https://www.ptt.cc/bbs/StupidClown/M.1435710970.A.31E.html\n","saved\n","title: [公告] 10月份置底閒聊文\n","href: https://www.ptt.cc/bbs/StupidClown/M.1569938128.A.51D.html\n","saved\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"MkRd8kgpuIQh","colab_type":"code","outputId":"9978aa63-838b-490f-f091-4d67c5fa8bab","executionInfo":{"status":"ok","timestamp":1572075857697,"user_tz":-480,"elapsed":1769,"user":{"displayName":"陳宇威","photoUrl":"","userId":"00090422027206355406"}},"colab":{"base_uri":"https://localhost:8080/","height":134}},"source":["%ls"],"execution_count":0,"outputs":[{"output_type":"stream","text":["'Fw: [問卦] 鮭魚握壽司被蒸熟了該怎麼辦？.html'\n"," \u001b[0m\u001b[01;34msample_data\u001b[0m/\n","'[公告] 10月份置底閒聊文.html'\n","'[公告]本板即日起不可PO問卷文.html'\n","'[公告] 笨板板規.html'\n","'[無言] 牙線棒卡在牙套上.html'\n","'[無言] 誰會委託同事買餐巾紙?.html'\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"1vyg4HrFLfVJ","colab_type":"code","colab":{}},"source":["#需滿18歲要加cookies\n","import requests\n","\n","def fetch(url):\n"," response = requests.get(url)\n"," response = requests.get(url, cookies={'over18': '1'}) # 一直向 server 回答滿 18 歲了 !\n"," return response\n","\n","url = 'https://www.ptt.cc/bbs/Gossiping/index.html'\n","resp = fetch(url) # step-1\n","\n","print(resp.text) # result of setp-1"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"lDF2XVSvNs_4","colab_type":"text"},"source":["- 更多可參考[爬蟲教學 CrawlerTutorial](https://github.com/leVirve/CrawlerTutorial)"]},{"cell_type":"markdown","metadata":{"id":"sdsBCb4BkOkK","colab_type":"text"},"source":["### 以wiki亞洲國家資訊為例"]},{"cell_type":"markdown","metadata":{"id":"rINKbz4-kXl8","colab_type":"text"},"source":["- 參考來源[Web Scraping Wikipedia Tables using BeautifulSoup and Python](https://medium.com/analytics-vidhya/web-scraping-wiki-tables-using-beautifulsoup-and-python-6b9ea26d8722)"]},{"cell_type":"code","metadata":{"id":"MwJK3gS2kpmJ","colab_type":"code","colab":{}},"source":["import requests\n","\n","website_url = requests.get('https://en.wikipedia.org/wiki/\\\n"," List_of_Asian_countries_by_area').text\n","\n","from bs4 import BeautifulSoup\n","soup = BeautifulSoup(website_url,'lxml')\n","print(soup.prettify())"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ysX4Mq1JlKTT","colab_type":"text"},"source":[""]},{"cell_type":"code","metadata":{"id":"ELYuAc4plPCN","colab_type":"code","colab":{}},"source":["My_table = soup.find(\"table\",{\"class\":\"wikitable sortable\"})\n","My_table"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"vPKOb288lhI1","colab_type":"code","colab":{}},"source":["links = My_table.findAll('a')\n","links"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"m7n_vzVClthW","colab_type":"code","colab":{}},"source":["Country = [ link.get('title') for link in links if link.get('title') != None]"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"lD4Uf9lVDNxL","colab_type":"code","colab":{}},"source":["Country"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"5QaTp8NcmdGZ","colab_type":"code","colab":{}},"source":["import pandas as pd\n","\n","df = pd.DataFrame()\n","df['Country'] = Country\n","df"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"0Hdt3qwunW_g","colab_type":"code","colab":{}},"source":["df = df.sort_values(by=\"Country\")\n","df.reset_index(drop = True)\n","df"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vnv7fzVMaPCz","colab_type":"text"},"source":["## 練習"]},{"cell_type":"markdown","metadata":{"id":"Ta5rh-Up1vQe","colab_type":"text"},"source":["### 練習1"]},{"cell_type":"markdown","metadata":{"id":"E76LBd6BaSRE","colab_type":"text"},"source":["- 試著看懂並執行、拆解以下程式\n","- 程式來源https://github.com/jwlin/web-crawler-tutorial/blob/master/ch3/ptt_gossiping.py"]},{"cell_type":"code","metadata":{"id":"8HJHhY5qaJtf","colab_type":"code","outputId":"cc214cef-ddf0-456e-9a12-2bc8f4c9868c","executionInfo":{"status":"ok","timestamp":1572076619044,"user_tz":-480,"elapsed":22301,"user":{"displayName":"陳宇威","photoUrl":"","userId":"00090422027206355406"}},"colab":{"base_uri":"https://localhost:8080/","height":581}},"source":["import requests\n","import time\n","import json\n","from bs4 import BeautifulSoup\n","\n","\n","PTT_URL = 'https://www.ptt.cc'\n","\n","\n","def get_web_page(url):\n"," resp = requests.get(\n"," url=url,\n"," cookies={'over18': '1'}\n"," )\n"," if resp.status_code != 200:\n"," print('Invalid url:', resp.url)\n"," return None\n"," else:\n"," return resp.text\n","\n","\n","def get_articles(dom, date):\n"," soup = BeautifulSoup(dom, 'html5lib')\n","\n"," # 取得上一頁的連結\n"," paging_div = soup.find('div', 'btn-group btn-group-paging')\n"," prev_url = paging_div.find_all('a')[1]['href']\n","\n"," articles = [] # 儲存取得的文章資料\n"," divs = soup.find_all('div', 'r-ent')\n"," for d in divs:\n"," if d.find('div', 'date').text.strip() == date: # 發文日期正確\n"," # 取得推文數\n"," push_count = 0\n"," push_str = d.find('div', 'nrec').text\n"," if push_str:\n"," try:\n"," push_count = int(push_str) # 轉換字串為數字\n"," except ValueError:\n"," # 若轉換失敗，可能是'爆'或 'X1', 'X2', ...\n"," # 若不是, 不做任何事，push_count 保持為 0\n"," if push_str == '爆':\n"," push_count = 99\n"," elif push_str.startswith('X'):\n"," push_count = -10\n","\n"," # 取得文章連結及標題\n"," if d.find('a'): # 有超連結，表示文章存在，未被刪除\n"," href = d.find('a')['href']\n"," title = d.find('a').text\n"," author = '' # author = d.find('div', 'author').text if d.find('div', 'author') else ''\n"," articles.append({\n"," 'title': title,\n"," 'href': href,\n"," 'push_count': push_count,\n"," 'author': author\n"," })\n"," return articles, prev_url\n","\n","\n","def get_author_ids(posts, pattern):\n"," ids = set()\n"," for post in posts:\n"," if pattern in post['author']:\n"," ids.add(post['author'])\n"," return ids\n","\n","if __name__ == '__main__':\n"," current_page = get_web_page(PTT_URL + '/bbs/Gossiping/index.html')\n"," if current_page:\n"," articles = [] # 全部的今日文章\n"," today = time.strftime(\"%m/%d\").lstrip('0') # 今天日期, 去掉開頭的 '0' 以符合 PTT 網站格式\n"," current_articles, prev_url = get_articles(current_page, today) # 目前頁面的今日文章\n"," while current_articles: # 若目前頁面有今日文章則加入 articles，並回到上一頁繼續尋找是否有今日文章\n"," articles += current_articles\n"," current_page = get_web_page(PTT_URL + prev_url)\n"," current_articles, prev_url = get_articles(current_page, today)\n","\n"," # 印出所有不同的 5566 id\n"," # print(get_author_ids(articles, '5566'))\n","\n"," # 儲存或處理文章資訊\n"," print('今天有', len(articles), '篇文章')\n"," threshold = 50\n"," print('熱門文章(> %d 推):' % (threshold))\n"," for a in articles:\n"," if int(a['push_count']) > threshold:\n"," print(a)\n"," with open('gossiping.json', 'w', encoding='utf-8') as f:\n"," json.dump(articles, f, indent=2, sort_keys=True, ensure_ascii=False)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["今天有 776 篇文章\n","熱門文章(> 50 推):\n","{'title': '[新聞] 星宇航空首架機交付 張國煒親自駕回台', 'href': '/bbs/Gossiping/M.1572072025.A.F0A.html', 'push_count': 57, 'author': ''}\n","{'title': '[新聞] 同志遊行前夕 蘇貞昌：相互尊重讓台灣更', 'href': '/bbs/Gossiping/M.1572070242.A.738.html', 'push_count': 58, 'author': ''}\n","{'title': '[新聞] 違法三缺一？中國觸怒民怨的「麻將館禁令」', 'href': '/bbs/Gossiping/M.1572068601.A.93D.html', 'push_count': 60, 'author': ''}\n","{'title': '[新聞] 進台北市區注意！明天同志遊行中午起交通', 'href': '/bbs/Gossiping/M.1572066682.A.275.html', 'push_count': 54, 'author': ''}\n","{'title': '[新聞] 非洲豬瘟肆虐 菲律賓養豬業每月損失約6億', 'href': '/bbs/Gossiping/M.1572066826.A.A34.html', 'push_count': 80, 'author': ''}\n","{'title': '[問卦] 有沒有做出PttChrome計算推樓插件的八卦', 'href': '/bbs/Gossiping/M.1572066384.A.30B.html', 'push_count': 99, 'author': ''}\n","{'title': '[爆卦] 中國食品凍漲擋不住了', 'href': '/bbs/Gossiping/M.1572065077.A.9C0.html', 'push_count': 99, 'author': ''}\n","{'title': '[新聞] 貧窮線調高！ 北市月收不到2萬4293元就', 'href': '/bbs/Gossiping/M.1572064119.A.A22.html', 'push_count': 55, 'author': ''}\n","{'title': '[爆卦] GD權志龍退伍了!', 'href': '/bbs/Gossiping/M.1572064886.A.775.html', 'push_count': 61, 'author': ''}\n","{'title': 'Re: [新聞] 李亞萍哭認余苑綺「是末期了」：心裡有數', 'href': '/bbs/Gossiping/M.1572063783.A.974.html', 'push_count': 58, 'author': ''}\n","{'title': '[新聞] 卓榮泰說沒民進黨柯文哲會落選？對手丁守', 'href': '/bbs/Gossiping/M.1572063861.A.27F.html', 'push_count': 51, 'author': ''}\n","{'title': '[新聞] 謝震武紅遍政論節目 竟讓劉寶傑好委屈', 'href': '/bbs/Gossiping/M.1572063933.A.371.html', 'push_count': 71, 'author': ''}\n","{'title': '[問卦] 癌症三期了會治療還是放棄？', 'href': '/bbs/Gossiping/M.1572061470.A.980.html', 'push_count': 54, 'author': ''}\n","{'title': '[新聞] 韓國瑜「國外遊學」支票 蔡英文：韓也說', 'href': '/bbs/Gossiping/M.1572061561.A.6B0.html', 'push_count': 99, 'author': ''}\n","{'title': 'Re: [問卦] 故宮的東西在世界上算厲害嗎', 'href': '/bbs/Gossiping/M.1572061164.A.F36.html', 'push_count': 99, 'author': ''}\n","{'title': '[新聞] 地府只能收美金？中國新法令冥紙禁印人', 'href': '/bbs/Gossiping/M.1572060525.A.E97.html', 'push_count': 56, 'author': ''}\n","{'title': '[新聞] 強國人為何想偷渡英國？\\u3000華春瑩嗆CNN：', 'href': '/bbs/Gossiping/M.1572059049.A.526.html', 'push_count': 99, 'author': ''}\n","{'title': 'Re: [新聞] 旺中「跳船」？《旺報》社長砲轟韓國瑜', 'href': '/bbs/Gossiping/M.1572059203.A.FE5.html', 'push_count': 69, 'author': ''}\n","{'title': '[問卦] 認真問,去駕訓班學手排.還是自己考自排?', 'href': '/bbs/Gossiping/M.1572058150.A.FBA.html', 'push_count': 59, 'author': ''}\n","{'title': '[問卦] 可以數位故宮那能不能數位遊學?!', 'href': '/bbs/Gossiping/M.1572056208.A.0EE.html', 'push_count': 64, 'author': ''}\n","{'title': 'Re: [新聞] 快訊／韓國瑜晚間開支票：只要當總統，大', 'href': '/bbs/Gossiping/M.1572051391.A.DCD.html', 'push_count': 99, 'author': ''}\n","{'title': '[新聞] 美軍盼晶片在地生產 台積電評估赴美建新', 'href': '/bbs/Gossiping/M.1572051828.A.2F6.html', 'push_count': 99, 'author': ''}\n","{'title': '[新聞] 台北警公務車載女友！女方「屁蛋妹」身份', 'href': '/bbs/Gossiping/M.1572050526.A.01D.html', 'push_count': 53, 'author': ''}\n","{'title': 'Re: [新聞] 快訊／韓國瑜晚間開支票：只要當總統，大', 'href': '/bbs/Gossiping/M.1572048367.A.98A.html', 'push_count': 99, 'author': ''}\n","{'title': '[問卦] 你周遭朋友，買過春的男生多嗎？', 'href': '/bbs/Gossiping/M.1572037109.A.44C.html', 'push_count': 88, 'author': ''}\n","{'title': '[問卦] 香港反送中為何不平息？', 'href': '/bbs/Gossiping/M.1572029681.A.250.html', 'push_count': 51, 'author': ''}\n","{'title': '[新聞] 死刑「注射針」插歪！他痛苦3倍時間才身亡', 'href': '/bbs/Gossiping/M.1572026898.A.906.html', 'push_count': 70, 'author': ''}\n","{'title': 'Re: [新聞] 藏人自焚說 柯文哲：用字不精確但我意思', 'href': '/bbs/Gossiping/M.1572026908.A.C09.html', 'push_count': 53, 'author': ''}\n","{'title': 'Re: [問卦] 五月天跪中國了?', 'href': '/bbs/Gossiping/M.1572020507.A.C3B.html', 'push_count': 99, 'author': ''}\n","{'title': '[新聞] 告別立院演說 王金平：太陽花學運不流血落幕 我無愧天地', 'href': '/bbs/Gossiping/M.1572019252.A.AE8.html', 'push_count': 99, 'author': ''}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"wyXAXDIN1MBI","colab_type":"text"},"source":["### 練習2"]},{"cell_type":"markdown","metadata":{"id":"l9CxzWKIcqEZ","colab_type":"text"},"source":["- 試著看懂並執行、拆解以下程式\n","- 程式來源https://github.com/jwlin/web-crawler-tutorial/blob/master/ch3/yahoo_movie.py"]},{"cell_type":"code","metadata":{"id":"mq4Xi5X-cc9A","colab_type":"code","outputId":"1666f312-3144-4b81-99ce-45f0511f9a08","executionInfo":{"status":"ok","timestamp":1572076585531,"user_tz":-480,"elapsed":2183,"user":{"displayName":"陳宇威","photoUrl":"","userId":"00090422027206355406"}},"colab":{"base_uri":"https://localhost:8080/","height":211}},"source":["import requests\n","import re\n","import json\n","from bs4 import BeautifulSoup\n","\n","\n","Y_MOVIE_URL = 'https://tw.movies.yahoo.com/movie_thisweek.html'\n","\n","# 以下網址後面加上 \"/id=MOVIE_ID\" 即為該影片各項資訊\n","Y_INTRO_URL = 'https://tw.movies.yahoo.com/movieinfo_main.html' # 詳細資訊\n","Y_PHOTO_URL = 'https://tw.movies.yahoo.com/movieinfo_photos.html' # 劇照\n","Y_TIME_URL = 'https://tw.movies.yahoo.com/movietime_result.html' # 時刻表\n","\n","\n","def get_web_page(url):\n"," resp = requests.get(url)\n"," if resp.status_code != 200:\n"," print('Invalid url:', resp.url)\n"," return None\n"," else:\n"," return resp.text\n","\n","\n","def get_movies(dom):\n"," soup = BeautifulSoup(dom, 'html5lib')\n"," movies = []\n"," rows = soup.find_all('div', 'release_info_text')\n"," for row in rows:\n"," movie = dict()\n"," movie['expectation'] = row.find('div', 'leveltext').span.text.strip()\n"," movie['ch_name'] = row.find('div', 'release_movie_name').a.text.strip()\n"," movie['eng_name'] = row.find('div', 'release_movie_name').find('div', 'en').a.text.strip()\n"," movie['movie_id'] = get_movie_id(row.find('div', 'release_movie_name').a['href'])\n"," movie['poster_url'] = row.parent.find_previous_sibling('div', 'release_foto').a.img['src']\n"," movie['release_date'] = get_date(row.find('div', 'release_movie_time').text)\n"," movie['intro'] = row.find('div', 'release_text').text.replace(u'詳全文', '').strip()\n"," trailer_a = row.find_next_sibling('div', 'release_btn color_btnbox').find_all('a')[1]\n"," movie['trailer_url'] = trailer_a['href'] if 'href' in trailer_a.attrs.keys() else ''\n"," movies.append(movie)\n"," return movies\n","\n","\n","def get_date(date_str):\n"," # e.g. \"上映日期：2017-03-23\" -> match.group(0): \"2017-03-23\"\n"," pattern = '\\d+-\\d+-\\d+'\n"," match = re.search(pattern, date_str)\n"," if match is None:\n"," return date_str\n"," else:\n"," return match.group(0)\n","\n","\n","def get_movie_id(url):\n"," # 20180515: URL 格式有變, e.g., 'https://movies.yahoo.com.tw/movieinfo_main/%E6%AD%BB%E4%BE%8D2-deadpool-2-7820.html\n"," # e.g., \"https://tw.rd.yahoo.com/referurl/movie/thisweek/info/*https://tw.movies.yahoo.com/movieinfo_main.html/id=6707\"\n"," # -> match.group(0): \"/id=6707\"\n"," try:\n"," movie_id = url.split('.html')[0].split('-')[-1]\n"," except:\n"," movie_id = url\n"," return movie_id\n","\n","\n","def get_trailer_url(url):\n"," # e.g., 'https://tw.rd.yahoo.com/referurl/movie/thisweek/trailer/*https://tw.movies.yahoo.com/video/美女與野獸-最終版預告-024340912.html'\n"," return url.split('*')[1]\n","\n","\n","def get_complete_intro(movie_id):\n"," page = get_web_page(Y_INTRO_URL + '/id=' + movie_id)\n"," if page:\n"," soup = BeautifulSoup(page, 'html5lib')\n"," infobox = soup.find('div', 'gray_infobox_inner')\n"," print(infobox.text.strip())\n","\n","\n","def main():\n"," page = get_web_page(Y_MOVIE_URL)\n"," if page:\n"," movies = get_movies(page)\n"," for movie in movies:\n"," print(movie)\n"," # get_complete_intro(movie[\"movie_id\"])\n"," with open('movie.json', 'w', encoding='utf-8') as f:\n"," json.dump(movies, f, indent=2, sort_keys=True, ensure_ascii=False)\n","\n","\n","if __name__ == '__main__':\n"," main()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["{'expectation': '90%', 'ch_name': '雙子殺手', 'eng_name': 'Gemini Man', 'movie_id': '10017', 'poster_url': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/October2019/YSeFXqDiSK0XrIcSdHi1-486x720.JPG', 'release_date': '2019-10-23', 'intro': '威爾史密斯飾演的一名頂尖殺手亨利布羅根，卻被一位神秘的年輕殺手追殺，而且這位年輕的殺手竟然能事先預測亨利的一舉一動。《雙子殺手》由金像獎得主李安執導，知名製作人傑瑞布洛克海默、大衛艾利森、戴娜戈柏和唐葛蘭傑製作。其他演員還有瑪麗伊莉莎白文斯蒂德、克里夫歐文和班奈狄克王。', 'trailer_url': 'https://movies.yahoo.com.tw/video/%E9%9B%99%E5%AD%90%E6%AE%BA%E6%89%8B-%E6%9C%80%E6%96%B0%E9%A0%90%E5%91%8A-035402781.html?movie_id=10017'}\n","{'expectation': '88%', 'ch_name': '電影版 吹響吧！上低音號～誓言的終章～', 'eng_name': 'Sound! Euphonium, the Movie -Our Promise: A Brand New Day-', 'movie_id': '10306', 'poster_url': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/October2019/ZXplEx71mOQQlcTpxiYd-992x1418.JPG', 'release_date': '2019-10-23', 'intro': '順利在去年的全日本管樂競演會中出場的北宇治高中管樂社，升上二年級的黃前久美子和三年級的加部友惠，兩人開始一起負責指導從四月開始新加入的一年級新生。\\n\\xa0\\n由於身為全國大賽的出場學校，因此吸引了大量的一年級新生入部，其中，有四名新生來到了低音部，包括乍看之下似乎毫無問題的久石奏、不融入周圍的鈴木美玲、想要和美玲成為朋友的鈴木五月，以及從不提及私事的月永求。面對接連而來的Sunrise祭、選拔賽以及競演會，以「全國大賽金獎」為目標的管樂社，卻接連發生問題……！？北宇治高中管樂社，風波不斷的日子開始了！', 'trailer_url': ''}\n","{'expectation': '56%', 'ch_name': '日本國民導演—山田洋次影展', 'eng_name': '', 'movie_id': '10299', 'poster_url': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/October2019/CLNjaW36KsCZoGzmmhMn-509x720.jpg', 'release_date': '2019-10-24', 'intro': '現年88歲的日本國民導演-山田洋次，從事電影工作超過一甲子的時間，已累績八十多部的導演作品，至今仍持續推出高品質的新作，半世紀以來，一直深受不同世代觀眾的共鳴。2019年適逢《男人真命苦》問世五十週年，光點台北因此特別規劃山田洋次回顧影展，精選八部代表作品，從1960年代到二十一世紀，一窺這位日本庶民大師五十年來的創作軌跡！10.04周五起正式售票，10.24-11.06影展期間於光點台北電影館播映，歡迎影迷們共襄盛舉！\\n\\xa0\\n本次影展共選《男人真命苦》、《家族》、《故鄉》、《幸福的黃手帕》、《遠山的呼喚》、《兒子》《黃昏清兵衛》、《東京小屋的回憶》八部經典之作，透過不同時期的電影回顧，激起屬於每個時代的獨特記憶。其中1969年正式上映的松竹電影《男人真命苦》，由渥美清、倍賞千惠子、前田吟、森川信等人主演，描述東京平凡市井小民─寅次郎的故事，本片不只創下破億日幣票房，在接下來的二十六年間以相同班底連續推出48集，同時也締造金氏世界紀錄最長系列片，成為日本人的共同記憶！\\n\\xa0\\n山田洋次從影歷程幾乎是半部日本電影史，耕耘多年的他繼承小津安二郎、木下惠介等前輩樹立的「庶民劇」傳統，終其一生致力於書寫市井小民的喜怒哀樂，也如同小津一樣，長期與固定的劇組合作，不僅培養出工作夥伴間的良好默契，也宛如陪伴觀眾成長的左鄰右舍一般親切。\\n\\xa0\\n「身為一介小市民，我要將日常生活中觸動─人心的事，透過某種契機使之成形，在構思中建立骨骼、填上血肉，完成具體的作品」，多年來山田洋次的秉持著相同的理念，在他的電影裡可以感受到最真摯、最貼近生活的創作，欲重溫這些經典作品的影迷們，10.24-11.06期間於光點台北電影館千萬別錯過【日本國民導演—山田洋次 影展】。\\n\\xa0\\n【售票資訊】10.04（五）起開始售票\\n\\xa0\\n全票：240元/張 ，光點會員200元/張\\n\\xa0\\n套票：720元/4張乙套\\n\\xa0\\n✔專屬優惠好禮─凡於現場一次購買『#兩組套票』，即贈 大春煉皂 提供【經典米萃皂】乙份。(數量有限，送完為止)\\n\\xa0\\n●購票一律請親至光點台北售票服務台並於現場確認場次\\n\\xa0\\n●光點會員請持會員卡及有效證件購票享會員優惠\\n\\xa0\\n●愛心票僅供65歲以上老人、身心障礙人士與乙名必要陪同者(需同時入場)購買，購票時請出示相關證件\\n\\xa0\\n●套票售出概不退換', 'trailer_url': ''}\n","{'expectation': '89%', 'ch_name': '金翅雀', 'eng_name': 'The Goldfinch', 'movie_id': '10072', 'poster_url': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/October2019/WR18DRSUJn9Z1GdkwUNI-2985x4263.jpg', 'release_date': '2019-10-25', 'intro': '《金翅雀》故事敘述小名「席歐」的13歲少年席爾鐸戴克(奧克斯弗格雷 飾)與母親參觀大都會藝術博物館，當他們在欣賞母親最愛的「金翅雀」這幅畫作時，館內突然發生大爆炸，席歐幸運地逃過一劫，卻也因此與母親從此天人永隔，這場突如其來的鉅變改變了席歐的一生，讓他展開充滿波折的人生與探索的旅程；成年後的席歐(安索艾格特 飾)經歷了無止盡的悲傷與自責、一連串的重生、贖罪與溫暖的愛之後，當他回首生命的這一切，他心中放不下的還是那幅改變他一生的畫作：一隻腳被細細的鎖鏈綁在棲木上的小鳥兒，那個看似美麗，卻永遠無法獲得自由，同時也是母親在世時摯愛的作品：「金翅雀」。', 'trailer_url': 'https://movies.yahoo.com.tw/video/%E9%87%91%E7%BF%85%E9%9B%80-%E4%B8%AD%E6%96%87%E9%A0%90%E5%91%8A-072238436.html?movie_id=10072'}\n","{'expectation': '94%', 'ch_name': '陪你很久很久', 'eng_name': 'Stand by me', 'movie_id': '10160', 'poster_url': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/September2019/IyCzEAuwXvQbjJFZZ4TI-510x720.jpg', 'release_date': '2019-10-25', 'intro': '★今年最熱血、最青春、最動感的校園愛情電影！\\n★李淳首挑大樑，攜手兩大超人氣女星邵雨薇、蔡瑞雪，共解最複雜的愛情習題！\\n★顛覆傳統浪漫小清新，荒唐、逗趣、充滿驚喜，新世代青春爆笑喜劇！\\n\\xa0\\n憨直的九餅（李淳飾）從小就暗戀著薄荷（邵雨薇飾），兩人一路相互陪伴，卻始終維持著「好朋友」的關係。直到上了大學後，九餅意外地與甜美高中生夏天（蔡瑞雪飾）成為同房室友，又在迎新試膽大會時，不小心將薄荷推向校園天菜麥子學長（宋柏緯飾）懷中，眼看著心愛的她即將被追走，九餅究竟要如何守住薄荷，為自己的青春奮力一搏！', 'trailer_url': 'https://movies.yahoo.com.tw/video/%E9%99%AA%E4%BD%A0%E5%BE%88%E4%B9%85%E5%BE%88%E4%B9%85-%E9%9D%92%E6%98%A5%E7%86%B1%E8%A1%80%E7%89%88%E9%A0%90%E5%91%8A-132846573.html?movie_id=10160'}\n","{'expectation': '62%', 'ch_name': '伊索遊戲', 'eng_name': 'Aesop’s Game', 'movie_id': '10216', 'poster_url': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/August2019/fJBtPhAz38ToLmsSaIVb-800x1142.jpg', 'release_date': '2019-10-25', 'intro': '「一隻烏龜從天而降砸傷了路人，不知是意外還是惡作劇，警方正在追查這隻肇事逃逸的烏龜…」故事從烏龜開始，加上兔子，還有狗狗來亂入？顛覆你所知道的伊索寓言！\\n\\xa0\\n龜田美羽（龜），內向的女大學生，和家人感情很好，唯一的朋友卻是隻烏龜。\\n兔草早織（兔），超人氣的星二代，出生「明星家族」，卻天生一副戀愛體質。\\n戌井小柚（犬），身手不凡，和爸爸一起經營「復仇」生意，天天幫人尋仇。\\n當三位少女相遇，你說這是部酸酸甜甜的青春電影？……大錯特錯！\\n\\xa0\\n綁架、背叛、復仇、揭開虛偽的假象，一場超乎想像的鬥智心理遊戲正要開始，結局會和你想像的……完全不同！！！校園純愛戀曲是假的，懸疑驚悚才是真的！當綁架事件爆發，最難預測的才不是青春，而是這部電影的劇情走向！\\n\\xa0\\n【關於電影】\\n\\xa0\\n延續神片《一屍到底》最強騙術\\n三位導演、三倍翻轉，一次滿足！\\n本片由日本觀影人次超過220萬、票房突破31億日圓、2018年最受矚目的話題作品《一屍到底》導演上田慎一郎親自操刀原創劇本，借三名導演之力搬上大銀幕，承襲《一屍到底》一路「騙」很大、在騙局中加洋蔥的反轉風格，將再次帶給觀眾就算被騙到底，也甘之如飴的觀影體驗。此次與上田慎一郎共同執導的導演之一是在《一屍到底》擔任副導的中泉裕矢，他在2011年推出第一部導演長片《圓罪》，其後作品《與母親的旅程》、《來去拍片尾》都在日本國內影展蔚為話題。另一位導演則是在《一屍到底》中擔任劇照師的淺沼直也，他曾以《若冬天燃起》在SKIP CITY國際電影節獲得2017年短片單元最佳作品，是日本當前備受期待的新銳創作者。三人在2015年便曾合作過短片集《四個與貓的故事》，當時三人各自擔任其中一部的導演，因此這次決定再度攜手合作。截然不同的三人共同擔任導演，將蹦出三種不同特色的火花，這部花費三年以上製作的電影，終於要登上大銀幕和觀眾見面了！\\n\\xa0\\n以《一屍到底》快速竄紅的導演上田慎一郎表示自己從中學時期就開始自己嘗試拍片，當時沒有YouTube，也不知道有電影節這種管道，光是把作品拍出來就已經是對自己最大的獎勵了！長大之後，也有曾經抱持著：「作品非在電影節放映不可！」、「我的作品一定要被偉大的人認可！」然而在拍攝《一屍到底》時，反而試圖讓自己不去多想成敗，只專注於自己想拍的東西，最後的成果反而得到崇高肯定，上田慎一郎表示：「投入《伊索遊戲》創作後我也期許自己秉持初心，繼續拍自己想拍的電影！」上田導演坦言《伊索遊戲》這部作品前後花了三年構想，不過事實上前兩年都還在和另兩位導演爭論究竟要拍什麼樣的故事，「三個人一起的執導，絕對會有各自無法妥協的部分，不過我們也有著要將這些矛盾、糾結化為這部電影的魅力的覺悟！」本片也大膽啟用新銳演員石川瑠華、井桁弘恵、紅甘擔綱主演，上田慎一郎在電影上映前也再次向觀眾呼籲：「這部電影有三名導演、三位女主角，所以大家請務必三人以上同行前往電影院觀看！」', 'trailer_url': 'https://movies.yahoo.com.tw/video/%E4%BC%8A%E7%B4%A2%E9%81%8A%E6%88%B2-%E5%B0%8E%E6%BC%94%E5%95%8F%E5%80%99%E7%AF%87-105145492.html?movie_id=10216'}\n","{'expectation': '69%', 'ch_name': '西幽玹歌', 'eng_name': 'Thunderbolt Fantasy', 'movie_id': '10244', 'poster_url': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/September2019/EcNK84hDr6NAucwtBl3N-506x720.jpg', 'release_date': '2019-10-25', 'intro': '生來便懷有異能歌聲的少年浪巫謠，自小便跟在隱遁雪山的盲眼母親身邊，接受經年累月的苛刻訓練。\\n\\xa0\\n母親懷著野心，想將兒子的歌聲鍛鍊成天下無雙，然後送入宮廷。然而過於苛刻的訓練卻招致了不幸的事故，母親在浪巫謠眼前斷送了性命。\\n\\xa0\\n失去照顧者後，浪巫謠成了流浪之身。但他的異能卻總被無情的人們利用，成為他們滿足慾望的道具，漸漸消磨著少年的心靈。\\n\\xa0\\n儘管如此，浪巫謠罕見的歌聲終於吸引到了西幽皇女，因而得到了過去母親所夢想的飛黃騰達。但是等待著他的，卻是成為執政者玩物、賭上性命與其他樂師進行演奏競賽的殘忍遊戲。\\n\\xa0\\n就在某一天，浪巫謠聽聞有一名在西幽各地奪取魔劍、占為己有的大罪人──「啖劍太歲」。而啖劍太歲的下一個目標，正是皇帝藏在宮中的聖劍。\\n\\xa0\\n【關於電影】\\n\\xa0\\n在臺灣可以說是「無人不知、無人不曉」，從大人到小孩都非常喜愛的木偶劇「布袋戲」。 本次作品由對布袋戲深深著迷的Nitroplus「虛淵玄」擔任故事原著・劇本・總監修，與臺灣布袋戲中最具知名度及以其高品質製作聞名的「霹靂國際多媒體」（日文簡稱：霹靂社）合作， 以此奇蹟般的組合所完成的日本及臺灣地區共同企劃之影像作品。\\n\\xa0\\n於2018年12月24日（一）播放TV版第二季《Thunderbolt Fantasy 東離劍遊紀２》最後一集， 在那之後隨即發表「第三季製作決定！」的情報， 並且在3月22日（五）公佈第二季外傳《Thunderbolt Fantasy 西幽玹歌》正在進行製作的消息。\\xa0\\n\\xa0\\n2019年10月25日（五）於戲院上映的《Thunderbolt Fantasy 西幽玹歌》，以活躍於TV版第二季的「浪巫謠」為主角，講述他的過去、以及他在西幽發生的故事，為Nitroplus「虛淵玄」擔任故事原著・劇本・總監修的全新作品。\\n\\xa0\\n角色設計和至今為止的系列作品同樣，由Nitroplus所率領的設計團隊（新加入成員「minoa」）負責，同時亦邀請公仔製作公司「GOOD SMILE COMPANY」擔任戲偶造型顧問。 配樂的部份也同樣邀請到專注在偶像劇、動畫、電影界配樂、並且擔任許多歌手樂曲之提供的「澤野弘之」擔任製作。', 'trailer_url': 'https://movies.yahoo.com.tw/video/%E8%A5%BF%E5%B9%BD%E7%8E%B9%E6%AD%8C-%E6%AD%A3%E5%BC%8F%E9%A0%90%E5%91%8A-053315107.html?movie_id=10244'}\n","{'expectation': '80%', 'ch_name': '失憶的總理大臣', 'eng_name': '', 'movie_id': '10246', 'poster_url': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/October2019/w1eivF3UfcYZ5YI6T45k-800x1132.jpg', 'release_date': '2019-10-25', 'intro': '無良政客一夕失憶，竟然轉性善良大叔！？\\n史上民調最低的總理大臣黑田啟介（中井貴一飾），走到哪都被市民噓爆，黑粉滿街跑！某天，總理又被沿街抗議，沒想到一顆石頭飛來，砸中額頭而昏迷，醒來後，他竟完全喪失記憶，黑粉美夢一夕成真？！為了避免國家大亂，他的三位貼身秘書竭盡所能讓國事照常運作，全面隱瞞真相。而一向唯利是圖的無良總理，一夕之間轉性變成傾聽民心、致力國政的善良大叔，異常反差的態度也讓全民開始起疑。眼看總理「我失憶了」秘密即將露餡，政壇的狐群狗黨們為私利大亂鬥，美國總統偏偏在此時將參訪日本，一場顛覆政壇的狂喜劇即將鬧上全世界！\\n\\xa0\\n【關於電影】\\n\\xa0\\n喜劇大師三谷幸喜醞釀四十年《失憶的總理大臣》引爆全民共鳴\\n導演撿到槍電影質問首相安倍晉三幽默回應心得秒破冰！\\n日本喜劇大師三谷幸喜睽違多年，在影迷千呼萬喚之下，終於帶來最高傑作《失憶的總理大臣》！三谷幸喜端出醞釀四十年的故事，並集結黃金卡司包括日本奧斯卡影帝中井貴一、《月薪嬌妻》「全民小阿姨」石田百合子、《大叔之愛》田中圭、《告白》木村佳乃、老班底日本奧斯卡影帝佐藤浩市等一線明星，再次打造三谷幸喜無人能擋的魅力！三谷幸喜表示，故事靈感來自他高中的志願發想，「如果像我這樣沒有私利私慾、不追求權力金錢慾的人做總理，不是很好的政治家嗎？」但路人突然變成總理根本不可能，喜劇鬼才的他就想到石頭砸破腦袋的橋段，創作出這部不分國界引爆觀眾共鳴的喜劇新作。此外，膽大包天的《失憶的總理大臣》劇組甚至直接邀請日本現任首相安倍晉三看片，映後導演戰戰兢兢地問首相：「感想是？」沒想到安倍幽默拿出政治人物愛用語回：「我失去記憶了！」爆笑氣氛現場一秒破冰！三谷再次以縝密的故事編排和獨特「三谷式」幽默驚艷觀眾，電影對政客的惡搞更引起日本全民共鳴，展開「最希望哪位政治人物失憶」的熱烈討論，造就另一股社會話題炫風！\\n\\xa0\\n每一部都讓你笑到哭又感動哭！\\n三谷幸喜魅力無人能敵日本巨星爭相合作朝聖\\n鬼才導演三谷幸喜能編能導，作品橫跨電影、電視及舞台劇，過去不僅以叫好叫座的喜劇《魔幻時刻》、《鬼壓床了沒》、《有頂天大飯店》累積破百億票房，NHK大河劇《真田丸》也大獲好評，他所執導的舞台劇更是在日本及台灣都場場爆滿，是日本巨星阿部寬、妻夫木聰、綾瀨遙、松隆子、深津繪里、役所廣司等人爭相合作的對象，就算只能演小配角也沒問題！這次更加入首度合作的石田百合子及藤岡靛黃金陣容，加上三谷幸喜作品天馬行空卻又天衣無縫的巧合與誤會，荒謬劇情再度讓觀眾笑到岔氣，同時透過小人物的熱血初衷，帶來意想不到的逆轉結局，讓人忍不住笑著笑著就感動哭了！今年新作《失憶的總理大臣》以觀眾再熟悉不過的政治為題，三谷幸喜表示：「這不是政治諷刺劇，而是政治狂想曲！」大導透過架空的舞台為觀眾創造更多聯想空間，就算來自不同國家的觀眾，也能輕鬆帶入自己所熟悉的時事！', 'trailer_url': 'https://movies.yahoo.com.tw/video/%E5%A4%B1%E6%86%B6%E7%9A%84%E7%B8%BD%E7%90%86%E5%A4%A7%E8%87%A3-%E6%AD%A3%E5%BC%8F%E9%A0%90%E5%91%8A-122854501.html?movie_id=10246'}\n","{'expectation': '46%', 'ch_name': '阿嬤養的豬', 'eng_name': \"Esmeralda's Twilight\", 'movie_id': '10272', 'poster_url': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/September2019/crRK7LKzsnj0hdMz3Yv2-488x720.JPG', 'release_date': '2019-10-25', 'intro': '★2019墨西哥奧斯卡「阿里爾獎」奪最佳處女電影獎導演提名\\n★2019墨西哥奧斯卡「阿里爾獎」最佳女主角提名\\n★墨西哥電影界最高榮譽「墨西哥電影電視藝術學院獎」提名\\n★比《我不笨，我有話要說》更感人的溫馨催淚片\\xa0\\n最佛心的阿嬤，豬年行大運\\n★ 豬儂我儂，有阿嬤疼的小豬最窩心、像個寶\\n\\xa0\\n年度必看的感人溫馨催淚電影 內含洋蔥讓人揪心落淚\\n隨著丈夫的去世和她兒子不在身旁，老婦人已經失去了對生活的樂趣，直到她重新獲得希望…當一隻小豬進入她的生活。\\n\\xa0\\n在丈夫過世後，老婦人獨居在小鎮，與移居美國的兒子通上電話，成為每天生活唯一的動力，對習慣照顧人的她來說，生命像失去目標。\\n\\xa0\\n有天，一頭小豬意外闖進她的生活，她開始把小豬當成孩子照顧，三餐讓牠吃飽飽，梳毛散步都不少，帶給她與鄰居很多歡樂，不久這隻豬懷孕了，她傾盡所有的積蓄、精力與時間，準備迎接豬孫的到來，卻發現豬女兒身體不如以往了…\\n\\xa0\\n【關於電影】\\xa0\\n\\xa0\\n喪偶奶奶透過小豬重獲新生 純樸鄉村裡人與豬之間最溫馨的羈絆關係\\n《阿嬤養的豬》透過獨居婦人對動物的依戀，點出老人現實處境，熱帶風情的景色、美味的料理、人情味的可愛人物，打造出充滿愛與信念的樸素小鎮，細膩堆疊出人與小豬的真摯情感，已近薄暮的主角，面對人生的無常，選擇放手與釋懷，心境轉換帶出希望的光輝，更顯溫暖令人動容。', 'trailer_url': 'https://movies.yahoo.com.tw/video/%E9%98%BF%E5%AC%A4%E9%A4%8A%E7%9A%84%E8%B1%AC-%E4%B8%AD%E6%96%87%E9%A0%90%E5%91%8A-134459693.html?movie_id=10272'}\n","{'expectation': '53%', 'ch_name': '我的朋友霸王龍', 'eng_name': 'My TYRANO: Together, Forever', 'movie_id': '10274', 'poster_url': 'https://movies.yahoo.com.tw/x/r/w420/i/o/production/movies/September2019/FjcsFOd9I7bDwpFHNcpP-800x1142.jpg', 'release_date': '2019-10-25', 'intro': '★改編自全球暢銷繪本作家宮西達也的恐龍系列作品，取自〈永遠永遠在一起〉〈我愛你〉〈我相信你〉三部繪本，是《你看起來好像很好吃》同系列電影。\\n★由《名偵探柯南》動畫系列導演靜野孔文出任導演、監製，是一部中、日、韓三地聯合製作的動畫電影。\\xa0\\n★奧斯卡金獎配樂大師坂本龍一第一部於動畫電影界亮相的作品。\\n★由手塚治虫創辦的手塚製作公司擔任動畫製作。\\n\\xa0\\n在冰河世紀來臨前的恐龍時期，一隻正被魔鬼龍追逐的粉紅色小翼龍普娜，就在她正要被吞下肚之前，一隻巨大威猛的食肉霸王龍突然出現在牠們面前；\\n當他一出現，早已不見被嚇到的魔鬼龍蹤跡，此時只獨留下普娜在這隻霸王龍面前，他張開嘴靠近普娜，一口口咬了下去，但他吞下的竟然是紅果子而不是這隻小翼龍普娜。\\n”大個子，你不是吃肉的動物嗎？怎麼會吃紅色水果？”普娜在他面前喋喋不休地問著；說著說著，她下定決心跟隨這隻霸王龍，路途中遇見了與媽媽失散的三角龍男孩，他們決定陪著牠回家並結伴一起開始尋找綠洲。\\n\\xa0\\n一隻不會飛的翼龍、一隻不吃肉的霸王龍與孤獨的男孩三角龍，在前往尋找綠洲的旅程中，三人成為像是好朋友又像是家人的好夥伴。\\n\\xa0\\n一路上，他們是否可以避開魔鬼龍的追逐，安全地到達那大家口中所說的的充滿和平與富裕的綠洲呢？', 'trailer_url': 'https://movies.yahoo.com.tw/video/%E6%88%91%E7%9A%84%E6%9C%8B%E5%8F%8B%E9%9C%B8%E7%8E%8B%E9%BE%8D-%E4%B8%AD%E6%96%87%E9%A0%90%E5%91%8A-122910372.html?movie_id=10274'}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"rgZmPsh9zZlN","colab_type":"text"},"source":["### 練習3\n"]},{"cell_type":"markdown","metadata":{"id":"w3GT1nsfzhEy","colab_type":"text"},"source":["- 擷取並parse「批批踢JOKE版的一篇文章」\n","- 請依下列步驟練習：\n"," - 以GET方法將網頁https://www.ptt.cc/bbs/joke/M.1571755669.A.663.html 原始碼讀入\n"," - 依照上述步驟parse出推文內容及推文者\n"," - 透過for迴圈，整齊印出"]}]}
--------------------------------------------------------------------------------