![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
38 | ]
39 | },
40 | {
41 | "cell_type": "code",
42 | "metadata": {
43 | "id": "Jrf17eHRgugt",
44 | "colab_type": "code",
45 | "colab": {}
46 | },
47 | "source": [
48 | "%reload_ext autoreload\n",
49 | "%autoreload 2\n",
50 | "%matplotlib inline"
51 | ],
52 | "execution_count": 0,
53 | "outputs": []
54 | },
55 | {
56 | "cell_type": "code",
57 | "metadata": {
58 | "id": "2g2Us8Ibguhe",
59 | "colab_type": "code",
60 | "colab": {}
61 | },
62 | "source": [
63 | "from fastai import *\n",
64 | "from fastai.text import *"
65 | ],
66 | "execution_count": 0,
67 | "outputs": []
68 | },
69 | {
70 | "cell_type": "code",
71 | "metadata": {
72 | "id": "IEu729BZguh6",
73 | "colab_type": "code",
74 | "colab": {}
75 | },
76 | "source": [
77 | "path = untar_data(URLs.IMDB_SAMPLE)"
78 | ],
79 | "execution_count": 0,
80 | "outputs": []
81 | },
82 | {
83 | "cell_type": "code",
84 | "metadata": {
85 | "id": "ma9L0pzHguiS",
86 | "colab_type": "code",
87 | "colab": {},
88 | "outputId": "c594d394-bcde-4ecc-f073-27b1d4984986"
89 | },
90 | "source": [
91 | "df = pd.read_csv(path/'texts.csv')\n",
92 | "df.head()"
93 | ],
94 | "execution_count": 0,
95 | "outputs": [
96 | {
97 | "output_type": "execute_result",
98 | "data": {
99 | "text/html": [
100 | "| \n", 118 | " | label | \n", 119 | "text | \n", 120 | "is_valid | \n", 121 | "
|---|---|---|---|
| 0 | \n", 126 | "negative | \n", 127 | "Un-bleeping-believable! Meg Ryan doesn't even ... | \n", 128 | "False | \n", 129 | "
| 1 | \n", 132 | "positive | \n", 133 | "This is a extremely well-made film. The acting... | \n", 134 | "False | \n", 135 | "
| 2 | \n", 138 | "negative | \n", 139 | "Every once in a long while a movie will come a... | \n", 140 | "False | \n", 141 | "
| 3 | \n", 144 | "positive | \n", 145 | "Name just says it all. I watched this movie wi... | \n", 146 | "False | \n", 147 | "
| 4 | \n", 150 | "negative | \n", 151 | "This movie succeeds at being one of the most u... | \n", 152 | "False | \n", 153 | "
| idx | \n", 399 | "text | \n", 400 | "
|---|---|
| 0 | \n", 405 | "! ! ! ! ! this was directed by the guy who did ! big ! ! ? ! must be a ! of ! ! - ! style . ! ! ! ! ! this is a extremely well - made film . ! the acting , script and camera - work are all first - rate . ! the music is good , too , though it is | \n", 406 | "
| 1 | \n", 409 | "/ 2 hours . i expected more about ! son of ! sam and instead got a movie that ! to have very little to do with the 1977 serial ! . ! the talking dog was ! ( you know you 're in ! when all the movie ! burst into laughter ! ) . ! the ! movie ! very ! and not very interesting . ! the | \n", 410 | "
| 2 | \n", 413 | "series ! it a ! telling ! to the ! . ! ! , ! ! ! is a pretty \" modern \" girl when you think about it , even though the ! of ! ! ! ! 's world may ! a ! ! to us today . ! if you have ! ! ! ! , i 'll only say that self - sufficient ! ! does | \n", 414 | "
| 3 | \n", 417 | ", particularly the use of ! or ! on head and ! hair . ! ! this film is the only one that ! with some ! ! ! ' original ! ! as has been ! . ! ! may find it ! to note plot details that ! here that are ! ! in other versions of the story . ! ! ' description of the ! ! | \n", 418 | "
| 4 | \n", 421 | "! once wrote about that , \" we should all try to hang around with ! of other ! and try to know them . \" ! ! you 're ! your own ! . ! ! ! ! ! in one of those ! i ! . ! one time someone put ! around campus talking about the ! of the ! ! ! race and had the ! | \n", 422 | "
| epoch | \n", 520 | "train_loss | \n", 521 | "valid_loss | \n", 522 | "accuracy | \n", 523 | "time | \n", 524 | "
|---|---|---|---|---|
| 0 | \n", 529 | "4.842220 | \n", 530 | "4.403178 | \n", 531 | "0.281824 | \n", 532 | "00:57 | \n", 533 | "
| 1 | \n", 536 | "4.287532 | \n", 537 | "3.765579 | \n", 538 | "0.314429 | \n", 539 | "00:57 | \n", 540 | "
| 2 | \n", 543 | "3.825458 | \n", 544 | "3.423177 | \n", 545 | "0.345966 | \n", 546 | "00:57 | \n", 547 | "
| 3 | \n", 550 | "3.545260 | \n", 551 | "3.327716 | \n", 552 | "0.357733 | \n", 553 | "00:57 | \n", 554 | "
| 4 | \n", 557 | "3.352965 | \n", 558 | "3.245280 | \n", 559 | "0.363361 | \n", 560 | "00:57 | \n", 561 | "
| 5 | \n", 564 | "3.226402 | \n", 565 | "3.210527 | \n", 566 | "0.366980 | \n", 567 | "00:57 | \n", 568 | "
| 6 | \n", 571 | "3.125595 | \n", 572 | "3.181285 | \n", 573 | "0.369834 | \n", 574 | "00:57 | \n", 575 | "
| 7 | \n", 578 | "3.046173 | \n", 579 | "3.167163 | \n", 580 | "0.372082 | \n", 581 | "00:57 | \n", 582 | "
| 8 | \n", 585 | "3.001789 | \n", 586 | "3.164754 | \n", 587 | "0.372130 | \n", 588 | "00:57 | \n", 589 | "
"
11 | ]
12 | },
13 | {
14 | "cell_type": "code",
15 | "execution_count": 0,
16 | "metadata": {
17 | "colab": {},
18 | "colab_type": "code",
19 | "id": "i8F6rk1YwJ9P"
20 | },
21 | "outputs": [],
22 | "source": [
23 | "%reload_ext autoreload\n",
24 | "%autoreload 2\n",
25 | "%matplotlib inline"
26 | ]
27 | },
28 | {
29 | "cell_type": "code",
30 | "execution_count": 0,
31 | "metadata": {
32 | "colab": {},
33 | "colab_type": "code",
34 | "id": "btS8qaALwJ9b"
35 | },
36 | "outputs": [],
37 | "source": [
38 | "from fastai import *\n",
39 | "from fastai.text import *"
40 | ]
41 | },
42 | {
43 | "cell_type": "markdown",
44 | "metadata": {
45 | "colab_type": "text",
46 | "id": "LkXYHQhcwKAU"
47 | },
48 | "source": [
49 | "## Seq2Seq\n",
50 | "\n",
51 | "A sequence-to-sequence model is a model that takes a sequence of items and outputs another sequence of items using two networks that are trained end-to-end. This is perfect for machine translation since input sequences are directly related to output sequences. We will looking at preparing a dataset for Machine Translation task and implementing a seq2seq model. We will be using the parallel corpus available from [here](ftp://ftp.monash.edu/pub/nihongo/examples.utf.gz)"
52 | ]
53 | },
54 | {
55 | "cell_type": "code",
56 | "execution_count": 0,
57 | "metadata": {
58 | "colab": {},
59 | "colab_type": "code",
60 | "id": "I5KcBSgDwKAU"
61 | },
62 | "outputs": [],
63 | "source": [
64 | "def make_corpus(corpus_path):\n",
65 | " corpus = corpus_path.open().readlines()\n",
66 | " data_pair = []\n",
67 | " pat = r'#ID.+\\n'\n",
68 | " for c in corpus:\n",
69 | " if 'A: ' in c:\n",
70 | " clean_c = c.replace('A: ','')\n",
71 | " res = re.search(pat,clean_c)\n",
72 | " clean_c = clean_c.replace(res.group(0),'').split('\\t')\n",
73 | " data_pair.append((clean_c[0],clean_c[1]))\n",
74 | " return pd.DataFrame(data_pair,columns=['ja','en'])"
75 | ]
76 | },
77 | {
78 | "cell_type": "code",
79 | "execution_count": 0,
80 | "metadata": {
81 | "colab": {},
82 | "colab_type": "code",
83 | "id": "LBvnts3UwKAY",
84 | "outputId": "82c45ceb-1eae-44e3-e742-0d0ca8e2fb65"
85 | },
86 | "outputs": [
87 | {
88 | "data": {
89 | "text/html": [
90 | "| \n", 108 | " | ja | \n", 109 | "en | \n", 110 | "
|---|---|---|
| 0 | \n", 115 | "ムーリエルは２０歳になりました。 | \n", 116 | "Muiriel is 20 now. | \n", 117 | "
| 1 | \n", 120 | "すぐに戻ります。 | \n", 121 | "I will be back soon. | \n", 122 | "
| 2 | \n", 125 | "すぐに諦めて昼寝をするかも知れない。 | \n", 126 | "I may give up soon and just nap instead. | \n", 127 | "
| 3 | \n", 130 | "愛してる。 | \n", 131 | "I love you. | \n", 132 | "
| 4 | \n", 135 | "ログアウトするんじゃなかったよ。 | \n", 136 | "I shouldn't have logged off. | \n", 137 | "
"
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {},
13 | "source": [
14 | "## GANs with mnist dataset\n",
15 | "We'll look at how to make a simple generator and discriminator network and try to generate images from the mnist dataset"
16 | ]
17 | },
18 | {
19 | "cell_type": "code",
20 | "execution_count": null,
21 | "metadata": {},
22 | "outputs": [],
23 | "source": [
24 | "%reload_ext autoreload\n",
25 | "%autoreload 2\n",
26 | "%matplotlib inline"
27 | ]
28 | },
29 | {
30 | "cell_type": "code",
31 | "execution_count": null,
32 | "metadata": {},
33 | "outputs": [],
34 | "source": [
35 | "import fastai\n",
36 | "from fastai.vision import *\n",
37 | "from fastai.callbacks import *\n",
38 | "from fastai.vision.gan import *"
39 | ]
40 | },
41 | {
42 | "cell_type": "code",
43 | "execution_count": null,
44 | "metadata": {},
45 | "outputs": [],
46 | "source": [
47 | "path = untar_data(URLs.MNIST)"
48 | ]
49 | },
50 | {
51 | "cell_type": "code",
52 | "execution_count": null,
53 | "metadata": {},
54 | "outputs": [],
55 | "source": [
56 | "path.ls()"
57 | ]
58 | },
59 | {
60 | "cell_type": "code",
61 | "execution_count": null,
62 | "metadata": {},
63 | "outputs": [],
64 | "source": [
65 | "bs,size=128,28"
66 | ]
67 | },
68 | {
69 | "cell_type": "code",
70 | "execution_count": null,
71 | "metadata": {},
72 | "outputs": [],
73 | "source": [
74 | "il = ImageList.from_folder(path, convert_mode='L')"
75 | ]
76 | },
77 | {
78 | "cell_type": "code",
79 | "execution_count": null,
80 | "metadata": {},
81 | "outputs": [],
82 | "source": [
83 | "defaults.cmap='binary'"
84 | ]
85 | },
86 | {
87 | "cell_type": "code",
88 | "execution_count": null,
89 | "metadata": {},
90 | "outputs": [],
91 | "source": [
92 | "sd = il.split_by_folder(train='training', valid='testing')"
93 | ]
94 | },
95 | {
96 | "cell_type": "code",
97 | "execution_count": null,
98 | "metadata": {},
99 | "outputs": [],
100 | "source": [
101 | "ll = sd.label_from_folder()"
102 | ]
103 | },
104 | {
105 | "cell_type": "code",
106 | "execution_count": null,
107 | "metadata": {},
108 | "outputs": [],
109 | "source": [
110 | "x,y = ll.train[0]"
111 | ]
112 | },
113 | {
114 | "cell_type": "code",
115 | "execution_count": null,
116 | "metadata": {},
117 | "outputs": [],
118 | "source": [
119 | "x.show()\n",
120 | "print(y,x.shape)"
121 | ]
122 | },
123 | {
124 | "cell_type": "markdown",
125 | "metadata": {},
126 | "source": [
127 | "Since these are numbers we don't want to apply any warping, flip or zoom augmentation as they will cease to look like numbers. We only apply some random padding"
128 | ]
129 | },
130 | {
131 | "cell_type": "code",
132 | "execution_count": null,
133 | "metadata": {},
134 | "outputs": [],
135 | "source": [
136 | "tfms = ([*rand_pad(padding=3, size=size, mode='zeros')], [])"
137 | ]
138 | },
139 | {
140 | "cell_type": "code",
141 | "execution_count": null,
142 | "metadata": {},
143 | "outputs": [],
144 | "source": [
145 | "ll.transform(tfms)\n",
146 | "data = ll.databunch(bs=bs).normalize()"
147 | ]
148 | },
149 | {
150 | "cell_type": "markdown",
151 | "metadata": {},
152 | "source": [
153 | "Always a good idea to look at these stats if you're not sure what format the image is in. "
154 | ]
155 | },
156 | {
157 | "cell_type": "code",
158 | "execution_count": null,
159 | "metadata": {},
160 | "outputs": [],
161 | "source": [
162 | "def get_stats(tensor): return tensor.mean(),tensor.std(),tensor.min(),tensor.max()"
163 | ]
164 | },
165 | {
166 | "cell_type": "code",
167 | "execution_count": null,
168 | "metadata": {},
169 | "outputs": [],
170 | "source": [
171 | "get_stats(data.train_ds[1][-1].data)"
172 | ]
173 | },
174 | {
175 | "cell_type": "code",
176 | "execution_count": null,
177 | "metadata": {},
178 | "outputs": [],
179 | "source": [
180 | "def get_data(bs,size):\n",
181 | " return (GANItemList.from_folder(path, noise_sz=100,convert_mode='L')\n",
182 | " .split_none()\n",
183 | " .label_from_func(noop)\n",
184 | " .transform(tfms,size=size,tfm_y=True)\n",
185 | " .databunch(bs=bs).normalize(stats = [torch.tensor([0.5]), torch.tensor([0.5])],do_x=False,do_y=True))"
186 | ]
187 | },
188 | {
189 | "cell_type": "code",
190 | "execution_count": null,
191 | "metadata": {},
192 | "outputs": [],
193 | "source": [
194 | "data = get_data(bs,size)"
195 | ]
196 | },
197 | {
198 | "cell_type": "code",
199 | "execution_count": null,
200 | "metadata": {
201 | "scrolled": true
202 | },
203 | "outputs": [],
204 | "source": [
205 | "data.show_batch(2)"
206 | ]
207 | },
208 | {
209 | "cell_type": "markdown",
210 | "metadata": {},
211 | "source": [
212 | "We define some helper functions to get quickly get `conv_blocks` i.e layers with 2d convolutions,relu activation and batch normalization as we use them quite often and don't want to make mistakes"
213 | ]
214 | },
215 | {
216 | "cell_type": "code",
217 | "execution_count": null,
218 | "metadata": {},
219 | "outputs": [],
220 | "source": [
221 | "def conv2d(ni,nf,kernel_size=4,stride=2,bn=True):\n",
222 | " layers = [nn.Conv2d(ni,nf,kernel_size=kernel_size,stride=stride,padding=1,bias=False),nn.LeakyReLU(0.2,True)]\n",
223 | " if bn: layers.append(nn.BatchNorm2d(nf))\n",
224 | " return nn.Sequential(*layers)"
225 | ]
226 | },
227 | {
228 | "cell_type": "markdown",
229 | "metadata": {},
230 | "source": [
231 | "`nn.ConvTranspose2d` is a deconvolution or a fractional convolution. Instead of downsampling (which is what is done with stride > 1 convolutions), we upsample (increase the grid size)"
232 | ]
233 | },
234 | {
235 | "cell_type": "code",
236 | "execution_count": null,
237 | "metadata": {},
238 | "outputs": [],
239 | "source": [
240 | "def convt2d(ni,nf,kernel_size=4,stride=2,padding=1,bn=True): \n",
241 | " layers = [nn.ConvTranspose2d(ni,nf,kernel_size=kernel_size,stride=stride,padding=1,bias=False),nn.ReLU(True)]\n",
242 | " if bn: layers.append(nn.BatchNorm2d(nf))\n",
243 | " return nn.Sequential(*layers)"
244 | ]
245 | },
246 | {
247 | "cell_type": "code",
248 | "execution_count": null,
249 | "metadata": {},
250 | "outputs": [],
251 | "source": [
252 | "from fastai.vision.gan import AvgFlatten"
253 | ]
254 | },
255 | {
256 | "cell_type": "code",
257 | "execution_count": null,
258 | "metadata": {},
259 | "outputs": [],
260 | "source": [
261 | "critic = nn.Sequential(conv2d(3, 8,bn=False), #14\n",
262 | " conv2d(8,8,kernel_size=3,stride=1),#7\n",
263 | " conv2d(8,16),#7\n",
264 | " conv2d(16, 32), #4\n",
265 | " nn.Conv2d(32,1,kernel_size=3,stride=1,padding=0,bias=False), #2\n",
266 | " AvgFlatten()) #make into vector"
267 | ]
268 | },
269 | {
270 | "cell_type": "code",
271 | "execution_count": null,
272 | "metadata": {},
273 | "outputs": [],
274 | "source": [
275 | "generator = nn.Sequential(convt2d(100,8,padding=0), #2\n",
276 | " convt2d(8,16), #4\n",
277 | " convt2d(16,32,stride=3), #12\n",
278 | " convt2d(32,16), #24\n",
279 | " convt2d(16,16,stride=1), #24\n",
280 | " nn.ConvTranspose2d(16, 3, kernel_size=4, stride=1, padding=1,bias=False), #24\n",
281 | " nn.Tanh())"
282 | ]
283 | },
284 | {
285 | "cell_type": "code",
286 | "execution_count": null,
287 | "metadata": {},
288 | "outputs": [],
289 | "source": [
290 | "learn = GANLearner.wgan(data, generator, critic, switch_eval=False,\n",
291 | " opt_func = partial(optim.Adam, betas = (0.,0.99)), wd=0.)"
292 | ]
293 | },
294 | {
295 | "cell_type": "code",
296 | "execution_count": null,
297 | "metadata": {
298 | "scrolled": true
299 | },
300 | "outputs": [],
301 | "source": [
302 | "learn.fit(10,1e-4)"
303 | ]
304 | },
305 | {
306 | "cell_type": "code",
307 | "execution_count": null,
308 | "metadata": {},
309 | "outputs": [],
310 | "source": [
311 | "bs,size=128,64\n",
312 | "data = get_data(bs,size)"
313 | ]
314 | },
315 | {
316 | "cell_type": "code",
317 | "execution_count": null,
318 | "metadata": {},
319 | "outputs": [],
320 | "source": [
321 | "generator = basic_generator(in_size=size,n_channels=3,n_features=64,n_extra_layers=1)\n",
322 | "critic = basic_critic(in_size=size,n_channels=3,n_features=64,n_extra_layers=1)"
323 | ]
324 | },
325 | {
326 | "cell_type": "code",
327 | "execution_count": null,
328 | "metadata": {},
329 | "outputs": [],
330 | "source": [
331 | "learn = GANLearner.wgan(data, generator, critic, switch_eval=False,\n",
332 | " opt_func = partial(optim.Adam, betas = (0.,0.99)), wd=0.)"
333 | ]
334 | },
335 | {
336 | "cell_type": "code",
337 | "execution_count": null,
338 | "metadata": {
339 | "scrolled": true
340 | },
341 | "outputs": [],
342 | "source": [
343 | "learn.fit(10,1e-4)"
344 | ]
345 | },
346 | {
347 | "cell_type": "code",
348 | "execution_count": null,
349 | "metadata": {},
350 | "outputs": [],
351 | "source": []
352 | }
353 | ],
354 | "metadata": {
355 | "kernelspec": {
356 | "display_name": "Python 3",
357 | "language": "python",
358 | "name": "python3"
359 | },
360 | "language_info": {
361 | "codemirror_mode": {
362 | "name": "ipython",
363 | "version": 3
364 | },
365 | "file_extension": ".py",
366 | "mimetype": "text/x-python",
367 | "name": "python",
368 | "nbconvert_exporter": "python",
369 | "pygments_lexer": "ipython3",
370 | "version": "3.6.7"
371 | }
372 | },
373 | "nbformat": 4,
374 | "nbformat_minor": 2
375 | }
376 |
--------------------------------------------------------------------------------