![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
29 | ]
30 | },
31 | {
32 | "cell_type": "markdown",
33 | "metadata": {
34 | "id": "d8U6fiicXLUM"
35 | },
36 | "source": [
37 | "# **Google Colab Tutorial**\n",
38 | "\n",
39 | "\n",
40 | "Should you have any question, contact TA via "
29 | ]
30 | },
31 | {
32 | "cell_type": "markdown",
33 | "metadata": {
34 | "id": "eItOSGyKakne"
35 | },
36 | "source": [
37 | "# **Pytorch Tutorial**\n"
38 | ]
39 | },
40 | {
41 | "cell_type": "code",
42 | "metadata": {
43 | "id": "0AatWga6aTh_"
44 | },
45 | "source": [
46 | "import torch"
47 | ],
48 | "execution_count": 1,
49 | "outputs": []
50 | },
51 | {
52 | "cell_type": "markdown",
53 | "metadata": {
54 | "id": "hsZeHjlzan6W"
55 | },
56 | "source": [
57 | "**1. Pytorch Documentation Explanation with torch.max**\n",
58 | "\n"
59 | ]
60 | },
61 | {
62 | "cell_type": "code",
63 | "metadata": {
64 | "colab": {
65 | "base_uri": "https://localhost:8080/"
66 | },
67 | "id": "kWSXhKOLapaG",
68 | "outputId": "c06ac6e4-7f5a-4841-868c-35cddfdf671c"
69 | },
70 | "source": [
71 | "x = torch.randn(4,5)\n",
72 | "y = torch.randn(4,5)\n",
73 | "z = torch.randn(4,5)\n",
74 | "print(x)\n",
75 | "print(y)\n",
76 | "print(z)"
77 | ],
78 | "execution_count": 2,
79 | "outputs": [
80 | {
81 | "output_type": "stream",
82 | "text": [
83 | "tensor([[-0.2319, 0.7826, 0.5202, -0.1384, 0.6040],\n",
84 | " [-0.5828, -0.2883, 0.3508, 1.5920, 0.1148],\n",
85 | " [ 0.5468, -1.3432, -0.3204, 0.9779, -0.0244],\n",
86 | " [ 0.0494, -0.4104, 2.3866, 0.2881, 1.0155]])\n",
87 | "tensor([[-1.2664, 1.0201, 0.4730, -1.8292, 0.8858],\n",
88 | " [-2.5277, 0.1873, -0.1353, -0.6365, -0.7611],\n",
89 | " [ 0.9117, 0.3282, -0.5094, -1.0268, -1.8023],\n",
90 | " [-0.1177, -0.1617, -1.0410, 0.9007, 0.8385]])\n",
91 | "tensor([[-1.5717, -1.5700, -3.2840, -0.1819, 1.5947],\n",
92 | " [ 1.4966, -0.1422, 1.4710, -2.4297, 1.3951],\n",
93 | " [-0.5066, 0.5822, 0.9158, -0.2434, 0.3539],\n",
94 | " [-0.0303, 0.1333, -0.6207, 1.2630, 0.3462]])\n"
95 | ],
96 | "name": "stdout"
97 | }
98 | ]
99 | },
100 | {
101 | "cell_type": "code",
102 | "metadata": {
103 | "colab": {
104 | "base_uri": "https://localhost:8080/"
105 | },
106 | "id": "sQW_dRseasEt",
107 | "outputId": "2ab028f7-e7ed-4d58-ca3f-7bb3a16d16f7"
108 | },
109 | "source": [
110 | "# 1. max of entire tensor (torch.max(input) → Tensor)\n",
111 | "m = torch.max(x)\n",
112 | "print(m)"
113 | ],
114 | "execution_count": 3,
115 | "outputs": [
116 | {
117 | "output_type": "stream",
118 | "text": [
119 | "tensor(2.3866)\n"
120 | ],
121 | "name": "stdout"
122 | }
123 | ]
124 | },
125 | {
126 | "cell_type": "code",
127 | "metadata": {
128 | "colab": {
129 | "base_uri": "https://localhost:8080/"
130 | },
131 | "id": "1PqGQti8asmb",
132 | "outputId": "a9072e53-274f-4160-aa2c-6279bc38b130"
133 | },
134 | "source": [
135 | "# 2. max along a dimension (torch.max(input, dim, keepdim=False, *, out=None) → (Tensor, LongTensor))\n",
136 | "m, idx = torch.max(x,0)\n",
137 | "print(m)\n",
138 | "print(idx)"
139 | ],
140 | "execution_count": 4,
141 | "outputs": [
142 | {
143 | "output_type": "stream",
144 | "text": [
145 | "tensor([0.5468, 0.7826, 2.3866, 1.5920, 1.0155])\n",
146 | "tensor([2, 0, 3, 1, 3])\n"
147 | ],
148 | "name": "stdout"
149 | }
150 | ]
151 | },
152 | {
153 | "cell_type": "code",
154 | "metadata": {
155 | "colab": {
156 | "base_uri": "https://localhost:8080/"
157 | },
158 | "id": "jOlacFQNat5h",
159 | "outputId": "1f8b3a88-5589-486d-aaaf-a0e7bc825dab"
160 | },
161 | "source": [
162 | "# 2-2\n",
163 | "m, idx = torch.max(input=x,dim=0)\n",
164 | "print(m)\n",
165 | "print(idx)"
166 | ],
167 | "execution_count": 5,
168 | "outputs": [
169 | {
170 | "output_type": "stream",
171 | "text": [
172 | "tensor([0.5468, 0.7826, 2.3866, 1.5920, 1.0155])\n",
173 | "tensor([2, 0, 3, 1, 3])\n"
174 | ],
175 | "name": "stdout"
176 | }
177 | ]
178 | },
179 | {
180 | "cell_type": "code",
181 | "metadata": {
182 | "colab": {
183 | "base_uri": "https://localhost:8080/"
184 | },
185 | "id": "sLysSuo1avgS",
186 | "outputId": "d9e1163b-abcf-4701-91b2-ee879939638e"
187 | },
188 | "source": [
189 | "# 2-3\n",
190 | "m, idx = torch.max(x,0,False)\n",
191 | "print(m)\n",
192 | "print(idx)"
193 | ],
194 | "execution_count": 6,
195 | "outputs": [
196 | {
197 | "output_type": "stream",
198 | "text": [
199 | "tensor([0.5468, 0.7826, 2.3866, 1.5920, 1.0155])\n",
200 | "tensor([2, 0, 3, 1, 3])\n"
201 | ],
202 | "name": "stdout"
203 | }
204 | ]
205 | },
206 | {
207 | "cell_type": "code",
208 | "metadata": {
209 | "colab": {
210 | "base_uri": "https://localhost:8080/"
211 | },
212 | "id": "CwA6osCvawxl",
213 | "outputId": "cf4e1219-e22a-4b02-f037-41a733ad21db"
214 | },
215 | "source": [
216 | "# 2-4\n",
217 | "m, idx = torch.max(x,dim=0,keepdim=True)\n",
218 | "print(m)\n",
219 | "print(idx)"
220 | ],
221 | "execution_count": 7,
222 | "outputs": [
223 | {
224 | "output_type": "stream",
225 | "text": [
226 | "tensor([[0.5468, 0.7826, 2.3866, 1.5920, 1.0155]])\n",
227 | "tensor([[2, 0, 3, 1, 3]])\n"
228 | ],
229 | "name": "stdout"
230 | }
231 | ]
232 | },
233 | {
234 | "cell_type": "code",
235 | "metadata": {
236 | "colab": {
237 | "base_uri": "https://localhost:8080/"
238 | },
239 | "id": "IgYZZTO0ayKb",
240 | "outputId": "6d66eef2-a95d-420f-afad-60f24d2f0a89"
241 | },
242 | "source": [
243 | "# 2-5\n",
244 | "p = (m,idx)\n",
245 | "torch.max(x,0,False,out=p)\n",
246 | "print(p[0])\n",
247 | "print(p[1])\n"
248 | ],
249 | "execution_count": 8,
250 | "outputs": [
251 | {
252 | "output_type": "stream",
253 | "text": [
254 | "tensor([0.5468, 0.7826, 2.3866, 1.5920, 1.0155])\n",
255 | "tensor([2, 0, 3, 1, 3])\n"
256 | ],
257 | "name": "stdout"
258 | }
259 | ]
260 | },
261 | {
262 | "cell_type": "code",
263 | "metadata": {
264 | "colab": {
265 | "base_uri": "https://localhost:8080/",
266 | "height": 318
267 | },
268 | "id": "RuhKYn_8azTX",
269 | "outputId": "fd302638-1983-43cd-972e-4c38641c59a7"
270 | },
271 | "source": [
272 | "# 2-6\n",
273 | "p = (m,idx)\n",
274 | "torch.max(x,0,False,p)\n",
275 | "print(p[0])\n",
276 | "print(p[1])"
277 | ],
278 | "execution_count": 9,
279 | "outputs": [
280 | {
281 | "output_type": "error",
282 | "ename": "TypeError",
283 | "evalue": "ignored",
284 | "traceback": [
285 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
286 | "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
287 | "\u001b[0;32m"
29 | ]
30 | },
31 | {
32 | "cell_type": "markdown",
33 | "metadata": {
34 | "id": "wzVBe3h7Xh-2"
35 | },
36 | "source": [
37 | "\n",
38 | "# **HW15 Meta Learning: Few-shot Classification**\n",
39 | "\n",
40 | "Please mail to ntu-ml-2021spring-ta@googlegroups.com if you have any questions.\n",
41 | "\n",
42 | "Useful Links:\n",
43 | "1. [Go to hyperparameter setting.](#hyp)\n",
44 | "1. [Go to meta algorithm setting.](#modelsetting)\n",
45 | "1. [Go to main loop.](#mainloop)"
46 | ]
47 | },
48 | {
49 | "cell_type": "markdown",
50 | "metadata": {
51 | "id": "RdpzIMG6XsGK"
52 | },
53 | "source": [
54 | "## **Step 0: Check GPU**"
55 | ]
56 | },
57 | {
58 | "cell_type": "code",
59 | "metadata": {
60 | "id": "zjjHsZbaL7SV"
61 | },
62 | "source": [
63 | "!nvidia-smi"
64 | ],
65 | "execution_count": null,
66 | "outputs": []
67 | },
68 | {
69 | "cell_type": "code",
70 | "metadata": {
71 | "cellView": "form",
72 | "id": "gWpc6vW3MQhv"
73 | },
74 | "source": [
75 | "#@markdown ### Install `qqdm`\n",
76 | "# Check if installed\n",
77 | "try:\n",
78 | " import qqdm\n",
79 | "except:\n",
80 | " ! pip install qqdm > /dev/null 2>&1\n",
81 | "print(\"Done!\")"
82 | ],
83 | "execution_count": null,
84 | "outputs": []
85 | },
86 | {
87 | "cell_type": "markdown",
88 | "metadata": {
89 | "id": "bQ3wvyjnXwGX"
90 | },
91 | "source": [
92 | "## **Step 1: Download Data**\n",
93 | "\n",
94 | "Run the cell to download data, which has been pre-processed by TAs. \n",
95 | "The dataset has been augmented, so extra data augmentation is not required.\n"
96 | ]
97 | },
98 | {
99 | "cell_type": "code",
100 | "metadata": {
101 | "id": "g7Gt4Jucug41"
102 | },
103 | "source": [
104 | "workspace_dir = '.'\n",
105 | "\n",
106 | "# gdown is a package that downloads files from google drive\n",
107 | "!gdown --id 1FLDrQ0k-iJ-mk8ors0WItqvwgu0w9J0U \\\n",
108 | " --output \"{workspace_dir}/Omniglot.tar.gz\""
109 | ],
110 | "execution_count": null,
111 | "outputs": []
112 | },
113 | {
114 | "cell_type": "markdown",
115 | "metadata": {
116 | "id": "AMGFHI9XX9ms"
117 | },
118 | "source": [
119 | "### Decompress the dataset\n",
120 | "\n",
121 | "Since the dataset is quite large, please wait and observe the main program [here](#mainprogram). \n",
122 | "You can come back here later by [*back to pre-process*](#preprocess)."
123 | ]
124 | },
125 | {
126 | "cell_type": "code",
127 | "metadata": {
128 | "id": "AvvlAQBUug42"
129 | },
130 | "source": [
131 | "# Use `tar' command to decompress\n",
132 | "!tar -zxf \"{workspace_dir}/Omniglot.tar.gz\" \\\n",
133 | " -C \"{workspace_dir}/\""
134 | ],
135 | "execution_count": null,
136 | "outputs": []
137 | },
138 | {
139 | "cell_type": "markdown",
140 | "metadata": {
141 | "id": "T5P9eT0fYDqV"
142 | },
143 | "source": [
144 | "### Data Preview\n",
145 | "\n",
146 | "Just look at some data in the dataset."
147 | ]
148 | },
149 | {
150 | "cell_type": "code",
151 | "metadata": {
152 | "colab": {
153 | "base_uri": "https://localhost:8080/",
154 | "height": 297
155 | },
156 | "id": "7VtgHLurYE5x",
157 | "outputId": "961971b2-8b61-4d03-c06e-571a778ab52d"
158 | },
159 | "source": [
160 | "from PIL import Image\n",
161 | "from IPython.display import display\n",
162 | "for i in range(10, 20):\n",
163 | " im = Image.open(\"Omniglot/images_background/Japanese_(hiragana).0/character13/0500_\" + str (i) + \".png\")\n",
164 | " display(im)"
165 | ],
166 | "execution_count": null,
167 | "outputs": [
168 | {
169 | "output_type": "display_data",
170 | "data": {
171 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABEUlEQVR4nGP8z4AbMOGRQ5F8uekLmux/BNjHs+0/CkDWqcKJppMFQv1jYGBgYGb69w/FMsb/DAwMDD+bnjMwMHxbb6nIyMDAwMBWqoyk8/+zRwwMDD//vGFmYGBgYBDhQNbJ8Pvvp7t/OL0nBENMZUG1s2vFsz+C71nYsPnz5QSz/Vt1f//DGgj/GV0MbEMYTqDJQrz7Sc/62VVZZtefKIEAlfy3XthM3TBD7Qu2EGL07ZPWncHGzog9bP/+/v1EvvUv9rBlYvk37VsgE1ad/34+zeeL+4UaKywMDAwM7w7/unH46puSKlZUjYz/GRj+z278y2xkbW7Cy4ApyfD1838mQVY0lzLAAx47IDqBDQpJAN4Euv7fFejQAAAAAElFTkSuQmCC\n",
172 | "text/plain": [
173 | "![](\"https://i.imgur.com/9aHlvfX.png\")
\n",
367 | "\n",
368 | "On the 10-th line, what we take gradients on are those $\\theta$ representing \n",
369 | "**the original model parameters** (outer loop) instead of those in the \n",
370 | "**inner loop**, so we need to use `functional_forward` to compute the output \n",
371 | "logits of input image instead of `forward` in `nn.Module`.\n",
372 | "\n",
373 | "The following defines these functions.\n",
374 | "\n",
375 | ""
376 | ]
377 | },
378 | {
379 | "cell_type": "markdown",
380 | "metadata": {
381 | "id": "iuYQiPeQYc__"
382 | },
383 | "source": [
384 | "### Model block definition"
385 | ]
386 | },
387 | {
388 | "cell_type": "code",
389 | "metadata": {
390 | "id": "GgFbbKHYg3Hk"
391 | },
392 | "source": [
393 | "def ConvBlock(in_ch: int, out_ch: int):\n",
394 | " return nn.Sequential(\n",
395 | " nn.Conv2d(in_ch, out_ch, 3, padding=1),\n",
396 | " nn.BatchNorm2d(out_ch),\n",
397 | " nn.ReLU(),\n",
398 | " nn.MaxPool2d(kernel_size=2, stride=2)\n",
399 | " )\n",
400 | "\n",
401 | "def ConvBlockFunction(x, w, b, w_bn, b_bn):\n",
402 | " x = F.conv2d(x, w, b, padding=1)\n",
403 | " x = F.batch_norm(x,\n",
404 | " running_mean=None,\n",
405 | " running_var=None,\n",
406 | " weight=w_bn, bias=b_bn,\n",
407 | " training=True)\n",
408 | " x = F.relu(x)\n",
409 | " x = F.max_pool2d(x, kernel_size=2, stride=2)\n",
410 | " return x"
411 | ],
412 | "execution_count": null,
413 | "outputs": []
414 | },
415 | {
416 | "cell_type": "markdown",
417 | "metadata": {
418 | "id": "iQEzgWN7fi7B"
419 | },
420 | "source": [
421 | "### Model definition"
422 | ]
423 | },
424 | {
425 | "cell_type": "code",
426 | "metadata": {
427 | "id": "0bFBGEQoHQUW"
428 | },
429 | "source": [
430 | "class Classifier(nn.Module):\n",
431 | " def __init__(self, in_ch, k_way):\n",
432 | " super(Classifier, self).__init__()\n",
433 | " self.conv1 = ConvBlock(in_ch, 64)\n",
434 | " self.conv2 = ConvBlock(64, 64)\n",
435 | " self.conv3 = ConvBlock(64, 64)\n",
436 | " self.conv4 = ConvBlock(64, 64)\n",
437 | " self.logits = nn.Linear(64, k_way)\n",
438 | "\n",
439 | " def forward(self, x):\n",
440 | " x = self.conv1(x)\n",
441 | " x = self.conv2(x)\n",
442 | " x = self.conv3(x)\n",
443 | " x = self.conv4(x)\n",
444 | " x = x.view(x.shape[0], -1)\n",
445 | " x = self.logits(x)\n",
446 | " return x\n",
447 | "\n",
448 | " def functional_forward(self, x, params):\n",
449 | " '''\n",
450 | " Arguments:\n",
451 | " x: input images [batch, 1, 28, 28]\n",
452 | " params: model parameters, \n",
453 | " i.e. weights and biases of convolution\n",
454 | " and weights and biases of \n",
455 | " batch normalization\n",
456 | " type is an OrderedDict\n",
457 | "\n",
458 | " Arguments:\n",
459 | " x: input images [batch, 1, 28, 28]\n",
460 | " params: The model parameters, \n",
461 | " i.e. weights and biases of convolution \n",
462 | " and batch normalization layers\n",
463 | " It's an `OrderedDict`\n",
464 | " '''\n",
465 | " for block in [1, 2, 3, 4]:\n",
466 | " x = ConvBlockFunction(\n",
467 | " x,\n",
468 | " params[f'conv{block}.0.weight'],\n",
469 | " params[f'conv{block}.0.bias'],\n",
470 | " params.get(f'conv{block}.1.weight'),\n",
471 | " params.get(f'conv{block}.1.bias'))\n",
472 | " x = x.view(x.shape[0], -1)\n",
473 | " x = F.linear(x,\n",
474 | " params['logits.weight'],\n",
475 | " params['logits.bias'])\n",
476 | " return x"
477 | ],
478 | "execution_count": null,
479 | "outputs": []
480 | },
481 | {
482 | "cell_type": "markdown",
483 | "metadata": {
484 | "id": "gmJq_0B9Yj0G"
485 | },
486 | "source": [
487 | "### Create Label\n",
488 | "\n",
489 | "This function is used to create labels. \n",
490 | "In a N-way K-shot few-shot classification problem,\n",
491 | "each task has `n_way` classes, while there are `k_shot` images for each class. \n",
492 | "This is a function that creates such labels.\n"
493 | ]
494 | },
495 | {
496 | "cell_type": "code",
497 | "metadata": {
498 | "colab": {
499 | "base_uri": "https://localhost:8080/"
500 | },
501 | "id": "GQF5vgLvg5aX",
502 | "outputId": "5df41e04-290c-428b-b06f-cc749f09f027"
503 | },
504 | "source": [
505 | "def create_label(n_way, k_shot):\n",
506 | " return (torch.arange(n_way)\n",
507 | " .repeat_interleave(k_shot)\n",
508 | " .long())\n",
509 | "\n",
510 | "# Try to create labels for 5-way 2-shot setting\n",
511 | "create_label(5, 2)"
512 | ],
513 | "execution_count": null,
514 | "outputs": [
515 | {
516 | "output_type": "execute_result",
517 | "data": {
518 | "text/plain": [
519 | "tensor([0, 0, 1, 1, 2, 2, 3, 3, 4, 4])"
520 | ]
521 | },
522 | "metadata": {
523 | "tags": []
524 | },
525 | "execution_count": 9
526 | }
527 | ]
528 | },
529 | {
530 | "cell_type": "markdown",
531 | "metadata": {
532 | "id": "2nCFv9PGw50J"
533 | },
534 | "source": [
535 | "### Accuracy calculation"
536 | ]
537 | },
538 | {
539 | "cell_type": "code",
540 | "metadata": {
541 | "id": "FahDr0xQw50S"
542 | },
543 | "source": [
544 | "def calculate_accuracy(logits, val_label):\n",
545 | " \"\"\" utility function for accuracy calculation \"\"\"\n",
546 | " acc = np.asarray([(\n",
547 | " torch.argmax(logits, -1).cpu().numpy() == val_label.cpu().numpy())]\n",
548 | " ).mean() \n",
549 | " return acc"
550 | ],
551 | "execution_count": null,
552 | "outputs": []
553 | },
554 | {
555 | "cell_type": "markdown",
556 | "metadata": {
557 | "id": "9Hl7ro2mYzsI"
558 | },
559 | "source": [
560 | "### Define Dataset\n",
561 | "\n",
562 | "Define the dataset. \n",
563 | "The dataset returns images of a random character, with (`k_shot + q_query`) images, \n",
564 | "so the size of returned tensor is `[k_shot+q_query, 1, 28, 28]`. \n"
565 | ]
566 | },
567 | {
568 | "cell_type": "code",
569 | "metadata": {
570 | "id": "-tJ2mot9hHPb"
571 | },
572 | "source": [
573 | "class Omniglot(Dataset):\n",
574 | " def __init__(self, data_dir, k_way, q_query):\n",
575 | " self.file_list = [f for f in glob.glob(\n",
576 | " data_dir + \"**/character*\", \n",
577 | " recursive=True)]\n",
578 | " self.transform = transforms.Compose(\n",
579 | " [transforms.ToTensor()])\n",
580 | " self.n = k_way + q_query\n",
581 | "\n",
582 | " def __getitem__(self, idx):\n",
583 | " sample = np.arange(20)\n",
584 | "\n",
585 | " # For random sampling the characters we want.\n",
586 | " np.random.shuffle(sample) \n",
587 | " img_path = self.file_list[idx]\n",
588 | " img_list = [f for f in glob.glob(\n",
589 | " img_path + \"**/*.png\", recursive=True)]\n",
590 | " img_list.sort()\n",
591 | " imgs = [self.transform(\n",
592 | " Image.open(img_file)) \n",
593 | " for img_file in img_list]\n",
594 | " # `k_way + q_query` examples for each character\n",
595 | " imgs = torch.stack(imgs)[sample[:self.n]] \n",
596 | " return imgs\n",
597 | "\n",
598 | " def __len__(self):\n",
599 | " return len(self.file_list)"
600 | ],
601 | "execution_count": null,
602 | "outputs": []
603 | },
604 | {
605 | "cell_type": "markdown",
606 | "metadata": {
607 | "id": "Gm5iVp90Ylii"
608 | },
609 | "source": [
610 | "## **Step 3: Core MAML**\n",
611 | "\n",
612 | "Here is the main Meta Learning algorithm. \n",
613 | "The algorithm is exactly the same as the paper. \n",
614 | "What the function does is to update the parameters using \"the data of a meta-batch.\"\n",
615 | "Here we implement the second-order MAML (inner_train_step = 1), according to [the slides of meta learning in 2019 (p. 13 ~ p.18)](http://speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2019/Lecture/Meta1%20(v6).pdf#page=13&view=FitW)\n",
616 | "\n",
617 | "As for the mathematical derivation of the first-order version, please refer to [p.25 of the slides in 2019](http://speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2019/Lecture/Meta1%20(v6).pdf#page=25&view=FitW).\n",
618 | "\n",
619 | "The following is the algorithm with some explanation."
620 | ]
621 | },
622 | {
623 | "cell_type": "code",
624 | "metadata": {
625 | "id": "KjNxrWW_yNck"
626 | },
627 | "source": [
628 | "def OriginalMAML(\n",
629 | " model, optimizer, x, n_way, k_shot, q_query, loss_fn,\n",
630 | " inner_train_step=1, inner_lr=0.4, train=True):\n",
631 | " criterion, task_loss, task_acc = loss_fn, [], []\n",
632 | "\n",
633 | " for meta_batch in x:\n",
634 | " # Get data\n",
635 | " support_set = meta_batch[: n_way * k_shot] \n",
636 | " query_set = meta_batch[n_way * k_shot :] \n",
637 | " \n",
638 | " # Copy the params for inner loop\n",
639 | " fast_weights = OrderedDict(model.named_parameters())\n",
640 | " \n",
641 | " ### ---------- INNER TRAIN LOOP ---------- ###\n",
642 | " for inner_step in range(inner_train_step): \n",
643 | " # Simply training\n",
644 | " train_label = create_label(n_way, k_shot) \\\n",
645 | " .to(device)\n",
646 | " logits = model.functional_forward(\n",
647 | " support_set, fast_weights)\n",
648 | " loss = criterion(logits, train_label)\n",
649 | " # Inner gradients update! vvvvvvvvvvvvvvvvvvvv #\n",
650 | " \"\"\" Inner Loop Update \"\"\" #\n",
651 | " grads = torch.autograd.grad( #\n",
652 | " loss, fast_weights.values(), #\n",
653 | " create_graph=True) #\n",
654 | " # Perform SGD #\n",
655 | " fast_weights = OrderedDict( #\n",
656 | " (name, param - inner_lr * grad) #\n",
657 | " for ((name, param), grad) #\n",
658 | " in zip(fast_weights.items(), grads)) #\n",
659 | " # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #\n",
660 | "\n",
661 | " ### ---------- INNER VALID LOOP ---------- ###\n",
662 | " val_label = create_label(n_way, q_query).to(device)\n",
663 | " \n",
664 | " # Collect gradients for outer loop\n",
665 | " logits = model.functional_forward(\n",
666 | " query_set, fast_weights) \n",
667 | " loss = criterion(logits, val_label)\n",
668 | " task_loss.append(loss)\n",
669 | " task_acc.append(\n",
670 | " calculate_accuracy(logits, val_label))\n",
671 | "\n",
672 | " # Update outer loop\n",
673 | " model.train()\n",
674 | " optimizer.zero_grad()\n",
675 | "\n",
676 | " meta_batch_loss = torch.stack(task_loss).mean()\n",
677 | " if train:\n",
678 | " meta_batch_loss.backward() # <--- May change later!\n",
679 | " optimizer.step()\n",
680 | " task_acc = np.mean(task_acc)\n",
681 | " return meta_batch_loss, task_acc"
682 | ],
683 | "execution_count": null,
684 | "outputs": []
685 | },
686 | {
687 | "cell_type": "markdown",
688 | "metadata": {
689 | "id": "MF5ZahPdxKbp"
690 | },
691 | "source": [
692 | "## Variations of MAML\n",
693 | "\n",
694 | "### First-order approximation of MAML (FOMAML)\n",
695 | "\n",
696 | "Slightly modify the MAML mentioned earlier, applying first-order approximation to decrease amount of computation.\n",
697 | "\n",
698 | "### Almost No Inner Loop (ANIL)\n",
699 | "\n",
700 | "The algorithm from [this paper](https://arxiv.org/abs/1909.09157), using the technique of feature reuse to decrease amount of computation."
701 | ]
702 | },
703 | {
704 | "cell_type": "markdown",
705 | "metadata": {
706 | "id": "qyQ7ZUN4foh-"
707 | },
708 | "source": [
709 | "To finish the modification required, we need to change some blocks of the MAML algorithm. \n",
710 | "Below, we have replace three parts that may be modified as functions. \n",
711 | "Please choose to replace the functions with their alternative versions to complete the algorithm."
712 | ]
713 | },
714 | {
715 | "cell_type": "markdown",
716 | "metadata": {
717 | "id": "Ne5cOja0H8H7"
718 | },
719 | "source": [
720 | "### Part 1: Inner loop update"
721 | ]
722 | },
723 | {
724 | "cell_type": "markdown",
725 | "metadata": {
726 | "id": "LChAX51sIFwi"
727 | },
728 | "source": [
729 | "MAML"
730 | ]
731 | },
732 | {
733 | "cell_type": "code",
734 | "metadata": {
735 | "id": "Aqgb0kEVzQol"
736 | },
737 | "source": [
738 | "def inner_update_MAML(fast_weights, loss, inner_lr):\n",
739 | " \"\"\" Inner Loop Update \"\"\"\n",
740 | " grads = torch.autograd.grad(\n",
741 | " loss, fast_weights.values(), create_graph=True)\n",
742 | " # Perform SGD\n",
743 | " fast_weights = OrderedDict(\n",
744 | " (name, param - inner_lr * grad)\n",
745 | " for ((name, param), grad) in zip(fast_weights.items(), grads))\n",
746 | " return fast_weights"
747 | ],
748 | "execution_count": null,
749 | "outputs": []
750 | },
751 | {
752 | "cell_type": "markdown",
753 | "metadata": {
754 | "id": "QnQ_BN-L2Gd7"
755 | },
756 | "source": [
757 | "Alternatives"
758 | ]
759 | },
760 | {
761 | "cell_type": "code",
762 | "metadata": {
763 | "id": "Ug5LIO6V15cd"
764 | },
765 | "source": [
766 | "def inner_update_alt1(fast_weights, loss, inner_lr):\n",
767 | " grads = torch.autograd.grad(\n",
768 | " loss, fast_weights.values(), create_graph=False)\n",
769 | " # Perform SGD\n",
770 | " fast_weights = OrderedDict(\n",
771 | " (name, param - inner_lr * grad)\n",
772 | " for ((name, param), grad) in zip(fast_weights.items(), grads))\n",
773 | " return fast_weights\n",
774 | "\n",
775 | "def inner_update_alt2(fast_weights, loss, inner_lr):\n",
776 | " grads = torch.autograd.grad(\n",
777 | " loss, list(fast_weights.values())[-2:], create_graph=True)\n",
778 | " # Split out the logits\n",
779 | " for ((name, param), grad) in zip(\n",
780 | " list(fast_weights.items())[-2:], grads):\n",
781 | " fast_weights[name] = param - inner_lr * grad\n",
782 | " return fast_weights"
783 | ],
784 | "execution_count": null,
785 | "outputs": []
786 | },
787 | {
788 | "cell_type": "markdown",
789 | "metadata": {
790 | "id": "1ZfaWPMt164t"
791 | },
792 | "source": [
793 | "### Part 2: Collect gradients"
794 | ]
795 | },
796 | {
797 | "cell_type": "markdown",
798 | "metadata": {
799 | "id": "-W7zL2nN164u"
800 | },
801 | "source": [
802 | "MAML \n",
803 | "(Actually do nothing as gradients are computed by PyTorch automatically.)"
804 | ]
805 | },
806 | {
807 | "cell_type": "code",
808 | "metadata": {
809 | "id": "sgcuPPm2zSFL"
810 | },
811 | "source": [
812 | "def collect_gradients_MAML(\n",
813 | " special_grad: OrderedDict, fast_weights, model, len_data):\n",
814 | " \"\"\" Actually do nothing (just backwards later) \"\"\"\n",
815 | " return special_grad"
816 | ],
817 | "execution_count": null,
818 | "outputs": []
819 | },
820 | {
821 | "cell_type": "markdown",
822 | "metadata": {
823 | "id": "2OxEME6l2QOO"
824 | },
825 | "source": [
826 | "Alternatives"
827 | ]
828 | },
829 | {
830 | "cell_type": "code",
831 | "metadata": {
832 | "id": "fWLYwZlM2RZO"
833 | },
834 | "source": [
835 | "def collect_gradients_alt(\n",
836 | " special_grad: OrderedDict, fast_weights, model, len_data):\n",
837 | " \"\"\" Special gradient calculation \"\"\"\n",
838 | " diff = OrderedDict(\n",
839 | " (name, params - fast_weights[name]) \n",
840 | " for (name, params) in model.named_parameters())\n",
841 | " for name in diff:\n",
842 | " special_grad[name] = special_grad.get(name, 0) + diff[name] / len_data\n",
843 | " return special_grad"
844 | ],
845 | "execution_count": null,
846 | "outputs": []
847 | },
848 | {
849 | "cell_type": "markdown",
850 | "metadata": {
851 | "id": "ahqE-Sf92TID"
852 | },
853 | "source": [
854 | "### Part 3: Outer loop gradients calculation"
855 | ]
856 | },
857 | {
858 | "cell_type": "markdown",
859 | "metadata": {
860 | "id": "-wr0hSd02TIE"
861 | },
862 | "source": [
863 | "MAML \n",
864 | "(Simply call PyTorch `backward`.)"
865 | ]
866 | },
867 | {
868 | "cell_type": "code",
869 | "metadata": {
870 | "id": "_hBSQ02xzTXb"
871 | },
872 | "source": [
873 | "def outer_update_MAML(model, meta_batch_loss, grad_tensors):\n",
874 | " \"\"\" Simply backwards \"\"\"\n",
875 | " meta_batch_loss.backward()"
876 | ],
877 | "execution_count": null,
878 | "outputs": []
879 | },
880 | {
881 | "cell_type": "markdown",
882 | "metadata": {
883 | "id": "Q4zxf6yr2TIE"
884 | },
885 | "source": [
886 | "Alternatives"
887 | ]
888 | },
889 | {
890 | "cell_type": "code",
891 | "metadata": {
892 | "id": "DEyCwYmI2bdC"
893 | },
894 | "source": [
895 | "def outer_update_alt(model, meta_batch_loss, grad_tensors):\n",
896 | " \"\"\" Replace the gradients\n",
897 | " with precalculated tensors \"\"\"\n",
898 | " for (name, params) in model.named_parameters():\n",
899 | " params.grad = grad_tensors[name]"
900 | ],
901 | "execution_count": null,
902 | "outputs": []
903 | },
904 | {
905 | "cell_type": "markdown",
906 | "metadata": {
907 | "id": "z1jck3KE2g1D"
908 | },
909 | "source": [
910 | "### Complete the algorithm\n",
911 | "Here we have wrapped the algorithm in `MetaAlgorithmGenerator`. \n",
912 | "You can get your modified algorithm by filling in like this:\n",
913 | "```python\n",
914 | "MyAlgorithm = MetaAlgorithmGenerator(inner_update=inner_update_alt2)\n",
915 | "```\n",
916 | "Default the three blocks will be filled with that of `MAML`."
917 | ]
918 | },
919 | {
920 | "cell_type": "code",
921 | "metadata": {
922 | "id": "XosNxVMDxL6V"
923 | },
924 | "source": [
925 | "def MetaAlgorithmGenerator(\n",
926 | " inner_update = inner_update_MAML, \n",
927 | " collect_gradients = collect_gradients_MAML, \n",
928 | " outer_update = outer_update_MAML):\n",
929 | "\n",
930 | " global calculate_accuracy\n",
931 | "\n",
932 | " def MetaAlgorithm(\n",
933 | " model, optimizer, x, n_way, k_shot, q_query, loss_fn,\n",
934 | " inner_train_step=1, inner_lr=0.4, train=True): \n",
935 | " criterion = loss_fn\n",
936 | " task_loss, task_acc = [], []\n",
937 | " special_grad = OrderedDict() # Added for variants!\n",
938 | "\n",
939 | " for meta_batch in x:\n",
940 | " support_set = meta_batch[: n_way * k_shot] \n",
941 | " query_set = meta_batch[n_way * k_shot :] \n",
942 | " \n",
943 | " fast_weights = OrderedDict(model.named_parameters())\n",
944 | " \n",
945 | " ### ---------- INNER TRAIN LOOP ---------- ###\n",
946 | " for inner_step in range(inner_train_step): \n",
947 | " train_label = create_label(n_way, k_shot).to(device)\n",
948 | " logits = model.functional_forward(support_set, fast_weights)\n",
949 | " loss = criterion(logits, train_label)\n",
950 | "\n",
951 | " fast_weights = inner_update(fast_weights, loss, inner_lr)\n",
952 | "\n",
953 | " ### ---------- INNER VALID LOOP ---------- ###\n",
954 | " val_label = create_label(n_way, q_query).to(device)\n",
955 | " # FIXME: W for val?\n",
956 | " special_grad = collect_gradients(\n",
957 | " special_grad, fast_weights, model, len(x))\n",
958 | " \n",
959 | " # Collect gradients for outer loop\n",
960 | " logits = model.functional_forward(query_set, fast_weights) \n",
961 | " loss = criterion(logits, val_label)\n",
962 | " task_loss.append(loss)\n",
963 | " task_acc.append(calculate_accuracy(logits, val_label))\n",
964 | "\n",
965 | " # Update outer loop\n",
966 | " model.train()\n",
967 | " optimizer.zero_grad()\n",
968 | "\n",
969 | " meta_batch_loss = torch.stack(task_loss).mean()\n",
970 | " if train:\n",
971 | " # Notice the update part!\n",
972 | " outer_update(model, meta_batch_loss, special_grad)\n",
973 | " optimizer.step()\n",
974 | " task_acc = np.mean(task_acc)\n",
975 | " return meta_batch_loss, task_acc\n",
976 | " return MetaAlgorithm"
977 | ],
978 | "execution_count": null,
979 | "outputs": []
980 | },
981 | {
982 | "cell_type": "code",
983 | "metadata": {
984 | "id": "jEsPtV-GzbDv",
985 | "cellView": "form"
986 | },
987 | "source": [
988 | "#@title Here is the answer hidden, please fill in yourself!\n",
989 | "Give_me_the_answer = True #@param {\"type\": \"boolean\"}\n",
990 | "\n",
991 | "def HiddenAnswer():\n",
992 | " MAML = MetaAlgorithmGenerator()\n",
993 | " FOMAML = MetaAlgorithmGenerator(inner_update=inner_update_alt1)\n",
994 | " ANIL = MetaAlgorithmGenerator(inner_update=inner_update_alt2)\n",
995 | " return MAML, FOMAML, ANIL"
996 | ],
997 | "execution_count": null,
998 | "outputs": []
999 | },
1000 | {
1001 | "cell_type": "code",
1002 | "metadata": {
1003 | "id": "2P__5N2Yz9O4"
1004 | },
1005 | "source": [
1006 | "# `HiddenAnswer` is hidden in the last cell.\n",
1007 | "if Give_me_the_answer:\n",
1008 | " MAML, FOMAML, ANIL = HiddenAnswer()\n",
1009 | "else: \n",
1010 | " # TODO: Please fill in the function names \\\n",
1011 | " # as the function arguments to finish the algorithm.\n",
1012 | " MAML = MetaAlgorithmGenerator()\n",
1013 | " FOMAML = MetaAlgorithmGenerator()\n",
1014 | " ANIL = MetaAlgorithmGenerator()"
1015 | ],
1016 | "execution_count": null,
1017 | "outputs": []
1018 | },
1019 | {
1020 | "cell_type": "markdown",
1021 | "metadata": {
1022 | "id": "nBoRBhVlZAST"
1023 | },
1024 | "source": [
1025 | "## **Step 4: Initialization**\n",
1026 | "\n",
1027 | "After defining all components we need, the following initialize a model before training."
1028 | ]
1029 | },
1030 | {
1031 | "cell_type": "markdown",
1032 | "metadata": {
1033 | "id": "Ip-i7aseftUF"
1034 | },
1035 | "source": [
1036 | "\n",
1037 | "### Hyperparameters \n",
1038 | "[Go back to top!](#top)"
1039 | ]
1040 | },
1041 | {
1042 | "cell_type": "code",
1043 | "metadata": {
1044 | "id": "0wFHmVcBhE4M"
1045 | },
1046 | "source": [
1047 | "n_way = 5\n",
1048 | "k_shot = 1\n",
1049 | "q_query = 1\n",
1050 | "inner_train_step = 1\n",
1051 | "inner_lr = 0.4\n",
1052 | "meta_lr = 0.001\n",
1053 | "meta_batch_size = 32\n",
1054 | "max_epoch = 30\n",
1055 | "eval_batches = test_batches = 20\n",
1056 | "train_data_path = './Omniglot/images_background/'\n",
1057 | "test_data_path = './Omniglot/images_evaluation/' "
1058 | ],
1059 | "execution_count": null,
1060 | "outputs": []
1061 | },
1062 | {
1063 | "cell_type": "markdown",
1064 | "metadata": {
1065 | "id": "Uvzo7NVpfu5V"
1066 | },
1067 | "source": [
1068 | "### Dataloader initialization"
1069 | ]
1070 | },
1071 | {
1072 | "cell_type": "code",
1073 | "metadata": {
1074 | "id": "3I13GJavhP0_"
1075 | },
1076 | "source": [
1077 | "def dataloader_init(datasets, num_workers=2):\n",
1078 | " train_set, val_set, test_set = datasets\n",
1079 | " train_loader = DataLoader(train_set,\n",
1080 | " # The \"batch_size\" here is not \\\n",
1081 | " # the meta batch size, but \\\n",
1082 | " # how many different \\\n",
1083 | " # characters in a task, \\\n",
1084 | " # i.e. the \"n_way\" in \\\n",
1085 | " # few-shot classification.\n",
1086 | " batch_size=n_way,\n",
1087 | " num_workers=num_workers,\n",
1088 | " shuffle=True,\n",
1089 | " drop_last=True)\n",
1090 | " val_loader = DataLoader(val_set,\n",
1091 | " batch_size=n_way,\n",
1092 | " num_workers=num_workers,\n",
1093 | " shuffle=True,\n",
1094 | " drop_last=True)\n",
1095 | " test_loader = DataLoader(test_set,\n",
1096 | " batch_size=n_way,\n",
1097 | " num_workers=num_workers,\n",
1098 | " shuffle=True,\n",
1099 | " drop_last=True)\n",
1100 | " train_iter = iter(train_loader)\n",
1101 | " val_iter = iter(val_loader)\n",
1102 | " test_iter = iter(test_loader)\n",
1103 | " return (train_loader, val_loader, test_loader), \\\n",
1104 | " (train_iter, val_iter, test_iter)\n",
1105 | "\n",
1106 | "train_set, val_set = torch.utils.data.random_split(\n",
1107 | " Omniglot(train_data_path, k_shot, q_query), [3200, 656])\n",
1108 | "test_set = Omniglot(test_data_path, k_shot, q_query)\n",
1109 | "\n",
1110 | "(train_loader, val_loader, test_loader), \\\n",
1111 | "(train_iter, val_iter, test_iter) = dataloader_init(\n",
1112 | " (train_set, val_set, test_set))"
1113 | ],
1114 | "execution_count": null,
1115 | "outputs": []
1116 | },
1117 | {
1118 | "cell_type": "markdown",
1119 | "metadata": {
1120 | "id": "KVund--bfw0e"
1121 | },
1122 | "source": [
1123 | "### Model & optimizer initialization"
1124 | ]
1125 | },
1126 | {
1127 | "cell_type": "code",
1128 | "metadata": {
1129 | "id": "Kxug882ihF2B"
1130 | },
1131 | "source": [
1132 | "def model_init():\n",
1133 | " meta_model = Classifier(1, n_way).to(device)\n",
1134 | " optimizer = torch.optim.Adam(meta_model.parameters(), \n",
1135 | " lr=meta_lr)\n",
1136 | " loss_fn = nn.CrossEntropyLoss().to(device)\n",
1137 | " return meta_model, optimizer, loss_fn\n",
1138 | "\n",
1139 | "meta_model, optimizer, loss_fn = model_init()"
1140 | ],
1141 | "execution_count": null,
1142 | "outputs": []
1143 | },
1144 | {
1145 | "cell_type": "markdown",
1146 | "metadata": {
1147 | "id": "gj8cLRNLf2zg"
1148 | },
1149 | "source": [
1150 | "### Utility function to get a meta-batch"
1151 | ]
1152 | },
1153 | {
1154 | "cell_type": "code",
1155 | "metadata": {
1156 | "id": "zrkCSsxOhC-N"
1157 | },
1158 | "source": [
1159 | "def get_meta_batch(meta_batch_size,\n",
1160 | " k_shot, q_query, \n",
1161 | " data_loader, iterator):\n",
1162 | " data = []\n",
1163 | " for _ in range(meta_batch_size):\n",
1164 | " try:\n",
1165 | " # a \"task_data\" tensor is representing \\\n",
1166 | " # the data of a task, with size of \\\n",
1167 | " # [n_way, k_shot+q_query, 1, 28, 28]\n",
1168 | " task_data = iterator.next() \n",
1169 | " except StopIteration:\n",
1170 | " iterator = iter(data_loader)\n",
1171 | " task_data = iterator.next()\n",
1172 | " train_data = (task_data[:, :k_shot]\n",
1173 | " .reshape(-1, 1, 28, 28))\n",
1174 | " val_data = (task_data[:, k_shot:]\n",
1175 | " .reshape(-1, 1, 28, 28))\n",
1176 | " task_data = torch.cat(\n",
1177 | " (train_data, val_data), 0)\n",
1178 | " data.append(task_data)\n",
1179 | " return torch.stack(data).to(device), iterator"
1180 | ],
1181 | "execution_count": null,
1182 | "outputs": []
1183 | },
1184 | {
1185 | "cell_type": "markdown",
1186 | "metadata": {
1187 | "id": "O5JCtob4fyh_"
1188 | },
1189 | "source": [
1190 | "\n",
1191 | "### Choose the meta learning algorithm\n",
1192 | "[Go back to top!](#top)"
1193 | ]
1194 | },
1195 | {
1196 | "cell_type": "code",
1197 | "metadata": {
1198 | "id": "3av6pAI7OxOP"
1199 | },
1200 | "source": [
1201 | "# You can change this to `FOMAML` or `ANIL`\n",
1202 | "MetaAlgorithm = MAML"
1203 | ],
1204 | "execution_count": null,
1205 | "outputs": []
1206 | },
1207 | {
1208 | "cell_type": "markdown",
1209 | "metadata": {
1210 | "id": "pWQczA3FwjEG"
1211 | },
1212 | "source": [
1213 | "\n",
1214 | "## **Step 5: Main program for training & testing**"
1215 | ]
1216 | },
1217 | {
1218 | "cell_type": "markdown",
1219 | "metadata": {
1220 | "id": "8EirEnaof7ep"
1221 | },
1222 | "source": [
1223 | "### Start training!\n",
1224 | "\n",
1225 | "[Go back to top!](#top)"
1226 | ]
1227 | },
1228 | {
1229 | "cell_type": "code",
1230 | "metadata": {
1231 | "id": "JQZjJrLAhBWw"
1232 | },
1233 | "source": [
1234 | "for epoch in range(max_epoch):\n",
1235 | " print(\"Epoch %d\" % (epoch + 1))\n",
1236 | " train_meta_loss = []\n",
1237 | " train_acc = []\n",
1238 | " # The \"step\" here is a meta-gradinet update step\n",
1239 | " for step in tqdm(range(\n",
1240 | " len(train_loader) // meta_batch_size)): \n",
1241 | " x, train_iter = get_meta_batch(\n",
1242 | " meta_batch_size, k_shot, q_query, \n",
1243 | " train_loader, train_iter)\n",
1244 | " meta_loss, acc = MetaAlgorithm(\n",
1245 | " meta_model, optimizer, x, \n",
1246 | " n_way, k_shot, q_query, loss_fn)\n",
1247 | " train_meta_loss.append(meta_loss.item())\n",
1248 | " train_acc.append(acc)\n",
1249 | " print(\" Loss : \", \"%.3f\" % (np.mean(train_meta_loss)), end='\\t')\n",
1250 | " print(\" Accuracy: \", \"%.3f %%\" % (np.mean(train_acc) * 100))\n",
1251 | "\n",
1252 | " # See the validation accuracy after each epoch.\n",
1253 | " # Early stopping is welcomed to implement.\n",
1254 | " val_acc = []\n",
1255 | " for eval_step in tqdm(range(\n",
1256 | " len(val_loader) // (eval_batches))):\n",
1257 | " x, val_iter = get_meta_batch(\n",
1258 | " eval_batches, k_shot, q_query, \n",
1259 | " val_loader, val_iter)\n",
1260 | " # We update three inner steps when testing.\n",
1261 | " _, acc = MetaAlgorithm(meta_model, optimizer, x, \n",
1262 | " n_way, k_shot, q_query, \n",
1263 | " loss_fn, \n",
1264 | " inner_train_step=3, \n",
1265 | " train=False) \n",
1266 | " val_acc.append(acc)\n",
1267 | " print(\" Validation accuracy: \", \"%.3f %%\" % (np.mean(val_acc) * 100))"
1268 | ],
1269 | "execution_count": null,
1270 | "outputs": []
1271 | },
1272 | {
1273 | "cell_type": "markdown",
1274 | "metadata": {
1275 | "id": "u5Ew8-POf9sw"
1276 | },
1277 | "source": [
1278 | "### Testing the result"
1279 | ]
1280 | },
1281 | {
1282 | "cell_type": "code",
1283 | "metadata": {
1284 | "id": "CYN_zGB3g_5_"
1285 | },
1286 | "source": [
1287 | "test_acc = []\n",
1288 | "for test_step in tqdm(range(\n",
1289 | " len(test_loader) // (test_batches))):\n",
1290 | " x, test_iter = get_meta_batch(\n",
1291 | " test_batches, k_shot, q_query, \n",
1292 | " test_loader, test_iter)\n",
1293 | " # When testing, we update 3 inner-steps\n",
1294 | " _, acc = MetaAlgorithm(meta_model, optimizer, x, \n",
1295 | " n_way, k_shot, q_query, loss_fn, \n",
1296 | " inner_train_step=3, train=False)\n",
1297 | " test_acc.append(acc)\n",
1298 | "print(\" Testing accuracy: \", \"%.3f %%\" % (np.mean(test_acc) * 100))"
1299 | ],
1300 | "execution_count": null,
1301 | "outputs": []
1302 | },
1303 | {
1304 | "cell_type": "markdown",
1305 | "metadata": {
1306 | "id": "rtD8X3RLf-6w"
1307 | },
1308 | "source": [
1309 | "## **Reference**\n",
1310 | "1. Chelsea Finn, Pieter Abbeel, & Sergey Levine. (2017). [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.](https://arxiv.org/abs/1909.09157)\n",
1311 | "1. Aniruddh Raghu, Maithra Raghu, Samy Bengio, & Oriol Vinyals. (2020). [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML.](https://arxiv.org/abs/1909.09157)"
1312 | ]
1313 | }
1314 | ]
1315 | }
1316 |
--------------------------------------------------------------------------------
/HW2/HW02.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW2/HW02.pdf
--------------------------------------------------------------------------------
/HW2/README2_1.md:
--------------------------------------------------------------------------------
1 | # Homework 2-1: Phoneme Classification
2 |
3 | Detail : [Link](https://speech.ee.ntu.edu.tw/~hylee/ml/ml2021-course-data/hw/HW02/HW02.pdf)
4 |
5 | Code : [Link](homework2_1.ipynb)
6 |
--------------------------------------------------------------------------------
/HW2/README2_2.md:
--------------------------------------------------------------------------------
1 | # Homework 2-2: Hessian Matrix
2 |
3 | Detail : [Link](https://speech.ee.ntu.edu.tw/~hylee/ml/ml2021-course-data/hw/HW02/HW02.pdf)
4 |
5 | [code](homework2_2.ipynb)
6 |
--------------------------------------------------------------------------------
/HW2/homework2_2.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "nbformat": 4,
3 | "nbformat_minor": 0,
4 | "metadata": {
5 | "colab": {
6 | "name": "Untitled0.ipynb",
7 | "provenance": [],
8 | "authorship_tag": "ABX9TyNh31QPg5onyqlA0B16rpPP",
9 | "include_colab_link": true
10 | },
11 | "kernelspec": {
12 | "name": "python3",
13 | "display_name": "Python 3"
14 | },
15 | "language_info": {
16 | "name": "python"
17 | }
18 | },
19 | "cells": [
20 | {
21 | "cell_type": "markdown",
22 | "metadata": {
23 | "id": "view-in-github",
24 | "colab_type": "text"
25 | },
26 | "source": [
27 | ""
28 | ]
29 | },
30 | {
31 | "cell_type": "markdown",
32 | "metadata": {
33 | "id": "mEVFnEg4a6yE"
34 | },
35 | "source": [
36 | "# **Homework 2-2 Hessian Matrix**\n",
37 | "\n",
38 | "## Hessian Matrix\n",
39 | "Imagine we are training a neural network and we are trying to find out whether the model is at **local minima like, saddle point, or none of the above**. We can make our decision by calculating the Hessian matrix.\n",
40 | "\n",
41 | "In practice, it is really hard to find a point where the gradient equals zero or all of the eigenvalues in Hessian matrix are greater than zero. In this homework, we make the following two assumptions:\n",
42 | "1. View gradient norm less than 1e-3 as **gradient equals to zero**.\n",
43 | "2. If minimum ratio is greater than 0.5 and gradient norm is less than 1e-3, then we assume that the model is at “local minima like”.\n",
44 | "\n",
45 | "> Minimum ratio is defined as the proportion of positive eigenvalues."
46 | ]
47 | },
48 | {
49 | "cell_type": "markdown",
50 | "metadata": {
51 | "id": "AfEv2ee-bATF"
52 | },
53 | "source": [
54 | "## IMPORTANT NOTICE\n",
55 | "In this homework, students with different student IDs will get different answers. Make sure to fill in your `student_id` in the following block correctly. Otherwise, your code may not run correctly and you will get a wrong answer."
56 | ]
57 | },
58 | {
59 | "cell_type": "code",
60 | "metadata": {
61 | "id": "LR4SxRxQaqVo"
62 | },
63 | "source": [
64 | "student_id = '40673034h' # fill with your student ID\n",
65 | "\n",
66 | "assert student_id != 'your_student_id', 'Please fill in your student_id before you start.'"
67 | ],
68 | "execution_count": 1,
69 | "outputs": []
70 | },
71 | {
72 | "cell_type": "markdown",
73 | "metadata": {
74 | "id": "CS0o8eJsbEIX"
75 | },
76 | "source": [
77 | "## Calculate Hessian Matrix\n",
78 | "The computation of Hessian is done by TA, you don't need to and shouldn't change the following code. The only thing you need to do is to run the following blocks and determine whether the model is at `local minima like`, `saddle point`, or `none of the above` according to the value of `gradient norm` and `minimum ratio`."
79 | ]
80 | },
81 | {
82 | "cell_type": "markdown",
83 | "metadata": {
84 | "id": "cr3BIXy4bG0Y"
85 | },
86 | "source": [
87 | "### Install Package to Compute Hessian.\n",
88 | "\n",
89 | "The autograd-lib library is used to compute Hessian matrix. You can check the full document here https://github.com/cybertronai/autograd-lib."
90 | ]
91 | },
92 | {
93 | "cell_type": "code",
94 | "metadata": {
95 | "colab": {
96 | "base_uri": "https://localhost:8080/"
97 | },
98 | "id": "s5glDmIebEqw",
99 | "outputId": "1b536f61-3e6b-46bb-8802-c293c117025a"
100 | },
101 | "source": [
102 | "!pip install autograd-lib"
103 | ],
104 | "execution_count": 2,
105 | "outputs": [
106 | {
107 | "output_type": "stream",
108 | "text": [
109 | "Collecting autograd-lib\n",
110 | " Downloading https://files.pythonhosted.org/packages/b9/ac/a3927e1e2a886a12b914bce86965bec3b925ad14ffb696b2f84d9f8ee949/autograd_lib-0.0.7-py3-none-any.whl\n",
111 | "Requirement already satisfied: gin-config in /usr/local/lib/python3.7/dist-packages (from autograd-lib) (0.4.0)\n",
112 | "Collecting pytorch-lightning\n",
113 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/58/01/5df6324efdc3f79025ea7eaf19478936c401a16dae4fd3fbd29f7d426974/pytorch_lightning-1.2.6-py3-none-any.whl (829kB)\n",
114 | "\u001b[K |████████████████████████████████| 839kB 5.2MB/s \n",
115 | "\u001b[?25hRequirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from autograd-lib) (0.11.1)\n",
116 | "Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning->autograd-lib) (1.19.5)\n",
117 | "Collecting torchmetrics>=0.2.0\n",
118 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/3a/42/d984612cabf005a265aa99c8d4ab2958e37b753aafb12f31c81df38751c8/torchmetrics-0.2.0-py3-none-any.whl (176kB)\n",
119 | "\u001b[K |████████████████████████████████| 184kB 8.0MB/s \n",
120 | "\u001b[?25hRequirement already satisfied: tqdm>=4.41.0 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning->autograd-lib) (4.41.1)\n",
121 | "Collecting fsspec[http]>=0.8.1\n",
122 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/91/0d/a6bfee0ddf47b254286b9bd574e6f50978c69897647ae15b14230711806e/fsspec-0.8.7-py3-none-any.whl (103kB)\n",
123 | "\u001b[K |████████████████████████████████| 112kB 11.4MB/s \n",
124 | "\u001b[?25hRequirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning->autograd-lib) (2.4.1)\n",
125 | "Collecting PyYAML!=5.4.*,>=5.1\n",
126 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB)\n",
127 | "\u001b[K |████████████████████████████████| 276kB 10.7MB/s \n",
128 | "\u001b[?25hRequirement already satisfied: torch>=1.4 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning->autograd-lib) (1.8.1+cu101)\n",
129 | "Collecting future>=0.17.1\n",
130 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB)\n",
131 | "\u001b[K |████████████████████████████████| 829kB 12.7MB/s \n",
132 | "\u001b[?25hRequirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from seaborn->autograd-lib) (1.4.1)\n",
133 | "Requirement already satisfied: pandas>=0.23 in /usr/local/lib/python3.7/dist-packages (from seaborn->autograd-lib) (1.1.5)\n",
134 | "Requirement already satisfied: matplotlib>=2.2 in /usr/local/lib/python3.7/dist-packages (from seaborn->autograd-lib) (3.2.2)\n",
135 | "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from fsspec[http]>=0.8.1->pytorch-lightning->autograd-lib) (3.8.1)\n",
136 | "Collecting aiohttp; extra == \"http\"\n",
137 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/88/c0/5890b4c8b04a79b7360e8fe4490feb0bb3ab179743f199f0e6220cebd568/aiohttp-3.7.4.post0-cp37-cp37m-manylinux2014_x86_64.whl (1.3MB)\n",
138 | "\u001b[K |████████████████████████████████| 1.3MB 21.0MB/s \n",
139 | "\u001b[?25hRequirement already satisfied: requests; extra == \"http\" in /usr/local/lib/python3.7/dist-packages (from fsspec[http]>=0.8.1->pytorch-lightning->autograd-lib) (2.23.0)\n",
140 | "Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (3.12.4)\n",
141 | "Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (1.32.0)\n",
142 | "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (1.15.0)\n",
143 | "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (1.8.0)\n",
144 | "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (0.36.2)\n",
145 | "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (0.12.0)\n",
146 | "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (1.28.0)\n",
147 | "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (3.3.4)\n",
148 | "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (0.4.3)\n",
149 | "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (1.0.1)\n",
150 | "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (54.2.0)\n",
151 | "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.4->pytorch-lightning->autograd-lib) (3.7.4.3)\n",
152 | "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.23->seaborn->autograd-lib) (2.8.1)\n",
153 | "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.23->seaborn->autograd-lib) (2018.9)\n",
154 | "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.2->seaborn->autograd-lib) (1.3.1)\n",
155 | "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.2->seaborn->autograd-lib) (2.4.7)\n",
156 | "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.2->seaborn->autograd-lib) (0.10.0)\n",
157 | "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->fsspec[http]>=0.8.1->pytorch-lightning->autograd-lib) (3.4.1)\n",
158 | "Collecting async-timeout<4.0,>=3.0\n",
159 | " Downloading https://files.pythonhosted.org/packages/e1/1e/5a4441be21b0726c4464f3f23c8b19628372f606755a9d2e46c187e65ec4/async_timeout-3.0.1-py3-none-any.whl\n",
160 | "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp; extra == \"http\"->fsspec[http]>=0.8.1->pytorch-lightning->autograd-lib) (20.3.0)\n",
161 | "Collecting multidict<7.0,>=4.5\n",
162 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/7c/a6/4123b8165acbe773d1a8dc8e3f0d1edea16d29f7de018eda769abb56bd30/multidict-5.1.0-cp37-cp37m-manylinux2014_x86_64.whl (142kB)\n",
163 | "\u001b[K |████████████████████████████████| 143kB 29.1MB/s \n",
164 | "\u001b[?25hRequirement already satisfied: chardet<5.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp; extra == \"http\"->fsspec[http]>=0.8.1->pytorch-lightning->autograd-lib) (3.0.4)\n",
165 | "Collecting yarl<2.0,>=1.0\n",
166 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/f1/62/046834c5fc998c88ab2ef722f5d42122230a632212c8afa76418324f53ff/yarl-1.6.3-cp37-cp37m-manylinux2014_x86_64.whl (294kB)\n",
167 | "\u001b[K |████████████████████████████████| 296kB 21.3MB/s \n",
168 | "\u001b[?25hRequirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests; extra == \"http\"->fsspec[http]>=0.8.1->pytorch-lightning->autograd-lib) (1.24.3)\n",
169 | "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests; extra == \"http\"->fsspec[http]>=0.8.1->pytorch-lightning->autograd-lib) (2020.12.5)\n",
170 | "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests; extra == \"http\"->fsspec[http]>=0.8.1->pytorch-lightning->autograd-lib) (2.10)\n",
171 | "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (4.2.1)\n",
172 | "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3.6\" in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (4.7.2)\n",
173 | "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (0.2.8)\n",
174 | "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (1.3.0)\n",
175 | "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.7/dist-packages (from rsa<5,>=3.1.4; python_version >= \"3.6\"->google-auth<2,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (0.4.8)\n",
176 | "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning->autograd-lib) (3.1.0)\n",
177 | "Building wheels for collected packages: PyYAML, future\n",
178 | " Building wheel for PyYAML (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
179 | " Created wheel for PyYAML: filename=PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl size=44620 sha256=b80020fe2f1cc81e7a85d1a230840ae9e3051ccf18808e5d38b12578f4c8080e\n",
180 | " Stored in directory: /root/.cache/pip/wheels/a7/c1/ea/cf5bd31012e735dc1dfea3131a2d5eae7978b251083d6247bd\n",
181 | " Building wheel for future (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
182 | " Created wheel for future: filename=future-0.18.2-cp37-none-any.whl size=491058 sha256=eb42531dfa9bff327ad80fe7f584e2231e31f800574c57c0cfaa780ca568c4a0\n",
183 | " Stored in directory: /root/.cache/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e\n",
184 | "Successfully built PyYAML future\n",
185 | "Installing collected packages: torchmetrics, async-timeout, multidict, yarl, aiohttp, fsspec, PyYAML, future, pytorch-lightning, autograd-lib\n",
186 | " Found existing installation: PyYAML 3.13\n",
187 | " Uninstalling PyYAML-3.13:\n",
188 | " Successfully uninstalled PyYAML-3.13\n",
189 | " Found existing installation: future 0.16.0\n",
190 | " Uninstalling future-0.16.0:\n",
191 | " Successfully uninstalled future-0.16.0\n",
192 | "Successfully installed PyYAML-5.3.1 aiohttp-3.7.4.post0 async-timeout-3.0.1 autograd-lib-0.0.7 fsspec-0.8.7 future-0.18.2 multidict-5.1.0 pytorch-lightning-1.2.6 torchmetrics-0.2.0 yarl-1.6.3\n"
193 | ],
194 | "name": "stdout"
195 | }
196 | ]
197 | },
198 | {
199 | "cell_type": "markdown",
200 | "metadata": {
201 | "id": "Ts3YO-DrbNhW"
202 | },
203 | "source": [
204 | "### Import Libraries"
205 | ]
206 | },
207 | {
208 | "cell_type": "code",
209 | "metadata": {
210 | "id": "kYWpBK-sbPX8"
211 | },
212 | "source": [
213 | "import numpy as np\n",
214 | "from math import pi\n",
215 | "from collections import defaultdict\n",
216 | "from autograd_lib import autograd_lib\n",
217 | "\n",
218 | "import torch\n",
219 | "import torch.nn as nn\n",
220 | "from torch.utils.data import DataLoader, Dataset\n",
221 | "\n",
222 | "import warnings\n",
223 | "warnings.filterwarnings(\"ignore\")"
224 | ],
225 | "execution_count": 3,
226 | "outputs": []
227 | },
228 | {
229 | "cell_type": "markdown",
230 | "metadata": {
231 | "id": "qy_Hs63GbQ_a"
232 | },
233 | "source": [
234 | "### Define NN Model\n",
235 | "The NN model here is used to fit a single variable math function.\n",
236 | "$$f(x) = \\frac{\\sin(5\\pi x)}{5\\pi x}.$$"
237 | ]
238 | },
239 | {
240 | "cell_type": "code",
241 | "metadata": {
242 | "id": "cIX42-5mbTVl"
243 | },
244 | "source": [
245 | "class MathRegressor(nn.Module):\n",
246 | " def __init__(self, num_hidden=128):\n",
247 | " super().__init__()\n",
248 | " self.regressor = nn.Sequential(\n",
249 | " nn.Linear(1, num_hidden),\n",
250 | " nn.ReLU(),\n",
251 | " nn.Linear(num_hidden, 1)\n",
252 | " )\n",
253 | "\n",
254 | " def forward(self, x):\n",
255 | " x = self.regressor(x)\n",
256 | " return x"
257 | ],
258 | "execution_count": 4,
259 | "outputs": []
260 | },
261 | {
262 | "cell_type": "markdown",
263 | "metadata": {
264 | "id": "StneuhWqbVGL"
265 | },
266 | "source": [
267 | "### Get Pretrained Checkpoints\n",
268 | "The pretrained checkpoints is done by TA. Each student will get a different checkpoint."
269 | ]
270 | },
271 | {
272 | "cell_type": "code",
273 | "metadata": {
274 | "colab": {
275 | "base_uri": "https://localhost:8080/"
276 | },
277 | "id": "ibMNFpMgbW2-",
278 | "outputId": "8a469925-0a79-4f51-dde1-437072c4de3f"
279 | },
280 | "source": [
281 | "!gdown --id 1ym6G7KKNkbsqSnMmnxdQKHO1JBoF0LPR"
282 | ],
283 | "execution_count": 5,
284 | "outputs": [
285 | {
286 | "output_type": "stream",
287 | "text": [
288 | "Downloading...\n",
289 | "From: https://drive.google.com/uc?id=1ym6G7KKNkbsqSnMmnxdQKHO1JBoF0LPR\n",
290 | "To: /content/data.pth\n",
291 | "\r 0% 0.00/34.5k [00:00nli', B, A) # do batch-wise outer product\n",
403 | "\n",
404 | " # full Hessian\n",
405 | " hess[layer] += torch.einsum('nli,nkj->likj', BA, BA) # do batch-wise outer product, then sum over the batch\n",
406 | "\n",
407 | "# function to compute the minimum ratio\n",
408 | "def compute_minimum_ratio(model, criterion, train, target):\n",
409 | " model.zero_grad()\n",
410 | " # compute Hessian matrix\n",
411 | " # save the gradient of each layer\n",
412 | " with autograd_lib.module_hook(save_activations):\n",
413 | " output = model(train)\n",
414 | " loss = criterion(output, target)\n",
415 | "\n",
416 | " # compute Hessian according to the gradient value stored in the previous step\n",
417 | " with autograd_lib.module_hook(compute_hess):\n",
418 | " autograd_lib.backward_hessian(output, loss='LeastSquares')\n",
419 | "\n",
420 | " layer_hess = list(hess.values())\n",
421 | " minimum_ratio = []\n",
422 | "\n",
423 | " # compute eigenvalues of the Hessian matrix\n",
424 | " for h in layer_hess:\n",
425 | " size = h.shape[0] * h.shape[1]\n",
426 | " h = h.reshape(size, size)\n",
427 | " h_eig = torch.symeig(h).eigenvalues # torch.symeig() returns eigenvalues and eigenvectors of a real symmetric matrix\n",
428 | " num_greater = torch.sum(h_eig > 0).item()\n",
429 | " minimum_ratio.append(num_greater / len(h_eig))\n",
430 | "\n",
431 | " ratio_mean = np.mean(minimum_ratio) # compute mean of minimum ratio\n",
432 | "\n",
433 | " return ratio_mean"
434 | ],
435 | "execution_count": 8,
436 | "outputs": []
437 | },
438 | {
439 | "cell_type": "markdown",
440 | "metadata": {
441 | "id": "78xJoeY_bmSr"
442 | },
443 | "source": [
444 | "### Mathematical Derivation\n",
445 | "\n",
446 | "Method used here: https://en.wikipedia.org/wiki/Gauss–Newton_algorithm\n",
447 | "\n",
448 | "> **Notations** \\\\\n",
449 | "> $\\mathbf{A}$: the input of the layer. \\\\\n",
450 | "> $\\mathbf{B}$: the backprop value. \\\\\n",
451 | "> $\\mathbf{Z}$: the output of the layer. \\\\\n",
452 | "> $L$: the total loss, mean squared error was used here, $L=e^2$. \\\\\n",
453 | "> $w$: the weight value.\n",
454 | "\n",
455 | "Assume that the input dimension of the layer is $n$, and the output dimension of the layer is $m$.\n",
456 | "\n",
457 | "The derivative of the loss is\n",
458 | "\n",
459 | "\\begin{align*}\n",
460 | " \\left(\\frac{\\partial L}{\\partial w}\\right)_{nm} &= \\mathbf{A}_m \\mathbf{B}_n,\n",
461 | "\\end{align*}\n",
462 | "\n",
463 | "which can be written as\n",
464 | "\n",
465 | "\\begin{align*}\n",
466 | " \\frac{\\partial L}{\\partial w} &= \\mathbf{B} \\times \\mathbf{A}.\n",
467 | "\\end{align*}\n",
468 | "\n",
469 | "The Hessian can be derived as\n",
470 | "\n",
471 | "\\begin{align*}\n",
472 | " \\mathbf{H}_{ij}&=\\frac{\\partial^2 L}{\\partial w_i \\partial w_j} \\\\\n",
473 | " &= \\frac{\\partial}{\\partial w_i}\\left(\\frac{\\partial L}{\\partial w_j}\\right) \\\\\n",
474 | " &= \\frac{\\partial}{\\partial w_i}\\left(\\frac{2e\\partial e}{\\partial w_j}\\right) \\\\\n",
475 | " &= 2\\frac{\\partial e}{\\partial w_i}\\frac{\\partial e}{\\partial w_j}+2e\\frac{\\partial^2 e}{\\partial w_j \\partial w_i}.\n",
476 | "\\end{align*}\n",
477 | "\n",
478 | "We neglect the second-order derivative term because the term is relatively small ($e$ is small)\n",
479 | "\n",
480 | "\\begin{align*}\n",
481 | " \\mathbf{H}_{ij}\n",
482 | " &\\propto \\frac{\\partial e}{\\partial w_i}\\frac{\\partial e}{\\partial w_j},\n",
483 | "\\end{align*}\n",
484 | "\n",
485 | "and as the error $e$ is a constant\n",
486 | "\n",
487 | "\\begin{align*}\n",
488 | " \\mathbf{H}_{ij}\n",
489 | " &\\propto \\frac{\\partial L}{\\partial w_i}\\frac{\\partial L}{\\partial w_j},\n",
490 | "\\end{align*}\n",
491 | "\n",
492 | "then the full Hessian becomes\n",
493 | "\n",
494 | "\\begin{align*}\n",
495 | " \\mathbf{H} &\\propto (\\mathbf{B}\\times\\mathbf{A})\\times(\\mathbf{B}\\times\\mathbf{A}).\n",
496 | "\\end{align*}\n"
497 | ]
498 | },
499 | {
500 | "cell_type": "code",
501 | "metadata": {
502 | "id": "EDCQuAdlboHT"
503 | },
504 | "source": [
505 | "# the main function to compute gradient norm and minimum ratio\n",
506 | "def main(model, train, target):\n",
507 | " criterion = nn.MSELoss()\n",
508 | "\n",
509 | " gradient_norm = compute_gradient_norm(model, criterion, train, target)\n",
510 | " minimum_ratio = compute_minimum_ratio(model, criterion, train, target)\n",
511 | "\n",
512 | " print('gradient norm: {}, minimum ratio: {}'.format(gradient_norm, minimum_ratio))"
513 | ],
514 | "execution_count": 9,
515 | "outputs": []
516 | },
517 | {
518 | "cell_type": "markdown",
519 | "metadata": {
520 | "id": "75_j79LlbsTY"
521 | },
522 | "source": [
523 | "After running this block, you will get the value of `gradient norm` and `minimum ratio`. Determine whether the model is at `local minima like`, `saddle point`, or `none of the above`, and then submit your choice to NTU COOL."
524 | ]
525 | },
526 | {
527 | "cell_type": "code",
528 | "metadata": {
529 | "colab": {
530 | "base_uri": "https://localhost:8080/"
531 | },
532 | "id": "fsGdnelqbqBu",
533 | "outputId": "5dfd75e3-69f0-4637-8f22-2683ccbaa46a"
534 | },
535 | "source": [
536 | "if __name__ == '__main__':\n",
537 | " # fix random seed\n",
538 | " torch.manual_seed(0)\n",
539 | "\n",
540 | " # reset compute dictionaries\n",
541 | " activations = defaultdict(int)\n",
542 | " hess = defaultdict(float)\n",
543 | "\n",
544 | " # compute Hessian\n",
545 | " main(model, train, target)"
546 | ],
547 | "execution_count": 10,
548 | "outputs": [
549 | {
550 | "output_type": "stream",
551 | "text": [
552 | "gradient norm: 0.000772382496506907, minimum ratio: 0.48046875\n"
553 | ],
554 | "name": "stdout"
555 | }
556 | ]
557 | },
558 | {
559 | "cell_type": "markdown",
560 | "metadata": {
561 | "id": "wU-vs9zXbvpc"
562 | },
563 | "source": [
564 | "● gradient norm < 1e-3 and minimum ratio > 0.5 => local minima like\n",
565 | "\n",
566 | "● gradient norm < 1e-3 and minimum ratio <= 0.5 => saddle point\n",
567 | "\n",
568 | "● gradient norm >= 1e-3 => none of the above."
569 | ]
570 | }
571 | ]
572 | }
--------------------------------------------------------------------------------
/HW3/HW03.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW3/HW03.pdf
--------------------------------------------------------------------------------
/HW4/HW04.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW4/HW04.pdf
--------------------------------------------------------------------------------
/HW4/hw4_report.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW4/hw4_report.pdf
--------------------------------------------------------------------------------
/HW5/HW05.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW5/HW05.pdf
--------------------------------------------------------------------------------
/HW6/HW06.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW6/HW06.pdf
--------------------------------------------------------------------------------
/HW6/hw6_report.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW6/hw6_report.pdf
--------------------------------------------------------------------------------
/HW7/HW07.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW7/HW07.pdf
--------------------------------------------------------------------------------
/HW8/HW08.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW8/HW08.pdf
--------------------------------------------------------------------------------
/HW8/hw8_report.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW8/hw8_report.pdf
--------------------------------------------------------------------------------
/HW9/HW09.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HW9/HW09.pdf
--------------------------------------------------------------------------------
/HomeworkParticipation.JPG:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Offliners/NTUML-2021Spring/21e38b137d76a6f6e57673c1871cd93e413bee02/HomeworkParticipation.JPG
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | 
2 |
3 | Course Syllabus : [Link](https://speech.ee.ntu.edu.tw/~hylee/ml/2021-spring.html)
4 |
5 | ## Outline
6 | |#|Homework|Slide|Code|Public score|Private Score|Score|
7 | |-|-|-|-|-|-|-|
8 | |0|Colab Tutorial|[Link](HW0/Google_Colab_Tutorial.pdf)|[Colab Tutorial](HW0/Colab_Tutorial.ipynb)|x|x|x|
9 | |0|Pytorch Tutorial|[Link](HW0/Pytorch_Tutorial_1.pdf) [Link](HW0/Pytorch_Tutorial_2.pdf)|[Pytorch Tutorial](HW0/Pytorch_Tutorial.ipynb)|x|x|x|
10 | |1|Regression|[Link](HW1/HW01.pdf)|[COVID-19 Cases Prediction](HW1/homework1.ipynb)|`0.86928`|`0.92417`|9|
11 | |2|Classification|[Link](HW2/HW02.pdf)|[Phoneme Classification](HW2/homework2_1.ipynb)|`0.75171`|`0.74957`|6|
12 | |2|Linear Algebra|[Link](HW2/HW02.pdf)|[Hessian Matrix](HW2/homework2_2.ipynb)|x|x|2|
13 | |3|CNN|[Link](HW3/HW03.pdf)|[Image Classification](HW3/homework3.ipynb)|`0.77956`|`0.77166`|8|
14 | |4|Self-Attention|[Link](HW4/HW04.pdf)|[Phoneme classification](HW4/homework4.ipynb)|`0.95928`|`0.95833`|10+0.5([report](HW4/hw4_report.pdf))|
15 | |5|Transformer|[Link](HW5/HW05.pdf)|[Machine Translation](HW5/homework5.ipynb)|`26.44`|`25.54`|8|
16 | |6|GAN|[Link](HW6/HW06.pdf)|[Anime Face Generation](HW6/homework6.ipynb)|`0.710`, `7716.47`|`0.710`, `7716.47`|10+0.5([report](HW6/hw6_report.pdf))|
17 | |7|BERT|[Link](HW7/HW07.pdf)|[Question Answering](HW7/homework7.ipynb)|`0.81178`|`0.81833`|9|
18 | |8|Anomaly Detection|[Link](HW8/HW08.pdf)|[Anomaly Detection](HW8/homework8.ipynb)|`0.89049`|`0.88126`|10+0.5([report](HW8/hw8_report.pdf))|
19 | |9|Explainable AI|[Link](HW9/HW09.pdf)|[Explainable AI](HW9/homework9.ipynb)|x|x|8.4|
20 | |10|Attack|[Link](HW10/HW10.pdf)|[Adversarial Attack](HW10/homework10.ipynb)|`0.000`|`0.010`|10+0.5([report](HW10/hw10_report.pdf))|
21 | |11|Adaptation|[Link](HW11/HW11.pdf)|[Domain Adaptation](HW11/homework11.ipynb)|`0.81598`|`0.81690`|10+0.5([report](HW11/hw11_report.pdf))|
22 | |12|RL|[Link](HW12/HW12.pdf)|[Reinforcement Learning](HW12/homework12.ipynb)|`280`|`280`|10+0.5([report](HW12/hw12_report.pdf))|
23 | |13|Compression|[Link](HW13/HW13.pdf)|[Food Classification](HW13/homework13.ipynb)|`0.80167`|`0.81350`|9.5|
24 | |14|Life-Long Learning|[Link](HW14/HW14.pdf)|[Permuted MNIST](HW14/homework14.ipynb)|x|x|8.8|
25 | |15|Meta Learning|[Link](HW15/HW15.pdf)|[Few-shot image classification](HW15/homework15.ipynb)|x|x|9.2|
26 |
27 |
45 |
46 | ## Useful Tips
47 | ### Prevent Google Colab from disconnecting (valid utill 2021/06/25)
48 | Press `F12`，and enter this code in console，then press `enter`
49 | ```javascript
50 | function ClickConnect(){
51 | console.log("Connnect Clicked - Start");
52 | document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click();
53 | console.log("Connnect Clicked - End");
54 | };
55 | setInterval(ClickConnect, 60000)
56 | ```
57 |
58 | ### Auto save output file on Google Colab
59 | insert code cell at the bottom
60 | ```python
61 | from google.colab import files
62 | files.download("output_file.csv") # "output_file.csv" must be your output file name
63 | ```
64 |
65 | ### Display information of GPU of Google Colab
66 | insert code cell to check which GPU is assigned
67 | ```shell
68 | !nvidia-smi
69 | ```
70 | ### "Sorry, something went wrong. Reload?" when viewing *.ipynb on Github
71 | Copy the URL to https://nbviewer.jupyter.org/
72 |
73 | ## Student Source
74 |
75 | 
76 |
80 | 
81 |
85 | 
86 |
![\"Colaboratory](\"/img/colab_favicon.ico\")