├── .gitignore ├── docs ├── OS │ ├── Chap01.md │ ├── Chap07.md │ ├── index.md │ ├── Chap02.md │ ├── final.md │ ├── Chap09.md │ ├── Chap04.md │ ├── Chap06.md │ └── Chap08.md ├── assets │ ├── os │ │ ├── RR.png │ │ ├── 2.1.png │ │ ├── 2.5.png │ │ ├── 2.6.png │ │ ├── 2.7.png │ │ ├── 3.1.png │ │ ├── 3.10.png │ │ ├── 3.12.png │ │ ├── 3.19.png │ │ ├── 3.2.png │ │ ├── 3.20.png │ │ ├── 3.23.png │ │ ├── 3.24.png │ │ ├── 3.4.png │ │ ├── 3.5.png │ │ ├── 3.6.png │ │ ├── 3.7.png │ │ ├── 3.8.png │ │ ├── 4.1.png │ │ ├── 4.13.png │ │ ├── 4.14.png │ │ ├── 4.2.png │ │ ├── 4.3.png │ │ ├── 4.4.png │ │ ├── 4.5.png │ │ ├── 4.6.png │ │ ├── 4.7.png │ │ ├── 4.8.png │ │ ├── 5.17.png │ │ ├── 6.1.png │ │ ├── 6.17.png │ │ ├── 6.18.png │ │ ├── 6.19.png │ │ ├── 6.6.png │ │ ├── 6.9.png │ │ ├── 8.1.png │ │ ├── 8.10.png │ │ ├── 8.11.png │ │ ├── 8.14.png │ │ ├── 8.18.png │ │ ├── 8.19.png │ │ ├── 8.2.png │ │ ├── 8.20.png │ │ ├── 8.21.png │ │ ├── 8.22.png │ │ ├── 8.23.png │ │ ├── 8.24.png │ │ ├── 8.3.png │ │ ├── 8.4.png │ │ ├── 8.5.png │ │ ├── 8.6.png │ │ ├── 8.8.png │ │ ├── 9.10.png │ │ ├── 9.12.png │ │ ├── 9.14.png │ │ ├── 9.17.png │ │ ├── 9.6.png │ │ ├── FCFS.png │ │ ├── SJF.png │ │ ├── VAX.png │ │ ├── libc.png │ │ ├── page.png │ │ ├── FCFS-2.png │ │ ├── IA-32.png │ │ ├── SJF-2.png │ │ ├── page2.png │ │ ├── pagesize.png │ │ ├── priority.png │ │ └── 2015midterm.png │ └── favicon.png ├── css │ ├── style.css │ └── katex.css ├── js │ └── katex.js ├── ML │ ├── index.md │ ├── 06.md │ ├── 08.md │ ├── 02.md │ ├── 09.md │ ├── 07.md │ ├── 01.md │ ├── 04.md │ ├── 03.md │ └── 05.md ├── JavaScript │ ├── 03 │ │ ├── insertionSort.md │ │ ├── selectionSort.md │ │ ├── bubbleSort.md │ │ ├── mergeSort.md │ │ └── quickSort.md │ ├── index.md │ ├── 01 │ │ ├── divideAndConquer.md │ │ ├── slidingWindow.md │ │ ├── multiplePointers.md │ │ └── frequencyCounters.md │ ├── 02 │ │ ├── recursionEasy.md │ │ └── recursionHard.md │ ├── 04 │ │ ├── MaxBinaryHeap.md │ │ ├── MaxPriorityQueue.md │ │ ├── BinarySearchTree.md │ │ ├── SinglyLinkedList.md │ │ └── DoublyLinkedList.md │ └── 05 │ │ ├── Graph.md │ │ └── Dijkstra.md ├── PCP │ ├── index.md │ ├── 2-3.md │ ├── 1-3.md │ ├── 1-2.md │ ├── 2-2.md │ ├── 1-4.md │ ├── 1-5.md │ └── 2-1.md └── index.md ├── .gitattributes ├── custom └── main.html ├── README.md └── mkdocs.yml /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | site/ 3 | -------------------------------------------------------------------------------- /docs/OS/Chap01.md: -------------------------------------------------------------------------------- 1 | # Chapter 1 Introduction -------------------------------------------------------------------------------- /.gitattributes: -------------------------------------------------------------------------------- 1 | *.md linguist-detectable=true 2 | *.md linguist-documentation=false 3 | -------------------------------------------------------------------------------- /docs/assets/os/RR.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/RR.png -------------------------------------------------------------------------------- /docs/assets/favicon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/favicon.png -------------------------------------------------------------------------------- /docs/assets/os/2.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/2.1.png -------------------------------------------------------------------------------- /docs/assets/os/2.5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/2.5.png -------------------------------------------------------------------------------- /docs/assets/os/2.6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/2.6.png -------------------------------------------------------------------------------- /docs/assets/os/2.7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/2.7.png -------------------------------------------------------------------------------- /docs/assets/os/3.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.1.png -------------------------------------------------------------------------------- /docs/assets/os/3.10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.10.png -------------------------------------------------------------------------------- /docs/assets/os/3.12.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.12.png -------------------------------------------------------------------------------- /docs/assets/os/3.19.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.19.png -------------------------------------------------------------------------------- /docs/assets/os/3.2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.2.png -------------------------------------------------------------------------------- /docs/assets/os/3.20.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.20.png -------------------------------------------------------------------------------- /docs/assets/os/3.23.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.23.png -------------------------------------------------------------------------------- /docs/assets/os/3.24.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.24.png -------------------------------------------------------------------------------- /docs/assets/os/3.4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.4.png -------------------------------------------------------------------------------- /docs/assets/os/3.5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.5.png -------------------------------------------------------------------------------- /docs/assets/os/3.6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.6.png -------------------------------------------------------------------------------- /docs/assets/os/3.7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.7.png -------------------------------------------------------------------------------- /docs/assets/os/3.8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/3.8.png -------------------------------------------------------------------------------- /docs/assets/os/4.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/4.1.png -------------------------------------------------------------------------------- /docs/assets/os/4.13.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/4.13.png -------------------------------------------------------------------------------- /docs/assets/os/4.14.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/4.14.png -------------------------------------------------------------------------------- /docs/assets/os/4.2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/4.2.png -------------------------------------------------------------------------------- /docs/assets/os/4.3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/4.3.png -------------------------------------------------------------------------------- /docs/assets/os/4.4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/4.4.png -------------------------------------------------------------------------------- /docs/assets/os/4.5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/4.5.png -------------------------------------------------------------------------------- /docs/assets/os/4.6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/4.6.png -------------------------------------------------------------------------------- /docs/assets/os/4.7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/4.7.png -------------------------------------------------------------------------------- /docs/assets/os/4.8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/4.8.png -------------------------------------------------------------------------------- /docs/assets/os/5.17.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/5.17.png -------------------------------------------------------------------------------- /docs/assets/os/6.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/6.1.png -------------------------------------------------------------------------------- /docs/assets/os/6.17.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/6.17.png -------------------------------------------------------------------------------- /docs/assets/os/6.18.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/6.18.png -------------------------------------------------------------------------------- /docs/assets/os/6.19.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/6.19.png -------------------------------------------------------------------------------- /docs/assets/os/6.6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/6.6.png -------------------------------------------------------------------------------- /docs/assets/os/6.9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/6.9.png -------------------------------------------------------------------------------- /docs/assets/os/8.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.1.png -------------------------------------------------------------------------------- /docs/assets/os/8.10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.10.png -------------------------------------------------------------------------------- /docs/assets/os/8.11.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.11.png -------------------------------------------------------------------------------- /docs/assets/os/8.14.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.14.png -------------------------------------------------------------------------------- /docs/assets/os/8.18.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.18.png -------------------------------------------------------------------------------- /docs/assets/os/8.19.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.19.png -------------------------------------------------------------------------------- /docs/assets/os/8.2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.2.png -------------------------------------------------------------------------------- /docs/assets/os/8.20.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.20.png -------------------------------------------------------------------------------- /docs/assets/os/8.21.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.21.png -------------------------------------------------------------------------------- /docs/assets/os/8.22.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.22.png -------------------------------------------------------------------------------- /docs/assets/os/8.23.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.23.png -------------------------------------------------------------------------------- /docs/assets/os/8.24.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.24.png -------------------------------------------------------------------------------- /docs/assets/os/8.3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.3.png -------------------------------------------------------------------------------- /docs/assets/os/8.4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.4.png -------------------------------------------------------------------------------- /docs/assets/os/8.5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.5.png -------------------------------------------------------------------------------- /docs/assets/os/8.6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.6.png -------------------------------------------------------------------------------- /docs/assets/os/8.8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/8.8.png -------------------------------------------------------------------------------- /docs/assets/os/9.10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/9.10.png -------------------------------------------------------------------------------- /docs/assets/os/9.12.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/9.12.png -------------------------------------------------------------------------------- /docs/assets/os/9.14.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/9.14.png -------------------------------------------------------------------------------- /docs/assets/os/9.17.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/9.17.png -------------------------------------------------------------------------------- /docs/assets/os/9.6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/9.6.png -------------------------------------------------------------------------------- /docs/assets/os/FCFS.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/FCFS.png -------------------------------------------------------------------------------- /docs/assets/os/SJF.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/SJF.png -------------------------------------------------------------------------------- /docs/assets/os/VAX.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/VAX.png -------------------------------------------------------------------------------- /docs/assets/os/libc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/libc.png -------------------------------------------------------------------------------- /docs/assets/os/page.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/page.png -------------------------------------------------------------------------------- /docs/assets/os/FCFS-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/FCFS-2.png -------------------------------------------------------------------------------- /docs/assets/os/IA-32.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/IA-32.png -------------------------------------------------------------------------------- /docs/assets/os/SJF-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/SJF-2.png -------------------------------------------------------------------------------- /docs/assets/os/page2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/page2.png -------------------------------------------------------------------------------- /docs/assets/os/pagesize.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/pagesize.png -------------------------------------------------------------------------------- /docs/assets/os/priority.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/priority.png -------------------------------------------------------------------------------- /docs/assets/os/2015midterm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/walkccc/CS/HEAD/docs/assets/os/2015midterm.png -------------------------------------------------------------------------------- /docs/css/style.css: -------------------------------------------------------------------------------- 1 | img[alt=normal] { 2 | width: 80%; 3 | } 4 | 5 | img[alt=small] { 6 | width: 50%; 7 | } -------------------------------------------------------------------------------- /docs/OS/Chap07.md: -------------------------------------------------------------------------------- 1 | # Chapter 7 Deadlocks 2 | 3 | ## 7.1 System Model 4 | 5 | ## 7.2 Deadlock Characterization 6 | 7 | ### 7.2.1 Necessary Conditions 8 | 9 | ### 7.2.2 Resource-Allocation Graph -------------------------------------------------------------------------------- /custom/main.html: -------------------------------------------------------------------------------- 1 | 2 | {% extends "base.html" %} 3 | 4 | {% block site_meta %} 5 | {{ super() }} 6 | 7 | {% endblock %} 8 | -------------------------------------------------------------------------------- /docs/js/katex.js: -------------------------------------------------------------------------------- 1 | document.addEventListener("DOMContentLoaded", function() { 2 | renderMathInElement(document.body, { 3 | delimiters: [ 4 | {left: "$$", right: "$$", display: true}, 5 | {left: "$", right: "$", display: false} 6 | ] 7 | }); 8 | }); -------------------------------------------------------------------------------- /docs/ML/index.md: -------------------------------------------------------------------------------- 1 | # Machine Learning | notes 2 | 3 | In this page, I'll work on my notes when I watched the [Machine Learning course video](https://www.youtube.com/playlist?list=PLJV_el3uVTsPy9oCRY30oBPNLCo89yu49) provided by Professor [Hung-yi Lee](http://speech.ee.ntu.edu.tw/~tlkagk/index.html). 4 | -------------------------------------------------------------------------------- /docs/JavaScript/03/insertionSort.md: -------------------------------------------------------------------------------- 1 | # Insertion Sort 2 | 3 | ## `insertionSort(arr)` 4 | 5 | ```js 6 | const insertionSort = arr => { 7 | for (let i = 1; i < arr.length; i++) { 8 | let curr = arr[i]; 9 | 10 | for (var j = i - 1; j >= 0 && arr[j] > curr; j--) 11 | arr[j + 1] = arr[j]; 12 | 13 | arr[j + 1] = curr; 14 | } 15 | 16 | return arr; 17 | }; 18 | ``` 19 | -------------------------------------------------------------------------------- /docs/JavaScript/index.md: -------------------------------------------------------------------------------- 1 | # JavaScript Data Structures and Algorithms 2 | 3 | This page contains notes from the course: [JavaScript Algorithms and Data Structures Masterclass](https://www.udemy.com/js-algorithms-and-data-structures-masterclass/learn/v4/content) taught by Colt Steele. 4 | 5 | All codes are written in ES9 fashion and are formatted by [prettier](https://github.com/prettier/prettier). 6 | -------------------------------------------------------------------------------- /docs/css/katex.css: -------------------------------------------------------------------------------- 1 | .katex { 2 | font-size: 1.1em !important; 3 | font: normal 1.1em KaTeX_Main, Times New Roman, serif; 4 | line-height: 1.2; 5 | white-space: normal; 6 | text-indent: 0; 7 | } 8 | 9 | .katex-display > .katex { 10 | display: inline-block; 11 | white-space: nowrap; 12 | max-width: 100%; 13 | text-align: initial; 14 | } 15 | 16 | .katex-html { 17 | overflow-y: hidden; 18 | } 19 | -------------------------------------------------------------------------------- /docs/PCP/index.md: -------------------------------------------------------------------------------- 1 | # Parallel and Concurrent Programming with C++ 2 | 3 | This page contains notes from the course: [Parallel and Concurrent Programming with C++](https://www.linkedin.com/learning/parallel-and-concurrent-programming-with-c-plus-plus-part-1/learn-parallel-programming-basics?u=2131553) taught by Barron Stone and Olivia Chiu Stone in LinkedIn Learning. 4 | 5 | I use highlight here to indicate important knowledge. 6 | -------------------------------------------------------------------------------- /docs/OS/index.md: -------------------------------------------------------------------------------- 1 | # Operating System | notes 2 | 3 | In this page, I'll work on my notes when I had class of [Operating System, Spring 2018](http://newslab.csie.ntu.edu.tw/course/OS2018/index.php) by Professor [Tei-Wei Kuo](https://www.csie.ntu.edu.tw/~ktw/eng-index.htm). Professor Kuo has also upload the course videos to [YouTube](https://www.youtube.com/playlist?list=PLco3ZjBUnBUKNn0ANhQ1N7aJbYUmMlgc8). 4 | 5 | The materials are mainly from the [Operating System Concepts, 9th Edition](https://www.amazon.com/Operating-System-Concepts-Abraham-Silberschatz-ebook/dp/B00APSZCEQ). 6 | -------------------------------------------------------------------------------- /docs/JavaScript/03/selectionSort.md: -------------------------------------------------------------------------------- 1 | # Selection Sort 2 | 3 | ## `swap(arr, i, j)` 4 | 5 | ```js 6 | const swap = (arr, i, j) => { 7 | [arr[i], arr[j]] = [arr[j], arr[i]]; 8 | }; 9 | ``` 10 | 11 | ## `selectionSort(arr)` 12 | 13 | ```js 14 | const selectionSort = arr => { 15 | for (let i = 0; i < arr.length; i++) { 16 | let smallest = i; 17 | for (let j = i + 1; j < arr.length; j++) { 18 | if (arr[smallest] > arr[j]) { 19 | smallest = j; 20 | } 21 | } 22 | if (i !== smallest) swap(arr, i, smallest); 23 | } 24 | return arr; 25 | }; 26 | ``` 27 | -------------------------------------------------------------------------------- /docs/JavaScript/03/bubbleSort.md: -------------------------------------------------------------------------------- 1 | # Bubble Sort 2 | 3 | ## `swap(arr, i, j)` 4 | 5 | ```js 6 | const swap = (arr, i, j) => { 7 | [arr[i], arr[j]] = [arr[j], arr[i]]; 8 | }; 9 | ``` 10 | 11 | ## `bubbleSort(arr)` 12 | 13 | ```js 14 | const bubbleSort = arr => { 15 | for (let i = arr.length - 1; i > 0; i--) { 16 | let noSwaps = true; 17 | 18 | for (let j = 0; j < i; j++) { 19 | if (arr[j] > arr[j + 1]) { 20 | swap(arr, j, j + 1); 21 | noSwaps = false; 22 | } 23 | } 24 | 25 | if (noSwaps) break; 26 | } 27 | 28 | return arr; 29 | }; 30 | ``` 31 | -------------------------------------------------------------------------------- /docs/JavaScript/03/mergeSort.md: -------------------------------------------------------------------------------- 1 | # Merge Sort 2 | 3 | ## `merge(arr1, arr2)` 4 | 5 | ```js 6 | const merge = (arr1, arr2) => { 7 | let ret = []; 8 | let i = 0; 9 | let j = 0; 10 | 11 | while (i < arr1.length && j < arr2.length) { 12 | if (arr1[i] < arr2[j]) ret.push(arr1[i++]); 13 | else ret.push(arr2[j++]); 14 | } 15 | 16 | while (i < arr1.length) ret.push(arr1[i++]); 17 | while (j < arr2.length) ret.push(arr2[j++]); 18 | 19 | return ret; 20 | }; 21 | ``` 22 | 23 | ## `mergeSort(arr)` 24 | 25 | ```js 26 | const mergeSort = arr => { 27 | if (arr.length <= 1) return arr; 28 | 29 | let mid = Math.floor(arr.length / 2); 30 | let left = mergeSort(arr.slice(0, mid)); 31 | let right = mergeSort(arr.slice(mid)); 32 | 33 | return merge(left, right); 34 | }; 35 | ``` 36 | -------------------------------------------------------------------------------- /docs/JavaScript/03/quickSort.md: -------------------------------------------------------------------------------- 1 | # Quick Sort 2 | 3 | ## `pivot(arr, left = 0, right = arr.length - 1)` 4 | 5 | ```js 6 | const pivot = (arr, left, right) => { 7 | const swap = (arr, i, j) => { 8 | [arr[i], arr[j]] = [arr[j], arr[i]]; 9 | }; 10 | 11 | let pivot = arr[left]; 12 | let swapIndex = left; 13 | 14 | for (let i = left + 1; i <= right; i++) 15 | if (pivot > arr[i]) 16 | swap(arr, ++swapIndex, i); 17 | swap(arr, left, swapIndex); 18 | 19 | return swapIndex; 20 | }; 21 | ``` 22 | 23 | ## `quickSort(arr)` 24 | 25 | ```js 26 | const quickSort = (arr, left = 0, right = arr.length - 1) => { 27 | if (left < right) { 28 | let pivotIndex = pivot(arr, left, right); 29 | quickSort(arr, left, pivotIndex - 1); 30 | quickSort(arr, pivotIndex + 1, right); 31 | } 32 | 33 | return arr; 34 | }; 35 | ``` 36 | -------------------------------------------------------------------------------- /docs/index.md: -------------------------------------------------------------------------------- 1 | # Computer Science Notes 2 | 3 | ## Getting Started 4 | 5 | In this website, I'll work on my notes about several topics: 6 | 7 | - [Parallel and Concurrent Programming with C++](./PCP/) 8 | - [Operating System](./OS/) 9 | - [Maching Learning](./ML/) 10 | - [JavaScript Data Structures and Algorithms](./JavaScript/) 11 | 12 | ## How I generate this website 13 | 14 | I use the static site generator [MkDocs](http://www.mkdocs.org/) and the beautiful theme [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/) to build this website! 15 | 16 | All mathematical equations are rendered by [KaTeX](https://katex.org/), and I added `overflow-x: auto` to prevent overflow issue on small screen devices, so you can scroll horizontally in some math display equations. 17 | 18 | ## More Informations 19 | 20 | For more informations please visit [**my GitHub site**](https://github.com/walkccc). 21 | 22 | By Jay Chen on May 14, 2018. 23 | -------------------------------------------------------------------------------- /docs/ML/06.md: -------------------------------------------------------------------------------- 1 | # Deep Learning 2 | 3 | ## Three Steps 4 | 5 | ### Step 1: Function Set 6 | 7 | [Lecture 5](../5) 提到的，我們可以將多個 logistic regression models 連接起來，而每一個 model 就像是一個 neuron，給定了網路架構，就定義了一個 function set。 8 | 9 | 給定了架構，再給上每個 neuron 的參數（$w^l, b^l$），整個網路就像是一個 function。 10 | 11 | ![](https://imgur.com/GWWtv4F.png) 12 | 13 | 每個 $W^l$ 都是一個矩陣，都是我們可以用 GPU 的平行運算去加速矩陣運算： 14 | 15 | $$ 16 | \begin{aligned} 17 | y & = f(x) \\\\ 18 | & = \sigma(W^L \cdots \sigma(W^2 \sigma(W^1x + b^1) + b^2) \cdots + b^L) 19 | \end{aligned} 20 | $$ 21 | 22 | #### Output Layer 23 | 24 | ![](https://imgur.com/hAiDUEU.png) 25 | 26 | ### Step 2: Goodness of Function 27 | 28 | ![](https://imgur.com/ZoAcoSz.png) 29 | 30 | ![](https://imgur.com/M9yq5ag.png) 31 | 32 | ### Step 3: Find the best function 33 | 34 | 我們一樣可以用 Gradient Descent 來求解，這用到 [Backpropagation](../7) 的原理。 35 | 36 | ![](https://imgur.com/ahD269D.png) 37 | 38 | 其中， 39 | 40 | $$ 41 | \nabla L = \left[ \begin{array} { c } { \frac { \partial L } { \partial w _ { 1 } } } \\\\ { \frac { \partial L } { \partial w _ { 2 } } } \\\\ { \vdots } \\\\ { \frac { \partial L } { \partial b _ { 1 } } } \\\\ { \vdots } \end{array} \right] 42 | $$ -------------------------------------------------------------------------------- /docs/PCP/2-3.md: -------------------------------------------------------------------------------- 1 | # Asynchronous 2 | 3 | ## Thread pool 4 | 5 | === "begin" 6 | 7 | ```cpp 8 | // Chopping vegetables with a thread pool 9 | #include 10 | 11 | void vegetable_chopper(int vegetable_id) { 12 | printf("Thread %d chopped vegetable %d.\n", std::this_thread::get_id(), 13 | vegetable_id); 14 | } 15 | 16 | int main() { 17 | std::thread choppers[100]; 18 | for (int i = 0; i < 100; i++) { 19 | choppers[i] = std::thread(vegetable_chopper, i); 20 | } 21 | for (auto& c : choppers) { 22 | c.join(); 23 | } 24 | } 25 | ``` 26 | 27 | === "end" 28 | 29 | ```cpp hl_lines="2 10 12" 30 | // Chopping vegetables with a thread pool 31 | #include 32 | 33 | void vegetable_chopper(int vegetable_id) { 34 | printf("Thread %d chopped vegetable %d.\n", std::this_thread::get_id(), 35 | vegetable_id); 36 | } 37 | 38 | int main() { 39 | boost::asio::thread_pool pool(4); // 4 threads 40 | for (int i = 0; i < 100; i++) { 41 | boost::asio::post(pool, [i]() { vegetable_chopper(i); }); 42 | } 43 | pool.join(); 44 | } 45 | ``` 46 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Computer Science Notes 2 | 3 | ## Getting Started 4 | 5 | In this website, I'll work on my notes about several topics: 6 | 7 | - [Operating System](https://walkccc.github.io/CS/OS/) 8 | - [Maching Learning](https://walkccc.github.io/CS/ML/) 9 | - [JavaScript Data Structures and Algorithms](https://walkccc.github.io/CS/JavaScript/) 10 | 11 | Feel free to give me your feedback if any adjustment is needed with the sorted solutions. You can press the "pencil icon" in the upper right corner to edit the contents or simply [open an issue](https://github.com/walkccc/CS/issues/new) in [my repository](https://github.com/walkccc/CS/). 12 | 13 | ## How I generate this website 14 | 15 | I use the static site generator [MkDocs](http://www.mkdocs.org/) and the beautiful theme [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/) to build this website! 16 | 17 | All mathematical equations are rendered by [KaTeX](https://katex.org/), and I added `overflow-x: auto` to prevent overflow issue on small screen devices, so you can scroll horizontally in some math display equations. 18 | 19 | ## More Informations 20 | 21 | For more informations please visit [**my GitHub site**](https://github.com/walkccc). 22 | 23 | By Jay Chen on May 14, 2018. 24 | -------------------------------------------------------------------------------- /docs/ML/08.md: -------------------------------------------------------------------------------- 1 | # "Hello world" of deep learning 2 | 3 | ## Three Steps 4 | 5 | - Step 1: define a set of function 6 | - Step 2: goodness of function 7 | - Step 3: pick the best function 8 | 9 | 老師課堂上的 task 是使用 [Keras](https://keras.io/) 10 | 11 | ```python 12 | model = Sequential() # declare a model 13 | model.add(Dense(input_dim=28*28, output_dim=500)) # Dense: Fully connected 14 | model.add(Activation('sigmoid')) 15 | model.add(Dense(units=500, activation='relu')) 16 | model.add(Dense(units=10, activation='softmax')) # output between 0 and 1 17 | model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) 18 | model.fit(s_train, y_train, batch_size=100, epochs=20) # pick the best function 19 | ``` 20 | 21 | ## Mini-batch 22 | 23 | **Pseudo code:** 24 | 25 | 1. Randomly initialize network parameters 26 | 2. while (!all mini-batches have been picked) 27 | 28 | pick the $i^\text{th}$ batch $L' = C^1 + C^{31} + \cdots$ 29 | update parameters once 30 | 31 | 每跑一次 while-loop，即是一個 epoch。 32 | 33 | - 如果 batch_size = 1：SGP 34 | - 如果 batch_size = #training data：（full batch）Gradient Descent 35 | 36 | ## Matrix Operation 37 | 38 | ### Stochastic Gradient Descent 39 | 40 | ![](https://imgur.com/pQKWG8A.png) 41 | 42 | ### Mini-batch 43 | 44 | ![](https://imgur.com/dMmzSOx.png) 45 | 46 | GPU 的加速，就是因為取 batch 後，能夠平行運算，所以若沒有 batch_size，是無法用 GPU 加速的。 -------------------------------------------------------------------------------- /docs/JavaScript/01/divideAndConquer.md: -------------------------------------------------------------------------------- 1 | # Divide and Conquer Pattern 2 | 3 | This pattern involves dividing a data set into smaller chunks and then repeating a process with a subset of data. 4 | 5 | This pattern can tremendously **decrease time complexity**. 6 | 7 | --- 8 | 9 | ## `search(arr, val)` 10 | 11 | > Given a **sorted** array of integers, write a function called `search`, that accepts a value and returns the index where the value passed to the function is located. If the value is not found, return -1. 12 | 13 | ```js 14 | search([1, 2, 3, 4, 5, 6], 4); // 3 15 | search([1, 2, 3, 4, 5, 6], 6); // 5 16 | search([1, 2, 3, 4, 5, 6], 11); // -1 17 | ``` 18 | 19 | - Solution 1 (naive) 20 | 21 | - Time: $O(n)$ 22 | 23 | ```js 24 | const search = (arr, val) => { 25 | for (let i = 0; i < arr.length; i++) 26 | if (arr[i] === val) 27 | return i; 28 | return -1; 29 | }; 30 | ``` 31 | 32 | - Solution 2 (binary search) 33 | 34 | - Time: $O(\log n)$ 35 | 36 | ```js 37 | const search = (arr, val) => { 38 | let left = 0; 39 | let right = arr.length - 1; 40 | 41 | while (left <= right) { 42 | let mid = Math.floor((left + right) / 2); 43 | let curr = arr[mid]; 44 | 45 | if (arr[mid] < val) { 46 | left = mid + 1; 47 | } else if (arr[mid] > val) { 48 | right = mid - 1; 49 | } else { 50 | return mid; 51 | } 52 | } 53 | 54 | return -1; 55 | }; 56 | ``` 57 | -------------------------------------------------------------------------------- /docs/ML/02.md: -------------------------------------------------------------------------------- 1 | # Lecture 2: Where does the error come from? 2 | 3 | ## Bias and Variance of Estimator 4 | 5 | Error 有兩種來源： 6 | 7 | - bias 8 | - variance 9 | 10 | 分析 error 的來源，可以挑選適當的方法改善 model。 11 | 12 | 以進化前的寶可夢為輸入，以進化後的真實 CP 值為輸出，真實的函數記為 $\hat f$（上帝視角才知道） 13 | 14 | 從訓練數據，我們找到 $f^\*$（$\hat f$ 的估計） 15 | 16 | ![](https://i.imgur.com/Q8eohUL.png) 17 | 18 | 用表格來看 19 | 20 | | | 簡單模型 | 複雜模型 | 21 | |:--:|:--:|:--:| 22 | | variance | 小 | 大 | 23 | | bias | 大 | 小 | 24 | 25 | - 簡單模型更少受 training data 影響。 26 | - 複雜模型會盡力去擬合 training data 的變化。 27 | 28 | Bias 即 $d(\bar f, \hat f)$ 29 | 30 | 簡單模型的 model space 較小，可能根本沒有包含到 target。 31 | 32 | - 在 underfitting 的情況下，error 大部分來自 bias。 33 | - 在 overfitting 的情況下，error 大部分來自 variance。 34 | 35 | --- 36 | 37 | - 如果 model 連 training data 都 fit 不好，那就是 underfitting 38 | - 如果 model 可以 fit training data，但 testing error 大，那就是 overfitting 39 | 40 | --- 41 | 42 | - 在 bias 大的情況下，需要重新設計 model，例如增加更多 features，或著讓 model 更複雜 43 | - 在 variance 大的情況下，需要更多資料，或者 regularization。More data 很有效，但卻不一定可行，regularization 希望曲線平滑，但可能會損害 bias，造成 model space 無法包含 target $\hat f$。 44 | 45 | ## Model Selection 46 | 47 | 在選擇模型時，要考慮 2 種 error 的折衷，使 total error 最小。 48 | 49 | 不應這樣做： 50 | 51 | ![](https://i.imgur.com/QZZrg8C.png) 52 | 53 | 因為這樣做，在 public testing error 的表現不能完全的反應在 private testing error。 54 | 55 | ### Cross Validation 56 | 57 | ![](https://i.imgur.com/NiYfKs9.png) 58 | 59 | 將 training set 分成 training set 和 validation set，在 training set 上訓練 model 1-3，選擇在 validation set 上 error 最小的 model。 60 | 61 | 如果嫌 training set 中 data 太少的話，可以在確定 model 後在全部 training set 上再訓練一遍該 model。 62 | 63 | 這樣做，在 public testing set 上的 error 才會代表在 private testing set 上的 error。 64 | 65 | 不能用 public testing set 去調整 model！ 66 | 67 | ### $N$-fold Cross Validation 68 | 69 | ![](https://i.imgur.com/L93md2I.png) 70 | 71 | 將 training set 分成 $N$ 折，每次只有一折作為 validation set，其它折作為 training set，在各 model 中選擇 $N$ 次訓練得到的 $N$ 個 validation error 均值最小的 model。 72 | -------------------------------------------------------------------------------- /docs/JavaScript/02/recursionEasy.md: -------------------------------------------------------------------------------- 1 | # Recursion Problem Set (easy) 2 | 3 | ## `power(n, x)` 4 | 5 | > Write a function called `power` which accepts a base and an exponent. The function should return the power of the base to the exponent. 6 | > This function should mimic the functionality of `Math.pow()` - do not worry about negative bases and exponents. 7 | 8 | ```js 9 | const power = (n, x) => { 10 | if (x == 0) return 1; 11 | if (x == 1) return n; 12 | let t = power(n, Math.floor(x / 2)); 13 | return t * t * power(n, x % 2); 14 | }; 15 | ``` 16 | 17 | ## `factorial(n)` 18 | 19 | > Write a function `factorial` which accepts a number and returns the factorial of that number. A factorial is the product of an integer and all the integers below it; e.g., factorial four (4!) is equal to 24, because 4 \* 3 \* 2 \* 1 equals 24. **factorial zero (0!) is always 1.** 20 | 21 | ```js 22 | const factorial = n => { 23 | if (n == 0) return 1; 24 | return n * factorial(n - 1); 25 | }; 26 | ``` 27 | 28 | ## `productOfArray(arr)` 29 | 30 | > Write a function called `productOfArray` which takes in an array of numbers and returns the product of them all. 31 | 32 | ```js 33 | const productOfArray = arr => { 34 | if (arr.length == 0) return 1; 35 | return arr[0] * productOfArray(arr.splice(1)); 36 | }; 37 | ``` 38 | 39 | ## `recursiveRange(num)` 40 | 41 | > Write a function called `recursiveRange` which accepts a number and adds up all the numbers from 0 to the number passed to the function. 42 | 43 | ```js 44 | const recursiveRange = num => { 45 | if (num == 0) return 0; 46 | return num + recursiveRange(num - 1); 47 | }; 48 | ``` 49 | 50 | ## `fib(num)` 51 | 52 | > Write a recursive function called `fib` which accepts a number and returns the nth number in the Fibonacci sequence. Recall that the Fibonacci sequence is the sequence of whole numbers 1, 1, 2, 3, 5, 8, which starts with 1 and 1, and where every number thereafter is equal to the sum of the previous two numbers. 53 | 54 | ```js 55 | const fib = num => { 56 | if (num == 1 || num == 2) return 1; 57 | return fib(num - 1) + fib(num - 2); 58 | }; 59 | ``` 60 | -------------------------------------------------------------------------------- /docs/JavaScript/04/MaxBinaryHeap.md: -------------------------------------------------------------------------------- 1 | ```js 2 | class MaxBinaryHeap { 3 | constructor() { 4 | this.arr = []; 5 | } 6 | 7 | insert(elem) { 8 | this.arr = [...this.arr, elem]; 9 | this.bubbleUp(); 10 | } 11 | 12 | bubbleUp() { 13 | let index = this.arr.length - 1; 14 | const elem = this.arr[index]; 15 | 16 | while (index > 0) { 17 | let parentIndex = Math.floor((index - 1) / 2); 18 | let parent = this.arr[parentIndex]; 19 | 20 | if (elem <= parent) break; 21 | 22 | this.arr[parentIndex] = elem; 23 | this.arr[index] = parent; 24 | index = parentIndex; 25 | } 26 | } 27 | 28 | extractMax() { 29 | const max = this.arr[0]; 30 | const end = this.arr.pop(); 31 | 32 | if (this.arr.length > 0) { 33 | this.arr[0] = end; 34 | this.sinkDown(); 35 | } 36 | 37 | return max; 38 | } 39 | 40 | sinkDown() { 41 | let index = 0; 42 | const { length } = this.arr; 43 | const elem = this.arr[index]; 44 | 45 | while (true) { 46 | let leftChildIndex = 2 * index + 1; 47 | let rightChildIndex = 2 * index + 2; 48 | let leftChild = null; 49 | let rightChild = null; 50 | let swap = null; 51 | 52 | if (leftChildIndex < length) { 53 | leftChild = this.arr[leftChildIndex]; 54 | if (leftChild > elem) { 55 | swap = leftChildIndex; 56 | } 57 | } 58 | 59 | if (rightChildIndex < length) { 60 | rightChild = this.arr[rightChildIndex]; 61 | if ( 62 | (swap === null && rightChild > elem) || 63 | (swap !== null && rightChild > leftChild) 64 | ) { 65 | swap = rightChildIndex; 66 | } 67 | } 68 | 69 | if (swap === null) break; 70 | 71 | this.arr[index] = this.arr[swap]; 72 | this.arr[swap] = elem; 73 | index = swap; 74 | } 75 | } 76 | } 77 | ``` 78 | 79 | ```js 80 | let heap = new MaxBinaryHeap(); 81 | 82 | heap.insert(41); 83 | heap.insert(39); 84 | heap.insert(33); 85 | heap.insert(18); 86 | heap.insert(27); 87 | heap.insert(12); 88 | heap.insert(55); 89 | // [55, 39, 41, 18, 27, 12, 33] 90 | 91 | heap.extractMax(); // [41, 39, 33, 18, 27, 12] 92 | ``` 93 | -------------------------------------------------------------------------------- /docs/JavaScript/04/MaxPriorityQueue.md: -------------------------------------------------------------------------------- 1 | ```js 2 | class Node { 3 | constructor(val, priority) { 4 | this.val = val; 5 | this.priority = priority; 6 | } 7 | } 8 | ``` 9 | 10 | ```js 11 | class MaxPriorityQueue { 12 | constructor() { 13 | this.arr = []; 14 | } 15 | 16 | enqueue(val, priority) { 17 | const newNode = new Node(val, priority); 18 | this.arr = [...this.arr, newNode]; 19 | this.bubbleUp(); 20 | } 21 | 22 | bubbleUp() { 23 | let index = this.arr.length - 1; 24 | const elem = this.arr[index]; 25 | 26 | while (index > 0) { 27 | let parentIndex = Math.floor((index - 1) / 2); 28 | let parent = this.arr[parentIndex]; 29 | 30 | if (elem.priority <= parent.priority) break; 31 | 32 | this.arr[parentIndex] = elem; 33 | this.arr[index] = parent; 34 | index = parentIndex; 35 | } 36 | } 37 | 38 | dequeue() { 39 | const max = this.arr[0]; 40 | const end = this.arr.pop(); 41 | 42 | if (this.arr.length > 0) { 43 | this.arr[0] = end; 44 | this.sinkDown(); 45 | } 46 | 47 | return max; 48 | } 49 | 50 | sinkDown() { 51 | let index = 0; 52 | const { length } = this.arr; 53 | const elem = this.arr[index]; 54 | 55 | while (true) { 56 | let leftChildIndex = 2 * index + 1; 57 | let rightChildIndex = 2 * index + 2; 58 | let leftChild = null; 59 | let rightChild = null; 60 | let swap = null; 61 | 62 | if (leftChildIndex < length) { 63 | leftChild = this.arr[leftChildIndex]; 64 | if (leftChild.priority > elem.priority) { 65 | swap = leftChildIndex; 66 | } 67 | } 68 | 69 | if (rightChildIndex < length) { 70 | rightChild = this.arr[rightChildIndex]; 71 | if ( 72 | (swap === null && rightChild.priority > elem.priority) || 73 | (swap !== null && rightChild.priority > leftChild.priority) 74 | ) { 75 | swap = rightChildIndex; 76 | } 77 | } 78 | 79 | if (swap === null) break; 80 | 81 | this.arr[index] = this.arr[swap]; 82 | this.arr[swap] = elem; 83 | index = swap; 84 | } 85 | } 86 | } 87 | ``` 88 | 89 | ```js 90 | let queue = new MaxPriorityQueue(); 91 | 92 | queue.enqueue('Drink water', 1); 93 | queue.enqueue('Sleep', 5); 94 | queue.enqueue('Eat', 2); 95 | ``` 96 | -------------------------------------------------------------------------------- /docs/ML/09.md: -------------------------------------------------------------------------------- 1 | # Tips for Deep Learning 2 | 3 | ## Recipe of Deep Learning 4 | 5 | ![](https://imgur.com/DKQKjDt.png) 6 | 7 | ## Vanishing Gradient Problem 8 | 9 | ![](https://imgur.com/PWU5CNw.png) 10 | 11 | ![](https://imgur.com/yQFRcgO.png) 12 | 13 | 以上圖的例子來看：由於通過神經元的 sigmoid function，數值大的 input 會被壓縮到 $0$ 到 $1$ 之間，以至於彼此明明差異很大的 input，在 output 時的差異卻沒像本來那麼明顯。 14 | 15 | ## ReLU (Rectified Linear Unit) 16 | 17 | ![](https://imgur.com/9uYm0Cw.png) 18 | 19 | pros: 20 | 21 | 1. fast to compute 22 | 2. biological reason 23 | 3. infinite sigmoid with different biases 24 | 4. **vanishing gradient problem** 25 | 26 | ### variant 27 | 28 | | Leack ReLU | Parametric ReLU | 29 | | :--------------------------------: | :--------------------------------: | 30 | | ![](https://imgur.com/9lz8M8T.png) | ![](https://imgur.com/menbnf3.png) | 31 | 32 | $\alpha$ 會由 gradient descent 中學習。 33 | 34 | ## Maxout 35 | 36 | ***ReLU is a special cases of Maxout*** 37 | 38 | - Learnable activation function 39 | 40 | - Activation function in maxout network can be any piecewise linear convex function 41 | - How many pieces depending on how many elements in a group 42 | 43 | ## Adaptive Learning Rate 44 | 45 | ### RMSProp 46 | 47 | $$ 48 | \begin{array} { c l } 49 | { w ^ { 1 } \leftarrow w ^ { 0 } - \frac { \eta } { \sigma ^ { 0 } } g ^ { 0 } } & { \sigma ^ { 0 } = g ^ { 0 } } \\\\ 50 | { w ^ { 2 } \leftarrow w ^ { 1 } - \frac { \eta } { \sigma ^ { 1 } } g ^ { 1 } } & { \sigma ^ { 1 } = \sqrt { \alpha \left( \sigma ^ { 0 } \right) ^ { 2 } + ( 1 - \alpha ) \left( g ^ { 2 } \right) ^ { 2 } } } \\\\ 51 | { w ^ { 3 } \leftarrow w ^ { 2 } - \frac { \eta } { \sigma ^ { 2 } } g ^ { 2 } } & { \sigma ^ { 2 } = \sqrt { \alpha \left( \sigma ^ { 1 } \right) ^ { 2 } + ( 1 - \alpha ) \left( g ^ { 2 } \right) ^ { 2 } } } \\\\ 52 | { \vdots } & { \vdots } \\\\ 53 | { w ^ { t + 1 } \leftarrow w ^ { t } - \frac { \eta } { \sigma ^ { t } } g ^ { t } } & { \sigma ^ { t } = \sqrt { \alpha \left( \sigma ^ { t - 1 } \right) ^ { 2 } + ( 1 - \alpha ) \left( g ^ { t } \right) ^ { 2 } } } 54 | \end{array} 55 | $$ 56 | 57 | Root Mean Square of the gradients with previous gradients being decayed. 58 | 59 | ### Momentum 60 | 61 | ![](https://imgur.com/CRg6kxR.png) 62 | 63 | ### Adam 64 | 65 | Adam = RMSProp + Momentum 66 | 67 | ## Early Stopping 68 | 69 | ## Regularization 70 | 71 | ## Dropout -------------------------------------------------------------------------------- /docs/ML/07.md: -------------------------------------------------------------------------------- 1 | # Backpropagation 2 | 3 | Backpropagation 是一種能有效計算梯度的方法。 4 | 5 | 我們的網路： 6 | 7 | $$x^n \to \text{NN}(\theta) \to y^n \leftrightarrow_{C^n} \hat y^n$$ 8 | 9 | ## Gradient Descent Review 10 | 11 | 假設我們有以下參數：$\theta = \\{w_1, w_2, \dots, b_1, b_2, \dots\\}$ 12 | 13 | 我們會逐次更新參數如下：$\theta^0 \to \theta^1 \to \theta^2 \to \cdots$ 14 | 15 | 但因為有太多參數，所以我們必需「有效地」更新參數。 16 | 17 | 給定 loss function： 18 | 19 | $$ 20 | \begin{aligned} 21 | L ( \theta ) & = \sum _ { n = 1 } ^ { N } C ^ { n } ( \theta ) \\\\ 22 | \to \frac { \partial L ( \theta ) } { \partial w } & = \sum _ { n = 1 } ^ { N } \frac { \partial C ^ { n } ( \theta ) } { \partial w } 23 | \end{aligned} 24 | $$ 25 | 26 | ## Backpropagation 27 | 28 | ![](https://imgur.com/CFHtwpb.png) 29 | 30 | 假設只考慮其中一個 neuron，則 $z = x_1w_1 + x_2w_2 + b$，我們所要求的是： 31 | 32 | $$\frac { \partial C } { \partial w } = \frac { \partial z } { \partial w } \frac { \partial C } { \partial z }$$ 33 | 34 | 透過 forward pass 和 backward pass 可分別算出： 35 | 36 | - Forward pass: Compute $\partial z / \partial w$ for all parameters 37 | - Backward pass: Compute $\partial C / \partial z$ for all activation function inputs $z$ 38 | 39 | ### Forward pass 40 | 41 | - $\partial z / \partial w_1 = x_1$ 42 | - $\partial z / \partial w_2 = x_2$ 43 | 44 | 其實就是 input value 前面所接的值。 45 | 46 | 你要算出這個 neural network 裡面的每一個 weight 對它的 activation function 的 input $z$ 的偏微分，你就把你的 input 丟進去。然後，計算每一個 neuron 的 output 就結束了。 47 | 48 | ### Backward pass 49 | 50 | 要計算出 $\frac { \partial C } { \partial z }$ 是非常複雜的，但我們可以透過 chain rule 做些拆解： 51 | 52 | ![](https://imgur.com/C67xQpB.png) 53 | 54 | $$ 55 | \begin{aligned} 56 | \frac { \partial C } { \partial z } & = \frac { \partial a } { \partial z } \frac { \partial C } { \partial a } \\\\ 57 | & = \sigma ^ { \prime } ( z ) \left[ \frac { \partial z ^ { \prime } } { \partial a } \frac { \partial C } { \partial z ^ { \prime } } + \frac { \partial z ^ { \prime \prime } } { \partial a } \frac { \partial C } { \partial z ^ { \prime \prime } } \right] \\\\ 58 | & = \sigma ^ { \prime } ( z ) \left[ w _ { 3 } \frac { \partial C } { \partial z ^ { \prime } } + w _ { 4 } \frac { \partial C } { \partial z ^ { \prime \prime } } \right] 59 | \end{aligned} 60 | $$ 61 | 62 | 想像有另一個 network，方向是反過來的，下面這個網路即和上式所代表的意義一樣： 63 | 64 | ![](https://imgur.com/R5CVdpT.png) 65 | 66 | #### Case 1. Output Layer 67 | 68 | ![](https://imgur.com/jPZZPaa.png) 69 | 70 | $$ 71 | \begin{aligned} 72 | \frac { \partial C } { \partial z ^ { \prime } } & = \frac { \partial y _ { 1 } } { \partial z ^ { \prime } } \frac { \partial C } { \partial y _ { 1 } } \\\\ 73 | \frac { \partial C } { \partial z ^ { \prime \prime } } & = \frac { \partial y _ { 2 } } { \partial z ^ { \prime \prime } } \frac { \partial C } { \partial y _ { 2 } } 74 | \end{aligned} 75 | $$ 76 | 77 | #### Case 2. Not Output Layer 78 | 79 | ![](https://imgur.com/4LO37Jp.png) 80 | 81 | Compute $\partial C / \partial z$ recursively until we reach the output layer. 82 | 83 | ## Summary 84 | 85 | ![](https://imgur.com/PIJqv1u.png) -------------------------------------------------------------------------------- /docs/JavaScript/05/Graph.md: -------------------------------------------------------------------------------- 1 | ```js 2 | class Graph { 3 | constructor() { 4 | this.adjList = {}; 5 | } 6 | 7 | addVertex(vertex) { 8 | if (!this.adjList[vertex]) this.adjList[vertex] = []; 9 | } 10 | 11 | addEdge(v1, v2) { 12 | this.adjList[v1] = [...this.adjList[v1], v2]; 13 | this.adjList[v2] = [...this.adjList[v2], v1]; 14 | } 15 | 16 | removeEdge(v1, v2) { 17 | this.adjList[v1] = this.adjList[v1].filter(v => v !== v2); 18 | this.adjList[v2] = this.adjList[v2].filter(v => v !== v1); 19 | } 20 | 21 | removeVertex(vertex) { 22 | while (this.adjList[vertex].length) { 23 | const adjVertex = this.adjList[vertex].pop(); 24 | this.removeEdge(vertex, adjVertex); 25 | } 26 | delete this.adjList[vertex]; 27 | } 28 | 29 | dfsRecursive(start) { 30 | const ret = []; 31 | const visited = {}; 32 | const adjList = this.adjList; 33 | 34 | (function dfs(vertex) { 35 | if (!vertex) return null; 36 | 37 | visited[vertex] = true; 38 | ret.push(vertex); 39 | 40 | adjList[vertex].forEach(neighbor => { 41 | if (!visited[neighbor]) { 42 | return dfs(neighbor); 43 | } 44 | }); 45 | })(start); 46 | 47 | return ret; 48 | } 49 | 50 | dfsIterative(start) { 51 | const ret = []; 52 | const visited = {}; 53 | const stack = [start]; 54 | let currVertex; 55 | 56 | visited[start] = true; 57 | 58 | while (stack.length) { 59 | currVertex = stack.pop(); 60 | ret.push(currVertex); 61 | 62 | this.adjList[currVertex].forEach(neighbor => { 63 | if (!visited[neighbor]) { 64 | visited[neighbor] = true; 65 | stack.push(neighbor); 66 | } 67 | }); 68 | } 69 | 70 | return ret; 71 | } 72 | 73 | bfs(start) { 74 | const ret = []; 75 | const visited = {}; 76 | const queue = [start]; 77 | let currVertex; 78 | 79 | visited[start] = true; 80 | 81 | while (queue.length) { 82 | currVertex = queue.shift(); 83 | ret.push(currVertex); 84 | 85 | this.adjList[currVertex].forEach(neighbor => { 86 | if (!visited[neighbor]) { 87 | visited[neighbor] = true; 88 | queue.push(neighbor); 89 | } 90 | }); 91 | } 92 | 93 | return ret; 94 | } 95 | } 96 | ``` 97 | 98 | ```js 99 | let g = new Graph(); 100 | 101 | g.addVertex('A'); 102 | g.addVertex('B'); 103 | g.addVertex('C'); 104 | g.addVertex('D'); 105 | g.addVertex('E'); 106 | g.addVertex('F'); 107 | 108 | g.addEdge('A', 'B'); 109 | g.addEdge('A', 'C'); 110 | g.addEdge('B', 'D'); 111 | g.addEdge('C', 'E'); 112 | g.addEdge('D', 'E'); 113 | g.addEdge('D', 'F'); 114 | g.addEdge('E', 'F'); 115 | 116 | // A 117 | // / \ 118 | // B C 119 | // | | 120 | // D - E 121 | // \ / 122 | // F 123 | 124 | g.dfsRecursive('A'); // ['A', 'B', 'D', 'E', 'C', 'F'] 125 | g.dfsIterative('A'); // ['A', 'C', 'E', 'F', 'D', 'B'] 126 | g.bfs('A'); // ['A', 'B', 'C', 'D', 'E', 'F'] 127 | ``` 128 | -------------------------------------------------------------------------------- /docs/JavaScript/02/recursionHard.md: -------------------------------------------------------------------------------- 1 | # Recursion Problem Set (hard) 2 | 3 | ## `reverse(str)` 4 | 5 | > Write a recursive function called `reverse` which accepts a string and returns a new string in reverse. 6 | 7 | ```js 8 | reverse('awesome'); // 'emosewa' 9 | reverse('rithmschool'); // 'loohcsmhtir' 10 | ``` 11 | 12 | - Solution: 13 | 14 | ```js 15 | const reverse = str => { 16 | if (str.length <= 1) return str; 17 | return reverse(str.slice(1)) + str[0]; 18 | }; 19 | ``` 20 | 21 | ## `isPalindrome(str)` 22 | 23 | > Write a recursive function called `isPalindrome` which returns true if the string passed to it is a palindrome (reads the same forward and backward). Otherwise it returns false. 24 | 25 | ```js 26 | isPalindrome('awesome'); // false 27 | isPalindrome('foobar'); // false 28 | isPalindrome('tacocat'); // true 29 | isPalindrome('amanaplanacanalpanama'); // true 30 | isPalindrome('amanaplanacanalpandemonium'); // false 31 | ``` 32 | 33 | - Solution: 34 | 35 | ```js 36 | const isPalindrome = str => { 37 | if (str.length == 1) return true; 38 | if (str.length == 2) return str[0] == str[1]; 39 | if (str[0] === str.slice(-1)) return isPalindrome(str.slice(1, -1)); 40 | return false; 41 | }; 42 | ``` 43 | 44 | ## `someRecursive(arr, callback)` 45 | 46 | > Write a recursive function called `someRecursive` which accepts an array and a callback. The function returns true if a single value in the array returns true when passed to the callback. Otherwise it returns false. 47 | 48 | ```js 49 | const isOdd = val => val % 2 !== 0; 50 | someRecursive([1, 2, 3, 4], isOdd); // true 51 | someRecursive([4, 6, 8, 9], isOdd); // true 52 | someRecursive([4, 6, 8], isOdd); // false 53 | someRecursive([4, 6, 8], val => val > 10); // false 54 | ``` 55 | 56 | - Solution: 57 | 58 | ```js 59 | const someRecursive = (arr, callback) => { 60 | if (callback(arr[0])) return true; 61 | if (arr.length == 1) return callback(arr[0]); 62 | return someRecursive(arr.slice(1), callback); 63 | }; 64 | ``` 65 | 66 | ## `flatten` 67 | 68 | > Write a recursive function called `flatten` which accepts an array of arrays and returns a new array with all values flattened. 69 | 70 | ```js 71 | flatten([1, 2, 3, [4, 5]]); // [1, 2, 3, 4, 5] 72 | flatten([1, [2, [3, 4], [[5]]]]); // [1, 2, 3, 4, 5] 73 | flatten([[1], [2], [3]]); // [1, 2, 3] 74 | flatten([[[[1], [[[2]]], [[[[[[[3]]]]]]]]]]); // [1, 2, 3] 75 | ``` 76 | 77 | - Solution: 78 | 79 | ```js 80 | const flatten = oldArr => { 81 | let newArr = []; 82 | for (let i = 0; i < oldArr.length; i++) { 83 | if (Array.isArray(oldArr[i])) { 84 | newArr = newArr.concat(flatten(oldArr[i])); 85 | } else { 86 | newArr = [...newArr, oldArr[i]]; 87 | } 88 | } 89 | return newArr; 90 | }; 91 | ``` 92 | 93 | ## `capitalizeFirst` 94 | 95 | > Write a recursive function called `capitalizeFirst`. Given an array of strings, capitalize the first letter of each string in the array. 96 | 97 | ```js 98 | capitalizeFirst(['car', 'taco', 'banana']); // ['Car', 'Taco', 'Banana'] 99 | ``` 100 | 101 | - Solution: 102 | 103 | ``` 104 | 105 | ``` 106 | -------------------------------------------------------------------------------- /docs/PCP/1-3.md: -------------------------------------------------------------------------------- 1 | # 3. Mutual Exclusion 2 | 3 | ## Data race 4 | 5 | === "begin" 6 | 7 | ```cpp 8 | // Two shoppers adding items to a shared notepad 9 | #include 10 | 11 | unsigned int garlic_count = 0; 12 | 13 | void shopper() { 14 | for (int i = 0; i < 10; ++i) { 15 | ++garlic_count; 16 | } 17 | } 18 | 19 | int main() { 20 | std::thread barron(shopper); 21 | std::thread olivia(shopper); 22 | barron.join(); 23 | olivia.join(); 24 | printf("We should buy %u garlic.\n", garlic_count); 25 | } 26 | ``` 27 | 28 | === "end" 29 | 30 | ```cpp hl_lines="7" 31 | // Two shoppers adding items to a shared notepad 32 | #include 33 | 34 | unsigned int garlic_count = 0; 35 | 36 | void shopper() { 37 | for (int i = 0; i < 10000000; ++i) { 38 | ++garlic_count; 39 | } 40 | } 41 | 42 | int main() { 43 | std::thread barron(shopper); 44 | std::thread olivia(shopper); 45 | barron.join(); 46 | olivia.join(); 47 | printf("We should buy %u garlic.\n", garlic_count); 48 | } 49 | ``` 50 | 51 | ## Mutual exclusion 52 | 53 | === "begin" 54 | 55 | ```cpp 56 | // Two shoppers adding items to a shared notepad 57 | 58 | #include 59 | #include 60 | 61 | unsigned int garlic_count = 0; 62 | std::mutex pencil; 63 | 64 | void shopper() { 65 | pencil.lock(); 66 | for (int i = 0; i < 10000000; ++i) { 67 | ++garlic_count; 68 | } 69 | pencil.unlock(); 70 | } 71 | 72 | int main() { 73 | std::thread barron(shopper); 74 | std::thread olivia(shopper); 75 | barron.join(); 76 | olivia.join(); 77 | printf("We should buy %u garlic.\n", garlic_count); 78 | } 79 | ``` 80 | 81 | === "end" 82 | 83 | ```cpp hl_lines="15 16 17" 84 | /** 85 | * Two shoppers adding items to a shared notepad 86 | */ 87 | #include 88 | #include 89 | #include 90 | 91 | unsigned int garlic_count = 0; 92 | std::mutex pencil; 93 | 94 | void shopper() { 95 | for (int i = 0; i < 5; ++i) { 96 | printf("Shopper %d is thinking...\n", std::this_thread::get_id()); 97 | std::this_thread::sleep_for(std::chrono::milliseconds(500)); 98 | pencil.lock(); 99 | ++garlic_count; 100 | pencil.unlock(); 101 | } 102 | } 103 | 104 | int main() { 105 | std::thread barron(shopper); 106 | std::thread olivia(shopper); 107 | barron.join(); 108 | olivia.join(); 109 | printf("We should buy %u garlic.\n", garlic_count); 110 | } 111 | ``` 112 | 113 | ## Atomic objects 114 | 115 | === "begin" 116 | 117 | ```cpp 118 | // Two shoppers adding items to a shared notepad 119 | #include 120 | 121 | unsigned int garlic_count = 0; 122 | 123 | void shopper() { 124 | for (int i = 0; i < 10000000; ++i) { 125 | ++garlic_count; 126 | } 127 | } 128 | 129 | int main() { 130 | std::thread barron(shopper); 131 | std::thread olivia(shopper); 132 | barron.join(); 133 | olivia.join(); 134 | printf("We should buy %u garlic.\n", garlic_count); 135 | } 136 | ``` 137 | 138 | === "end" 139 | 140 | ```cpp hl_lines="2 5" 141 | // Two shoppers adding items to a shared notepad 142 | #include 143 | #include 144 | 145 | std::atomic garlic_count(0); 146 | 147 | void shopper() { 148 | for (int i = 0; i < 10000000; ++i) { 149 | ++garlic_count; 150 | } 151 | } 152 | 153 | int main() { 154 | std::thread barron(shopper); 155 | std::thread olivia(shopper); 156 | barron.join(); 157 | olivia.join(); 158 | printf("We should buy %u garlic.\n", garlic_count.load()); 159 | } 160 | ``` 161 | -------------------------------------------------------------------------------- /docs/JavaScript/04/BinarySearchTree.md: -------------------------------------------------------------------------------- 1 | ```js 2 | class Node { 3 | constructor(val) { 4 | this.val = val; 5 | this.left = null; 6 | this.right = null; 7 | } 8 | } 9 | ``` 10 | 11 | ```js 12 | class BinarySearchTree { 13 | constructor() { 14 | this.root = null; 15 | } 16 | 17 | insert(val) { 18 | const newNode = new Node(val); 19 | 20 | if (this.root === null) { 21 | this.root = newNode; 22 | return this; 23 | } 24 | 25 | let curr = this.root; 26 | 27 | while (true) { 28 | if (val === curr.val) return undefined; 29 | if (val < curr.val) { 30 | if (curr.left === null) { 31 | curr.left = newNode; 32 | return this; 33 | } 34 | curr = curr.left; 35 | } else { 36 | if (curr.right === null) { 37 | curr.right = newNode; 38 | return this; 39 | } 40 | curr = curr.right; 41 | } 42 | } 43 | } 44 | 45 | find(val) { 46 | if (this.root === null) return false; 47 | 48 | let curr = this.root; 49 | let found = false; 50 | 51 | while (curr && !found) { 52 | if (val < curr.val) { 53 | curr = curr.left; 54 | } else if (val > curr.val) { 55 | curr = curr.right; 56 | } else { 57 | found = true; 58 | } 59 | } 60 | 61 | if (!found) return undefined; 62 | return curr; 63 | } 64 | 65 | contains(val) { 66 | if (this.root === null) return false; 67 | 68 | let curr = this.root; 69 | let found = false; 70 | 71 | while (curr && !found) { 72 | if (val < curr.val) { 73 | curr = curr.left; 74 | } else if (val > curr.val) { 75 | curr = curr.right; 76 | } else { 77 | return true; 78 | } 79 | } 80 | 81 | return false; 82 | } 83 | 84 | bfs() { 85 | let node = this.root; 86 | const ret = []; 87 | const queue = [node]; 88 | 89 | while (queue.length) { 90 | node = queue.shift(); 91 | ret.push(node.val); 92 | if (node.left) queue.push(node.left); 93 | if (node.right) queue.push(node.right); 94 | } 95 | 96 | return ret; 97 | } 98 | 99 | dfs() { 100 | let node = this.root; 101 | const ret = []; 102 | const stack = [node]; 103 | 104 | while (stack.length) { 105 | node = stack.pop(); 106 | ret.push(node.val); 107 | if (node.right) stack.push(node.right); 108 | if (node.left) stack.push(node.left); 109 | } 110 | 111 | return ret; 112 | } 113 | 114 | preOrder() { 115 | const ret = []; 116 | 117 | const traverse = node => { 118 | ret.push(node.val); 119 | if (node.left) traverse(node.left); 120 | if (node.right) traverse(node.right); 121 | }; 122 | 123 | traverse(this.root); 124 | 125 | return ret; 126 | } 127 | 128 | postOrder() { 129 | const ret = []; 130 | 131 | const traverse = node => { 132 | if (node.left) traverse(node.left); 133 | if (node.right) traverse(node.right); 134 | ret.push(node.val); 135 | }; 136 | 137 | traverse(this.root); 138 | 139 | return ret; 140 | } 141 | 142 | inOrder() { 143 | const ret = []; 144 | 145 | const traverse = node => { 146 | if (node.left) traverse(node.left); 147 | ret.push(node.val); 148 | if (node.right) traverse(node.right); 149 | }; 150 | 151 | traverse(this.root); 152 | 153 | return ret; 154 | } 155 | } 156 | ``` 157 | 158 | ```js 159 | var tree = new BinarySearchTree(); 160 | 161 | tree.insert(10); 162 | tree.insert(6); 163 | tree.insert(3); 164 | tree.insert(8); 165 | tree.insert(15); 166 | tree.insert(20); 167 | 168 | // 10 169 | // 6 15 170 | // 3 8 20 171 | 172 | tree.bfs(); // [10, 6, 15, 3, 8, 20] 173 | tree.dfs(); // [10, 6, 3, 8, 15, 20] 174 | tree.preOrder(); // [10, 6, 3, 8, 15, 20] 175 | tree.postOrder(); // [3, 8, 6, 20, 15, 10] 176 | tree.inOrder(); // [3, 6, 8, 10, 15, 20] 177 | ``` 178 | -------------------------------------------------------------------------------- /docs/JavaScript/04/SinglyLinkedList.md: -------------------------------------------------------------------------------- 1 | ```js 2 | class Node { 3 | constructor(val) { 4 | this.val = val; 5 | this.next = null; 6 | } 7 | } 8 | ``` 9 | 10 | ```js 11 | class SinglyLinkedList { 12 | constructor() { 13 | this.head = null; 14 | this.tail = null; 15 | this.length = 0; 16 | } 17 | 18 | push(val) { 19 | const newNode = new Node(val); 20 | 21 | if (!this.head) { 22 | this.head = newNode; 23 | this.tail = this.head; 24 | } else { 25 | this.tail.next = newNode; 26 | this.tail = newNode; 27 | } 28 | 29 | this.length++; 30 | return this; 31 | } 32 | 33 | pop() { 34 | if (!this.head) return undefined; 35 | 36 | let poppedNode = this.head; 37 | let newTail = poppedNode; 38 | 39 | if (this.length === 1) { 40 | this.head = null; 41 | this.tail = null; 42 | } else { 43 | while (poppedNode.next) { 44 | newTail = poppedNode; 45 | poppedNode = poppedNode.next; 46 | } 47 | this.tail = newTail; 48 | this.tail.next = null; 49 | } 50 | 51 | this.length--; 52 | return poppedNode; 53 | } 54 | 55 | shift() { 56 | if (this.length === 0) return undefined; 57 | 58 | let oldHead = this.head; 59 | 60 | if (this.length === 1) { 61 | this.head = null; 62 | this.tail = null; 63 | } else { 64 | this.head = oldHead.next; 65 | oldHead.next = null; 66 | } 67 | 68 | this.length--; 69 | return oldHead; 70 | } 71 | 72 | unshift(val) { 73 | const newNode = new Node(val); 74 | 75 | if (this.length === 0) { 76 | this.head = newNode; 77 | this.tail = newNode; 78 | } else { 79 | newNode.next = this.head; 80 | this.head = newNode; 81 | } 82 | 83 | this.length++; 84 | return this; 85 | } 86 | 87 | get(index) { 88 | if (index < 0 || index >= this.length) return null; 89 | 90 | let count = 0; 91 | let curr = this.head; 92 | 93 | while (count !== index) { 94 | curr = curr.next; 95 | count++; 96 | } 97 | 98 | return curr; 99 | } 100 | 101 | set(index, val) { 102 | let foundNode = this.get(index); 103 | 104 | if (foundNode) { 105 | foundNode.val = val; 106 | return true; 107 | } 108 | 109 | return false; 110 | } 111 | 112 | insert(index, val) { 113 | if (index < 0 || index > this.length) return false; 114 | if (index === 0) return this.unshift(val); 115 | if (index === this.length) return [...this, val]; 116 | 117 | const newNode = new Node(val); 118 | let prev = this.get(index - 1); 119 | let next = prev.next; 120 | 121 | prev.next = newNode; 122 | newNode.next = next; 123 | 124 | this.length++; 125 | return true; 126 | } 127 | 128 | remove(index) { 129 | if (index < 0 || index >= this.length) return undefined; 130 | if (index === 0) return this.shift(); 131 | if (index === this.length - 1) return this.pop(); 132 | 133 | let prev = this.get(index - 1); 134 | let removedNode = prev.next; 135 | 136 | prev.next = removedNode.next; 137 | removedNode.next = null; 138 | 139 | this.length--; 140 | return removedNode; 141 | } 142 | 143 | reverse() { 144 | let node = this.head; 145 | this.head = this.tail; 146 | this.tail = node; 147 | 148 | let oldNext; 149 | let prev = null; 150 | 151 | for (let i = 0; i < this.length; i++) { 152 | oldNext = node.next; 153 | node.next = prev; 154 | prev = node; 155 | node = oldNext; 156 | } 157 | 158 | return this; 159 | } 160 | 161 | print() { 162 | const arr = []; 163 | let curr = this.head; 164 | 165 | while (curr) { 166 | arr.push(curr.val); 167 | curr = curr.next; 168 | } 169 | 170 | console.log(arr); 171 | } 172 | } 173 | ``` 174 | 175 | ```js 176 | let list = new SinglyLinkedList(); 177 | 178 | for (let i = 0; i < 5; i++) list.push(i); 179 | ``` 180 | -------------------------------------------------------------------------------- /docs/JavaScript/04/DoublyLinkedList.md: -------------------------------------------------------------------------------- 1 | ```js 2 | class Node { 3 | constructor(val) { 4 | this.val = val; 5 | this.next = null; 6 | this.prev = null; 7 | } 8 | } 9 | ``` 10 | 11 | ```js 12 | class DoublyLinkedList { 13 | constructor() { 14 | this.head = null; 15 | this.tail = null; 16 | this.length = 0; 17 | } 18 | 19 | push(val) { 20 | const newNode = new Node(val); 21 | 22 | if (this.length === 0) { 23 | this.head = newNode; 24 | this.tail = newNode; 25 | } else { 26 | this.tail.next = newNode; 27 | newNode.prev = this.tail; 28 | this.tail = newNode; 29 | } 30 | 31 | this.length++; 32 | return this; 33 | } 34 | 35 | pop() { 36 | if (!this.head) return undefined; 37 | 38 | let poppedNode = this.tail; 39 | 40 | if (this.length === 1) { 41 | this.head = null; 42 | this.tail = null; 43 | } else { 44 | this.tail = poppedNode.prev; 45 | this.tail.next = null; 46 | poppedNode.prev = null; 47 | } 48 | 49 | this.length--; 50 | return poppedNode; 51 | } 52 | 53 | shift() { 54 | if (this.length === 0) return undefined; 55 | 56 | let oldHead = this.head; 57 | 58 | if (this.length === 1) { 59 | this.head = null; 60 | this.tail = null; 61 | } else { 62 | this.head = oldHead.next; 63 | this.head.prev = null; 64 | oldHead.next = null; 65 | } 66 | 67 | this.length--; 68 | return oldHead; 69 | } 70 | 71 | unshift(val) { 72 | const newNode = new Node(val); 73 | 74 | if (this.length === 0) { 75 | this.head = newNode; 76 | this.tail = newNode; 77 | } else { 78 | this.head.prev = newNode; 79 | newNode.next = this.head; 80 | this.head = newNode; 81 | } 82 | 83 | this.length++; 84 | return this; 85 | } 86 | 87 | get(index) { 88 | if (index < 0 || index >= this.length) return null; 89 | 90 | let curr = null; 91 | if (index <= this.length / 2) { 92 | let count = 0; 93 | curr = this.head; 94 | while (count !== index) { 95 | curr = curr.next; 96 | count++; 97 | } 98 | } else { 99 | let count = this.length - 1; 100 | curr = this.tail; 101 | while (count !== index) { 102 | curr = curr.prev; 103 | count--; 104 | } 105 | } 106 | 107 | return curr; 108 | } 109 | 110 | set(index, val) { 111 | let foundNode = this.get(index); 112 | 113 | if (foundNode) { 114 | foundNode.val = val; 115 | return true; 116 | } 117 | 118 | return false; 119 | } 120 | 121 | insert(index, val) { 122 | if (index < 0 || index > this.length) return false; 123 | if (index === 0) return this.unshift(val); 124 | if (index === this.length) return [...this, val]; 125 | 126 | const newNode = new Node(val); 127 | let prev = this.get(index - 1); 128 | let next = prev.next; 129 | 130 | (prev.next = newNode), (newNode.prev = prev); 131 | (newNode.next = next), (next.prev = newNode); 132 | 133 | this.length++; 134 | return true; 135 | } 136 | 137 | remove(index) { 138 | if (index < 0 || index >= this.length) return undefined; 139 | if (index === 0) return this.shift(); 140 | if (index === this.length) return this.pop(); 141 | 142 | let removedNode = this.get(index); 143 | 144 | removedNode.prev.next = removedNode.next; 145 | removedNode.next.prev = removedNode.prev; 146 | removedNode.next = null; 147 | removedNode.prev = null; 148 | 149 | this.length--; 150 | return removedNode; 151 | } 152 | 153 | print() { 154 | const arr = []; 155 | let curr = this.head; 156 | 157 | while (curr) { 158 | arr.push(curr.val); 159 | curr = curr.next; 160 | } 161 | 162 | console.log(arr); 163 | } 164 | } 165 | ``` 166 | 167 | ```js 168 | let list = new DoublyLinkedList(); 169 | 170 | for (let i = 0; i < 5; i++) list.push(i); 171 | ``` 172 | -------------------------------------------------------------------------------- /docs/JavaScript/01/slidingWindow.md: -------------------------------------------------------------------------------- 1 | # Sliding Window Pattern 2 | 3 | This pattern involves creating a **window** which can either be an array or number from one position to another. 4 | 5 | Depending on a certain condition, the window either increases or closes (and a new window is created). 6 | 7 | Very useful for keeping track of a subset of data in an array/string etc. 8 | 9 | --- 10 | 11 | ## `maxSubarraySum(arr, n)` 12 | 13 | > Write a function called `maxSubarraySum` which accepts an array of integers and a number called **n**. The function should calculate the maximum sum of **n** consecutive elements in the array. 14 | 15 | ```js 16 | maxSubarraySum([1, 2, 5, 2, 8, 1, 5], 2); // 10 17 | maxSubarraySum([1, 2, 5, 2, 8, 1, 5], 4); // 17 18 | maxSubarraySum([4, 2, 1, 6], 1); // 6 19 | maxSubarraySum([], 4); // null 20 | ``` 21 | 22 | - Solution 1 (naive): 23 | 24 | - Time: $O(n^2)$ 25 | 26 | ```js 27 | const maxSubarraySum = (arr, n) => { 28 | if (n > arr.length) return null; 29 | 30 | let ret = -Infinity; 31 | 32 | for (let i = 0; i < arr.length - n + 1; i++) { 33 | let temp = 0; 34 | for (let j = 0; j < n; j++) 35 | temp += arr[i + j]; 36 | 37 | ret = Math.max(ret, temp); 38 | } 39 | 40 | return ret; 41 | }; 42 | ``` 43 | 44 | - Solution 2 (refactor): 45 | 46 | - Time: $O(n)$ 47 | - Space: $O(1)$ 48 | 49 | ```js 50 | const maxSubarraySum = (arr, n) => { 51 | if (arr.length < n) return null; 52 | 53 | let ret = 0; 54 | let temp = 0; 55 | 56 | for (let i = 0; i < n; i++) ret += arr[i]; 57 | 58 | temp = ret; 59 | for (let i = n; i < arr.length; i++) { 60 | temp = temp - arr[i - n] + arr[i]; 61 | ret = Math.max(ret, temp); 62 | } 63 | 64 | return ret; 65 | }; 66 | ``` 67 | 68 | ## `minSubArrayLen(arr, num)` 69 | 70 | > Write a function called `minSubArrayLen` which accepts two parameters - an array of positive integers and a positive integer. 71 | > 72 | > This function should return the minimal length of a contiguous subarray of which the sum is greater than or equal to the integer passed to the function. If there isn’t one, return 0 instead. 73 | 74 | ```js 75 | minSubArrayLen([2, 3, 1, 2, 4, 3], 7); // 2 -> because [4, 3] is the smallest subarray 76 | minSubArrayLen([2, 1, 6, 5, 4], 9); // 2 -> because [5, 4] is the smallest subarray 77 | minSubArrayLen([3, 1, 62, 19], 52); // 1 -> because [62] is greater than 52 78 | ``` 79 | 80 | - Solution 81 | 82 | - Time: $O(n)$ 83 | - Space: $O(1)$ 84 | 85 | ```js 86 | const minSubArrayLen = (arr, num) => { 87 | let i = 0; // start 88 | let j = 0; // end 89 | let sum = 0; 90 | let ret = Infinity; 91 | 92 | while (i < arr.length) { 93 | if (sum < num && j < arr.length) { 94 | sum += arr[j]; 95 | j++; 96 | } else if (sum >= num) { 97 | ret = Math.min(ret, j - i); 98 | sum -= arr[i]; 99 | i++; 100 | } else { 101 | break; 102 | } 103 | } 104 | 105 | return ret === Infinity ? 0 : ret; 106 | }; 107 | ``` 108 | 109 | ## `findLongestSubstring(str)` 110 | 111 | > Write a function called `findLongestSubstring`, which accepts a string and returns the length of the longest substring with all distinct characters. 112 | 113 | ```js 114 | findLongestSubstring(''); // 0 115 | findLongestSubstring('rithmschool'); // 7 116 | findLongestSubstring('thecatinthehat'); // 7 117 | findLongestSubstring('bbbbbb'); // 1 118 | ``` 119 | 120 | - Solution: 121 | 122 | - Time: $O(n)$ 123 | 124 | ```js 125 | const findLongestSubstring = str => { 126 | let ret = 0; 127 | let seen = {}; 128 | let i = 0; 129 | 130 | for (let j = 0; j < str.length; j++) { 131 | let char = str[j]; 132 | if (seen[char]) i = Math.max(i, seen[char]); 133 | ret = Math.max(ret, j - i + 1); 134 | seen[char] = j + 1; 135 | } 136 | 137 | return ret; 138 | }; 139 | ``` 140 | -------------------------------------------------------------------------------- /docs/JavaScript/05/Dijkstra.md: -------------------------------------------------------------------------------- 1 | ```js 2 | class Node { 3 | constructor(val, priority) { 4 | this.val = val; 5 | this.priority = priority; 6 | } 7 | } 8 | ``` 9 | 10 | ```js 11 | class MinPriorityQueue { 12 | constructor() { 13 | this.arr = []; 14 | } 15 | 16 | enqueue(val, priority) { 17 | const newNode = new Node(val, priority); 18 | this.arr = [...this.arr, newNode]; 19 | this.bubbleUp(); 20 | } 21 | 22 | bubbleUp() { 23 | let index = this.arr.length - 1; 24 | const elem = this.arr[index]; 25 | 26 | while (index > 0) { 27 | let parentIndex = Math.floor((index - 1) / 2); 28 | let parent = this.arr[parentIndex]; 29 | 30 | if (elem.priority >= parent.priority) break; 31 | 32 | this.arr[parentIndex] = elem; 33 | this.arr[index] = parent; 34 | index = parentIndex; 35 | } 36 | } 37 | 38 | dequeue() { 39 | const min = this.arr[0]; 40 | const end = this.arr.pop(); 41 | 42 | if (this.arr.length > 0) { 43 | this.arr[0] = end; 44 | this.sinkDown(); 45 | } 46 | 47 | return min; 48 | } 49 | 50 | sinkDown() { 51 | let index = 0; 52 | const { length } = this.arr; 53 | const elem = this.arr[index]; 54 | 55 | while (true) { 56 | let leftChildIndex = 2 * index + 1; 57 | let rightChildIndex = 2 * index + 2; 58 | let leftChild = null; 59 | let rightChild = null; 60 | let swap = null; 61 | 62 | if (leftChildIndex < length) { 63 | leftChild = this.arr[leftChildIndex]; 64 | if (leftChild.priority < elem.priority) { 65 | swap = leftChildIndex; 66 | } 67 | } 68 | 69 | if (rightChildIndex < length) { 70 | rightChild = this.arr[rightChildIndex]; 71 | if ( 72 | (swap === null && rightChild.priority < elem.priority) || 73 | (swap !== null && rightChild.priority < leftChild.priority) 74 | ) { 75 | swap = rightChildIndex; 76 | } 77 | } 78 | 79 | if (swap === null) break; 80 | 81 | this.arr[index] = this.arr[swap]; 82 | this.arr[swap] = elem; 83 | index = swap; 84 | } 85 | } 86 | } 87 | ``` 88 | 89 | ```js 90 | class WeightedGraph { 91 | constructor() { 92 | this.adjList = {}; 93 | } 94 | 95 | addVertex(vertex) { 96 | if (!this.adjList[vertex]) this.adjList[vertex] = []; 97 | } 98 | 99 | addEdge(v1, v2, weight) { 100 | this.adjList[v1] = [...this.adjList[v1], { node: v2, weight }]; 101 | this.adjList[v2] = [...this.adjList[v2], { node: v1, weight }]; 102 | } 103 | 104 | Dijkstra(start, end) { 105 | const nodes = new MinPriorityQueue(); 106 | const dist = {}; 107 | const prev = {}; 108 | let path = []; 109 | let smallest; 110 | 111 | for (let vertex in this.adjList) { 112 | if (vertex === start) { 113 | dist[vertex] = 0; 114 | nodes.enqueue(vertex, 0); 115 | } else { 116 | dist[vertex] = Infinity; 117 | nodes.enqueue(vertex, Infinity); 118 | } 119 | prev[vertex] = null; 120 | } 121 | 122 | while (nodes.arr.length) { 123 | smallest = nodes.dequeue().val; 124 | if (smallest === end) { 125 | while (prev[smallest]) { 126 | path.push(smallest); 127 | smallest = prev[smallest]; 128 | } 129 | break; 130 | } 131 | 132 | if (smallest || dist[smallest] !== Infinity) { 133 | for (let neighbor in this.adjList[smallest]) { 134 | let nextNode = this.adjList[smallest][neighbor]; 135 | let candidate = dist[smallest] + nextNode.weight; 136 | let nextNeighbor = nextNode.node; 137 | 138 | if (candidate < dist[nextNeighbor]) { 139 | dist[nextNeighbor] = candidate; 140 | prev[nextNeighbor] = smallest; 141 | nodes.enqueue(nextNeighbor, candidate); 142 | } 143 | } 144 | } 145 | } 146 | 147 | return path.concat(smallest).reverse(); 148 | } 149 | } 150 | ``` 151 | 152 | ```js 153 | let g = new WeightedGraph(); 154 | 155 | g.addVertex('A'); 156 | g.addVertex('B'); 157 | g.addVertex('C'); 158 | g.addVertex('D'); 159 | g.addVertex('E'); 160 | g.addVertex('F'); 161 | 162 | g.addEdge('A', 'B', 4); 163 | g.addEdge('A', 'C', 2); 164 | g.addEdge('B', 'E', 3); 165 | g.addEdge('C', 'D', 2); 166 | g.addEdge('C', 'F', 4); 167 | g.addEdge('D', 'E', 3); 168 | g.addEdge('D', 'F', 1); 169 | g.addEdge('E', 'F', 1); 170 | 171 | g.Dijkstra('A', 'E'); // ['A', 'C', 'D', 'F', 'E'] 172 | ``` 173 | -------------------------------------------------------------------------------- /docs/JavaScript/01/multiplePointers.md: -------------------------------------------------------------------------------- 1 | # Multiple Pointers Pattern 2 | 3 | Creating **pointers** or values that correspond to an index or position and move towards the beginning, end or middle based on a certain condition. 4 | 5 | **Very** efficient for solving problems with minimal Space as well. 6 | 7 | --- 8 | 9 | ## `sumZero(arr)` 10 | 11 | > Write a function called `sumZero` which accepts a **sorted** array of integers. The function should find the **first** pair where the sum is 0. Return an array that includes both values that sum to zero or undefined if a pair does not exist. 12 | 13 | ```js 14 | sumZero([-3, -2, -1, 0, 1, 2, 3]); // [-3, 3] 15 | sumZero([-2, 0, 1, 3]); // undefined 16 | sumZero([1, 2, 3]); // undefined 17 | ``` 18 | 19 | - Solution 1 (naive) 20 | 21 | - Time: $O(n^2)$ 22 | - Space: $O(1)$ 23 | 24 | ```js 25 | const sumZero = arr => { 26 | for (let i = 0; i < arr.length; i++) 27 | for (let j = i + 1; j < arr.length; j++) 28 | if (arr[i] + arr[j] === 0) 29 | return [arr[i], arr[j]]; 30 | }; 31 | ``` 32 | 33 | - Solution 2 (refactor): 34 | 35 | - Time: $O(n)$ 36 | - Space: $O(1)$ 37 | 38 | ```js 39 | const sumZero = arr => { 40 | let left = 0; 41 | let right = arr.length - 1; 42 | 43 | while (left < right) { 44 | let sum = arr[left] + arr[right]; 45 | 46 | if (sum === 0) return [arr[left], arr[right]]; 47 | else if (sum < 0) left++; 48 | else right--; 49 | } 50 | }; 51 | ``` 52 | 53 | ## `countUniqueValues(arr)` 54 | 55 | > Implement a function called `countUniqueValues`, which accepts a sorted array, and counts the unique values in the array. There can be negative numbers in the array, but it will always be sorted. 56 | 57 | ```js 58 | countUniqueValues([1, 1, 1, 1, 1, 2]); // 2 59 | countUniqueValues([-2, -1, -1, 0, 1]); // 4 60 | countUniqueValues([]); // 0 61 | ``` 62 | 63 | - Solution: 64 | 65 | - Time: $O(n)$ 66 | - Space: $O(1)$ 67 | 68 | ```js 69 | const countUniqueValues = arr => { 70 | if (arr.length === 0) return 0; 71 | 72 | let i = 0; 73 | for (let j = 1; j < arr.length; j++) 74 | if (arr[i] !== arr[j]) 75 | arr[++i] = arr[j]; 76 | 77 | return i + 1; 78 | }; 79 | ``` 80 | 81 | ## `averagePair(arr, val)` 82 | 83 | > Write a function called `averagePair`. Given a sorted array of integers and a target average, determine if there is a pair of values in the array where the average of the pair equals the target average. There may be more than one pair that matches the average target. 84 | 85 | ```js 86 | averagePair([1, 2, 3], 2.5); // true 87 | averagePair([1, 3, 3, 5, 6, 7, 10, 12, 19], 8); // true 88 | averagePair([-1, 0, 3, 4, 5, 6], 4.1); // false 89 | averagePair([], 4); // false 90 | ``` 91 | 92 | - Solution: 93 | 94 | - Time: $O(n)$ 95 | - Space: $O(1)$ 96 | 97 | ```js 98 | const averagePair = (arr, num) => { 99 | let left = 0; 100 | let right = arr.length - 1; 101 | 102 | while (left < right) { 103 | let average = (arr[left] + arr[right]) / 2; 104 | 105 | if (average === num) return true; 106 | else if (average < num) left++; 107 | else right--; 108 | } 109 | 110 | return false; 111 | }; 112 | ``` 113 | 114 | ## `isSubsequence(str1, str2)` 115 | 116 | > Write a function called `isSubsequence` which takes in two strings and checks whether the characters in the first string form a subsequence of the characters in the second string. In other words, the function should check whether the characters in the first string appear somewhere in the second string, **without their order changing**. 117 | 118 | ```js 119 | isSubsequence('hello', 'hello world'); // true 120 | isSubsequence('sing', 'sting'); // true 121 | isSubsequence('abc', 'acb'); // false (order matters) 122 | ``` 123 | 124 | - Solution 1 (iterative): 125 | 126 | - Time: $O(n + m)$ 127 | - Space: $O(1)$ 128 | 129 | ```js 130 | const isSubsequence = (str1, str2) => { 131 | if (!str1) return true; 132 | 133 | let i = 0; 134 | for (let j = 0; j < str2.length; j++) { 135 | if (i == str1.length - 1) return true; 136 | if (str1[i] === str2[j]) i++; 137 | } 138 | 139 | return false; 140 | }; 141 | ``` 142 | 143 | - Solution 2 (recursive but not $O(1)$ space): 144 | 145 | ```js 146 | const isSubsequence = (str1, str2) => { 147 | if (str1.length === 0) return true; 148 | if (str2.length === 0) return false; 149 | if (str2[0] === str1[0]) return isSubsequence(str1.slice(1), str2.slice(1)); 150 | return isSubsequence(str1, str2.slice(1)); 151 | }; 152 | ``` 153 | -------------------------------------------------------------------------------- /mkdocs.yml: -------------------------------------------------------------------------------- 1 | # Project information 2 | site_name: Computer Science Notes 3 | site_description: Computer Science Notes, Operating System, Machine Learning, Parallel and Concurrent Programming with C++ 4 | site_author: Jay Chen 5 | site_url: https://walkccc.github.io/CS 6 | 7 | # Repository 8 | repo_name: walkccc/CS 9 | repo_url: https://github.com/walkccc/CS 10 | 11 | # Copyright 12 | copyright: Built by Jay Chen © 2017 - 2020 13 | 14 | # Configuration 15 | theme: 16 | name: material 17 | language: en 18 | custom_dir: custom/ 19 | favicon: assets/favicon.png 20 | palette: 21 | primary: indigo 22 | accent: blue 23 | font: 24 | text: Roboto 25 | code: Roboto Mono 26 | icon: 27 | logo: material/console 28 | repo: fontawesome/brands/github 29 | features: 30 | - tabs 31 | 32 | # Customization 33 | extra: 34 | social: 35 | - icon: fontawesome/brands/github-square 36 | link: https://github.com/walkccc 37 | - icon: fontawesome/brands/linkedin 38 | link: https://www.linkedin.com/in/walkccc/ 39 | - icon: fontawesome/solid/code 40 | link: https://walkccc.github.io/LeetCode 41 | 42 | extra_css: 43 | - css/katex.css 44 | - css/custom.css 45 | - https://cdn.jsdelivr.net/npm/katex@0.12.0/dist/katex.min.css 46 | 47 | extra_javascript: 48 | - js/katex.js 49 | - https://cdn.jsdelivr.net/npm/katex@0.12.0/dist/katex.min.js 50 | - https://cdn.jsdelivr.net/npm/katex@0.12.0/dist/contrib/auto-render.min.js 51 | 52 | # Extensions 53 | markdown_extensions: 54 | - pymdownx.highlight: 55 | linenums: true 56 | linenums_style: pymdownx.table 57 | - pymdownx.superfences 58 | - pymdownx.tabbed 59 | - admonition 60 | - toc: 61 | permalink: true 62 | 63 | nav: 64 | - Preface: index.md 65 | - Parallel and Concurrent Programming: 66 | - Preface: PCP/index.md 67 | - Part 1: 68 | - 2. Threads and Processes: PCP/1-2.md 69 | - 3. Mutual Exclusion: PCP/1-3.md 70 | - 4. Locks: PCP/1-4.md 71 | - 5. Liveness: PCP/1-5.md 72 | - Part 2: 73 | - 1. Synchronization: PCP/2-1.md 74 | - 2. Barriers: PCP/2-2.md 75 | - 3. Asynchronous: PCP/2-3.md 76 | - OS: 77 | - Preface: OS/index.md 78 | - PART ONE | OVERVIEW: 79 | - Chapter 1 Introduction: OS/Chap01.md 80 | - Chapter 2 Operating-System Structures: OS/Chap02.md 81 | - PART TWO | PROCESS MANAGEMENT: 82 | - Chapter 3 Processes: OS/Chap03.md 83 | - Chapter 4 Threads: OS/Chap04.md 84 | - Chapter 5 Process Synchronization: OS/Chap05.md 85 | - Chapter 6 CPU Scheduling: OS/Chap06.md 86 | - Chapter 7 Deadlocks: OS/Chap07.md 87 | - PART THREE | MEMORY MANAGEMENT: 88 | - Chapter 8 Main Memory: OS/Chap08.md 89 | - Chapter 9 Virtual Memory: OS/Chap09.md 90 | - Previous Exams: 91 | - Midterm: OS/midterm.md 92 | - Final: OS/final.md 93 | - ML: 94 | - Preface: ML/index.md 95 | - Lec 1 - Regression - Case Study: ML/01.md 96 | - Lec 2 - Where does the error com from?: ML/02.md 97 | - Lec 3 - Gradient Descent: ML/03.md 98 | - Lec 4 - Classification: ML/04.md 99 | - Lec 5 - Logistic Regression: ML/05.md 100 | - Lec 6 - Deep Learning: ML/06.md 101 | - Lec 7 - Backpropagation: ML/07.md 102 | - 'Lec 8 - "Hello world" of Deep Learning': ML/08.md 103 | - Lec 9 - Tips for Deep Learning: ML/09.md 104 | - JavaScript DSA: 105 | - Preface: JavaScript/index.md 106 | - 01 - Problem Solving Patterns: 107 | - Frequency Counters: JavaScript/01/frequencyCounters.md 108 | - Multiple Pointers: JavaScript/01/multiplePointers.md 109 | - Sliding Window: JavaScript/01/slidingWindow.md 110 | - Divide and Conquer: JavaScript/01/divideAndConquer.md 111 | - 02 - Recursion Problem Set: 112 | - Recursion Problem Set (easy): JavaScript/02/recursionEasy.md 113 | - Recursion Problem Set (hard): JavaScript/02/recursionHard.md 114 | - 03 - Sorting Algorithms: 115 | - Bubble Sort: JavaScript/03/bubbleSort.md 116 | - Selection Sort: JavaScript/03/selectionSort.md 117 | - Insertion Sort: JavaScript/03/insertionSort.md 118 | - Merge Sort: JavaScript/03/mergeSort.md 119 | - Quick Sort: JavaScript/03/quickSort.md 120 | - 04 - Data Structures: 121 | - Singly Linked List: JavaScript/04/SinglyLinkedList.md 122 | - Doubly Linked List: JavaScript/04/DoublyLinkedList.md 123 | - Binary Search Tree: JavaScript/04/BinarySearchTree.md 124 | - Max Binary Heap: JavaScript/04/MaxBinaryHeap.md 125 | - Max Priority Queue: JavaScript/04/MaxPriorityQueue.md 126 | - 05 - Graphs: 127 | - Graph: JavaScript/05/Graph.md 128 | - Dijkstra Algorithm: JavaScript/05/Dijkstra.md 129 | -------------------------------------------------------------------------------- /docs/PCP/1-2.md: -------------------------------------------------------------------------------- 1 | # 2. Threads and Processes 2 | 3 | ## Thread versus process 4 | 5 | ```cpp 6 | // Threads that waste CPU cycles 7 | #include 8 | #include 9 | #include 10 | 11 | // a simple function that wastes CPU cycles "forever" 12 | void cpu_waster() { 13 | printf("CPU Waster Process ID: %d\n", getpid()); 14 | printf("CPU Waster Thread ID %d\n", std::this_thread::get_id()); 15 | while (true) continue; 16 | } 17 | 18 | int main() { 19 | printf("Main Process ID: %d\n", getpid()); 20 | printf("Main Thread ID: %d\n", std::this_thread::get_id()); 21 | std::thread thread1(cpu_waster); 22 | std::thread thread2(cpu_waster); 23 | 24 | while (true) { // keep the main thread alive "forever" 25 | std::this_thread::sleep_for(std::chrono::seconds(1)); 26 | } 27 | } 28 | ``` 29 | 30 | ## Execution scheduling 31 | 32 | ```cpp 33 | // Two threads chopping vegetables 34 | #include 35 | #include 36 | 37 | bool chopping = true; 38 | 39 | void vegetable_chopper(const char* name) { 40 | unsigned int vegetable_count = 0; 41 | while (chopping) { 42 | ++vegetable_count; 43 | } 44 | printf("%s chopped %u vegetables.\n", name, vegetable_count); 45 | } 46 | 47 | int main() { 48 | std::thread olivia(vegetable_chopper, "Olivia"); 49 | std::thread barron(vegetable_chopper, "Barron"); 50 | 51 | printf("Barron and Olivia are chopping vegetables...\n"); 52 | std::this_thread::sleep_for(std::chrono::seconds(1)); 53 | chopping = false; 54 | barron.join(); 55 | olivia.join(); 56 | } 57 | ``` 58 | 59 | ## Thread life cycle 60 | 61 | === "begin" 62 | 63 | ```cpp 64 | // Two threads cooking soup 65 | #include 66 | #include 67 | 68 | void chef_olivia() { 69 | printf("Olivia started & waiting for sausage to thaw...\n"); 70 | std::this_thread::sleep_for(std::chrono::seconds(3)); 71 | printf("Olivia is done cutting sausage.\n"); 72 | } 73 | 74 | int main() { 75 | printf("Barron requests Olivia's help.\n"); 76 | std::thread olivia(chef_olivia); 77 | 78 | printf("Barron continues cooking soup.\n"); 79 | std::this_thread::sleep_for(std::chrono::seconds(1)); 80 | 81 | printf("Barron patiently waits for Olivia to finish and join...\n"); 82 | olivia.join(); 83 | 84 | printf("Barron and Olivia are both done!\n"); 85 | } 86 | ``` 87 | 88 | === "end" 89 | 90 | ```cpp hl_lines="14 18 22" 91 | // Two threads cooking soup 92 | #include 93 | #include 94 | 95 | void chef_olivia() { 96 | printf("Olivia started & waiting for sausage to thaw...\n"); 97 | std::this_thread::sleep_for(std::chrono::seconds(3)); 98 | printf("Olivia is done cutting sausage.\n"); 99 | } 100 | 101 | int main() { 102 | printf("Barron requests Olivia's help.\n"); 103 | std::thread olivia(chef_olivia); 104 | printf(" Olivia is joinable? %s\n", olivia.joinable() ? "true" : "false"); 105 | 106 | printf("Barron continues cooking soup.\n"); 107 | std::this_thread::sleep_for(std::chrono::seconds(1)); 108 | printf(" Olivia is joinable? %s\n", olivia.joinable() ? "true" : "false"); 109 | 110 | printf("Barron patiently waits for Olivia to finish and join...\n"); 111 | olivia.join(); 112 | printf(" Olivia is joinable? %s\n", olivia.joinable() ? "true" : "false"); 113 | 114 | printf("Barron and Olivia are both done!\n"); 115 | } 116 | ``` 117 | 118 | ## Detached thread 119 | 120 | === "begin" 121 | 122 | ```cpp 123 | // Barron finishes cooking while Olivia cleans 124 | #include 125 | #include 126 | 127 | void kitchen_cleaner() { 128 | while (true) { 129 | printf("Olivia cleaned the kitchen.\n"); 130 | std::this_thread::sleep_for(std::chrono::seconds(1)); 131 | } 132 | } 133 | 134 | int main() { 135 | std::thread olivia(kitchen_cleaner); 136 | for (int i = 0; i < 3; ++i) { 137 | printf("Barron is cooking...\n"); 138 | std::this_thread::sleep_for(std::chrono::milliseconds(600)); 139 | } 140 | printf("Barron is done!\n"); 141 | olivia.join(); 142 | } 143 | ``` 144 | 145 | === "end" 146 | 147 | ```cpp hl_lines="14" 148 | // Barron finishes cooking while Olivia cleans 149 | #include 150 | #include 151 | 152 | void kitchen_cleaner() { 153 | while (true) { 154 | printf("Olivia cleaned the kitchen.\n"); 155 | std::this_thread::sleep_for(std::chrono::seconds(1)); 156 | } 157 | } 158 | 159 | int main() { 160 | std::thread olivia(kitchen_cleaner); 161 | olivia.detach(); 162 | for (int i = 0; i < 3; ++i) { 163 | printf("Barron is cooking...\n"); 164 | std::this_thread::sleep_for(std::chrono::milliseconds(600)); 165 | } 166 | printf("Barron is done!\n"); 167 | } 168 | ``` 169 | -------------------------------------------------------------------------------- /docs/OS/Chap02.md: -------------------------------------------------------------------------------- 1 | # Chapter 2 Operating-System Structures 2 | 3 | Objectives: 4 | 5 | - To describe the services an operating system provides to users, processes, and other systems. 6 | - To discuss the various ways of structuring an operating system. 7 | - To explain how operating systems are installed and customized and how they boot. 8 | 9 | ## 2.1 Operating-System Services 10 | 11 | ![normal](../assets/os/2.1.png) 12 | 13 | - User interface (UI) 14 | - command-line interface (CLI) 15 | - batch interface 16 | - graphical user interface 17 | 18 | - Program execution. OS load a program into memory $\to$ run that program $\to$ end execution 19 | - normally 20 | - abnormally (error) 21 | 22 | - I/O operations. A running program may require I/O: 23 | - file 24 | - I/O device: recording to a CD or DVD ... 25 | 26 | - File-system manipulation. 27 | - read/write files 28 | - create/delete them by name 29 | - search 30 | - list file (ls) 31 | 32 | - Communications. 33 | - shared memory 34 | - message passing: packets of information in predefined formats are moved between processes by the operating system 35 | 36 | - Error detection 37 | - Resource allocation 38 | - Accounting. users can be billed 39 | - Protection and security 40 | 41 | ## 2.2 User and Operating-System Interface 42 | 43 | ### 2.2.1 Command Interpreters 44 | 45 | On systems with multiple command interpreters to choose from, the interpreters are known as **shells**. 46 | 47 | Two approaches: 48 | 49 | 1. the command interpreter itself contains the code to execute the command. 50 | - fast but the interpreter tends to be big! $\to$ painful in revision! 51 | - e.g. `cd`, `ls`, `del` 52 | 2. the command interpreter merely uses the command to identify a file to be loaded into memory and executed $\to$ search exec files 53 | - parameter passing 54 | - being slow 55 | - inconsistent interpretation of parameters 56 | - e.g. `rm` 57 | 58 | ### 2.2.2 Graphical User Interfaces 59 | 60 | - desktop 61 | - icons 62 | - folder 63 | - mouse 64 | - gestures on the touchscreen 65 | 66 | ### 2.2.3 Choice of Interface 67 | 68 | - shell scripts 69 | - e.g. `UNIX` and `Linux`. 70 | 71 | ## 2.3 Systems Calls 72 | 73 | ![small](../assets/os/2.5.png) 74 | 75 | System calls provide an interface to the services made available by an operating system. 76 | 77 | e.g. writing a simple program to read data from one file and copy them to another file causes a lot of system calls! 78 | 79 | - C/C++ 80 | 81 | Each read and write must return status information regarding various possible error conditions. 82 | 83 | - application programming interface (API): it specifies a set of functions 84 | - Windows API 85 | - POSIX API 86 | - UNIX 87 | - Linux 88 | - macOS 89 | - Java API 90 | 91 | **libc**: UNIX and Linux for programs written in C 92 | 93 | Why prefer API rather than invoking actual system calls? 94 | 95 | - protability (expected to run on any system) 96 | - actual system calls can be more difficult to learn 97 | 98 | The relationship between an **API**, the **system-call interface**, and the **OS** 99 | 100 | ![normal](../assets/os/2.6.png) 101 | 102 | The caller need know nothing about how the system call is implemented or what it does during execution. Rather, the caller need only obey the API and understand what the operating system will do as a result of the execution of that system call. 103 | 104 | *Make explicit to implicit*. 105 | 106 | Three general methods are used to pass parameters to the operating system. 107 | 108 | - through registers (Linux and Solaris) 109 | - block, 110 | - table, 111 | - memory, 112 | - and the address of the 113 | 114 | - placed or pushed onto the stack $\to$ popped off the stack by the OS 115 | 116 | ![normal](../assets/os/2.7.png) 117 | 118 | ## 2.4 Types of System Calls 119 | 120 | ### 2.4.1 Process Control 121 | 122 | A running program halts either 123 | 124 | - normally: `end()` 125 | - abnormally: `abort()` 126 | 127 | error $\to$ dump (written to disk, may be examined by a debugger) 128 | 129 | More severe errors can be indicated by a higher-level error parameter. 130 | 131 | e.g. Standard C Library 132 | 133 | ![normal](../assets/os/libc.png) 134 | 135 | ### 2.4.2 File Management 136 | 137 | ### 2.4.3 Device Management 138 | 139 | ### 2.4.4 Information Maintenance 140 | 141 | Many systems provide system calls to `dump()` memory. This provision is useful for debugging. A program trace lists each system call as it is executed. Even microprocessors provide a CPU mode known as single step, in which a trap is executed by the CPU after every instruction. The trap is usually caught by a debugger. 142 | 143 | ### 2.4.5 Communication 144 | 145 | ### 2.4.6 Protection 146 | 147 | ## 2.5 System Programs 148 | 149 | ## 2.6 Operating-System Design and Implementation 150 | 151 | ### 2.6.1 Design Goals 152 | 153 | ### 2.6.2 Mechanisms and Policies 154 | 155 | ### 2.6.3 Implementation -------------------------------------------------------------------------------- /docs/JavaScript/01/frequencyCounters.md: -------------------------------------------------------------------------------- 1 | # Frequency Counters Pattern 2 | 3 | This pattern uses objects or sets to collect values/frequencies of values. 4 | 5 | This can often avoid the need for nested loops or $O(n^2)$ operations with arrays/strings. 6 | 7 | --- 8 | 9 | ## `same(arr1, arr2)` 10 | 11 | > Write a function called `same`, which accepts two arrays. The function should return true if every value in the array has it's corresponding value squared in the second array. The frequency of values must be the same. 12 | 13 | ```js 14 | same([1, 2, 3], [4, 1, 9]); // true 15 | same([1, 2, 3], [1, 9]); // false 16 | same([1, 2, 1], [4, 4, 1]); // false (must be same frequency) 17 | ``` 18 | 19 | - Solution 1 (naive) 20 | 21 | - Time: $O(n^2)$ 22 | - Space: $O(1)$ 23 | 24 | ```js 25 | const same = (arr1, arr2) => { 26 | if (arr1.length !== arr2.length) return false; 27 | 28 | for (let i = 0; i < arr1.length; i++) { 29 | let correctIndex = arr2.indexOf(arr1[i] ** 2); 30 | if (correctIndex === -1) return false; 31 | arr2.splice(correctIndex, 1); 32 | } 33 | 34 | return true; 35 | }; 36 | ``` 37 | 38 | - Solution 2 (refactor) 39 | 40 | - Time: $O(n)$ 41 | - Space: $O(n)$ 42 | 43 | ```js 44 | const same = (arr1, arr2) => { 45 | if (arr1.length !== arr2.length) return false; 46 | 47 | const freqCount1 = {}; 48 | const freqCount2 = {}; 49 | 50 | for (let val of arr1) freqCount1[val] = (freqCount1[val] || 0) + 1; 51 | for (let val of arr2) freqCount2[val] = (freqCount2[val] || 0) + 1; 52 | for (let key in freqCount1) { 53 | if (!(key ** 2 in freqCount2)) return false; 54 | if (freqCount1[key] !== freqCount2[key ** 2]) return false; 55 | } 56 | 57 | return true; 58 | }; 59 | ``` 60 | 61 | ## `validAnagram(arr1, arr2)` 62 | 63 | > Given two strings, write a function called `validAnagram` to determine if the second string is an anagram of the first. An anagram is a word, phrase, or name formed by rearranging the letters of another, such as _cinema_, formed from _iceman_. 64 | 65 | ```js 66 | validAnagram('', ''); // true 67 | validAnagram('aaz', 'zza'); // false 68 | validAnagram('anagram', 'nagaram'); // true 69 | ``` 70 | 71 | - Solution 72 | 73 | - Time: $O(n)$ 74 | - Space: $O(n)$ 75 | 76 | ```js 77 | const validAnagram = (arr1, arr2) => { 78 | if (arr1.length !== arr2.length) return false; 79 | 80 | const freqCount1 = {}; 81 | 82 | for (let val of arr1) freqCount1[val] = (freqCount1[val] || 0) + 1; 83 | for (let val of arr2) { 84 | if (!freqCount1[val]) return false; 85 | freqCount1[val]--; 86 | } 87 | 88 | return true; 89 | }; 90 | ``` 91 | 92 | ## `sameFrequency(num1, num2)` 93 | 94 | > Write a function called `sameFrequency`. Given two positive integers, find out if the two numbers have the same frequency of digits. 95 | 96 | ```js 97 | sameFrequency(182, 281); // true 98 | sameFrequency(3589578, 5879385); // true 99 | sameFrequency(34, 14); // false 100 | sameFrequency(22, 222); // false 101 | ``` 102 | 103 | - Solution 104 | 105 | - Time: $O(n)$ 106 | 107 | ```js 108 | const sameFrequency = (num1, num2) => { 109 | strNum1 = num1.toString(); 110 | strNum2 = num2.toString(); 111 | 112 | if (strNum1.length !== strNum2.length) return false; 113 | 114 | const freqCount1 = {}; 115 | 116 | for (let val of strNum1) freqCount1[val] = (freqCount1[val] || 0) + 1; 117 | for (let val of strNum2) { 118 | if (!freqCount1[val]) return false; 119 | freqCount1[val]--; 120 | } 121 | 122 | return true; 123 | }; 124 | ``` 125 | 126 | ## `areThereDuplicates(...args)` 127 | 128 | > Implement a function called, `areThereDuplicates` which accepts a variable number of arguments, and checks whether there are any duplicates among the arguments passed in. You can solve this using the frequency counter pattern OR the multiple pointers pattern. 129 | 130 | ```js 131 | areThereDuplicates(1, 2, 3); // false 132 | areThereDuplicates(1, 2, 2); // true 133 | areThereDuplicates('a', 'b', 'c', 'a'); // true 134 | ``` 135 | 136 | - Solution 1 (frequency counter) 137 | 138 | - Time: $O(n)$ 139 | - Space: $O(n)$ 140 | 141 | ```js 142 | const areThereDuplicates = (...args) => { 143 | let lookup = {}; 144 | 145 | for (let val of args) { 146 | if (lookup[val]) return true; 147 | lookup[val] = (lookup[val] || 0) + 1; 148 | } 149 | 150 | return false; 151 | }; 152 | ``` 153 | 154 | - Solution 2 (multiple pointers) 155 | 156 | - Time: $O(n\log n)$ 157 | - Space: $O(1)$ 158 | 159 | ```js 160 | const areThereDuplicates = (...args) => { 161 | args.sort(); 162 | 163 | let i = 0; 164 | 165 | for (let j = 1; j < args.length; i++, j++) 166 | if (args[i] === args[j]) 167 | return true; 168 | 169 | return false; 170 | }; 171 | ``` 172 | 173 | - Solution 3 174 | 175 | ```js 176 | const areThereDuplicates = () => { 177 | return new Set(arguments).size !== arguments.length; 178 | }; 179 | ``` 180 | -------------------------------------------------------------------------------- /docs/ML/01.md: -------------------------------------------------------------------------------- 1 | # Lecture 1: Regression - Case Study 2 | 3 | ## Regression: Output a scalar 4 | 5 | Regression 的例子舉例有： 6 | 7 | - 道瓊斯指數預測 8 | - 自動駕駛中的方向盤角度預測 9 | - 推薦系統中購買可能性的預測 10 | 11 | ## Example Application 12 | 13 | - Estimating the Combat Power (CP) of a pokemon after evolution 14 | 15 | 一隻寶可夢可由 5 個參數表示： 16 | 17 | $$x = (x_{cp}, x_s, x_{hp}, x_w, x_h).$$ 18 | 19 | ![](https://i.imgur.com/StzhhXI.png) 20 | 21 | 我們希望能計算出，給定寶可夢的一些 features $x$，能過算出他進化後的 CP 值 $y$ 22 | 23 | ## Machine Learning Steps 24 | 25 | Machine Learning 主要有三個步驟 26 | 27 | ### Step 1: Model 28 | 29 | 我們假設預測 $y$ 值的 function 為 $y = b + w \cdot x_{cp}$，許多不同的 $w$ 和 $b$ 可構成不同的 $f_1, f_2, \dots$，例如： 30 | 31 | $$ 32 | \begin{aligned} 33 | f_1: y & = 10.0 + 9.0 \cdot x_{cp} \\\\ 34 | f_2: y & = 9.8 + 9.2 \cdot x_{cp} \\\\ 35 | f_3: y & = -0.8 - 1.2 \cdot x_{cp} \\\\ 36 | & \vdots 37 | \end{aligned} 38 | $$ 39 | 40 | 如此一來，我們就有了 linear model: 41 | 42 | $$y = b + \sum w_ib_i$$ 43 | 44 | 其中， 45 | 46 | - $x_i: x_{cp}, x_s, x_{hp}, x_w, x_h$：寶可夢的 features 47 | - $w_i$：不同 features，給它不同的 weight 48 | - $b$：bias 49 | 50 | ### Step 2: Goodness of Function 51 | 52 | 有了一堆 functions 後，我們必需要知道，這些 functions 有多好？因此我們需要一個評比的標準，我們可以用 Loss function $L$，它的 input 為 $f$，output 為 how bad it is？ 53 | 54 | loss function 可定義如下： 55 | 56 | $$ 57 | \begin{aligned} 58 | L(f) & = \sum_{n = 1}^{10} (\hat y^n - f(x_{cp}^n))^2 \\\\ 59 | L(w, b) & = \sum_{n = 1}^{10} (\hat y^n - (b + w \cdot x_{cp}^n))^2 60 | \end{aligned} 61 | $$ 62 | 63 | ### Step 3: Best Function 64 | 65 | 最後，我們要選出 best function，即：選擇 loss 最小的 function（參數）： 66 | 67 | $$ 68 | \begin{aligned} 69 | f ^ { * } & = \arg \min _ { f } L ( f ) \\\\ 70 | w ^ { * } , b ^ { * } & = \arg \min _ { w , b } L ( w , b ) \\\\ 71 | & = \arg \min _ { w , b } \sum _ { n = 1 } ^ { 10 } \left( \hat { y } ^ { n } - \left( b + w \cdot x _ { c p } ^ { n } \right) \right) ^ { 2 } 72 | \end{aligned} 73 | $$ 74 | 75 | 求解最佳參數的方法可以是 Gradient Descent。 76 | 77 | ## Gradient Descent 78 | 79 | ![](https://i.imgur.com/mkxAZGT.png) 80 | 81 | Gradient Descent 的步驟如下： 82 | 83 | 1. （隨機）選擇參數初始值（e.g. $w^0$），上標表示時間點 84 | 2. 計算 loss function $L(w)$ 在 $w = w^0$ 上時的 gradient，即：$\left. \frac { d L } { d w } \right| _ { w = w ^ { 0 } }$ 85 | 86 | 3. 向負梯度方向迭代更新 87 | 88 | - $\frac{dL}{dw} \Big|_{w = w^0} < 0 \to$ Increase $w$ 89 | - $\frac{dL}{dw} \Big|_{w = w^0} > 0 \to$ Decrease $w$ 90 | 91 | \begin{align} 92 | w ^ { 1 } & \leftarrow w ^ { 0 } - \eta \left. \frac { d L } { d w } \right| _ { w = w ^ { 0 } } \\\\ 93 | w ^ { 2 } & \leftarrow w ^ { 1 } - \eta \left. \frac { d L } { d w } \right| _ { w = w ^ { 1 } } 94 | \end{align} 95 | 96 | 其中，$\eta$ 為 learning rate，用來控制一次走多遠、學習速度 97 | 98 | 重複這 3 個步驟，直到 $w$ 在 local minima 收斂為止。 99 | 100 | 順帶一提，只有一個參數 $w$ 時，$\nabla L$ 就是 $\frac{\partial L}{\partial w}$ 為 $L$ 的 gradient，所以才叫 Gradient Descent。 101 | 102 | 若有兩個參數 $w, b$ 時，$\nabla L = \left[ \begin{array} { l } { \frac { \partial L } { \partial w } } \\\\ { \frac { \partial L } { \partial b } } \end{array} \right]$，本質上概念還是相同的 103 | 104 | Gradient Descent 可能使參數停在損失函數的局部最小值、微分為 $0$ 的點，或者微分為極小值的點。Linear regression 中不必擔心局部最小值的問題，因為 loss function 是 convex 的。 105 | 106 | 在得到 best function 之後，我們真正在意的是它**在 testing data 上的表現**。選擇不同的 model，會得到不同的 best function，它們在testing data 上有不同表現。 107 | 108 | 複雜模型的 model space 涵蓋了簡單模型的 model space，因此在 training data 上的錯誤率更小，但並不意味著在 testing data 上錯誤率也會小，因為 Model 太複雜有很高機率會出現 overfitting。 109 | 110 | ## What are the hidden factors? 111 | 112 | 如果我們收集更多寶可夢進化前後的 CP 值會發現，進化後的 CP 值不只依賴於進化前的 CP 值，可能還有其它的隱藏因素（例如：寶可夢所屬的物種）。同時考慮進化前 CP 值 $x_{cp}$ 和物種 $x_s$，我們可以回到 Step 1: Model 重新選擇新的 model： 113 | 114 | $$ 115 | \begin{aligned} 116 | \text{if } & x_s = \text{pidgey}: & y = b_1 + w_1 \cdot x_{cp} \\\\ 117 | \text{if } & x_s = \text{weedle}: & y = b_2 + w_2 \cdot x_{cp} \\\\ 118 | \text{if } & x_s = \text{caterpie}: & y = b_3 + w_3 \cdot x_{cp} \\\\ 119 | \text{if } & x_s = \text{eevee}: & y = b_4 + w_4 \cdot x_{cp} 120 | \end{aligned} 121 | $$ 122 | 123 | 這仍是一個線性模型，因為它可以寫作： 124 | 125 | $$ 126 | \begin{aligned} 127 | y = & b_1 \cdot \boldsymbol{\delta(x_s = \text{pidgey}) } & + & w_1 \cdot \boldsymbol{\delta(x_s = \text{pidgey}) \cdot x_{cp}} + \\\\ 128 | & b_2 \cdot \boldsymbol{\delta(x_s = \text{weedle}) } & + & w_2 \cdot \boldsymbol{\delta(x_s = \text{weedle}) \cdot x_{cp}} + \\\\ 129 | & b_3 \cdot \boldsymbol{\delta(x_s = \text{caterpie})} & + & w_3 \cdot \boldsymbol{\delta(x_s = \text{caterpie}) \cdot x_{cp}} + \\\\ 130 | & b_4 \cdot \boldsymbol{\delta(x_s = \text{eevee}) } & + & w_4 \cdot \boldsymbol{\delta(x_s = \text{eevee}) \cdot x_{cp}} 131 | \end{aligned} 132 | $$ 133 | 134 | 上式中的粗體項都是 linear model $y = b + \Sigma w_i \cdot x_i$ 中的 feature $x_i$。 135 | 136 | 這個模型在 testing data 上有更好的表現，如果同時考慮寶可夢的其它屬性，選一個很複雜的模型，結果可能會導致 overfitting。 137 | 138 | 對線性模型來說，我們希望選出的 best function 能平滑一點，也就是權重係數小一些，因為這樣的話，在 testing data 受 noise 影響時，預測值所受的影響會更小。 139 | 140 | 所以在 Step 2: Goodness of Function 設計 loss function 時，我們可以加一個正則項 $\lambda\Sigma w_i^2$： 141 | 142 | $$L = \sum _ { n } \left( \hat { y } ^ { n } - \left( b + \sum w _ { i } x _ { i }^n \right) \right) ^ { 2 } + \lambda \sum \left( w _ { i } \right) ^ { 2 }$$ 143 | 144 | 越大的 $\lambda$，training error 可能會越大，但沒關係，我們希望函數平滑，但也不能太平滑，於是我們調整 $\lambda$，選擇使 testing error 最小的 $\lambda$。 145 | -------------------------------------------------------------------------------- /docs/ML/04.md: -------------------------------------------------------------------------------- 1 | # Lecture 4: Classification: Probabilistic Generative Model 2 | 3 | 分類問題常被用在各種領域，例如： 4 | 5 | - 金融交易成功與否 6 | - 醫學診斷分類 7 | - 手寫辨識 8 | - 人臉辨試 9 | 10 | --- 11 | 12 | 課堂上的例子：根據寶可夢各屬性（features）去預測其類別（class）。 13 | 14 | Training data：寶可夢的各屬性（features）及其類別（class）。 15 | 16 | ## How to do Classification? 17 | 18 | ### Classification as Regression? 19 | 20 | ![](https://i.imgur.com/IoR3woc.png) 21 | 22 | 像上圖右側，即使分類正確了，還是會因為遠大於 $1$，造成 loss 過大，因此不適合使用 regression。 23 | 24 | 此外，當類別超過 $2$ 種時，也會產生許多問題，因為這樣表示，各類別之間是有大小遠近關係的，但在分類問題這樣是不合理的。 25 | 26 | ## Ideal Alternatives 27 | 28 | - Function (Model): 29 | 30 | $$ 31 | f(x) = 32 | \begin{cases} 33 | g(x) > 0 & \text{output: class 1}, \\\\ 34 | \text{else} & \text{output: class 2} 35 | \end{cases} 36 | $$ 37 | 38 | - Loss function: 39 | 40 | $$L(f) = \sum_n \delta(f(x^n) \ne \hat y^n)$$ 41 | 42 | - Find the best function 43 | 44 | ### Bayes' Theorem 45 | 46 | 假設有兩個類別，我們想要估算來自 training data 的機率。 47 | 48 | ![](https://i.imgur.com/Jv8rfsI.png) 49 | 50 | Given an $x$, which class does it belong to: 51 | 52 | $$ 53 | P \left( C _ { 1 } | x \right) = \frac { P ( x | C _ { 1 } ) P \left( C _ { 1 } \right) } { P ( x | C _ { 1 } ) P \left( C _ { 1 } \right) + P ( x | C _ { 2 } ) P \left( C _ { 2 } \right) } 54 | $$ 55 | 56 | Generative Model: 57 | 58 | $$ 59 | P ( x ) = P ( x | C _ { 1 } ) P \left( C _ { 1 } \right) + P ( x | C _ { 2 } ) P \left( C _ { 2 } \right) 60 | $$ 61 | 62 | ### Prior 63 | 64 | 舉例來說：我們挑選出 79 隻水系（$C_1$）和 61 隻一般系（$C_2$）的寶可夢，那麼 65 | 66 | \begin{align} 67 | P(C_1) & = 79 / (79 + 61) = 0.56 \\\\ 68 | P(C_2) & = 61 / (79 + 61) = 0.44 69 | \end{align} 70 | 71 | ### Probability from Class 72 | 73 | 現在我們想要知道的事，從水系（$C_1$）挑出一隻「海龜」的機率是多少？ 74 | 75 | ### Probability from Class - Feature 76 | 77 | 因為寶可夢的 features 太多，為了方便，我們只考慮兩項，Defense 和 SP Defense，並假設這 79 隻水系寶可夢都是由 Gaussian distribution sample 出來的。 78 | 79 | ### Maximum Likelihood 80 | 81 | ![](https://i.imgur.com/Wf9hd5Z.png) 82 | 83 | 因為每個點 $x$ 是獨立的，所以我們可以透過以下連乘來得出 $L(\mu, \Sigma)$ 84 | 85 | Likelihood of Gaussian with mean $\mu$ and covariance matrix $\Sigma$ = the probability of the Gaussian samples $x^1, x^2, x^3, \dots, x^{79}$: 86 | 87 | $$L(\mu, \Sigma) = f_{\mu, \Sigma}(x^1) f_{\mu, \Sigma}(x^2) f_{\mu, \Sigma}(x^3) \dots f_{\mu, \Sigma}(x^{79})$$ 88 | 89 | 所以我們的目的是，找出一個 $(\mu^\*, \Sigma^\*)$ 能讓上式的 likelihood 最大： 90 | 91 | $$\mu^\*, \Sigma^\* = \arg \max_{\mu, \Sigma} L(\mu, \Sigma)$$ 92 | 93 | 透過微分可得以下結果： 94 | 95 | $$ 96 | \begin{aligned} 97 | \mu^\* & = \frac 1 {79} \sum_{n = 1}^{79} x^n \\\\ 98 | \Sigma^\* & = \frac 1 {79} \sum_{n = 1}^{79} (x^n - \mu^\*)(x^n - \mu^\*)^T 99 | \end{aligned} 100 | $$ 101 | 102 | 所以我們可計算出： 103 | 104 | ![](https://i.imgur.com/HOWXjFd.png) 105 | 106 | 現在可以開始分類了， 107 | 108 | $$P(C_1 \mid x) = \frac{P(x \mid C_1) P(C_1)}{P(x \mid C_1) P(C_1) + P(x \mid C_2) P(C_2)}$$ 109 | 110 | 其中， 111 | 112 | - $P(C_1) = 79 / (79 + 61) = 0.56$ 113 | - $P(C_2) = 61 / (79 + 61) = 0.44$ 114 | 115 | $$P(x \mid C_1) = f_{\mu^1, \Sigma^1}(x) = \frac 1 {(2\pi)^{D / 2}} \frac 1 {|\Sigma^1|^{1 / 2}} exp \Big(- \frac 1 2 (x - \mu^1)^T (\Sigma^1)^{-1} (x - \mu^1) \Big)$$ 116 | 117 | $$P(x \mid C_2) = f_{\mu^2, \Sigma^2}(x) = \frac 1 {(2\pi)^{D / 2}} \frac 1 {|\Sigma^2|^{1 / 2}} exp \Big(- \frac 1 2 (x - \mu^2)^T (\Sigma^2)^{-1} (x - \mu^2) \Big)$$ 118 | 119 | 若 $P(C_1 \mid x) > 0.5$，$x$ 就屬於 class 1（水系） 120 | 121 | 但結果卻不怎麼樣，因為參數太多了，可能有 overfit 疑慮。 122 | 123 | ### Modifying Model 124 | 125 | 原本對兩個分佈（水系、一般系）都有對應的 covariance matrix （$\Sigma_1, \Sigma_2$），但這樣參數量太大，可能會 overfit，所以我們降低參數量去避免 overfit，只考慮一個 convariance matrix（$\Sigma$），修改 loss function 如下： 126 | 127 | $$L(\mu^1, \mu^2, \Sigma) = f_{\mu^1, \Sigma}(x^1) f_{\mu^2, \Sigma}(x^2) \cdots f_{\mu^1, \Sigma}(x^{79}) \times f_{\mu^1, \Sigma}(x^{80}) f_{\mu^1, \Sigma}(x^{81}) \cdots f_{\mu^1, \Sigma}(x^{140})$$ 128 | 129 | $\mu^1$ 和 $\mu^2$ 維持不變，$\Sigma = \frac{79}{140}\Sigma^1 + \frac{61}{140}\Sigma^2$ 130 | 131 | ### Three Steps 132 | 133 | - Function Set (Model): 134 | 135 | 輸入一個 $x$：$P(C_1 \mid x) = \frac{P(x \mid C_1) P(C_1)}{P(x \mid C_1) P(C_1) + P(x \mid C_2) P(C_2)}$ 136 | 137 | $$ 138 | \begin{cases} 139 | P(C_1 \mid x) > 0.5 & \text{output: class 1}, \\\\ 140 | \text{else} & \text{output: class 2} 141 | \end{cases} 142 | $$ 143 | 144 | - Goodness of a function 145 | 146 | The mean $\mu$ and covariance $\Sigma$ that maximizing the likelihood (the probability of generating data) 147 | 148 | - Find the test function: easy 149 | 150 | ### Posterior Probability 151 | 152 | $$ 153 | \begin{aligned} 154 | P(C_1 \mid x) 155 | & = \frac{P(x \mid C_1) P(C_1)}{P(x \mid C_1) P(C_1) + P(x \mid C_2) P(C_2)} \\\\ 156 | & = \frac 1 {1 + \frac{P(x \mid C_2) P(C_2)}{P(x \mid C_1) P(C_1)}} \\\\ 157 | & = \frac 1 {1 + e^{-z}} = \sigma(z) 158 | \end{aligned} 159 | $$ 160 | 161 | 其中，$z = \ln \frac{P(x \mid C_1) P(C_1)}{P(x \mid C_2) P(C_2)}$ (sigmoid) 162 | 163 | 經過很長一段推導（這裡就不再贅述，可直接看老師[投影片](http://speech.ee.ntu.edu.tw/~tlkagk/courses/ML_2016/Lecture/Classification%20(v3).pdf)），可得出： 164 | 165 | $$ 166 | \begin{aligned} 167 | P(C_1 \mid x) & = \sigma(z) \\\\ 168 | & = \sigma(w \cdot x + b) 169 | \end{aligned} 170 | $$ 171 | 172 | 其中， 173 | 174 | - $w^T = (\mu^1 - \mu^2)^T \Sigma^{-1}$ 175 | - $b = -\frac 1 2 (\mu^1)^T \Sigma^{-1} \mu^1 + \frac 1 2 (\mu^2)^T \Sigma^{-1} \mu^2 + \ln \frac{N_1}{N_2}$ 176 | -------------------------------------------------------------------------------- /docs/PCP/2-2.md: -------------------------------------------------------------------------------- 1 | # Barriers 2 | 3 | ## Race condition 4 | 5 | ```cpp 6 | // Deciding how many bags of chips to buy for the party 7 | #include 8 | #include 9 | 10 | unsigned int bags_of_chips = 1; // start with one on the list 11 | std::mutex pencil; 12 | 13 | void cpu_work(unsigned long workUnits) { 14 | unsigned long x = 0; 15 | for (unsigned long i; i < workUnits * 1000000; ++i) { 16 | ++x; 17 | } 18 | } 19 | 20 | void barron_shopper() { 21 | cpu_work(1); // do a bit of work first 22 | std::scoped_lock lock(pencil); 23 | bags_of_chips *= 2; 24 | printf("Barron DOUBLED the bags of chips.\n"); 25 | } 26 | 27 | void olivia_shopper() { 28 | cpu_work(1); // do a bit of work first 29 | std::scoped_lock lock(pencil); 30 | bags_of_chips += 3; 31 | printf("Olivia ADDED 3 bags of chips.\n"); 32 | } 33 | 34 | int main() { 35 | std::thread shoppers[10]; 36 | for (int i = 0; i < 10; i += 2) { 37 | shoppers[i] = std::thread(barron_shopper); 38 | shoppers[i + 1] = std::thread(olivia_shopper); 39 | } 40 | for (auto& s : shoppers) { 41 | s.join(); 42 | } 43 | printf("We need to buy %u bags_of_chips.\n", bags_of_chips); 44 | } 45 | ``` 46 | 47 | ## Barrier 48 | 49 | === "begin" 50 | 51 | ```cpp 52 | // Deciding how many bags of chips to buy for the party 53 | #include 54 | #include 55 | 56 | unsigned int bags_of_chips = 1; // start with one on the list 57 | std::mutex pencil; 58 | 59 | void cpu_work(unsigned long workUnits) { 60 | unsigned long x = 0; 61 | for (unsigned long i; i < workUnits * 1000000; ++i) { 62 | ++x; 63 | } 64 | } 65 | 66 | void barron_shopper() { 67 | cpu_work(1); // do a bit of work first 68 | std::scoped_lock lock(pencil); 69 | bags_of_chips *= 2; 70 | printf("Barron DOUBLED the bags of chips.\n"); 71 | } 72 | 73 | void olivia_shopper() { 74 | cpu_work(1); // do a bit of work first 75 | std::scoped_lock lock(pencil); 76 | bags_of_chips += 3; 77 | printf("Olivia ADDED 3 bags of chips.\n"); 78 | } 79 | 80 | int main() { 81 | std::thread shoppers[10]; 82 | for (int i = 0; i < 10; i += 2) { 83 | shoppers[i] = std::thread(barron_shopper); 84 | shoppers[i + 1] = std::thread(olivia_shopper); 85 | } 86 | for (auto& s : shoppers) { 87 | s.join(); 88 | } 89 | printf("We need to buy %u bags_of_chips.\n", bags_of_chips); 90 | } 91 | ``` 92 | 93 | === "barrier" 94 | 95 | ```cpp hl_lines="2 8 19 27 28 29 30 32" 96 | // Deciding how many bags of chips to buy for the party 97 | #include 98 | #include 99 | #include 100 | 101 | unsigned int bags_of_chips = 1; // start with one on the list 102 | std::mutex pencil; 103 | boost::barrier fist_bump(10); 104 | 105 | void cpu_work(unsigned long workUnits) { 106 | unsigned long x = 0; 107 | for (unsigned long i; i < workUnits * 1000000; ++i) { 108 | ++x; 109 | } 110 | } 111 | 112 | void barron_shopper() { 113 | cpu_work(1); // do a bit of work first 114 | fist_bump.wait(); 115 | std::scoped_lock lock(pencil); 116 | bags_of_chips *= 2; 117 | printf("Barron DOUBLED the bags of chips.\n"); 118 | } 119 | 120 | void olivia_shopper() { 121 | cpu_work(1); // do a bit of work first 122 | { 123 | std::scoped_lock lock(pencil); 124 | bags_of_chips += 3; 125 | } 126 | printf("Olivia ADDED 3 bags of chips.\n"); 127 | fist_bump.wait(); 128 | } 129 | 130 | int main() { 131 | std::thread shoppers[10]; 132 | for (int i = 0; i < 10; i += 2) { 133 | shoppers[i] = std::thread(barron_shopper); 134 | shoppers[i + 1] = std::thread(olivia_shopper); 135 | } 136 | for (auto& s : shoppers) { 137 | s.join(); 138 | } 139 | printf("We need to buy %u bags_of_chips.\n", bags_of_chips); 140 | } 141 | ``` 142 | 143 | === "latch" 144 | 145 | ```cpp hl_lines="2 8 32" 146 | // Deciding how many bags of chips to buy for the party 147 | #include 148 | #include 149 | #include 150 | 151 | unsigned int bags_of_chips = 1; // start with one on the list 152 | std::mutex pencil; 153 | boost::latch fist_bump(10); 154 | 155 | void cpu_work(unsigned long workUnits) { 156 | unsigned long x = 0; 157 | for (unsigned long i; i < workUnits * 1000000; ++i) { 158 | ++x; 159 | } 160 | } 161 | 162 | void barron_shopper() { 163 | cpu_work(1); // do a bit of work first 164 | fist_bump.wait(); 165 | std::scoped_lock lock(pencil); 166 | bags_of_chips *= 2; 167 | printf("Barron DOUBLED the bags of chips.\n"); 168 | } 169 | 170 | void olivia_shopper() { 171 | cpu_work(1); // do a bit of work first 172 | { 173 | std::scoped_lock lock(pencil); 174 | bags_of_chips += 3; 175 | } 176 | printf("Olivia ADDED 3 bags of chips.\n"); 177 | fist_bump.count_down(); 178 | } 179 | 180 | int main() { 181 | std::thread shoppers[10]; 182 | for (int i = 0; i < 10; i += 2) { 183 | shoppers[i] = std::thread(barron_shopper); 184 | shoppers[i + 1] = std::thread(olivia_shopper); 185 | } 186 | for (auto& s : shoppers) { 187 | s.join(); 188 | } 189 | printf("We need to buy %u bags_of_chips.\n", bags_of_chips); 190 | } 191 | ``` 192 | -------------------------------------------------------------------------------- /docs/OS/final.md: -------------------------------------------------------------------------------- 1 | # Previous Operating System Midterms at NTUCSIE 2 | 3 | ## [Spring 2011](https://www.ptt.cc/bbs/NTU-Exam/M.1340817215.A.742.html) 4 | 5 | The exam is 150 minutes long. The total score is 112pts. Please read the questions carefully. 6 | 7 | 1. Terminologies (21pts). 8 | 9 | - Condition Variable (Hint: Monitor) 10 | 11 | - Stable Storage 12 | 13 | - 50-Percent Rule for the First-Fit Algorithm (Hint: Dynamic Partitioning) 14 | 15 | - Virtual Memory 16 | 17 | - Lazy Swapper 18 | 19 | - Network Information Service (NIS or Yellow Pages from SUN) 20 | 21 | - Scan (or Elevator) Algorithm (Hint: Disk Scheduling) 22 | 23 | 24 | 2. Please answer the following question on process synchronization: (18pts) 25 | 26 | - Synchronization hardware might make programming easier for process synchronization. Please explain why interrupt disabling works for uniprocessor environments to avoid the race condition. (6pts) 27 | 28 | - There are three requirements for a solution to critical section problem. Please tell me which requirement might not be satisfied for solutions based on semaphores with priority-waiting queues. You must provide your arguments. (6pts) 29 | 30 | - When the Two-Phase Locking protocol (2PL) is adopted for process synchronization, and there is only exclusive lock, can we have a deadlock? You must provide your arguments. (6pts) 31 | 32 | 33 | 3. Given the following snapshot of the system: Please determine whether there exists a safe sequence. (5pts) 34 | 35 | 4. With binding at the compiling time, is it appropriate to have Paging? With binding at the loading time, is it appropriate to have Paging? You must provide explanation. (6pts) 36 | 37 | 5. Given a computer system with 64-bit virtual address and 8 bytes per page entry, let the physical address be of 48 bits, and the system is byte-addressable. Assume that every page is of 4KB. Please answer the following questions: (20pts). 38 | 39 | - What is the maximum number of frames? (5pts) 40 | 41 | - Suppose that we have multi-level paging. How many levels do we have in multi-level paging? (5pts) 42 | 43 | - Suppose that TLB is adopted for paging, and multi-level page tables are all in the main memory. Let the memory access time and TLB access time be 100ns and 10ns, respectively. What the TLB hit ratio should be so that the effective memory access time would be no more than 120ns? (5pts) 44 | 45 | - Continued with the previous question, What is the maximum number of entries for an inverted page table? (5pts) 46 | 47 | 6. Please answer the following questions for demand paging: (20pts) 48 | 49 | - What is the maximum number of pages needed for the following instruction: Add (R1)+, -(R2), +(R3) (4pts) 50 | 51 | - The effective access time is defined as (1 - p) * ma + p * pft, where p, ma, and pft are the probability of a page fault, the memory access time for paging, and the page fault time, respectively. Please give me two approaches to reduce the pft. Please give me two approaches to reduce p. (16pts) 52 | 53 | 7. Given the following reference string, which reference causes a page fault under the Second Chance (Clock) Algorithm. Please also show us which page is replaced when a page replacement occurs? Suppose that we have 3 available frames with 0, 1, and 3 as shown in the following graph, all reference bits are 0, and the selection pointer starts at Page 0. (10pts) 54 | 55 | $$1 \, 0 \, 3 \, 1 \, 4 \, 3 \, 2 \, 1 \, 3 \, 0 \, 2 \, 1 \, 5$$ 56 | 57 | 8. Modern operating system often only recognize few file types. Please give two advantages why file types are usually supported by applications, instead of the operating system. (8pts) 58 | 59 | 9. Give me one reason when we need scheduling in I/O subsystem, beside performance improvement. (4pts) 60 | 61 | ## [Spring 2012](https://www.ptt.cc/bbs/NTU-Exam/M.1418545643.A.933.html) 62 | 63 | 1. Terminologies. (24pts) 64 | 65 | - **Deadlock** 66 | 67 | A set of process is in a deadlock state when every process in the set is waiting for an event that can be caused by only another process in the set. 68 | 69 | - **Segmentation** 70 | 71 | Segmentation is a memory management scheme that supports the user view of memory. 72 | 73 | - **Working Set** (Hint: Thrashing and Frame Allocation) 74 | 75 | The working set is an approximation of a program's locality. 76 | 77 | - **Optimal Algorithm for Page Replacement (OPT)** 78 | 79 | Replace the page that will not be used for the longest period of time. 80 | 81 | - **Hard Link** (Hint: Acyclic-Graph Directory) 82 | 83 | A hard link of a file name let its directory entry point to the i-node that describes the file's contents. 84 | 85 | - **Network-Attached Storage (NAS)** 86 | 87 | Clients access NAS via a remote procedure call interface such as NFS (for UNIX) or CIFS (for Windows) where NAS has a file system running on it. 88 | 89 | - **Raw Disk** 90 | 91 | Ans: A partition as a large sequential array of logical blocks. 92 | 93 | 2. Please provide one advantage and one disadvantage of a deadlock avoidance algorithm (such as the Banker's Algorithm), compared to a deadlock 94 | prevention algorithm (such as the one without "Hold-and-Wait"). (6pts) 95 | 96 | - Advantage: Better resource utilization and/or better system throughput/performance. 97 | - Dsadvantage: Processes must claim maximum usages of each resource. 98 | 99 | ## [Spring 2013](https://www.ptt.cc/bbs/NTU-Exam/M.1403667958.A.FA7.html) 100 | 101 | ## [Spring 2014](https://www.ptt.cc/bbs/NTU-Exam/M.1437210714.A.B8A.html) 102 | 103 | ## [Spring 2015](https://www.ptt.cc/bbs/NTU-Exam/M.1472670507.A.427.html) -------------------------------------------------------------------------------- /docs/OS/Chap09.md: -------------------------------------------------------------------------------- 1 | # Chapter 9 Virtual Memory 2 | 3 | ## 9.1 Background 4 | 5 | > For instance, the routines on U.S. government computers that balance the budget have not been used in many years. 6 | 7 | !!! note "Virtual memory" 8 | It separates the logical memory perceived by users from physical memory. 9 | 10 | !!! note "Virtual address space" 11 | Logical (virtual) view of how a process is stored in memory. 12 | 13 | !!! note "Sparse address spaces" 14 | Virtual address spaces that include holes. 15 | 16 | ## 9.2 Demand Paging 17 | 18 | !!! note "Demand paging" 19 | A strategy to load pages only as they are needed. 20 | 21 | !!! note "Lazy swapper" 22 | A lazy swapper never swaps a page into memory unless that page will be needed. 23 | 24 | - swapper: entire processes 25 | - **pager**: individual pages of a process 26 | 27 | ### 9.2.1 Basic Concepts 28 | 29 | If the valid bit is set to "invalid", there are 2 possibilities: 30 | 31 | - the page is invalid 32 | - the page is valid but is currently on the dist. 33 | 34 | ![normal](../assets/os/9.6.png) 35 | 36 | !!! note "Pure demand paging" 37 | Never bring a page into memory until it is required. 38 | 39 | !!! note "Secondary memory" 40 | This memory holds those pages that are not present in main memory. 41 | 42 | ### 9.2.2 Performance of Demand Paging 43 | 44 | For most computer systems, the memory-access time, denoted $ma$, ranges from 10 to 200 nanoseconds. 45 | 46 | Let $p$ be the probability of a page fault ($0 \le p \le 1$). We would expect $p$ to be close to zero. 47 | 48 | $$\text{effective access time} = (1 - p) \times ma + p \times \text{page fault time}.$$ 49 | 50 | ## 9.3 Copy-on-Write 51 | 52 | Many operating systems provide a pool of free pages for such requests. 53 | 54 | !!! note "Zero-fill-on-demand" 55 | Zero-fill-on-demand pages have been zeroed-out before being allocated, thus erasing the previous contents. 56 | 57 | !!! note "`vfork()` (virtual memory fork)" 58 | It operates differently from `fork()` with copy-on-write. With `vfork()`, the parent process is suspended, and the child process uses the address space of the parent. 59 | 60 | ## 9.4 Page Replacement 61 | 62 | ### 9.4.1 Basic Page Replacement 63 | 64 | 1. Find the location of the desired page on the disk. 65 | 2. Find a free frame: 66 | 67 | a. If there is a free frame, use it. 68 | 69 | b. If there is no free frame, use a page-replacement algorithm to select a **victim frame**. 70 | 71 | c. Write the victim frame to the disk; change the page and frame tables accordingly. 72 | 73 | 3. Read the desired page into the newly freed frame; change the page and frame tables. 74 | 4. Continue the user process from where the page fault occurred. 75 | 76 | ![normal](../assets/os/9.10.png) 77 | 78 | !!! note "Modify bit (dirty bit)" 79 | The modify bit for a page is set by the hardware whenever any byte in the page is written into. 80 | 81 | When we select a page for replacement. 82 | 83 | - If the modify bit is set, we must write the page to the disk. 84 | - If the modify bit is not set, we need not write the page to the disk. 85 | 86 | We must solve two major problems to implement demand paging: 87 | 88 | 1. frame-allocation algorithm: decide how many frames to allocate to each process. 89 | 2. page-replacement algorithm: when page replacement is required. 90 | 91 | !!! note "Reference string" 92 | The string of memory references. 93 | 94 | ### 9.4.2 FIFO Page Replacement 95 | 96 | ![normal](../assets/os/9.12.png) 97 | 98 | !!! note "Belady's anomaly" 99 | For some page-replacement algorithms, the page-fault rate may increase as the number of allocated frames increases. 100 | 101 | ### 9.4.3 Optimal Page Replacement 102 | 103 | !!! note "Optimal page-replacement algorithm" 104 | Replace the page that will not be used for the longest period of time, which will never suffer from Belady's anomaly. 105 | 106 | ![normal](../assets/os/9.14.png) 107 | 108 | ### 9.4.4 LRU Page Replacement 109 | 110 | The LRU policy is often used as a page-replacement algorithm and is considered to be good. The major problem is ***how*** to implement LRU replacement. 111 | 112 | - Counter 113 | - cons: overflow 114 | - Stack 115 | 116 | ### 9.4.5 LRU-Approximation Page Replacement 117 | 118 | #### 9.4.5.1 Additional-Reference-Bits Algorithm 119 | 120 | The operating system shifts the reference bit for each page into the high-order bit of its 8-bit byte, shifting the other bits right by 1 bit and discarding the low-order bit. 121 | 122 | These 8-bit shift registers contain the history of page use for the last eight time periods. 123 | 124 | e.g. A page with a history register value of 11000100 has been used more recently than one with a value of 01110111. 125 | 126 | #### 9.4.5.2 Second-Chance Algorithm 127 | 128 | ![normal](../assets/os/9.17.png) 129 | 130 | In the worst case, when all bits are set, the pointer cycles through the whole queue, giving each page a second chance. It clears all the reference bits before selecting the next page for replacement. ($\to$ FIFO) 131 | 132 | #### 9.4.5.3 Enhanced Second-Chance Algorithm 133 | 134 | ordered pair = (reference bit, modify bit) 135 | 136 | 1. (0, 0) neither recently used nor modified—best page to replace 137 | 2. (0, 1) not recently used but modified—not quite as good, because the page will need to be written out before replacement 138 | 3. (1, 0) recently used but clean—probably will be used again soon 139 | 4. (1, 1) recently used and modified—probably will be used again soon, and the page will be need to be written out to disk before it can be replaced 140 | 141 | ### 9.4.6 Counting-Based Page Replacement 142 | 143 | - least frequently used (LFU) 144 | - most frequently used (MFU) 145 | 146 | ### 9.4.7 Page-Buffering Algorithms 147 | 148 | Systems keep a pool of free frames 149 | 150 | 1. The desired page is read into a free frame from the pool before the victime is written out. 151 | 2. When the victim is later written out, its frame is added to the free-frame pool. 152 | 153 | There are two variations: 154 | 155 | 1. To maintain a list of modified pages. 156 | - Whenever the paging device is idle, a modified page is written to the dist and its modify bit is reset. 157 | 2. To keep a pool of free frames but to remember which page was in each frame. 158 | - No I/O is needed in this case $\to$ "Swapped-in" time is saved! 159 | 160 | 161 | ### 9.4.8 Applications and Page Replacement -------------------------------------------------------------------------------- /docs/ML/03.md: -------------------------------------------------------------------------------- 1 | # Lecture 3: Gradient Descent 2 | 3 | ## Review: Gradient Descent 4 | 5 | ![](https://i.imgur.com/MGrouqh.png) 6 | 7 | 在 Gradient Descent 中，我們想解決的是下列問題： 8 | 9 | $$ 10 | \theta ^ { * } = \arg \min _ \theta L ( \theta ) 11 | $$ 12 | 13 | 其中， 14 | 15 | - $L$: loss function 16 | - $\theta$: parameters 17 | 18 | 假設 $\theta$ 有兩個參數 ${\theta_1, \theta_2}$，我們隨機選取 $\theta ^ { 0 } = \left[ \begin{array} { l } { \theta _ { 1 } ^ { 0 } } \\\\ { \theta _ { 2 } ^ { 0 } } \end{array} \right]$，所以 $\nabla L ( \theta ) = \left[ \begin{array} { l } { \partial L \left( \theta _ { 1 } \right) / \partial \theta _ { 1 } } \\\\ { \partial L \left( \theta _ { 2 } \right) / \partial \theta _ { 2 } } \end{array} \right]$ 19 | 20 | $$ 21 | \begin{aligned} 22 | \left[ \begin{array} { l } { \theta _ { 1 } ^ { 1 } } \\\\ { \theta _ { 2 } ^ { 1 } } \end{array} \right] = \left[ \begin{array} { c } { \theta _ { 1 } ^ { 0 } } \\\\ { \theta _ { 2 } ^ { 0 } } \end{array} \right] - \eta \left[ \begin{array} { l } { \partial L \left( \theta _ { 1 } ^ { 0 } \right) / \partial \theta _ { 1 } } \\\\ { \partial L \left( \theta _ { 2 } ^ { 0 } \right) / \partial \theta _ { 2 } } \end{array} \right] \to \theta ^ { 1 } = \theta ^ { 0 } - \eta \nabla L \left( \theta ^ { 0 } \right) \\\\ 23 | \left[ \begin{array} { l } { \theta _ { 1 } ^ { 2 } } \\\\ { \theta _ { 2 } ^ { 2 } } \end{array} \right] = \left[ \begin{array} { c } { \theta _ { 1 } ^ { 1 } } \\\\ { \theta _ { 2 } ^ { 1 } } \end{array} \right] - \eta \left[ \begin{array} { l } { \partial L \left( \theta _ { 1 } ^ { 1 } \right) / \partial \theta _ { 1 } } \\\\ { \partial L \left( \theta _ { 2 } ^ { 1 } \right) / \partial \theta _ { 2 } } \end{array} \right] \to \theta ^ { 2 } = \theta ^ { 1 } - \eta \nabla L \left( \theta ^ { 1 } \right) 24 | \end{aligned} 25 | $$ 26 | 27 | ## Tip 1: Tuning your learning rates 28 | 29 | 我們必需要小心的設我們的 learning rate $\eta$ 30 | 31 | $$ 32 | \theta ^i = \theta ^{i - 1} - \eta \nabla L \left( \theta ^{i - 1} \right) 33 | $$ 34 | 35 | ### Adaptive Learning Rates 36 | 37 | 因為一開始 learning rate 更新的很慢，所以我們可以一開始先用較大的 learning rate 來加速，之後再逐漸下降 learning rate 來避免無法達到 loss 的 minimum，例如：$\eta^t = \eta / \sqrt{t + 1}$。 38 | 39 | #### Adagrad 40 | 41 | Adagrad 是一個常見的適性方法：Divide the learning rate of each parameter by the ***root mean square of its previous derivatives*** 42 | 43 | - Vanilla Gradient Descent 44 | 45 | $$w^{t + 1} \leftarrow w^t - \eta^t g^t$$ 46 | 47 | 其中， 48 | 49 | - $\eta ^ { t } = \frac \eta { \sqrt { t + 1 } }$ 50 | - $g ^ { t } = \frac { \partial L \left( \theta ^ { t } \right) } { \partial w }$ 51 | 52 | - Adagrad 53 | 54 | $$ 55 | \begin{aligned} 56 | w^{t + 1} & \leftarrow w^t - \frac{\eta^t}{\sigma^t} g^t \\\\ 57 | \equiv w^{t + 1} & \leftarrow w^t - \frac{\frac \eta { \sqrt { t + 1 } }}{\sqrt { \frac { 1 } { t + 1 } \sum _ { i = 0 } ^ { t } \left( g ^i \right) ^ { 2 } }} g^t \\\\ 58 | \equiv w^{t + 1} & \leftarrow w^t - \frac{\eta}{\sqrt{\sum_{i = 0}^t (g^i)^2}} g^t \\\\ 59 | \end{aligned} 60 | $$ 61 | 62 | 其中， 63 | 64 | $\sigma^t$：***root mean square*** of the previous derivatives of parameter $w$ 65 | 66 | $g^t$ 代表 first derivative，分母則是利用歷史的 first derivative 去估計 second derivative。 67 | 68 | ## Tip 2: Stochastic Gradient Descent 69 | 70 | - Gradient Descent 71 | 72 | Loss is the summation over all training examples: 73 | 74 | $$L = \sum_n \Big(\hat y^n - \Big(b + \sum w_i x_i^n \Big) \Big)^2$$ 75 | 76 | 更新參數方式： 77 | 78 | $$\theta^i = \theta^{i - 1} - \eta \nabla L (\theta^{i - 1})$$ 79 | 80 | - Stochastic Gradient Descent 81 | 82 | Pick an example $x^n$ 83 | 84 | Loss for only one example: 85 | 86 | $$L = \Big(\hat y^n - \Big(b + \sum w_i x_i^n \Big) \Big)^2$$ 87 | 88 | 更新參數方式： 89 | 90 | $$\theta ^i = \theta^{i - 1} - \eta \nabla L^n (\theta^{i - 1})$$ 91 | 92 | Stochastic Gradient Descent 能夠加速訓練，一般 Gradient Descent 是在看到所有樣本（訓練數據）後，才計算出 $L$，再更新參數。而 Stochastic Gradient Descent 是每看到一次樣本 $n$ 就計算一次 $L^n$，並更新參數。 93 | 94 | ## Tip 3: Feature Scaling 95 | 96 | 例如我們的 model 為：$y = b + w_1x_1 + w_2x_2$ 97 | 98 | ![](https://i.imgur.com/l6b3osk.png) 99 | 100 | 將 features 從左圖 scaling 成右圖，能確保每一個 features 「貢獻」的數字，不會因為不同參數間的大小範圍而差太多。 101 | 102 | ![](https://i.imgur.com/XDNSGri.png) 103 | 104 | 若不同參數的大小範圍差太多，那 loss function 等高線方向（負梯度、參數更新方向）不會指向 loss minimum。Feature scaling 讓不同參數的取相同區間範圍的值，變成正圓，更容易新參數。 105 | 106 | 做法如下，對 $x^1, x^2, x^3, \dots, x^r, \dots, x^R$ 共 $R$ 筆資料，每一個維度 $i$ 做標準化： 107 | 108 | ![](https://i.imgur.com/oKTFaPu.png) 109 | 110 | ## Theory 111 | 112 | 接下來要說說，為什麼 Gradient Descent 是可行的呢？ 113 | 114 | 我們欲解的問題為： 115 | 116 | $$ 117 | \theta ^ { * } = \arg \min _ { \theta } L ( \theta ) \quad \text { by gradient descent } 118 | $$ 119 | 120 | 考慮以下陳述是否正確： 121 | 122 | > 每次更新參數，我們就得到 $\theta$ 使得 $L(\theta)$ 更小： 123 | > 124 | > $$L \left( \theta ^ { 0 } \right) > L \left( \theta ^ { 1 } \right) > L \left( \theta ^ { 2 } \right) > \cdots$$ 125 | 126 | ### Formal Derivation 127 | 128 | 假設 $\theta$ 有兩個變數 $\{\theta_1, \theta_2\}$，我們如何輕易的找出給定一個點，離他最近最小的值？ 129 | 130 | ![](https://i.imgur.com/tEyoPf4.png) 131 | 132 | ### Taylor Series 133 | 134 | #### Single-variable 135 | 136 | Let $h(x)$ be any function infinitely differentiable around $x = x_0$. 137 | 138 | $$ 139 | \begin{aligned} { h } ( { x } ) & = \sum _ { k = 0 } ^ { \infty } \frac { { h } ^ { ( k ) } \left( x _ { 0 } \right) } { k ! } \left( x - x _ { 0 } \right) ^ { k } \\\\ & = h \left( x _ { 0 } \right) + h ^ { \prime } \left( x _ { 0 } \right) \left( x - x _ { 0 } \right) + \frac { h ^ { \prime \prime } \left( x _ { 0 } \right) } { 2 ! } \left( x - x _ { 0 } \right) ^ { 2 } + \cdots \end{aligned} 140 | $$ 141 | 142 | 當 $x$ 接近 $x_0$ 時：$h(x) \approx h(x_0) + h'(x_0)(x - x_0)$ 143 | 144 | #### Multi-variable 145 | 146 | Let $h(x)$ be any function infinitely differentiable around $x = x_0$ and $y = y_0$. 147 | 148 | $$ 149 | h ( x , y ) = h \left( x _ { 0 } , y _ { 0 } \right) + \frac { \partial h \left( x _ { 0 } , y _ { 0 } \right) } { \partial x } \left( x - x _ { 0 } \right) + \frac { \partial h \left( x _ { 0 } , y _ { 0 } \right) } { \partial y } \left( y - y _ { 0 } \right) 150 | $$ 151 | 152 | 當 $x$ 接近 $x_0$ 和 $y_0$ 時：$h ( x , y ) \approx h \left( x _ { 0 } , y _ { 0 } \right) + \frac { \partial h \left( x _ { 0 } , y _ { 0 } \right) } { \partial x } \left( x - x _ { 0 } \right) + \frac { \partial h \left( x _ { 0 } , y _ { 0 } \right) } { \partial y } \left( y - y _ { 0 } \right)$ 153 | 154 | ### Back to Formal Derivation 155 | 156 | ![small](https://i.imgur.com/tyoqtM7.png) 157 | 158 | Based on Taylor Series: If the red circle is ***small enough*** (i.e., $\theta_1$ and $\theta_2$ is close to $a$ and $b$), in the red circle 159 | 160 | $$ 161 | \begin{aligned} 162 | { L } ( \theta ) & \approx { L } ( a , b ) + \frac { \partial { L } ( a , b ) } { \partial \theta _ { 1 } } \left( \theta _ { 1 } - a \right) + \frac { \partial { L } ( a , b ) } { \partial \theta _ { 2 } } \left( \theta _ { 2 } - b \right) \\\\ 163 | & \approx s + u \left( \theta _ { 1 } - a \right) + v \left( \theta _ { 2 } - b \right) 164 | \end{aligned} 165 | $$ 166 | 167 | 其中， 168 | 169 | - $s = L(a, b)$ 170 | - $u = \frac { \partial { L } ( a , b ) } { \partial \theta _ { 1 } } , v = \frac { \partial { L } ( a , b ) } { \partial \theta _ { 2 } }$ 171 | 172 | Goal: Find $\theta_1$ and $\theta_2$ in the *red circle* **minimizing** $L(\theta)$ 173 | 174 | 在 red circle 內代表的是一個圓的方程式： 175 | 176 | $$ 177 | \begin{aligned} 178 | \left( \theta _ { 1 } - a \right) ^ { 2 } + \left( \theta _ { 2 } - b \right) ^ { 2 } & \le d ^ { 2 } \\\\ 179 | (\Delta \theta_1)^2 + (\Delta \theta_2)^2 & \le d^2 180 | \end{aligned} 181 | $$ 182 | 183 | To minimize $L(\theta)$: 184 | 185 | $$ 186 | \begin{aligned} 187 | \left[ \begin{array} { l } { \Delta \theta _ { 1 } } \\\\ { \Delta \theta _ { 2 } } \end{array} \right] & = - \eta \left[ \begin{array} { l } { u } \\\\ { v } \end{array} \right] \\\\ 188 | \Rightarrow \left[ \begin{array} { l } { \theta _ { 1 } } \\\\ { \theta _ { 2 } } \end{array} \right] & = \left[ \begin{array} { l } { a } \\\\ { b } \end{array} \right] - \eta \left[ \begin{array} { l } { u } \\\\ { v } \end{array} \right] \\\\ 189 | & = \left[ \begin{array} { l } { a } \\\\ { b } \end{array} \right] - \eta \left[ \begin{array} { c } { \frac { \partial { L } ( a , b ) } { \partial \theta _ { 1 } } } \\\\ { \frac { \partial { L } ( a , b ) } { \partial \theta _ { 2 } } } \end{array} \right] 190 | \end{aligned} 191 | $$ -------------------------------------------------------------------------------- /docs/PCP/1-4.md: -------------------------------------------------------------------------------- 1 | # 4. Locks 2 | 3 | ## Recursive mutex 4 | 5 | === "begin" 6 | 7 | ```cpp 8 | // Two shoppers adding garlic and potatoes to a shared notepad 9 | #include 10 | #include 11 | 12 | unsigned int garlic_count = 0; 13 | unsigned int potato_count = 0; 14 | std::mutex pencil; 15 | 16 | void add_garlic() { 17 | pencil.lock(); 18 | ++garlic_count; 19 | pencil.unlock(); 20 | } 21 | 22 | void add_potato() { 23 | pencil.lock(); 24 | ++potato_count; 25 | pencil.unlock(); 26 | } 27 | 28 | void shopper() { 29 | for (int i = 0; i < 10000; ++i) { 30 | add_garlic(); 31 | add_potato(); 32 | } 33 | } 34 | 35 | int main() { 36 | std::thread barron(shopper); 37 | std::thread olivia(shopper); 38 | barron.join(); 39 | olivia.join(); 40 | printf("We should buy %u garlic.\n", garlic_count); 41 | printf("We should buy %u potatoes.\n", potato_count); 42 | } 43 | ``` 44 | 45 | === "end" 46 | 47 | ```cpp hl_lines="19" 48 | // Two shoppers adding garlic and potatoes to a shared notepad 49 | 50 | #include 51 | #include 52 | 53 | unsigned int garlic_count = 0; 54 | unsigned int potato_count = 0; 55 | std::recursive_mutex pencil; 56 | 57 | void add_garlic() { 58 | pencil.lock(); 59 | ++garlic_count; 60 | pencil.unlock(); 61 | } 62 | 63 | void add_potato() { 64 | pencil.lock(); 65 | ++potato_count; 66 | add_garlic(); 67 | pencil.unlock(); 68 | } 69 | 70 | void shopper() { 71 | for (int i = 0; i < 10000; ++i) { 72 | add_garlic(); 73 | add_potato(); 74 | } 75 | } 76 | 77 | int main() { 78 | std::thread barron(shopper); 79 | std::thread olivia(shopper); 80 | barron.join(); 81 | olivia.join(); 82 | printf("We should buy %u garlic.\n", garlic_count); 83 | printf("We should buy %u potatoes.\n", potato_count); 84 | } 85 | ``` 86 | 87 | ## Try lock 88 | 89 | === "begin" 90 | 91 | ```cpp 92 | // Two shoppers adding items to a shared notepad 93 | #include 94 | #include 95 | #include 96 | 97 | unsigned int items_on_notepad = 0; 98 | std::mutex pencil; 99 | 100 | void shopper(const char* name) { 101 | int items_to_add = 0; 102 | while (items_on_notepad <= 20) { 103 | if (items_to_add) { // add item(s) to shared items_on_notepad 104 | pencil.lock(); 105 | items_on_notepad += items_to_add; 106 | printf("%s added %u item(s) to notepad.\n", name, items_to_add); 107 | items_to_add = 0; 108 | std::this_thread::sleep_for( 109 | std::chrono::milliseconds(300)); // time spent writing 110 | pencil.unlock(); 111 | } else { // look for other things to buy 112 | std::this_thread::sleep_for( 113 | std::chrono::milliseconds(100)); // time spent searching 114 | ++items_to_add; 115 | printf("%s found something else to buy.\n", name); 116 | } 117 | } 118 | } 119 | 120 | int main() { 121 | auto start_time = std::chrono::steady_clock::now(); 122 | std::thread barron(shopper, "Barron"); 123 | std::thread olivia(shopper, "Olivia"); 124 | barron.join(); 125 | olivia.join(); 126 | auto elapsed_time = std::chrono::duration_cast( 127 | std::chrono::steady_clock::now() - start_time) 128 | .count(); 129 | printf("Elapsed Time: %.2f seconds\n", elapsed_time / 1000.0); 130 | } 131 | ``` 132 | 133 | === "end" 134 | 135 | ```cpp hl_lines="13" 136 | // Two shoppers adding items to a shared notepad 137 | #include 138 | #include 139 | #include 140 | 141 | unsigned int items_on_notepad = 0; 142 | std::mutex pencil; 143 | 144 | void shopper(const char* name) { 145 | int items_to_add = 0; 146 | while (items_on_notepad <= 20) { 147 | if (items_to_add && 148 | pencil.try_lock()) { // add item(s) to shared items_on_notepad 149 | items_on_notepad += items_to_add; 150 | printf("%s added %u item(s) to notepad.\n", name, items_to_add); 151 | items_to_add = 0; 152 | std::this_thread::sleep_for( 153 | std::chrono::milliseconds(300)); // time spent writing 154 | pencil.unlock(); 155 | } else { // look for other things to buy 156 | std::this_thread::sleep_for( 157 | std::chrono::milliseconds(100)); // time spent searching 158 | ++items_to_add; 159 | printf("%s found something else to buy.\n", name); 160 | } 161 | } 162 | } 163 | 164 | int main() { 165 | auto start_time = std::chrono::steady_clock::now(); 166 | std::thread barron(shopper, "Barron"); 167 | std::thread olivia(shopper, "Olivia"); 168 | barron.join(); 169 | olivia.join(); 170 | auto elapsed_time = std::chrono::duration_cast( 171 | std::chrono::steady_clock::now() - start_time) 172 | .count(); 173 | printf("Elapsed Time: %.2f seconds\n", elapsed_time / 1000.0); 174 | } 175 | ``` 176 | 177 | ## Shared mutext 178 | 179 | === "begin" 180 | 181 | ```cpp 182 | // Several users reading a calendar, but only a few users updating it 183 | #include 184 | #include 185 | #include 186 | #include 187 | 188 | char WEEKDAYS[7][10] = {"Sunday", "Monday", "Tuesday", "Wednesday", 189 | "Thursday", "Friday", "Saturday"}; 190 | int today = 0; 191 | std::mutex marker; 192 | 193 | void calendar_reader(const int id) { 194 | for (int i = 0; i < 7; ++i) { 195 | marker.lock(); 196 | printf("Reader-%d sees today is %s\n", id, WEEKDAYS[today]); 197 | std::this_thread::sleep_for(std::chrono::milliseconds(100)); 198 | marker.unlock(); 199 | } 200 | } 201 | 202 | void calendar_writer(const int id) { 203 | for (int i = 0; i < 7; ++i) { 204 | marker.lock(); 205 | today = (today + 1) % 7; 206 | printf("Writer-%d updated date to %s\n", id, WEEKDAYS[today]); 207 | std::this_thread::sleep_for(std::chrono::milliseconds(100)); 208 | marker.unlock(); 209 | } 210 | } 211 | 212 | int main() { 213 | // create ten reader threads ...but only two writer threads 214 | std::array readers; 215 | for (unsigned int i = 0; i < readers.size(); ++i) { 216 | readers[i] = std::thread(calendar_reader, i); 217 | } 218 | std::array writers; 219 | for (unsigned int i = 0; i < writers.size(); ++i) { 220 | writers[i] = std::thread(calendar_writer, i); 221 | } 222 | 223 | // wait for readers and writers to finish 224 | for (unsigned int i = 0; i < readers.size(); ++i) { 225 | readers[i].join(); 226 | } 227 | for (unsigned int i = 0; i < writers.size(); ++i) { 228 | writers[i].join(); 229 | } 230 | } 231 | ``` 232 | 233 | === "end" 234 | 235 | ```cpp hl_lines="5 11 15 18" 236 | // Several users reading a calendar, but only a few users updating it 237 | #include 238 | #include 239 | #include 240 | #include 241 | #include 242 | 243 | char WEEKDAYS[7][10] = {"Sunday", "Monday", "Tuesday", "Wednesday", 244 | "Thursday", "Friday", "Saturday"}; 245 | int today = 0; 246 | std::shared_mutex marker; 247 | 248 | void calendar_reader(const int id) { 249 | for (int i = 0; i < 7; ++i) { 250 | marker.lock_shared(); 251 | printf("Reader-%d sees today is %s\n", id, WEEKDAYS[today]); 252 | std::this_thread::sleep_for(std::chrono::milliseconds(100)); 253 | marker.unlock_shared(); 254 | } 255 | } 256 | 257 | void calendar_writer(const int id) { 258 | for (int i = 0; i < 7; ++i) { 259 | marker.lock(); 260 | today = (today + 1) % 7; 261 | printf("Writer-%d updated date to %s\n", id, WEEKDAYS[today]); 262 | std::this_thread::sleep_for(std::chrono::milliseconds(100)); 263 | marker.unlock(); 264 | } 265 | } 266 | 267 | int main() { 268 | // create ten reader threads ...but only two writer threads 269 | std::array readers; 270 | for (unsigned int i = 0; i < readers.size(); ++i) { 271 | readers[i] = std::thread(calendar_reader, i); 272 | } 273 | std::array writers; 274 | for (unsigned int i = 0; i < writers.size(); ++i) { 275 | writers[i] = std::thread(calendar_writer, i); 276 | } 277 | 278 | // wait for readers and writers to finish 279 | for (unsigned int i = 0; i < readers.size(); ++i) { 280 | readers[i].join(); 281 | } 282 | for (unsigned int i = 0; i < writers.size(); ++i) { 283 | writers[i].join(); 284 | } 285 | } 286 | ``` 287 | -------------------------------------------------------------------------------- /docs/ML/05.md: -------------------------------------------------------------------------------- 1 | # Lecture 5: Logistic Regression 2 | 3 | ## Three Steps 4 | 5 | ### Step 1: Function Set 6 | 7 | 我們想找的是：$P_{w, b} (C_1 \mid x)$ 8 | 9 | $$ 10 | f_{w, b} = 11 | \begin{cases} 12 | P_{w, b} (C_1 \mid x) \ge 0.5 & \text{output: } C_1, \\\\ 13 | \text{else} & \text{output: } C_2 14 | \end{cases} 15 | $$ 16 | 17 | $$ 18 | \begin{aligned} 19 | P_{w, b} (C_1 \mid x) 20 | & = \sigma(z) \\\\ 21 | & = \sigma(w \cdot x + b) 22 | \end{aligned} 23 | $$ 24 | 25 | 我們會有以下的 Function set（包含各種不同的 $w$ 和 $b$）: 26 | 27 | $$f_{w, b}(x) = P_{w, b} (C_1 \mid x)$$ 28 | 29 | ![](https://i.imgur.com/3vbJcUq.png) 30 | 31 | ### Step 2: Goodness of a Function 32 | 33 | 若 Training Data 為： 34 | 35 | $$ 36 | \begin{array} { l l l l } 37 | { x ^ { 1 } } & { x ^ { 2 } } & { x ^ { 3 } } & \cdots & { x ^ { N } } \\\\ 38 | { C _ { 1 } } & { C _ { 1 } } & { C _ { 2 } } & \cdots & { C _ { 1 } } 39 | \end{array} 40 | $$ 41 | 42 | 接下來一樣要決定一個 function 的好壞，假設 data 是從 $f_{w, b}(x) = P_{w, b} (C_1 \mid x)$ 產生。 43 | 44 | Given a set of $w$ and $b$, what is its probability of generating the data? 45 | 46 | $$ 47 | L ( w , b ) = f _ { w , b } \left( x ^ { 1 } \right) f _ { w , b } \left( x ^ { 2 } \right) \left( 1 - f _ { w , b } \left( x ^ { 3 } \right) \right) \cdots f _ { w , b } \left( x ^ { N } \right) 48 | $$ 49 | 50 | The most likely $w^\*$ and $b^\*$ is the one with the largest $L(w, b)$: 51 | 52 | $$ 53 | w ^ { * } , b ^ { * } = \arg \max _ { w , b } L ( w , b ) 54 | $$ 55 | 56 | 求上式等同於求： 57 | 58 | $$ 59 | \begin{aligned} 60 | w ^ { * } , b ^ { * } & = \arg \min _ { w , b } - \ln L ( w , b ) \\\\ 61 | & = \arg \min _ { w , b } - \ln f _ { w , b } \left( x ^ { 1 } \right) - \ln f _ { w , b } \left( x ^ { 2 } \right) - \ln \left( 1 - f _ { w , b } \left( x ^ { 3 } \right) \right) \cdots - \ln f _ { w , b } \left( x ^ { N } \right) \\\\ 62 | & = \arg \min _ { w , b } \sum _ { n } - \left[ \hat { y } ^ { n } \ln f _ { w , b } \left( x ^ { n } \right) + \left( 1 - \hat { y } ^ { n } \right) \ln \left( 1 - f _ { w , b } \left( x ^ { n } \right) \right) \right] 63 | \end{aligned} 64 | $$ 65 | 66 | 其中， 67 | 68 | - $\hat y^n$：$1$ for $C_1$, $0$ for $C_2$ 69 | 70 | $\Sigma_n$ 項等同於求以下兩個分佈 $p$、$q$ 的 cross entropy 71 | 72 | - Distribution $p$: 73 | 74 | $$ 75 | \begin{array} { l } { p ( x = 1 ) = \hat { y } ^ { n } } \\\\ { p ( x = 0 ) = 1 - \hat { y } ^ { n } } \end{array} 76 | $$ 77 | 78 | - Distribution $q$: 79 | 80 | $$ 81 | \begin{array} { l } {{ q } ( x = 1 ) = f \left( x ^ { n } \right) } \\\\ {{ q } ( x = 0 ) = 1 - f \left( x ^ { n } \right) } \end{array} 82 | $$ 83 | 84 | $$ 85 | H ( p , q ) = - \sum _ { x } p ( x ) \ln ( q ( x ) ) 86 | $$ 87 | 88 | 問題是：為什麼在 Logistic Regression 不用 rms 當 loss function 了？ 89 | 90 | 答：做微分後，某些項次會為 $0$，導致參數更新過慢。 91 | 92 | ### Step 3: Find the best function 93 | 94 | 決定完 loss function 後，我們要從一個 set 中，找出 best function，先對 $w_i$ 進行偏微分： 95 | 96 | $$ 97 | \frac { - \ln L ( w , b ) } { \partial w _ { i } } = \sum _ { n } - \left[ \hat { y } ^ { n } \frac { \ln f _ { w , b } \left( x ^ { n } \right) } { \partial w _ { i } } + \left( 1 - \hat { y } ^ { n } \right) \frac { \ln \left( 1 - f _ { w , b } \left( x ^ { n } \right) \right) } { \partial w _ { i } } \right] \tag{*} 98 | $$ 99 | 100 | 其中， 101 | 102 | - $f _ { w , b } ( x ) = \sigma ( z ) = 1 / (1 + e^{-z})$ 103 | - $z = w \cdot x + b = \sum _ { i } w _ { i } x _ { i } + b$ 104 | 105 | $$ 106 | \begin{aligned} 107 | \frac { \partial \ln f _ { w , b } ( x ) } { \partial w _ { i } } & = \frac { \partial \ln f _ { w , b } ( x ) } { \partial z } \frac { \partial z } { \partial w _ { i } } \\\\ 108 | & = \frac { \partial \ln \sigma ( z ) } { \partial z } \cdot x_i \\\\ 109 | & = \frac { 1 } { \sigma ( z ) } \frac { \partial \sigma ( z ) } { \partial z } \cdot x_i \\\\ 110 | & = \frac{1}{\sigma(z)} \sigma(z) (1 - \sigma(z)) \cdot x_i \\\\ 111 | & = (1 - \sigma(z)) \cdot x_i \\\\ 112 | & = \left( 1 - f _ { w , b } \left( x \right) \right) \cdot x_i 113 | \end{aligned} 114 | $$ 115 | 116 | $$ 117 | \begin{aligned} 118 | \frac { \partial \ln \left( 1 - f _ { w , b } ( x ) \right) } { \partial w _ { i } } & = \frac { \partial \ln \left( 1 - f _ { w , b } ( x ) \right) } { \partial z } \frac { \partial z } { \partial w _ { i } } \\\\ 119 | & = \frac { \partial \ln ( 1 - \sigma ( z ) ) } { \partial z } \cdot x_i \\\\ 120 | & = - \frac { 1 } { 1 - \sigma ( z ) } \frac { \partial \sigma ( z ) } { \partial z } \cdot x_i \\\\ 121 | & = - \frac{1}{1 - \sigma(z)} \sigma(z) (1 - \sigma(z)) \cdot x_i \\\\ 122 | & = -\sigma(z) \cdot x_i \\\\ 123 | & = -f _ { w , b } \left( x \right) \cdot x_i 124 | \end{aligned} 125 | $$ 126 | 127 | 透過上面計算出來的結果，我們可以對 $*$ 式進行代換： 128 | 129 | $$ 130 | \begin{aligned} 131 | \frac { - \ln L ( w , b ) } { \partial w _ { i } } & = \sum _ { n } - \left[ \hat { y } ^ { n } \frac { \ln f _ { w , b } \left( x ^ { n } \right) } { \partial w _ { i } } + \left( 1 - \hat { y } ^ { n } \right) \frac { \ln \left( 1 - f _ { w , b } \left( x ^ { n } \right) \right) } { \partial w _ { i } } \right] \\\\ 132 | & = \sum _ { n } - \left[ \hat { y } ^ { n } \left( 1 - f _ { w , b } \left( x ^ { n } \right) \right) x _ { i } ^ { n } - \left( 1 - \hat { y } ^ { n } \right) f _ { w , b } \left( x ^ { n } \right) x _ { i } ^ { n } \right] \\\\ 133 | & = \sum _ { n } - \left[ \hat { y } ^ { n } - \hat { y } ^ { n } f _ { w , b } \left( x ^ { n } \right) - f _ { w , b } \left( x ^ { n } \right) + \hat { y } ^ { n } f _ { W , b } \left( x ^ { n } \right) \right] x _ { i } ^ { n } \\\\ 134 | & = \sum _ { n } - \left( \hat { y } ^ { n } - f _ { w , b } \left( x ^ { n } \right) \right) x _ { i } ^ { n } 135 | \end{aligned} 136 | $$ 137 | 138 | 參數更新方式如下： 139 | 140 | $$ 141 | w _ { i } \leftarrow w _ { i } - \eta \sum _ { n } - \left( \hat { y } ^ { n } - f _ { w , b } \left( x ^ { n } \right) \right) x _ { i } ^ { n } 142 | $$ 143 | 144 | ## Logistic v.s. Linear 145 | 146 | 下面對 Logistic Regression 和 Linear Regression 做比較： 147 | 148 | | | Logistic Regression | Linear Regression | 149 | | :-----------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------: | 150 | | $f_{w, b}(x)$ | $\sigma (\sum_i w_ix_i + b)$ | $\sum_i w_ix_i + b$ | 151 | | Output | between $0$ and $1$ | any value | 152 | | Training data | $(x^n, \hat y^n)$ | $(x^n, \hat y^n)$ | 153 | | $\hat y^n$ | $1$ for class 1, $0$ for class 2 | a real number | 154 | | $L(f)$ | $\sum_n C(f(x^n), \hat y^n) = \sum_n - \left[ \hat { y } ^ { n } \ln f \left( x ^ { n } \right) + \left( 1 - \hat { y } ^ { n } \right) \ln \left( 1 - f \left( x ^ { n } \right) \right) \right]$ | $\frac 1 2 \sum_n (f(x^n) - \hat y^n)^2$ | 155 | | update method | $w_i \leftarrow w_i - \eta \sum_n - (\hat y^n - f_{w, b}(x^n)) x_i^n$ | 156 | 157 | ## Why not Logistic Regression with Square Error? 158 | 159 | 若我們的 loss function 改寫成 square error 版本： 160 | 161 | $$ 162 | L ( f ) = \frac { 1 } { 2 } \sum _ { n } \left( f _ { w , b } \left( x ^ { n } \right) - \hat { y } ^ { n } \right) ^ { 2 } 163 | $$ 164 | 165 | 在 Step 3: Find the best function 對 $w_i$ 做微分時： 166 | 167 | $$ 168 | \begin{aligned} 169 | \frac { \partial \left( f _ { w , b } ( x ) - \hat { y } \right) ^ { 2 } } { \partial w _ { i } } & = 2 \left( f _ { w , b } ( x ) - \hat { y } \right) \frac { \partial f _ { w , b } ( x ) } { \partial z } \frac { \partial z } { \partial w _ { i } } \\\\ 170 | & = 2 \left( f _ { w , b } ( x ) - \hat { y } \right) f _ { w , b } ( x ) \left( 1 - f _ { w , b } ( x ) \right) x _ { i } 171 | \end{aligned} 172 | $$ 173 | 174 | 不論 $\hat y^n = 1$ 或 $\hat y^n = 0$，都可能導致 $\partial L / \partial w _ { i } = 0$，無法有效更新參數。 175 | 176 | ## Generative v.s. Discriminative 177 | 178 | - Benefit of generative model 179 | - With the assumption of probability distribution, 180 | less training data is needed 181 | - With the assumption of probability distribution, more robust to the noise 182 | - Priors and class-dependent probabilities can be estimated from different sources. 183 | 184 | ## Multi-class Classification 185 | 186 | 我們用 3 個 classes 來做例子： 187 | 188 | ![](https://i.imgur.com/9892I0H.png) 189 | 190 | ## Limitation of Logistic Regression 191 | 192 | ![](https://i.imgur.com/ApJvKMF.png) 193 | 194 | 給定上述的 4 個 features，我們無法有效的分類，因為沒有任何線性的切法是可以完美將紅點和藍點分開，因此我們有以下兩種方法： 195 | 196 | - Feature Transformation: Not always easy to find a good transformation 197 | - Cascading logistic regression models -------------------------------------------------------------------------------- /docs/PCP/1-5.md: -------------------------------------------------------------------------------- 1 | # Liveness 2 | 3 | ## Deadlock 4 | 5 | === "begin" 6 | 7 | ```cpp 8 | // Two philosophers, thinking and eating sushi 9 | #include 10 | #include 11 | 12 | int sushi_count = 5000; 13 | 14 | void philosopher(std::mutex& first_chopstick, std::mutex& second_chopstick) { 15 | while (sushi_count > 0) { 16 | first_chopstick.lock(); 17 | second_chopstick.lock(); 18 | if (sushi_count) { 19 | sushi_count--; 20 | } 21 | second_chopstick.unlock(); 22 | first_chopstick.unlock(); 23 | } 24 | } 25 | 26 | int main() { 27 | std::mutex chopstick_a, chopstick_b; 28 | std::thread barron(philosopher, std::ref(chopstick_a), std::ref(chopstick_b)); 29 | std::thread olivia(philosopher, std::ref(chopstick_b), std::ref(chopstick_a)); 30 | barron.join(); 31 | olivia.join(); 32 | printf("The philosophers are done eating.\n"); 33 | } 34 | ``` 35 | 36 | === "end" 37 | 38 | ```cpp hl_lines="22" 39 | // Two philosophers, thinking and eating sushi 40 | #include 41 | #include 42 | 43 | int sushi_count = 5000; 44 | 45 | void philosopher(std::mutex& first_chopstick, std::mutex& second_chopstick) { 46 | while (sushi_count > 0) { 47 | first_chopstick.lock(); 48 | second_chopstick.lock(); 49 | if (sushi_count) { 50 | sushi_count--; 51 | } 52 | second_chopstick.unlock(); 53 | first_chopstick.unlock(); 54 | } 55 | } 56 | 57 | int main() { 58 | std::mutex chopstick_a, chopstick_b; 59 | std::thread barron(philosopher, std::ref(chopstick_a), std::ref(chopstick_b)); 60 | std::thread olivia(philosopher, std::ref(chopstick_a), std::ref(chopstick_b)); 61 | barron.join(); 62 | olivia.join(); 63 | printf("The philosophers are done eating.\n"); 64 | } 65 | ``` 66 | 67 | === "scoped_lock" 68 | 69 | ```cpp hl_lines="9 19" 70 | // Two philosophers, thinking and eating sushi 71 | #include 72 | #include 73 | 74 | int sushi_count = 5000; 75 | 76 | void philosopher(std::mutex& first_chopstick, std::mutex& second_chopstick) { 77 | while (sushi_count > 0) { 78 | std::scoped_lock lock(first_chopstick, second_chopstick); 79 | if (sushi_count) { 80 | sushi_count--; 81 | } 82 | } 83 | } 84 | 85 | int main() { 86 | std::mutex chopstick_a, chopstick_b; 87 | std::thread barron(philosopher, std::ref(chopstick_a), std::ref(chopstick_b)); 88 | std::thread olivia(philosopher, std::ref(chopstick_b), std::ref(chopstick_a)); 89 | barron.join(); 90 | olivia.join(); 91 | printf("The philosophers are done eating.\n"); 92 | } 93 | ``` 94 | 95 | ## Abandoned lock 96 | 97 | === "begin" 98 | 99 | ```cpp 100 | // Two philosophers, thinking and eating sushi 101 | #include 102 | #include 103 | 104 | int sushi_count = 5000; 105 | 106 | void philosopher(std::mutex& chopsticks) { 107 | while (sushi_count > 0) { 108 | chopsticks.lock(); 109 | if (sushi_count) { 110 | --sushi_count; 111 | } 112 | chopsticks.unlock(); 113 | } 114 | } 115 | 116 | int main() { 117 | std::mutex chopsticks; 118 | std::thread barron(philosopher, std::ref(chopsticks)); 119 | std::thread olivia(philosopher, std::ref(chopsticks)); 120 | barron.join(); 121 | olivia.join(); 122 | printf("The philosophers are done eating.\n"); 123 | } 124 | ``` 125 | 126 | === "end" 127 | 128 | ```cpp hl_lines="8 14" 129 | // Two philosophers, thinking and eating sushi#include 130 | #include 131 | 132 | int sushi_count = 5000; 133 | 134 | void philosopher(std::mutex& chopsticks) { 135 | while (sushi_count > 0) { 136 | std::scoped_lock lock(chopsticks); 137 | if (sushi_count) { 138 | --sushi_count; 139 | } 140 | if (sushi_count == 10) { 141 | printf("This philosopher has had enough!\n"); 142 | break; 143 | } 144 | } 145 | } 146 | 147 | int main() { 148 | std::mutex chopsticks; 149 | std::thread barron(philosopher, std::ref(chopsticks)); 150 | std::thread olivia(philosopher, std::ref(chopsticks)); 151 | barron.join(); 152 | olivia.join(); 153 | printf("The philosophers are done eating.\n"); 154 | } 155 | ``` 156 | 157 | ## Starvation 158 | 159 | === "begin" 160 | 161 | ```cpp 162 | // Too many philosophers, thinking and eating sushi 163 | #include 164 | #include 165 | #include 166 | 167 | int sushi_count = 5000; 168 | 169 | void philosopher(std::mutex& chopsticks) { 170 | int sushi_eaten = 0; 171 | while (sushi_count > 0) { 172 | std::scoped_lock lock(chopsticks); 173 | if (sushi_count) { 174 | --sushi_count; 175 | ++sushi_eaten; 176 | } 177 | } 178 | printf("Philosopher %d ate %d.\n", std::this_thread::get_id(), sushi_eaten); 179 | } 180 | 181 | int main() { 182 | std::mutex chopsticks; 183 | std::array philosophers; 184 | for (size_t i = 0; i < philosophers.size(); ++i) { 185 | philosophers[i] = std::thread(philosopher, std::ref(chopsticks)); 186 | } 187 | for (size_t i = 0; i < philosophers.size(); ++i) { 188 | philosophers[i].join(); 189 | } 190 | printf("The philosophers are done eating.\n"); 191 | } 192 | ``` 193 | 194 | === "end" 195 | 196 | ```cpp hl_lines="22" 197 | // Too many philosophers, thinking and eating sushi 198 | #include 199 | #include 200 | #include 201 | 202 | int sushi_count = 5000; 203 | 204 | void philosopher(std::mutex& chopsticks) { 205 | int sushi_eaten = 0; 206 | while (sushi_count > 0) { 207 | std::scoped_lock lock(chopsticks); 208 | if (sushi_count) { 209 | --sushi_count; 210 | ++sushi_eaten; 211 | } 212 | } 213 | printf("Philosopher %d ate %d.\n", std::this_thread::get_id(), sushi_eaten); 214 | } 215 | 216 | int main() { 217 | std::mutex chopsticks; 218 | std::array philosophers; 219 | for (size_t i = 0; i < philosophers.size(); ++i) { 220 | philosophers[i] = std::thread(philosopher, std::ref(chopsticks)); 221 | } 222 | for (size_t i = 0; i < philosophers.size(); ++i) { 223 | philosophers[i].join(); 224 | } 225 | printf("The philosophers are done eating.\n"); 226 | } 227 | ``` 228 | 229 | ## Livelock 230 | 231 | === "begin" 232 | 233 | ```cpp 234 | // Exceptionally polite philosophers, thinking and eating sushi 235 | #include 236 | #include 237 | 238 | int sushi_count = 5000; 239 | 240 | void philosopher(std::mutex& first_chopstick, std::mutex& second_chopstick) { 241 | while (sushi_count > 0) { 242 | first_chopstick.lock(); 243 | second_chopstick.lock(); 244 | if (sushi_count) { 245 | --sushi_count; 246 | } 247 | second_chopstick.unlock(); 248 | first_chopstick.unlock(); 249 | } 250 | } 251 | 252 | int main() { 253 | std::mutex chopstick_a, chopstick_b; 254 | std::thread barron(philosopher, std::ref(chopstick_a), std::ref(chopstick_b)); 255 | std::thread olivia(philosopher, std::ref(chopstick_b), std::ref(chopstick_a)); 256 | barron.join(); 257 | olivia.join(); 258 | printf("The philosophers are done eating.\n"); 259 | } 260 | ``` 261 | 262 | === "end" 263 | 264 | ```cpp hl_lines="10 11" 265 | // Exceptionally polite philosophers, thinking and eating sushi 266 | #include 267 | #include 268 | 269 | int sushi_count = 5000; 270 | 271 | void philosopher(std::mutex& first_chopstick, std::mutex& second_chopstick) { 272 | while (sushi_count > 0) { 273 | first_chopstick.lock(); 274 | if (!second_chopstick.try_lock()) { 275 | first_chopstick.unlock(); 276 | } else { 277 | if (sushi_count) { 278 | --sushi_count; 279 | } 280 | second_chopstick.unlock(); 281 | first_chopstick.unlock(); 282 | } 283 | } 284 | } 285 | 286 | int main() { 287 | std::mutex chopstick_a, chopstick_b; 288 | std::thread barron(philosopher, std::ref(chopstick_a), std::ref(chopstick_b)); 289 | std::thread olivia(philosopher, std::ref(chopstick_b), std::ref(chopstick_a)); 290 | std::thread steve(philosopher, std::ref(chopstick_a), std::ref(chopstick_b)); 291 | std::thread nikki(philosopher, std::ref(chopstick_b), std::ref(chopstick_a)); 292 | barron.join(); 293 | olivia.join(); 294 | steve.join(); 295 | nikki.join(); 296 | printf("The philosophers are done eating.\n"); 297 | } 298 | ``` 299 | 300 | === "yield" 301 | 302 | ```cpp hl_lines="12 13 14" 303 | // Exceptionally polite philosophers, thinking and eating sushi 304 | #include 305 | #include 306 | 307 | int sushi_count = 5000; 308 | 309 | void philosopher(std::mutex& first_chopstick, std::mutex& second_chopstick) { 310 | while (sushi_count > 0) { 311 | first_chopstick.lock(); 312 | if (!second_chopstick.try_lock()) { 313 | first_chopstick.unlock(); 314 | // allow current thread to wait a moment (don't pick the first_chopstick 315 | // so fast, so other threads can have chances to pick the first_chopstick) 316 | std::this_thread::yield(); 317 | } else { 318 | if (sushi_count) { 319 | --sushi_count; 320 | } 321 | second_chopstick.unlock(); 322 | first_chopstick.unlock(); 323 | } 324 | } 325 | } 326 | 327 | int main() { 328 | std::mutex chopstick_a, chopstick_b; 329 | std::thread barron(philosopher, std::ref(chopstick_a), std::ref(chopstick_b)); 330 | std::thread olivia(philosopher, std::ref(chopstick_b), std::ref(chopstick_a)); 331 | std::thread steve(philosopher, std::ref(chopstick_a), std::ref(chopstick_b)); 332 | std::thread nikki(philosopher, std::ref(chopstick_b), std::ref(chopstick_a)); 333 | barron.join(); 334 | olivia.join(); 335 | steve.join(); 336 | nikki.join(); 337 | printf("The philosophers are done eating.\n"); 338 | } 339 | ``` 340 | -------------------------------------------------------------------------------- /docs/PCP/2-1.md: -------------------------------------------------------------------------------- 1 | # Synchronization 2 | 3 | ## Condition variable 4 | 5 | === "busy waiting" 6 | 7 | ```cpp 8 | // Two hungry threads, anxiously waiting for their turn to take soup 9 | #include 10 | #include 11 | 12 | int soup_servings = 10; 13 | std::mutex slow_cooker_lid; 14 | 15 | void hungry_person(int id) { 16 | int put_lid_back = 0; 17 | while (soup_servings > 0) { 18 | std::unique_lock lid_lock( 19 | slow_cooker_lid); // pick up the slow cooker lid 20 | if ((id == soup_servings % 2) && 21 | (soup_servings > 0)) { // is it your turn to take soup? 22 | --soup_servings; // it's your turn; take some soup! 23 | } else { 24 | ++put_lid_back; // it's not your turn; put the lid back... 25 | } 26 | } 27 | printf("Person %d put the lid back %u times.\n", id, put_lid_back); 28 | } 29 | 30 | int main() { 31 | std::thread hungry_threads[2]; 32 | for (int i = 0; i < 2; ++i) { 33 | hungry_threads[i] = std::thread(hungry_person, i); 34 | } 35 | for (auto& ht : hungry_threads) { 36 | ht.join(); 37 | } 38 | } 39 | ``` 40 | 41 | === "notify_one" 42 | 43 | ```cpp hl_lines="4 8 18 22 23" 44 | // Two hungry threads, anxiously waiting for their turn to take soup 45 | #include 46 | #include 47 | #include 48 | 49 | int soup_servings = 10; 50 | std::mutex slow_cooker_lid; 51 | std::condition_variable soup_taken; 52 | 53 | void hungry_person(int id) { 54 | int put_lid_back = 0; 55 | while (soup_servings > 0) { 56 | std::unique_lock lid_lock( 57 | slow_cooker_lid); // pick up the slow cooker lid 58 | while ((id != soup_servings % 2) && 59 | (soup_servings > 0)) { // is it your turn to take soup? 60 | ++put_lid_back; // it's not your turn; put the lid back... 61 | soup_taken.wait(lid_lock); // ...and wait... 62 | } 63 | if (soup_servings > 0) { 64 | --soup_servings; // it's your turn; take some soup! 65 | lid_lock.unlock(); // put back the lid 66 | soup_taken.notify_one(); // notify another thread to take their turn 67 | } 68 | } 69 | printf("Person %d put the lid back %u times.\n", id, put_lid_back); 70 | } 71 | 72 | int main() { 73 | std::thread hungry_threads[2]; 74 | for (int i = 0; i < 2; ++i) { 75 | hungry_threads[i] = std::thread(hungry_person, i); 76 | } 77 | for (auto& ht : hungry_threads) { 78 | ht.join(); 79 | } 80 | } 81 | ``` 82 | 83 | === "notify_all" 84 | 85 | ```cpp hl_lines="15 23 30 31" 86 | // Two hungry threads, anxiously waiting for their turn to take soup 87 | #include 88 | #include 89 | #include 90 | 91 | int soup_servings = 10; 92 | std::mutex slow_cooker_lid; 93 | std::condition_variable soup_taken; 94 | 95 | void hungry_person(int id) { 96 | int put_lid_back = 0; 97 | while (soup_servings > 0) { 98 | std::unique_lock lid_lock( 99 | slow_cooker_lid); // pick up the slow cooker lid 100 | while ((id != soup_servings % 5) && 101 | (soup_servings > 0)) { // is it your turn to take soup? 102 | ++put_lid_back; // it's not your turn; put the lid back... 103 | soup_taken.wait(lid_lock); // ...and wait... 104 | } 105 | if (soup_servings > 0) { 106 | --soup_servings; // it's your turn; take some soup! 107 | lid_lock.unlock(); // put back the lid 108 | soup_taken.notify_all(); // notify another threads to take their turn 109 | } 110 | } 111 | printf("Person %d put the lid back %u times.\n", id, put_lid_back); 112 | } 113 | 114 | int main() { 115 | std::thread hungry_threads[5]; 116 | for (int i = 0; i < 5; ++i) { 117 | hungry_threads[i] = std::thread(hungry_person, i); 118 | } 119 | for (auto& ht : hungry_threads) { 120 | ht.join(); 121 | } 122 | } 123 | ``` 124 | 125 | ## Producer-consumer 126 | 127 | === "segmentation fault" 128 | 129 | ```cpp 130 | // Threads serving and eating soup 131 | #include 132 | #include 133 | 134 | class ServingLine { 135 | public: 136 | void serve_soup(int i) { 137 | soup_queue.push(i); 138 | } 139 | 140 | int take_soup() { 141 | int bowl = soup_queue.front(); 142 | soup_queue.pop(); 143 | return bowl; 144 | } 145 | 146 | private: 147 | std::queue soup_queue; 148 | }; 149 | 150 | ServingLine serving_line = ServingLine(); 151 | 152 | void soup_producer() { 153 | for (int i = 0; i < 1000000; ++i) { // serve a 10,000 bowls of soup 154 | serving_line.serve_soup(1); 155 | } 156 | serving_line.serve_soup(-1); // indicate no more soup 157 | printf("Producer is done serving soup!\n"); 158 | } 159 | 160 | void soup_consumer() { 161 | int soup_eaten = 0; 162 | while (true) { 163 | int bowl = serving_line.take_soup(); 164 | if (bowl == -1) { // check for last bowl of soup 165 | printf("Consumer ate %d bowls of soup.\n", soup_eaten); 166 | serving_line.serve_soup(-1); // put back last bowl for other consumers to take 167 | return; 168 | } else { 169 | soup_eaten += bowl; // eat the soup 170 | } 171 | } 172 | } 173 | 174 | int main() { 175 | std::thread olivia(soup_producer); 176 | std::thread barron(soup_consumer); 177 | std::thread steve(soup_consumer); 178 | olivia.join(); 179 | barron.join(); 180 | steve.join(); 181 | } 182 | ``` 183 | 184 | === "conditional_variable" 185 | 186 | ```cpp hl_lines="2 3 10 12 13 17 18 19 20 28 29" 187 | // Threads serving and eating soup 188 | #include 189 | #include 190 | #include 191 | #include 192 | 193 | class ServingLine { 194 | public: 195 | void serve_soup(int i) { 196 | std::unique_lock ladle_lock(ladle); 197 | soup_queue.push(i); 198 | ladle_lock.unlock(); 199 | soup_served.notify_one(); 200 | } 201 | 202 | int take_soup() { 203 | std::unique_lock ladle_lock(ladle); 204 | while (soup_queue.empty()) { 205 | soup_served.wait(ladle_lock); 206 | } 207 | int bowl = soup_queue.front(); 208 | soup_queue.pop(); 209 | return bowl; 210 | } 211 | 212 | private: 213 | std::queue soup_queue; 214 | std::mutex ladle; 215 | std::condition_variable soup_served; 216 | }; 217 | 218 | ServingLine serving_line = ServingLine(); 219 | 220 | void soup_producer() { 221 | for (int i = 0; i < 10000; ++i) { // serve a 10,000 bowls of soup 222 | serving_line.serve_soup(1); 223 | } 224 | serving_line.serve_soup(-1); // indicate no more soup 225 | printf("Producer is done serving soup!\n"); 226 | } 227 | 228 | void soup_consumer() { 229 | int soup_eaten = 0; 230 | while (true) { 231 | int bowl = serving_line.take_soup(); 232 | if (bowl == -1) { // check for last bowl of soup 233 | printf("Consumer ate %d bowls of soup.\n", soup_eaten); 234 | serving_line.serve_soup(-1); // put back last bowl for other consumers to take 235 | return; 236 | } else { 237 | soup_eaten += bowl; // eat the soup 238 | } 239 | } 240 | } 241 | 242 | int main() { 243 | std::thread olivia(soup_producer); 244 | std::thread barron(soup_consumer); 245 | std::thread steve(soup_consumer); 246 | olivia.join(); 247 | barron.join(); 248 | steve.join(); 249 | } 250 | ``` 251 | 252 | ## Semaphore 253 | 254 | === "begin" 255 | 256 | ```cpp 257 | // Connecting cell phones to a charger 258 | #include 259 | #include 260 | #include 261 | #include 262 | 263 | class Semaphore { 264 | public: 265 | Semaphore(unsigned long init_count) { count_ = init_count; } 266 | 267 | void acquire() { // decrement the internal counter 268 | std::unique_lock lck(m_); 269 | while (!count_) { 270 | cv_.wait(lck); 271 | } 272 | --count_; 273 | } 274 | 275 | void release() { // increment the internal counter 276 | std::unique_lock lck(m_); 277 | ++count_; 278 | lck.unlock(); 279 | cv_.notify_one(); 280 | } 281 | 282 | private: 283 | std::mutex m_; 284 | std::condition_variable cv_; 285 | unsigned long count_; 286 | }; 287 | 288 | Semaphore charger(4); 289 | 290 | void cell_phone(int id) { 291 | charger.acquire(); 292 | printf("Phone %d is charging...\n", id); 293 | srand(id); // charge for "random" amount between 1-3 seconds 294 | std::this_thread::sleep_for(std::chrono::milliseconds(rand() % 2000 + 1000)); 295 | printf("Phone %d is DONE charging!\n", id); 296 | charger.release(); 297 | } 298 | 299 | int main() { 300 | std::thread phones[10]; 301 | for (int i = 0; i < 10; ++i) { 302 | phones[i] = std::thread(cell_phone, i); 303 | } 304 | for (auto& p : phones) { 305 | p.join(); 306 | } 307 | } 308 | ``` 309 | 310 | === "end" 311 | 312 | ```cpp hl_lines="32" 313 | // Connecting cell phones to a charger 314 | #include 315 | #include 316 | #include 317 | #include 318 | 319 | class Semaphore { 320 | public: 321 | Semaphore(unsigned long init_count) { count_ = init_count; } 322 | 323 | void acquire() { // decrement the internal counter 324 | std::unique_lock lck(m_); 325 | while (!count_) { 326 | cv_.wait(lck); 327 | } 328 | --count_; 329 | } 330 | 331 | void release() { // increment the internal counter 332 | std::unique_lock lck(m_); 333 | ++count_; 334 | lck.unlock(); 335 | cv_.notify_one(); 336 | } 337 | 338 | private: 339 | std::mutex m_; 340 | std::condition_variable cv_; 341 | unsigned long count_; 342 | }; 343 | 344 | Semaphore charger(1); 345 | 346 | void cell_phone(int id) { 347 | charger.acquire(); 348 | printf("Phone %d is charging...\n", id); 349 | srand(id); // charge for "random" amount between 1-3 seconds 350 | std::this_thread::sleep_for(std::chrono::milliseconds(rand() % 2000 + 1000)); 351 | printf("Phone %d is DONE charging!\n", id); 352 | charger.release(); 353 | } 354 | 355 | int main() { 356 | std::thread phones[10]; 357 | for (int i = 0; i < 10; ++i) { 358 | phones[i] = std::thread(cell_phone, i); 359 | } 360 | for (auto& p : phones) { 361 | p.join(); 362 | } 363 | } 364 | ``` -------------------------------------------------------------------------------- /docs/OS/Chap04.md: -------------------------------------------------------------------------------- 1 | # Chapter 4 Threads 2 | 3 | ## 4.1 Overview 4 | 5 | !!! note "Thread–Lightweight process (LWP)" 6 | A basit unit of CPU utilization. 7 | 8 | A thread shares 9 | 10 | - code section 11 | - data section 12 | - OS resources (e.g. open files and signals) 13 | 14 | A thread have its own 15 | 16 | - thread ID 17 | - program counter 18 | - register set 19 | - stack 20 | 21 | ![normal](../assets/os/4.1.png) 22 | 23 | ### 4.1.1 Motivation 24 | 25 | It is generally more efficient to use one process that contains multiple threads since process creation is time consuming and resource intensive. 26 | 27 | ![normal](../assets/os/4.2.png) 28 | 29 | ### 4.1.2 Benefits 30 | 31 | The benefits of multithreaded: 32 | 33 | 1. **Responsiveness** 34 | 2. **Resource sharing** 35 | 3. **Economy** 36 | 4. **Scalability/Utilization** 37 | 38 | ## 4.2 Multicore Programming 39 | 40 | A more recent, similar trend in system design is to place multiple computing cores on a single chip. 41 | 42 | !!! note "Multicore or Multiprocessor systems" 43 | The cores appear across CPU chips or within CPU chips. 44 | 45 | Consider an application with 4 threads. 46 | 47 | - With a single core 48 | 49 | ![small](../assets/os/4.3.png) 50 | 51 | - With multiple cores 52 | 53 | ![small](../assets/os/4.4.png) 54 | 55 | | Parallelism | Concurrency | 56 | | :--: | :--: | 57 | | Perform more than one task simultaneously. | Allow all the tasks to make progress. | 58 | 59 | !!! note "Amdahl's Law" 60 | If $S$ is the portion cannot be accelerated by $N$ cores (serially). 61 | 62 | $$speedup \le \frac{1}{S + \frac{(1 - S)}{N}}$$ 63 | 64 | ### 4.2.1 Programming Challenges 65 | 66 | 1. Identifying tasks: Dividing Activities 67 | 2. Balance (Equal value) 68 | 3. Data splitting 69 | 4. Data dependency 70 | 5. Testing and debugging 71 | 72 | ### 4.2.2 Types of Parallelism 73 | 74 | !!! note "Data parallelism" 75 | Distribute subsets of the same data across multiple computing cores. 76 | 77 | Each core performs the same operation. 78 | 79 | e.g. 80 | 81 | $$\sum_{i = 0}^{N - 1} arr[i] = \sum_{i = 0}^{N / 2 - 1} arr[i] (\text{thread } A) + \sum_{i = N / 2}^{N - 1} arr[i] (\text{thread } B).$$ 82 | 83 | !!! note "Task parallelism" 84 | Distribute tasks (threads) across multiple computing cores. 85 | 86 | Each thread performs a unique operation. 87 | 88 | ## 4.3 Multithreading Models 89 | 90 | ### 4.3.1 Many-to-One Model 91 | 92 | - pros: 93 | - Efficiency 94 | 95 | - cons: 96 | - One blocking syscall blocks all 97 | - No parallelism for multiple processors 98 | 99 | e.g. Green threads (Solaris) 100 | 101 | ![small](../assets/os/4.5.png) 102 | 103 | ### 4.3.2 One-to-One Model 104 | 105 | - pros: 106 | - One syscall blocks one thread 107 | 108 | - cons: 109 | - Overheads in creating a kernel thread 110 | 111 | e.g. Windows NT/2000/XP, Linux, OS/2, Solaris 9 112 | 113 | ![small](../assets/os/4.6.png) 114 | 115 | ### 4.3.3 Many-to-Many Model 116 | 117 | - pros: 118 | - A combination of parallelism and efficiency 119 | 120 | e.g. Solaris 2 & 9, IRIX, HP-UX, Tru64 UNIX 121 | 122 | ![small](../assets/os/4.7.png) 123 | ![small](../assets/os/4.8.png) 124 | 125 | ## 4.4 Thread Libraries 126 | 127 | - User level 128 | - Kernel level 129 | 130 | Three main thread libraries: 131 | 132 | - POSIX Pthreads: User or kernel level 133 | - Windows: Kernel level 134 | - Java: Level depending on the thread library on the host system. 135 | 136 | Two general strategie for creating threads: 137 | 138 | - Asynchronous threading: parent doesn't know children. 139 | - Synchronous threading: parent must wait for all of its children. (***fork-join***) 140 | 141 | All of the following examples use synchronous threading. 142 | 143 | ### 4.4.1 Pthreads 144 | 145 | !!! note "Pthreads" 146 | The POSIX standard (IEEE 1003.1c) defining an API for thread creation and synchronization. This is a ***specification*** for thread behavior, not and ***implementation***. 147 | 148 | ### 4.4.2 Windows Threads 149 | 150 | ### 4.4.3 Java Threads 151 | 152 | Creating a `Thread` object does not specifically create the new thread. $\to$ `start()` does! 153 | 154 | 1. It allocates memory and initializes a new thread in the JVM. 155 | 2. It calls the `run()` method, making the thread eligible to be run by the JVM. 156 | (Note again that we never call the `run()` method directly. Rather, we call the `start()` method, and it calls the `run()` method on our behalf) 157 | 158 | ## 4.5 Implicit Threading 159 | 160 | ### 4.5.1 Thread Pools 161 | 162 | The issue of multithreaded server: 163 | 164 | - Time to create the thread 165 | - Concurrency 166 | 167 | !!! note "Thread pool" 168 | Create a number of threads at process startup and place them into a pool, where they sit and wait for work. 169 | 170 | The benefits of thread pools: 171 | 172 | 1. Speed 173 | 2. Limited # of threads, which is good for OS. 174 | 3. Seperating the task of creating tasks allows us to use different strategies. 175 | 4. Dynamic or static thread pools 176 | 177 | e.g. `QueueUserWorkItem()`, `java.util.concurrent`. 178 | 179 | ### 4.5.2 OpenMP 180 | 181 | !!! note "OpenMP" 182 | A set of compiler directivese and APIs to support parallel programming in shared memory environment. 183 | 184 | Threads with divided workload are created automatically based on # of cores or a set bound. 185 | 186 | !!! note "Parallel regions" 187 | Blocks of code that may run in parallel. 188 | 189 | When OpenMP encounters 190 | 191 | ```c 192 | #pragma omp parallel 193 | ``` 194 | 195 | it creates as many threads are there are processing cores in the system. 196 | 197 | e.g. 198 | 199 | ```c 200 | #pragma omp parallel for 201 | for (i = 0; i < N; ++i) 202 | c[i] = a[i] + b[i]; 203 | ``` 204 | 205 | ### 4.5.3 Grand Central Dispatch 206 | 207 | A ma 208 | sOS/iOS combination of extensions to the C. 209 | 210 | Like OpenMP, GCD manges most of the details of threading. 211 | 212 | ```c 213 | ^{ printf("I am a block."); } 214 | ``` 215 | 216 | GCD schedules blocks for run-time execution by placing them on a **dispatch queue**. 217 | 218 | - Serial (FIFO) 219 | - Concurrent (FIFO) 220 | - low 221 | - default 222 | - high 223 | 224 | ```c 225 | dispatch_queue_t queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0); 226 | dispatch_async(queue, ^{ printf("I am a block.") }); 227 | ``` 228 | 229 | ### 4.5.4 Other Approaches 230 | 231 | e.g. Intel's Threading Building Blocks (TBB), `java.util.concurrent`. 232 | 233 | ## 4.6 Threading Issues 234 | 235 | ### 4.6.1 The `fork()` and `exec()` System Calls 236 | 237 | !!! info "`fork()` issue" 238 | 239 | - Duplicate all threads? 240 | - Is the new process single-threaded? 241 | 242 | 243 | !!! info "`exec()` issue" 244 | If a thread invokese the `exec()`, the program specified in the parameter to `exec()` will replace the entire process—including all threads. Thus if `exec()` is called immediately after forking, then duplicating all threads is unnecessary. 245 | 246 | ### 4.6.2 Signal Handling 247 | 248 | All signals follows: 249 | 250 | 1. A signal is generated by the occurrence of a particular event. 251 | 2. The signal is delivered to a process. 252 | 3. Once delivered, the signal must be handled. 253 | 254 | Two types of signals: 255 | 256 | - Synchronous signal: delivered to the same process that performed the operation causing the signal. 257 | - illegal memory access 258 | - division by 0 259 | 260 | - Asynchronous signal 261 | - generated by an external process (e.g. ^C) 262 | - having a timer expire 263 | 264 | A signal may be handled by: 265 | 266 | 1. A default signal handler 267 | 2. A user-defined signal handler 268 | 269 | Signal delivering: 270 | 271 | 1. Deliver the signal to the thread to which the signal applies. (e.g. division by 0) 272 | 2. Deliver the signal to every thread in the process. (e.g. ^C) 273 | 3. Deliver the signal to certain threads in the process. 274 | 4. Assign a specific thread to receive all signals for the process. 275 | 276 | Functions/Methods for delivering a signal: 277 | 278 | - UNIX: 279 | 280 | ```c 281 | kill(pid_t pid, int signal) 282 | ``` 283 | 284 | - POSIX: 285 | 286 | ```c 287 | pthread_kill(pthread_t tid, int signal) 288 | ``` 289 | 290 | - Windows: 291 | 292 | - Asynchronous Procedure Calls (APCs) 293 | 294 | ### 4.6.3 Thread Cancellation 295 | 296 | !!! note "Target thread" 297 | A thread that is to be canceled. 298 | 299 | Cancellation of a target thread may occur in two different scenarios: 300 | 301 | 1. **Asynchronous cancellation**. One thread immediately terminates the target thread. 302 | 2. **Deferred cancellation**. The target thread periodically checks whether it should terminate, allowing it an opportunity to terminate itself in an orderly fashion. 303 | 304 | !!! info "" 305 | Canceling a thread asynchronously may not free a necessary system-wide resource. 306 | 307 | ```c 308 | pthread_tid; 309 | 310 | /* create the thread */ 311 | pthread_create(&tid, 0, worker, NULL); 312 | ... 313 | /* cancel the thread */ 314 | pthread_cancel(tid); 315 | ``` 316 | 317 | Pthreads supports three cancellation modes: 318 | 319 | | Mode | State | Type | 320 | | :--: | :--: | :--: | 321 | | Off | Disabled | – | 322 | | Deferred | Enabled | Deferred | 323 | | Asynchronous | Enabled | Asynchronous | 324 | 325 | !!! note "Cancellation point" 326 | Cancellation occurs only when a thread reaches a cancellation point. 327 | 328 | !!! note "Cleanup handler" 329 | If a cancellation request is found to be pending, this function allows any resources a thread may have acquired to be released before the thread is terminated. 330 | 331 | e.g. deferred cancellation: 332 | 333 | ```c 334 | while (1) { 335 | /* do some work for awhile */ 336 | 337 | /* check if there is a cancellation request */ 338 | pthread_testcancel(); 339 | } 340 | ``` 341 | 342 | ### 4.6.4 Thread-Local Storage 343 | 344 | !!! note "Thread-Local Storage (TLS)" 345 | Each thread might need its own copy of certain data. 346 | 347 | TLS is similar to `static` data. 348 | 349 | | TLS | Local variables | 350 | | :--: | :--: | 351 | | visible across function invocations | visible only during a single function invocation | 352 | 353 | ### 4.6.5 Scheduler Activations 354 | 355 | !!! note "Lightweight process (LWP)" 356 | A virtual processor (kernel threads) on which the application can schedule a user thread to run. (many-to-many or two-level) 357 | 358 | ![small](../assets/os/4.13.png) 359 | 360 | !!! note "Scheduler activation" 361 | The kernel provides an application with a set of virtual processors (LWPs), and the application can schedule user threads onto an available virtual processor. 362 | 363 | !!! note "Upcall" 364 | The kernel must inform an application about certain events. 365 | 366 | ## 4.7 Operating-System Examples 367 | 368 | ### 4.7.1 Windows Threads 369 | 370 | The general components of a thread include: 371 | 372 | - A thread ID uniquely identifying the thread 373 | - A register set representing the status of the processor 374 | - A user stack, employed when the thread is running in user mode 375 | - A kernel stack, employed when the thread is running in kernel mode 376 | - A private storage area used by various run-time libraries and dynamic link libraries (DLLs) 377 | 378 | **Context** (register set, stacks, and private sotrage area) of the thread: 379 | 380 | - Kernel space 381 | - ETHREAD—executive thread block: 382 | - a pointer to the process 383 | - the address of the routine 384 | - a pointer to the corresponding KTHREAD 385 | 386 | - KTHREAD—kernel thread block 387 | - scheduling/synchronization information 388 | - the kernel stack 389 | - a pointer to TEB 390 | 391 | - User space 392 | - TEB—thread environment block 393 | - the thread ID 394 | - a user-mode stack 395 | - an array for TLS 396 | 397 | ![small](../assets/os/4.14.png) 398 | 399 | ### 4.7.2 Linux Threads 400 | 401 | `clone()` vs `fork()` 402 | 403 | - term ***task***—rather than ***process*** or ***thread*** 404 | - Several per-process data structures 405 | - ***points*** to the same data structures for open files, signal handling, virtual memory, etc. -------------------------------------------------------------------------------- /docs/OS/Chap06.md: -------------------------------------------------------------------------------- 1 | # Chapter 6 CPU Scheduling 2 | 3 | ## 6.1 Basic Concepts 4 | 5 | The objective of multiprogramming is to have some process running at all times, to maximize CPU utilization. 6 | 7 | A process is executed until it must wait, typically for the completion of some I/O request. 8 | 9 | ### 6.1.1 CPU–I/O Burst Cycle 10 | 11 | Process execution = cycle of CPU + I/O wait. 12 | 13 | ![small](../assets/os/6.1.png) 14 | 15 | ### 6.1.2 CPU Scheduler 16 | 17 | **Short-term scheduler** (CPU scheduler): select process in the ready queue. 18 | 19 | ### 6.1.3 Preemptive Scheduling 20 | 21 | CPU-scheduling decisions when a process: 22 | 23 | 1. (nonpreemptive) Switches from the running state $\to$ the waiting state (e.g. I/O request, `wait()` for child) 24 | 2. (preemptive) Switches from the running state $\to$ the ready state (e.g. interrupt) 25 | 3. (preemptive) Switches from the waiting state $\to$ the ready state (e.g. completion of I/O) 26 | 4. (nonpreemptive) Terminates 27 | 28 | ### 6.1.4 Dispatcher 29 | 30 | !!! note "Dispatcher" 31 | The module that gives control of the CPU to *the process selected by the short-term scheduler*. It should be fast. 32 | 33 | - Switching context 34 | - Switching to user mode 35 | - Jumping to the proper location in the user program to restart that program 36 | 37 | !!! note "Dispatch latency" 38 | The time it takes for the dispatcher to stop one process and start another running. 39 | 40 | Stop a process $\leftrightarrow$ Start a process 41 | 42 | ## 6.2 Scheduling Criteria 43 | 44 | - **CPU utilization** 45 | - **Throughput** 46 | - **Turnaround time**: $\text{Completion Time} - \text{Start Time}$. 47 | - **Waiting time**: The sum of the periods spent waiting in the ready queue. 48 | - **Response time** 49 | 50 | ## 6.3 Scheduling Algorithms 51 | 52 | ### 6.3-1 First-Come, First-Served Scheduling 53 | 54 | - The process which requests the CPU first is allocated the CPU. 55 | - Properties: 56 | - Nonpreemptive FCFS. 57 | - CPU might be hold for an extended period. 58 | 59 | - Critical problem: Convoy effect! 60 | 61 | - Example: 62 | 63 | - Given processes: 64 | 65 | | Process | Busrt Time | 66 | | :-----: | :--------: | 67 | | $P_1$ | 24 | 68 | | $P_2$ | 3 | 69 | | $P_3$ | 3 | 70 | 71 | - Consider order: $P_1 \to P_2 \to P_3$: 72 | 73 | - Gantt chart: 74 | 75 | ![normal](../assets/os/FCFS.png) 76 | 77 | - Average waiting time = (0 + 24 + 27) / 3 = 17 ms. 78 | 79 | - Consider order: $P_2 \to P_3 \to P_1$: 80 | 81 | - Gantt chart: 82 | 83 | ![normal](../assets/os/FCFS-2.png) 84 | 85 | - Average waiting time = (0 + 3 + 6) / 3 = 9 ms. 86 | 87 | !!! note "Convoy effect" 88 | All the other processes wait for the one big process to get off the CPU. 89 | 90 | ### 6.3.2 Shortest-Job-First Scheduling 91 | 92 | - Properties: 93 | - Nonpreemptive SJF 94 | - ***Shortest-next-CPU-burst first*** 95 | 96 | - Problem: Measure the future! 97 | 98 | - Example 1: 99 | 100 | - Given processes: 101 | 102 | | Process | Burst Time | 103 | | :-----: | :--------: | 104 | | $P_1$ | 6 | 105 | | $P_2$ | 8 | 106 | | $P_3$ | 7 | 107 | | $P_4$ | 3 | 108 | 109 | - By SJF scheduling: 110 | 111 | - Gantt chart: 112 | 113 | ![normal](../assets/os/SJF.png) 114 | 115 | - Average waiting time = (3 + 16 + 9 + 0) / 4 = 7 ms. 116 | 117 | !!! info "" 118 | SJF is used frequently in long-term (job) scheduling, but it cannot be implemented at the level of short-term CPU scheduling. 119 | 120 | !!! note "Exponential average" 121 | Let $t_n$ be time of $n$th CPU burst, and $\tau_{n + 1}$ be the next CPU burst. 122 | 123 | $$ 124 | \begin{aligned} 125 | \tau_{n + 1} & = \alpha t_n + (1 - \alpha)\tau_n, \quad 0 \le \alpha \le 1. \\\\ 126 | & = \alpha t_n + (1 - \alpha)\alpha t_{n - 1} + \cdots + (1 - \alpha)^j \alpha t_{n - j} + \cdots + (1 - \alpha)^{n + 1}\tau_0. 127 | \end{aligned} 128 | $$ 129 | 130 | - Example 2: 131 | 132 | - Given processes: 133 | 134 | | Process | Arrival Time | Burst Time | 135 | | :-----: | :----------: | :--------: | 136 | | $P_1$ | 0 | 8 | 137 | | $P_2$ | 1 | 4 | 138 | | $P_3$ | 2 | 9 | 139 | | $P_4$ | 3 | 5 | 140 | 141 | By preemptive SJF scheduling: 142 | 143 | - Gantt chart: 144 | 145 | ![normal](../assets/os/SJF-2.png) 146 | 147 | - Average waiting time = [(10 - 1) + (1 - 1) + (17 - 2) + (5 - 3)] / 4 = 26 / 4 = 6.5 ms. 148 | 149 | ### 6.3.3 Priority Scheduling 150 | 151 | - Properties: 152 | - CPU is assigned to the process with the highest priority — A framework for various scheduling algorithms: 153 | - FCFS: Equal-priority with tie-breaking 154 | - SJF: Priority = 1 / next CPU burst length 155 | 156 | - Example: 157 | 158 | - Given processes: 159 | 160 | | Process | Burst Time | Priority | 161 | | :-----: | :--------: | :------: | 162 | | $P_1$ | 10 | 3 | 163 | | $P_2$ | 1 | 1 | 164 | | $P_3$ | 2 | 4 | 165 | | $P_4$ | 1 | 5 | 166 | | $P_5$ | 5 | 2 | 167 | 168 | - By preemptive SJF scheduling: 169 | 170 | - Gantt chart: 171 | 172 | ![normal](../assets/os/priority.png) 173 | 174 | - Average waiting time = (6 + 0 + 16 + 18 + 1) / 5 = 8.2 ms. 175 | 176 | !!! info "Problem with priority scheduling" 177 | - indefinite blocking: low-priority processes could starve to death! 178 | - starvation 179 | 180 | !!! note "Aging" 181 | Aging gradually increase the priority of processes that wait in the system for a long time. 182 | 183 | ### 6.3.4 Round-Robin Scheduling 184 | 185 | - RR is similar to FCFS except that preemption is added to switch between processes. 186 | - Goal: fairness, time sharing. 187 | 188 | - Example: 189 | 190 | - Given processes with quantum = 4ms: 191 | 192 | | Process | Burst Time | 193 | | :-----: | :--------: | 194 | | $P_1$ | 24 | 195 | | $P_2$ | 3 | 196 | | $P_3$ | 3 | 197 | 198 | - Gantt chart: 199 | 200 | ![normal](../assets/os/RR.png) 201 | 202 | - Average waiting time = [(10 - 4) + 4 + 7] / 3 = 5.66 ms. 203 | 204 | !!! info "" 205 | Although the time quantum should be large compared with the context switch time, it should not be too large. 206 | 207 | ### 6.3.5 Multilevel Queue Scheduling 208 | 209 | - Intra-queue scheduling 210 | - Independent choice of scheduling algorithms 211 | 212 | - Inter-queue scheduling 213 | - Fixed-priority preemptive scheduling 214 | - e.g., foreground queues always have absolute priority over the background queues. 215 | - Time slice between queues 216 | - e.g., 80% CPU is given to foreground processes, and 20% CPU is given to background processes. 217 | 218 | Each queue has absolute priority over lower-priority queues. 219 | 220 | If an interactive editing process entered the ready queue while a batch process was running, the batch process would be preempted. 221 | 222 | ![normal](../assets/os/6.6.png) 223 | 224 | ### 6.3.6 Multilevel Feedback Queue Scheduling 225 | 226 | !!! info "" 227 | Multilevel feedback queue allows a process to move between queues. 228 | 229 | A multilevel feedback queue is defined by: 230 | 231 | - \#queues 232 | - The scheduling algorithm for each queue 233 | - The method used to determine when to 234 | - upgrade a process to a higher priority queue 235 | - demote a process to a lower priority queue 236 | 237 | - The method used to determine which queue a process will enter when that process needs service 238 | 239 | ## 6.4 Thread Scheduling 240 | 241 | ### 6.4.1 Contention Scope 242 | 243 | !!! note "Process Contention Scope (PCS)" 244 | On systems implementing the [many-to-one](./Chap04/#431-many-to-one-model) and [many-to-many](./Chap04/#433-many-to-many-model) models, the thread library schedules user-level threads to run on an available LWP, since competition for the CPU takes place among threads belonging to the same process. 245 | 246 | It is important to note that PCS will typically preempt the thread currently running in favor of a higher-priority thread 247 | 248 | !!! note "System Contention Scope (SCS)" 249 | Competition for the CPU with SCS scheduling takes place among all threads in the system. Systems using the [one-to-one](./Chap04/#432-one-to-one-model) model. 250 | 251 | ### 6.4.2 Pthread Scheduling 252 | 253 | - PCS: `PTHREAD_SCOPE_PROCESS` 254 | - SCS: `PTHREAD_SCOPE_SYSTEM` 255 | 256 | 2 methods: 257 | 258 | - `pthread_attr_setscope(pthread attr_t *attr, int scope)` 259 | - `pthread_attr_getscope(pthread attr_t *attr, int *scope)` 260 | 261 | ## 6.5 Multiple-Processor Scheduling 262 | 263 | ### 6.5.1 Approaches to Multiple-Processor Scheduling 264 | 265 | - **Asymmetric multiprocessing**: only *the master server* process accesses the system data structures, reducing the need for data sharing. 266 | - **Symmetric multiprocessing (SMP)**: each processor is self-scheduling. 267 | 268 | ### 6.5.2 Processor Affinity 269 | 270 | !!! note "Processor Affinity" 271 | A process has an affinity for the processor on which it is currently running. 272 | 273 | - **Soft affinity** 274 | - **Hard affinity** (e.g. Linux: `sched_setaffinity()`) 275 | 276 | !!! note "Non-Uniform Memory Access (NUMA)" 277 | A CPU has faster access to some parts of main memory than to other parts. 278 | 279 | ![small](../assets/os/6.9.png) 280 | 281 | ### 6.5.3 Load Balancing 282 | 283 | On systems with a common run queue, load balancing is often unnecessary, because once a processor becomes idle, it immediately extracts a runnable process from the common run queue. 284 | 285 | - **Push migration**: pushing processes from overloaded to less-busy processors. 286 | - **Pull migration**: pulling a waiting task from a busy processor. 287 | 288 | ### 6.5.4 Multicore Processors 289 | 290 | SMP systems that use multicore processors are faster and consume less power than systems in which each processor has its own physical chip. 291 | 292 | There are two ways to multithread a processing core: 293 | 294 | - **Coarse-grained**: a thread executes on a processor until a long-latency event such as a memory stall occurs. 295 | - **Fine-grained (interleaved)**: switches between threads at a much finer level of granularity 296 | 297 | ## 6.6 Real-Time CPU Scheduling 298 | 299 | - **Soft real-time systems** 300 | - **Hard real-time systems** 301 | 302 | ### 6.6.1 Minimizing Latency 303 | 304 | !!! note "Event latency" 305 | The amount of time that elapses from when an event occurs to when it is serviced. 306 | 307 | There are 2 types: 308 | 309 | 1. Interrupt latency 310 | 2. Dispatch latency 311 | 312 | !!! note "Interrupt latency" 313 | The period of time from the arrival of an interrupt at the CPU to the start of the interrupt service routine (ISR) that services the interrupt. 314 | 315 | !!! note "Dispatch latency" 316 | The amount of time required for the scheduling dispatcher to stop one process and start another. 317 | 318 | ### 6.6.2 Priority-Based Scheduling 319 | 320 | The processes are considered **periodic**. That is, they require the CPU at constant intervals (periods). ($t$: fixed processing time, $d$: deadline, and $p$: period.) 321 | 322 | $$0 \le t \le d \le p.$$ 323 | 324 | $$rate = 1 / p.$$ 325 | 326 | **Admission-control**: 327 | 328 | 1. Admits the process. 329 | 2. Rejects the request. 330 | 331 | ### 6.6.3 Rate-Monotonic Scheduling 332 | 333 | - Example 1: 334 | 335 | - Given processes: (the deadline for each process requires that it complete its CPU burst by the start of its next period.) 336 | 337 | | Process | Period | ProcTime | 338 | | :-----: | :---------: | :--------: | 339 | | $P_1$ | $p_1 = 50$ | $t_1 = 20$ | 340 | | $P_2$ | $p_2 = 100$ | $t_2 = 35$ | 341 | 342 | ![normal](../assets/os/6.17.png) 343 | 344 | - Example 2: 345 | 346 | - Given processes: 347 | 348 | | Process | Period | ProcTime | 349 | | :-----: | :--------: | :--------: | 350 | | $P_1$ | $p_1 = 50$ | $t_1 = 25$ | 351 | | $P_2$ | $p_2 = 80$ | $t_2 = 35$ | 352 | 353 | ![normal](../assets/os/6.18.png) 354 | 355 | ### 6.6.4 Earliest-Deadline-First Scheduling 356 | 357 | ![normal](../assets/os/6.19.png) 358 | 359 | ### 6.6.5 Proportional Share Scheduling 360 | 361 | Proportional share schedulers operate by allocating $T$ shares among all applications. An application can receive $N$ shares of time. 362 | 363 | ### 6.6.6 POSIX Real-Time Scheduling 364 | 365 | - POSIX.1b — Extensions for real-time computing 366 | - `SCHED_FIFO` 367 | - `SCHED_RR` 368 | - `SCHED_OTHER` 369 | 370 | - API 371 | - `pthread_attr_getsched_policy(pthread_attr_t *attr, int *policy)` 372 | - `pthread_attr_setsched_policy(pthread_attr_t *attr, int *policy)` 373 | 374 | ## 6.7 Operating-System Examples 375 | 376 | ### 6.7.1 Example: Linux Scheduling 377 | 378 | ### 6.7.2 Example: Windows Scheduling 379 | 380 | ### 6.7.3 Example: Solaris Scheduling 381 | 382 | ## 6.8 Algorithm Evaluation 383 | 384 | ### 6.8.1 Deterministic Modeling 385 | 386 | !!! note "Deterministic modeling" 387 | It is one type of analytic evaluation. This method takes a particular predetermined workload and defines the performance of each algorithm for that workload. 388 | 389 | ### 6.8.2 Queueing Models 390 | 391 | - Little's formula ($n$: # of processes in the queue, $\lambda$: arrival rate, $W$: average waiting time in the queue.) 392 | 393 | $$n = \lambda \times W.$$ 394 | 395 | ### 6.8.3 Simulations 396 | 397 | - Properties: 398 | - Accurate but expensive 399 | 400 | - Procedures: 401 | - Program a model of the computer system 402 | - Drive the simulation with various data sets 403 | 404 | ### 6.8.4 Implementation -------------------------------------------------------------------------------- /docs/OS/Chap08.md: -------------------------------------------------------------------------------- 1 | # Chapter 8 Main Memory 2 | 3 | ## 8.1 Background 4 | 5 | The CPU fetches instructions from memory according to the value of the program counter. 6 | 7 | Memory Management Motivation: 8 | 9 | - Keep several processes in memory to improve a system's performance 10 | 11 | ### 8.1.1 Basic Hardware 12 | 13 | !!! info "" 14 | **Main memory** and the **registers** built into the processor itself are the only general-purpose storage that the CPU can access directly. 15 | 16 | We need to make sure that each process has a separate memory space. 17 | 18 | We can provide this protection by using two registers, usually **a base** and **a limit**. 19 | 20 | ![normal](../assets/os/8.1.png) 21 | ![normal](../assets/os/8.2.png) 22 | 23 | ### 8.1.2 Address Binding 24 | 25 | The binding of instructions and data to memory addresses can be done at any step along the way: 26 | 27 | - **Compile time**: absolute code. 28 | - **Load time**: relocatable code. 29 | - **Execution time** 30 | 31 | ![small](../assets/os/8.3.png) 32 | 33 | A major portion of this chapter is devoted to showing how these various bindings can be implemented effectively in a computer system and to discussing appropriate hardware support. 34 | 35 | ### 8.1.3 Logical Versus Physical Address Space 36 | 37 | !!! note "Logical address" 38 | An address generated by the CPU. 39 | 40 | !!! note "Physical address" 41 | An address seen by the memory unit and loaded into the memory-address register. 42 | 43 | - Generating identical logical and physical addresses: 44 | - Compile-time address-binding 45 | - Load-time address-binding 46 | 47 | - Generating different logical and physical addresses: 48 | - Execution-time address-binding 49 | - logical address $\to$ **virtual address** 50 | 51 | !!! note "Logical address space" 52 | The set of all logical addresses generated by a program. 53 | 54 | !!! note "Physical address space" 55 | The set of all physical addresses corresponding to these logical addresses. 56 | 57 | !!! note "Memory-management unit (MMU)" 58 | Run-time map from virtual to physical addresses. 59 | 60 | ![normal](../assets/os/8.4.png) 61 | 62 | The user program never sees the real physical addresses and deals with logical addresses. 63 | 64 | The memory-mapping hardware converts logical addresses into physical addresses 65 | 66 | ### 8.1.4 Dynamic Loading 67 | 68 | !!! note "Dynamic loading" 69 | A routine is not loaded until it is called. 70 | 71 | ### 8.1.5 Dynamic Linking and Shared Libraries 72 | 73 | !!! note "Dynamically linked libraries" 74 | System libraries that are linked to user programs when the programs are run. 75 | 76 | !!! note "Static linking" 77 | System libraries are treated like any other object module and are combined by the loader into the binary program image. 78 | 79 | !!! note "Dynamic linking" 80 | Linking is postponed until execution time. e.g. language subroutine libraries. 81 | 82 | With dynamic linking, a **stub** is included in the image for each library-routine reference. 83 | 84 | !!! note "Stub" 85 | A small piece of code that indicates how to locate the appropriate memory-resident library routine or how to load the library if the routine is not already present. 86 | 87 | !!! note "Shared libraries" 88 | Only programs that are compiled with the new library version are affected by any incompatible changes incorporated in it. Other programs linked before the new library was installed will continue using the older library. 89 | 90 | ## 8.2 Swapping 91 | 92 | A process must be in memory to be executed. A process, however, can be swapped temporarily out of memory to a **backing store** and then brought back into memory for continued execution. 93 | 94 | ### 8.2.1 Standard Swapping 95 | 96 | ![normal](../assets/os/8.5.png) 97 | 98 | The context-switch time in such a swapping system is fairly high. 99 | 100 | Let's assume that the user process is 100 MB in size and the backing store is a standard hard disk with a transfer rate of 50 MB per second. The actual transfer of the 100-MB process to or from main memory takes 101 | 102 | $$\frac{100\text{MB}}{50\text{MB}/s} = 2s.$$ 103 | 104 | ### 8.2.2 Swapping on Mobile Systems 105 | 106 | Mobile systems do not support swapping in general but might have paging (iOS and Android) 107 | 108 | ## 8.3 Contiguous Memory Allocation 109 | 110 | The memory is usually divided into two partitions: 111 | 112 | - one for the resident operating system and 113 | - one for the user processes. 114 | 115 | !!! note "Contiguous memory allocation" 116 | Each process is contained in a single section of memory that is contiguous to the section containing the next process. 117 | 118 | ### 8.3.1 Memory Protection 119 | 120 | - The **relocation register** contains the value of the smallest *physical address*. 121 | - The **limit register** contains the range of *logical addresses*. 122 | 123 | ![normal](../assets/os/8.6.png) 124 | 125 | ### 8.3.2 Memory Allocation 126 | 127 | One of the simplest methods for allocating memory is to divide memory into several fixed-sized **partitions**. 128 | 129 | Thus, the degree of multiprogramming is bound by the number of partitions. 130 | 131 | - In this **multiple-partition method**, when a partition is free, a process is selected from the input queue and is loaded into the free partition. 132 | 133 | - In the **variable-partition** scheme, the operating system keeps a table indicating which parts of memory are available and which are occupied. 134 | 135 | **Dynamic storage-allocation problem** concerns how to satisfy a request of size $n$ from a list of free holes. There are many solutions to this problem: 136 | 137 | - **First fit** 138 | - **Best fit** 139 | - **Worst fit** 140 | 141 | Simulations have shown that both first fit and best fit are better than worst fit in terms of decreasing time and storage utilization. Neither first fit nor best fit is clearly better than the other in terms of storage utilization, but first fit is generally faster. 142 | 143 | ### 8.3.3 Fragmentation 144 | 145 | Both the first-fit and best-fit strategies for memory allocation suffer from **external fragmentation**. 146 | 147 | !!! note "External fragmentation" 148 | There is enough total memory space to satisfy a request but the available spaces are not contiguous 149 | 150 | !!! note "50-percent rule" 151 | Even with some optimization, given $N$ allocated blocks, another $0.5 N$ blocks will be lost to fragmentation. That is, one-third of memory may be unusable! 152 | 153 | !!! note "Internal fragmentation" 154 | The memory allocated to a process may be slightly larger than the requested memory. The difference between these two numbers is internal fragmentation. 155 | 156 | !!! note "Compaction" 157 | The goal is to shuffle the memory contents so as to place all free memory together in one large block. 158 | 159 | Compaction is possible only if 160 | 161 | - relocation is dynamic and 162 | - is done at execution time. 163 | 164 | Another possible solution to the external-fragmentation problem is to permit the logical address space of the processes to be noncontiguous, thus allowing a process to be allocated physical memory wherever such memory is available: 165 | 166 | - Segmentation 167 | - Paging 168 | 169 | ## 8.4 Segmentation 170 | 171 | !!! note "Segmentation" 172 | A memory-management scheme that supports this programmer view of memory. 173 | 174 | ### 8.4.1 Basic Method 175 | 176 | segments are numbered and are referred to by a segment number, rather than by a segment name. Thus, a logical address consists of a *two tuple*: 177 | 178 |
179 | <*segment-number, offset*> 180 |
181 | 182 | ### 8.4.2 Segmentation Hardware 183 | 184 | A logical address consists of two parts: 185 | 186 | - a segment number: $s$ 187 | - an offset into that segment: $d$ 188 | 189 | **Segment table** consists of 190 | 191 | - a segment base: contains the starting physical address where the segment resides in memory. 192 | - a segment limit: specifies the length of the segment. 193 | 194 | ![normal](../assets/os/8.8.png) 195 | 196 | ## 8.5 Paging 197 | 198 | Paging avoids external fragmentation and the need for compaction, whereas segmentation does not. 199 | 200 | - breaking physical memory into fixed-sized blocks called **frames** 201 | - breaking logical memory into blocks of the same size called **pages** 202 | 203 | Every address generated by the CPU is divided into two parts: 204 | 205 | - a **page number (p)** 206 | - a **page offset (d)** 207 | 208 | ![normal](../assets/os/8.10.png) 209 | 210 | ![normal](../assets/os/8.11.png) 211 | 212 | If the size of the logical address space is $2^m$, and a page size is $2^n$ bytes. The logical address is as follows: 213 | 214 | ![normal](../assets/os/page.png) 215 | 216 | On my MacBook Pro (15-inch, 2017), I obtain the page size of 4096 bytes = 4 KB. 217 | ![normal](../assets/os/pagesize.png) 218 | 219 | In the worst case, a process would need $n$ pages plus $1$ byte. It would be allocated $n + 1$ frames, resulting in internal fragmentation of almost an entire frame. 220 | 221 | !!! note "Frame table" 222 | It has one entry for each physical page frame, indicating whether the latter is free or allocated and, if it is allocated, to which page of which process or processes. 223 | 224 | ### 8.5.2 Hardware Support 225 | 226 | - The page table is implemented as a set of dedicated **registers**. 227 | - pros: fast 228 | - cons: entries could be small 229 | 230 | - The page table is kept in main memory, and a **page-table base register** (**PTBR**) points to the page table. Changing page tables requires changing only this one register, substantially reducing context-switch time. 231 | - pros: entries could be large 232 | - cons: slow (the time required to access a user memory location) 233 | - With this scheme, ***two*** memory accesses are needed to access a byte (one for the page-table entry, one for the byte) 234 | 235 | !!! note "Translation look-aside buffer (TLB)" 236 | The TLB is associative, high-speed memory. Each entry in the TLB consists of two parts: 237 | 238 | - a key (or tag) 239 | - a value 240 | 241 | The TLB contains only a few of the page-table entries (typically between 32 and 1024 entries). When a logical address is generated by the CPU, its page number is presented to the TLB. If the page number is found, its frame number is immediately available and is used to access memory. As just mentioned, *these steps are executed as part of the instruction pipeline within the CPU, adding no performance penalty* compared with a system that does not implement paging. 242 | 243 | !!! note "TLB miss" 244 | The page number is not in the TLB. 245 | 246 | ![normal](../assets/os/8.14.png) 247 | 248 | !!! info "TLB replacement policies" 249 | If the TLB is already full of entries, an existing entry must be selected for replacement. Replacement policies range from **least recently used (LRU)** through round-robin to random. 250 | 251 | !!! info "Wire down" 252 | Some TLBs allow certain entries to be **wired down**, meaning that they cannot be removed from the TLB. Typically, TLB entries for key kernel code are wired down. 253 | 254 | Some TLBs store **address-space identifiers** (**ASIDs**) in each TLB entry. 255 | 256 | !!! note "ASID" 257 | An ASID uniquely identifies each process and is used to provide address-space protection for that process. 258 | 259 | !!! note "Hit ratio" 260 | The percentage of times that the page number of interest is found in the TLB. 261 | 262 | ### 8.5.3 Protection 263 | 264 | We can create hardware to provide read-only, read-write, or execute-only protection. 265 | 266 | !!! note "Valid–invalid bit" 267 | When this bit is set to valid, the associated page is in the process's logical address space and is thus a legal page. 268 | 269 | !!! note "Page-table length register (PTLR)" 270 | To indicate the size of the page table. 271 | 272 | ### 8.5.4 Shared Pages 273 | 274 | To be sharable, the code must be reentrant. The read-only nature of shared code should not be left to the correctness of the code; the operating system should enforce this property. 275 | 276 | ## 8.6 Structure of the Page Table 277 | 278 | ### 8.6.1 Hierarchical Paging 279 | 280 | Hierarchical paging is also known as **forward-mapped page table**. 281 | 282 | ![normal](../assets/os/8.18.png) 283 | 284 | ![normal](../assets/os/page2.png) 285 | 286 | The VAX minicomputer from Digital Equipment Corporation (DEC): 287 | 288 | ![normal](../assets/os/VAX.png) 289 | 290 | ### 8.6.2 Hashed Page Tables 291 | 292 | !!! note "Clustered page tables" 293 | Each entry in the hash table refers to several pages (such as 16) rather than a single page, which are particularly useful for address spaces. 294 | 295 | ![normal](../assets/os/8.19.png) 296 | 297 | ### 8.6.3 Inverted Page Tables 298 | 299 | !!! note "Inverted page table" 300 | An inverted page table has one entry for each real page (or frame) of memory. Each entry consists of the virtual address of the page stored in that real memory location, with information about the process that owns the page. 301 | 302 | ![normal](../assets/os/8.20.png) 303 | 304 | For the ***IBM RT***, each virtual address in the system consists of a triple: 305 | 306 |
307 | <*process-id, page-number, offset*> 308 |
309 | 310 | !!! info "Shared memory & Inverted page tables" 311 | Shared memory cannot be used with inverted page tables; because there is only one virtual page entry for every physical page, one physical page cannot have two (or more) shared virtual addresses. 312 | 313 | ### 8.6.4 Oracle SPARC Solaris 314 | 315 | !!! note "TLB walk" 316 | - If a match is found in the TSB, the CPU copies the TSB entry into the TLB, and the memory translation completes. 317 | - If no match is found in the TSB, the kernel is interrupted to search the hash table. 318 | 319 | ## 8.7 Example: Intel 32 and 64-bit Architectures 320 | 321 | ### 8.7.1 IA-32 Architecture 322 | 323 | ![normal](../assets/os/8.21.png) 324 | 325 | segmentation unit + paging unit = memory-management unit (MMU) 326 | 327 | #### 8.7.1.1 IA-32 Segmentation 328 | 329 | The logical address is a pair (selector, offset), where the selector is a 16-bit number: 330 | 331 | ![normal](../assets/os/IA-32.png) 332 | 333 | - $s$: the segment number 334 | - $g$: GDT or LDT 335 | - $p$: protection. 336 | 337 | ![normal](../assets/os/8.22.png) 338 | 339 | #### 8.7.1.2 IA-32 Paging 340 | 341 | ![normal](../assets/os/8.23.png) 342 | 343 | !!! note "Page address extension (PAE)" 344 | It allows 32-bit processors to access a physical address space larger than 4 GB. 345 | 346 | ![normal](../assets/os/8.24.png) 347 | 348 | ### 8.7.2 x86-64 349 | 350 | ## 8.8 Example: ARM Architecture 351 | 352 | ## 8.9 Summary 353 | 354 | - Hardware support 355 | - Performance 356 | - Fragmentation 357 | - Relocation 358 | - Swapping 359 | - Sharing 360 | - Protection --------------------------------------------------------------------------------