├── README.md ├── chapter1.md ├── chapter10.md ├── chapter11.md ├── chapter12.md ├── chapter13.md ├── chapter14.md ├── chapter15.md ├── chapter16.md ├── chapter17.md ├── chapter18.md ├── chapter19.md ├── chapter2.md ├── chapter20.md ├── chapter3.md ├── chapter4.md ├── chapter5.md ├── chapter6.md ├── chapter7.md ├── chapter8.md └── chapter9.md /README.md: -------------------------------------------------------------------------------- 1 | # Data Structures and Algorithms Notes 2 | Notes while reading and working through Data Structures and Algorithms Made Easy in Java (https://www.amazon.com/Data-Structures-Algorithms-Made-Easy/dp/1468101277). 3 | 4 | * [Chapter 1: Introduction](chapter1.md) 5 | * [Chapter 2: Recursion and Backtracking](chapter2.md) 6 | * [Chapter 3: Linked Lists](chapter3.md) 7 | * [Chapter 4: Stacks](chapter4.md) 8 | * [Chapter 5: Queues](chapter5.md) 9 | * [Chapter 6: Trees](chapter6.md) 10 | * [Chapter 7: Heaps & Priority Queues](chapter7.md) 11 | * [Chapter 8: Disjoint Sets](chapter8.md) 12 | * [Chapter 9: Graph Algorithms](chapter9.md) 13 | * [Chapter 10: Sorting](chapter10.md) 14 | * [Chapter 11: Searching](chapter11.md) 15 | * [Chapter 12: Selection Algorithms](chapter12.md) 16 | * [Chapter 13: Symbol Tables](chapter13.md) 17 | * [Chapter 14: Hashing](chapter14.md) 18 | * [Chapter 15: String Algorithms](chapter15.md) 19 | * [Chapter 16: Algorithm Design Techniques](chapter16.md) 20 | * [Chapter 17: Greedy Algorithms](chapter17.md) 21 | * [Chapter 18: Divide and Conquer Algorithms](chapter18.md) 22 | * [Chapter 19: Dynamic Programming](chapter19.md) 23 | * [Chapter 20: Complexity Classes](chapter20.md) 24 | -------------------------------------------------------------------------------- /chapter1.md: -------------------------------------------------------------------------------- 1 | ## Chapter 1: Introduction 2 | * 1 byte = 8 bits (00000000 through 11111111) 3 | * Data structure = special format for organizing and storing data 4 | * Linear data structures: elements accessed in a sequential order 5 | * Nonlinear data structures: elements stored/accessed in a nonlinear manner (trees/graphs) 6 | * Abstract Data Types (ADTs) = combination of declaration of data structures and operations that can be performed on them 7 | * When defining ADTs don't worry about implementation details 8 | * Algorithm = step by step instructions to solve a given problem 9 | * Types of algorithmic analysis 10 | * Worst case 11 | * Best case 12 | * Average case 13 | * Big-O Notation = Asymptotic tight upper bound on an algorithm's performance for inputs of large size 14 | * Omega Notation = Tighter lower bound of an algorithm's performance for inputs of large size 15 | * Theta Notation = average case of an algorithm's performance (an average of all the complexities of the algorithm) 16 | * Represents the upper and lower bounds for the bounds of an algorithm's performance for large inputs 17 | * Used especially if upper and lower bounds are the same 18 | * Recurrence problems 19 | * A recurrence is when an algorithm divides itself into smaller components plus a set amount of work per component 20 | * Guess the big-o, omega, or theta Notation 21 | * Prove it correct by induction (or incorrect and guess again) 22 | * Amortized analysis = worst case analysis for a sequence of operations rather than for individual operations 23 | * The operation in the sequence with the worst performance will be amortized over the entire sequence 24 | -------------------------------------------------------------------------------- /chapter10.md: -------------------------------------------------------------------------------- 1 | ## Chapter 10: Sorting 2 | * Categorized by: 3 | * Number of comparisons 4 | * Number of swaps 5 | * Memory usage 6 | * Recursion 7 | * Stability (are items with the same sort key value in the same order in the result?) 8 | * Adaptability (how does pre-sortedness affect performance) 9 | * For sorts that use comparisons 10 | * Best case is O(n log n) 11 | * Worse case is O(n^2) 12 | 13 | ### Bubble Sort 14 | * O(n^2) in best or worst case 15 | * Smaller elements bubble to the top 16 | ``` 17 | def bubble_sort(array): 18 | for iteration in range(len(array)-1, -1, -1): 19 | for index in range(iteration-1): 20 | if array[index] > array[index+1]: 21 | array[index], array[index+1] = array[index+1], array[index] 22 | ``` 23 | * O(n) in improved version for an already sorted array 24 | ``` 25 | def bubble_sort_improved(array): 26 | swapped = True 27 | for iteration in range(len(array)-1, -1, -1): 28 | if not swapped: 29 | break 30 | swapped = False 31 | for index in range(iteration-1): 32 | if array[index] > array[index+1]: 33 | array[index], array[index+1] = array[index+1], array[index] 34 | swapped = True 35 | ``` 36 | 37 | ### Selection Sort 38 | * In-place sorting algorithm 39 | * Repeatedly selects the smallest element 40 | * Algorithm 41 | * Find the minimum value in the list 42 | * Swap it with the value in the current position 43 | * Repeat for all elements until entire array is sorted 44 | * O(n^2) 45 | ``` 46 | def selection_sort(array): 47 | for i in range(len(array)): 48 | min = i 49 | for j in range(i+1, len(array)): 50 | if array[j] < array[min]: 51 | min = j 52 | array[min], array[i] = array[i], array[min] 53 | ``` 54 | 55 | ### Insertion Sort 56 | * Simple and efficient comparison sort 57 | * Each iteration removes an element from the input data and inserts it into the correct position in the list being sorted 58 | * Stable, in-place, adaptive 59 | * O(n^2) but practically more efficient than selection sort or bubble sort 60 | * Algorithm 61 | * Remove an element from the input data 62 | * Insert it into the correct position in the already sorted list until no input elements remain 63 | * This can be done in place 64 | ``` 65 | def insertion_sort(array): 66 | for i in range(1, len(array)): 67 | # assign next current element to insert into sorted portion of list 68 | curr = array[i] 69 | j = i 70 | 71 | # look at where in sorted list current element belongs 72 | while array[j-1] > curr and j >= 1: 73 | array[j] = array[j-1] 74 | j -= 1 75 | array[j] = curr 76 | ``` 77 | 78 | ### Shell Sort 79 | * More complex generalization of insertion sort 80 | * Improves upon insertion sort by generating and sorting subarrays until the gap size is 1 81 | * Makes several passes of the array, for a decreasing gap size 82 | * In each pass, sort the elements in the (i+gap*0, i+gap*1, i+gap*2, i+gap*3, …) subarrays 83 | * Decrease gap size, eventually reaching 1, which is one pass of insertion sort 84 | ``` 85 | def shell_sort(array): 86 | gaps = [701, 301, 132, 57, 23, 10, 4, 1] 87 | 88 | # Start with the largest gap and work down to a gap of 1 89 | for gap in gaps: 90 |     # Do a gapped insertion sort for this gap size. 91 |     # The first gap elements array[0..gap-1] are already in gapped order 92 |     # keep adding one more element until the entire array is gap sorted 93 |     for i in range(gap, len(array), 1): 94 |         # add array[i] to the elements that have been gap sorted 95 |         # save array[i] in temp and make a hole at position i 96 |         temp = array[i] 97 | 98 |         # shift earlier gap-sorted elements up until the correct location for array[i] is found 99 | j = i 100 |         while j >= gap and array[j - gap] > temp: 101 |             array[j] = array[j - gap] 102 | j -= gap 103 | 104 |         # put temp (the original array[i]) in its correct location 105 |         array[j] = temp 106 | ``` 107 | 108 | ### Merge Sort 109 | * MergeSort each half 110 | * Merge the two halves 111 | * Two finger algorithm 112 | * O(n) 113 | * O(n log n) 114 | ``` 115 | def merge_sort(array, low, high): 116 | if high <= low: 117 | return 118 | 119 | mid = (low + high)/2 120 | merge_sort(array, low, mid) 121 | merge_sort(array, mid+1, high) 122 | merge(array, low, high) 123 | 124 | def merge(array, low, high): 125 | mid = (low + high)/2 126 | left = low 127 | right = mid+1 128 | 129 | sorted_array = [] 130 | while left <= mid and right <= high: 131 | if array[left] <= array[right]: 132 | sorted_array.append(array[left]) 133 | left += 1 134 | else: 135 | sorted_array.append(array[right]) 136 | right += 1 137 | 138 | while left <= mid: 139 | sorted_array.append(array[left]) 140 | left += 1 141 | 142 | while right <= high: 143 | sorted_array.append(array[right]) 144 | right += 1 145 | 146 | for i in range(len(sorted_array)): 147 | array[low] = sorted_array[i] 148 | low += 1 149 | ``` 150 | 151 | ### Quick Sort 152 | * Divides a large array into two smaller subarrays: low elements and high elements 153 | * Algorithm: 154 | * Pick an element (right-most or median of 3 random elements) 155 | * Partition 156 | * Reorder the array so that all the elements with values < pivot are before pivot 157 | * All elements with values > pivot come after pivot 158 | * Recursively apply the previous steps to the low and high subarrays 159 | * Theta(n log n) 160 | * O(n^2) when array is sorted or all the same element 161 | 162 | ### Linear Time Sorting 163 | * Assume n keys are integers {0, 1, …, K-1} and each fits in a **word** 164 | * Can do a lot more than comparisons 165 | * For small enough k, can sort in O(n) time 166 | 167 | #### Counting Sort 168 | * Including assumptions above 169 | * Integer inputs 170 | * Relatively small k 171 | * O(n+k) 172 | ``` 173 | L = array of k empty lists 174 | for j in range(n): 175 | L[key(A[j])].append(A[j]) 176 | 177 | output = [] 178 | for i in range(k): 179 | output.extend(L[i]) 180 | ``` 181 | 182 | ### Radix Sort 183 | * Imagine each integer in the input as base b 184 | * # digits = d = log_b k 185 | * Algorithm: 186 | * Sort integers by least significant digit 187 | * Sort integers by next least significant digit 188 | * … 189 | * Sort integers by most significant digit 190 | * Each of the above sorts by digit uses counting sort 191 | * Total time = O((n+b)*d) = O((n+b) * log_b k) 192 | * For b = Theta(n) ==> O(n log_n k) 193 | * When k <= n^c ==> **O(nc)** 194 | -------------------------------------------------------------------------------- /chapter11.md: -------------------------------------------------------------------------------- 1 | ## Chapter 11: Searching 2 | * Unordered linear search = O(n) 3 | * Binary Search = O(log n) 4 | 5 | ``` 6 | def binary_search_recursive(array, val, lo, hi): 7 | if lo > hi: 8 | return -1 9 | 10 | mid = lo + (hi - lo)/2 11 | if array[mid] == val: 12 | return mid 13 | elif array[mid] < val: 14 | return binary_search_recursive(array, val, mid+1, hi) 15 | else: 16 | return binary_search_recursive(array, val, lo, mid-1) 17 | ``` 18 | 19 | ``` 20 | def binary_search_iterative(array, val): 21 | lo = 0 22 | hi = len(array) - 1 23 | 24 | while lo <= hi: 25 | mid = lo + (hi - lo)/2 26 | if array[mid] == val: 27 | return mid 28 | elif array[mid] < val: 29 | lo = mid + 1 30 | else: 31 | hi = mid - 1 32 | 33 | return -1 34 | ``` 35 | -------------------------------------------------------------------------------- /chapter12.md: -------------------------------------------------------------------------------- 1 | ## Chapter 12: Selection Algorithms 2 | * Find the k-th largest number in a sequence 3 | * Selection by sorting 4 | * Sort list then select desired element 5 | * Advantageous if selecting many elements 6 | * O(n log n) to sort 7 | * Amortized to O(log n) if selecting n elements 8 | * Finding the k-th smallest element in sorted order 9 | * Balanced tree method 10 | * Put first k elements into balanced tree 11 | * For each remaining element 12 | * if > largest in tree, continue 13 | * else, remove largest in tree and replace with current element = O(log k) 14 | * k-th smallest is largest in tree 15 | * **Partition method (similar to quicksort - a.k.a quickselect)** 16 | * Choose a pivot element and partition and sort around it 17 | * if partition index < k, k-th smallest is in right side so partition that side 18 | * else if partition index == k, return element 19 | * else, partition index > k, so partition left side 20 | * O(n^2) worst case 21 | * Theta(n log k) on average ==> linear in n 22 | -------------------------------------------------------------------------------- /chapter13.md: -------------------------------------------------------------------------------- 1 | SKIPPED 2 | -------------------------------------------------------------------------------- /chapter14.md: -------------------------------------------------------------------------------- 1 | ## Chapter 14: Hashing 2 | * Worst case = O(n) 3 | * Average case = O(1) 4 | * Hashing = mapping keys to memory locations 5 | * This is typically achieved by converting the key to a number and taking the remainder of the table size 6 | * Hashing is mixing and recombining the value in some way such that it creates as unique a value as possible 7 | * Universal hashing 8 | * h(k) = [ ( ak + b ) mod p ] mod m 9 | * m is table size 10 | * a and b are random integers [0, p) 11 | * p is prime number > universe of k ==> large prime 12 | * Hash Table 13 | * **A generalized array whose indexes can be selected based on a hash of the value stored in that index** 14 | * Direct addressing = Direct access table = using an array’s index as the key 15 | * Applicable when we can afford to allocate an array with one position for every possible key 16 | * Hashing allows us to map keys to values 17 | * Useful when the number of possible keys is large compared to the actual number of keys 18 | * When table is full, double table size, like array 19 | * When number of elements shrinks to size/4, shrink table to size/2 ==> avoids repeated O(n) operations when adding/removing around the doubling/halving number of elements 20 | * Hash Functions 21 | * Used to transform the key into the index 22 | * Ideally should map each possible key to a unique slot index, but difficult to achieve in practice 23 | * An efficient hash function should be designed so that it distributes the index values of inserted objects uniformly across the table 24 | * Must be calculated quickly, minimize collisions, and efficiently resolve collisions 25 | * Collision = when to records are stored in the same location 26 | * Collision Resolution Techniques 27 | * Chaining = the value at each index is a linked list, which allows collisions to be resolved by having several elements in a single index 28 | * Open Addressing 29 | * **in notes below, `m` indicates the table size** 30 | * If a collision occurs, the value is stored at a different index 31 | * **Insert - If initial hash index is filled, probe for a new index until empty or DeleteMe index is found, then insert value in table** 32 | * **Search - If value at hash index is not desired key, probe for a new index until desired key or None is found** 33 | * **Delete - If value at hash index is not desired key, probe for a new index until desired key is found, then delete the value at the table and replace with a DeleteMe flag, which indicates that it is empty at present** 34 | * Linear probing = stored at the next open index 35 | * h(k, i) = (h'(k) + i) % m 36 | * may result in poor distribution throughout the table (cluster) 37 | * Quadratic probing = stored in a subsequent index, progressing in a quadratic manner 38 | * h(k, i) = (h'(k) + c1*i + c2*i^2) % m 39 | * may result in a better distribution throughout the table as keys get spread out 40 | * Double hashing = use a primary hash function for initial index, then combine with a second hash function for the probing sequence 41 | * h(k, i) = (h1(k) + i*h2(k)) % m 42 | * h2(k) and m must be relatively prime ==> this can be achieved by making m a power of 2 and making h2 always return an odd number 43 | * Expected number of probes = 1 / (1 - load factor) 44 | * Load factor = ( # elements in hash table ) / ( size of hash table) 45 | * If average number of elements in an index > load factor, rehash elements with a bigger hash table size 46 | * Bloom Filter 47 | * Probabilistic data structure which was designed to check whether an element is present in a set with memory and time efficiency 48 | * Tells us that element is either NOT in set or MAY BE in set 49 | * Initialize an array of all zeros 50 | * When an element is added, use k hash functions to hash the element itself 51 | * Set the array index to 1 for each of the k indexes yielded by the hash functions 52 | * If an element is queried and the indexes yielded by the hash functions are not all 1, then the element DEFINITELY HAS NOT been added 53 | * If they ARE all, then it is likely that it has been added but they all may have had collisions 54 | 55 | ### Rabin-Karp Algorithm 56 | * Implements rolling hash ADT with O(1) operations 57 | * `r` maintains an internal string `x` 58 | * Treat `x` as multi digit number `u` in base a (a = size of alphabet — ASCII) 59 | * This can be done because the string `x` is an array of its characters, which can be represented as an array of ASCII integers 60 | * For example, for `a = 10`, `x = [ 1, 2, 3, 4, 5 ] = 1*10^4 + 2*10^3 + 3*10^2 + 4*10^1 + 5*10^0` 61 | * This is the number that will then be used as input to the hash function 62 | * **This is advantageous because when rolling, only two O(1) operations must be performed: append new element to end & remove first element** 63 | * To achieve this: 64 | * Add element to end: multiply `x[ i+1 : ]` by `a` and add the new element 65 | * Remove element from front: `x_i - a ^ (L-1) * x[i]` 66 | * Generalized: `h(x_i+1) = a * ( h(x_i) - a ^ ( L-1 ) * x[i] ) + x[ i + L ] mod m`, where L is the length of s 67 | * `r.append(char)` = add char to end of `x` 68 | * `r.append(char)` = delete first char from `x` (assuming it is `char`) 69 | * `r()` = hash value of x = h(x) 70 | ``` 71 | # looking for substring s in corpus t 72 | 73 | # initialize rolling hash for s = rs 74 | for c in s: 75 | rs.append(c) 76 | 77 | # initialize rolling hash for t = rt 78 | for c in t[:len(s)]: 79 | rt.append(t) 80 | 81 | # check equality 82 | if rs() == rt(): 83 | # substring of t MAY match s 84 | # check whether s == t[ i-len(s)+1 : i+1 ] 85 | if s == t[ i-len(s)+1 : i+1 ]: 86 | return True 87 | 88 | # rs and rt not initially equal 89 | # roll through and look for equality 90 | for i in range(len(s), len(t)): 91 | rt.skip(t[i-len(s)]) 92 | rt.append(t[i]) 93 | if rs() == rt(): 94 | # substring of t MAY match s 95 | # check whether s == t[ i-len(s)+1 : i+1 ] 96 | if s == t[ i-len(s)+1 : i+1 ]: 97 | return True 98 | else: 99 | # happens with probability <= 1/s 100 | pass 101 | ``` 102 | -------------------------------------------------------------------------------- /chapter15.md: -------------------------------------------------------------------------------- 1 | ## Chapter 15: String Algorithms 2 | 3 | ### String Matching Algorithms 4 | * Is pattern P (length m) a substring of string T (length n)? 5 | * Brute Force = O(nm) 6 | ``` 7 | def brute_force(T, P): 8 | n = len(T) 9 | m = len(P) 10 | 11 | for i in range(n-m+1): 12 | j = 0 13 | while j < m and T[i + j] == P[j]: 14 | j += 1 15 | 16 | if j == m: 17 | return i 18 | ``` 19 | * Rabin-Karp = O(nm) worst case (bad hash function) but O(n+m) with good hash function 20 | * See chapter 14 notes 21 | * Knuth-Morris-Pratt (KMP) = O(n+m) 22 | * Avoids having to backtrack in the text T 23 | * Uses a prefix table to identify prefixes that are substrings in P of the same characters 24 | * When a partial match is found in a substring of T but the next character does not match the next desired character in P, check whether the suffix of the matched characters in P is also a prefix in P 25 | * This is a check as to whether a new successful attempt has already begun, for example, the substrings will match up until the positions indicated 26 | * The suffix of the successful substring of P `41` is a suffix and the prefix, `3` is the next character that should be checked for equality, since the suffix `41` is known to be correct 27 | ``` 28 | | 29 | P = 413412 30 | T = 123413413412 31 | | 32 | 33 | ``` 34 | * To achieve this in code, a prefix table must be built based on pattern P, which traverses P looking for internally matching substrings 35 | ``` 36 | int F[]; // assume F is a global array 37 | public void PrefixTable(int P[], int m) { 38 | int i = 1, j = 0, F[0] = 0; 39 | while (i < m) { 40 | if (P[i] == P[j]) { 41 | F[i] = j+1 42 | i++; 43 | j++; 44 | } else if (j > 0) { 45 | j = F[j-1]; 46 | } else { 47 | F[i] = 0; 48 | i++; 49 | } 50 | } 51 | } 52 | ``` 53 | * The prefix table helps the search because it tells you where in the prefix table to look if there is a failed match when looking for P in T 54 | * The matching component is as follows: 55 | ``` 56 | public int KMP(char T[], int n, int P[], int m) { 57 | int i = 0, j = 0; 58 | PrefixTable(P, m); 59 | while (i < n) { 60 | if (T[i] == P[j]) { 61 | if (j == m-1) { 62 | return i-j; 63 | } else { 64 | i++; 65 | j++; 66 | } 67 | } else if (j > 0) { 68 | // have we completed a prefix of P already? 69 | j = F[j-1]; 70 | } else { 71 | i++; 72 | } 73 | } 74 | return -1; 75 | } 76 | ``` 77 | 78 | ### Data Structures for Storing Sets of Strings 79 | * Hash Tables 80 | * Good, but loses the location of strings 81 | * For example, to find all letters beginning with a certain character, the entire table must be scanned 82 | * Binary Search Trees 83 | * Good in terms of storage, but bad in terms of search time 84 | * This is because the entire string must be searched when comparing a string vs a target string 85 | * Tries 86 | * Prefix-based tree 87 | * Tree in which each node contains pointers that represent up to every letter in the alphabet 88 | * Insert and Search in O(L) where L is length of a single word 89 | * When deleting a word, if it has children, just set `endOfWord = false` so as to not delete part of the tree (see https://www.youtube.com/watch?v=AXjmTQ8LEoI) 90 | * If two strings share a prefix, they will have a common ancestor in the tree 91 | ``` 92 | TrieNode { 93 | children map[character]TrieNode 94 | endOfWord bool 95 | } 96 | ``` 97 | * Ternary Search Trees (TSTs) 98 | * Combines aspects of BSTs and tries 99 | * left points to the TST containing all the strings which are alphabetically less than data 100 | * right points to the TST containing all the strings which are alphabetically greater than data 101 | * eq points to the TST containing all the strings which contain `data` as a prefix 102 | ``` 103 | TSTNode { 104 | char data; 105 | boolean endOfWord; 106 | TSTNode left; 107 | TSTNode eq; 108 | TSTNode right; 109 | } 110 | ``` 111 | 112 | ### Suffix Trees 113 | * Tree representation of a single string and possible suffixes 114 | * Construction of the tree is complicated but it solves many string problems in O(n) time 115 | * Exact string matching 116 | * Longest repeated substring 117 | * Longest palindrome 118 | * Longest common substring of two strings 119 | * Longest common prefix of two strings 120 | * Suffix tree definition for string T of length n 121 | * contains n leaves which are numbered 1 to n 122 | * each internal node (except root) should have at least 2 children 123 | * each edge is labeled by a non-empty substring of T 124 | * no two edges of a node begin with the same character 125 | * the paths from root to leaves represent all suffixes of T 126 | * Construction of suffix trees — there exist O(n) algorithms for constructing 127 | -------------------------------------------------------------------------------- /chapter16.md: -------------------------------------------------------------------------------- 1 | SKIPPED 2 | -------------------------------------------------------------------------------- /chapter17.md: -------------------------------------------------------------------------------- 1 | ## Chapter 17: Greedy Algorithms 2 | * Greedy is best suited for looking at the immediate situation rather than looking at future states 3 | * Assumes that a local good selection makes for a global optimal solution 4 | * Two basic properties of optimal Greedy algorithms 5 | * Greedy choice property = the globally optimal solution can be obtained by making a locally optimal solution (and may depend on past choices but not future choices), and reducing the problem 6 | * Optimal substructure = optimal solution to the problem contains optimal solutions to the subproblems 7 | * However, in many situations, there is no guarantee that making locally optimal improvements in a locally optimal solution gives the optimal global solution 8 | * Often useful in conjunction with heaps 9 | -------------------------------------------------------------------------------- /chapter18.md: -------------------------------------------------------------------------------- 1 | ## Chapter 18: Divide and Conquer Algorithms 2 | * Recursively break down a problem into 2+ subproblems of the same type, until they become simple enough to be solved directly, and then combine solutions to subproblems to get solution to original problem 3 | * Sometimes useful when Greedy fails 4 | * Subproblems must be smaller instances of the same type of problem as the original problem 5 | 6 | ### Advantages of Divide and Conquer 7 | * Solving difficult problems (towers of Hanoi) 8 | * Parallelism 9 | * Memory access: once a subproblem is small, it can be solved within the cache rather than in slower main memory 10 | 11 | ### Disadvantages of Divide and Conquer 12 | * Recursion is slow 13 | * May be more complicated than iterative approaches 14 | 15 | ### Applications 16 | * Binary search 17 | * Merge sort and quick sort 18 | * Median finding 19 | * Min and max finding 20 | * Matrix multiplication 21 | * Closest pair problem 22 | -------------------------------------------------------------------------------- /chapter19.md: -------------------------------------------------------------------------------- 1 | ## Chapter 19: Dynamic Programming 2 | * DP = recursion + memoization 3 | * Useful when we fail to get optimal solutions with Greedy or divide and conquer 4 | * Differs from divide and conquer because d&c subproblems are independent but DP subproblems may overlap 5 | * Reduces exponential complexity to polynomial 6 | * Properties of DP Strategy 7 | * Optimal substructure = an optimal solution contains optimal solutions to subproblems 8 | * Overlapping subproblems = a recursive solution contains a small number of distinct subproblems repeated many times 9 | * Top down vs bottom up 10 | * DP problems 11 | * String longest common subsequence, longest increasing subsequence, longest common substring 12 | * Efficient graph algorithms (Bellman-Ford) 13 | * Chain matrix multiplication 14 | * Subset sum 15 | * Knapsack 16 | * Traveling Salesman 17 | 18 | ### OCW Notes 19 | * Useful for optimization problems (min/max) 20 | * DP = Careful brute force 21 | * DP = memoize and reuse solutions to subproblems that help solve the problem = **guessing + recursion + memoization** 22 | * DP = shortest paths in some DAG 23 | * **RUNNING TIME = ( # subproblems ) x ( time per subproblem )** 24 | * Bottom up DP = topological sort of subproblem DAG 25 | * Guess…try ALL guesses and take the best one 26 | * **Subproblem dependencies should be acyclic** 27 | * Five Steps to DP: 28 | 1. Subproblems (**THINK ABOUT SUFFIXES**) ==> # subproblems 29 | 2. Guess (part of solution) ==> # choices for guess 30 | 3. Recurrence ==> time/subproblem 31 | 4. Recurse and memoize OR build DP table bottom-up ==> check subproblem recurrence is acyclic (i.e., has topological order) 32 | 5. Solve original problem ==> total time 33 | * Parent pointer = remember which guess was best…follow parent pointers to get the optimal subproblem solutions that build the optimal solution 34 | 35 | ### Subproblems for Strings/Sequences 36 | * Suffixes — x[i:] for all i 37 | * Prefixes — x[:i] for all i 38 | * All substrings — x[i:j] for all i <= j 39 | 40 | ### Knapsack 41 | * List of items, each with size s_i and value v_i (both integers) 42 | * Knapsack of size S 43 | * maximize sum of values for a subset of items of total size <= S 44 | 2. Guessing: is item i in subset or not? 45 | 1. Subproblem = suffix `i:` of items & remaining capacity X <= S 46 | 3. DP(i, X) = max( DP(i+1, X), DP(i+1, X-s_i) + v_i ) 47 | 4. Total time = O(nS) — pseudopolynomial time because S is one of the inputs 48 | 49 | -------------------------------------------------------------------------------- /chapter2.md: -------------------------------------------------------------------------------- 1 | ## Chapter 2: Recursion and Backtracking 2 | * Recursion is most useful for tasks that can be defined in terms of similar subtasks 3 | * Recursion adds overhead for each recursive call (space on the stack frame), as compared with the equivalent iterative operation 4 | * Generally iterative calls are more efficient 5 | * Backtracking is a method of exhaustive search using divide and conquer, by trying all possibilities 6 | -------------------------------------------------------------------------------- /chapter20.md: -------------------------------------------------------------------------------- 1 | ## Chapter 20: Complexity Classes 2 | * P = set of decision problems that can be solved on a deterministic machine in polynomial time 3 | * Solutions are easy to find 4 | * NP = set of problems that can be solved on a non-deterministic machine in polynomial time 5 | * Hard to solve, easy to verify 6 | * Co-NP = "no" answer can be given in polynomial time on a non-deterministic machine 7 | * NP-hard = takes a long time to check whether the solution is correct 8 | * NP-complete = NP and NP-hard 9 | -------------------------------------------------------------------------------- /chapter3.md: -------------------------------------------------------------------------------- 1 | ## Chapter 3: Linked Lists 2 | * Data structure for storing collections of data, such that successive elements are connected by pointers with last element pointing to null 3 | * Can grow and shrink in size during execution and does not waste memory 4 | * Primary operations for the linked list ADT 5 | * Insert 6 | * Delete (and return element at specified position) 7 | * Delete list 8 | * Count 9 | * Find n-th 10 | * Arrays 11 | * One memory block is allocated for an entire array 12 | * Array elements can be accessed in constant time 13 | * The size of the data type is multiplied by the index and added to the memory address of the array itself 14 | * The result is the memory address of the desired element, after one multiplication and one addition operation 15 | * Dynamic arrays can make it easy to add and remove elements 16 | * Linked lists vs arrays 17 | * Arrays are simpler and have faster access to the elements 18 | * But they require a single block of memory and inserting an element requires shifting elements after it 19 | * Linked lists can be expanded in constant time 20 | * But access time for linked lists is linear in the worst case, they require extra manipulation when deleting the last item, and they require additional memory for reference pointers 21 | * Doubly linked list 22 | * Pointer to previous node and next node 23 | * Convenience at the expense of more space used and more operations required when inserting or deleting 24 | * Circular Linked List 25 | * Each node has a successor 26 | * Only node needed for access is the tail node (which provides access to the head node) 27 | * Used in managing the computing resources of a computer 28 | * Can be used to implement stacks and queues 29 | * Unrolled Linked Lists 30 | * Store a circular linked list in the element of each node 31 | * All nodes should contain no more than sqrt(n) elements, except the last one (n elements in the full list) 32 | * When inserting an element, elements may need to be moved to the next circular linked list 33 | * This shift operation requires removing a node from the tail (head.next) of the CLL in a block and inserting it to the head of the next CLL 34 | * O(sqrt(n)) because there are O(sqrt(n)) blocks and each insertion is O(1) 35 | * Can save a significant amount of memory as compared with doubly linked lists 36 | * Skip Lists 37 | * Linked list with additional pointers to skip intermediate nodes 38 | * Can be used as an alternative to balanced binary trees 39 | * Skip list allows for a quick search, insertion, and deletion of elements 40 | * When a node is generated, it is randomly determined how many forward pointers it will have 41 | * Can have as many levels of forward pointers as desired, but cap them at MaxLevel 42 | * O(log n) performance for search if forward pointers are distributed in a base 2 manner (and only 2x the number o pointers) 43 | * O(log n) performance for insert and delete operations 44 | * The sorting technique 45 | * List pointers in array and sort them 46 | * Duplicates will be next to one another 47 | * The search technique 48 | * List all pointers in one array 49 | * Use a hash table to traverse the array and find the first repeat 50 | * Floyd cycle finding algorithm 51 | * Use two pointers to identify whether a linked list is a cycle or whether it terminates 52 | * Slow pointer moves one node at a time 53 | * Fast pointer moves two nodes at a time 54 | * If they meet, there is a cycle 55 | * If one hits null, the list terminates 56 | * **To find the start of the cycle** 57 | * Once finding where the fast and slow pointers meet, slow will have moved s+x nodes and fast will have moved 2s+2x (s is distance to start of loop, x is some number of nodes traversed in the loop) 58 | * Move the slow pointer back to the head 59 | * Advance each one node at a time now 60 | * When they meet again, they will be at the start of the cycle 61 | * This works because the formerly fast pointer moved an additional distance s+x in the loop to arrive at position x, and advancing it by only s (s+x-x), it will be at the start of the loop 62 | * Reversing a list 63 | * Set current to head 64 | * While curr.next isn't null 65 | * Set next to curr.next 66 | * set curr.next to prev 67 | * set prev to curr 68 | * set curr to next 69 | -------------------------------------------------------------------------------- /chapter4.md: -------------------------------------------------------------------------------- 1 | ## Chapter 4: Stacks 2 | * LIFO data structure 3 | * Ordered list in which insertion and deletion occur on one end, called the top 4 | * Push onto the top 5 | * Pop off of the top 6 | * Underflow = pop off an empty stack 7 | * Overflow = push onto a full stack 8 | * Stack ADT 9 | * Push 10 | * Pop 11 | * Top (retrieve top element) 12 | * Size 13 | * IsFull/IsEmpty 14 | * Implementation 15 | * Simple array 16 | * O(1) for all operations 17 | * Maintain an index for the top of the stack 18 | * Access the top element with this index 19 | * Dynamic array 20 | * Similar to the fixed array implementation 21 | * Double the array size every time the array is full 22 | * O(1) for all operations except when the array needs to be doubled and copied, which is O(n) 23 | * Too many doublings may cause memory overflow 24 | * Linked list 25 | * Push onto stack by inserting at head of list 26 | * Pop off stack by removing from head of list 27 | * O(1) for all operations except delete stack (O(n)) 28 | * Tradeoff is the expensive doubling operation for arrays when they need to resize vs extra space needed for references 29 | * Stack to convert infix to postfix notation (basic shunting yard algorithm) 30 | * 5*(2+3-4) -> 5 2 3 4 - + * 31 | * Loop through input characters 32 | * If char is a number, output it 33 | * If char is an operator, pop off stack and output operator until empty or lower priority operator is encountered, push onto the stack 34 | * If char is ( push onto stack 35 | * If char is ) pop off stack and output until reaching ( 36 | * When reaching end of input, pop off stack and output 37 | * Stack to evaluate postfix notation 38 | * 1 2 3 + - 1 * -> (1-2+3)*1 39 | * Push numbers onto the stack 40 | * When encountering a binary operator, pop two numbers off the stack, apply the operator, push result onto stack 41 | * When encountering a unary operator, pop one number off the stack, apply the operator, push result onto stack 42 | * Shunting yard algorithm 43 | * Can be used to convert from infix to postfix 44 | * While there are tokens to be read: 45 | * Read a token 46 | * If the token is a number, push it to the output queue 47 | * If the token is a function token, push it onto the operator stack 48 | * If the token is a function argument separator (comma) 49 | * Until the token at the top of the operator stack is a left parenthesis, pop operators off the operator stack onto the output queue 50 | * If no left parentheses are encountered, either the separator was misplaced or parentheses were mismatched 51 | * If the token is an operator, op1, then: 52 | * while there is an operator token, op2, at the top of the operator stack and either: 53 | * op1 is left-associative and its precedence is <= that of op2 54 | * op1 is right-associative and its precedence is < that of op2 55 | * pop op2 off the operator stack and onto the output queue 56 | * at the end of the iteration, push op1 onto the operator stack 57 | * If the token is a left parenthesis, push it onto the operator stack 58 | * If the token is a right parenthesis: 59 | * Until the token at the top of the stack is a left parenthesis, pop operators off the operator stack onto the output queue 60 | * Pop the left parenthesis from the operator stack, but not onto the output queue 61 | * If the token at the top of the operator stack is a function token, pop it onto the output queue 62 | * If the operator stack runs out without finding a left parenthesis, then there are mismatched parentheses 63 | * When there are no more tokens to be read: 64 | * While there are still operator tokens in the stack: 65 | * If the operator token on the top of the stack is a parenthesis, then there are mismatched parentheses 66 | * Pop the operator onto the output queue 67 | * Exit 68 | -------------------------------------------------------------------------------- /chapter5.md: -------------------------------------------------------------------------------- 1 | ## Chapter 5: Queues 2 | * FIFO data structure 3 | * Ordered list in which insertion occurs at the rear and deletion occurs at the front 4 | * Enqueue at the rear 5 | * Dequeue from the front 6 | * Underflow = dequeue an empty queue 7 | * Overflow = enqueue onto a full queue 8 | * Queue ADT 9 | * Enqueue 10 | * Dequeue 11 | * Front 12 | * QueueSize 13 | * IsEmpty 14 | * Implementation 15 | * Simple circular array 16 | * Treat last element and the first array elements as contiguous 17 | * Use two variables to keep track of indexes for front and rear 18 | * Enqueuing flows around the array if at the end of the array 19 | * O(1) for all functions 20 | * O(n) for space for array 21 | * Dynamic circular array 22 | * Linked list 23 | * Use pointers for front, rear (head, tail, respectively) 24 | * O(1) for all operations 25 | -------------------------------------------------------------------------------- /chapter6.md: -------------------------------------------------------------------------------- 1 | ## Chapter 6: Trees 2 | * Hierarchical data structure 3 | * Each node points to a number of nodes, not just one 4 | * Nonlinear 5 | * Order of elements is not important 6 | * Root node has no parents 7 | * Edge is the link from parent to child 8 | * Leaf node has no children 9 | * Sibling nodes have the same parent 10 | * Ancestor/descendant 11 | * Level refers to the set of nodes at a given depth 12 | * Height of node is length of path from that node to the deepest node 13 | * Depth of node is length of path from root to node 14 | * Height of tree is length of path from root to deepest node 15 | * Skew tree is one in which every node has one child 16 | 17 | ### 6.4 Binary tree 18 | * General terminology and classification 19 | * 0, 1, 2 children 20 | * Left, right subtrees 21 | * Strict binary tree is one in which each node has two children or no children 22 | * Full binary tree is one in which each node has exactly two children and all leaf nodes are st the same level 23 | * Complete binary tree is one in which all levels, except possibly the last, are full and leaf nodes are as far left as possible 24 | * Properties of Binary Trees 25 | * 2^height = number of nodes in a level of a binary tree 26 | * 2^(height+1)-1 = number of nodes in a full binary tree for a full tree of a given height 27 | * The number of null pointers in a binary tree with n nodes is n+1 28 | * Binary Tree ADT 29 | * Data element 30 | * Left pointer 31 | * Right pointer 32 | * Insert 33 | * Delete 34 | * Find node to delete 35 | * Find deepest node (last node in level order traversal) 36 | * Replace node to delete with deepest node 37 | * Delete deepest node 38 | * Search 39 | * Traverse 40 | * Size 41 | * Height = longest path from node down to a leaf 42 | * Lowest common ancestor 43 | * Binary Tree Traversals 44 | * Each node may be visited multiple times but us only processed once 45 | * Like searching the tree except that in traversal the goal is to move through the tree in a particular order 46 | * Pre-Order 47 | * Each node is processed before subtrees 48 | * Must keep track of processed nodes that must be returned to, so stack can be used to preserve LIFO 49 | * In-Order 50 | * Root is visited between the subtrees 51 | * Use stack again to keep track of nodes that need to be processed 52 | * Post-Order 53 | * Node is visited after the subtrees 54 | * Use stack again to keep track of nodes that need to be processed 55 | * Also keep track of previous node to know where in traversal process we are 56 | * If previous is left child, we are returning from processing left subtree, so push node and process right subtree 57 | * If previous is right child, we are returning from processing right subtree, so print current node 58 | * Level Order 59 | * Visit root 60 | * While traversing level L, keep all elements at level L+1 in queue 61 | * Go to next level and visit all nodes at that level 62 | * Use null to indicate the progression between levels 63 | * Repeat until all levels are completed 64 | 65 | ### 6.5 Generic trees (N-ary trees) 66 | * Representation (first child/next sibling) 67 | * At each node, link children of same parent (siblings) from left to right 68 | * Remove the links from parent to all children except the first child 69 | * In practice this is a binary tree with different names for left and right 70 | 71 | ### 6.6 Threaded (Stack or Queue less) Binary Tree Traversals 72 | * Stack/queue representation will waste storage space due to majority null pointers 73 | * Rather than wasting pointers to null, store predecessor/successor information to avoid stack/queue 74 | * Predecessor/successor pointers are called threads 75 | * Left threaded = only use left pointers for predecessor information 76 | * Right threaded = only use right pointers for successor information 77 | * Fully threaded = left pointers for predecessors and right pointers for successors 78 | * **Successor/predecessor information is based solely on whether tree type is preorder, inorder, or postorder** 79 | * Data structure modifications 80 | * LTag and RTag to indicate whether the left or right pointer points to child (tag = 1) or successor/predecessor (tag = 0) 81 | * Always included is a dummy node which indicates which node is the root of the tree and to which used successor or predecessor pointers can point 82 | * A pointer would be on used if a node has no child and no successor or no predecessor 83 | * Finding inorder successor inorder threaded binary tree 84 | * If current node has no right subtree, return the right child (which points to the successor) 85 | * If current node does have a right subtree, go as far as possible to the left of the right child of the current node 86 | * Finding pre-order successor of inorder threaded binary tree 87 | * If current node has a left subtree, return left child 88 | * If current node does not have a left subtree, traverse right child (inorder successor) pointers until the first true right child is reached 89 | * Insertion of node in inorder threaded binary tree 90 | * Node to which to append to has no child 91 | * Node to which to append to has a child 92 | 93 | ### 6.9 Binary search trees (BSTs) 94 | * All nodes in the left subtree contain values less than the current node 95 | * All nodes in the right subtree contain values is greater than the current node 96 | * Also subtrees must also be binary search trees 97 | * O(log n) search in a balanced tree 98 | * O(n) search in a skew tree 99 | * Insert 100 | * Attempt to find the element in the tree 101 | * If not found, insert in the last location of the path traversed 102 | * **Node is not inserted between other nodes, but rather simply added as a leaf of an existing node** 103 | * Delete 104 | * Find node to be deleted 105 | * If node has no children, delete it and set parent child to null 106 | * If node has one child, delete node and set its child to the child of its parent 107 | * If node has both children, replace node with largest element in its left subtree and recursively delete that largest element 108 | * **Lowest common ancestor (LCA) is lowest node that connects two nodes in the tree** 109 | * O(n) algorithm 110 | * To find LCA of two nodes, traverse from root to bottom in preorder 111 | * Then, the first node whose value is between the two target values is the LCA 112 | * Theta(log n) algorithm (for balanced tree) 113 | * Traverse from root down 114 | * while True 115 | * if node value is lower than both targets, set node to node.right 116 | * if node value is greater than both targets, set node to node.left 117 | * otherwise, node is within targets, return node 118 | 119 | ### 6.10 Balanced BSTs 120 | * Impose restrictions on tree height to avoid O(n) complexity of skew trees 121 | * HB(k) ==> height balance, k is left height - right height 122 | 123 | ### 6.11 AVL (Adelson-Velskii and Landis) Trees 124 | * HB(k), where k = 1 125 | * BST where left and right subtree heights differ by at most 1 126 | * Minimum/Maximum number of nodes in AVL tree 127 | * Minimum heigh ~ log n 128 | * N(h) = N(h-1) + N(h-2) + 1 129 | * Number of nodes at height h = Number in subtree with height h-1 + Number in subtree with height h-2 + root 130 | * Maximum height ~ log n 131 | * AVL Tree declaration includes height parameter for simplicity of operations 132 | * Rotations 133 | * Preserve AVL property when heights differ by 2 (assuming tree was balanced every time balance was violated) 134 | * After an insertion, only nodes that are on the path from the insertion point to the root might have their balances altered 135 | * **TO RESTORE AVL TREE PROPERTY, start at insertion point and keep going to the root of the tree** 136 | * While moving to the root, need to consider the first node that does not satisfy the AVL property 137 | * **From that node onwards, every node on the path to the root will have the issue** 138 | * Always need to care for the first node that is not satisfying the AVL property on the path from the insertion point to the root and fix it 139 | * Types of Violations 140 | * Suppose X must be rebalanced and it is the first node whose subtrees differ by height 2 141 | * Violation may have occurred in one of 4 ways 142 | 1. Insertion into left subtree of left child of X (solve with LL rotation) 143 | 2. Insertion into right subtree of left child of X (solve with LR rotation) 144 | 3. Insertion into left subtree of right child of X (solve with RL rotation) 145 | 4. Insertion into right subtree of right child of X (solve with RR rotation) 146 | * Single Rotations 147 | * LL Rotation 148 | * X is node to be rebalanced 149 | * create new node W 150 | * set X’s R child to its L child’s R child 151 | * set X as the R child of its L child 152 | * update heights 153 | * return pointer to W for X’s former parent to point to 154 | * RR Rotation 155 | * mirror of LL 156 | * Double Rotations 157 | * Perform two single rotations (one left and one right) 158 | * Insertion into AVL Tree 159 | * Insert as with any BST 160 | * Check for imbalance 161 | * Rotate if necessary 162 | * AVL Sort 163 | * insert n items in AVL tree = O(n log n) 164 | * inorder traversal to output = O(n) 165 | -------------------------------------------------------------------------------- /chapter7.md: -------------------------------------------------------------------------------- 1 | ## Chapter 7: Priority Queues and Heaps 2 | 3 | ### Priority Queue 4 | * Queue data structure in which dequeuing order is based on element value 5 | * Ascending priority queue always dequeues smallest element (smallest has highest priority) 6 | * Descending priority queue always dequeues largest element (largest has highest priority) 7 | * ADT 8 | * Insert (enqueue) 9 | * DeleteMin 10 | * DeleteMax 11 | * GetMin/GetMax 12 | * Can be represented with list, array, tree data structures but ideal is min/max heap 13 | 14 | ### Heap 15 | * O(log n) insert/delete and O(1) get min/max 16 | * Heap is a tree data structure with special properties 17 | * Heap property = each node is >= its children (for max heap) or <= its children (for min heap) 18 | * MUST be a complete tree (nodes are at levels h or h-1) 19 | * Heap ADT 20 | * array (elements in heap) 21 | * Count (number of elements in heap) 22 | * Capacity (of heap/array) 23 | * Heap type (min/max) 24 | * GetParent 25 | * For a node at location i, its parent is at (i-1)/2 26 | * GetLeft/GetRight 27 | * For a node at location i, its children are at 2i+1 and 2i+2 28 | * GetMax/GetMin = first element in the array, root of tree 29 | * Heapifying 30 | * After inserting an element, the heap property may be violated 31 | * To identify, check from root down to inserted element whether heap property has been violated 32 | * If it has been, swap node with the greater of its children 33 | * Repeat this process down until heap property is not violated 34 | * This is PercolateDown 35 | * Can also PercolateUp from a non root node up to the root 36 | * O(log n) because it is based on the height of the nearly balanced tree 37 | * Delete element 38 | * Only root is deleted because of heap 39 | * Store root in temp var to return 40 | * Replace last element with root 41 | * Delete last element 42 | * Percolate new root down 43 | * Decrease heap size 44 | * Return temp var 45 | * O(log n) 46 | * Insert element 47 | * Add to end of heap 48 | * Percolate up 49 | * Increase heap size 50 | * O(log n) 51 | * Heapsort 52 | * Add unsorted array to heap 53 | * Build max heap (heapify) O(n) 54 | * Swap root with last element of heap O(1) 55 | * Decrease heap size 56 | * Heapify root O(log n) 57 | * Repeat until done with all elements 58 | * Array is sorted in place by performing n heapify operations plus build heap from unsorted array = O(n) + O(n log n) ~ O(n log n) 59 | * Build heap from unsorted array 60 | * for (n-1)/2 down to 0: heapify node 61 | * No need to heapify beyond element (n-1)/2 because those have no children and are already heaps 62 | * O(n) 63 | * Only root node performs log n operations 64 | * Nodes one level above leaves perform O(1) operation, and there are n/4 of them 65 | * The whole series is then n times a series which converges to a constant so it is linear (sum of inverse powers of 2) 66 | -------------------------------------------------------------------------------- /chapter8.md: -------------------------------------------------------------------------------- 1 | SKIPPED 2 | -------------------------------------------------------------------------------- /chapter9.md: -------------------------------------------------------------------------------- 1 | ## Chapter 9: Graph Algorithms 2 | * A graph is a pair (V, E), where V is a set of vertices and E is a collection of pairs of vertices called edges 3 | * Directed edge ==> ordered pair of vertices 4 | * (u, v) - origin vertex u, destination vertex v 5 | * Undirected edge ==> unordered pair of vertices 6 | * Directed graph is a graph in which all edges are directed 7 | * Undirected graph is a graph in which all edges are undirected 8 | * **Adjacent** vertices are connected by an edge, which **is incident on** both vertices 9 | * Graph with no cycles is a tree (acyclic connected graph) 10 | * Two edges are parallel if they connect the same pair of vertices 11 | * **Degree** of a vertex is the number of edges incident on it 12 | * Subgraph is a subset of a graph’s edges that form a graph 13 | * **Path* in a graph is a sequence of adjacent vertices 14 | * Simple path is a path with no repeated vertices 15 | * Cycle is a path in which the first and last vertices are the same 16 | * A vertex is connected to another if there is a path that contains both of them 17 | * A graph is connected if there is a path from every vertex to every other vertex 18 | * **Directed Acyclic Graph** (DAG) is a directed graph with no cycles 19 | * **Weighted graphs** have weights assigned to each edge 20 | * Complete graph has all edges present 21 | * Sparse graph has relatively few edges (fewer than V log V) 22 | * |V| is the number of vertices 23 | * |E| is the number of edges (ranges from 0 to V * ( V + 1 ) / 2 ) 24 | 25 | ### Graph Representation 26 | * Store vertices as an array of vertices 27 | * Adjacency Matrix 28 | * Matrix of boolean values or weights to show whether two vertices are connected by an edge 29 | * Adj[u,v] = weight ==> u and v are connected by an edge 30 | * In a directed graph, Adj[u,v] = weight ==> there is an edge from u to v 31 | * An undirected graph only needs half of the matrix and all self edges are set to Weight 32 | * O(V^2) space and time to initialize 33 | * Adjacency List 34 | * **Array Adj of vertices, where each element is a pointer to a linked list that contains the neighbors of that element** 35 | * For OOP, you can use v.neighbors = Adj[v] 36 | * Linked list for each node that lists the different nodes that can be visited from the current node 37 | * V total linked lists 38 | * Order of adding edges is important because it will affect the order in which all processes process edges 39 | * Can be inefficient for deletes because the vertex must be deleted from the vertices list AND from all adjacency lists 40 | * O(E+V) space 41 | * Implicit representation 42 | * Use Adj(u) as a function to get the adjacency list for vertex u 43 | * Use v.neighbors() as a method to get the adjacency list for vertex v 44 | * No need to get or generate entire graph 45 | * Just keep getting neighbors until you find desired state 46 | * Good for representing graphs with many many states 47 | 48 | ### 9.5 Graph Traversals/Searches 49 | * Start from source (similar to root of Tree) 50 | * Depth First Search (DFS) 51 | * Breadth First Search (BFS) 52 | 53 | ### Depth First Search (DFS) 54 | * Similar to preorder tree traversal 55 | * Edge types 56 | * Tree edge = visit new vertex 57 | * Forward edge = from ancestor to descendent in the forest of trees along DFS visit path (does not exist in undirected graphs) 58 | * Backward edge = from descendent to ancestor in the forest of trees along DFS visit path 59 | * Cross edge = between a tree or subtrees that are not ancestor related (does not exist in undirected graphs) 60 | * For most problems, boolean classification (unvisited/visited) is sufficient, but some require three colors 61 | * Use a stack to keep track of previously visited indexes 62 | * General DFS concept 63 | * Start at vertex u in the graph 64 | * Mark vertices as visited when visited (as part of their data structure) 65 | * Consider the edges from u to all other vertices 66 | * If the edge leads to an already visited vertex, then backtrack to u 67 | * If it leads to an unvisited vertex, go to that vertex and that becomes current vertex (previous current is pushed to stack) 68 | * Repeat until reaching a dead end at u 69 | * Backtrack if u current vertex is unable to make progress (pop from stack) 70 | * Process terminates when backtracking leads back to the start vertex 71 | * DFS traversal forms a tree (no back edges) = called DFS tree 72 | * O(V+E) time complexity with adjacency lists 73 | * O(V^2) for adjacency matrix 74 | 75 | ## DFS Part 2 (OCW lecture notes) 76 | * DFS Forest consists of DFS trees and the tree edges in those trees 77 | * DFS trees consist of edges included in the DFS 78 | * DFS tree edges are the set of edges from parent u to vertex v, where the parent is not nil 79 | * **Directed graph is acyclic if it has no back edges** 80 | * Recursively explore graph, backtracking as necessary 81 | * Be careful to not repeat vertices 82 | * Vertices are either 83 | * White - undiscovered 84 | * Gray - discovered and may have undiscovered adjacent vertices 85 | * Black - finished 86 | * Vertices have two timestamps (or ticks/counters) 87 | * v.d = discovery time (or tick) 88 | * v.f = finish time (or tick), when DFS finishes v’s adjacency list and blackens v 89 | * Parenthesis Theorem 90 | * In a DFS, for any two vertices u and v, exactly one of the following conditions holds 91 | * [u.d, u.f] and [v.d, v.f] are entirely disjoint and neither u nor v is a descendant of the other in the depth-first forest 92 | * [u.d, u.f] is contained entirely within [v.d, v.f] and u is a descendant of v in a depth-first tree 93 | * [v.d, v.f] is contained entirely within [u.d, u.f] and v is a descendant of u in a depth-first tree 94 | * White-path theorem 95 | * In a depth-first forest, vertex v is a descendent of vertex u IIF at the time u.d that DFS discovers u, there is a path from u to v consisting entirely of white vertices 96 | ``` 97 | parent = {s:None} 98 | DFS-Visit(Adj, s): 99 | for v in Adj[s]: 100 | if v not in parent: 101 | parent[v] = s 102 | DFS-Visit(Adj, v) 103 | 104 | # DFS visits and tracks all vertices in the graph 105 | # V is the set of vertices in the graph 106 | DFS(V, Adj): 107 | parent = {} 108 | for s in V: 109 | if s not in parent: 110 | parent[s] = None 111 | DFS-Visit(Adj, s) 112 | ``` 113 | * G has a cycle <==> DFS has a back edge 114 | * Used for topological sort 115 | * Run DFS 116 | * Output reverse of finishing times/sequences of vertices 117 | * This works because there are no back edges (because graph is acyclic) 118 | 119 | ### Breadth First Search (BFS) 120 | * **Used to find shortest paths** 121 | * Similar to level order tree traversal 122 | * Vertices are either 123 | * White - undiscovered 124 | * Gray - discovered and may have undiscovered adjacent vertices 125 | * Black - discovered and has all adjacent vertices discovered 126 | * Construct breadth first tree starting at root s 127 | * Whenever a white vertex is discovered, it is added to the tree along with the edge 128 | * Uses queue to store vertices at subsequent levels 129 | * General BFS Concept 130 | * Starts at a given vertex, which is level 0 131 | * Mark vertices as visited when visited (as part of their data structure) 132 | * Enqueue all vertices 1 level away 133 | * Visit all vertices at level 1 (1 step away from start point) 134 | * Then visit all vertices at level 2 135 | * Repeat until all levels of the graph are completed 136 | * O(V+E) time complexity with adjacency lists 137 | * O(V^2) for adjacency matrix 138 | ``` 139 | # python BFS 140 | BFS(s, Adj): 141 | level = {s:0} 142 | parent = {s:None} 143 | i = 1 144 | frontier = [s] # everything reachable in i-1 moves 145 | while frontier: 146 | next = [] # everything reachable in i moves 147 | for u in frontier: 148 | for v in Adj[u]: 149 | if v not in level: 150 | level[v] = i 151 | parent[v] = u 152 | next.append(u) 153 | frontier = next 154 | i += 1 155 | ``` 156 | * parent points back to s (source) and form shortest path back to s from any given vertex 157 | 158 | ### DFS vs BFS 159 | * DFS is more memory efficient (does not require storage of child pointers on each level) 160 | * Usage depends on whether it is important to get to the bottom of the tree or whether the desired data is near the top of the tree 161 | * BFS is better for shortest paths 162 | 163 | ### 9.6 Topological Sort 164 | * Topological sort is an ordering of vertices in a DAG in which each node comes before all nodes to which it has outgoing edges 165 | * An example is course prerequisites in a major 166 | * If all pairs of consecutive vertices in the sorted order are connected by edges, then they form a directed Hamiltonian path, and the sort order is unique 167 | * Otherwise, if the Hamiltonian path does not exist, the DAG can have 2+ topological sort orderings 168 | * General Topological Sort Algorithm 169 | * Calculate indegree for all vertices (number of vertices leading into it) and store locally for each vertex 170 | * Enqueue all vertices of indegree 0 171 | * While queue is not empty 172 | * Dequeue vertex v as next vertex in sort order 173 | * Decrement all indegrees of edges adjacent to v 174 | * Enqueue a vertex as soon as its indegree falls to 0 175 | * O(V+E) time complexity with adjacency lists 176 | 177 | ### 9.7 Shortest Path Algorithms 178 | * GOAL: ∂(u, v) = { min( weighted_path from u to v if there exists any such path ), INF otherwise } 179 | * Maintain two data structures 180 | * d(v) = current total path weight to v from source 181 | * ∏(v) = predecessor on current best path to v from source 182 | * General structure, assuming no negative cycles 183 | * **Initialize for all u in V:** 184 | * **d[v] = INF** 185 | * **∏[u] = nil** 186 | * **d[s] = 0** 187 | * Repeat until all E have d[v] <= d[u] + w(u,v) 188 | * select edge (u,v) … somehow 189 | * if d[v] > d[u] + w(u,v) 190 | * d[v] = d[u] + w(u,v) 191 | * ∏[v] = u 192 | * Utilizes the notion of relaxation = testing an edge to see if we can improve the current shortest path estimate 193 | * Relax an edge when it is used to update the distance to a vertex 194 | * Parent is then updated 195 | * Shortest path algorithms only differ in how many times they relax edges and the order in which they relax edges 196 | * Dijkstra relaxes each edge only one time 197 | * Bellman-Ford relaxes each edge 198 | * **Optimal substructure = subpaths of a shortest path are shortest paths** 199 | * Triangle inequality 200 | * ∂(s, v) <= ∂(s, u) + w(u, v) 201 | 202 | ### Further Shortest Path Notes 203 | * Unweighted graph, weighted graph, weighted graph with negative edges 204 | * Unweighted shortest path 205 | * BFS 206 | * Newly discovered vertices have a distance of their parents distance plus one 207 | * Set their parent to their parents node 208 | 209 | ### Weighted DAG Shortest Path (non-negative edges) 210 | * Topological sort the DAG 211 | * Path from u to v implies that u is before v in ordering 212 | * One pass over vertices in topologically sorted order 213 | * Relax each edge that leaves each vertex 214 | * O(V+E) 215 | 216 | ### Dijkstra (non-negative, weighted graph) 217 | * Complexity 218 | * Theta(V^2 + E) with array for queue 219 | * Theta(V log V + E log V) for min heap 220 | * O(log V) for extract min (and update heap) 221 | * O(log V) for decrease key operation 222 | * Theta(v log V + E) for Fibonacci heap 223 | * O(log V) for extract min (and update heap) 224 | * O(1) for decrease key operation 225 | * Generalization of breadth first search 226 | * BFS cannot guarantee that the vertex at the front of the queue is the closest to the source vertex in terms of weighted distance 227 | * Uses greedy algorithm: always picks the next closest vertex to the source 228 | * Uses a priority queue to store on visited vertices by distance from source (instead of a regular queue in regular BFS) 229 | * **As new vertices are discovered they are added to the priority queue** 230 | * Does not work on graphs with negative edges 231 | * Distance for each vertex is stored as the total weighted distance from the source 232 | * A distance can be updated if a new shorter distance is found 233 | * The element is updated in the priority queue with this new distance 234 | * Disadvantages are doing a blind search which wastes resources and not being able to handle negative edges 235 | ``` 236 | Dijkstra(G, w, s): 237 | Initialize(G, s) 238 | S = Ø // set of vertices whose shortest path from s has been found 239 | Q = G.V // set of vertices whose shortest path from s remains yet to be found 240 | // priority queue, prioritized by v.d = total distance from s thus far 241 | while Q != Ø: 242 | u = extract-min(Q) 243 | S = S U {u} // add u to S 244 | for v in Adj(u): 245 | Relax(u, v, w) 246 | 247 | Initialize(G, s): 248 | for v in G: 249 | v.π = nil 250 | v.d = INF 251 | s.d = 0 252 | 253 | Relax(u, v, w): 254 | if v.d > u.d + w(u,v): 255 | v.d = u.d + w(u,v) 256 | v.π = u 257 | ``` 258 | 259 | ### Bellman-Ford 260 | * Bellman-Ford Intuition 261 | * With each pass from 1 up to V-1, you're establishing ∂ for the nodes each level further away from s 262 | * Thus, each pass creates progressively more optimal subpaths until they cannot be optimized further 263 | * After V-1 passes, all ∂ will have been found 264 | * Nodes unreachable due to negative weight cycles will have ∂(s, v) = undef 265 | * Nodes unreachable otherwise will have ∂(s, v) = INF 266 | * Initialize regular queue with s 267 | * Initialize hash table to indicate which vertices are in the queue 268 | * At each iteration, dequeue v 269 | * Find all vertices w adjacent to v such that: dist[v] + weight(v,w) < old dist[w] 270 | * Then update old distance and path for w and enqueue if not already there 271 | * Repeat until queue is empty 272 | * O(E*V) with adjacency lists 273 | * Disadvantage is that it does not work if there are negative-cost cycles 274 | ```` 275 | Bellman-Ford(G, w, s): 276 | Initialize(G, s) 277 | for i=1 to size(V)-1: 278 | for each edge (u, v): 279 | Relax(u, v, w) 280 | 281 | // check for negative weight cycle 282 | for each edge (u, v): 283 | if v.d > u.d + w(u, v): 284 | Report negative weight cycle exists 285 | ``` 286 | 287 | ### Bidirectional Search 288 | * Source s, Target t 289 | * Run Dijkstra, alternating forwards from s and backwards from t, when they meet the algorithm will be complete 290 | * Maintain priority queues for Q_forward, Q_backward 291 | * When an element has been extracted from both, the frontiers will have met and the search can be terminated 292 | * **∂(s, t) = min(d_forward[x] + d_backward[x])** 293 | * x may or may not be the same vertex that terminates the search 294 | 295 | ### 9.8 Minimum Spanning Tree (MST) Algorithms 296 | * Spanning Tree = subgraph (that is a tree) that contains all vertices in a graph AND has minimum total weight 297 | * Prim’s Algorithm 298 | * Almost exact same as Dijkstra 299 | * Relax function is all that differs: 300 | ``` 301 | PrimRelax(u, v, w): 302 | if v.d > w(u, v): 303 | v.d = w(u, v) 304 | ``` 305 | * Kruskal’s Algorithm 306 | --------------------------------------------------------------------------------