Hi.
7 | 8 |This is a place to find information about some of the more fundamental algorithms used in computer science. This information is widely available on the net, but hopefully the way it's presented and discussed here will resonate with you.
9 | 10 |Most of these are things you wouldn't need to write yourself. Modern libraries and languages tend to have quality implementations for all of this. Nonetheless, I truly believe that understanding how things work is key to improving how we work.
11 | 12 |The source code for this site is available on github.
13 | 14 |Enjoy.
15 | 16 | -------------------------------------------------------------------------------- /_layouts/default.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 |Where a bubble sort relies on a number of small swaps, insertion sort relies on inserting a single element in the right for a given iteration. Every iteration through the collection leaves a greater segment sorted.
11 | 12 |
14 | function sort(values) {
15 | var length = values.length;
16 | for(var i = 1; i < length; ++i) {
17 | var temp = values[i];
18 | var j = i - 1;
19 | for(; j >= 0 && values[j] > temp; --j) {
20 | values[j+1] = values[j];
21 | }
22 | values[j+1] = temp;
23 | }
24 | };
25 | sort([7, 4, 5, 2, 9, 1]);
26 | //finished
27 |
28 |
29 | Click step to sort the array.
31 |Like bubble sorts, insertion sorts is efficient for an already sorted or nearly sorted collection. Insertion sort will always be at least as efficient as a bubble sort.
45 | 46 |Insertion sort is a good choice for small or mostly sorted collections. It performs well, has little memory overhead and is simple to understand and implement.
48 | 49 | 53 | 54 | 55 | -------------------------------------------------------------------------------- /public/css/main.css: -------------------------------------------------------------------------------- 1 | html{font-family:verdana;} 2 | h1{font-size:1.2em;margin:3px 0;} 3 | h2{font-size:1em;margin:0;} 4 | ul{list-style:none;margin:0;padding:0;} 5 | li{margin:0;padding:0;} 6 | a{text-decoration:none;} 7 | code{background:#f6f6f6;border:1px solid #eee;padding:2px 5px;} 8 | pre{margin:5px;margin-bottom:20px;} 9 | #head{height:40px;position:relative;} 10 | #left{width:220px;padding:20px;float:left;} 11 | #left p{margin-bottom:20px;} 12 | #left a{padding:3px;color:#004A7F;border:1px solid transparent;background:#f0f0f0;} 13 | #left a:hover{background:#e6e6e6;border:1px solid #ccc;} 14 | #content{margin-left:300px;padding:20px;} 15 | #content p{line-height:1.4em;margin-top:0;} 16 | #passion {position:absolute;top:20px;right:20px;font-weight:bold;font-style:italic;color:#bbb;font-size:0.75em;} 17 | 18 | .example{user-select: none; -o-user-select:none; -moz-user-select: none; -khtml-user-select: none; -webkit-user-select: none;position:relative;} 19 | .example .top, .example .bottom{height:30px;overflow:hidden;font-size:80%;} 20 | .example .top div, .example .nodeList div, .example .bottom div{white-space: nowrap;float:left;margin-right:10px;padding:5px;text-align:center;border:1px solid transparent;} 21 | .example .top div{width:50px;} 22 | .example .bottom div, .example .nodeList div{min-width:50px;} 23 | .example .top .wide, .example .bottom .wide{margin-right:110px;} 24 | .example .bottom .bad{background: #fdd;} 25 | .example .bottom .good{background: #ddf;} 26 | .example .reference{position:absolute;top:62px;} 27 | .example .reference_200{width:200px;height:30px;background:url('/images/reference_200.png') no-repeat;} 28 | .example .reference_400{width:450px;height:30px;background:url('/images/reference_400.png') no-repeat;} 29 | .nodeList{margin-bottom:20px;height:20px;} 30 | .nodeList div{border:1px solid #666;background:#222;color:#ddd;height:20px;} 31 | .nodeList div.highlight{border:1px solid #eeb;background:#ffc;color:#222;} 32 | .nodeList div.used{background:url('/images/used.png') repeat-x;width:150px;color:#fff;} 33 | .step{position:absolute;right:10px;top:10px;padding:5px 10px;cursor:pointer;background:-webkit-gradient(linear,left top,left bottom,from(#eee),to(#ccc));background:-moz-linear-gradient(#eee,#ccc);background-color:#c0c2c0;text-decoration:none;color:#444;font-weight:bold;} 34 | .step:hover{color:#111;} 35 | .clear{overflow:hidden;} 36 | .hide{display:none;} 37 | 38 | #nav{margin-top:50px;height:50px;} 39 | #nav a{color:#5aa;} 40 | #prev{float:left;} 41 | #next{float:right;} 42 | 43 | pre .keyword{color:#069;} 44 | pre .control{color:#00f;} 45 | pre .string{color:#833;} 46 | pre .comment{color:#383;} 47 | pre span.highlight{background:#ffc;padding:3px 0px;} 48 | .r{-moz-border-radius:4px;-webkit-border-radius:4px;border-radius:4px;} 49 | -------------------------------------------------------------------------------- /_posts/2012-5-15-quicksort.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: post 3 | title: "Quick Sort" 4 | permalink: "sort/quicksort.html" 5 | disqus_id: "/algo/quicksort" 6 | --- 7 | 8 |Quick sort is a efficient sorting algorithm which works by dividing a collection into smaller and smaller partitions which become increasingly sorted. The approach is to take a single element (called the pivot) and make it so that all values before the pivot are smaller, and all elements after the pivot are larger. This places the pivot at its final position. The step is repeated for the two parts of the collection (before the pivot and after), and then again for the 4 quarters (based on the 2 pivots sorted in the previous iteration), and so on until the collection is sorted.
10 | 11 |
13 | function partition(values, start, end) {
14 | var value = values[start] ;
15 | swap(values, start, end);
16 | for (var i = start; i < end; ++i) {
17 | if (values[i] <= value) { swap(values, start++, i); }
18 | }
19 | swap(values, end, start);
20 | return start;
21 | };
22 | function swap(values, i, j) {
23 | if (i == j) { return; }
24 | var temp = values[i]; values[i] = values[j]; values[j] = temp;
25 | };
26 |
27 | function quickSort(values, start, end){
28 | if (start < end ) {
29 | var pivot = partition(values, start, end);
30 | quickSort(values, start, pivot);
31 | quickSort(values, pivot+1, end);
32 | }
33 | };
34 | function sort(values) {
35 | quickSort(values, 0, values.length-1);
36 | };
37 | sort([7, 4, 5, 2, 9, 1]);
38 | //finished
39 |
40 |
41 | Click step to sort the array.
43 |Bubble sort is the most basic way to sort a collection. It works by sequentially going through your array and comparing two values at a time, swapping them if necessary. It then repeats the process until no swaps are required. 10 | 11 |
13 | function sort(values) {
14 | var length = values.length - 1;
15 | do {
16 | var swapped = false;
17 | for(var i = 0; i < length; ++i) {
18 | if (values[i] > values[i+1]) {
19 | var temp = values[i];
20 | values[i] = values[i+1];
21 | values[i+1] = temp;
22 | swapped = true;
23 | }
24 | }
25 | }
26 | while(swapped == true)
27 | };
28 | sort([7, 4, 5, 2, 9, 1]);
29 | //finished
30 |
31 |
32 | Click step to sort the array. (This is a very slow algorithm, you probably want to give up once you have a general feel for it.)
34 |As you've no doubt seen, bubble sort requires a lot of steps in order to sort even a short collection. In fact, sorting this collection required almost as many steps as all of the steps from the previous sections combined. It takes a lot of iterations and comparison to small values to their final destination.
48 | 49 |There are two interesting things to know about bubble sorts. First, large values are always sorted first. If you run through it again (if you are insane), you'll notice that the 9 then the 7 then the 5 and so on get sorted first. Knowing this, we could make the code slightly smarter and stop repeatedly iterating over already sorted values.
50 | 51 |The other interesting thing about a bubble sort is that it only takes one iteration to detect that a collection is already sorted. That is, if the first iteration doesn't set swapped = true, no additional iterations are required. However, this is not a unique property of bubble sorts.
There is no scenario in which a bubble sort is the most efficient way to sort (and more often than not it's much less efficient). In fact, many very influential people feel that it shouldn't be taught given how useless (and horrible) an algorithm it is.
55 | 56 |Personally, I think it's still worth understanding.
57 | 61 | 62 | -------------------------------------------------------------------------------- /_posts/2012-5-15-binarysearch.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: post 3 | title: "Binary Search" 4 | permalink: "search/binarysearch.html" 5 | disqus_id: "/algo/binarysearch" 6 | --- 7 | 8 |Binary search relies on a divide and conquer strategy to find a value within an already-sorted collection. The algorithm is deceptively simple. Pretend I was thinking of a number between 1 and 100. Every guess you take, I'll say higher or lower. The most efficient way to discover my number is to first guess 50. Higher. 75. Lower. 62. Higher 68. Yes!
10 | 11 |function findIndex(values, target) {
13 | return binarySearch(values, target, 0, values.length - 1);
14 | };
15 |
16 | function binarySearch(values, target, start, end) {
17 | if (start > end) { return -1; } //does not exist
18 |
19 | var middle = Math.floor((start + end) / 2);
20 | var value = values[middle];
21 |
22 | if (value > target) { return binarySearch(values, target, start, middle-1); }
23 | if (value < target) { return binarySearch(values, target, middle+1, end); }
24 | return middle; //found!
25 | }
26 | findIndex([1, 4, 6, 7, 12, 13, 15, 18, 19, 20, 22, 24], 20);
27 | //finished
28 |
29 |
30 | Click step to find 20 within our sorted array
Every iteration eliminates half of the remaining possibilities. This makes binary searches very efficient - even for large collections. Our implementation relies on recursion, though it is equally as common to see an iterative approach.
52 | 53 |Binary search requires a sorted collection. This means the collection must either be sorted before searching, or inserts/updates must be smart. Also, binary searching can only be applied to a collection that allows random access (indexing).
54 | 55 |Binary searching is frequently used thanks to its performance characteristics over large collections. The only time binary searching doesn't make sense is when the collection is being frequently updated (relative to searches), since re-sorting will be required.
57 | 58 |Hash tables can often provide better (though somewhat unreliable) performance. Typically, it's relatively clear when data belongs in a hash table (for frequent lookups) versus when a search is needed.
59 | 60 | 64 | 65 | 66 | -------------------------------------------------------------------------------- /_posts/2012-5-16-LinearSearch.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: post 3 | title: "Linear Search" 4 | permalink: "search/linear.html" 5 | disqus_id: "/algo/linearsearch" 6 | --- 7 | 8 |A linear search is the most basic of search algorithm you can have. A linear search sequentially moves through your collection (or data structure) looking for a matching value.
10 | 11 |
13 | function findIndex(values, target) {
14 | for(var i = 0; i < values.length; ++i){
15 | if (values[i] == target) { return i; }
16 | }
17 | return -1;
18 | }
19 | findIndex([7, 3, 6, 1, 0], 6)
20 |
21 |
22 | Click step to step through the above implementation and find 6 within the following list
24 |The worst case performance scenario for a linear search is that it needs to loop through the entire collection; either because the item is the last one, or because the item isn't found. In other words, if you have N items in your collection, the worst case scenario to find an item is N iterations. This is known as O(N) using the Big O Notation. The speed of search grows linearly with the number of items within your collection.
31 | 32 |Linear searches don't require the collection to be sorted.
33 | 34 |In some cases, you'll know ahead of time that some items will be disproportionally searched for. In such situations, frequently requested items can be kept at the start of the collection. This can result in exceptional performance, regardless of size, for these frequently requested items.
35 | 36 |Despite its less than stellar performance, linear searching is extremely common. Many of the built-in methods that you are familiar with, like ruby's find_index, or much of jQuery, rely on linear searches. When you are dealing with a relatively small set of data, it's often good enough (and for really small unordered data is can even be faster than alternatives).
Beyond this though, the general concept of sequential/linear access is something that is often overlooked. The more abstract libraries get, the more risk you run of unknowingly doing something linearly. .NET's LINQ is a great example. Most of LINQ works against IEnumerable which only exposes a forward moving enumerator. So what do you think happens when you call the Count() method? Thankfully, LINQ is smart and, if possible, it'll rely on a fast Count or Length property. However, if the actual implementation doesn't have those, it'll loop through the enumerator.
That doesn't make LINQ's Any, or Ruby's include? "evil". It's just good to know what these high level methods might be doing (and often, what they are doing, is a linear search).
42 | 43 |
47 | 48 | -------------------------------------------------------------------------------- /_posts/2012-5-16-Arrays.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: post 3 | title: "Arrays and Dynamic Arrays" 4 | permalink: "structures/arrays.html" 5 | disqus_id: "/algo/arrays" 6 | --- 7 | 8 |In the traditional definition of an array, the key concept is that elements of an array occupy a contiguous block of memory. This has a couple important consequences. First, once created, an array cannot grow (because the adjacent memory might already be taken). Secondly, arrays can be randomly accessed. When you use square brackets to access an array element [X]), you are actually saying move forward within the memory from the start of the allocation by X. That explains why indexes start at 0, because the first item is retrieved by moving forward by 0.
Many languages have, what they call, "arrays" that can grow. Technically, these are dynamic arrays. Java and C# have ArrayLists which are also dynamic arrays. A dynamic array wraps a real array and allows it to grow. How? Once the array fills up, a new, larger, memory location is found and the original is copied to the new space.
16 | function ArrayList(initialLength) {
17 | this.length = 0;
18 | this.array = new Array(initialLength);
19 |
20 | this.add = function(value) {
21 | if (this.length == this.array.length){
22 | this.grow();
23 | }
24 | this.array[this.length++] = value;
25 | };
26 | this.grow = function() {
27 | var original = this.array;
28 | this.array = new Array(this.length * 2);
29 | for(var i = 0; i < this.length; ++i) {
30 | this.array[i] = original[i];
31 | };
32 | }
33 | }
34 | var array = new ArrayList(1);
35 | array.add(2);
36 | array.add(9);
37 | array.add(4);
38 |
39 |
40 | Click step to add the values to our dynamic array
42 |When an insert occurs in a filled dynamic array, the growth algorithm must be executed. This means that inserts provide inconsistent performance as well as non-linear growth. The implementation of the growth algorithm obviously has a great impact. Like our implementation, many simply double the size; however, most are able to provide more efficient copying than what we've done.
59 | 60 | 61 |Strings are the most common example of fixed arrays we use. They truly should be considered arrays of characters. And, like all arrays, their size is set when initialized. This is why many people warn that string are immutable and that appending multiple times causes poor performance. Like a dynamic array, a string must be reallocated to a larger space whenever values are appended. However, unlike a dynamic array this is handled at compile time and no buffering is used. In other words, appending a value to a string will allocate just enough space for the new combined value. This is where StringBuilder classes come into play. A StringBuilder is nothing more than a dynamic arrays for strings with some extra buffer space. When it fills up, it too must be copied to a new, larger, memory block.
The most important thing to do, when dealing with a dynamic array which will see many inserts, is to specify an adequate initial size. Many implementations start with a small default, say 20. This means that if you are inserting 10 000 values, it'll have to grow 9 times. It'll also fragment your memory (leaving 8 empty holes), causing additional compaction.
65 | 66 |Some languages, such as Ruby rely solely on dynamic arrays. Others, like Java and C# expose both types. In this day and age, it's hard to come up with real-world use cases for absolutely requiring fixed arrays.
67 | 68 | 72 | 73 | 74 | -------------------------------------------------------------------------------- /_posts/2012-5-15-linkedlists.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: post 3 | title: "Linked List" 4 | permalink: "structures/linkedlists.html" 5 | disqus_id: "/algo/linkedlists" 6 | --- 7 | 8 |Where the behavior of arrays is largely defined by using contiguous blocks of memory, link lists are defined by the opposite: their ability to use non-contiguous memory. How do linked lists represent a cohesive collection of items then? In its simplest implementation, each node of a linked lists is a combination of the value and a reference to the next item in the list. Therefore, instead of relying on absolute position from a starting point, as array do, linked lists require each node to know the location of its sibling.
10 | 11 | function LinkedList() {
13 | this.head = null;
14 | this.tail = null;
15 |
16 | this.add = function(value) {
17 | var node = new Node(value);
18 | if (this.head == null) { this.head = node; }
19 | if (this.tail != null) { this.tail.next = node; }
20 | this.tail = node;
21 | };
22 | }
23 |
24 | function Node(value) {
25 | this.value = value;
26 | this.next = null;
27 | }
28 |
29 | var list = new LinkedList();
30 | list.add(1);
31 | list.add(2);
32 | list.add(3);
33 |
34 |
35 | Click step to add the values to our linked list
37 |Manipulating the linked lists comes down to updating the affected head, tail and next references.
this.removeAt = function(index) {
53 | var prev = null;
54 | var node = this.head;
55 | var i = 0;
56 | while (node != null && i++ < index) {
57 | prev = node;
58 | node = node.next;
59 | }
60 | if (prev == null) { this.head = node.next;}
61 | else { prev.next = node.next; }
62 | };
63 | list.removeAt(1);
64 |
65 |
66 | Click step to remove the value from our linked list
68 |In a world where memory isn't measured in gigabytes, linked lists have many advantages. Namely, they allow a structure to grow with minimal impact on available memory. Linked list can grow even within fragmented memory, and do so while providing constantly good insert performance. However, one does not have random access to individual nodes, resulting in constantly poor read performance. The overhead of a simple linked lists is essentially one reference per value.
81 | 82 |It is also common for each node to have a reference to the previous node. This is known as a doubly-linked list. This doubles the memory overhead as well as making manipulation more complicated (two references must be maintained rather than just one). However, it does make insertion and deletions within the list simpler.
83 | 84 |Link lists' memory friendliness isn't the great benefit it once was. Dynamic arrays, with spare space to grow, are generally a better fit for today's programs. However, linked lists are still a solid implementation for collections which provide sequential access, like queues and stacks.
86 | 87 | 91 | 92 | 93 | -------------------------------------------------------------------------------- /_posts/2012-5-15-hashtables.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: post 3 | title: "HashTables" 4 | permalink: "structures/hashtables.html" 5 | disqus_id: "/algo/hashtables" 6 | --- 7 | 8 |A hash table is a more advanced data structure which typically makes use of one or more other data structure. The general idea is to store the value within a bucket based on the hashing of some key. The simplest example stores values within an array. By applying the hashing algorithm to our key, we get the index to store our value in. Generally speaking, hash tables are used to store a key=>value pair, though in our examples the key will be the value - to keep things straightforward.
10 | 11 |
13 | function HashTable(size) {
14 | this.size = size;
15 | this.buckets = new Array(size);
16 |
17 | this.add = function(value) {
18 | var index = this.hash(value);
19 | this.buckets[index] = value;
20 | };
21 | this.hash = function(value) {
22 | var sum = 0;
23 | for (var i = 0; i < value.length; ++i) {
24 | sum += value[i].charCodeAt() - 97;
25 | }
26 | return sum % this.size;
27 | };
28 | }
29 |
30 | var hash = new HashTable(3);
31 | hash.add('fear');
32 | hash.add('is the');
33 | hash.add('little death');
34 |
35 |
36 | Click step to add the values to our hash table
38 |The hash function plays a pivotal role. Our naive implementation merely sums the ascii value of each character (minus 97). In order to fit within our fixed array, the remainder is taken (via the modulo operator (%)). This last step guarantees that our hash function returns a value between 0 and the size of the array (0 to 2 in our specific case).
Hash tables must deal with collisions - that is, two values returning the same bucket index. This is more common than you might think (see the birthday problem). One approach is for each bucket be its own data structure - say, a linked list.
52 | 53 |function HashTable(size) {
55 | this.size = size;
56 | this.buckets = new Array(size);
57 | for(var i = 0; i < size; ++i) {
58 | this.buckets[i] = new LinkedList();
59 | }
60 | this.add = function(value) {
61 | var index = this.hash(value);
62 | this.buckets[index].add(value);
63 | };
64 | this.hash = function(value) { ... };
65 | }
66 |
67 | var hash = new HashTable(5);
68 | hash.add('i');
69 | hash.add('will');
70 | hash.add('face');
71 | hash.add('my');
72 | hash.add('fear');
73 |
74 |
75 | Click step to add the values to our hash table
77 |As a general rule hash algorithms are lightweight and fast. This means that, without collisions, read and write performance is good (and constant). However, as collisions occur, read and write performance become dependent on the underlying collision resolution implementation. Also, if keys are not properly distributed, performance between individual buckets can vary.
96 | 97 |Hash tables are used frequently in modern applications. They are often used to store values per key in order to efficiently do lookups without relying on a linear search. However, many developers believe that reading and writing to a hash table is a constant 0(1) operation (that is, the hash table is collision-free). This is rarely the case. In the worst case, every value might go in the same bucket.
99 | 100 |Hashing functions and the collision strategy used by modern languages and libraries are efficient and should be used. The only thing to be careful of is using custom objects as a key without specifying a custom hashing solution.
101 | 102 | 106 | 107 | 108 | -------------------------------------------------------------------------------- /public/js/jquery.example.js: -------------------------------------------------------------------------------- 1 | (function($) 2 | { 3 | var defaults = {}; 4 | $.fn.example = function(options) 5 | { 6 | var opts = $.extend({}, defaults, options); 7 | return this.each(function() 8 | { 9 | if (this.example) { return false; } 10 | 11 | var instructions = options.instructions; 12 | var $container = $(this); 13 | var $code = options.code; 14 | var $nodes = $container.find('div.nodeList div'); 15 | var $topNodes, $bottomNodes = null; 16 | var step = -1; 17 | 18 | var self = 19 | { 20 | initialize: function() 21 | { 22 | $container.addClass('example'); 23 | var $top = $('