├── README.md ├── .gitignore ├── rails ├── rails_as_she_is_spoke.md └── growing_rails_apps_in_practice.md ├── planning ├── essentialism_ru.md ├── peak_performance_ru.md ├── superbetter_ru.md ├── the_practicing_mind.md └── genius_mom.md ├── ruby ├── effective_ruby.md ├── well_grounded_rubyist.md ├── ruby_under_a_microscope.md ├── exceptional_ruby.md ├── clean_ruby.md └── working_with_ruby_threads.md └── development ├── working_with_unix_processes_terms.md ├── designing_data_intensive_apps_short.md ├── shape_up.md ├── working_with_tcp_sockets.md └── working_with_unix_processes.md /README.md: -------------------------------------------------------------------------------- 1 | # books-notes 2 | Notes for the books I've read 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | code/ 2 | *.txt 3 | *.html 4 | poodr-notes/ 5 | development/designing_data_intensive_apps.md 6 | -------------------------------------------------------------------------------- /rails/rails_as_she_is_spoke.md: -------------------------------------------------------------------------------- 1 | ## Bad Naming Within Rails: A Broken Window 2 | 3 | 4 | patterns - not plan to use a pattern, but work and​ realize that you've used a pattern 5 | 6 | rails- abstract factory, no sti -------------------------------------------------------------------------------- /planning/essentialism_ru.md: -------------------------------------------------------------------------------- 1 | Оценка возможности (соглашаться или нет): 2 | 3 | - сформулируйте возможность 4 | - 3 минимальных критерия, к-м должен удовлетворять вариант 5 | - 3 оптимальных критерия в порядку убывания из важности 6 | 7 | Возможность должна удовлетфорять всем минимальным критериям и хотя бы двум из трёх оптимальных. 8 | -------------------------------------------------------------------------------- /planning/peak_performance_ru.md: -------------------------------------------------------------------------------- 1 | - идея чередования "тяжёлых" и лёгких периодов (как в спорте) 2 | - осознанность - полное погружение в работу или полное расслабление 3 | - если тратим "силу воли" на что-то - потом меньше сил, даже если дело в другой сфере 4 | - разбивать день на периоды и у каждого своя цель 5 | - час работы - 15мин перерыв 6 | - перерыв на "природу" - погулять в парке, посмотреть фотки природы (можно свои, кстати, разбирать с выходных) 7 | - ритуалы и триггеры (перед работой/выступлением и т.п.) 8 | -------------------------------------------------------------------------------- /planning/superbetter_ru.md: -------------------------------------------------------------------------------- 1 | # Superbetter 2 | 3 | Самоэффективность/аватары 4 | 5 | - self-efficacy (челленджи => баллы) 6 | - power-ups 7 | - physical (water, shower) 8 | - emotional (music, dance) 9 | - social (love sprint, work in a cafe) 10 | - mental (2 things forward this week, ) 11 | 12 | - face the bad guys 13 | - bad habits 14 | - the lord of impossibility 15 | - sticky chair 16 | - warm bed 17 | 18 | For one day: 19 | - 3 power-ups 20 | - 1 bad guy battle 21 | - 1 quest -------------------------------------------------------------------------------- /planning/the_practicing_mind.md: -------------------------------------------------------------------------------- 1 | 1) Attention - no distractions 2 | 2) Focus on the process instead of the product 3 | 3) You are perfect at any moment of your growing, as a flower at any stage ) 4 | 4) Focus on the present instead of the past/future. Focusing on the past/future is worrying. 5 | 5) Observe your practice instead of judging the results, meditation helps 6 | 6) Observe distractions and remove them 7 | 7) Do tasks slowly instead of doing it in a hussle 8 | 8) Trigger for the bad habits . e.g. picking up a phone => trigger 9 | 10 | 11 | -------------------------------------------------------------------------------- /ruby/effective_ruby.md: -------------------------------------------------------------------------------- 1 | ## Spaceship <=> 2 | - Implement object ordering by defining a “ <=> ” operator and including the Comparable module 3 | - The <=> operator should return nil if the left operand can’t be compared 4 | with the right. 5 | - If you implement `<=>`` for a class you should consider aliasing eql? to 6 | `==` 7 | 8 | ## Prefer Class Instance Variables to Class Variables 9 | Class variables (those which begin with “ @@ ”) are handled differently. They’re attached to a class and visible to all instances of that class. 10 | 11 | Classes are objects and therefore have their own private set of instance 12 | variables. 13 | 14 | ## Duping Cloning 15 | `dup` returns unfrozen object 16 | `clone` can return frozen (if the caller is frozen) 17 | 18 | dup and clone return shallow copies! 19 | 20 | ## Manage Resources with Blocks and ensure 21 | Write an ensure clause to release any acquired resources. 22 | Use the block and ensure pattern with a class method to abstract away 23 | resource management. 24 | ```ruby 25 | class Lock 26 | def self.acquire 27 | lock = new # Initialize the resource 28 | lock.exclusive_lock! 29 | yield(lock) # Give it to the block 30 | ensure 31 | # Make sure it gets unlocked. 32 | lock.unlock if lock 33 | end 34 | end 35 | ``` 36 | ## Know the Difference Between the Variants of eval 37 | Passing binding: 38 | ```ruby 39 | def glass_case_of_emotion 40 | x = "I'm in a " + __method__.to_s.tr('_', ' ') 41 | binding 42 | end 43 | x = "I'm in a test" 44 | p eval('x') 45 | p eval('x', binding) 46 | p eval("x", glass_case_of_emotion) 47 | ``` 48 | `instance_eval`: 49 | 50 | The object you invoke `instance_eval` on becomes the context for the evaluation. This allows you to reach into an object and access its private methods and instance variables. 51 | 52 | ```ruby 53 | Widget.instance_eval do 54 | def table_name; "widgets"; end 55 | end 56 | Widget.table_name 57 | Widget.singleton_methods(false) 58 | ``` 59 | `class_eval` evaluates a string or a block in the context of a class. 60 | 61 | ```ruby 62 | Widget.class_eval do 63 | attr_accessor(:name) 64 | def sold?; false; end 65 | end 66 | ``` 67 | `class_eval` is defined in the Module module as a singleton method which means it can only 68 | 69 | `instance_exec`, `class_exec`. `module_exec` 70 | The eval versions accept strings or blocks the exec variants only accept blocks. 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | be used on modules and classes. 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | -------------------------------------------------------------------------------- /ruby/well_grounded_rubyist.md: -------------------------------------------------------------------------------- 1 | ## Chapter 10. Collections central: Enumerable and Enumerator 2 | 3 | Mixing `Enumerable` to your own classes: 4 | ```ruby 5 | class C 6 | include Enumerable 7 | end 8 | ``` 9 | Enumerators are objects that encapsulate knowledge of how to iterate through a particular collection. 10 | 11 | ### 10.1 Gaining enumerability through each 12 | `each` yields items to a supplied code block, one at a time. 13 | `each` does different things for different types of objects. 14 | ```ruby 15 | class Rainbow 16 | include Enumerable 17 | def each 18 | yield "red" 19 | yield "orange" 20 | yield "yellow" 21 | yield "green" 22 | yield "blue" 23 | yield "indigo" 24 | yield "violet" 25 | end 26 | end 27 | 28 | r = Rainbow.new 29 | r.each do |color| 30 | puts "Next color: #{color}" 31 | end 32 | 33 | y_color = r.find {|color| color.start_with?('y') } 34 | Enumerable.instance_methods(false).sort 35 | ``` 36 | 37 | Defining each , together with mixing in Enumerable , buys you a great deal of functionality for your objects. Much of the searching and querying functionality you see in Ruby arrays, hashes, and other collection objects comes directly from Enumerable - `map`, `select`, `find`, etc 38 | 39 | ### 10.2 Enumerable Boolean queries 40 | `any?`, `one?`, `none?`, `include?` 41 | 42 | ```ruby 43 | states.any? {|state| state =~ / / } 44 | states.one? {|state| state =~ /West/ } 45 | ``` 46 | 47 | For hashes: 48 | ```ruby 49 | states.include?("Louisiana") # consults keys 50 | states.all? {|state, abbr| state =~ / / } 51 | ``` 52 | ### 10.3 Enumerable searching and selecting 53 | `find`, `find_all`, `reject` 54 | 55 | `grep` 56 | ```ruby 57 | colors = %w{ red orange yellow green blue indigo violet } 58 | colors.grep(/o/) 59 | ``` 60 | `group_by`: 61 | ```ruby 62 | colors.group_by {|color| color.size } 63 | ``` 64 | 65 | `partition`: 66 | ```ruby 67 | enum.partition {| obj | block } => [ true_array, false_array ] 68 | ``` 69 | 70 | ### 10.8 Sorting enumerables 71 | 1 Define a comparison method for the class ( <=> ). 72 | 2 Place the multiple instances in a container, probably an array. 73 | 3 Sort the container. 74 | 75 | 76 | ```ruby 77 | class Painting 78 | ... 79 | def <=>(other_painting) 80 | self.price <=> other_painting.price 81 | end 82 | end 83 | 84 | paintings.sort 85 | ``` 86 | #### 10.8.2 Defining sort-order logic with a block 87 | ```ruby 88 | year_sort = [pa1, pa2, pa3, pa4, pa5].sort do |a,b| 89 | a.year <=> b.year 90 | end 91 | ``` 92 | 93 | ### 10.9 Enumerators and the next dimension of enumerability 94 | An iterator is a method that yields one or more values to a code block. An enumerator is an object, not a method. 95 | 96 | An enumerator is a simple enumerable object. It has an each method. It has an each method, and it employs the Enumerable module to define all the usual methods— select , inject , map , and friends—directly on top of its each . 97 | 98 | #### 10.9.1 Creating enumerators with a code block 99 | ```ruby 100 | e = Enumerator.new do |y| 101 | y << 1 102 | y << 2 103 | y << 3 104 | end 105 | ``` 106 | `y` is a yielder, an instance of `Enumerator::Yielder`, automatically passed to your block. 107 | y.yield(1) - same 108 | 109 | Upon being asked to iterate, the enumerator consults the yielder and makes the next move—the next yield—based on the instructions that the yielder has stored. 110 | 111 | The enumerator e is an enumerating machine 112 | -------------------------------------------------------------------------------- /development/working_with_unix_processes_terms.md: -------------------------------------------------------------------------------- 1 | Processes: any code is executed in a process. 2 | Pid - process id. 3 | Ppid - process parent id. 4 | 5 | File descriptors - represent open files(resources). Devices, sockets, pids and pipes (resources) are also treated like files. 6 | `file.fileno` - unique number of the open resource. 7 | Standart streams: STDIN (input), STDOUT (output), STDERR (error), they are also resources with fileno 0, 1, 2. 8 | Processes have limits for the number of file descriptors. The number is usually very large. You can change it. 9 | 10 | *Environment variables* are often used to accept input to a program. They are global to a process and are inhereted by child processes. 11 | 12 | `$PROGRAM_NAME`, `$0` - process names 13 | 14 | Exit codes - last mark of a program, when it exits. 15 | 0 - success, 1 - unsuccessful (usually), you can use custom codes. 16 | `at_exit` hook - executed at exit 17 | `abort` - exit unsuccessfully (code 1), executes `at_exit` before ending 18 | 19 | *Forking* - creating a copy of an existing process. Copies all the memory and file descriptors of the parent process. 20 | `fork` may be used with `if` or more often with a block. 21 | `fork` returns 2 times - for the child and the parent processes. 22 | 23 | Copy-on-write - copying only when there's a need to modify, available for forking since ruby 2.0 24 | 25 | *Orphaned processes* - child processes that run after the parent dies. They're not treated differently from other processes. 26 | 27 | `Process.wait` - a blocking call, the parent process will wait for the child to exit. Returns status of an exited child. 28 | `wait2` - return child pid and status 29 | `waitpid`, `waitpid2` - wait for the specific process. 30 | Waiting when no children are running will cause `Errno::ECHILD` 31 | 32 | *Zombie Processes* - dead processes, whose status hasn't been waited on. 33 | If you want to "fire and forget" you need to detauch process. 34 | 35 | Nonblocking calls to keep track of children: 36 | trapping the :CHLD signal - `trap(:CHLD)` (will tell the parent when the child exits) 37 | `Process.wait(-1, Process::WNOHANG)` - nonblocking call 38 | 39 | *Signals* 40 | Signals are sent from one process to another process, using the kernel as a middleman. 41 | You can send signals like `SIGINT`, `SIGKILL` or others to a process (`SIG` part is optional). Signals have their default actions like terminate, terminate and dump, ignore, pause or resume. 42 | `SIGUSR1`, `SIGUSR2` - special user-defined signals for the process. 43 | 44 | You can redefine signals, e.g. `trap(:INT) { print "Na na na, you can't get me" }` `SIGKILL` can't be redefined. 45 | 46 | Ignoring signals - `trap(:INT, "IGNORE")` 47 | 48 | *IPC* - inter-process communication 49 | Can be done with pipes or sockets. 50 | *Pipes* - one-way communication. One end is a reader, another is a writer. 51 | *Sockets* - 2-way communication, reading and writing. 52 | `child_socket, parent_socket = Socket.pair(:UNIX, :DGRAM, 0)` 53 | 54 | *Remote IPC* - RPC, ZeroMQ, TCP sockets. 55 | 56 | *Daemon processes* - processes that run in the background, not under the control of a user or at a terminal. `Process.daemon` 57 | 58 | *Process group* - usually a group is a parent with children, but you can also set a group arbitrarily. 59 | *Session Groups* - `git log | grep shipped | less` - each command will have it's own process group, but the same session group. 60 | 61 | `exec` - transform ruby process to any other process (pass string or array), won't go back 62 | `system` - returns true or false 63 | ``` and `%x[]` - returns output of a program 64 | `Process#spawn` - nonblocking call 65 | `IO.popen`, `open3` - sets up a pipe to communicate with the spawned process 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | -------------------------------------------------------------------------------- /ruby/ruby_under_a_microscope.md: -------------------------------------------------------------------------------- 1 | ## Chapter 2. Tokenization and Parsing 2 | Ruby converts your code into two different for- 3 | mats. First, it converts the text in your Ruby program into a series of tokens. 4 | Next, it uses an LALR parser to convert the input stream of tokens into a 5 | data structure called an abstract syntax tree. 6 | 7 | ## Chapter 3. Compilation 8 | Ruby converts your code into a third format: 9 | a series of bytecode instructions that are later used when your program is actually executed. 10 | 11 | Ruby 1.8 executes your program directly from the AST. 12 | 13 | Starting with version 1.9, Ruby compiles your code before executing it. 14 | Translate your code from one programming language to another 15 | 16 | Ruby compiler translates your Ruby code into another language that Ruby’s virtual machine understands. 17 | 18 | But unlike C or Java Ruby’s compiler runs automatically without you ever knowing. 19 | 20 | Ruby’s compiler works by iterating through the AST produced by the tokenizing and parsing processes, generating a series of bytecode instructions along the way. 21 | 22 | Ruby code is compiled to the YARV virtual machine language. 23 | 24 | When you run any Ruby program, you are actually using a virtual 25 | machine designed specifically to execute Ruby programs. 26 | 27 | ## Chapter 4. Control Structures and Method Dispatch 28 | YARV’s control structures 29 | internally Ruby categorizes methods into 11 types. 30 | ISEQ - def 31 | Ruby labels its own built-in method as `CFUNC` 32 | Ruby uses a hash to track the argument labels and default values. 33 | 34 | ## Chapter 5. Objects and Classes 35 | Ruby uses the `RObject` structure to represent instances of any custom 36 | classes you define in your code and of some classes predefined by Ruby 37 | itself. 38 | > Every Ruby object is the combination of a class pointer and an array of instance variables. 39 | 40 | Ruby uses special C structures to represent instances of many commonly used, built-in Ruby classes called ”generic” objects. 41 | 42 | Ruby uses the RString structure to represent an instance of the 43 | String class, RArray for an instance of the Array class, or RRegexp for an 44 | instance of the Regexp class. 45 | 46 | The `RClass` structure working with the rb_classext_struct structure saves a large set of information 47 | 48 | > A Ruby class is a Ruby object that also contains method definitions, 49 | > attribute names, a superclass pointer, and a constants table. 50 | 51 | Ruby saves both instance and class variable types in the same hash table. 52 | Classes also contain a series of hash tables that store their methods. 53 | Each Ruby class records its superclass using the super pointer. 54 | 55 | ## Chapter 6. Method Lookup and Constant Lookup 56 | 57 | 58 | ## Chapter 12 Garbage Collection 59 | Clean up objects your program no longer uses. 60 | MRI uses the same GC algorithm John McCarthy invented over 50 years ago: mark-and-sweep garbage collection. 61 | 62 | JRuby and Rubinius use copying garbage collection, they also employ `generational garbage collection` and `concurrent garbage collection`. 63 | 64 | Garbage Collectors Solve Three Problems: 65 | - allocate memory for use by new objects 66 | - identify which objects your program is no longer using. 67 | - reclaim memory from unused objects. 68 | 69 | When you create a new Ruby object, the garbage collector allocates memory for that object. Later, Ruby’s garbage collector determines when your program has stopped using the object so it can reuse that memory to create new Ruby objects. 70 | 71 | ### Mark and Sweep 72 | Internally all Ruby objects are represented by a C structure called RVALUE . 73 | 74 | MRI sets the length of the initial free list to about 10,000 RVALUE structures, which means that MRI can create 10,000 Ruby objects without allocating more memory. 75 | 76 | Ruby divides the allocated memory into subsections known as heaps in the MRI source code, each about 16k in size, creates a free list for each of these heaps. 77 | 78 | Eventually MRI uses up all remaining objects on the free list. At that point, the GC system stops your program, identifies objects that your code is no longer using, and reclaims their memory for allocation to new objects. I 79 | 80 | If no unused objects are found, Ruby asks the operating system for more memory. No memory ==> throw `OutOfMemory`` exception. 81 | 82 | The Lazy Sweeping algorithm reduces the amount of time a program is stopped by the garbage collector, sweeps only enough garbage objects back to the free 83 | list to create a few new Ruby objects. 84 | 85 | Main disadvantage -- the program has to stop. 86 | 87 | ### Copying Garbage Collector 88 | JRuby and Rubinius use another GC: 89 | - they allocate memory for new objects and reclaim memory from garbage objects using an algorithm called copying garbage collection. 90 | - handle old and young Ruby objects differently using generational 91 | garbage collection. 92 | - use concurrent garbage collection to perform some GC tasks at the 93 | same time that your application code is running. 94 | 95 | Copying garbage collectors allocate memory for new objects from 96 | a single large heap or memory segment. 97 | 98 | When that memory segment is used up, these collectors copy only the live objects over to a second memory segment, leaving the garbage objects behind. 99 | 100 | ### Generational Garbage Collection 101 | - a technique that treats new objects differently than older ones (young vs mature) 102 | 103 | #### Using the Semi-Space Algorithm for Young Objects 104 | Young objs become garbage frequently. 105 | When the Eden heap fills up with new objects, the garbage collector identifies most of them as garbage because new objects usually die young. 106 | Fewer live => fewer copying. 107 | 108 | When an object has survived several copyings, it's promoted to "mature". 109 | 110 | #### Garbage Collection for Mature Objects 111 | It's needed to run GC against them much less frequently. 112 | 113 | #### Concurrent Garbage Collection 114 | the garbage collector runs at the same time as your application code. 115 | Concurrent garbage collectors run in a separate thread from the pri- 116 | mary application. 117 | Most computers today contain microprocessors with multiple cores, which allow different threads to run in parallel, 1 core is dedicated to the GC thread. 118 | 119 | > MRI Ruby 2.1 also supports a form of concurrent garbage collection by performing the sweep portion of the mark-and-sweep algorithm in parallel while your Ruby code continues to run. 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | -------------------------------------------------------------------------------- /planning/genius_mom.md: -------------------------------------------------------------------------------- 1 | ## 1. Решите 1 раз 2 | Готовые решения: 3 | - подарки (опред подарок на каждый повод) 4 | - одежда (для дней) 5 | - еда (одни ингредиенты, одно и то же для гостей, матрица еды (что едите в какие дни) 6 | - выбрать традиции один раз 7 | 8 | ## 2. Начните с малого 9 | - убедитесь, что цель оправдывает средства (мб делаете то, что на самом деле неважно?) 10 | - подход "всё или ничего" ведёт в тупик, т.к. мы не можем сделать "всё" 11 | - делайте маленькие шаги к тому, что важно 12 | 13 | ## 3. Задайте волшебный вопрос "что я могу сделать сейчас, чтобы облегчить себе жизнь потом?" 14 | - займитесь необходимым, пока это не стало срочным 15 | - один маленький шаг в нужном направлении 16 | 17 | ## 4. Проживайте непростые периоды 18 | - раздражение в ожидании, пока сложный период кончится, нормально, но если всё время забегать вперёд/оглядываться назад, неудовлетворённость будет нарастать 19 | - не игнорировать огорчения, но не давать им управлять собой 20 | - делайте то, что нужно сейчас 21 | - не отвлекайтесь на то, что могло бы быть, или было 22 | - жизнь не сводится к сложным периодам 23 | 24 | ## 5. Правильный режим 25 | Режим, постоянство = безопасность 26 | Но иногда попытки держаться в жёстких рамках вызывают только усталость. 27 | Режим - не самоцель, это дорога к чему-то ещё, помогает делать дела. 28 | Понять, зачем именно нужен режим. 29 | Стройте свой режим на том единственном деле, после которого все остальные пункты плана станут проще или в них вообще отпадет необходимость. 30 | Утренний ритуал - потому что важно утреннее настроение. 31 | Важное (пример) - выбирать, о чем "разрешено" думать в этот день (что важно и что важнее всего в этот день) 32 | Можно добавить утренние ритуалы (кофе. etc), начиная с самого важного (на случай непредвиденного) 33 | Рабочий режим (пример): помидоры 40-60 минут, перерывы - социальное, творческое, активное. 34 | Подготовка к работе - спросите, что важно в вашей работе именно сегодня. 35 | Начинайте с самой необходимой вещи. 36 | Вечерние ритуалы - (пример) - восстановление порядка, подготовка к завтрашнему дню. 37 | Создавайте распорядок для любой задачи или времени суток, но начните с того, что важно для вас, а затем уже определите шаги, ведущие к этому. 38 | 39 | ## 6. Заведите правила дома 40 | Инструмент, который помогает поддержать по-настоящему значимое и не заблудиться. 41 | Правила нацелены на налаживание контактов, а не на оборону и контроль. 42 | Поддержка правил - не ценой мира в семье или в душе. 43 | Обратите внимание на ситуации, когда обороняетесь вместо того, чтобы налаживать связь (вызывают ярость, раздражение) 44 | Нет общих правил для всех (найдите свои, к-е не дадут "домино упасть", привести вас в ярость). 45 | 46 | ### Примеры 47 | - Школьные вещи на столещницу. 48 | - Посмотрите на рабочие поверхности своей кухни и обратите внимание на то, что могло бы отправиться в какое-то другое место. 49 | - Шкаф с одеждой: не покупать одежду, которую советует кто-то другой. 50 | - Правило для чтения - "начинать новую книжку в течение суток после завершения предыдущей" 51 | - Без телефонов за столом. 52 | - Правило для уборки - разобрать в одном месте прежде, чем устроить бедлам в другом. 53 | 54 | ### Поиск правил 55 | Эмоциональные срывы случаются, ок, просим прощения. 56 | Мы можем выбрать правила, которые помогут нам расти и становиться добрее. 57 | 58 | ## 7. У каждой вещи своё место 59 | 60 | Вещи случайным образом лежат по стопкам, кучам и корзинам => пространство превращается в ящик с хламом. 61 | Не нужно становиться минималистом, просто определите для каждой вещи свое место. 62 | 63 | Освободите место для того, что важно - например, шкафы для книг, а не безделушек. 64 | Расхламление не всегда помогает, иногда нужно не меньше вещей, а более эффективные привычки по обращению с ними. 65 | Сосредоточьтесь на маленьких ежедневных привычках: 66 | 67 | 1) Убирайте все по местам, как только у вас появится такая возможность, — так вы нейтрализуете «намагничивание беспорядка» 68 | 2) Перед покупкой решите, куда положите вещь до того, как придёте домой (мб положить некуда и покупать не нужно). 69 | 3) Выбрасывайте мусор (сломанные и ненужные вещи не оставлять "на всякий случай") 70 | 4) Убирайте по одной вещи в день (для разбора захламлённых мест - ящиков, поверхностей и тд) 71 | 5) Делайте еженедельные чистки, если вы готовы к большему после освоения привычки 4 72 | Выберите день недели и очистите одно из маленьких пространств. 73 | Сделайте список маленьких пространств, очищайте одно за раз. 74 | 6) Прислушайтесь к тому, что "говорят" вещи. Например, редко используемые вещи стоят на видном месте, приходится отодвигать и тд. 75 | 76 | Уборка игрушек: уборка раз в день. Убираемся, чтобы устроить новый беспорядок. Но это того стоит. 77 | 78 | Цель - не чистота, а получение большего удовольствия от жизни (дома). 79 | Ничто из этого не имеет никакого отношения к вашей ценности как личности. 80 | 81 | ## 8. Впустите других людей в свою жизнь. 82 | 83 | Игра, чтобы лучше узнать друг друга: «Электрический стул» - человек садится на стул и все задают разные вопросы. 84 | Впустите людей в свой дом. 85 | Аргументы против: не умеете готовить, плотный график (в конце дня не хочется делать что-то дополнительное), боитесь, что вашу дружбу отвергнут. 86 | Что если вы не станете к ним прислушиваться? Не извиняйтесь за свой дом, еду, несостоятельность. 87 | 88 | Идеи: 89 | - регулярный ужин 90 | - завтрак семьями по выходным 91 | - десертные вечера 92 | 93 | Впустите людей в свою повседневную жизнь: норм делиться чувствами, даже если не всё совсем ужасно. 94 | Кризис - не единственная причина просить о помощи. 95 | 96 | Делитесь событиями, которые вроде бы не безумно важные, но ценны тем, что они настоящие и происходят с нами. 97 | Нормально, если приглашенный в гости человек не станет вашим другом на всю жизнь, это всё равно того стоило. 98 | 99 | ## 9. Делите на партии 100 | 101 | Однотипные задачи делаем вместе, как на конвейере. Формирование партий дает отдых вашему мозгу, т.к. работаете на автопилоте. 102 | Некоторые вещи получаются лучше, если их автоматизировать (но не все), не обязательно всё делать вдумчиво. 103 | 104 | Партиями могут стать действия, к-е выполняем изо дня в день. 105 | Стирка - сортируйте по тому, где оказываются. 106 | Складывание - от самых крупных, по типам. 107 | 108 | На кухне: 109 | Освобождение поверхностей - освобождайте по одной за раз до конца, не отвлекаясь на другое, потом переходите к другой. 110 | Решите один раз, как вы прибираетесь (e.g. - сначала стол, потом рабочая поверхность, потом ...) - делать потом на автомате будет легче. 111 | 112 | ### Бумаги 113 | 114 | Зона "безотлагательных" бумаг - отдельно от остальных + напоминалка о её просмотре (например, регулярно, раз в 2 недели) 115 | Каталоги, скидки и тд - сразу в переработку 116 | Художественная зона - отдельная корзина с рисунками. Когда заполнится - сортировка на "шедевры" и мусор. 117 | "Зона будущего" - записи, вырезки и тд => потом за раз перенос всего в эл. вид 118 | 119 | ### Еда 120 | Список блюд, к-е легко приготовить. 121 | Подготовка партиями (лук и тд). 122 | Подготовка "полуфабрикатов" для заморозки - например, курица с приправами порционно. 123 | 124 | Формируйте партии, только если это упрощает вам жизнь. 125 | 126 | ## 10. Выбирайте ключевое 127 | 128 | Когда вы удаляете всё, что отвлекает вас от важного и оставляете только то, что является ключевым. 129 | Примеры - мало сортов чая, только базовые материалы для нового хобби. 130 | 131 | - определить важное 132 | - избавиться от неважного 133 | - оставить ключевое 134 | 135 | «Ключевой» не значит «минимальный», надо просто избавиться от того, что отвлекает от важного. 136 | 137 | ## 11. Действуйте в правильной последовательности 138 | 139 | Последовательность для всего: 140 | - вспомните о том, что важно 141 | - успокойте безумие (см, что бесит, предотвратить) 142 | - доверьтесь себе 143 | 144 | Можно поменять восприятие. Например, упражнения - ритуал заботы о теле, избавление от стресса. Уборка - освобождение места для нового. 145 | Успокоить безумие - например, организовать хранение; уборка - не с самого грязного, а самого важного. 146 | 147 | ## 12. Планируйте отдых 148 | 149 | Отдых должен быть регулярным, а не "в крайнем случае". 150 | 151 | Назвать несколько занятий, которые помогают вам почувствовать себя собой и отдохнуть. 152 | Список занятий, к-е не позволяют чувствовать себя собой. 153 | 154 | Сезонный отдых: В каждые три месяца освободите один день для отдыха - будете делать только то, что важно для вас. 155 | Если не можете выделить даже 4 дня в году, значит проблема более глубока - надо понять, почему не можете себе разрешить. 156 | 157 | Еженедельный отдых - не обязательно целый день, но регулярно. Например, прогулки, покупка кофе и тд 158 | Ежедневный отдых - найти что-то маленькое для удовольствия + сон. 159 | 160 | ## 13. Будьте к себе добры 161 | Вы не что-то, что нужно чинить, придавать форму и ежедневно оценивать. 162 | Вы ценный человек прямо сейчас, и этот человек заслуживает вашей доброты, потому что он ваш друг. 163 | Вы бы не стали осуждать подругу за то, что она ещё не достигла каких-то целей. Относитесь к себе так же. 164 | - Цените себя сегодняшнюю 165 | - Поразмышляйте о себе завтрашней 166 | «Если вы хотите добиться высоких результатов, забудьте о постановке целей. Вместо этого сосредоточьтесь на своей системе» 167 | 168 | Создавая систему вокруг того, что имеет значение, не надо чрезмерно сосредоточиваться на конечной цели. 169 | Отмечайте моменты роста и цените их. 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | Отзыв: 205 | Понравились рассуждения о том, что можно звать гостей в неубранный дом. 206 | 207 | Что не навязывает своё мнение. Типа "я упарываюсь по тому, в каком порядке убираться, но вам необязательно тоже так делать" 208 | 209 | 210 | -------------------------------------------------------------------------------- /rails/growing_rails_apps_in_practice.md: -------------------------------------------------------------------------------- 1 | # Notes from "Growing rails applications in practice" 2 | 3 | # introduction 4 | In this book you'll find techniques to scale large, monolithic Rails applications, learn how to use discipline, consistency and code organization to make your code grow more gently. 5 | Not revolutionary patterns or new gems, but using tools you already know the smarter way 6 | # New rules for Rails 7 | ## Beautiful controllers 8 | Controllers are not fun (( 9 | Lots of code, not sure where to place code 10 | Requires complex environment, hard to test. 11 | Lack of clear ror guidelines. 12 | It leads to the situation where every controller is a bit different. 13 | ## The case for consistent controller design 14 | A good approach is to use a standard controller design for every single user interaction. 15 | Having a default design approach speeds up development, leads to the better team work. 16 | ## Normalizing user interactions 17 | Normalize every user interaction to crud, even if doesn't look like. 18 | ## A better controller implementation 19 | Short, dry, minimum glue code 20 | - The controller actions are delegating most of their work to helper methods like load_note or build_note. 21 | - There is a private method note_scope which is used by all member action with a given ID and index method. 22 | - private method #note_params for attributes 23 | - every controller action reads or changes a single model 24 | - you might want to use a custom model for forms that are complicated or do not persist to a database 25 | ## Why have controllers at all? 26 | - Security (auth, auth) 27 | - Parse, whitelist parameters, loading and instantiating a model 28 | - Decide which view to render 29 | - No heavy lifting 30 | Inherited resources and such gems: don't use for large apps - confusing configuration, too much magic 31 | 32 | # Relearning ActiveRecord 33 | Activerecord is useful to validate data, form roundtrips, lifecycle callbacks 34 | error handling and input validation 35 | but it's easy to shoot yourself in the foot. 36 | ## Understanding the ActiveRecord lifecycle 37 | Ar requires your models to be written in a certain way 38 | "Infinite protocol" of AR - "objects on rails" 39 | Approaches: 40 | - get rid of 'bad' methods generated by AR and leave only 'good' methods you can control (fig-leaf). but that's fighting AR 41 | - alternate implementation: do stuff using validations and callbacks 42 | So don't rely on custom model methods that are not callbacks. 43 | Enforce the integrity of the validations and callbacks. But remember about update_attributes which doesn't fire callbacks. 44 | Too many callbacks => look in 'Dealing with fat models' 45 | 9. The true API of ActiveRecord models 46 | using any API methods: 47 | 1) instantiate 48 | 2) manipulate 49 | 3) manipulating puts object in dirty state, not saved 50 | 4) after validations: all changes are commited to db in a single transaction 51 | Benefits: clear views, no need to learn objects custom api, impossible accidental misuse, possible to use standart controller design, many useful libs 52 | It's not exclusive to AR models, it's possible for unpersisted classes too.4. User interactions without a database 53 | ## 4. User interactions without a database 54 | There are many ui interactions that don't need changes in db: sign in form, search form, payment form. 55 | Useful AR features for such forms: validations and form roundtrips, attributes setterst to cast, translations, transactional-style submissions. 56 | 10. Writing a better sign in form 57 | SignInForm < PlainModel 58 | (link to code, if possible) 59 | PlainModel is a class that makes POROs feel a lot like ActiveRecord. 60 | PlainModel is a wrapper around ActiveModel 61 | Virtus? 62 | SignInForm has the same lifecycle of AR model: 63 | - attributes set from params 64 | - object is asked to validate its attributes 65 | - if the object can be saved, some action is performed 66 | 11. Building PlainModel 67 | Custom wrapper around Active model 68 | Gem ActiveType from the book authors 69 | 12. Refactoring controllers from hell 70 | Hell - long, lots of logic 71 | Refactor with a class inheriting from ActiveType - clean view, clean controller 72 | Form object with callbacks - move the code from the controller. 73 | Unit tests controller are easier 74 | Any controller: 75 | - instantiate an object 76 | - assign attributes 77 | - try to save 78 | - render a view or redirect 79 | # Creating a system for growth 80 | # Dealing with fat models 81 | Problems: afraid of calling unneeded callback, too much callbacks and validations, different UI requires different support code from your model 82 | ## Why models grow fat 83 | More and more purposes, lots of use cases 84 | Different forms need different validations, fear of unneeded callbacks triggering, hard to test because of side-effects 85 | ## The case of the missing classes 86 | Code never goes away, you'll need to put it into a place or it'll infest an existing class 87 | ## Getting into a habit of organizing 88 | Instead of looking for existing AR class first look for new classes 89 | # A home for interaction-specific code 90 | Reduce fat model to slim core model. Create multiple interaction-specific form models. 91 | Model: minimum of validations, associations, universal methods. 92 | Not: validations for specific views, virtual attributes, callbacks for particular case, auth, helpers for views and rendering 93 | ## A modest approach to form models 94 | Problems with gems for form objects, presenters, etc 95 | Vanilla inheritance for form objects 96 | Use::Assign up < User 97 | ## More convenience for form models 98 | Tweaks from ActiveType + inheritance 99 | User::AsSignUp < ActiveType:: Record [User] 100 | ## Extracting service objects 101 | In a fat model there is a lot of code that doesn't need to be tied to AR. Such code should be extracted into poros. Slim down: walk through the code and find what code can be extracted into service objects. 102 | ### Example 103 | Move Note#search and connected methods to Note::Search 104 | Aggressively look for opportunities to extract into service objects 105 | 19. Did we just move code around? 106 | Yes, but we move from large monolithic blocks to multiple components 107 | X Organizing large codebases with namespaces 108 | Namespace models in sub-folders, it makes much easier to browse. 109 | app/models/invoice.rb 110 | app/models/invoice/item.rb (Invoice::Item, belongs_to :invoice) 111 | app/models/invoice/export.rb 112 | ... 113 | ### Real-world example 114 | ## Use the same structure everywhere 115 | Make sure namespacing is also adopted by all the other places that are organized by model. 116 | (helpers, controllers, views) 117 | app/views/projects/index.html.erb 118 | app/views/projects/reports/index.html.erb 119 | Have a good default of organizing files 120 | XI Taming stylesheets 121 | css files can spiral out of control easily: 122 | - afraid that change will break some screens 123 | - simple p or label already inherits millions of styles 124 | - you often have to use "!important" 125 | 22. How CSS grows out of control 126 | Maintainable css is hard. 127 | 23. An API for your stylesheets 128 | BEM (Block, Element, Modifier) - a set of rules 129 | It's like spaghetti vs OOP 130 | #### Blocks 131 | BEM stylesheets consist of a simple list of blocks: 132 | e.g. nav bar, article, row of buttons. 133 | You can think of them as classes. 134 | #### Elements 135 | Blocks comprise elements. 136 | Elements can only exist as the child of one specific block, their name is prefixed by block name. 137 | An element can't exist without a block. They are like methods of the class (block). 138 | The list of selectors is always flat, avoid nesting. 139 | .article{} 140 | .article__title{} 141 | .article__text {} 142 | #### Modifiers 143 | E.g. we need a block that behaves 90% like another block. 144 | Modifiers are the optional parameters of the blocks and elements. 145 | Implemented as conjunctive selectors on an existing block or modifier 146 | Overshadow existing style from the same block. 147 | .button.is_primary { .. } 148 | Modifiers are not prefixed with the name of their block. 149 | Choose names like is_small or has_children 150 | Block modifiers can also modify elements as they are within the same block. 151 | .article.is_summary . article__title {} 152 | 24. The BEM prime directive 153 | A block must never influence the style of other blocks 154 | - no inheritance of unwanted styles 155 | - no fear of breaking styles 156 | - easy to reuse 157 | Instead of: 158 | .article{} 159 | .Sidebar.Article{} 160 | Do: 161 | .sidebar{} 162 | .sidebar__article{} 163 | Or: 164 | .Article{} 165 | . Article.is_summary{} 166 | .Sidebar{} 167 | Escape from CSS hell by isolating blocks. 168 | 25. Full BEM layout example 169 | Do be wary of God blocks that grow too large 170 | 26. Organizing stylesheets 171 | One file for each block 172 | 27. BEM anti-patterns 173 | - badly named elements 174 | - giant god blocks 175 | - blocks for each screen 176 | - violation of prime directive 177 | You are creating API for your stylesheets 178 | 28. Living style guides 179 | Use gems like hologram to generate living docs. 180 | 29. Pragmatic BEM 181 | There are situations when you can slightly deviate from bem dogma, e.g.compound structures like tables. 182 | When to be pragmatic: 183 | - compound elements like tables, I'll 184 | ... 185 | If you are unsure, be strict 186 | 187 | # Building applications to last 188 | # On following fashions 189 | 30. Before/after code comparisons 190 | Compare before and after code when you are testing a new fashion/style 191 | + is the code easier? 192 | - amount of code, number of files, coupling- changed? 193 | - are there new issues? 194 | - requires new library? Worth it? 195 | Decide if using new pattern works for you? 196 | 31. Understanding trade-offs 197 | New patterns are incremental improvements, not silver bullets. 198 | Scan them for trade-offs 199 | 32. The value of consistency 200 | Don't adapt every new fashion, cause your app will be a patchwork of different styles. 201 | Consistent code style is the priority. 202 | XIV Surviving the upgrade pace of Rails 203 | 33. Gems increase the cost of upgrades 204 | So think before using a new gem 205 | 34. Upgrades are when you pay for monkey patches 206 | 35. Don’t live on the bleeding edge 207 | XV Owning your stack 208 | Decide whether to use specific gem: 209 | - is the code well-groomed or messy 210 | - tests? 211 | - active development? 212 | - are you ready to maintain in case? 213 | - functionality significant to accept the cost of integration and maintenance or is it easier to implement yourself? 214 | 36. Accepting storage services into your stack 215 | ... 216 | 37. Maxing out your current toolbox 217 | Go with a simple solution until your load or data volume justify the changeto easily switch later hide service access behind a simple API. 218 | TODO - example 219 | XVI The value of tests 220 | Learning to test is the most important skill. 221 | Benefits: no bugs, not scary releases, refactor without fear, ability to work on 1 part of the app without knowing the rest. 222 | But you need time to write tests and maintain them 223 | 38. Choosing test types effectively 224 | Most value: unit tests and full-stack integration tests (cucumber) 225 | - cover as much as possible with integration tests 226 | - every UI screen should be kissed by at least one integration test 227 | - need to test many edge cases => unit tests 228 | 39. How many tests are too many 229 | The dark house rule 230 | 40. When to repeat yourself in tests - and when not to 231 | Some repetition in tests is ok, they shouldn't be too clever. 232 | 41. Better design guided by tests 233 | "Use" the class in unit test before it (the class) even exists, test-driven design, improves quality. 234 | 42. Getting started with tests in legacy applications 235 | Write the full.integration tests, that walks along the "happy" path that is most central to your app. 236 | Whenever you make a change, create another test. 237 | You don't need full coverage to benefit from tests. 238 | 43. Overcoming resistance to testing in your team 239 | Usable test suite, 1 command to run. 240 | No code should be committed without green tests. 241 | 242 | -------------------------------------------------------------------------------- /development/designing_data_intensive_apps_short.md: -------------------------------------------------------------------------------- 1 | Causality - The dependency between events that arises when one thing “happens 2 | before” another thing in a system. 3 | Deterministic - a function that always produces the same output if you give it 4 | the same input (no random numbers, time of the day, etc) 5 | Hash - A function that turns an input into a random-looking number. Same input => same number. If there's the same number for different inputs, it's a collision. 6 | Idempotent - an operation that can be safely retried; if it is executed more 7 | than once, it has the same effect as if it was only executed once. 8 | Linearizable - Behaving as if there was only a single copy of data in the system, which is updated by atomic operations. 9 | Materialize - to perform a computation eagerly and write out its result, as opposed to calculating it on demand when requested. 10 | OLAP - Online analytic processing. Access pattern characterized by aggregating 11 | (e.g., count, sum, average) over a large number of records. 12 | OLTP - Online transaction processing. Access pattern characterized by fast queries that read or write a small number of records, usually indexed by key. 13 | Partitioning/Sharding - Splitting up a large dataset or computation that is too big for a single machine into smaller parts and spreading them across several machines. 14 | Quorum -- The minimum number of nodes that need to vote on an operation before it can be considered successful. 15 | Serializable -- A guarantee that if several transactions execute concurrently, they behave the same as if they had executed one at a time, in some serial order. 16 | Skew -- 17 | 1) Imbalanced load across partitions, such that some partitions have lots of 18 | requests or data, and others have much less. 19 | 2) A timing anomaly that causes events to appear in an unexpected, 20 | nonsequential order. 21 | System of record - source of truth. 22 | Total order -- A way of comparing things (e.g., timestamps) that allows you to always say which one of two things is greater and which one is lesser. 23 | Two-phase commit (2PC) -- An algorithm to ensure that several database nodes either all commit or all abort a transaction. 24 | Two-phase locking (2PL) -- An algorithm for achieving serializable isolation that works by a transaction acquiring a lock on all data it reads or writes, and holding the lock until the end of the transaction. 25 | 26 | Summaries: 27 | ### Chapter 1. Reliable, Scalable, and Maintainable Applications 28 | Functional requirements: (what it should do, such as allowing data to be 29 | stored, retrieved, searched, and processed in various ways). 30 | Nonfunctional requirements (general properties like security, reliability, 31 | compliance, scalability, compatibility, and maintainability). 32 | Reliability -- making systems work correctly, even when faults occur. 33 | Scalability -- having strategies for keeping performance good, even when 34 | load increases. 35 | Maintainability -- making life better for the engineering and operations teams who need to work with the system, good abstractions, reducing complexity and helping to adopt and change the system. 36 | 37 | ### Chapter 2. Data Models and Query Languages 38 | Data started out being represented as one big tree (the hierarchical model), not good for representing many-to-many relationships. 39 | The relational model was invented to solve that problem. 40 | Later - nonrelational “NoSQL” datastores: 41 | 1) Document databases -- data comes in self-contained documents and relationships between one document and another are rare. 42 | 2) Graph databases -- designed for cases, where anything is potentially related to everything. 43 | 44 | Document and graph databases typically don’t enforce a schema for the data they store, which can make it easier to adapt applications to changing requirements. 45 | But the app still assumes some structure, so it’s just a question of whether the schema is explicit (enforced on write) or implicit (handled on read). 46 | 47 | Other data models: 48 | - specialized genome database software like GenBank (dealing with very long strings, representing DNA) 49 | - custom solutions for projects like the Large Hadron Collider (LHC) 50 | - Full-text search -- arguably a kind of data model, used alongside databases. 51 | 52 | ### Chapter 3. Storage and Retrieval 53 | Storage engines fall into two broad categories: 54 | 1) OLTP (optimized for transaction processing) -- user-facing, huge volume of requests, touch a small number of records in each query. Use some kind of key, the storage engine uses an index to find the data for the requested key. Bottleneck - disk seek time. 55 | 2) OLAP (optimized for analytics) -- Data warehouses and similar analytic systems, used by business analysts. 56 | Handle lower volume of queries, but each query requires many millions of records. Bottleneck -- disk bandwidth (not seek time). Column-oriented storage are popular. 57 | 58 | OLTP: 59 | 1) The log-structured school (appending to files, never update) -- Bitcask, 60 | SSTables, LSM-trees, LevelDB, Cassandra, HBase, Lucene. 61 | 2) Update-in-place -- B-trees, used in all major relational databases and also many nonrelational ones. 62 | 63 | OLAP: it becomes important to encode data very compactly, to minimize the amount of data that the query needs to read from disk. 64 | 65 | ### Chapter 4. Encoding and Evolution 66 | The problem of rolling upgrade without downtime. 67 | It is important that all data flowing around the system is encoded in a way that provides backward compatibility (new code can read old data) and 68 | forward compatibility (old code can read new data). 69 | Data encoding formats and their compatibility properties: 70 | 1) Programming language–specific encodings, often fail to provide forward and backward compatibility. 71 | 2) Textual formats like JSON, XML, and CSV, compatibility depends on how you use them. Somewhat vague about datatypes, you have to be careful. 72 | 3) Binary schema–driven formats like Thrift, Protocol Buffers, and Avro, allow compact, efficient encoding with clearly defined forward and backward compatibility semantics. Data needs to be decoded before it is human-readable. 73 | 74 | Modes of dataflow: 75 | - Databases (encoding/decoding by the db) 76 | - RPC and REST APIs (encoding/decoding by a server and a client) 77 | - Asynchronous message passing (using message brokers or actors, encoded by the 78 | sender and decoded by the recipient) 79 | 80 | ### Chapter 5. Replication 81 | Replication - keeping a copy of the same data on several machines. 82 | Purposes: 83 | - High availability - keeping the system working even if one machine (or several machines, or an entire datacenter) goes down 84 | - Allowing an application to continue working when there is a network 85 | interruption 86 | - Latency - Placing data geographically close to users 87 | - Scalability - Being able to handle a higher volume of reads than a single machine could handle, by performing reads on replicas. 88 | 89 | Requires carefully thinking about concurrency, unavailable nodes and network interruptions. 90 | 91 | Approaches: 92 | - Single-leader replication 93 | All writes are sent to a single node (the leader), which sends a stream of 94 | data change events to the other replicas (followers). Reads are performed from any replica (but might be stale). 95 | - Multi-leader replication 96 | Clients send each write to one of several leader nodes, any of which can 97 | accept writes. The leaders send streams of data change events to each other 98 | and to any follower nodes. 99 | - Leaderless replication 100 | Clients send each write to several nodes, and read from several nodes in 101 | parallel in order to detect and correct nodes with stale data. 102 | 103 | Replication can be synchronous or asynchronous. 104 | Сonsistency models for behaving under replication lag: 105 | - Read-after-write consistency (Users should always see data that they submitted themselves.) 106 | - Monotonic reads (After users have seen the data at one point in time, they shouldn’t later see the data from some earlier point in time.) 107 | - Consistent prefix reads (Users should see the data in a state that makes causal sense: for example, seeing a question and its reply in the correct order.) 108 | 109 | ### Chapter 6. Partitioning 110 | Necessary when the amount of data is too big for a single machine. 111 | Goal -- spread the data and query load evenly across multiple machines, avoiding hot spots. 112 | Approaches: 113 | - Key range partitioning. Keys are sorted, and a partition owns all the 114 | keys from some minimum up to some maximum. 115 | Efficient range queries, but the risk of hot spots. 116 | Partitions are typically rebalanced dynamically by splitting range into 2 ranges. 117 | - Hash partitioning. A hash function is applied to each key, and a 118 | partition owns a range of hashes. Fixed number of partitions in advance, moving entire partitions if the new node is added. 119 | - Hybrid approaches (compound key - one part of the key to identify the partition and another part for the sort order). 120 | 121 | A secondary index also needs to be partitioned: 122 | - Document-partitioned indexes (local indexes). Only a single partition needs to be updated on write, but a read of the secondary index requires a scatter/gather across all partitions. 123 | - Term-partitioned indexes (global indexes), the secondary indexes are 124 | partitioned separately, using the indexed values. On write several partitions of the secondary index need to be updated; a read can be served from a single partition. 125 | 126 | ### Chapter 7. Transactions 127 | Transactions are an abstraction layer that allows an application to pretend that certain concurrency problems and certain kinds of hardware and software 128 | faults don’t exist. 129 | Transaction help to not end up with inconsistent data, e.g. keep denormalized data in sync. 130 | Isolation levels: read commited, snapshot isolation, serializable. 131 | Dirty reads - one client reads another client’s writes before they have been committed (read commited prevents this) 132 | Dirty writes - one client overwrites data that another client has written, but not yet committed (Almost all transaction implementations prevent this) 133 | Read skew (nonrepeatable reads) - A client sees different parts of the database at different points in time (prevented with snapshot isolation, usually implemented with multi-version concurrency control (MVCC).) 134 | Lost updates - Two clients concurrently perform a read-modify-write cycle. 135 | One overwrites the other ’s write without incorporating its changes, so data is lost. Some implementations of snapshot isolation prevent this, other require manual lock (SELECT FOR UPDATE). 136 | Write skew - A transaction reads smth, makes a desicion based on that data, makes write, but when the write is commited the prev.read data is no longer true. Only serializable isolation prevents this anomaly. 137 | Phantom reads - A transaction reads objects that match some search condition. Another client makes a write that affects the results of that search. 138 | Snapshot isolation prevents straightforward phantom reads, phantoms in the context of write skew require special treatment. 139 | Only serializable isolation protects against all of these issues. 140 | Approaches for serializability: 141 | - Literally executing transactions in a serial order 142 | - Two-phase locking (bad performance) 143 | - Serializable snapshot isolation (SSI) - uses an optimistic approach, allowing transactions to proceed without blocking. When a transaction wants to commit, it is checked, and it is aborted if the execution was not serializable. 144 | 145 | ### Chapter 8. The Trouble with Distributed Systems 146 | - Packets sent over the network may be lost or arbitrarily delayed. The response may be lost to, so you have no idea whether the message got through. 147 | - A node’s clock may be significantly out of sync with other nodes 148 | - A process may pause for a substantial amount of time at any point in its 149 | execution (e.g. garbage collector) and be considered dead by other nodes. 150 | 151 | In distributed systems, we try to build tolerance of partial failures into software, so that the system still functions while its parts are broken. 152 | 153 | Detecting faults is hard: timeouts can’t distinguish between network and node failures, and variable network delay sometimes causes a node to be falsely suspected of crashing. 154 | 155 | Major decisions cannot be safely made by a single node, so we require 156 | protocols that enlist help from other nodes and try to get a quorum to agree. 157 | 158 | It is possible to give hard real-time response guarantees and bounded delays in networks, but doing so is very expensive and results in lower utilization of hardware resources. Most non-safety-critical systems choose cheap and unreliable over expensive and reliable. 159 | 160 | ### Chapter 9. Consistency and Consensus 161 | Linearizability - a consistency model, its goal is to make replicated data appear as though there were only a single copy, and to make all operations act on it atomically. 162 | Easy to understand but slow. 163 | Casuality -- imposes an ordering on events in a system (based on cause and effect). It's a weaker consistency model: some things can be concurrent, so the version history is like a timeline with branching and merging. Much less sensitive to network problems. 164 | Achieving consensus means deciding something in such a way that all nodes agree on what was decided, and such that the decision is irrevocable. 165 | A wide range of problems are actually reducible to consensus and are equivalent to each other. 166 | - Linearizable compare-and-set registers 167 | - Atomic transaction commit - wether to commit or abort 168 | - Total order broadcast - decide the order 169 | - Locks and leases - which client has successfully acquired a lock 170 | - Membership/coordination service - which nodes are alive 171 | - Uniqueness constraint - which one to allow and which should fail with a constraint violation 172 | 173 | Single node or single leader -- the leader makes all the decisions, but if the leader fails, the whole system is unable to do any progress. 174 | Ways to handle this situation: 175 | - Wait for the leader to recover 176 | - Manually fail over by getting humans to choose a new leader node and 177 | reconfigure the system to use it 178 | - Use an algorithm to automatically choose a new leader 179 | 180 | Although a single-leader database can provide linearizability without executing 181 | a consensus algorithm on every write, it still requires consensus to maintain its leadership and for leadership changes. 182 | 183 | Tools like ZooKeeper play an important role in providing an “outsourced” 184 | consensus, failure detection, and membership service that applications can use. 185 | 186 | Not every system necessarily requires consensus: for example, leaderless and multi-leader replication systems typically do not use global consensus. 187 | 188 | ### Chapter 10. Batch Processinфаа 189 | The design philosophy of Unix tools such as awk, grep, and sort affected MapReduce dataflow engines. 190 | Principles: 191 | - inputs are immutable 192 | - outputs are intended to become the input to another (as yet 193 | unknown) program 194 | - complex problems are solved by composing small tools that “do one thing well.” 195 | Dataflow engines add their own pipe-like data transport mechanisms to avoid materializing intermediate state to the distributed filesystem. The initial input and final output of a job is still usually HDFS. 196 | The two main problems that distributed batch processing frameworks need to 197 | solve are: 198 | - Partitioning 199 | Mappers are partitioned according to input file blocks. The purpose is to bring all the related data—e.g., all the records with the same key—together in the same place. 200 | Post-MapReduce dataflow engines try to avoid sorting unless it is required, 201 | but they otherwise take a broadly similar approach to partitioning. 202 | - Fault tolerance 203 | Dataflow engines perform less materialization of intermediate state and keep more in memory, which means that they need to recompute more data if a node fails. Deterministic operators reduce the amount of data that needs to be recomputed. 204 | 205 | Join algorithms: 206 | - Sort-merge joins 207 | - Broadcast hash joins 208 | - Partitioned hash joins 209 | 210 | Distributed batch processing engines have a deliberately restricted 211 | programming model: callback functions (such as mappers and reducers) are 212 | assumed to be stateless and to have no externally visible side effects besides 213 | their designated output. 214 | 215 | The framework can guarantee that the final output of a job is the same as if no faults had occurred, even though in reality various tasks perhaps had to be retried. 216 | Batch processing job reads some input, produces some output data, without modifying the input, the output is derived from the input. 217 | The data is bounded, it has a known, fixed size. 218 | 219 | ### Chapter 11. Stream Processing 220 | Stream processing is very much like the batch processing, but run on the unbounded data. From this perspective, message brokers and event logs serve as the streaming equivalent of a filesystem. 221 | 222 | 2 types of message brokers: 223 | 1) AMQP/JMS-style message broker 224 | The broker assigns individual messages to consumers, and consumers acknowledge individual messages when they have been successfully processed. Messages are deleted from the broker once they have been acknowledged. 225 | This approach is appropriate as an asynchronous form of RPC -- task queue, where the order is not important and there's no need to go back. 226 | 2) Log-based message broker 227 | The broker assigns all messages in a partition to the same consumer node, 228 | and always delivers messages in the same order. Parallelism is achieved through partitioning. The broker retains messages on disk. 229 | 230 | The log-based approach has similarities to the replication logs found in databases and log-structured storage engines. 231 | It can also be useful to think of the writes to a database as a stream: we can capture the changelog. 232 | Representing databases as streams opens up powerful opportunities for integrating systems. You can keep derived data systems such as search indexes, caches, and analytics systems continually up to date by consuming the log of 233 | changes and applying them to the derived system. 234 | 235 | Purposes of stream processing: searching for event patterns (complex event processing), computing windowed aggregations (stream analytics), keeping derived data systems up to date (materialized views). 236 | 237 | Types of joins in stream processing: 238 | - Stream-stream joins - 2 activity events streams (the join operator searches for related events that occur within some window of time.) 239 | - Stream-table joins - One input stream consists of activity events, while the other is a database changelog. For each activity event, the join operator queries the database and outputs an enriched activity event. 240 | - Table-table joins - Both input streams are database changelogs. Every change on one side is joined with the latest state of the other side. 241 | 242 | A finer-grained recovery mechanism can be used, based on microbatching, checkpointing, transactions, or idempotent writes -- for achieving fault tolerance and exactly-once semantics in a stream processor. 243 | 244 | ### Chapter 12. The Future of Data Systems 245 | Certain systems are designated as systems of record, and other data is derived from them through transformations. 246 | In this way we can maintain indexes, materialized views, machine learning models, statistical summaries, and more. 247 | By making these derivations and transformations asynchronous and loosely coupled, a problem in one area is prevented from spreading to unrelated parts of the system, increasing the robustness and fault-tolerance of the system as a whole. 248 | If you want to change one of the processing steps, for example to change the structure of an index or cache, you can just rerun the new transformation code on the whole input dataset in order to rederive the output. If something goes wrong, you can fix the code and reprocess the data in order to recover. 249 | Derived state can be updated by observing changes in the underlying data. 250 | The derived state itself can further be observed by downstream consumers. 251 | We can even build user interfaces that dynamically update to reflect data changes and continue to work offline. 252 | 253 | Strong integrity guarantees can be implemented scalably with asynchronous event processing, by using end-to-end operation identifiers to make operations idempotent and by checking constraints asynchronously. 254 | Clients can either wait until the check has passed, or go ahead without waiting but risk having to apologize about a constraint violation. This approach is much more scalable and robust than the traditional approach of using distributed transactions, and fits with how many business processes work in practice. 255 | By structuring applications around dataflow and checking constraints asynchronously, we can avoid most coordination and create systems that maintain integrity but still perform well, even in geographically distributed scenarios and in the presence of faults. 256 | 257 | 258 | 259 | 260 | 261 | 262 | 263 | 264 | 265 | 266 | 267 | 268 | 269 | 270 | 271 | 272 | 273 | 274 | 275 | 276 | 277 | 278 | 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 | 288 | 289 | 290 | 291 | 292 | 293 | 294 | 295 | 296 | 297 | 298 | 299 | 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 316 | 317 | 318 | 319 | 320 | 321 | 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | 335 | 336 | 337 | 338 | 339 | 340 | 341 | 342 | 343 | 344 | -------------------------------------------------------------------------------- /development/shape_up.md: -------------------------------------------------------------------------------- 1 | https://basecamp.com/shapeup/shape-up.pdf 2 | 3 | ## Chapter 1. Introduction 4 | 5 | Six-week cycles. 6 | Shaping the work: A small senior group works in parallel tothe cycle teams, they define the key elements of the solution. 7 | Shaped projects are concrete enough that the teams know what to do but abstract enough that they have room to work out the interesting detailsthemselves. 8 | Appetite (asking how much time do we want to spend on it) instead of estimating (asking how much will it take). 9 | Making teams responsible - give full responsibility to design+dev teams. 10 | Risk: bet to 6 weeks, If a project runs over,by default it doesn’t get an extension. 11 | 12 | ## Chapter 2. Principles of Shaping 13 | 14 | Wireframes are too concrete. 15 | It's harder to re-think for designers if you give them a concrete thing. 16 | 17 | Words are too abstract. 18 | Properties of shaped work: 19 | - rough (unfinished) 20 | - solved (the main elements are there) 21 | - bounded (indicates what not to do) 22 | 23 | Steps to shaping: 24 | - set boundaries 25 | - rough out the elements 26 | - address risks and rabbit holes 27 | - write the pitch 28 | 29 | ## Chapter 3. Set Boundaries 30 | 31 | Appetite - time budget. 32 | Small Batch - a team of a designer and 2 devs can do in 1-2 weeks 33 | Big Batch - the same team can do in 6 weeks 34 | 35 | Bigger scope => narrow the problem or break it so that the part would fit in 6 weeks. 36 | 37 | ### Fixed time, variable scope 38 | Estimates start with a design and end with a number. 39 | Appetites start with a number and end with a design. 40 | 41 | Good is relative, there is no best solution. 42 | 43 | Default response to any idea => "maybe, some day". 44 | 45 | ### Narrow down the problem 46 | Dig deeper and find out why the problem needs to be solved. It may lead to much simpler and quicker solution. 47 | Sometimes it makes sense to ask “What could we build?” instead of “What’s really going wrong?”. 48 | 49 | ### Watch out for grab-bags 50 | Unclear ideas, often like "redesign" or "refactoring". 51 | Instead, figure out what's not working or working wrong. 52 | 53 | ### Boundaries in place 54 | Move to the next step when you have 3 things: a raw idea, an appetite, and a narrow problem definition. 55 | 56 | ## Chapter 4: Find the Elements 57 | 58 | An idea in words => the elements of a software solution. 59 | - work alone or with a trusted partner 60 | - not too many details in the sketches 61 | 62 | Prototyping techniques: breadboarding and fat marker sketches. 63 | 64 | ### Breadboarding 65 | - an electrical engineering prototype that has all the componentsand wiring of a real device but no industrial design. 66 | 67 | Draw 3 basic things: 68 | - places - you can navigate to (screens, dialogs, menus) 69 | - affordances - a user can act on (buttons, fields, interfaces too) 70 | - connection lines - show how the affordances take the user from place to place 71 | 72 | Shaping will take a cloudy idea to a specific approach. 73 | 74 | ## Chapter 5. Risks and Rabbit Holes 75 | 76 | An unanticipated problem that takes 2 weeks to solve, or even worse, a problem with no apparent solution. 77 | An option is to abandon the project and rethink it later. 78 | But we should look for the pitfalls we can find before betting on a project. 79 | 80 | ### Look for rabbit holes 81 | Analyze: walk through a use case in slow motion, how exactly would a user get from a starting point to the end. 82 | - does this require new tech work we've never done before? 83 | - are we making assumptions about how parts fit together and the design? 84 | - is there a hard decision we should handle in advance? 85 | 86 | ### Case study: Patching a hole 87 | A hole in the concept - how to display completed tasks? 88 | Decided to leave them exactly as they worked before. 89 | Don't try to create the best design. 90 | 91 | Present to technical experts, get an idea of what is possible in 6 weeks 92 | 93 | At the end of this stage, we have the elements of the solution, patches for potential rabbit holes,and fences around areas we’ve declared out of bounds. 94 | 95 | ## Chapter 6. Write the Pitch 96 | 97 | Now we need to put the conceptinto a form that other people will be able to understand, digest, and respond to. 98 | Ingredients for the pitch: 99 | - problem 100 | - appetite 101 | - solution 102 | - rabbit holes 103 | - no-gos 104 | 105 | ### Ingredient 1. Problem 106 | Jumping right into working on a solution is dangerous. 107 | We wouldn't want to spend 6 weeks on the solution that will only benefit a small percentage of customers. 108 | 109 | ### Ingredient 2. Appetite 110 | Not only solve the problem but come up with a way to do this in 6 weeks. 111 | 112 | > if we only care enough to spend two weeks on this now, how does this specific solution look? 113 | 114 | ### Ingredient 3. Solution 115 | During the elements phase we draw the sketch ideas in the right level of abstraction. 116 | Now we need to write at the right level of details. 117 | 118 | We need some techniques to help people see the idea while still not going too far intoirrelevant details. 119 | 120 | - embedded (in the page) sketches 121 | - annotated fat marker sketches 122 | 123 | ### Ingredient 4. Rabbit holes 124 | Writing to patch potential rabbit holes (e.g. specify details of the solution or edge cases that we don't need to handle) 125 | 126 | ### Ingredient 5. No Gos 127 | E.g. specify that there will be no wysiwyg in the form. 128 | 129 | Present a pitch: 130 | - post the write-up with all ingredients 131 | - stakeholders read it in their own time (async communication) 132 | - people comment on the pitch asynchronously 133 | 134 | ## Chapter 7. Bets, Not Backlogs 135 | 136 | Backlog are big time wasters. 137 | 138 | ### A few potential bets 139 | A betting table - look at pitches fromthe last six weeks or any pitches purposefully revived. 140 | The pitches are potential bets. The meetings are infrequent, short, andintensely productive. 141 | If we decide to bet on a pitch, it goes into the next cycle to build. 142 | 143 | The great pitches can be lobbied again in 6 weeks if it's not right time for them right now. 144 | 145 | ### Decentralized lists 146 | Everyone can still track pitches, bugs, requests, or things they want to doindependently without a central backlog (support, product, bugs) 147 | 148 | Regular but infrequent one-on-ones between departments help to cross-pollinate ideas for whatto do next. 149 | 150 | This way the conversation is always fresh. 151 | 152 | ### Important ideas come back 153 | It's easy to overvalue ideas but they are cheap. Really important ones come back to you. 154 | 155 | ## Chapter 8. Bet Six Weeks 156 | 157 | Some teams use 2-weeks cycles, but it's too little time to build smth meaningful. The time is not worth spending time planning and discussing. 158 | 159 | After the 6-weeks period we have a 2-weeks cooldown. This is a period with no scheduled work to breath, meet and consider what to do next. 160 | During cool-down, programmers and designers on project teams are free to work on whateverthey want. 161 | 162 | ### Team and project sizes 163 | 1 designer + 2 programmers or 1 designer + 1 programmer. 164 | The teams will either spend the entire cycle working on one project or multiple smaller projects. 165 | 166 | ### The betting table 167 | - is a meeting held during cool-down where stakeholders decide what to do in the next cycle. 168 | 169 | Our betting table at Basecamp consists of the CEO (who in our case is the last word on product),CTO, a senior programmer, and a product strategist. 170 | 171 | ### The meaning of a bet 172 | If we bet six weeks, then we commit to giving the team the entiresix weeks to work exclusively on that thing with no interruptions. 173 | 174 | No "just a few hours" or "just one day" interruptions. 175 | 176 | ### The circuit breaker 177 | By default, the project doesn't get an extention if it's not finished in 6 weeks. 178 | If a project doesn’t finish in the six weeks, it means we did something wrong in theshaping. 179 | We can use the shaping track on the next six weeks to come up with a newor better solution that avoids rabbit holes. 180 | 181 | The circuit breaker motivates teams to take more ownership over their projects. 182 | 183 | ### What about bugs? 184 | There is nothing special about bugs that makes them automatically more important thaneverything else. 185 | If we’re in a real crisis, then drop everything to fix it. But that's rare. Most bugs can wait 6 weeks or longer. 186 | 187 | Strategies to deal with bugs: 188 | 189 | - Use cool-down 190 | - Bring it to the betting table 191 | - Schedule a bug smash 192 | Once a year—usually around the holidays—we’ll dedicate a wholecycle to fixing bugs. 193 | 194 | ### Keep the slate clean 195 | Never carry scraps of your old work to a betting table. 196 | 197 | ### Questions to ask 198 | - Does the problem matter? 199 | - Is the appetite right? 200 | - Is the solution attractive? 201 | - Is this the right time? 202 | - Are the right people available? 203 | 204 | ### Make the announcement 205 | 206 | After the bets are made, one of us will write a message that tells everyone which projects we’rebetting on for the next cycle and who will be working on them. 207 | 208 | ## Chapter 9. Hand Over Responsibility 209 | 210 | ### Assign projects, not tasks 211 | We don’t start by assigning tasks to anyone. Nobody plays the role of the “taskmaster” or the “architect” who splits the project up into pieces for other people to execute. 212 | 213 | Done means deployed. 214 | Deploying - at the end of the cycle or in small batches if a team has multiple small projects. 215 | Testing and QA is done within the cycle. Announcements and docs updates can be done at different time. 216 | 217 | ### Kick-off 218 | The project starts with: 219 | - creating a project 220 | - posting a pitch 221 | - kick-off call 222 | 223 | ### Getting Oriented 224 | First days - not much communication, no tasks checked off. Everybody is figuring out how the existing system works and which starting point is the best. 225 | Don't ask for visible progress at this stage. 226 | 227 | ### Imagined vs Discovered tasks 228 | The team comes up with the tasks themselves. Naturally, the team starts off with the imagined tasks. While doing real work, we discover new tasks. 229 | > The way to really figure out what needs to be done is to start doing real work. 230 | 231 | ## Chapter 10. Get One Piece Done 232 | 233 | There should be one thing implemented that's possible to try out. 234 | Integrate one slice (the corresponding backend/frontend/integration) so that the thing would be really done. 235 | 236 | ### Programmers don't need to wait 237 | Affordances before pixel-perfect screens. 238 | The first interface designer gives to a programmer is very basic. 239 | Program just enough for the next step. Create a back-and-forth between design and programming on the same piece of a product. 240 | 3 criteria to choose what to build first: 241 | - core (w/o it other work will mean nothing) 242 | - small 243 | - novel (prefer the things you've never done before, it would eliminate uncertainty) 244 | 245 | ## Chapter 11. Map the Scopes 246 | 247 | ### Organize by structure, not by person. 248 | People often separate tasks by their role (programmer or designer). This could lead to a problem: these tasks won't add up to a finished part of a project early on. 249 | We call integrated slices of the project scopes. The overall scope is broken into separate scopes that can be finished independently. 250 | 251 | ### The Scope Map 252 | As the team starts doing real work on a project they learn what actually the tasks are and what's the structure of a project. Then they factor the project into scopes. 253 | We define and track the scopes as to-do lists. 254 | 255 | ### The Language of the Project 256 | Scopes become the language of the project at the macro level. It's easier to reason about what's done and not done and the order to be done in this way, in comparison to discussing individual tasks. 257 | 258 | First: make a list of unscoped tasks and start working on smth. 259 | Then, focus on "get one piece done", focus on one key interaction you can integrate. Create a new scope for this piece and put "todos" into it. 260 | After finishing this scope, revise the unscoped tasks that you have. At this point they don't represent the work needs to be done. 261 | You can extract a new scope and tasks to work on. While working on it, you'll discover a more suitable scopes structure and new todos. 262 | You'll see the whole project at the high level. 263 | 264 | ### Discovering scopes 265 | Scope mapping isn't planning. You need to start working before you can draw the map. 266 | Scopes arise from interdependencies. At the start of the project we don't see accurate scopes. 267 | At the beginning the team focuses on specific problems so that it's easier to identify that scope early. 268 | 269 | ### How to know if the scopes are right 270 | When you define the scopes correctly: 271 | - you can see the whole project, and don't worry about the missing details 272 | - conversations become more flowing, because you use the right language 273 | - you know where to put a new task when it comes up (scopes are like bucketss) 274 | 275 | The scopes should be redrawn if: 276 | - it's hard to say when the scope is "done". The tasks in the scope are unrelated. 277 | - the name isn't unique to the project (e.g. "front-end" or "bugs" are grab-bags) 278 | - the scope is too big to finish soon 279 | 280 | ### Layer Cakes 281 | The work can be judged by UI surface area because back-end work is thin and evenly distributed. 282 | 283 | ### Icebergs 284 | Sometimes there is significantly more backend work or UI work. 285 | For icebergs, it sometimes makes sense to factor out the UI as a separate scope. The backend work could be split into several scopes as well. 286 | The goal is to define different things you can finish and integrate in stages. 287 | But first, question if you need the UI or back-end complexity at all. 288 | 289 | ### Chowder 290 | - a list for loose tasks that don't fit anywhere. 291 | But be careful if the list gets too long (more than 3-5 items). 292 | 293 | ### Mark nice-to-haves with ~ 294 | A lot of new tasks and possible improvements come up while you're working on the problem, but the time is limited. 295 | 296 | ## Chapter 12. Show Progress 297 | 298 | Ideally, managers should see the status of the tasks without having to explicitly ask for them. 299 | Todos can be used for this but there are a couple of problems with them. 300 | 301 | ### The tasks that aren't there 302 | Sometimes there are lists with all tasks completed, that could mean that all the work is done but can also mean that the further tasks were not defined yet. 303 | Or the scope is defined early but there are no tasks yet. 304 | 305 | QA can add tasks to the scope at the end of the project. 306 | Todos actually grow when team makes progress, so it's hard to guess what time the project will take, esp. at early steps. 307 | 308 | ### Estimates don't show uncertainty 309 | 310 | If you try to estimate a task with a lot of uncertainties, like doing smth the team has never done before, or has uncertain dependencies, your estimate will likely be very vague. 311 | 312 | We came up to see status of the project w/o counting tasks and w/o numeric estimates. 313 | We shifted focus from what's done and not done to what's unknown and what's solved. 314 | 315 | ### Work is like a hill 316 | 317 | unknown phase (figuring out what to do) 318 | peak of the hill 319 | known phase (getting it done) 320 | 321 | ### Scopes on the hill 322 | 323 | You can combine the hill with the concept of scopes: putting them at different positions on the sides of the hill and using different colors for different scopes. 324 | 325 | ### Status w/o asking 326 | A feature in Basecamp so that managers could see the state of the scopes w/o asking: displaying the hill + scopes. 327 | 328 | ### Nobody says "I don't know" 329 | It's easier to see if someone is stuck - the chart says it instead of people. 330 | 331 | ### Prompts to refactor the scopes 332 | Sometimes the work is going fine but the dot on the hill is not moving. E.g. there is a need to refactor scopes, like split a scope into different ones. 333 | 334 | ### Build your way uphill 335 | Sometimes unexpected unknowns appear. It's often because someone did the uphill work with their head instead of their hands. 336 | The first third uphill - "I've thought" 337 | The second third - "I've validated" 338 | The third - "I'm far enough with what I've built" 339 | 340 | ### Solve in the right sequence 341 | Some scopes are riskier than others. 342 | Ask the question: if we were out of time at the end of the cycle - which we could finish easier? 343 | This motivates the team to push the scariest way uphill first. 344 | It's better to get a few critical scopes uphill early. 345 | 346 | > Work expands to fill the time available 347 | 348 | Choose the most important problems with the most unknowns first. At the end of the cycle the team should end up with the important work done and variety of "nice to haves" and "maybes" lingering around. 349 | 350 | ## Chapter 13. Decide when to stop 351 | 352 | Shipping on time means shipping smth imperfect. Is it good enough? 353 | 354 | ### Compare to baseline 355 | Baseline - what customers already have. 356 | If the current solution is better than the current alternatives, it makes us feel better with the progress we've made. 357 | 358 | ### Limits motivate trade-offs 359 | Trade-offs instead of cramming and pushing to finish. 360 | 361 | - Scope grows like a grass 362 | - Cutting scope is not lowering quality 363 | differentiating what is core and what is peripheral 364 | 365 | ### Scope hammering 366 | Even stronger than cutting :D 367 | 368 | musts vs nice-to-haves (~) 369 | ~ are usually never built! 370 | 371 | ### QA is for the edges 372 | Designers and programmers take responsibility for the basic quality of their work. 373 | QA person comes at the end of the cycle and hunts for edge cases. 374 | 375 | We don't depend on QA to ship quality features. 376 | 377 | QA generates discovered tasks, progr/designers triage them, and make some of them "must-haves". 378 | 379 | ### When to extend a project 380 | Rare cases! 381 | Cool down can be used to finish some must-haves but this shouldn't become a habit. 382 | 383 | ## Chapter 14. Move On. 384 | 385 | Let the storm pass, stay cool after shipping the feature (usually you'll get a lot of feature requests after it) 386 | Stay debt-free. Don't make immediate changes after receiving the feedback. 387 | Feedback needs to be shaped. If a request is truly important, shape it in the next cycle. 388 | 389 | ## How to implement 390 | - add people to do the pitches 391 | - post pitches as messages 392 | - shape 393 | - create a project 394 | - add team members 395 | - post a kick-off message with the shaped concept 396 | 397 | To-do lists for scopes: 398 | - the team gets oriented, starts discovering tasks and mapping them into scopes 399 | - the team creates a todo for each scope 400 | - add design and progr tasks to each scope todo items 401 | - add scopes and todos as they are discovered 402 | 403 | Track scopes on the hill chart 404 | 405 | ## Adjust to your size 406 | 407 | Separate basic truths from the specific practices. 408 | Shaping, betting and the building phase are basic. The specific practices are scale-dependent. 409 | 410 | ### Small enough to wing it 411 | 412 | A tiny team can throw out most of the structure (like 6-weeks, cooldown, formal pitching and betting) 413 | The phases of the work still hold true even if you don't work in cycles or have dedicated people to do the shaping and building. 414 | 415 | ### Big enough to specialize 416 | 417 | At some point when there are more people available to build there is a need for someone to carve out more time to figure out what to build. 418 | E.g. the founder spends more time shaping than building or elevating an employee to do more shaping work. 419 | Shapers work on shaping while builders build. 2 weeks cooldown is used for fixing bugs and address popping loose ends by builders, and for betting by shapers. 420 | 421 | ## How to begin Shape Up 422 | ### New vs Existing products 423 | 424 | Shaping, betting and building won't work if you work on a completely new product or using a new technology. You need to figure out the basic architecture first. 425 | Early phase: "R&D mode" - build enough to be confident about the architecture. 426 | Dedicate one team and six weeks to explore a few features. Don't separate them into shapers and builders yet. 427 | 428 | Then flip from "R&D" to "production" mode. At this point the first team is ready to separate the shaping and delegate shaped work to others in the form of bets. 429 | Sometimes the same approach can be used for an existing product when there is a new area of work with an unsettled architecture. 430 | 431 | ### Option A. One 6-week experiment 432 | 433 | Guarantee that no one will interrupt programmers and designers during the experiment. 434 | Don't worry about mapping the scopes or showing progress right away. 435 | You should see a big progress just by shaping work in advance, providing uninterrupted time, and letting the team figure out the details. 436 | 437 | After getting used to "Get one thing done", the team works on properly mapping the scopes. Then, work on "Show Progress". 438 | 439 | ### Option B. Start with shaping 440 | 441 | Present a project and put it through existing scheduling process. This can already make things better and open up to the new things like longer cycles and betting. 442 | 443 | ### Option C. Start with cycles 444 | 445 | Moving from like 2 weeks sprint to 6-weeks cycle removes the need to constantly plan meetings, gives more time to build and provides more room to breathe. 446 | 447 | Fix shipping first - get the team into a rhythm of finishing things. 448 | 449 | Focus on the end result - the outcome is all that matters, you shouldn't worry on "what if someone of the team members sits idle at some point". 450 | -------------------------------------------------------------------------------- /ruby/exceptional_ruby.md: -------------------------------------------------------------------------------- 1 | # What is a failure? 2 | An *exception* is the occurrence of an abnormal condition during the execution of a software element. 3 | A *failure* is the inability of a software element to satisfy its purpose. 4 | An *error* is the presence in the software of some element not satisfying its specification. 5 | 6 | ## Failure as a breach of contract 7 | Bertrand Meyer "Design by Contract", Eiffel language. 8 | A method’s contract states “given the following inputs, I promise to return 9 | certain outputs and/or cause certain side-effects”. 10 | 11 | The callers responsibility is to ensure that the method’s preconditions—the inputs it depends on—are met. 12 | The method’s responsibility to ensure that its postconditions — those outputs and side-effects—are fulfilled. 13 | 14 | It is the method’s responsibility to maintain the invariant of the object it is a member of. The invariant is the set of conditions that must be met for 15 | the object to be in a consistent state. E.g. a TrafficLight class might have an invariant that only one lamp can be turned on at a time. 16 | 17 | ## Reasons for failure 18 | - a case that was unanticipated by the programmer at the 19 | time of writing. 20 | - the program is running on may have run out of resources, e.g. memory 21 | - some needed external element (e.g. webservice) fail 22 | 23 | Whatever the reason for the failure, a robust program needs to have 24 | a plan for handling exceptional conditions. 25 | 26 | # The life-cycle of an exception 27 | ## It all starts with a raise (or a fail) 28 | `raise` and `fail` are synonyms. 29 | Opinion: using `fail` as a default, useing `raise` only when catching and re-raising an exception. 30 | 31 | ## Calling raise 32 | With no arguments, it creates a RuntimeError with no message. 33 | With just a string, it creates a RuntimeError with the given string as its message. 34 | 35 | Given an explicit class (which must be a subclass of Exception 1 ), it raises 36 | an object of the specified class. 37 | 38 | Third argument: to specify a custom backtrace for the exception (useful for assertion methods). 39 | ```ruby 40 | def assert(value) 41 | raise(RuntimeError, "Doh!", caller) unless value 42 | end 43 | assert(4 == 5) 44 | ``` 45 | ## Overriding raise for fun and profit 46 | `raise` and `fail` are ruby methods define in `Kernel` 47 | 48 | Overriding to `exit` when smth is raised. It won't affect exceptions raised in 49 | C extensions, e.g. with rb_raise(). 50 | 51 | ### Hammertime 52 | - an error console for Ruby, presents a menu of options when an exception is raised. 53 | 54 | ### raise internals 55 | What raise does: 56 | - Call #exception to get the exception object. 57 | - Set the backtrace. 58 | - Set the global exception variable 59 | - Throw the exception object up the call stack. 60 | 61 | ### Step 1: Call #exception to get the exception. 62 | ``` 63 | def raise(error_class_or_obj, message, backtrace) 64 | error = error_class_or_obj.exception(message) 65 | # ... 66 | end 67 | ``` 68 | `Exception` class defines #exception methods at both the class and 69 | instance level. 70 | Class level -- `Exception.new` 71 | Instance level: 72 | w/o args - return `self` 73 | with args - returns a duplicate of itself. The new obj has a new message and maybe a new backtrace. 74 | MEthod `exception` is similar to `to_proc` or `to_ary` -- it works like an implicit “exception coercion”. 75 | 76 | ### Implementing your own #exception methods 77 | #### Step 2: #set_backtrace 78 | Call #caller passing 0 for the “start” parameter. 79 | `caller(0)` => get a stack trace which includes the current line 80 | 81 | #### Step 3: Set the global exception variable 82 | The $! global variable always contains the currently “active” exception (if 83 | any). raise is responsible for setting this variable. 84 | 85 | Separate threads get their own $! variable. 86 | 87 | #### Step 4: Raise the exception up the call stack 88 | raise begins to unwind the call stack. 89 | Ruby works its way up the current call stack one method call or block evaluation at a time, looking for `ensure` or `rescue` statements. 90 | 91 | If the top level of execution is reached and no `rescue` or `ensure` are met, Ruby prints out the exception message and a stack trace, and terminates the program with a failure status. 92 | 93 | ### ensure 94 | ensure clauses will always be executed, whether an exception is raised or 95 | not. They are useful for cleaning up resources. 96 | 97 | Keep ensure clauses clean, the secondary exception raised there can leave the cleanup unperformed and may lead to difficult debugging. 98 | 99 | #### ensure with explicit return 100 | If you explicitly return from a method inside an ensure block, the method will 101 | return as if no exception had been raised at all. The exception will be thrown away. 102 | You should avoid using explicit returns inside ensure blocks. 103 | 104 | ### Coming to the rescue 105 | > the rescue clause should be a short sequence of sim- 106 | ple instructions designed to bring the object back to a stable 107 | state and to either retry the operation or terminate with failure. 108 | 109 | By default rescue only captures StandardError and derived classes. 110 | 111 | Such exceptions as NoMemoryError, LoadError, SignalException, etc won't be catched. 112 | 113 | The exceptions outside of the `StandardError` hierarchy typically represent conditions that can’t reasonably be handled by a generic catch-all rescue clause, so they are not catched by default. 114 | 115 | You may supplu a list of classes: 116 | ``` 117 | rescue SomeError, SomeOtherError => error 118 | # ... 119 | end 120 | ``` 121 | #### Dynamic rescue clauses 122 | ```ruby 123 | def ignore_exceptions(*exceptions) 124 | yield 125 | rescue *exceptions => e 126 | puts "IGNORED: ’#{e}’" 127 | end 128 | puts "Doing risky operation" 129 | ignore_exceptions(IOError, SystemCallError) do 130 | open("NONEXISTENT_FILE") 131 | end 132 | puts "Carrying on..." 133 | ``` 134 | rescue can use `===` for matching, but with some limitations. 135 | ``` 136 | starts_with_a = Object.new 137 | def starts_with_a.===(e) 138 | /^A/ =~ e.name 139 | end 140 | 141 | rescue starts_with_a => error 142 | ``` 143 | The arguments to rescue must all be classes or modules. But so long as we satisfy that requirement, we can define the match conditions to be anything we want. 144 | 145 | #### rescue as a statement modifier 146 | ```ruby 147 | # return value of a statement or bil 148 | f = open("nonesuch.txt") rescue nil 149 | ``` 150 | 151 | ### If at first you don’t succeed, retry, retry again 152 | ```ruby 153 | tries = 0 154 | begin 155 | tries += 1 156 | puts "Trying #{tries}..." 157 | raise "Didn’t work" 158 | rescue 159 | retry if tries < 3 160 | puts "I give up" 161 | end 162 | ``` 163 | Be very careful that your “giving up” criterion will be met eventually. 164 | 165 | ### raise during exception handling 166 | In C++ this will be concidered a bug. 167 | In ruby we can always raise a new Exception. 168 | 169 | One possibility is that we raise an entirely new Exception (the old will be thrown away). 170 | 171 | Never allow the original exception to be thrown away. Instead, use Nested Exceptions. 172 | 173 | #### Nested Exceptions 174 | - exceptions which have a slot for a reference to the originating exception. 175 | ```ruby 176 | class MyError < StandardError 177 | attr_reader :original 178 | def initialize(msg, original=$!) 179 | super(msg) 180 | @original = original; 181 | end 182 | end 183 | begin 184 | begin 185 | raise "Error A" 186 | rescue => error 187 | raise MyError, "Error B" 188 | end 189 | rescue => error 190 | puts "Current failure: #{error.inspect}" 191 | puts "Original failure: #{error.original.inspect}" 192 | end 193 | ``` 194 | The default argument to original be the global error variable ($!), the object will “magically” discover the originating exception on initialization. 195 | 196 | #### More ways to re-raise 197 | Re-raising with custom message, custom backtrace. 198 | 199 | #### Disallowing double-raise 200 | Overriding `raise` to disallow. 201 | 202 | #### else 203 | else after a rescue clause is the opposite of rescue; 204 | It's only hit when no exception is raised. 205 | 206 | Why would we ever use else instead of simply putting the else-code after 207 | the code which might raise an exception? 208 | 209 | Exceptions raised in an else block are not captured by the preceding rescue clauses. Instead they are propagated up to the next higher scope of execution. 210 | 211 | The order of processing. 212 | ```ruby 213 | begin 214 | puts "in body" 215 | rescue 216 | puts "in rescue" 217 | else 218 | puts "in else" 219 | ensure 220 | puts "in ensure" 221 | end 222 | puts "after end" 223 | # in body 224 | # in else 225 | # in ensure 226 | # after end 227 | ``` 228 | 229 | #### Uncaught exceptions 230 | Ruby will handle the un-rescued exception by printing a stack trace and terminating the program. 231 | 232 | Before the program ends, it'll execute git hooks: 233 | ```ruby 234 | trap("EXIT") { puts "in trap" } 235 | at_exit { puts "in at_exit" } 236 | END { puts "in END" } 237 | 238 | raise "Not handled" 239 | ``` 240 | 241 | They all have access to global variables like $!. 242 | 243 | Useful for logging all fatal exception-induced crashes. 244 | 245 | #### Exceptions in threads 246 | work a little differently. By default, the thread terminates and the exception is held until another thread joins it, then the exception is re-raised in the joining thread. 247 | 248 | #### Are Ruby exceptions slow? 249 | Raising an exception is an expensive operation in any language, but there is no penalty to have excetions-using code. 250 | 251 | We should reserve exceptions for genuinely exceptional circumstances. 252 | 253 | ## Responding to failures 254 | ### Failure flags and benign values 255 | If the failure isn’t a major one, it may be sufficient to return a failure flag. 256 | When the system’s success doesn’t depend on the outcome of the method in question, using a benign value may be the right choice. 257 | Also writing tests will be easier using this values. 258 | ```ruby 259 | begin 260 | response = HTTP.get_response(url) 261 | JSON.parse(response.body) 262 | rescue Net::HTTPError 263 | {"stock_quote" => ""} 264 | end 265 | ``` 266 | 267 | ### Reporting failure to the console 268 | ``` 269 | warn "Uh oh, something bad happened" 270 | $stderr.puts 'Smth!' 271 | ``` 272 | `warn` also uses `$stderr` 273 | 274 | #### Warnings as errors 275 | ```ruby 276 | if $DEBUG 277 | module Kernel 278 | def warn(message) 279 | raise message 280 | end 281 | end 282 | warn "Uh oh" 283 | end 284 | ``` 285 | This way warn will only raise an exception if you run Ruby with the -d flag. 286 | 287 | ### Remote failure reporting 288 | - A central log server 289 | - An email to the developers 290 | - A post to a third-party exception-reporting service 291 | 292 | #### A true story of cascading failures 293 | A simple failure reporter, which used our team’s Gmail account to email the developers with the details of the failure. 294 | 295 | Update which caused lots of failures => Google throttling the account => SMTP errors => the workers started crashing hard => unrelated systems which used Gmail acc started failing. 296 | 297 | This is a classic example of a failure cascade. 298 | 299 | #### Bulkheads 300 | The bulkhead enforces a principle of damage containment in a sheep. 301 | 302 | ``` 303 | begin 304 | SomeExternalService.some_request 305 | rescue Exception => error 306 | logger.error "Exception intercepted calling SomeExternalService" 307 | logger.error error.message 308 | logger.error error.backtrace.join("\n") 309 | end 310 | ``` 311 | • External services 312 | • External processes 313 | 314 | #### The Circuit Breaker pattern 315 | - a way to mitigate cascade failures. 316 | When the breaker is Closed, the subsystem is allowed to operate normally. 317 | A counter tracks the number of failures that have occurred. 318 | When the number exceeds a threshold, the breaker trips, and enters the open 319 | state. In this state, the subsystem is not permitted to operate. 320 | After either a timeout elapses or (depending on the implementation) a hu- 321 | man intervenes and resets the breaker, it enters the half-open state. 322 | In this state the system can run, but a single failure will send it back into the “open” state. 323 | 324 | The Circuit Breaker pattern eliminates a common cause of failure cascades. 325 | 326 | #### Ending the program 327 | When you call `exit`, you're actually raising an exception (`SystemExit`). 328 | `exit 1` == `raise SystemExit.new(1)` 329 | 330 | If you want to print a failure message and then raise SystemExit with 331 | a failure status code: `abort "Uh oh"` 332 | 333 | *really* want to end the program quickly (no cleanup, exit hooks): 334 | ``` 335 | exit! false 336 | ``` 337 | 338 | ## Alternatives to exceptions 339 | Fail fast is often the best policy, but not always. 340 | Sometimes you need a way to proceed through the steps of the provisioning 341 | process and then get a report at the end telling you what parts succeeded 342 | and what parts failed. 343 | 344 | ### Sideband data 345 | ### Multiple return values 346 | in the form of array splatting. 347 | 348 | The downside of this approach is that anyone calling the method must know 349 | that it returns multiple values. 350 | 351 | A variation on multiple return values is to return a simple structure with 352 | attributes for the result value and for the status: 353 | ``` 354 | def foo 355 | # ... 356 | OpenStruct.new(:result => 42, :status => :success) 357 | end 358 | ``` 359 | Fine for simple cases, but it quickly breaks down when we want return the status information from several levels deep of method execution. 360 | 361 | ### Output parameters 362 | When writing code that executes shell commands, you can provide a transcript parameter for capturing the output of the entire session: 363 | ```ruby 364 | require ’stringio’ 365 | def make_user_accounts(host, transcript=StringIO.new) 366 | transcript.puts "* Making user accounts..." 367 | # ... code that captures STDOUT and STDERR to the transcript 368 | end 369 | ... 370 | transcript = StringIO.new 371 | make_user_accounts('192.168.1.123', transcript) 372 | ``` 373 | 374 | ### Caller-supplied fallback strategy 375 | Inject a failure policy into the process. 376 | ```ruby 377 | def make_user_accounts(host, failure_policy=method(:raise)) 378 | # ... 379 | rescue => error 380 | failure_policy.call(error) 381 | end 382 | 383 | policy = lambda {|e| puts e.message} 384 | make_user_accounts("192.168.1.123", policy) 385 | ``` 386 | ### Global variables 387 | We’ll use a thread-local variable instead of global variable. 388 | ```ruby 389 | class Provisioner 390 | def provision 391 | # ... 392 | (Thread.current[:provisioner_errors] ||= []) << "Error getting key file..." 393 | end 394 | end 395 | p = Provisioner.new 396 | p.provision 397 | if Array(Thread.current[:provisioner_errors]).size > 0 398 | # handle failures... 399 | end 400 | ``` 401 | 402 | Disadvantage: unless we are diligent about resetting the variable, there is no 403 | way of knowing for sure if its value is from the preceding call to #provision, 404 | or from some other prior and unrelated call. 405 | You can mitigate that issue by using a helper, which cleans the error variables when it's done. 406 | 407 | ### Process reification 408 | Represent the process itself as an object, and give the object an attribute for collecting status data: 409 | 410 | ```ruby 411 | class Provisionment 412 | attr_reader :problems 413 | def initialize 414 | @problems = [] 415 | end 416 | def perform 417 | # ... 418 | @problems << "Failure downloading key file..." 419 | end 420 | end 421 | p = Provisionment.new 422 | p.perform 423 | if p.problems.size > 0 424 | # ... handle errors ... 425 | end 426 | ``` 427 | 428 | ### Beyond exceptions 429 | Lisp conditions. 430 | 431 | ## Your failure handling strategy 432 | ### Exceptions shouldn’t be expected 433 | Consider carefully whether a situation is truly unusual before 434 | raising an Exception. 435 | E.g. `#save` doesn’t raise an exception when the record is invalid. 436 | 437 | When writing an application you expect invalid input from users. 438 | Don't hand invalid input with exceptions. 439 | 440 | ### A rule of thumb for raising 441 | > ask yourself, ‘Will this code still run if I remove all the excep- 442 | tion handlers?” If the answer is “no”, then maybe exceptions are 443 | being used in non-exceptional circumstances. 444 | 445 | ### Use throw for expected cases 446 | Sometimes you really want the kind of non-local return that raising an ex- 447 | ception can provide, but for a non-exceptional circumstance. 448 | 449 | Use `throw` and `catch`. 450 | 451 | In Ruby, throw and catch are all about providing a way to quickly break out of multiple levels of method calls in non-exceptional circumstances. 452 | 453 | ```ruby 454 | def last_modified(time) 455 | response[’Last-Modified’] = time 456 | if request.env[’HTTP_IF_MODIFIED_SINCE’] > time 457 | throw :halt, response 458 | end 459 | end 460 | ``` 461 | ### What constitutes an exceptional case? 462 | The answer to all of these is: it depends! 463 | 464 | ### Caller-supplied fallback strategy 465 | > In most cases, the caller should determine how to handle an error, not the callee. 466 | 467 | Great example: `fetch`. 468 | ```ruby 469 | h.fetch(:optional_key){ DEFAULT_VALUE } 470 | h.fetch(:required_key) { 471 | raise "Required key not found!" 472 | } 473 | ``` 474 | 475 | Another example: 476 | Enumerable: 477 | ```ruby 478 | arr.detect(lambda{"None found"}) {|x| ... } 479 | ``` 480 | 481 | This is a terrifically useful idiom to use in your own methods. 482 | Not sure => consider whether you can delegate the decision to the caller in some way. 483 | 484 | ```ruby 485 | # render_user yields the passed block 486 | # Fall back to a benign placeholder value: 487 | render_user(u){ "UNNAMED USER" } 488 | # Fall back to an exception: 489 | render_user(u){ raise "User missing a name" } 490 | ``` 491 | 492 | ### Some questions before raising 493 | 1: Is the situation truly unexpected? 494 | 2: Am I prepared to end the program? 495 | 3: Can I punt the decision up the call chain? 496 | 4: Am I throwing away valuable diagnostics? 497 | When you have just received the results of a long, expensive operation, 498 | that’s probably not a good time to raise an exception because of a trivial formatting error. Use some kind of sideband. 499 | 5: Would continuing result in a less informative exception? 500 | In cases like this, it’s better to raise earlier than later. 501 | ```ruby 502 | response_code = might_return_nil() or raise "No response code" 503 | ``` 504 | 505 | ### Isolate exception handling code 506 | Exception-handling code is messy. When it interrupts the flow of business logic, it makes program harder to understand. 507 | I consider the begin keyword to be a code smell in Ruby. 508 | The “better way” is to use Ruby’s implicit begin blocks: 509 | ```ruby 510 | def foo 511 | # mainline logic goes here 512 | rescue # ------------------- 513 | # failure handling goes here 514 | end 515 | ``` 516 | #### Isolating exception policy with a Contingency Method 517 | ```ruby 518 | def with_io_error_handling 519 | yield 520 | rescue 521 | # handle IOError 522 | end 523 | with_io_error_handling { something_that_might_fail } 524 | # ... 525 | with_io_error_handling { something_else_that_might_fail } 526 | ``` 527 | ### Exception Safety 528 | Some methods are critical, we can't afford them to fail. E.g. error-reporting methods in one of the prev examples. 529 | 530 | Methods that could potentially destroy user data or violate security 531 | protocols if they fail are critical. 532 | 533 | #### The three guarantees 534 | The weak guarantee: If an exception is raised, the object will be left in a 535 | consistent state. 536 | The strong guarantee: If an exception is raised, the object will be rolled 537 | back to its beginning state. 538 | The nothrow guarantee: No exceptions will be raised from the method. If 539 | an exception is raised during the execution of the method it will be 540 | handled internally. 541 | 542 | An implicit fourth(or zero-s) level: no guarantees. 543 | 544 | No guarantees => cascading failure. 545 | 546 | ### When can exceptions be raised? 547 | There are some exceptions, like NoMemoryError and SignalException, that can be raised at any point in execution. 548 | 549 | #### Exception safety testing 550 | For some methods, it is a good idea to have well-defined exception semantics, in the form of one of the Three Guarantees. 551 | Exception safety testing. 552 | 553 | In the playback phase, the test script is executed once for each call 554 | point found in the record phase. Each time, a different call point is 555 | forced to raise an exception. After each execution, the assertions are 556 | checked to verify that they have not been violated as a result of the 557 | exception that was raised. 558 | 559 | #### Validity vs. consistency 560 | Validity Are the business rules for the data met? 561 | Consistency Can the object operate without crashing, or exhibiting unde- 562 | fined behavior? 563 | 564 | We can preserve consistency even while permitting the object to be invalid 565 | from a business perspective. 566 | 567 | Typically an object’s invariant includes rules to maintain consistency, but it does not necessarily make assertions about validity 568 | 569 | ### Be specific when eating exceptions 570 | `rescue Exception` is prone to bugs. 571 | If you can’t match on the class of an Exception, try to at least match on the message. 572 | ```ruby 573 | begin 574 | # ... 575 | rescue => error 576 | raise unless error.message =~ /foo bar/ 577 | end 578 | ``` 579 | ### Namespace your own exceptions 580 | That gives client code something to match on when calling into the library. 581 | Your code can also raise such exceptions as IOError, SystemCallError. 582 | 583 | A way to be considerate to your library clients is to put code in the top-level 584 | API calls which catches non-namespaced Exceptions and wraps them in a namespaced exception before re-raising it (Namespaced exceptions). 585 | 586 | ```ruby 587 | rescue MyLibrary::Error 588 | raise 589 | rescue => error 590 | # This assumes MyLibrary::Error supports nesting: 591 | raise MyLibrary::Error.new( 592 | "#{error.class}: #{error.message}", 593 | error) 594 | end 595 | ``` 596 | 597 | ### Tagging exceptions with modules 598 | ```ruby 599 | error.extend(AcmeHttp::Error) 600 | ``` 601 | 602 | We can extract out the tagging code into a general-purpose helper method: 603 | ```ruby 604 | def tag_errors 605 | yield 606 | rescue Exception => error 607 | error.extend(AcmeHttp::Error) 608 | raise 609 | end 610 | ``` 611 | 612 | ### The no-raise library API 613 | The Typhoeus 15 library furnishes an excellent example of a no-raise library 614 | API. All of the information about a Typhoeus HTTP request—no matter 615 | what its final disposition—is encapsulated in the Typhoeus::Response class. 616 | 617 | ### Three essential exception classes 618 | - divide our Exceptions up by which module or subsystem 619 | they come from. 620 | - by software layer—e.g. separate exceptions for UI-level failures and for model-level failures. 621 | - by severity levels: fatal, non-fatal 622 | 623 | #### Three classes of exception 624 | - user error 625 | - logic error 626 | - transient error => retry 627 | 628 | 629 | 630 | 631 | 632 | 633 | 634 | 635 | 636 | 637 | 638 | 639 | 640 | 641 | 642 | 643 | 644 | 645 | 646 | 647 | 648 | 649 | 650 | 651 | 652 | 653 | 654 | 655 | 656 | 657 | 658 | 659 | 660 | 661 | 662 | 663 | 664 | 665 | 666 | 667 | 668 | 669 | 670 | 671 | 672 | 673 | 674 | 675 | 676 | 677 | 678 | 679 | 680 | 681 | 682 | 683 | 684 | 685 | 686 | 687 | 688 | 689 | 690 | 691 | 692 | 693 | 694 | 695 | 696 | 697 | 698 | 699 | 700 | 701 | 702 | 703 | 704 | 705 | 706 | 707 | 708 | 709 | 710 | 711 | 712 | 713 | 714 | 715 | 716 | 717 | 718 | 719 | 720 | 721 | 722 | 723 | 724 | 725 | 726 | 727 | 728 | 729 | 730 | 731 | 732 | 733 | 734 | 735 | 736 | 737 | 738 | 739 | 740 | 741 | 742 | 743 | 744 | 745 | 746 | 747 | 748 | 749 | 750 | 751 | 752 | 753 | 754 | 755 | 756 | 757 | 758 | 759 | 760 | 761 | 762 | 763 | 764 | 765 | 766 | 767 | 768 | 769 | 770 | 771 | 772 | 773 | 774 | 775 | 776 | 777 | 778 | 779 | 780 | 781 | 782 | 783 | 784 | 785 | 786 | 787 | 788 | 789 | 790 | 791 | 792 | 793 | 794 | 795 | 796 | 797 | 798 | 799 | 800 | 801 | 802 | 803 | 804 | 805 | 806 | 807 | 808 | 809 | 810 | 811 | 812 | 813 | -------------------------------------------------------------------------------- /development/working_with_tcp_sockets.md: -------------------------------------------------------------------------------- 1 | ## Chapter 1. Your First Socket 2 | ```ruby 3 | require 'socket' 4 | socket = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM) 5 | ``` 6 | Creates a socket of type `STREAM` in the `INET` domain. 7 | STREAM - communicating via stream (provided by TCP). 8 | DGRAM - datagram, refers to a UDP socket. 9 | 10 | ### Understanding Endpoints 11 | When 2 sockets need to communicate they need to know where to find each other. 12 | Sockets use Ip-addresses to route messages to specific hosts. 13 | 14 | Ipv4 - 4 numbers <= 255 15 | Ipv6 16 | 17 | ### Loopbacks 18 | A virtual interface. 19 | Any data sent to the loopback interface is immediately received on the same interface. 20 | `localhost`, `127.0.0.1` 21 | 22 | ### Ports 23 | Each socket must bind to a unique port number. 24 | Defaults - http - 80, ftp - 21 25 | 26 | ### Second Socket 27 | Creating a socket in the Ipv6 domain: 28 | ```ruby 29 | require 'socket' 30 | socket = Socket.new(:INET6, :STREAM) 31 | ``` 32 | 33 | ## Chapter 2. Establishing Connections 34 | TCP connections are made between 2 points. 35 | One socket is a initiator (client), and another one is a listener (server). 36 | 37 | ## Chapter 3. Server Lifecycle 38 | A server socket lifecycle: create-bind-listen-accept-close. 39 | 40 | ### Bind 41 | Bind a port, low-level implementation: 42 | ```ruby 43 | # First, create a new TCP socket. 44 | socket = Socket.new(:INET, :STREAM) 45 | # Create a C struct to hold the address for listening. 46 | addr = Socket.pack_sockaddr_in(4481, '0.0.0.0') 47 | # Bind to it. 48 | socket.bind(addr) 49 | ``` 50 | `Errno::EADDRINUSE` is raised if the port is in use already. 51 | 52 | #### What port should I bind to? 53 | - Don't bind to ports in the 0-1024 range (reserved for system use) 54 | - and 49,000-65,535 range (used by services who need port for temporary purposes) 55 | - 1025-48,999 are ok 56 | Look at the IANA list of registered ports and make sure your choice doesn't conflict. 57 | 58 | #### What address should I bind to? 59 | Most of the time you'll want `0.0.0.0` 60 | 61 | ### Servers Listen 62 | ```ruby 63 | # listen to the incoming connections 64 | socket.listen(5) 65 | ``` 66 | The argument is the number of pending connections your server will tolerate. 67 | This list of connections is called `the listen queue`. 68 | If the listen queue is full when the new client arrives, the `Errno::ECONNREFUSED` will be raised. 69 | The number of pending connections can't be bigger then `Socket::SOMAXCONN`. 70 | You may want to set it to maximum number: `server.listen(Socket::SOMAXCONN)` 71 | 72 | ### Server Accepts 73 | ```ruby 74 | server = Socket.new(:INET, :STREAM) 75 | #... 76 | # bind, listen... 77 | # Accept a connection. 78 | connection, _ = server.accept 79 | ``` 80 | Accept is blocking call. 81 | `accept` pops the next pending connection of the listen queue. 82 | Accept returns an Array. 83 | ```ruby 84 | server.accept 85 | => [#, #]` 86 | ``` 87 | The first - the connection, the second - Addrinfo (represents a host and port) 88 | 89 | Each connection is represented by a new `Socket` object, so the server socket remains untouched and continues to accept connections. 90 | 91 | #### Connection Addresses 92 | remote_address - second return value from `accept` and `remote_address` of the connection. 93 | local address - `connection.local_address` 94 | 95 | Each TCP connection is defined by this unique grouping of local-host, local-port, remote-host, and remote-port. You cannot have two 96 | simultaneous connections to a remote host if the local ports and remote ports are identical. 97 | 98 | ### The Accept Loop 99 | ```ruby 100 | # create socket, bind, listen, ... 101 | # Enter an endless loop of accepting and 102 | # handling connections. 103 | loop do 104 | connection, _ = server.accept 105 | # handle connection 106 | connection.close 107 | end 108 | ``` 109 | 110 | ### Servers Close 111 | It's important to close connections because of the: 112 | - resource usage (get rid of the stuff you don't need immediately, not relying on GC) 113 | - os limit of open file descriptors 114 | 115 | You can also close only writing of reading with `connection.close_write` or `connection.close_read` 116 | 117 | `shutdown` - another type of closing, it'll also shut down the copies of the connection (made with `Socket#dup`, which is not a very common way to copy connections, usually `Process#fork` will be used) 118 | 119 | ### Ruby Wrappers 120 | `servers = Socket.tcp_server_sockets(4481)` - returns 2 sockets, 1 to listen to Ipv4, the other - to listen to the Ipv6 121 | 122 | #### Connection handling 123 | ```ruby 124 | # Create the listener socket. 125 | server = TCPServer.new(4481) 126 | # returns an instance of `TCPServer`, which has some differences with the `Socket` interface. 127 | 128 | # Enter an endless loop of accepting and 129 | # handling connections. 130 | Socket.accept_loop(server) do |connection| 131 | # handle connection 132 | connection.close 133 | end 134 | ``` 135 | 136 | You can pass multiple listening sockets when using `Socket.accept_loop` 137 | 138 | Wrapping all into one: 139 | ```ruby 140 | Socket.tcp_server_loop(4481) { |conn| conn.close } 141 | ``` 142 | 143 | ## Chapter 4. Client Lifecycle 144 | - create - bind - connect - close 145 | 146 | ### Client binds 147 | If the socket omits the call to `bind` (the client doesn't usually call bind) it will be assigned to the port in the ephemeral range 148 | 149 | Clients don't call bind because they don't need to be accessible from a known port number - a client can connect to the server from any port. 150 | 151 | ### Clients Connect 152 | ```ruby 153 | socket = Socket.new(:INET, :STREAM) 154 | # Initiate a connection to google.com on port 80. 155 | remote_addr = Socket.pack_sockaddr_in(80, 'google.com') 156 | socket.connect(remote_addr) 157 | ``` 158 | 159 | If the client connects to the server which is not ready to accept connections, it will wait for a while and then raise `Errno::ETIMEDOUT` if the server remains unresponsive. 160 | 161 | ### Ruby Wrappers 162 | ```ruby 163 | socket = TCPSocket.new('google.com', 80) 164 | # with a block 165 | Socket.tcp('google.com', 80) do |connection| 166 | connection.write "GET / HTTP/1.1\r\n" 167 | connection.close 168 | end 169 | ``` 170 | 171 | ## Chapter 5. Exchanging Data 172 | It can be helpful to think of a TCP connection as a series of tubes connecting a local socket to a remote socket, which can send and receive chunks of data. 173 | 174 | ### Strams 175 | TCP has a stream-based nature. That's why we pass `:STREAM` option when creating a socket. No `:STREAM` option => no tcp-socket. 176 | 177 | A TCP connection provides an ordered stream of communication with no beginning and no end. 178 | 179 | ## Chapter 6. Sockets Can Read 180 | ### Simple reads: 181 | ```ruby 182 | # inside a tcp_server_loop 183 | puts connection.read 184 | ``` 185 | But this way the server will never finish reading the data and never exit. 186 | 187 | ### Read Length 188 | ```ruby 189 | while data = connection.read(one_kb) do 190 | puts data 191 | end 192 | ``` 193 | You can read a certain amount of data, but if the client responds with less than that amount, the server will continue waiting. 194 | 195 | ### The EOF Event 196 | ```ruby 197 | client.write('gekko') 198 | client.close # close the socket => send the EOF 199 | ``` 200 | 201 | ### Partial Reads 202 | You pass the maximum length of data to the `readpartial` 203 | ```ruby 204 | Socket.tcp_server_loop(4481) do |connection| 205 | begin 206 | # Read data in chunks of 1 hundred kb or less. 207 | while data = connection.readpartial(one_hundred_kb) do 208 | puts data 209 | end 210 | rescue EOFError 211 | end 212 | ``` 213 | 214 | ## Chapter 7. Sockets Can Write 215 | ```ruby 216 | Socket.tcp_server_loop(4481) do |connection| 217 | # Simplest way to write data to a connection. 218 | connection.write('Welcome!') 219 | connection.close 220 | end 221 | ``` 222 | 223 | ## Chapter 8. Buffering 224 | ### Write Buffers 225 | When you `write` successfully it doesn't mean that the data is sent. It only means that the data was passed to the OS kernel. 226 | The kernel will decide whether to send it as is or combine it with other data. 227 | Buffering is helpful for perfomance, cause the network is very slow. 228 | So you can `write` as much as you need and the kernel will figure out the rest. 229 | (unless you send large files or bigdata, then you need to split it cause of the RAM) 230 | 231 | ### Read Buffers 232 | The optimal size to read dependa on the data you read. 233 | Mongrel, Unicorn, Puma, Passenger, and Net::HTTP do `readpartial(1024 * 16)` (16kb) 234 | redis-rb reads 1kb 235 | 236 | ## Chapter 9. Our First Client/Server 237 | ## Chapter 10. Socket Options 238 | ### SO_TYPE 239 | Socket type 240 | ```ruby 241 | socket = TCPSocket.new('google.com', 80) 242 | # Get an instance of Socket::Option representing the type of the socket. 243 | opt = socket.getsockopt(Socket::SOL_SOCKET, Socket::SO_TYPE) 244 | opt.int == Socket::SOCK_STREAM #=> true 245 | opt.int == Socket::SOCK_DGRAM #=> false 246 | 247 | # same 248 | opt = socket.getsockopt(:SOCKET, :TYPE) 249 | ``` 250 | 251 | ### SO_REUSE_ADDR 252 | - it's OK for another socket to bind to the same local address that the server is using if it's currently in the TCP TIME_WAIT state. 253 | 254 | ### TIME_WAIT state 255 | - when you `close` the socket, but there is some data in the buffers. 256 | If another socket tries to bind on the same address, the `Errno::EADDRINUSE` will be raised. 257 | But if you pass `SO_REUSE_ADDR`, you'll allow you to bind to an address still in use by another socket in the TIME_WAIT state. 258 | 259 | ```ruby 260 | server.setsockopt(:SOCKET, :REUSEADDR, true) 261 | server.getsockopt(:SOCKET, :REUSEADDR) #=> true 262 | ``` 263 | 264 | ## Chapter 11. Non-blocking IO 265 | ### Non-blocking Reads 266 | `read` - blocks until it receives EOF 267 | `readpartial` - returns available data immediately, blocks until then 268 | `read_nonblock` - never blocks 269 | 270 | `Errno::EAGAIN` will be raised if there is no data available to read. 271 | 272 | Calling `IO.select` with an Array of sockets as the first argument will block until one of the sockets becomes readable. So `retry` will be called when there's available data to read. 273 | ```ruby 274 | begin 275 | connection.read_nonblock(4096) 276 | rescue Errno::EAGAIN 277 | IO.select([connection]) 278 | retry 279 | end 280 | ``` 281 | Using `IO.select` gives the flexibility to monitor multiple sockets simultaneously or periodically check for readability while doing other work. 282 | 283 | `read_nonblock` method first checks the buffers. If there's some data, it returns immediately. 284 | If not, it asks the kernel for the data (using `select(2)`). Reads, if data is available. If not available - `read` blocks, `read_nonblock` raises an exception. 285 | 286 | ### Non-blocking Writes 287 | `writes` - writes all data available 288 | `write_nonblock` - may return a partial write 289 | 290 | Write blocks when: 291 | - The receiving end of the TCP connection has not yet acknowledged 292 | receipt of pending data. 293 | - The receiving end of the TCP connection cannot yet handle more data. 294 | 295 | ### Non-blocking Accept 296 | Accept just pops a connection off of the listen queue. 297 | If there's nothing on the queue, `accept` will block, `accept_nonblock` will raise `EAGAIN` 298 | 299 | ### Non-blocking Connect 300 | If connect_nonblock cannot make an immediate connection to the remote host, then it actually lets the operation continue in the background and raises `Errno::EINPROGRESS` 301 | 302 | ## Chapter 12. Multiplexing Connections 303 | - working with multiple active sockets at the same time. 304 | Prevent ourselves from getting stuck blocking a particular socket. 305 | 306 | `IO.select(for_reading, for_writing, for_writing)` 307 | first argument - an Array of IO objects which you want to read from 308 | second argument - an Array of IO objects which you want to write to 309 | third argument - an Array of IO objects for which you are interested in exceptional conditions 310 | 311 | returns an Array of Arrays Arrays. 312 | first element - array of IO objects that can be read from w/o blocking 313 | second - array of IO objects that can be written to without blocking 314 | third - array of IO objects which have applicable exceptional conditions. 315 | 316 | `IO.select` is blocking (sync method call). 317 | Blocks until the status of one of the passed `IO` objects changes. 318 | 319 | Fourth argument - timeout, prevents from blocking infinitely. 320 | If the timeout is reached before any of the IO statuses have changed, `IO.select` will return `nil` . 321 | 322 | You can also pass plain Ruby objects to IO.select , so long as they respond to `to_io` and return an IO object 323 | 324 | ### Events Other Than Read/Write 325 | 326 | ### High Performance Multiplexing 327 | There're more performant ways to do multiplexing with lots of connections besides `IO`. Most OS kernels have other methods than `select`, e.g. `epoll`(Linux) or `kqueue`. 328 | E.g. `nio4r` gem. 329 | 330 | ## Chapter 13. Nagle's algorithm 331 | - optimization applied to all TCP connections by default 332 | Applicable to apps which don't do buffering and send small amounts of data at a time. 333 | 334 | If you're working with a protocol like HTTP, where the request/response are large enough, this algorithm will take no effect. It's benefitial for the specific situations like implementing a telnet. 335 | 336 | Every Ruby web server disables Nagle's algorithm by default: `server.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1)` 337 | 338 | ## Chapter 14. Framing Messages 339 | Client/server should have an agreed-upon way to frame the beginning and end of messages, so that they wouldn't have to open a new connection each time. 340 | 341 | Reusing connections across multiple messages is the same concept behind the familiar keep-alive feature of HTTP. 342 | 343 | ### Using newlines 344 | `request = connection.gets` 345 | 346 | One real-world protocol that uses newlines to frame messages is HTTP. 347 | 348 | ### Using A Content Length 349 | The sender of the message first denotes the size of their message, packs that into a fixed-width integer representation and sends that over the connection, immediately followed by the message itself. 350 | 351 | Using this method the client/server always communicate using the same number of bytes for the message size. 352 | 353 | ## Chapter 15. Timeouts 354 | `IO.select([connection], nil, nil, timeout)` - select with timeout. 355 | 356 | These IO.select based timeout mechanisms are commonly used, even in Ruby's standard library, and offer more stability than something like native socket timeouts. 357 | 358 | ## Chapter 16. DNS Lookups 359 | ```ruby 360 | socket = TCPSocket.new('google.com', 80) 361 | ``` 362 | Ruby needs to do the DNS lookup, which may be slow and it can block your entire Ruby process. 363 | 364 | ### MRI and the GIL 365 | GIL - when 1 thread is active, the others are blocked. 366 | But it understands blocking IO, when a thread is doing a blocking IO, another thread can continue execution. 367 | 368 | But any time that a library uses the C extension API, the GIL blocks any other code from execution. 369 | Ruby uses a C extension for DNS lookups, so it won't release the GIL! 370 | 371 | `resolv` is a pure-ruby implementation. 372 | `require 'resolv-replace'` - monkeypatches the Socket classes to use 'resolv'. 373 | 374 | 375 | ## Chapter 17. SSL Sockets 376 | SSL provides a mechanism for exchanging data securely over sockets using public key cryptography. 377 | You can upgrade "plain" sockets to secure SSL sockets. 378 | A single socket can't do both SSL and non-SSL communication. 379 | SSL-certificates. 380 | 381 | The SSL source would provide you with the cert and key. 382 | ```ruby 383 | # Create the TCP client socket. 384 | socket = TCPSocket.new('0.0.0.0', 4481) 385 | ssl_socket = OpenSSL::SSL::SSLSocket.new(socket) 386 | ssl_socket.connect 387 | ssl_socket.read 388 | ``` 389 | The server builds the SSL context and sets the `verify_mode`. 390 | 391 | ## Chapter 18. Network Architecture Patterns 392 | ### The Muse 393 | One 'control' socket is used for sending FTP commands and their arguments between server and client. 394 | Each time a file transfer needs to be made, a new TCP socket is used. 395 | `connection.respond` writes response out on the connection. 396 | FTP uses a `\r\n` to signify message boundaries, just like HTTP. 397 | 398 | ## Chapter 19. Serial 399 | 1) Client connects. 400 | 2) Client/server exchange requests and responses. 401 | 3) Client disconnects. 402 | 4) Back to step #1. 403 | 404 | Advantage - simplicity. 405 | Disadvantage - no concurrency, slow response 406 | 407 | ## Chapter 20. Process per Connection 408 | After accepting a connection, the server will fork a child process whose sole purpose will be the handling of that new connection. The child process handles the connection, then exits. 409 | 410 | `fork` - create a new process at runtime. A new process (child) will be an exact copy if the original (parent). Then they can go their separate ways and do whatever they need to do. 411 | 412 | At any given time there'll be a single parent process waiting to accept connections and many children handling individual connections. 413 | 414 | `Process.detach(pid)` after `fork` - stop caring about the child process, its resources will be fully cleaned up when it exits 415 | 416 | ### Considerations 417 | Advantages: simplicity, complete separation (no locks or race condition). 418 | Disadvantag: no upper bound for the forked processes. May be a problem for the large number of clients. 419 | 420 | Examples - shotgun, inetd 421 | 422 | ## Chapter 21. Thread per Connection 423 | Like Process Per Connection, but spawn a thread instead of spawning a process. 424 | 425 | ### Threads vs. Processes 426 | Spawning: threads are much cheaper to spawn, they share memory. 427 | Synchronizing: need mutexes, locks, and synchronizing access between threads when working with data structures that will be accessed by multiple threads. 428 | Processes don't need them, cause they have their own copy of everything. 429 | Parallelism: theads' parallelism is influenced by the MRI GIL. 430 | 431 | ### Considerations 432 | Advantages: simplicity, no need to care about synchronizing (cause each connection is handled by a single, independent thread) 433 | Lighter on resources. 434 | Disadvantage: the number of threads can grow and grow until the system is overwhelmed. 435 | Less parralel execution cause of the GIL. 436 | 437 | Examples: webrick, mongrel. 438 | 439 | ## Chapter 22. Preforking 440 | Forks a bunch of processes when the server boots up, before any connections arrive. 441 | 1) Main server process creates a listening socket. 442 | 2) Main server process forks a horde of child processes. 443 | 3) Each child process accepts connections on the shared socket and handles them 444 | independently. 445 | 4) Main server process keeps an eye on the child processes. 446 | 447 | The kernel balances the load and ensures that one, and only one, copy of the socket will be able to accept any particular connection. 448 | 449 | Forwards an INT signal (interruption) received by the parent to its child processes, cleans up children. 450 | ```ruby 451 | trap(:INT) { 452 | # kill child processes 453 | } 454 | ``` 455 | In case a child exits (unusial situation): 456 | ```ruby 457 | loop do 458 | pid = Process.wait 459 | $stderr.puts "Process #{pid} quit unexpectedly" 460 | child_pids.delete(pid) 461 | child_pids << spawn_child 462 | end 463 | ``` 464 | Some preforking servers (e.g. unicorn) take more active role in caring in monitoring their children. E.g. they can see if a child process takes a long time to process a request. In this case the parent can kill the child process and spawn a new one. 465 | 466 | ### Considerations 467 | Prevents too many processes from being spawned (not like in the "process per connection"). 468 | Advantage over a threaded pattern -- complete separation. 469 | Disadvantage: more processes => more memory (they're expensive!) 470 | 471 | Example - unicorn. 472 | 473 | ## Chapter 23. Thread Pool 474 | Spawn a number of threads when the server boots and defer connection handling to each independent thread. 475 | Same as "preforking", but with threads. 476 | 477 | Example - puma. 478 | 479 | ## Chapter 24. Evented (Reactor) 480 | 1. The server monitors the listening socket for incoming connections. 481 | 2. Upon receiving a new connection it adds it to the list of sockets to monitor. 482 | 3. The server now monitors the active connection as well as the listening socket. 483 | 4. Upon being notified that the active connection is readable the server reads a chunk of data from that connection and dispatches the relevant callback. 484 | 5. Upon being notified that the active connection is still readable the server reads another chunk and dispatches the callback again. 485 | 6. The server receives another new connection; it adds that to the list of sockets to monitor. 486 | 7. The server is notified that the first connection is ready for writing, so the response is written out on that connection. 487 | 488 | The Evented pattern is single-threaded so it needs to represent each independent connection with an object so they don't end up trampling on each other's state. 489 | 490 | The connection stores the actual underlying IO object in its @client instance variable and makes it accessible to the world with an `attr_reader`. 491 | 492 | When the Reactor reads data from the client connection, checks to see if it's received a complete request. If it has then it asks the `@handler` to build the response and assigns that to `@response`. 493 | 494 | When the client connection is ready to be written to, it writes what it can from the `@response` out to the client connection 495 | 496 | The `@handles` Hash looks something `{6 => #}`where the keys are file descriptor numbers and the values are Connection objects. 497 | 498 | In the main loop: 499 | - ask each of the active connections if they want to be monitored for reading or writing 500 | - grab a reference to the underlying IO object for each of the eligible connections. 501 | - pass these IO instances to `IO.select` with no timeout 502 | - sneak the @control_socket into the connections to monitor for reading so it can detect new incoming client connections 503 | 504 | Handle readable sockets: 505 | - `@control_socket` is readable => it's a new client connection; `accept` and add `Connection` to `@handles` 506 | - other readable socket => read the data and trigger the `on_data` method of 507 | the appropriate Connection 508 | - 'writable' sockets => call `on_writable` 509 | 510 | ### Considerations 511 | Advantages: 512 | - is able to handle extremely high levels of concurrency, thouthands of connections 513 | - no need to worry about the syncronization 514 | Disadvantage: 515 | - never block the Reactor 516 | E.g you need to retreive data from the remote server and the link is slow. That would block the Reactor! 517 | You need to make sure that any blocking IO is handled by the Reactor itself. 518 | The slow remote connection would need to be encapsulated inside its own subclass of `Connection` that defined its own on_data and on_writable methods. 519 | 520 | It requires you to rethink all of the IO that your app does. 521 | 522 | Examples: 523 | - EventMachine 524 | - Celluloid::IO 525 | - Twisted 526 | 527 | ## Chapter 25. Hybrids 528 | ### nginx 529 | - an extremely high-performance network server written in C, is often used in the Ruby world as an HTTP proxy in front of web application servers, but it can speak HTTP, SMTP, and others. 530 | 531 | nginx uses the Preforking pattern, but in each of the forked processes it uses an Evented pattern. 532 | 533 | ### Puma 534 | The puma rubygem provides "a Ruby web server built for concurrency". 535 | At a high level Puma uses a Thread Pool to provide concurrency. 536 | 537 | Puma's request handling is always done by a pool of threads (thead pool). This is supported by a reactor that monitors any persistent connections (`keep-alive`). 538 | 539 | ### EventMachine 540 | - an event-driven I/O library. 541 | At its core EventMachine uses the Evented pattern. There is a single- 542 | threaded event loop that can handle network events on many concurrent connections. For the long-running or blocking operations it uses a Thread Pool. 543 | 544 | #### (My) Questions 545 | - What options can you pass when creating a socket in ruby? 546 | - What's a loopback? 547 | - What are the 2 points of the TCP connection? 548 | - What's the server socket lifecycle? 549 | - What port should you choose to bind a server socket? 550 | - What is the argument for `socket.listen`, how do you decide it's value? 551 | - What does the server accept do? 552 | - What arguments does it return? 553 | - What is the client lifecycle? 554 | - Why doesn't client usually call bind? 555 | - What happens if the socket doesn't call bind? 556 | - How does the server read? What problems may appear? 557 | - What's the difference between `readpartial` and `read`? 558 | - Why buffer? 559 | - How do you find out the amount of the data to send (at once)? 560 | - -||- to read? 561 | - what's the difference between `read`, `read_partial` and `read_nonblock`? 562 | - when `EAGAIN` is raised? 563 | - What's the benefit of using `IO.select` 564 | - When would a `read` block and what does `read_nonblock` do instead? 565 | - When would a `write` block? 566 | - What's multiplexing? 567 | - How does `IO.select` work? 568 | - What arguments does `IO.select` accept? 569 | - What does `IO.select` return? 570 | - What happens if the socket receives EOF, Accept or Connect while being monitored? 571 | - When is Nagle's algorithm useful? 572 | - How to signal the end of one message and the beginning of another if you use one connection to send multiple messages (how to frame)? 573 | - How to use `IO.select` with timeout? 574 | - What network architecture patterns are there? (a server which handles multiple connections) 575 | - How is Reactor pattern different from the others? 576 | - What patterns does `nginx` uses at it's core? 577 | - -||- `puma`? How does it handle keep-alive and non-keep-alive connections? -------------------------------------------------------------------------------- /development/working_with_unix_processes.md: -------------------------------------------------------------------------------- 1 | # Working with Unix Processes 2 | 3 | manpages sections 4 | 5 | • Section 1: General Commands 6 | • Section 2: System Calls 7 | • Section 3: C Library Functions 8 | • Section 4: Special Files 9 | 10 | ## Chapter 3. Processes: The Atoms of Unix 11 | *any code that is executed happens inside a process* 12 | One process can spawn and manage many others. 13 | 14 | ### Processes Have IDs 15 | Every process running on your system has a unique process identifier, hereby referred to as 'pid'. 16 | 17 | ```ruby 18 | puts Process.pid 19 | ``` 20 | 21 | ### Cross Referencing 22 | ```bash 23 | ps -p 24 | ``` 25 | 26 | ### In the Real World 27 | Process id can often be written into logs. 28 | You can also find the information about the process by the commands like `top` or `lsof`. 29 | 30 | ### System Calls 31 | `$$` - global variable that holds the value of the current PID. 32 | But `Process.pid` is much more expressive. 33 | 34 | ## Chapter 4. Processes Have Parents 35 | ```ruby 36 | puts Process.ppid 37 | ``` 38 | Parent for the `irb` process will be a bash process. 39 | There's not much real use of the `ppid`. It may be useful for detecting daemon processes. 40 | 41 | ## Chapter 5. Processes Have File Descriptors 42 | Pids represent running processes, and file descriptors represent open files. 43 | 44 | ### Everything is a file 45 | Devices, sockets, pids and pipes are treated like files. 46 | You can use the word 'resource' when talking about files in the general sense. 47 | 48 | ### Descriptors Represent Resources 49 | Any time that you open a resource in a running process it is assigned a file descriptor number. 50 | File descriptors are not shared between processes, they live and die inside a process. 51 | In Ruby, open resources are represented by the IO class. 52 | ```ruby 53 | passwd = File.open('/etc/passwd') 54 | puts passwd.fileno 55 | ``` 56 | Any resource is given a unique number, the kernel keeps track of the resources with the help of these numbers. 57 | 58 | - File descriptors are assigned the lowest unused number 59 | - once a resource is closed its file descriptor number becomes available again. 60 | 61 | Closed resources are not given a file descriptor number. 62 | An attempt to access it will cause an exception. 63 | ```ruby 64 | passwd = File.open('/etc/passwd') 65 | puts passwd.fileno 66 | passwd.close 67 | puts passwd.fileno 68 | # => closed stream (IOError) 69 | ``` 70 | 71 | ### Standard Streams 72 | Every Unix process comes with three open resources: 73 | - STDIN - standart input 74 | - STDOUT - standart output 75 | - STDERR - standart error 76 | 77 | ```ruby 78 | puts STDIN.fileno 79 | puts STDOUT.fileno 80 | puts STDERR.fileno 81 | ``` 82 | 83 | Many methods on Ruby's IO class map to system calls of the same name. 84 | E.g. open(2), close(2), read(2), write(2), pipe(2), fsync(2), stat(2). 85 | 86 | ## Chapter 6. Processes Have Resource Limits 87 | There are some resource limits imposed on a process by the kernel. 88 | The maximum number of allowed file descriptors: 89 | ```ruby 90 | p Process.getrlimit(:NOFILE) 91 | # => [1024, 1048576] 92 | ``` 93 | ### Soft Limits vs. Hard Limits 94 | First element -- soft limit. 95 | If it will be exceeded, the exception will be raised. 96 | The hard limit is a large number, you'll probably won't be able to open so much resources because of the hardware constraints. 97 | 98 | ### Bumping the Soft Limit 99 | You can change the soft limit: 100 | ```ruby 101 | Process.setrlimit(:NOFILE, 4096) 102 | # will set both soft and hard limits to 4096 103 | ``` 104 | 105 | Exceeding the soft limit will raise `Errno::EMFILE` 106 | 107 | ### Other Resources 108 | Check and modify limits of other system resources: 109 | ```ruby 110 | # The maximum number of simultaneous processes 111 | # allowed for the current user. 112 | Process.getrlimit(:NPROC) 113 | # The largest size file that may be created. 114 | Process.getrlimit(:FSIZE) 115 | # The maximum size of the stack segment of the 116 | # process. 117 | Process.getrlimit(:STACK) 118 | ``` 119 | 120 | ### In the Real World 121 | You won't need to change our process limits often. 122 | One of the cases is: dealing with a process which needs to handle thouthands of network connections, e.g. to test performance. 123 | You can also want to change limits when dealing with some third-party code. 124 | 125 | ### System Calls 126 | `Process.getrlimit` `Process.setrlimit` 127 | 128 | ## Chapter 7. Processes Have an Environment 129 | Environment variables are key-value pairs that hold data for a process. 130 | Processes inherit env variables from their parents. 131 | Environment variables are per-process and are global to each process. 132 | 133 | ```bash 134 | $ MESSAGE='wing it' ruby -e "puts ENV['MESSAGE']" 135 | ``` 136 | Equal to: 137 | ```ruby 138 | ENV['MESSAGE'] = 'wing it' 139 | system "echo $MESSAGE" 140 | ``` 141 | 142 | ### Hash like 143 | `ENV` uses hash-like API but is not a hash. 144 | It implements `Enumerable` and some of the hash API, but not all `Hash` methods are implemented, so don't rely on it. 145 | 146 | ```ruby 147 | puts ENV['EDITOR'] 148 | puts ENV.has_key?('PATH') 149 | puts ENV.is_a?(Hash) 150 | ``` 151 | 152 | ### In the Real World 153 | ```bash 154 | RAILS_ENV= production rails server 155 | ``` 156 | 157 | Environment variables are often used as a generic way to accept input into a command-line program. Using environment variables is often less overhead than explicitly parsing command line options. 158 | 159 | ### System calls 160 | C-library functions like `setenv(3)`, `getenv(3)`, also `environ(3)` 161 | 162 | ## Chapter 8. Processes Have Arguments 163 | Every process has access to a special array called `ARGV`. 164 | argv - Argument Vector. 165 | ```bash 166 | $ cat argv.rb 167 | p ARGV 168 | $ ruby argv.rb foo bar -va 169 | ["foo", "bar", "-va"] 170 | ``` 171 | `argv` is simply an `Array`. 172 | 173 | ### In the Real World 174 | Mostly accepting filenames or parsing command line command line input. 175 | You can use libraries like `optparse` or parse them by hand. 176 | 177 | ## Chapter 9. Processes Have Names 178 | Unix processes have very few inherent ways of communicating about their state. 179 | - Logfiles, but they operate on the filesystem level and are not inherent to the processes themselves 180 | - Using network (opening sockets) to communicate with other processes, but it also operates on the different level 181 | On the process level such mechanisms are: process names and exit codes. 182 | 183 | ### Naming Processes 184 | Every process has its name, e.g. `irb` 185 | You can access process name with the `$PROGRAM_NAME` variable. 186 | You can change it and so, change the name of the current process. 187 | 188 | In ruby you can only change `$PROGRAM_NAME` or its alias `$0`, no other ways to change the process name. 189 | 190 | ## Chapter 10. Processes Have Exit Codes 191 | When a process ends it has a last chance to make a mark: it's exit code (0 - 255) 192 | 0 is usually for succesful exits. 193 | 194 | ### How to Exit a Process 195 | #### exit 196 | ```ruby 197 | # hook that will be executed before exit 198 | at_exit { puts 'Last!' } 199 | # setting a custom exit code 200 | exit 4 201 | ``` 202 | #### exit! 203 | Sets an unsuccessful status code by default (1). 204 | Won't run `at_exit` hook. 205 | ```ruby 206 | # ypu can specify a custom code 207 | exit! 3 208 | ``` 209 | 210 | #### abort 211 | Kernel#abort provides a generic way to exit a process unsuccessfully. 212 | It'll set the exit code to 1. 213 | 214 | `at_exit` block will be invoked 215 | ```ruby 216 | # you can set a message which will be printed to STDERR 217 | abort "Something went horribly wrong." 218 | ``` 219 | 220 | #### raise 221 | If an unhandled exception will be raised, it'll cause the process to end. 222 | `Kernel#raise` won't end the process immediately, cause an exception may be rescued later. 223 | 224 | ## Chapter 11. Processes can fork 225 | `fork(2)` - creating a copy of the existing process. 226 | The new process will be a child of the current one. 227 | It'll have a copy of all the memory used by and file descriptors belonging to the parent process. 228 | In this way the two processes can share open files, sockets, etc. 229 | 230 | Can be used with `if` (not very common): 231 | ```ruby 232 | puts "parent process pid is #{Process.pid}" 233 | if fork 234 | puts "entered the if block from #{Process.pid}" 235 | else 236 | puts "entered the else block from #{Process.pid}" 237 | end 238 | # parent process is 21268 239 | # entered the if block from 21268 240 | # entered the else block from 21282 241 | ``` 242 | The if block is being executed by the parent process, while the code in the else block is being executed by the child process. The child process will exit after executing the else branch, the parent process will continue to run. 243 | 244 | `puts fork` returns 2 different values. First is a child process pid (returned from parent), second - `nil` from the child process 245 | 246 | ### Multicore Programming? 247 | Your code is able, but not guaranteed, to be distributed across multiple CPU cores. 248 | Forking may lead to a `fork bomb`, cause it will need a n*parent_process_memory memory. 249 | 250 | ### Using a Block 251 | `fork` is more commonly used with a block in ruby: 252 | ```ruby 253 | fork { puts Process.pid } 254 | ``` 255 | 256 | ### System Calls 257 | `Kernel#fork` maps to `fork(2)` 258 | 259 | ## Chapter 12. Orphaned Processes 260 | ### Out of Control 261 | When this code is executed, a parent will die before the child process is finished. 262 | The child process becomes orphaned, but it'll continue doing its job (printing into STDOUT). 263 | ```ruby 264 | fork do 265 | 5.times do 266 | sleep 1 267 | puts "I'm an orphan!" 268 | end 269 | end 270 | abort "Parent process died..." 271 | ``` 272 | 273 | ### Abandoned Children 274 | The OS doesn't treat children differently then other processes. 275 | 276 | ### Managing Orphans 277 | - daemon processing 278 | - managing processes that're not attached to the terminal (see in later chapters) 279 | 280 | ## Chapter 13. Processes Are Friendly 281 | ### Being CoW Friendly 282 | CoW delays the actual copying of memory until it needs to be written. 283 | CoW-friendly forking: on `fork` process memory is not copied. It's only copied when that data needs to be modified. 284 | 285 | ```ruby 286 | arr = [1, 2, 3] 287 | fork { p arr } # no copying 288 | fork { arr.push 4 } # copying cause of the need to write 289 | ``` 290 | Before the version 2.0, ruby used Mark-and-Sweep garbage collector, which needed to modify data while doing it's "mark" step, so the CoW wouldn't be effective. 291 | Since ruby 2.0 it change its GC method, so now forking is CoW-friendly. 292 | 293 | ## Chapter 14. Processes Can Wait 294 | The technique when the parent process exits before the child one is only suitable for the "fire and forget" situation. 295 | It's useful to do smth asynchronously in a child process. 296 | 297 | ### Babysitting 298 | But sometimes we need to keep track on the child processes. 299 | `Process.wait` is a blocking call instructing the parent process to wait for one of its child processes to exit before continuing. 300 | 301 | ```ruby 302 | fork { 5.times { sleep 1; p 'hi'; } } 303 | Process.wait 304 | abort 'end1' 305 | ``` 306 | 307 | ### Process.wait and Cousins 308 | Sometimes you have several children and want to know which one exited. 309 | Then you can use the `Process.wait` return value (pid) 310 | 311 | ### Communicating with Process.wait2 312 | `Process.wait2` returns `pid` and `status`. 313 | The `status` returned from `Process.wait2` is an instance of `Process::Status`, it has a lot of useful information attached to it. 314 | 315 | ### Waiting for Specific Children 316 | You can use `Process.waitpid` and `Process.waitpid2` to wait for the specific children. 317 | 318 | Actually `Process.wait` and `Process.waitpid` are aliases. So are `Process.wait2` and `Process.waitpid2` 319 | You can pass pid to `wait`, `wait2` and you can pass `-1` to `waitpid`, `waitpid2` to say "no pid". 320 | 321 | But you should use these methods as specified before to show clear intentions. 322 | 323 | ### Race Conditions 324 | If a child process exits before the `Process.wait` is executed, nothing happens. 325 | So the code is free of race conditions. 326 | But calling `Process.wait`, when there are no child processes, will raise `Errno::ECHILD`. It's always a good idea to keep track of how many child processes you have created. 327 | 328 | ### Real World 329 | The idea of looking in on your child processes is at the core of a common Unix programming pattern. It's sometimes called babysitting processes, master/ 330 | worker, or preforking. 331 | 332 | The core: a parent process forks several child processes and keeps track of them. 333 | E.g. `Unicorn` uses this pattern. 334 | 335 | ### System Calls 336 | `Process.wait` => `waitpid(2)` 337 | 338 | ## Chapter 15. Zombie Processes 339 | `Process.wait` can be called even the long time after the process finished. 340 | So the Kernel will retain the status of exited child processes until the parent process will call `Process.wait`. 341 | So if you don't `wait`, the kernel will keep that information forever. It's not a good use of the resources. 342 | If you want to fork processes in "fire and forget" manner, you need to detouch them. 343 | 344 | ```ruby 345 | pid = fork { sleep 1 } 346 | puts pid 347 | sleep 348 | ``` 349 | 350 | ```bash 351 | ps -ho pid,state -p [pid of a zombie process] 352 | # => Z 353 | ``` 354 | Any dead process whose status hasn't been waited on is a zombie process. 355 | 356 | gem `spawnling` - makes sure that the processes are properly detauched. 357 | 358 | ## Chapter 16. Processes Can Get Signals 359 | `Process.wait` is a nice way to keep track of children, but it's a blocking call. 360 | What if the parent wants to do it's own job while keeping track of children? 361 | 362 | ### Trapping SIGCHILD 363 | ```ruby 364 | ... 365 | # By trapping the :CHLD signal our process will be notified by the kernel 366 | # when one of its children exits. 367 | trap(:CHLD) do 368 | # Since Process.wait queues up any data that it has for us we can ask for it 369 | # here, since we know that one of our child processes has exited. 370 | puts Process.wait 371 | dead_processes += 1 372 | # We exit explicitly once all the child processes are accounted for. 373 | exit if dead_processes == child_processes 374 | end 375 | # ... parent process does some job 376 | ``` 377 | ### SIGCHILD and Concurrency 378 | Signal delivery is unreliable. If your code is handling a CHLD signal while another child process dies you may or may not receive a second CHLD signal. 379 | 380 | To properly handle CHLD you must call Process.wait in a loop and look for as many dead child processes as are available. 381 | `Process.wait` is a blocking call, but you can pass a second argument to it. 382 | `Process.wait(-1, Process::WNOHANG)` - nonblocking. 383 | 384 | ```ruby 385 | child_processes = 3 386 | dead_processes = 0 387 | 388 | #... forking 389 | 390 | $stdout.sync = true 391 | trap(:CHLD) do 392 | # Since Process.wait queues up any data that it has for us we can ask for it 393 | # here, since we know that one of our child processes has exited. 394 | # We loop over a non-blocking Process.wait to ensure that any dead child 395 | # processes are accounted for. 396 | begin 397 | while pid = Process.wait(-1, Process::WNOHANG) 398 | puts pid 399 | dead_processes += 1 400 | # We exit ourselves once all the child processes are accounted for. 401 | puts 'exiting'; exit if dead_processes == child_processes 402 | end 403 | rescue Errno::ECHILD 404 | puts 'rescued' 405 | end 406 | end 407 | #... parent job 408 | ``` 409 | ### Signals Primer 410 | Signals are asynchronous communication. 411 | When a process receive a signal from the kernel it can: ignore the signal, perform a specified action, perform the default action. 412 | 413 | ### Where do Signals Come From? 414 | Signals are sent from one process to another process, using the kernel as a middleman. 415 | The original purpose of signals was to specify different ways that a process should be killed. 416 | Killing one process from another: `Process.kill(:INT, )` 417 | 418 | ### The Big Picture 419 | Default actions of the signals: 420 | `Term` - terminate 421 | `Core` - terminate and dump core 422 | `Ign` - ignore the signal 423 | `Stop` - pause 424 | `Cont` - resume 425 | See the table in the book for the whole picture. 426 | Signal names start with `SIG` (`SIGINT`), but this part is optional. 427 | 428 | By default most signals will terminate a process. 429 | `SIGUSR1` and `SIGUSR2` signals - defined specifically for your process. 430 | 431 | ### Redefining Signals 432 | `trap(:INT) { print "Na na na, you can't get me" }` 433 | => you won't be able to interrupt a process from the code or with `Ctrl + C` 434 | 435 | ### Ingoring 436 | `trap(:INT, "IGNORE")` 437 | 438 | ### Signal Handlers are Global 439 | Trapping a signal is a bit like using a global variable, it's only suitable in certain situations. 440 | 441 | ### Being Nice about Redefining Signals 442 | There is no way to preserve system default behaviour, but you can preserve other Ruby handlers that have been defined. 443 | 444 | Preserve any previous QUIT handlers that have been defined. 445 | ```ruby 446 | old_handler = trap(:QUIT) { 447 | # do some cleanup 448 | puts 'All done!' 449 | old_handler.call if old_handler.respond_to?(:call) 450 | } 451 | ``` 452 | Process can receive signals at any given time. 453 | 454 | ### In the Real World 455 | With signals, any process can communicate with any other process on the system, if it knows its pid. 456 | Signals are mostly used by long running processes like servers and daemons. 457 | Unicorn -- see `SIGNALS` file. 458 | 459 | ## Chapter 17. Processes Can Communicate 460 | IPC - inter-process communication 461 | 2 most common ways: pipes and socket pairs 462 | 463 | ### Pipes 464 | A pipe is a unidirectional stream of data. One process is its "end" and the other is the second "end". 465 | One process is the reader, another is the writer, not vice versa. 466 | 467 | ```ruby 468 | reader, writer = IO.pipe # [IO, IO] 469 | writer.write("Into the pipe I go...") 470 | writer.close 471 | puts reader.read 472 | ``` 473 | `IO` is a superclass to `File`, `TCPSocket`, `UDPSocket` and others. 474 | If you skip closing the writer then the reader will lock and continue to read indefinitely. 475 | 476 | ### Sharing Pipes 477 | You can use a pipe to communicate between child and parent processes. 478 | 479 | ### Streams vs. Messages 480 | In IO there's no beginning or end. 481 | When reading data from that IO stream you read it in one chunk at a time, stopping when you come across the delimiter 482 | 483 | ### Sockets: 484 | ```ruby 485 | require 'socket' 486 | child_socket, parent_socket = Socket.pair(:UNIX, :DGRAM, 0) #=> [#, #] 487 | ``` 488 | `process.recv` => read 489 | `process.send` => write 490 | 491 | ### Remote IPC? 492 | IPC - communication only between processes on the same machine. 493 | Remote IPC: 494 | - TCP sockets 495 | - RPC 496 | - ZeroMQ 497 | 498 | ## Chapter 18. Daemon Processes 499 | Daemon processes - processes that run in the background, not under the control of a user or at a terminal. 500 | Many processes are running in the background when the os is running. 501 | 502 | ### The First Process 503 | When the kernel is bootstrapped it spawns a process called the `init` process. 504 | It has `ppid` 0 and `pid` 1. It's the grandparent of all processes. 505 | 506 | ### Creating a Daemon Process 507 | Example - `rackup` command, it has an option to daemonize the process. 508 | In ruby `1.9+` there is a command `Process.daemon`, but what it does exactly? 509 | 510 | ```ruby 511 | # exit the parent process, keep the child running 512 | exit if fork 513 | # The process becomes a session leader of a new session 514 | # The process becomes the process group leader of a new process group 515 | # The process has no controlling terminal 516 | # returns the id of the new session group it create 517 | Process.setsid 518 | # The forked process that had just become a process group and session group leader forks again and then exits, 519 | exit if fork 520 | # changes the current working directory to the root directory 521 | # avoids problems where the directory that the daemon was started from gets deleted or unmounted 522 | Dir.chdir "/" 523 | # sets all the standart streams to /dev/null 524 | # daemon doesn't use them 525 | # but some programs expect them to always be available 526 | STDIN.reopen "/dev/null" 527 | STDOUT.reopen "/dev/null", "a" 528 | STDERR.reopen "/dev/null", "a" 529 | ``` 530 | 531 | When a process is orphaned it's `ppid` will be 1. 532 | 533 | ### Process Groups and Session Groups 534 | Refer to how processes are handled by the terminal. 535 | Each process belongs to a group, and each group has a unique integer id. 536 | Usually a group is a parent with children, but you can also set a group arbitrarily: 537 | ```ruby 538 | Process.setpgrp(new_group_id) 539 | ``` 540 | If you fork a process, the group id will be the parent process pid. 541 | When the parent process is being controlled by a terminal and is killed by a signal the child processes will be also killed. 542 | 543 | #### Session Groups 544 | A session group is one level of abstraction higher up then the process group. 545 | E.g. `git log | grep shipped | less` - each command will have it's own process group, but the same session group. 546 | Ctrl-C will kill them all. 547 | 548 | ## Chapter 19. Spawning Terminal Processes 549 | 550 | ### fork + exec 551 | `exec` allows us to transform our ruby process to any other process 552 | E.g. `ls` or a python script 553 | `exec 'ls', '--help'` 554 | Your ruby process will be transformed and won't go back. 555 | To fix that you can use `fork` to create a new process and `exec` to transform it to another process. 556 | If you need the ouput of the `exec` process, you can use `Process.wait` to wait for it. 557 | 558 | ### Arguments to exec 559 | Pass string => start up a shell process and the process'll interpret the shell 560 | Pass array => skip the shell and set up the array directly as the `ARGV` to the new process. 561 | 562 | Avoid passing a string where possible. 563 | 564 | #### Kernel#system 565 | `system('ls', '--help')` 566 | return value: `true` if the exit code is `0`, `false` if there is another code. 567 | `nil` if execution failed 568 | 569 | #### Kernel#` 570 | ``ls --help`` 571 | return value -- `STDOUT` of the program collected to a String 572 | `%x[]` is the exact same thing 573 | 574 | #### Process#spawn 575 | - is a non-blocking call. (`Kernel#system` is a blocking call). 576 | 577 | takes many options that allow you to control the behaviour of the child process. 578 | 579 | #### IO.popen 580 | Implementing pipes in pure ruby. 581 | It still does `fork` + `exec` underneath. 582 | 583 | `IO.popen('ls')` - returns a file descriptor. 584 | 585 | It also sets up a pipe to communicate with the spawned process. 586 | ```ruby 587 | IO.popen('less', 'w') { |stream| 588 | # stream is an IO object 589 | stream.puts "some\ndata" 590 | } 591 | ``` 592 | 593 | #### open3 594 | Open3 allows simultaneous access to the STDIN, STDOUT, and STDERR of a spawned 595 | process. 596 | ```ruby 597 | # This is available as part of the standard library. 598 | require 'open3' 599 | Open3.popen3('grep', 'data') { |stdin, stdout, stderr| 600 | stdin.puts "some\ndata" 601 | stdin.close 602 | puts stdout.read 603 | } 604 | ``` 605 | 606 | ### In the Real World 607 | One common drawback of all methods: forking, and so using lots of memory. 608 | Another way - `posix-spawn`. 609 | 610 | ### System Calls 611 | `Kernel#system` - system 612 | `Kernel#exec` - `execve` 613 | `IO.popen` - `popen` 614 | `posix-spawn` - `posix_spawn` 615 | 616 | ## Chapter 20. Ending 617 | ### Abstraction 618 | A kernel has an extremely abstract and simple view of its processes. 619 | All processes look the same to the kernel. 620 | Unix programming is programming language agnostic. 621 | 622 | ### Communication 623 | The kernel provides very abstract ways of communicating between processes. 624 | 625 | ## Chapter 21. Appendix: How Resque Manages Processes 626 | ## Chapter 22. Unicorn 627 | At a very high level Unicorn is a pre-forking web server. 628 | You tell unicorn how many processes you would like to have. 629 | It initializes network sockets and loads an app. 630 | Then uses fork to create the worker processes. 631 | The master process will keep track of the worker processes, see if they have an ok response time, etc. 632 | 633 | Waits for any process, second option makes the call non-blocking. 634 | ```ruby 635 | wpid, status = Process.waitpid2(-1, Process::WNOHANG) 636 | ``` 637 | 638 | ## Chapter 23 Preforking Servers 639 | ### 1. Efficient use of memory 640 | Preforking uses memory more efficiently than does spawning multiple unrelated 641 | processes. 642 | 643 | ### 2. Efficient Load Balancing 644 | Typically the workflow happens in the same process: A socket is opened, the process waits for connections on that socket. The connection is handled, closed, and the loop starts over again. 645 | Preforking servers: the master process opens a socket before loading the app. 646 | The master process doesn't accept connections. 647 | Forked processes will get a copy of that socket, they accept connections. 648 | The kernel balances the load. 649 | 650 | ### 3. Efficient Sysandmining 651 | Easier administrating (by a human). 652 | 653 | ### Basic Example 654 | `Process.waitall` - runs a loop waiting for all child processes to exit and returns array of process statuses. 655 | 656 | #### Questions [ru] 657 | - что такое файловые дескрипторы? 658 | - как узнать его в руби? 659 | - как присваивается номер? 660 | - почему номер присваивается не начиная с 0, чем заняты первые номера? 661 | - какие стандартные потоки/ресурсы? 662 | - какая от них польза? 663 | - какие лимиты есть у процессов? можно ли их менять? 664 | - для чего нужны env-переменные 665 | - env - это хэш? 666 | - как парсить аргументы? 667 | - способы завершить процесс 668 | - что будет с дочерними процессами, если родительский процесс завершится? 669 | - что такое CoW, какие плюсы от этого при форке? 670 | - fork в ruby CoW-friendly или нет? 671 | - Зачем нужны `Process.wait` and `waitpid2`? 672 | - что такое зомби-процессы? Как их избежать? 673 | - что такое ipc и какие основные способы? 674 | - чем отличаются pipes от sockets? 675 | - что если нужно ipc между разными машинами? 676 | - что такое session groups и process groups 677 | - что происходит при демонизировании процесса? 678 | - что такое процесс-демон? 679 | - как spawnуть процесс? 680 | - чем отличается system от exec и от `` 681 | - чем отличается `Process.spawn` от них? 682 | - rescue fork 683 | - как unicorn запускает и убивает процессы? 684 | - какие преимущества preforking servers? (по сравнению с non-preforking, e.g. Mongrel) 685 | 686 | -------------------------------------------------------------------------------- /ruby/clean_ruby.md: -------------------------------------------------------------------------------- 1 | # Lost In Translation 2 | ## Modeling a Business 3 | Mental models are often incomplete, unstable and unscientific, so developers are often approached with an incomplete idea. 4 | There may be reasons why one aspect or another goes unnoticed until later in the development. 5 | 6 | ## Our First Bugs 7 | One of the greatest challenges in writing software is communication. 8 | Mental models are both difficult to communicate and often misunderstood - that's how our first bugs enter the program. 9 | E.g. typos or nil lead to incorrect results. 10 | Over time incorrect expectations and incomplete mental models lead to change - that's why Agile and Lean are so popular. 11 | Responding to change is more valuable than following a plan (Agile Manifesto) 12 | We can confidently say: We'll be wrong. 13 | 14 | To avoid frustration and to prevent bugs we can separate our business processes from our domain models. 15 | Our app architecture should respond well to changes in our understanding of what we're building. 16 | 17 | > A key, longstanding hallmark of a good program is that it separates what is stable from what changes in the interest of good maintenance 18 | 19 | # Object Orientation 20 | ## Where we are 21 | OOP is pervasive, but the definition of “object-oriented” seems to have a number of interpretations: 22 | Let's pick one: 23 | Object-oriented programs execute algorithms by sending messages among objects, where the objects are an encapsulation of data and functions that perform operations. 24 | 25 | So we represent things with objects and those objects know how to perform actions. These objects talk to each other by sending messages. 26 | 27 | ### What Are Objects 28 | The name of the object reveals the intention of what it represents. 29 | In a typical ruby app we create an object Account with an amount of cents. 30 | 31 | ### Doing More 32 | We need to add money to an account, so we add that ability. 33 | ```ruby 34 | class Account 35 | def add_money(cents) 36 | @cents += cents 37 | end 38 | end 39 | ``` 40 | Next we need to transfer money from one account to another. 41 | We add a method #transfer_to(account, amount) 42 | One account in sending a message to another. 43 | 44 | This is where we begin to complicate our code, but it isn't obvious. 45 | 46 | ### Restricted OO 47 | We’ve gone from a class that represents a thing, to a class that represents a thing as well as defining its actions. 48 | We built a class to represent objects that store information (how much money) as well as perform actions (transfer money). 49 | 50 | With OO, the perspective the programmer takes is from within the object itself. 51 | The traditional approach allows you to see the state of the object, the public methods, the private and protected methods, the attributes. 52 | The problem is that we need to understand a lot about the class of an object, but still know little about the actual execution of a program. 53 | We know that methods may be called and the state may be changed but we're still guessing what the actual behaviour is. 54 | 55 | If we add more methods to Account, that'll exaggerate the problem easily. 56 | 57 | ```ruby 58 | > account.public_methods(false) 59 | => 60 | [:add_money, :transfer_to, :recalculate, :hold_funds, :subtract, 61 | :transactions, :split, :close] 62 | ``` 63 | If we only care about transfers, these extra features are noise. 64 | 65 | We can call this “Restricted OO” because our goal of readable code is supported only by the restrictions that we place on our architecture. 66 | 67 | Our perspective is from within an individual class => we don't have a broad understanding of how and when domain models interact 68 | 69 | We are forced to understand increasingly more about our system in its entirety to make sense of specific scenarios. 70 | 71 | ### Full OO 72 | With Full OO we observe our objects from a perspective of being among them, not within them. 73 | We can see exactly what object sends what message to what other object and we can clearly see and specify where those abilities are assigned. 74 | 75 | > A very important goal for DCI is to give the user an illusion of working directly with her mental model when working with the computer. 76 | 77 | This mental model we seek is the network of interacting objects. It is the representation of how objects interact with what other objects and when. 78 | DCI elevates our perceived network of objects to a readable, executable algorithm. 79 | 80 | ## Where We've been 81 | What did the first OO languages intend? 82 | Simula was the first language to introduce objects, it was intended to simulate the real world. 83 | 84 | > A computer program is a series of instructions which contains all the information necessary for a computer to perform some task. 85 | 86 | Reading code as if it’s a series of instructions is pedantic, no one does it this way. 87 | 88 | ## Where We're Going 89 | But couldn’t they? Shouldn’t they? 90 | We can and we should. 91 | 92 | Step-by-step instructions are often how we think about our business process. 93 | That's a reason why BDD became so popular. The goal of BDD is writing better code and executable specification. 94 | 95 | DCI is similar to BDD. It encourages to focus on real world scenarious. 96 | You're describing interactions with the actual objects that interact. 97 | 98 | DCI pushes us to wite software that we better understand in terms of object collaboration and runtime execution. 99 | 100 | # Being and Doing 101 | Traditionally we create classes of things and then add behavior. 102 | One problem is that classes grow like weed. 103 | 104 | We need to define what the things are and then focus on programming what they do. 105 | 106 | Example: a User class with #activate method 107 | Over time we add more features that represent other aspects of behavior. 108 | 109 | The actual activation is something that a user will do, not what it is. 110 | Combining what the user is (representation of data) with what it does complicates our mental model. 111 | The classes should represent data or model the behavior of the data object, but not both. 112 | 113 | Let's create module Activator with the #activate method. 114 | 115 | Often refactoring is just moving the code around, well intentioned, but accomplishes little. 116 | 117 | If we just include Activator to User class, we'll still have the discussed problems. 118 | 119 | ## Managing Change 120 | Let's include a User class from typical rails app with name, email and password validations. 121 | This class is unlikely to change. 122 | If we follow SRP, new methods shouldn't belong here. 123 | 124 | Making changes outside of the user we'll have a clearer separation of concerns, better test targeting and better reusability. 125 | We'll create an Activator object: 126 | 127 | ```ruby 128 | class Activator 129 | def initialize(user) 130 | @user = user 131 | end 132 | 133 | def activate 134 | ... 135 | @user.activated_at = Time.now 136 | @user.save 137 | end 138 | end 139 | ``` 140 | 141 | It's probably better to call it UserActivator - depending on your domain. 142 | 143 | ### Implicitness, Explicitness, and Encapsulation 144 | The OO purists may claim that we've broken encapsulation - Activator changes the user's state. 145 | We've created an implicit contract - the Activator assumes that the user has a certain interface. 146 | 147 | But explicit structure is more obvious. 148 | That's the point for discussion. 149 | 150 | ## Simplifying the Code 151 | Wouldn't be lots of bits and pieces, that we create to separate data and interaction harder to understand and maintain? 152 | If you build an app in Rails you've probably seen controllers, that do a lot of things: find records, call methods on them, send emails, etc, etc. 153 | 154 | Then you realize, that you need to extract some functionality, e.g. to avoid the code duplication. 155 | 156 | One solution is skinny controllers, fat models. 157 | You might decide to use callbacks. 158 | ``` 159 | after_create :send_welcome, :notify_admin_of_new_member 160 | ``` 161 | 162 | ## Wrong Turn 163 | Later in your development you realize that the administrators prefer to create users themselves. 164 | Now you don't need one of callbacks in this situation. 165 | 166 | You might add attr_accessor :dont_send_to_admin? and check it for sending callbacks. 167 | 168 | This option is awful. 169 | We could create an alternate class method #create_without_admin_notification! and call skip_callback! inside it, but it's even more terrible. 170 | 171 | Callbacks are often a developers first foray into spaghetti code. 172 | 173 | Setting flags and adding more methods to a global namespace are just two ways that we increase the cognitive load on ourselves. 174 | 175 | # Objects in Context 176 | DCI is an approach that removes the distraction of unrelated code - the objects don't have distracting abilities, the classes aren't cluttered with extra features. 177 | Let’s look at our earlier example of transferring money between accounts to see how we can apply the idea of a DCI context. 178 | 179 | We set yhr router to map a POST to our TransfersController: 180 | ```ruby 181 | resources :accounts do 182 | resources :transfers, :only => [:index, :create] 183 | end 184 | ``` 185 | Handling the request: 186 | ```ruby 187 | def create 188 | account = Account.find(params[:account_id]) 189 | account.transfer_to(params[:destination_id], params[:amount]) 190 | redirect_to account_transfers_path(account) 191 | end 192 | ``` 193 | Let’s alter this example to add the ability to alter only now that we need it. 194 | ```ruby 195 | def create 196 | source = Account.find(params[:account_id]) 197 | destination = Account.find(params[:destination_id]) 198 | source.extend(Transferrer) 199 | source.transfer_to(destination, params[:amount]) 200 | redirect_to account_transfers_path(source) 201 | end 202 | ``` 203 | This example assumes that you’ve got a Transferrer module that defines 204 | transfer_to. 205 | 206 | There is a possibility of duplicating code. For now we aint gonna need it. 207 | But will our application architecture help us or hurt us when we finally do need to reuse this code? 208 | Reusing the code in the controller as it is will not be easy. 209 | 210 | ## Clearer Perspective 211 | __The context__ is the object responsible for organizing the data and roles for carrying out your use cases. 212 | It’s a powerful model for encapsulating your business logic. 213 | 214 | Where is the business logic in common approaches? It's everywhere: controllers, models, split into modules and so on. 215 | We can build a good system in such way but it'll be more difficult to understand. 216 | 217 | The power of the DCI context object will get you closer to the intention-revealing code. 218 | 219 | ## Describing the system 220 | The "D" is for data modeling: what we know as traditional objects, though bereft of knowledge about scenarios. It captures the static structure of objects[...] 221 | The "C" is for context: the mapping of roles onto objects on a peruse-case basis[...] 222 | The "I" is for interaction: an algorithm of a stateless role, written in terms only of other roles, that defines in readable terms what the system does, and how it does it. 223 | 224 | We’ve explored the separation of data objects and their responsibilities but we pulled them together in our controller. 225 | That's logical, but if the controllers become bloated by responsibilities such a way. 226 | 227 | We can limit the controllers to handling requests and responses, and put our business logic into contexts. 228 | 229 | ```ruby 230 | class MoneyTransfer 231 | def initialize(source, destination) 232 | @source, @destination = source, destination 233 | end 234 | 235 | def execute(amount) 236 | @source.transfer_to(@destination, amount) 237 | end 238 | end 239 | ``` 240 | 241 | We will pass accounts to this context when it’s initialized, we want to refer to them in terms relevant to our context. 242 | We call these __Methodless roles__ . 243 | 244 | Our concern with DCI is with the interactions between and among roles. 245 | Objects play different roles at different times during its execution. 246 | 247 | A Methodless Role is the role an object plays in the execution of an algorithm. 248 | Naming is very important for this context (source and destination, not account1, account2) 249 | 250 | Back in our context, we’re still missing the ability to do anything. 251 | 252 | Let’s alter the context: 253 | ```ruby 254 | class MoneyTransfer 255 | def initialize(source, destination) 256 | @source = source 257 | @destination = destination 258 | assign_transferrer(@source) 259 | end 260 | 261 | def execute(amount) 262 | @source.transfer_to(@destination, amount) 263 | end 264 | 265 | private 266 | 267 | def assign_transferrer(account) 268 | account.extend(Transferrer) 269 | end 270 | 271 | module Transferrer 272 | def transfer_to(destination, amount) 273 | self.balance -= amount 274 | destination.balance += amount 275 | end 276 | end 277 | end 278 | ``` 279 | We have just added what we call a __Methodful Role__. 280 | In prev example objects playing roles in this context as source and destination had no special roles. 281 | The Transferrer is the Methodful Role, it adds methods to the data object. 282 | 283 | ## Implementing Your System 284 | Added tests for the MoneyTransfer. 285 | Now we have something useful and reusable, but what does our controller look like? 286 | ```ruby 287 | # TransfersController#create 288 | source = Account.find(params[:account_id]) 289 | destination = Account.find(params[:destination_id]) 290 | transfer = MoneyTransfer.new(source, destination) 291 | transfer.execute(params[:amount]) 292 | ... 293 | ``` 294 | Although we’ve put our business logic in a reusable container, this controller action is still complicated. 295 | It would be great to have code like this: 296 | ```ruby 297 | transfer = MoneyTransfer.new(params[:account_id], params[:destination_id]) 298 | account = transfer.execute(params[:amount]) 299 | ``` 300 | 301 | Alter the tests and modify MoneyTransfer: 302 | ```ruby 303 | class MoneyTransfer 304 | def initialize(source_id, destination_id) 305 | @source = find_account(source_id) 306 | @destination = find_account(destination_id) 307 | assign_transferrer(@source) 308 | end 309 | 310 | private 311 | 312 | def find_account(id) 313 | Account.find(id) 314 | end 315 | end 316 | ``` 317 | 318 | ## All Set Up 319 | We've put our business logic in a place that's reusable and easily tested. 320 | When explaining our implementation we can say 'this is how it works' instead of 'What accounts do'. 321 | 322 | With our focus on the program’s duties we can more quickly react to changes. 323 | The MoneyTransfer context in this example centralizes our understanding of the system. 324 | 325 | We won't do this for every procedure that our application needs to perform, but we have a powerfult tool to separate being from doing. 326 | 327 | To better manage our flow we need a way to organize related scenarios. 328 | 329 | # Gazelles And Gazebos 330 | __User stories__ describe an end user’s behavior that will be later translated into code. 331 | A __use case__ helps us to describe multiple scenarios for a business requirement. 332 | For use case you have a specific preconditions, specific actors and specific goals. 333 | 334 | > You can fit a gazelle inside a gazebo just as you can fit a user story inside of a use case, otherwise would be cruel. 335 | 336 | A use case describes at least two exit points for a business process: the success and the failure. 337 | Often there are many variations and multiple exit points in related scenarios. 338 | 339 | Use cases DO: 340 | - Hold Functional Requirements in an easy to read, easy to track text format 341 | - Represents the goal of an interaction between an actor and the system. 342 | The context is an object that encapsulates this concept. 343 | - Records a set of paths (scenarios) that traverse an actor from a trigger event (start of the use case) to the goal (success scenarios). 344 | - Records a set of scenarios that traverse an actor from a trigger event toward a goal but fall short of the goal (failure scenarios). 345 | A use case describes a complete interaction between the user and your system and it is the responsibility of your context to implement this. 346 | - Are multi-level: one use case can use/extend the functionality of another. 347 | A context can trigger other contexts within. 348 | 349 | Using DCI goal is to implement business logic in an organized set. 350 | 351 | In Rails, your controllers should handle the user actions that trigger these use cases. 352 | You might have multiple ways to trigger a use case. 353 | E.g. user can interact with objects and admin can interact with them in admin view. 354 | 355 | Put your use cases in executable code and trigger them from wherever your interface requires it. 356 | Begin by writing your use cases. 357 | 358 | # Crossing the Use Chasm 359 | We can write use cases to clarify the requirements of our program. Creating a context helps us to implement those requirements in executable code. 360 | With this approach we can leave the decisions like data persistence for later. 361 | 362 | ## Framing the problem 363 | Let’s create a scenario for this use case: a user submits a question and receives an answer from an expert. 364 | ```bash 365 | rails new ask_the_experts 366 | touch app/models/expert_questioning.rb 367 | ``` 368 | 369 | expert_questioning is the context for describing the use case. 370 | 371 | ### A Note On Naming And Code Use 372 | 373 | expert_questioning_context.rb seems attractive as a filename. 374 | We use _controller appendage for rails controllers. 375 | 376 | The convention for the _controller appendage in Rails simplifies the setup of routing requests to the appropriate class to handle them 377 | A UsersController will be automatically initialized when a request matches the :users route 378 | 379 | But choosing to add Context to our context classes and _context to their filenames does nothing to simplify our code nor make magic happen. I 380 | 381 | A gerund is often an intention revealing choice for a context. Our domain models are __objects__ represented by nouns, and our object interactions are __methods__ represented by verbs. 382 | Our context is something __happening__ represented by a gerund. 383 | 384 | Think about how you will describe your process in spoken language before you put it into terms of a programming language. 385 | 386 | ### Expanding Our Use Case 387 | Our __Primary Actor__ will be a user of the system. 388 | 389 | ```ruby 390 | # Primary Actor: a regular user 391 | ``` 392 | Specify a __Goal__ 393 | ```ruby 394 | # Goal: user gains information about a specific finance question 395 | ``` 396 | List __Supporting Actors__ (additional users, other parts of your system, or other systems) 397 | ```ruby 398 | # Supporting Actors: a financial expert 399 | ``` 400 | __Preconditions__ 401 | ```ruby 402 | # Preconditions: user is authenticated and authorized 403 | ``` 404 | So our questions are: 405 | Who is doing this? What is the goal? Who else is involved? And what else do I need for this to work? 406 | 407 | Without creating any working code, we’ve begun with a clear understanding of what this program is going to do. 408 | 409 | ## Preparing the Context 410 | Let's begin defining what we need to get things working. 411 | ```bash 412 | touch spec/models/expert_questioning_spec.rb 413 | ``` 414 | We specify the requirements of the initialization method where we define what arguments we’ll need. 415 | ```ruby 416 | # spec/models/expert_questioning_spec.rb 417 | it 'initializes with a questioner and a question' do 418 | ... 419 | end 420 | ... 421 | it 'errors without a questioner and question' do 422 | ... 423 | end 424 | ... 425 | ``` 426 | The described_class method in RSpec will return the class used in the current describe block. 427 | It'll be easy to change the name of the class without changing the spec. 428 | 429 | Let's create a simpler helper file giving us only what we need. 430 | ```bash 431 | touch spec/spec_helper_lite.rb 432 | ``` 433 | We don't need booting the Rails app for now (using spec/spec_helper.rb) 434 | For our spec_helper_lite we’ll add references to RSpec’s libraries, we'll add references for our necessary files to our expert_questioning_spec.rb 435 | 436 | Let's define the class and the initialize method, so that the tests would pass: 437 | ```ruby 438 | # Primary Actor: a regular user 439 | # Goal: user gains information about a specific finance question 440 | # Supporting Actors: a financial expert 441 | # Preconditions: user is authenticated and authorized 442 | class ExpertQuestioning 443 | def initialize(questioner, question_text) 444 | end 445 | end 446 | ``` 447 | ## Decoupling the File System 448 | require_relative is usefult, but it couples us to the file system. We want to move files aroumd easily. 449 | Instead we can use a simple require to by a #needs helper in spec_helper_lite.rb, which will add a certain directory to $LOAD_PATH. 450 | We'll be able to load our model with: 451 | ```ruby 452 | needs 'models' 453 | require 'expert_questioning' 454 | ``` 455 | We could add app/models (and other needed dies) to load path, but that would cause an overhead. It's better to keep things simple and small. 456 | 457 | ## Defining Success 458 | We'd like to nofify experts about the new question and the users about the question answered. 459 | 460 | The __Trigger__ for this use case is the submitting of a question by the __Primary Actor__: 461 | 462 | Actions, triggered by the primary actor: 463 | - A notification is sent to the user that her question will be answered soon. 464 | - An available expert is assigned to the question. 465 | - The expert is notified of the unanswered question. 466 | 467 | Actions, triggered by the supporting actor: 468 | - The expert submits an answer to the question. 469 | - The user is notified of the answer to her question. 470 | 471 | There are no extra details (implementation), just what needs to be done. 472 | Our implementation can be as simple as throwing paper airplanes around for a notification delivery system. 473 | 474 | The five steps we’ve created represent just one scenario. 475 | There are also __Extensions__ to this use case that we’ll see where we have alternate outcomes and failures. 476 | 477 | ## Brainstorming 478 | As our program is used, we’re more likely to run into edge cases and alternate outcomes that our simple mental model didn’t cover. 479 | Failures, alternatives, other actors. 480 | __Extentions__ for our procedure: 481 | - no expert is available 482 | => the user is notified that no expert is available for her question 483 | - the expert determines that the question is answered already. 484 | => the expert assigns the question to an existing answer 485 | 486 | The Lean approach teaches us that determining the aspects of this use case should be exhaustive. 487 | We already have a problem with sending the user 2 contradictory notifications if there are not experts. 488 | 489 | We should send a user a notification that her question will be answered soon after assinging an expert. 490 | By using this approach we can avoic such errors before writing any actual code. 491 | 492 | ## Setting Boundaries 493 | Our ExpertQuestioning will begin when a user submits a question. We’ll need a break in the procedure where we wait for the expert to answer 494 | a question. 495 | Now we’re venturing into mental models that are more complex, so it’s important to find good names. 496 | An #execute method will probably not reflect our understanding. 497 | 498 | Following naming conventions limits required information to understand a system. But there is a downside too: we should focus on communicating the 499 | business process in our code not fitting our process to our code. 500 | 501 | Let’s go with “start” and “finish” since we could say that we’d start with asking and finish by answering. 502 | ```ruby 503 | class ExpertQuestioning 504 | def start 505 | # 1. An available expert is assigned to the question. 506 | # 1a. No expert is available. 507 | # 1a.1 The user is notified that no expert is available. 508 | # 2. A notification is sent to the user that her question 509 | will be answered soon. 510 | # 3. The expert is notified of the unanswered question. 511 | end 512 | def finish 513 | # 4. The expert submits an answer to the question. 514 | # 515 | 4a. The expert determines that the question has already 516 | been answered. 517 | # 518 | 4a.1 The expert assigns the question to an existing 519 | answer. 520 | # 5. The user is notified of the answer to her question. 521 | end 522 | end 523 | ``` 524 | 525 | Create tests: 526 | ```ruby 527 | # when starting: 528 | # it assigns an available expert to the question 529 | # it notifies the questioner that the answer is queued 530 | # it notifies the assigned expert of the questio 531 | # when finishing: 532 | # answers the questio 533 | # notifies the questioner of the answer 534 | ``` 535 | All of these tests are pending for now, and each one can be a point of discussion for the implementation 536 | Each one could become another use case. 537 | 538 | # Screenplay In Action 539 | We've set up the context. Now we need to decide how to implement the parts. 540 | 541 | ## Finding the Expert 542 | For now details of selecting the expert is a distraction. 543 | Our main concern is the asking and answering of a question. so let's leave a note for ourselves: 544 | ```ruby 545 | def expert 546 | # TODO: Expert selection requires consideration 547 | Expert.first_available 548 | end 549 | # bundle exec rake notes 550 | ``` 551 | 552 | # Responding to Rails 553 | We could create a responder to offload the behavior of redirection, rendering and other controller responsibilities. 554 | When the flow of our code is typical, it’s OK for the details to be hidden somewhere else. 555 | But when we need to understand special decisions or other elements of our program we want them to be as clear as possible. 556 | 557 | If we create a responder: 558 | ``` 559 | def create 560 | transfer = MoneyTransfer.new(params[:account_id], params[:destination_id]) 561 | respond_with transfer.execute(params[:amount]) 562 | end 563 | ``` 564 | 565 | The behavior (rendering view, redirection) depends on the return value of the execute method. 566 | Where does this responsibility belong? Should the responder be a part of the context? 567 | 568 | ## Considering Dependencies 569 | So does framework depends on business or vice versa? 570 | 571 | The framework depending on the business logic might look like this: 572 | ```ruby 573 | class SomeController < ActionController::Base 574 | def action 575 | if BusinessClass.logic_method 576 | redirect_to :some_path 577 | else 578 | render :some_other_action 579 | end 580 | end 581 | end 582 | ``` 583 | When our needs are simple, this approach is quick and easy to get an application going. 584 | But when we want to understand what process BusinessClass and logic_method represent, we have knowledge of the behavior in both BusinessClass and SomeController. 585 | Our controller acts as the connection to the framework and it knows much about the BusinessClass. 586 | 587 | To understand our program we need to keep more things in our minds and piece them together by matching up methods and behavior. 588 | 589 | We can simplify our program and ease our understanding by writing a controller like the following: 590 | ```ruby 591 | class SomeController < ActionController::Base 592 | def action 593 | respond_with BusinessClass.logic_method 594 | end 595 | end 596 | ``` 597 | Now the controller only needs to know two things (BusinessClass and logic_method). 598 | 599 | ## Alternate Dependency Approaches 600 | By using respond_with , our controller expects an object that behaves like a Responder. 601 | Rails will automatically create a Responder for us and follow the conventions. 602 | 603 | Relying on Rails to create a default responder for us might be hiding important information - we'd need to dig through documentation and framework code. 604 | For Rails responder all we need is an object that responds to call: 605 | ```ruby 606 | class SomeController < ActionController::Base 607 | def action 608 | respond_with BusinessClass.logic_method, 609 | :responder => lambda{ ... } 610 | end 611 | end 612 | 613 | # or 614 | 615 | class SomeController < ActionController::Base 616 | self.responder = lambda{ ... } 617 | def action 618 | respond_with BusinessClass.logic_method 619 | end 620 | end 621 | ``` 622 | We can keep a responder in the Business class. It might help us to keep the concerns in one place, but we either give up any useful behavior from our framework or inherit from ActionController::Responder creating an explicit dependency. 623 | 624 | Instead of implementing the interface that our framework expects, we can tell the framework related parts what to do. 625 | Dependency Injection: 626 | ```ruby 627 | class TransfersController < ApplicationController 628 | def create 629 | transfer = MoneyTransfer.new(params[:account_id], params[:destination_id]) 630 | transfer.command(self) 631 | transfer.execute(params[:amount]) 632 | end 633 | end 634 | ``` 635 | We are telling the transfer object to command the current controller what to do. This method could be named set_listener, guide, or any other term that best reveals the intentions. 636 | 637 | ## Framework Glue 638 | We need to implement #go_to method. 639 | We use this method instead of built-in redirect_to or render for 2 reasons: 640 | - better naming for our domain 641 | - 642 | 643 | 644 | 645 | 646 | 647 | -------------------------------------------------------------------------------- /ruby/working_with_ruby_threads.md: -------------------------------------------------------------------------------- 1 | ### The promise of multi-threading 2 | Processes copy memory, while threads share memory. 3 | Process spawning is slower than thread spawning. 4 | 5 | Threads can you give more 'units' of concurrency for the available resources. 6 | 7 | ## Chapter 1. You're Always in a Thread 8 | There's always at least one: the main thread. 9 | 10 | ```ruby 11 | Thread.main 12 | Thread.current == Thread.main # true 13 | ``` 14 | When the main thread exits, all other threads are immediately terminated and the Ruby process exits. 15 | 16 | ```ruby 17 | Thread.new { Thread.current == Thread.main } .value # false 18 | ``` 19 | 20 | ## Chapter 2. Threads of Execution 21 | Threads are powerfull and let you shoot yourself in the foot. 22 | ### Shared address space 23 | Share all of the same references in memory (ie. variables), as well as the AST (ie. the compiled source code). 24 | 25 | ### Many threads of execution 26 | Multiple threads of execution operating on the same code at the same time. 27 | It's no longer possible to step through these simultaneous threads of execution in any kind of predictable manner, a certain amount of randomness introduced in the way that threads are scheduled. 28 | 29 | ### Native threads 30 | All of the Ruby implementations studied in this book ultimately map one Ruby thread to one native, operating system thread. 31 | 32 | ### Non-deterministic context switching 33 | It refers to the work that's done by your operating system thread scheduler, you have no control on them. 34 | 35 | The thread scheduler can 'pause' or 'unpause' a thread at any time. 36 | But there are primitives you can use to say, "Hey, thread scheduler, I'm doing something important here, don't let anybody else cut in until I'm done." 37 | 38 | #### Context switching in practice 39 | The `Queue` class is a thread-safe data structure that ships with Ruby 40 | A *race condition* involves two threads racing to perform an operation on some 41 | shared state. 42 | 43 | E.g. `||=` is not safe. 44 | It's good practice to avoid lazy instantiation in the presence of multiple threads. The race may result in the underlying data becoming incorrect. 45 | 46 | Some race conditions may only be exposed under heavy load. 47 | 48 | An operation `@results ||= Queue.new` is not atomic. An atomic operation is one which cannot be interrupted before it's complete. 49 | 50 | With proper protection in place a multi-step operation can be treated as a single-step. 51 | 52 | ### Why is this so hard? 53 | Any time that you have two or more threads trying to modify the same thing at 54 | the same time, you're going to have issues. 55 | 1) don't allow concurrent modification, or 2) protect concurrent modificatio 56 | 57 | ## Lifecycle of a Thread 58 | `require 'thread'` doesn't load the `Thread` constant. Thread is loaded 59 | by default, but requiring 'thread' brings in some utility classes like `Queue`. 60 | 61 | ### Thread.new 62 | ```ruby 63 | Thread.new { ... } 64 | Thread.fork { ... } 65 | Thread.start(11, 2 ) { |x, y| x + y } 66 | ``` 67 | - Executes the block: the thread will yield to that block. The end of the block will be reached or an exception will be raised. 68 | - Returns an instance of Thread. 69 | `Thread.new` returns a Thread instance representing the subthread that was just spawned. 70 | 71 | ### Thread#join 72 | You can use `#join` to wait for it to finish. 73 | ```ruby 74 | thread = Thread.new { puts 'I am in thread'; sleep 3 } 75 | thread.join 76 | puts "You'll have to wait 3 seconds to see this" 77 | ``` 78 | Calling #join on the spawned thread will join the current thread of execution 79 | with the spawned one. With join, the current thread will sleep until the spawned thread exits. 80 | 81 | #### Thread#join and exceptions 82 | When one thread raises an unhandled exception, it terminates the thread where the exception was raised, but doesn't affect other threads. 83 | 84 | A thread that crashes from an unhandled exception won't be noticed until another thread attempts to join it. 85 | `join` => re-raise exception in the current thread 86 | 87 | ### Thread#value 88 | first joins with the thread, and then returns the last expression from the block of code the thread executed. 89 | 90 | ### Thread#status 91 | Possible values: 92 | - `run` 93 | - `sleep` - blocked waiting for a mutex, or waiting on IO 94 | - `false` - finished or killed 95 | - `nil` - raised an unhandled exception 96 | - `aborting` - running, yet dying 97 | 98 | ### Thread.stop 99 | puts the current thread to sleep and tells the thread scheduler to schedule some other thread. It will remain in this sleeping state until its alternate, `Thread#wakeup` is invoked. 100 | 101 | ### Thread.pass 102 | Just asks the thread scheduler to schedule some other thread, without putting the thread to sleep. Can't guarantee that the scheduler will take the hint. 103 | 104 | ### Avoid Thread#raise 105 | This method should not be used -- doesn't respect `ensure` blocks. Allows a caller external to the thread to raise an exception inside the thread. 106 | 107 | ### Avoid Thread#kill 108 | Same reason as above. 109 | 110 | ## Chapter 4. Concurrent != Parallel 111 | 1) Do multiple threads run your code concurrently? Yes. 112 | 2) Do multiple threads run your code in parallel? Maybe. 113 | 114 | Concurrent -- working on 2 things, possibly switching between them. 115 | Parralel -- working on 2 things at the same time. 116 | 117 | More resources are required in order to work in parallel. 118 | 119 | ### You can't guarantee anything will be parallel 120 | All you can do is to organize your code to be concurrent, making it parallel is not in your hands. 121 | 122 | You should assume that your concurrent code will be running in parallel because it typically will, but you can't guarantee it. 123 | 124 | ## Chapter 5. The GIL and MRI 125 | ### The global lock 126 | GIL - Global Interpreter Lock 127 | 128 | There is one, and only one, GIL per instance of MRI (or per MRI process). 129 | If one of those MRI processes spawns multiple threads, that group of threads will share the GIL for that process. 130 | 131 | Ruby code will never run in parallel on MRI. 132 | 133 | ### An inside look at MRI 134 | Remember that each MRI thread is backed by a native thread, and from the kernel's point of view, they're all executing in parallel. 135 | The GIL is implemented as a mutex. The operating system will guarantee that one, and only one, thread can hold the mutex at any time. 136 | 137 | ### The special case: blocking IO 138 | What happens when a thread executes some Ruby code that blocks on IO? 139 | ```ruby 140 | require 'open-uri' 141 | 3 .times.map do 142 | Thread.new do 143 | open('http://zombo.com') 144 | end 145 | end.each(&:value) 146 | ``` 147 | `open` make take a long time depending on the status of the resource. 148 | When a thread is blocked waiting for IO, it won't be executing any Ruby code. 149 | So when it's blocked on IO, the GIL is released. 150 | 151 | Under the hood, each thread is using a ppoll(2) system call to be notified when their connection attempt succeeds or fails. After recieving the answer, the ruby code will need to be executed, so the process starts over again. 152 | 153 | ### Why? 154 | 1) To protect MRI internals from race conditions 155 | 2) To facilitate the C extension API; MRI can function safely, even in the presence of C extensions that may not be thread-safe 156 | 3) To reduce the likelihood of race conditions in your Ruby code 157 | 158 | ### Misconceptions 159 | #### Myth: the GIL guarantees your code will be thread-safe. 160 | The GIL reduces the likelihood of a race condition, but can't prevent it. 161 | 162 | #### Myth: the GIL prevents concurrency. 163 | The GIL prevents parallel execution of Ruby code, but it doesn't prevent concurrency. 164 | You can use multiple threads to parallelize code that is IO-bound. 165 | 166 | ## Chapter 6. Real Parallel Threading with JRuby and Rubinius 167 | JRuby and Rubinius don't have a GIL - they allow real parallel execution of the code. 168 | 169 | JRuby and Rubinius don't have a GIL. 170 | 1) JRuby and Rubinius do protect their internals from race conditions with many fine-grained locks. 171 | 2) JRruby doesn't support C extension api. For gems that need to run natively, they support Java-based extensions. 172 | Rubinius does support the MRI C extension API and they help gem authors fix thread-safety issues. 173 | 3) They don't reduce the likelihood of race conditions in your Ruby code 174 | 175 | ## Chapter 7. How Many Threads Are Too Many? 176 | It depends. The only way to be sure is to measure and compare. 177 | But there are some heuristics you can use to get started. 178 | 179 | ### ALL the threads 180 | OSX has a hard limit on the number of threads that one process can spawn. 181 | On a Linux machine it's possible to spawn a lot of threads, but you probably don't want to. 182 | 183 | #### Context Switching 184 | Spawning a thread is cheap, but there is overhead for the thread scheduler to manage those threads. 185 | 4 cores => 4 threads can be executing instruction at any given time. 186 | Sometimes it makes sense to spawn more threads than the number of CPU cores. 187 | 188 | ### IO-bound 189 | IO-bound code -- bound by the IO capabilities of your machine (speed of external requests or read/write to disc) 190 | It does make sense to spawn more threads than CPU cores if your code spends time waiting for a response from some IO device. 191 | 192 | There will be a sweet spot between utilizing available resources and context switching overhead. Finding the sweet spot is really important. 193 | 194 | ### CPU-bound 195 | CPU-bound code is inherently bound by the rate at which the CPU can execute 196 | instructions. 197 | E.g. math calculations. 198 | 4 cores => code may run in parallel across 4 threads, utilize all the power, no overhead. 199 | more threads => stiall utilizes all the power, but with the overhead. 200 | MRI - performance isn't impacted with the introduction 201 | of more threads because of GIL. 202 | 203 | ### So... how many should you use? 204 | The situation won't be so clear. 205 | The only way to a surefire answer is to measure. 206 | 207 | ## Chapter 8. Thread safety 208 | ### What's really at stake? 209 | Thread-safe code => it can run in a multi-threaded context and your underlying data will be safe. 210 | Not thread-safe => the worst that can happen is incorrect data. 211 | 'check-then-set' race condition: 212 | Multi-step operation. 213 | E.g. checking the order status, then collecting payment and setting the status. 214 | It's likely that the operation will be interrupted before it's finished. 215 | That leads to collecting payments twice. 216 | 217 | ### The computer is oblivious 218 | The computer is unaware of thread-safety issues. 219 | So there is no alarm for such errors. It's hard to find those them, sometimes they only appear in heavy load. 220 | 221 | ### Is anything thread-safe by default? 222 | `+=`, `||=` are single operations, but they are not atomic 223 | `Array`, `Hash` are not thread-safe by default. 224 | 225 | Any concurrent modifications to the same object are not thread-safe. 226 | 227 | ## Chapter 9. Protecting Data with Mutexes 228 | ### Mutual exclusion 229 | Mutex -- mutual exclusion. 230 | If you wrap some section of your code with a mutex, you guarantee that no two threads can enter that section at the same time. 231 | ```ruby 232 | mutex = Mutex.new 233 | counter = 0 234 | 10_000.times do 235 | Thread.new do 236 | mutex.lock 237 | counter += 1 238 | mutex.unlock 239 | end 240 | end 241 | ``` 242 | The first thread that hits this bit of code will lock the mutex and become it's owner. 243 | Until the owning thread unlocks the mutex, no other thread can lock it. 244 | More commonly used: 245 | ```ruby 246 | mutex.synchronize do 247 | shared_array << nil 248 | end 249 | ``` 250 | ### The contract 251 | The mutex is shared among all the threads that share the same Mutex instance. 252 | The block of code inside of a Mutex#synchronize call is often called a critical section. 253 | 254 | ### Making key operations atomic 255 | check-then-set race condition => mutex.synchronize block should include both the 'check' AND the 'set' 256 | 257 | ### Mutexes and memory visibility 258 | Mutexes carry an implicit memory barrier. So, if one thread holds a mutex to write a value, other threads can lock the same mutex to read it and they will see the correct, most recent value. 259 | 260 | ### Mutex performance 261 | Mutexes inhibit parallelism. 262 | Mutexes provides safety where it's needed, but at the cost of performance. 263 | Restrict the critical section to be as small as possible, while still preserving the safety of your data. 264 | 265 | ### The dreaded deadlockrf 266 | One thread is blocked waiting for a resource from another thread (like blocking on a mutex), while this other thread is itself blocked waiting for a resource. There's a deadlock if a situation arises where neither thread can move forward. 267 | Example: 2 threads, 2 mutexes. Each thread acquires one mutex, then attempts to acquire the other. 268 | 269 | Solutions: 270 | `Mutex#try_lock` 271 | `try_lock` will not wait if the mutex isn't available. 272 | It will return false if the mutex is not available and true if it does. 273 | In our example when `try_lock` returns `false`, both threads should release their mutexes and return to the starting state. 274 | 275 | With this approach, it's fine to use a blocking lock to get the first mutex, but try_lock should be used when acquiring the second mutex. 276 | 277 | `livelock` -- threads are in some loop with each other with none progressing. 278 | 279 | It's possible that Thread A acquires Mutex A, Thread B acquires Mutex B, then both try_lock the other mutex, both fail, both unlock their first mutex, then both reacquire their first mutex, etc., etc. 280 | 281 | A better solution is to define a mutex hierarchy. 282 | Any time that two threads both need to acquire multiple mutexes, make sure they do it in the same order. 283 | 284 | ## Chapter 10 Signaling Threads with Condition Variables 285 | ```ruby 286 | condvar = ConditionVariable.new 287 | mutex = Mutex.new 288 | Thread.new do 289 | 10.times do 290 | ... 291 | mutex.synchronize do 292 | results << smth 293 | condvar.signal 294 | end 295 | end 296 | end 297 | until comics_received >= 10 298 | mutex.synchronize do 299 | while results.empty? 300 | condvar.wait(mutex) 301 | end 302 | end 303 | ... 304 | comics_received += 1 305 | end 306 | ``` 307 | Condition variables are used for putting threads to sleep and waking them only once a certain condition is met. 308 | 309 | A `ConditionVariable` provides more efficient solution to the consumer-producer problem -- instead of checking many times for the `results.count` inside a mutex, we just wait for the signal from `condvar` 310 | 311 | `ConditionVariable#wait` - wait for the specified mutex. 312 | `ConditionVariable#signal` will wake up exactly one thread that's waiting on this ConditionVariable. 313 | `ConditionVariable#broadcast` will wake up all threads currently waiting on 314 | this ConditionVariable. 315 | 316 | ## Chapter 11. Thread-safe Data Structures 317 | ### Implementing a thread-safe, blocking queue 318 | ```ruby 319 | q = BlockingQueue 320 | BlockingQueue.new 321 | q.push 'thing' 322 | q.pop #=> 'thing' 323 | ``` 324 | Requirements: internally thread-safe + the `pop` operation, when called on an empty queue, should block until something is pushed onto the queue. 325 | 326 | Basic example with mutex and `ConditionalVariable`: 327 | Condvar rectifies returning nil from empty queue. 328 | ```ruby 329 | class BlockingQueue 330 | def initialize 331 | @storage = Array.new 332 | @mutex = Mutex.new 333 | @condvar = ConditionVariable.new 334 | end 335 | 336 | def push(item) 337 | @mutex.synchronize do 338 | @storage.push(item) 339 | @condvar.signal 340 | end 341 | end 342 | 343 | def pop 344 | @mutex.synchronize do 345 | while @storage.empty? 346 | @condvar.wait(@mutex) 347 | end 348 | 349 | @storage.shift 350 | end 351 | end 352 | end 353 | ``` 354 | ### Queue, from the standard lib 355 | Ruby's standard library ships `Queue` class. This is the only thread-safe 356 | data structure that ships with Ruby. It loads when you do `require 'thread'` 357 | `Queue` has a few more methods than our `BlockingQueue` , but its behaviour regarding push and pop is exactly the same. 358 | 359 | Typicall usage: distributing workloads to multiple threads, where one thread is pushing to the queue, and multiple threads are popping. The popping threads are put to sleep until there's some work for them to do. 360 | 361 | ### Array and Hash 362 | The core `Array` and `Hash` classes are not thread-safe by default, nor should they be. 363 | The JRuby Array and Hash are also not thread-safe. 364 | `thread_safe` gem implements `ThreadSafe::Array` and `ThreadSafe::Hash` which are thread-safe. 365 | 366 | ### Immutable data structures 367 | Immutable data structures are inherently thread-safe. 368 | 369 | ## Chapter 12. Writing Thread-safe Code 370 | Idiomatic Ruby code is most often thread-safe Ruby code. 371 | 372 | ### Avoid mutating globals 373 | #### Even globals can provide thread-safe guarantees 374 | If you need global variables, ensure that data consistency is 375 | preserved (e.g. with mutexes) 376 | 377 | #### Anything where there is only one shared instance is a global 378 | - Constants 379 | - the AST 380 | - Class variables/methods 381 | 382 | E.g. modifying class variable (`@@clouds`) is not safe. 383 | 384 | Modifying the AST at runtime is almost always a bad idea: 385 | changes to the AST shouldn't happen at runtime. 386 | 387 | E.g. `kaminari` issue with defining a method dynamically, then calling `alias_method` with that method, then removing it. 388 | 389 | ### Create more objects, rather than sharing one 390 | Sometimes you just need that global object. 391 | The most common example of this is a network connection. 392 | Solution: thread-locals, connection pools. 393 | 394 | ### Thread locals 395 | A thread-local lets you define a variable that is global to the scope of the current thread. 396 | ```ruby 397 | Thread.current[:redis] = Redis.new 398 | ``` 399 | So, if your program is running N threads, it will open N connections to Redis. 400 | This N:N connection mapping is fine for small numbers of threads, but gets out of hand when the number of threads starts to increase. For connections, a pool is often a better abstraction. 401 | 402 | ### Resource pools 403 | A pool object will open a number of connections, or in the more general sense, may 404 | allocate a number of any kind of resource that needs to be shared among threads. 405 | The pool is responsible for keeping track of which connections are 406 | checked out and which are available, preserving thread safety. 407 | When a thread wants to make use of a connection, it asks the pool to check out a connection. 408 | When the thread is done, it checks the connection back in to the pool. 409 | gem `connection_pool` 410 | 411 | ### Avoid lazy loading 412 | ### Prefer data structures over mutexes 413 | Mutexes are harder to understand. 414 | Rather than letting threads access shared objects and implementing the necessary synchronization, you pass shared objects through data structures. 415 | This ensures that only one thread could mutate an object at any given time. 416 | 417 | ## Chapter 13. Thread-safety on Rails 418 | It's rare that you would spawn threads inside your application logic, but if you're using a multi-threaded web server (like Puma 1 ) or a multi-threaded background job processor (like Sidekiq 2 ), then your application is running in a multi-threaded context. 419 | 420 | ### Gem dependencies 421 | You'll have to do your research. Most of the gems are thread-safe buty you may need to check a bug-tracker. 422 | 423 | ### The request is the boundary 424 | A multi-threaded web server will process each request with a separate thread. 425 | Don't share objects between requests => no objects shared between threads. 426 | 427 | ## Chapter 14. Wrap Your Threads in an Abstraction 428 | ### Single level of abstraction 429 | > "one level of abstraction per function." (high- and low-level code don't mix well) 430 | threads, mutexes -- low-level code 431 | Simple threads wrapper that will spawn a thread for each element of an array: 432 | ```ruby 433 | module Enumerable 434 | def concurrent_each 435 | threads = [] 436 | each do |element| 437 | threads << Thread.new { 438 | yield element 439 | } 440 | end 441 | threads.each(&:join) 442 | end 443 | end 444 | ``` 445 | ### Actor model 446 | In some cases, it will make sense for you to write your own simple abstractions on top of your multi-threaded code, but often you'll get more benefit from a more mature abstraction. 447 | 448 | An Actor is a long-lived 'entity' that communicates by sending messages. 449 | long-running entity - long-running thread. 450 | 451 | Each Actor has an 'address'. If you know the address of an Actor, you can send it a message. 452 | These messages go to the Actor's mailbox, where they're processed asynchronously when the Actor gets around to it. 453 | 454 | `Celluloid` - Actor Model + Ruby Object Model 455 | 456 | Including the `Celluloid` module into any Ruby class will turn instances of that class into full-fledged `Celluloid` actors. 457 | Each Celluloid actor is housed by a separate thread, one thread per actor. 458 | 459 | 'address' is a reference to the object. 460 | Sending messages to an actor is calling methods on an object. 461 | 462 | Regular method calls are sent to the actor's mailbox but, behind the scenes, Celluloid will block the caller until it receives the result, just like a regular method call. 463 | 464 | Without waiting for the result -- `async` 465 | 466 | Async + waiting for the result: 467 | `future` -- Celluloid kicks off that method asynchronously and returns you a `Celluloid::Future` object. Calling `value` will block until the value has been computed. 468 | 469 | Celluloid is a great solution to concurrency that puts the abstraction at the right level and wraps up a lot of best practices. 470 | 471 | ```ruby 472 | fetcher = XKCDFetcher.new 473 | fetcher.next 474 | fetcher.async.next 475 | fetcher.future.next 476 | ``` 477 | ## Chapter 15. How Sidekiq Uses Celluloid 478 | Sidekiq multi-threaded processing is implemented on top of Celluloid. 479 | 480 | > As soon as you introduce the Thread constant, you've probably just 481 | > introduced 5 new bugs into your code. 482 | 483 | Sidekq is a system composed of Celluloid Actors. 484 | Manager actor - holds the state of the whole system, mediates between collaborator actors. 485 | Collaborator actors: 486 | the Fetcher - fetches the jobs from Redis 487 | the Processors - perform the actual work of the jobs 488 | 489 | ### Into the source 490 | In actor-based system the building blocks are not classes or methods, but messages. 491 | 492 | ### fetch (Manager) 493 | Manager starts: 494 | ```ruby 495 | def start 496 | @ready.each { dispatch } 497 | end 498 | ``` 499 | `@ready` holds references to the `Processor` actors that are ready to process jobs. 500 | `dispatch` does some housekeeping and does `@fetcher.async.fetch` 501 | 502 | The Fetcher actor will process each fetch in turn, as new jobs get pushed into Redis. 503 | 504 | The Fetcher first tries to retrieve a unit of work. If it doesn't retrieve any work, it calls itself again (waiting) 505 | 506 | When it's finally able to retrieve work, it asynchronously sends the assign message to the Manager: `@mgr.async.assign(work)` 507 | 508 | ### `Manager#assign` 509 | receives the unit of work. 510 | 511 | First, the Manager grabs the next available Processor, then keeps tracks of its status appropriately in its internal data structures. 512 | ```ruby 513 | processor = @ready.pop 514 | @in_progress[processor_object_id] = work 515 | @busy << processor 516 | processor.async.process(work) # fire and forget, not waiting for a response. 517 | ``` 518 | The Manager actor lives in its own thread, it own these instance variables, and doesn't share them with other actors. So it can use regular ruby arrays. 519 | 520 | The Processor will perform the work, then send a message back to the Manager when it's finished. 521 | 522 | the `process` method is most entirely focused on actually performing the job. 523 | 524 | ### Wrap-up 525 | The most obvious difference I see between the Sidekiq codebase and a more traditional Ruby codebase is the lack of dependence upon return values. 526 | Return values are seldom used. Instead, actors expect a message to be sent back in return. 527 | 528 | Sidekiq is a great example of how simple it can be to integrate multi-threaded 529 | concurrency, via actors, with your business logic. 530 | 531 | ## Chapter 16. Puma's Thread Pool Implementation 532 | Puma 1 is a concurrent web server for Rack apps that uses multiple threads for concurrency. 533 | 534 | ### A what now? 535 | Puma's thread pool is responsible for spawning the worker threads and feeding them work. 536 | The thread pool wraps up all of the multi-threaded concerns, and the rest of the server only cares about the app logic. 537 | `Puma::ThreadPool` is actually a generic class. 538 | Once initialized, the pool is responsible for receiving work and feeding it to an available worker thread. 539 | auto-trimming feature - the number of active threads is kept minimum, but more threads can be spawned 540 | 541 | ### Code 542 | `todo` is a shared array that holds work to be done. 543 | Inside a mutex hold: 544 | Endless loop. 545 | `todo` is empty => no work to be done 546 | If there's no need to shutdown => wait 547 | Waiting: Increments a global counter saying that it's going to wait. 548 | Next, wait on the shared condition variable, this releases the mutex and puts the current thread to sleep. 549 | It won't be woken up again until there's some work to do. Since it released the shared mutex, another thread can go through the same routine. 550 | 551 | Once enough work arrives, this thread will get woken up. 552 | It will re-acquire the shared mutex and decrement `@waiting` counter. 553 | Then pops the unit of work from `todo`. (`work = todo.pop if continue`) 554 | 555 | Outside the mutex.synchronize block: 556 | ```ruby 557 | break unless continue # time to shut down => break 558 | block.call(work, *extra) 559 | ``` 560 | Not time to shut down => call `block` with the `work` object. 561 | The block is passed in to the constructor and is the block of code that each worker thread will perform. 562 | 563 | ## Chapter 17. Closing 564 | The safest path to concurrency: 565 | 1. Don't do it. 566 | 2. If you must do it, don't share data across threads. 567 | 3. If you must share data across threads, don't share mutable data. 568 | 4. If you must share mutable data across threads, synchronize access to that 569 | data. 570 | 571 | ### Ruby concurrency doesn't suck 572 | 573 | ## Chapter 18 Appendix: Atomic Compare-and-set Operations 574 | 575 | Compare-and-set (CAS) 576 | 577 | Same as `@counter += 1` 578 | ```ruby 579 | @counter = 0 580 | # Get the current value of `@counter`. 581 | current_value = @counter 582 | # Increment the retrieved value by 1. 583 | new_value = current_value + 1 584 | # Assign the new value back to `@counter`. 585 | @counter = new_value 586 | ``` 587 | This multi-step operation is that it's not atomic and may lead to a race condition. 588 | 589 | Solutions: using mutex or using `Atomic` 590 | ```ruby 591 | require 'atomic' 592 | @counter = Atomic.new(0) 593 | @counter.update { |current_value| 594 | current_value + 1 595 | } 596 | ``` 597 | The whole operation is retried if the compare_and_set fails. 598 | 599 | For some kinds of operations, locking (with mutexes) is very expensive, so `Atomic` is preferrable. 600 | 601 | #### Benchmarks 602 | The results differ wildly between implementations. 603 | With JRuby, lockless and locking variants work on par. 604 | With Rubinius the lockelss variant is much faster. 605 | 606 | ### CAS as a primitive 607 | Some new technologies, such as Clojure, take full advantage of this and are showing what's really possible with multi-threaded concurrency 608 | Ruby -- gem concurrent-ruby. 609 | 610 | ## Chapter 19. Appendix: Thread-safety and Immutability 611 | The main problem of thread-safety is concurrent modification. 612 | So, by definition, immutable objects are thread-safe. 613 | Having only immutable objects sounds like a simple solution, but pure functional programming languages are usually hard to master. 614 | 615 | ### Immutable Ruby objects 616 | Immutability is actually supported in core Ruby using the `Object#freeze` method. 617 | The typical method signature for immutable objects is: methods that would typically mutate the object instead return a new version of the object with the mutation applied. 618 | `gem 'hamster'` -- Efficient, Immutable, Thread-Safe Collection classes for Ruby 619 | 620 | ### Integrating immutability 621 | The simplest use case is this: when you need to share objects between 622 | threads, share immutable objects. 623 | 624 | It's very easy to pass out immutable objects to share, but if you need to have multiple threads modifying an immutable object you still need some form of synchronization. 625 | 626 | In this case, immutable data structures work great with CAS operations. 627 | ```ruby 628 | @queue_wrapper = Atomic.new(Hamster::Queue.new) 629 | ... 630 | @queue_wrapper.update { |queue| 631 | queue.enqueue(rand(100)) 632 | } 633 | .... 634 | consumers << Thread.new do 635 | @queue_wrapper.update { |queue| 636 | number = queue.head 637 | queue.dequeue 638 | } 639 | end 640 | ``` 641 | 642 | #### Questions 643 | 644 | 1-6) 645 | - What is gil? 646 | - How does it work (in system terms)? 647 | - How does GIL deal with waiting for IO-operations? 648 | 7) 649 | - When does it make sense to spawn more threads then the number of processor cores. 650 | - How to decide how many threads you need? 651 | - How to decide how many threads you need if your program involves heavy math calculations? 652 | - What is check-then-set race condition? 653 | 8) 654 | - What objects are thread-safe by default? 655 | 9) 656 | - what is mutex? 657 | - How to use mutex? 658 | - Provide a classic deadlock example? 659 | - What is a livelock? 660 | - How do you avoid deadlocks/livelocks? 661 | 10) 662 | - how do you use conditional variables? 663 | - what if you just check the results instead of signaling? 664 | 11) 665 | - What are the caveats of the multithreaded code? 666 | - What global objects are there in your ruby program? 667 | - How do you avoid problems with global objects? 668 | 16) 669 | - How does sidekiq work? 670 | - What do manager, fetched and processors do? 671 | 17. 672 | - What are the rules for the thread-safe programming? 673 | --------------------------------------------------------------------------------