├── .gitattributes ├── .gitignore ├── README.md ├── pictures ├── GPM.png ├── block.png ├── ctx-kv.png ├── d-w.png ├── defer.png ├── gc-less.png ├── gc-more.png ├── gc-process.gif ├── gc-root.png ├── gin.png ├── go-line.png ├── go-thread.png ├── gp-thread-blocking.png ├── java-thread.png ├── less32.png ├── load.png ├── map-get.png ├── memory-span.png ├── overflow.png ├── quad-heap.gif ├── ssa.png ├── stack-heap.png ├── sync-map.png └── y-w.png └── problems ├── array.md ├── curly-brackets-and-square-brackets.md ├── defer.md ├── error-handling.md ├── garbage-collector.md ├── gin.md ├── go-garbage-collector.md ├── go-memory-allocate.md ├── go-versions.md ├── godep.md ├── golang-sensitive-problem.md ├── gopath-and-goroot.md ├── how-context-works.md ├── how-goroutine-works.md ├── how-to-use-channel.md ├── map-in-go.md ├── more-sync.md ├── mutex.md ├── new-one-slice.md ├── package.md ├── panic-and-recover.md ├── return-multiple-values.md ├── slice-in-go.md ├── string.md ├── struct-interface-pointer.md ├── struct.md ├── swap-the-values-of-two-variables.md ├── sync-map.md ├── timer.md ├── type-assertion.md ├── use-new-or-make.md ├── waitgroup.md └── what-is-gpm-and-why-need-p.md /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | *.DS_Store 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Golang Internal 2 | 3 | 4 | 5 | Go language has many amazing features and syntax, such as how to swap the values of two variables. 6 | 7 | ```go 8 | package main 9 | 10 | import "fmt" 11 | 12 | func main() { 13 | a := 5 14 | b := 10 15 | fmt.Println("Before swapping, values of a and b are:", a, b) 16 | a, b = b, a 17 | fmt.Println("After swapping, values of a and b are:", a, b) 18 | } 19 | ``` 20 | 21 | Output: 22 | 23 | ``` 24 | Before swapping, values of a and b are: 5 10 25 | After swapping, values of a and b are: 10 5 26 | ``` 27 | 28 | 29 | 30 | This interesting syntax feature is used to test whether an engineer has enough understanding of the Go language. But why is this way of exchanging values valid? How is this syntax feature implemented in Go? 31 | 32 | The goal of this project is to answer a series of questions like this from the perspective of the underlying implementation in Go, by explaining the source code, including syntax, data structures, and a series of concurrency issues. 33 | 34 | 35 | 36 | ### Problems 37 | 38 | #### Variable and Keywords 39 | 40 | 1. [How to Swap the Values of Two Variables](problems/swap-the-values-of-two-variables.md) 41 | 2. [Use New or Make](problems/use-new-or-make.md) 42 | 3. [String literals, Byte and Rune](problems/string.md) 43 | 4. [Struct, Interface, Pointer](problems/struct-interface-pointer.md) 44 | 5. [How to Use Defer](problems/defer.md) 45 | 6. [Panic and Recover](problems/panic-and-recover.md) 46 | 47 | 48 | 49 | #### GPM 50 | 51 | 1. [How Goroutine Works](problems/how-goroutine-works.md) 52 | 2. [How to Use Channel](problems/how-to-use-channel.md) 53 | 3. [How Context Works](problems/how-context-works.md) 54 | 55 | 56 | 57 | #### GC 58 | 59 | 1. [Go Memory Allocate](problems/go-memory-allocate.md) 60 | 2. [How Garbage Collector Works? ](problems/go-garbage-collector.md) 61 | 62 | 63 | 64 | #### Map 65 | 66 | 1. [Map in Go](problems/map-in-go.md) 67 | 2. [New One: Slice](problems/new-one-slice.md) 68 | 69 | 70 | 71 | #### Concurrency 72 | 73 | 1. [Sync.map](problems/sync-map.md) 74 | 2. [More Sync: Sync.pool, Sync.once](problems/more-sync.md) 75 | 3. [Mutex](problems/mutex.md) 76 | 4. [Waitgroup](problems/waitgroup.md) 77 | 5. [Timer](problems/timer.md) 78 | 79 | 80 | 81 | #### Develop Tools 82 | 83 | 1. [Gin](problems/gin.md) 84 | 2. Godep 85 | 86 | 87 | 88 | #### Code Practices 89 | 90 | 1. [Array](problems/array.md) 91 | 2. Stack 92 | 3. Linked-List 93 | 4. String 94 | 5. Binary Tree 95 | 6. Binary Search 96 | 7. Graph 97 | 8. Dynamic Programming 98 | 9. Binary Search Tree 99 | 10. Hash Table 100 | 11. Binary 101 | 12. Heap 102 | 13. Trie 103 | 14. Recursion 104 | 15. Matrix 105 | 106 | 107 | 108 | #### What's more 109 | 110 | 1. Go Version Update 111 | 2. [What Are Golang Packages?](problems/package.md) 112 | 3. [Is Golang Case sensitive or Insensitive?](problems/golang-sensitive-problem.md) 113 | 4. [Return Multiple Values From A Function in Go?](problems/return-multiple-values.md) 114 | 5. [Type Assertion in Go](problems/type-assertion.md) 115 | 6. [How Is GoPATH Different From GoROOT Variables In Go?](problems/gopath-and-goroot.md) 116 | 7. [In Go, Are There Any Good Error Handling Practices?](problems/error-handling.md) 117 | 8. Expressions. Diff b/w Curly Brackets and Square Brackets? 118 | -------------------------------------------------------------------------------- /pictures/GPM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/GPM.png -------------------------------------------------------------------------------- /pictures/block.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/block.png -------------------------------------------------------------------------------- /pictures/ctx-kv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/ctx-kv.png -------------------------------------------------------------------------------- /pictures/d-w.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/d-w.png -------------------------------------------------------------------------------- /pictures/defer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/defer.png -------------------------------------------------------------------------------- /pictures/gc-less.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/gc-less.png -------------------------------------------------------------------------------- /pictures/gc-more.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/gc-more.png -------------------------------------------------------------------------------- /pictures/gc-process.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/gc-process.gif -------------------------------------------------------------------------------- /pictures/gc-root.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/gc-root.png -------------------------------------------------------------------------------- /pictures/gin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/gin.png -------------------------------------------------------------------------------- /pictures/go-line.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/go-line.png -------------------------------------------------------------------------------- /pictures/go-thread.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/go-thread.png -------------------------------------------------------------------------------- /pictures/gp-thread-blocking.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/gp-thread-blocking.png -------------------------------------------------------------------------------- /pictures/java-thread.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/java-thread.png -------------------------------------------------------------------------------- /pictures/less32.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/less32.png -------------------------------------------------------------------------------- /pictures/load.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/load.png -------------------------------------------------------------------------------- /pictures/map-get.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/map-get.png -------------------------------------------------------------------------------- /pictures/memory-span.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/memory-span.png -------------------------------------------------------------------------------- /pictures/overflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/overflow.png -------------------------------------------------------------------------------- /pictures/quad-heap.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/quad-heap.gif -------------------------------------------------------------------------------- /pictures/ssa.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/ssa.png -------------------------------------------------------------------------------- /pictures/stack-heap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/stack-heap.png -------------------------------------------------------------------------------- /pictures/sync-map.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/sync-map.png -------------------------------------------------------------------------------- /pictures/y-w.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/weiwensangsang/golang-internal/efeee8aa6e322a6cdaeda8407c8a9815d54ba250/pictures/y-w.png -------------------------------------------------------------------------------- /problems/array.md: -------------------------------------------------------------------------------- 1 | # Array 2 | 3 | ### Two Sum 4 | 5 | ```go 6 | func Solution(nums []int, target int) []int { 7 | indexMap := make(map[int]int) 8 | for currIndex, currNum := range nums { 9 | if requiredIdx, isPresent := indexMap[target-currNum]; isPresent { 10 | return []int{requiredIdx, currIndex} 11 | } 12 | indexMap[currNum] = currIndex 13 | } 14 | return []int{} 15 | } 16 | 17 | func TestSolution(t *testing.T) { 18 | input1 := []int{2, 7, 11, 15} 19 | input2 := 9 20 | expectedOutput := []int{0, 1} 21 | output := Solution(input1, input2) 22 | t.Logf("Solution(%v, %v) return %v, we want %v", input1, input2, output, expectedOutput) 23 | } 24 | ``` 25 | 26 | 27 | -------------------------------------------------------------------------------- /problems/curly-brackets-and-square-brackets.md: -------------------------------------------------------------------------------- 1 | # Brackets -------------------------------------------------------------------------------- /problems/defer.md: -------------------------------------------------------------------------------- 1 | # How to use defer? 2 | 3 | 4 | 5 | Let us start with code. What is the result for this code? defer is first in last out, This is very natural, the following statement will depend on the previous resource, so if the previous resource is released first, the subsequent statement cannot be executed. 6 | 7 | ```go 8 | package main 9 | 10 | import ( 11 | "fmt" 12 | ) 13 | 14 | func main() { 15 | defer_call() 16 | } 17 | 18 | func defer_call() { 19 | defer func() { fmt.Println("1") }() 20 | defer func() { fmt.Println("2") }() 21 | defer func() { fmt.Println("3") }() 22 | 23 | panic("ERROR") 24 | } 25 | ``` 26 | 27 | Little weird but it should be, 28 | 29 | ```text 30 | 3 31 | 2 32 | 1 33 | panic: ERROR 34 | ``` 35 | 36 | 37 | 38 | ### defer 39 | 40 | The execution logic of defer, return, and return value should be: return is executed first, and return is responsible for writing the result into the return value; then defer starts to perform some finishing work; finally, the function exits with the current return value. 41 | 42 | In fact, to understand the defer closure, as long as you understand the following points, you will fully understand 43 | 44 | 1. 45 | Closure is to manipulate variables in the parent function through pointers 46 | 47 | 2. 48 | The process defined by defer is equivalent to generating a function body and then pushing it onto the stack, and then popping it out for execution when it returns 49 | 50 | 3. 51 | defer is executed before return. It should be noted here that if the return variable is not declared, you need to declare a return value first and then assign it to return. 52 | 53 | 54 | 55 | ```go 56 | type number int 57 | 58 | func (n number) print() { fmt.Println(n) } 59 | func (n *number) pprint() { fmt.Println(*n) } 60 | 61 | func main() { 62 | var n number 63 | 64 | defer n.print() 65 | defer n.pprint() 66 | defer func() { n.print() }() 67 | defer func() { n.pprint() }() 68 | 69 | n = 3 70 | } 71 | ``` 72 | 73 | At first n = 0. 74 | 75 | 76 | 77 | The first defer, push the code **0.print()** into defer stack (linkedList). 78 | 79 | The second defer, push the code **(pointer of n).print()** into defer stack. 80 | 81 | The third defer, push the code **func() { (pointer of n).print() }()** into defer stack. 82 | 83 | The 4th defer, push the code **func() { (pointer of n).pprint() }()** into defer stack. 84 | 85 | 86 | 87 | Then n=3. 88 | 89 | Then return. 90 | 91 | Then esecute 4th defer, print 3. 92 | 93 | Then esecute third defer, print 3. 94 | 95 | Then esecute second defer, print 3. 96 | 97 | Then esecute first defer, print 0. 98 | 99 | 100 | 101 | ### What's more: How is defer implemented 102 | 103 | It can be seen that the defer function in the Go language is a **last-in-first-out** mechanism. Why is it like this? 104 | 105 | let's take a look at the implementation of defer(From src/runtime/runtime2.go): 106 | 107 | 108 | 109 | ```go 110 | type _defer struct { 111 | started bool 112 | heap bool 113 | // openDefer indicates that this _defer is for a frame with open-coded 114 | // defers. We have only one defer record for the entire frame (which may 115 | // currently have 0, 1, or more defers active). 116 | openDefer bool 117 | sp uintptr // sp at time of defer 118 | pc uintptr // pc at time of defer 119 | fn func() // can be nil for open-coded defers 120 | _panic *_panic // panic that is running defer 121 | link *_defer // next defer on G; can point to either heap or stack! 122 | 123 | // If openDefer is true, the fields below record values about the stack 124 | // frame and associated function that has the open-coded defer(s). sp 125 | // above will be the sp for the frame, and pc will be address of the 126 | // deferreturn call in the function. 127 | fd unsafe.Pointer // funcdata for the function associated with the frame 128 | varp uintptr // value of varp for the stack frame 129 | // framepc is the current pc associated with the stack frame. Together, 130 | // with sp above (which is the sp associated with the stack frame), 131 | // framepc/sp can be used as pc/sp pair to continue a stack trace via 132 | // gentraceback(). 133 | framepc uintptr 134 | } 135 | ``` 136 | 137 | ![defer](../pictures/defer.png) 138 | 139 | Intuitively, defer should be directly inserted by the compiler into the required function call. It seems to be a compile-time feature, and there should be no runtime performance problems. 140 | 141 | But the actual situation is that since defer is not linked to its dependent resources, **it is also allowed to appear in conditions and loop statements**, which makes the semantics of defer relatively complicated, and sometimes it is impossible to determine how many defer calls exist at compile time. This makes it possible for defer to be implemented on the heap or on the stack. 142 | 143 | 144 | 145 | The main steps for the Go compiler to compile the Go code written by the programmer into machine-executable binary code are: 146 | 1) The compiler entry is in the gc.Main() method of the cmd/compile/internal/gc/main.go package; 147 | 148 | 2) gc.Main() performs lexical analysis, syntax analysis and type checking, and generates AST; 149 | 150 | 3) Main() calls the walk.Walk() method of the cmd/compile/internal/walk/walk.go package to traverse and rewrite the AST nodes in the code to generate the final abstract syntax tree AST. It should be noted that some keywords and built-in functions will be converted into runtime function calls in the walk.Walk() method, for example, the two built-in functions panic and recover will be converted into runtime.gopanic and runtime.gorecover Two real runtime functions, the keyword new will also be converted to call the runtime.newobject function, and **keywords such as Channel, map, make, new and select will be converted into corresponding runtime functions; and the main processing of the defer keyword logic is not here;** 151 | 152 | 4) Then, Main() generates the SSA intermediate code from the abstract syntax tree AST, and specifically calls ssagen.buildssa(); 153 | 154 | 5) ssagen.buildssa() calls the state.stmtList() of the same file, state.stmtList() will call the state.stmt() method for each node passed in, and state.stmt() will The current AST node is converted into the corresponding intermediate code; **Note: the processing of the defer keyword is here in the state.stmt() method;** 155 | 156 | From cmd/compile/internal/ssagen/ssa.go, we can see there are three ways of defer 157 | 158 | ```go 159 | case ir.ODEFER: 160 | n := n.(*ir.GoDeferStmt) 161 | if base.Debug.Defer > 0 { 162 | var defertype string 163 | if s.hasOpenDefers { 164 | defertype = "open-coded" 165 | } else if n.Esc() == ir.EscNever { 166 | defertype = "stack-allocated" 167 | } else { 168 | defertype = "heap-allocated" 169 | } 170 | base.WarnfAt(n.Pos(), "%s defer", defertype) 171 | } 172 | if s.hasOpenDefers { 173 | s.openDeferRecord(n.Call.(*ir.CallExpr))// open defer 174 | } else { 175 | d := callDefer // heap 176 | if n.Esc() == ir.EscNever { 177 | d = callDeferStack //satch 178 | } 179 | s.callResult(n.Call.(*ir.CallExpr), d) 180 | } 181 | ``` 182 | 183 | From this logic we can know: 184 | 185 | 1) There are three ways to implement defer: 186 | 1) open coding, 187 | 188 | 2) stack allocation 189 | 190 | 3) and heap allocation 191 | 192 | 2) If open coding is allowed, this method is preferred to implement defer, which was introduced by Go1.14 and has the best performance; we will analyze later that the conditions for realizing open coding are: 193 | 1) The compiler does not set parameters -N, that is, no inline optimization is disabled; 194 | 195 | 2) the number of defer in the function does not exceed the number of bits in a byte, that is, no more than 8; 196 | 197 | 3) the product of the number of defer functions and the number of parameters is less than 15; 198 | 199 | 4) The defer keyword cannot be executed in a loop. The defer call scenario that meets these conditions is simple, and most of the information can be determined at compile time; 200 | 201 | 3) If there is no memory escape to the heap, the stack allocation method is preferred to implement defer. This is an optimization method introduced by Go1.13, which reduces the extra overhead of memory allocation on the heap and improves the performance by about 30%; 202 | 203 | 4) If the previous two methods do not meet the conditions, then the method of heap allocation is used to implement defer by default; 204 | 205 | The following first analyzes the implementation method of defer allocation on the heap first adopted by Go, then analyzes the allocation of defer on the stack, and finally the implementation method of open coding. 206 | 207 | 208 | 209 | 1. Go first introduced the implementation of defer allocation on the heap. The implementation principle is: the compiler first converts the defer delay statement into a runtime.deferproc call, and inserts the runtime.deferreturn function before the return of the defer function; Call the runtime.deferproc function to obtain a runtime._defer structure to record the relevant parameters of the defer function, and push the _defer structure onto the stack of the current Goroutine's defer delay chain head; when the defer function returns, call the runtime.deferreturn function from the Goroutine The linked list head takes out the runtime._defer structure and executes it sequentially, which ensures that multiple defers in the same function can be executed in LIFO order; the performance problem of this type of defer is that each defer statement must allocate memory on the heap, worst performance; 210 | 211 | 2. Go1.13 introduces the implementation of defer allocation on the stack. The implementation principle is: the compiler will directly create a runtime._defer structure on the stack without memory allocation, and pass the _defer structure pointer as a parameter. Enter the runtime.deferprocStack function, and at the same time, runtime._defer will be pushed into the delayed call list of Goroutine; when the function exits, deferreturn will be called, and the runtime._defer structure will be popped and executed; the cost of this type of defer is that it needs to run The parameters determined at time are assigned to the runtime._defer structure created on the stack, and the runtime performance is 30% higher than that allocated on the heap; 212 | 213 | 3. Go1.14 introduces an open coding method to implement defer, which realizes defer calls with almost zero cost. When -N is not set, inline is disabled, the product of the number of defer functions and the number of return values does not exceed 15, and the number of defer functions When there are no more than 8 defers and the defer does not appear in the loop statement, this type of open coding method will be used to implement the defer; the implementation principle is: the compiler will store the defer-related parameters according to the delay bits deferBits and the state.openDeferInfo structure, and return At the end of the statement, the defer call is executed according to whether the relevant bit of the delay bit is 1; the runtime cost of this type of defer is to determine whether the relevant defer function needs to be executed according to the condition and the delay bit, and the runtime performance of open coding is the best. 214 | 215 | 216 | 217 | ### One issue 218 | 219 | This issue fixed in Go 1.18 and pretty funny. What happens if there is a defer function in deadcode? 220 | https://github.com/golang/go/issues/51839 221 | 222 | 223 | 224 | ```go 225 | package main 226 | 227 | func main() { 228 | testRecover() 229 | 230 | } 231 | 232 | func testRecover() { 233 | if false { 234 | func() { 235 | defer func() { 236 | _ = recover() 237 | }() 238 | }() 239 | } 240 | } 241 | ``` 242 | 243 | 244 | Before 1.18, it will create a error message. 245 | 246 | ``` 247 | walk . 248 | RECOVER INTER-interface {} tc(1) internal compiler error: walkExpr: switch 1 unknown op RECOVER 249 | ``` 250 | 251 | 252 | How to fix? 253 | 254 | ```go 255 | func markHiddenClosureDead(n ir.Node) { 256 | if n.Op() != ir.OCLOSURE { 257 | return 258 | } 259 | clo := n.(*ir.ClosureExpr) 260 | if clo.Func.IsHiddenClosure() { 261 | clo.Func.SetIsDeadcodeClosure(true) 262 | } 263 | ir.VisitList(clo.Func.Body, markHiddenClosureDead) // add this new line to fix this issue. 264 | } 265 | ``` 266 | 267 | This is a clever way to use recursion to continuously parse ir nodes to fix this type of deadcode problem. 268 | -------------------------------------------------------------------------------- /problems/error-handling.md: -------------------------------------------------------------------------------- 1 | # Error Handling 2 | 3 | Golang's Error Handling has always been controversial, and the community continue to propose various improvement plans. As a language-level error support, Go's implementation of error is extremely simple, even rudimentary. So, what is the best practice for Golang Error Handling? 4 | 5 | ### Errors are values 6 | 7 | "Errors are values" originated from the design concept of error by Rob Pike, one of the founders of the Go language. Rob Pike believes that error is equal to other return values of functions, and it is just one of many return values, and there is nothing special about it. Therefore, errors are treated just like the return value of a function. 8 | 9 | After understanding Rob Pike's design philosophy, you can naturally understand why it appears repeatedly in the Go language: 10 | 11 | ```go 12 | if err != nil { 13 | return err 14 | } 15 | ``` 16 | 17 | Because error is an ordinary return value, it is also handled simply in the code flow. However, for most cases of the Go language, the error handling method only needs to determine the non-null return. Therefore, the Go language is often complained about for its lengthy and repetitive error handling. 18 | 19 | 20 | 21 | The error design of the Go language has been controversial in long-term practice. The main complaints of developers are: 22 | 23 | 1. Error processing is interspersed in the code of Go language, which separates the normal logic code and affects the readability. 24 | 2. A large number of repeated if err != nil fragments cannot be simplified. 25 | 3. A simple return err cannot be used in all scenarios. 26 | 27 | 28 | 29 | Go2 plans to help simplify error's separation of code logic and a large number of repetitions of if err != nil by introducing two keywords **handle** and **check**. 30 | 31 | Go1 32 | 33 | ```go 34 | func printSum(a, b string) error { 35 | x, err := strconv.Atoi(a) 36 | if err != nil { 37 | return err 38 | } 39 | y, err := strconv.Atoi(b) 40 | if err != nil { 41 | return err 42 | } 43 | fmt.Println("result:", x + y) 44 | return nil 45 | } 46 | ``` 47 | 48 | Go2： 49 | 50 | ```go 51 | func printSum(a, b string) error { 52 | handle err { return err } 53 | x := check strconv.Atoi(a) 54 | y := check strconv.Atoi(b) 55 | fmt.Println("result:", x + y) 56 | return nil 57 | } 58 | ``` -------------------------------------------------------------------------------- /problems/garbage-collector.md: -------------------------------------------------------------------------------- 1 | fixme: not start -------------------------------------------------------------------------------- /problems/gin.md: -------------------------------------------------------------------------------- 1 | # Gin 2 | 3 | https://github.com/gin-gonic/gin 4 | 5 | ### Gin 6 | 7 | > `Gin` is a `Web` framework written in `Golang`, a routing based on `Radix` tree, small memory footprint, no reflection, predictable `API` framework, the speed is nearly 40 times higher, support Middleware, routing group processing, JSON and other multi-mode validation, built-in `JSON`, `XML`, `HTML` and other rendering. It is an easy-to-use `API` framework; this series of articles will specifically reveal the source code of `Gin`. Please see an example below: 8 | 9 | 10 | 11 | ```go 12 | func main() { 13 | // init a rounter 14 | r := gin.New() 15 | 16 | // global middleware 17 | // Logger: into gin.DefaultWriter 18 | // By default gin.DefaultWriter = os.Stdout 19 | r.Use(gin.Logger()) 20 | 21 | // Recovery: recover any panic。Will be 500。 22 | r.Use(gin.Recovery()) 23 | 24 | r.GET("/benchmark", MyBenchLogger(), benchEndpoint) 25 | 26 | // authorized := r.Group("/", AuthRequired()) 27 | authorized := r.Group("/") 28 | 29 | // AuthRequired() 30 | authorized.Use(AuthRequired()) 31 | { 32 | authorized.POST("/login", loginEndpoint) 33 | authorized.POST("/submit", submitEndpoint) 34 | authorized.POST("/read", readEndpoint) 35 | 36 | testing := authorized.Group("testing") 37 | testing.GET("/analytics", analyticsEndpoint) 38 | } 39 | 40 | r.Run(":8080") 41 | } 42 | ``` 43 | 44 | 45 | 46 | ### Engine 47 | 48 | > The following figure describes the entire `Engine` related structure combination; `gin.Engine` is a container object, which is the basis of the entire framework; `gin.RouterGroup` is responsible for storing all middleware, including request paths; `gin.trees `Responsible for storing the mapping between routes and `handle` methods, using the `Radix` tree structure; the following will focus on a brief description of `Engine` parameters; other parts will be analyzed in subsequent series of articles. 49 | 50 | 51 | 52 | ![gin](../pictures/gin.png) 53 | 54 | ``` 55 | RouterGroup 56 | ``` 57 | 58 | > Responsible for storing all middleware, including request paths. 59 | 60 | ``` 61 | RedirectTrailingSlash 62 | ``` 63 | 64 | > Whether to enable redirection requests for routes with the suffix /, such as requesting /foo/ but only /foo routes exist, the client uses http status 307 to redirect to /foo for requests 65 | 66 | ``` 67 | RedirectFixedPath 68 | ``` 69 | 70 | > If this parameter is enabled, the router will try to fix the current request address by itself when no handler is registered. 71 | > 1. The first redundant element will be deleted (../ or //); 72 | > 2. Then the routing will perform a case-insensitive lookup on the new path; 73 | > 3. If the corresponding handler can be found normally, the route will be redirected to the correct handler and return 301 or 307. (For example: users visiting /FOO and /..//Foo may be redirected to /foo on the route) 74 | 75 | ``` 76 | HandleMethodNotAllowed 77 | ``` 78 | 79 | > Check if the current path allows routing using other request methods. 80 | 81 | ``` 82 | ForwardedByClientIP 83 | ``` 84 | 85 | > Whether to forward the client ip. 86 | 87 | ``` 88 | Remote IP Headers 89 | ``` 90 | 91 | > Obtain the client IP in the following cases. ForwardedByClientIP is true; context.Request.RemoteAddr has a value 92 | 93 | ``` 94 | Trusted Proxies 95 | ``` 96 | 97 | > Network origin list (IPv4 address, IPv4 CIDR, IPv6 address or trust IPv6 CIDR of a request header containing: when `(*gin.Engine).ForwardedByClientIP` is `true`. 98 | 99 | ``` 100 | App Engine 101 | ``` 102 | 103 | > If enabled, a header starting with "X-AppEngine..." will be added to the request. 104 | 105 | ``` 106 | UseRawPath 107 | ``` 108 | 109 | > If enabled, url.RawPath will be used to find parameters 110 | 111 | ``` 112 | UnescapePathValues 113 | ``` 114 | 115 | > If true, path values will not be escaped. If UseRawPath is false (which is the default), UnescapePathValues is actually true, as url.Path will be used, which has been escaped. 116 | 117 | ``` 118 | MaxMultipartMemory 119 | ``` 120 | 121 | > The value assigned to the 'maxMemory' parameter of ParseMultipartForm of http.Request. 122 | 123 | ``` 124 | RemoveExtraSlash 125 | ``` 126 | 127 | > Whether to remove extra backslashes (requests with extra slashes can be parsed at the beginning) 128 | 129 | ``` 130 | delims 131 | ``` 132 | 133 | > Delims represent a set of left and right delimiters for HTML template rendering. 134 | 135 | ``` 136 | secureJSONPrefix 137 | ``` 138 | 139 | > Set the json prefix in Context.SecureJSON China 140 | 141 | ``` 142 | HTMLRender 143 | ``` 144 | 145 | > Return HTMLRender interface (used to render HTMLProduction and HTMLDebug structure type templates) 146 | 147 | ``` 148 | FuncMap 149 | ``` 150 | 151 | > FuncMap map[string]interface{} in the html/template package, used to define the mapping from names to functions. 152 | 153 | ``` 154 | allNoRoute 155 | ``` 156 | 157 | > Make a copy of the global `handlers`, add `NoRoute` processing method 158 | 159 | ``` 160 | allNoMethod 161 | ``` 162 | 163 | > Make a copy of the global `handlers`, add the `NoMethod` processing method 164 | 165 | ``` 166 | noRoute 167 | ``` 168 | 169 | > noRoute adds a handler for NoRoute. It returns 404 code by default 170 | 171 | ``` 172 | noMethod 173 | ``` 174 | 175 | > noMethod adds a handler for NoMethod, which returns a 405 code by default 176 | 177 | ``` 178 | pool 179 | ``` 180 | 181 | > Mainly used to store `context` context object, used to optimize performance when processing `http` requests 182 | 183 | ``` 184 | trees 185 | ``` 186 | 187 | > Responsible for storing the mapping of routes and handle methods, using the Radix tree structure. 188 | 189 | ``` 190 | maxParams 191 | ``` 192 | 193 | > When allocating `context`, initialize the slice length of the parameter `Params` 194 | 195 | ``` 196 | trustedCIDRs 197 | ``` 198 | 199 | > IP network address pointer slice 200 | 201 | 202 | 203 | ### Handler request 204 | 205 | #### Init Gin instance 206 | 207 | > The `Gin` framework exposes two methods for initializing `Gin` instances, the `New` and `Default` methods. The difference between the two is that the `Default` method adds the use of `Logger` and `Recovery` comes with middleware processing. If you need to write these two middleware by yourself, you can add the `Use` method on the basis of calling `New` to register the middleware. Two core functions are mainly implemented in the initialization of the `Gin` instance: the initialization of the routing group `RouterGroup` and the initialization of the `pool` are mainly used to store the `context` context object, which is used to optimize the performance of `http` request processing. 208 | 209 | ```go 210 | // return Engine 211 | func New() *Engine { 212 | debugPrintWARNINGNew() 213 | engine := &Engine{ 214 | RouterGroup: RouterGroup{ //init RouterGroup 215 | Handlers: nil, 216 | basePath: "/", 217 | root: true, 218 | }, 219 | FuncMap: template.FuncMap{}, 220 | RedirectTrailingSlash: true, 221 | RedirectFixedPath: false, 222 | HandleMethodNotAllowed: false, 223 | ForwardedByClientIP: true, 224 | RemoteIPHeaders: []string{"X-Forwarded-For", "X-Real-IP"}, 225 | TrustedProxies: []string{"0.0.0.0/0"}, 226 | AppEngine: defaultAppEngine, 227 | UseRawPath: false, 228 | RemoveExtraSlash: false, 229 | UnescapePathValues: true, 230 | MaxMultipartMemory: defaultMultipartMemory, 231 | trees: make(methodTrees, 0, 9), //init router tree 232 | delims: render.Delims{Left: "{{", Right: "}}"}, 233 | secureJSONPrefix: "while(1);", 234 | } 235 | engine.RouterGroup.engine = engine 236 | engine.pool.New = func() interface{} { //init Pool 237 | return engine.allocateContext() 238 | } 239 | return engine 240 | } 241 | 242 | func Default() *Engine { 243 | debugPrintWARNINGDefault() 244 | engine := New() 245 | engine.Use(Logger(), Recovery()) 246 | return engine 247 | } 248 | ``` 249 | 250 | #### Register Middleware 251 | 252 | > There are mainly two kinds of middleware in `Gin` framework: global middleware and routing group middleware. 253 | 254 | - `Global Middleware`: Write records to `RouterGroup.Handlers` by calling `Gin.Use()` 255 | 256 | ```go 257 | func (engine *Engine) Use(middleware ...HandlerFunc) IRoutes { 258 | engine.RouterGroup.Use(middleware...) 259 | engine.rebuild404Handlers() 260 | engine.rebuild405Handlers() 261 | return engine 262 | } 263 | func (group *RouterGroup) Use(middleware ...HandlerFunc) IRoutes { 264 | group.Handlers = append(group.Handlers, middleware...) 265 | return group.returnObj() 266 | } 267 | ``` 268 | 269 | - `Router Group Middleware`: Return a newly generated `RouterGroup` pointer through `RouterGroup.Group`, which is used to separate each routing group to load different middleware. Note that `group.combineHandlers(handlers)` will copy a copy of the global middleware to the newly generated `RouterGroup`. 270 | 271 | ```go 272 | func (group *RouterGroup) Group(relativePath string, handlers ...HandlerFunc) *RouterGroup { 273 | return &RouterGroup{ 274 | Handlers: group.combineHandlers(handlers), 275 | basePath: group.calculateAbsolutePath(relativePath), 276 | engine: group.engine, 277 | } 278 | } 279 | ``` 280 | 281 | #### Router 282 | 283 | > Several methods for registering routes are specified in the `Gin` framework: `GET`, `POST`, `DELETE`, `PUT`, `HEAD`, `OPTIONS`, `ANY`, `PATCH`. Regardless of the request method, `RouterGroup.handler` will be called to add routes. In this method, `group.combineHandlers(handlers)` will also copy a copy of the global middleware and add the `handlers` of the corresponding route. Form a `List` into the tree nodes, and finally call `tree.addRoute` to add nodes. 284 | 285 | ```go 286 | func (group *RouterGroup) POST(relativePath string, handlers ...HandlerFunc) IRoutes { 287 | return group.handle(http.MethodPost, relativePath, handlers) 288 | } 289 | func (group *RouterGroup) GET(relativePath string, handlers ...HandlerFunc) IRoutes { 290 | return group.handle(http.MethodGet, relativePath, handlers) 291 | } 292 | func (group *RouterGroup) DELETE(relativePath string, handlers ...HandlerFunc) IRoutes { 293 | return group.handle(http.MethodDelete, relativePath, handlers) 294 | } 295 | func (group *RouterGroup) PATCH(relativePath string, handlers ...HandlerFunc) IRoutes { 296 | return group.handle(http.MethodPatch, relativePath, handlers) 297 | } 298 | func (group *RouterGroup) PUT(relativePath string, handlers ...HandlerFunc) IRoutes { 299 | return group.handle(http.MethodPut, relativePath, handlers) 300 | } 301 | func (group *RouterGroup) OPTIONS(relativePath string, handlers ...HandlerFunc) IRoutes { 302 | return group.handle(http.MethodOptions, relativePath, handlers) 303 | } 304 | func (group *RouterGroup) HEAD(relativePath string, handlers ...HandlerFunc) IRoutes { 305 | return group.handle(http.MethodHead, relativePath, handlers) 306 | } 307 | func (group *RouterGroup) Any(relativePath string, handlers ...HandlerFunc) IRoutes { 308 | group.handle(http.MethodGet, relativePath, handlers) 309 | group.handle(http.MethodPost, relativePath, handlers) 310 | group.handle(http.MethodPut, relativePath, handlers) 311 | group.handle(http.MethodPatch, relativePath, handlers) 312 | group.handle(http.MethodHead, relativePath, handlers) 313 | group.handle(http.MethodOptions, relativePath, handlers) 314 | group.handle(http.MethodDelete, relativePath, handlers) 315 | group.handle(http.MethodConnect, relativePath, handlers) 316 | group.handle(http.MethodTrace, relativePath, handlers) 317 | return group.returnObj() 318 | } 319 | func (group *RouterGroup) handle(httpMethod, relativePath string, handlers HandlersChain) IRoutes { 320 | absolutePath := group.calculateAbsolutePath(relativePath) 321 | handlers = group.combineHandlers(handlers) 322 | group.engine.addRoute(httpMethod, absolutePath, handlers) 323 | return group.returnObj() 324 | } 325 | func (engine *Engine) addRoute(method, path string, handlers HandlersChain) { 326 | assert1(path[0] == '/', "path must begin with '/'") 327 | assert1(method != "", "HTTP method can not be empty") 328 | assert1(len(handlers) > 0, "there must be at least one handler") 329 | 330 | debugPrintRoute(method, path, handlers) 331 | 332 | root := engine.trees.get(method) 333 | if root == nil { 334 | root = new(node) 335 | root.fullPath = "/" 336 | engine.trees = append(engine.trees, methodTree{method: method, root: root}) 337 | } 338 | root.addRoute(path, handlers) 339 | 340 | if paramsCount := countParams(path); paramsCount > engine.maxParams { 341 | engine.maxParams = paramsCount 342 | } 343 | } 344 | ``` 345 | 346 | #### Start 347 | 348 | - `HTTP` 349 | 350 | ```go 351 | func (engine *Engine) Run(addr ...string) (err error) { 352 | defer func() { debugPrintError(err) }() 353 | 354 | trustedCIDRs, err := engine.prepareTrustedCIDRs() 355 | if err != nil { 356 | return err 357 | } 358 | engine.trustedCIDRs = trustedCIDRs 359 | address := resolveAddress(addr) 360 | debugPrint("Listening and serving HTTP on %s\n", address) 361 | err = http.ListenAndServe(address, engine) 362 | return 363 | } 364 | ``` 365 | 366 | - `HTTPS` 367 | 368 | ```go 369 | func (engine *Engine) RunTLS(addr, certFile, keyFile string) (err error) { 370 | debugPrint("Listening and serving HTTPS on %s\n", addr) 371 | defer func() { debugPrintError(err) }() 372 | err = http.ListenAndServeTLS(addr, certFile, keyFile, engine) 373 | return 374 | } 375 | ``` 376 | 377 | - `Unix Socket` 378 | 379 | ```go 380 | func (engine *Engine) RunUnix(file string) (err error) { 381 | debugPrint("Listening and serving HTTP on unix:/%s", file) 382 | defer func() { debugPrintError(err) }() 383 | 384 | listener, err := net.Listen("unix", file) 385 | if err != nil { 386 | return 387 | } 388 | defer listener.Close() 389 | defer os.Remove(file) 390 | 391 | err = http.Serve(listener, engine) 392 | return 393 | } 394 | ``` 395 | 396 | - `File` 397 | 398 | ```go 399 | func (engine *Engine) RunFd(fd int) (err error) { 400 | debugPrint("Listening and serving HTTP on fd@%d", fd) 401 | defer func() { debugPrintError(err) }() 402 | 403 | f := os.NewFile(uintptr(fd), fmt.Sprintf("fd@%d", fd)) 404 | listener, err := net.FileListener(f) 405 | if err != nil { 406 | return 407 | } 408 | defer listener.Close() 409 | err = engine.RunListener(listener) 410 | return 411 | } 412 | ``` 413 | 414 | #### Handler request 415 | 416 | ```go 417 | func (engine *Engine) ServeHTTP(w http.ResponseWriter, req *http.Request) { 418 | c := engine.pool.Get().(*Context) 419 | c.writermem.reset(w) 420 | c.Request = req 421 | c.reset() 422 | 423 | engine.handleHTTPRequest(c) 424 | 425 | engine.pool.Put(c) 426 | } 427 | 428 | func (engine *Engine) handleHTTPRequest(c *Context) { 429 | httpMethod := c.Request.Method //http请求方法，get、post 430 | rPath := c.Request.URL.Path //path 431 | unescape := false 432 | if engine.UseRawPath && len(c.Request.URL.RawPath) > 0 { 433 | rPath = c.Request.URL.RawPath 434 | unescape = engine.UnescapePathValues 435 | } 436 | 437 | if engine.RemoveExtraSlash { 438 | rPath = cleanPath(rPath) 439 | } 440 | 441 | // scan httpMethod node in tree 442 | t := engine.trees 443 | for i, tl := 0, len(t); i < tl; i++ { 444 | if t[i].method != httpMethod { 445 | continue 446 | } 447 | root := t[i].root 448 | // find path 449 | value := root.getValue(rPath, c.params, unescape) 450 | if value.params != nil { 451 | c.Params = *value.params //Params 452 | } 453 | if value.handlers != nil { 454 | c.handlers = value.handlers //handlers 455 | c.fullPath = value.fullPath //fullPath 456 | c.Next() 457 | c.writermem.WriteHeaderNow() 458 | return 459 | } 460 | 461 | if httpMethod != "CONNECT" && rPath != "/" { 462 | if value.tsr && engine.RedirectTrailingSlash { 463 | redirectTrailingSlash(c) 464 | return 465 | } 466 | if engine.RedirectFixedPath && redirectFixedPath(c, root, engine.RedirectFixedPath) { 467 | return 468 | } 469 | } 470 | break 471 | } 472 | 473 | if engine.HandleMethodNotAllowed { 474 | for _, tree := range engine.trees { 475 | if tree.method == httpMethod { 476 | continue 477 | } 478 | 479 | if value := tree.root.getValue(rPath, nil, unescape); value.handlers != nil { 480 | c.handlers = engine.allNoMethod 481 | serveError(c, http.StatusMethodNotAllowed, default405Body) 482 | return 483 | } 484 | } 485 | } 486 | // http code = 404 487 | c.handlers = engine.allNoRoute 488 | serveError(c, http.StatusNotFound, default404Body) 489 | } 490 | ``` -------------------------------------------------------------------------------- /problems/go-garbage-collector.md: -------------------------------------------------------------------------------- 1 | # GC 2 | 3 | ### **tricolor mark-and-sweep algorithm** 4 | 5 | **** 6 | 7 | 8 | 9 | ![gc-process](../pictures/gc-process.gif) 10 | 11 | The three-color method divides the color of the object into three colors: black, gray, and white. 12 | 13 | 1. Black: The object has been marked, and all the attributes under the object have been marked (the object required by the program); Black object is save and should stay. 14 | 2. Gray: The object has been marked, but the properties under the object have not been marked (the GC needs to find garbage from this object); 15 | 3. White: the object has not been marked (object garbage); 16 | 17 | When the garbage collector starts working, the traversal visit starts from GC Roots, and the visit steps can be divided into the following steps: 18 | 19 | **GC Roots root objects will be marked in gray**; 20 | Then get the object from the gray collection, mark it black, and mark the object that this object references to as gray; 21 | Repeat step 2 until there are no gray sets to mark; 22 | After the end, the remaining unmarked white objects are unreachable by GC Roots and can be recycled. 23 | The process is roughly as follows: 24 | 25 | ![gc-root](../pictures/gc-root.png) 26 | 27 | ### Problems 28 | 29 | #### More Garabge 30 | 31 | Assuming that E has been marked (turned gray), D and E have broken the reference at this time. It is reasonable to say that the object E/F/G should be recycled, but because E has turned gray, it will still be It is regarded as a living object and continues to traverse. The final result is: this part of the object will still be marked as alive, that is, the current round of GC will not reclaim this part of memory. 32 | 33 | This part of the memory that should have been reclaimed but not reclaimed is called "floating garbage". The process is shown in the figure below: 34 | 35 | 36 | 37 | ![gc-more](../pictures/gc-more.png) 38 | 39 | 40 | 41 | In addition to the problem of multiple labels above, there is also the problem of missing labels. When the GC thread has traversed until E becomes gray and D becomes black, gray E disconnects reference to white G, and black D references white G. At this time, switch back to the GC thread and continue running, because E has no reference to G, so G will not be placed in the gray set. Although G is re-referenced by D, because D is already black, the traversal process will not be re-traversed. 42 | 43 | The final result is: G will always stay in the white set, and finally be cleared as garbage. This directly affects the correctness of the application, which is unacceptable, and this is also a problem that Go needs to solve during GC. 44 | 45 | ![gc-more](../pictures/gc-less.png) 46 | 47 | ### Memory Barrier 48 | 49 | A **memory barrier**, is a type of barrier instruction that causes a central processing unit (CPU) or compiler to enforce an ordering constraint on memoryoperations issued before and after the barrier instruction. This typically means that operations issued prior to the barrier are guaranteed to be performed before operations issued after the barrier. 50 | 51 | Then in order to ensure correctness in the marking algorithm, then we need to meet any of the following conditions: 52 | 53 | - Strong tri-color invariant: black objects will not point to white objects, only gray objects or black objects; 54 | 55 | - Weak tri-color invariant: Even if a black object points to a white object, starting from the gray object, there is always a path to find the white object; 56 | 57 | According to the different types of operations, we can divide the memory barrier into two types: Read barrier and Write barrier. In Go, both use Write barrier. The reason is also in "Uniprocessor Garbage Collection Techniques". mentioned: 58 | 59 | If a non copying collector is used the use of a read barrier is an unnecessary expense. there is no need to protect the mutator from seeing an invalid version of a pointer. Write barrier techniques are cheaper, because heap writes are several times less common than heap reads 60 | 61 | For a garbage collector that does not need object copying, the Read barrier (read barrier) is very expensive, because for this type of garbage collector, it is not necessary to save the version pointer problem of the read operation. Relatively speaking, the Write barrier (write barrier) code is smaller, because the write operation in the heap is much smaller than the read operation in the heap. 62 | 63 | Let's take a look at how the Write barrier (write barrier) does this. 64 | 65 | ### Dijkstra Write barrier 66 | 67 | Before Go 1.7, Dijkstra Write barrier was used, and the implementation used was similar to the following pseudocode: 68 | 69 | ```go 70 | writePointer(slot, ptr): 71 | shade(ptr) 72 | *slot = ptr 73 | ``` 74 | 75 | If the object is white, shade(ptr) will mark the object as gray. This can ensure strong three-color invariance, and it will ensure that the object pointed to by the ptr pointer is not white before being assigned to *slot. 76 | 77 | As follows, the D object pointed to by the root object is marked black and the object E pointed to by the D object is marked gray; if D breaks the reference to E and changes to refer to the B object, then the write barrier is triggered and the B object is marked gray . 78 | 79 | ![d-w](../pictures/d-w.png) 80 | 81 | Although Dijkstra Write barrier is very simple to implement and can also guarantee strong three-color invariance, it also has some shortcomings in "Proposal: Eliminate STW stack re-scanning": 82 | 83 | In particular, it presents a trade-off for pointers on stacks: either writes to pointers on the stack must have write barriers, which is prohibitively expensive, or stacks must be permagrey. 84 | 85 | Because objects on the stack are also considered root objects in garbage collection, either a write barrier is added to the objects on the stack, but this will greatly increase the overhead of writing pointers; or when a write operation on the stack occurs , marking the stack as permagrey. 86 | 87 | Go 1.7 chose to mark the stack as constant gray, but these stacks need to be re-scanned (re-scan) when marking the termination phase STW. The reason is described below: 88 | 89 | without stack write barriers, we can't ensure that the stack won't later contain a reference to a white object, so a scanned stack is only black until its goroutine executes again, at which point it conservatively reverts to grey. At the end of the cycle, the garbage collector must re-scan gray stacks to blacken them and finish marking any remaining heap pointers. 90 | 91 | 92 | 93 | ### Yuasa Write barrier 94 | 95 | Yuasa Write barrier is a deletion barrier technology proposed by Yuasa in "Real-time garbage collection on general-purpose machines". The idea is that when an evaluator removes a white pointer from a gray or white object, concurrently executing collectors are notified of this behavior through a write barrier. 96 | 97 | The algorithm uses the write barrier shown below to ensure the correctness of the program when incrementally or concurrently executing garbage collection. The pseudocode implementation is as follows: 98 | 99 | ```go 100 | writePointer(slot, ptr) 101 | shade(*slot) 102 | *slot = ptr 103 | ``` 104 | 105 | To prevent losing the path from the gray object to the white object, it should be assumed that the *slot may turn black. To ensure that ptr does not turn white before being assigned to the slot, shade(slot) will first mark the *slot gray, Furthermore, the write operation always creates a path from gray to gray or gray to white objects, so deleting the write barrier can ensure the weak three-color invariance, and the downstream objects referenced by the old objects must be referenced by gray objects. 106 | 107 | ![y-w](../pictures/y-w.png) 108 | 109 | ### Hybrid write barrier 110 | 111 | Before Go 1.7, Dijkstra Write barrier was used to ensure three-color invariance. Go must ensure that the reference of the object does not change when rescanning, so it will suspend the program (STW), mark all stack objects as gray and rescan, which usually takes 10-100 milliseconds. 112 | 113 | Through the introduction of Proposal: Eliminate STW stack re-scanning, we can know that in order to eliminate the performance loss caused by re-scanning, Go uses Hybrid write barrier (hybrid write barrier) in 1.8, which combines Yuasa write barrier and Dijkstra write barrier. The pseudocode of the implementation is as follows: 114 | 115 | ```go 116 | CopywritePointer(slot, ptr): 117 | shade(*slot) 118 | if current stack is grey: 119 | shade(ptr) 120 | *slot = ptr 121 | ``` 122 | 123 | > the write barrier shades the object whose reference is being overwritten, and, if the current goroutine's stack has not yet been scanned, also shades the reference being installed. 124 | 125 | In `go\src\runtime\malloc.go` ： 126 | 127 | ```go 128 | Copyfunc mallocgc(size uintptr, typ *_type, needzero bool) unsafe.Pointer { 129 | ... 130 | dataSize := size 131 | c := gomcache() 132 | var x unsafe.Pointer 133 | 134 | noscan := typ == nil || typ.ptrdata == 0 135 | // maxSmallSize=32768 32k 136 | if size <= maxSmallSize { 137 | // maxTinySize= 16 bytes 138 | if noscan && size < maxTinySize { 139 | ... 140 | } else { 141 | ... 142 | } 143 | 144 | } else { 145 | ... 146 | } 147 | ... 148 | 149 | if gcphase != _GCoff { 150 | gcmarknewobject(span, uintptr(x), size, scanSize) 151 | } 152 | ... 153 | return x 154 | } 155 | ``` 156 | 157 | -------------------------------------------------------------------------------- /problems/go-memory-allocate.md: -------------------------------------------------------------------------------- 1 | # Go Memory Allocate 2 | 3 | In order to allow programmers to better focus on the implementation of business code, the Go language has added a **garbage collection** mechanism to automatically reclaim unused memory. 4 | 5 | Go has two places to allocate memory: **a global heap space** for dynamically allocating memory, and the other is each **goroutine's own stack** space. 6 | 7 | ![stack-heap](../pictures/stack-heap.png) 8 | 9 | ### Stack memory 10 | 11 | The memory in the stack area is generally allocated and released automatically by the compiler, which stores the input parameters of the function and local variables. These parameters will be created with the creation of the function and destroyed when the function returns. (via CPU push & release). 12 | 13 | Stack memory is the simplest and most efficient way to allocate and reclaim. It is essentially a "linear allocation" of a continuous memory. 14 | 15 | 16 | 17 | ### Heap memory 18 | 19 | The memory in the heap area is generally managed and allocated by the compiler and the engineer themselves, and released by the Runtime GC. Allocation on the heap must find **a large enough memory** to store new variable data. On subsequent frees, the garbage collector scans the heap space looking for objects that are no longer in use. 20 | Anytime a value is shared outside the scope of a function’s stack frame, it will be placed (or allocated) on the heap. 21 | 22 | Stack allocation is cheap, heap allocation is expensive 23 | 24 | 25 | 26 | ### So, are variables on the heap or on the stack? 27 | 28 | The Go declaration syntax **does not mention the stack and the heap**, but leaves it to the Go compiler to decide where to allocate memory to ensure the correctness of the program. 29 | 30 | The Go FAQ mentioned such an explanation: 31 | 32 | > To put it in perspective, you don't need to know. Every variable in Go persists as long as there is a reference to it. The storage location of variables (heap or stack) has nothing to do with the semantics of the language. 33 | 34 | The Go compiler will allocate local variables to functions in that function's stack frame if possible. But if the compiler cannot prove that the variable is unreferenced after the function returns, the compiler must allocate the variable on the heap that will be garbage collected to avoid dangling pointer errors. Also, if the local variable is very large, it might make more sense to store it on the heap rather than the stack. 35 | 36 | In current compilers, a variable is a candidate for allocation on the heap if it is addressed. But basic **escape analysis** can identify variables that don't outlive the function's return value, and can therefore be allocated on the stack. 37 | 38 | 39 | 40 | ### Escape analysis 41 | 42 | Determine whether to allocate it on the heap by checking whether the scope of the variable exceeds the stack where it is located", in which the behavior of "the scope of the variable exceeds the stack where it is located" is called escape. Escape analysis is static analysis in most languages: static code analysis at compile time determines whether a value can be allocated on the stack frame or needs to "escape" to the heap. 43 | 44 | 1. To reduce GC pressure, the variables on the stack are directly reclaimed by the system after the function exits, and they do not need to be marked and then cleared 45 | 46 | 2. Reduce memory fragmentation 47 | 48 | 3. Reduce the overhead of allocating heap memory and improve the running speed of the program 49 | 50 | 51 | You can use the command **go build -gcflags -m** to find out whether variables escape to the heap. 52 | 53 | Because the stack is more efficient than the heap and does not require GC, Go will allocate memory to the stack as much as possible. 54 | 55 | When allocation to the stack may cause problems such as illegal memory access, the heap will be used. The main scenarios are: 56 | 57 | 1. When a value may be accessed after the function is called, the value is most likely to be allocated on the heap. 58 | 2. When the compiler detects that a value is too large, the value is allocated on the heap. 59 | 3. When compiling, the compiler doesn't know the size of the value (slice, map...) the value will be allocated on the heap. 60 | 61 | 62 | 63 | ### Stack size 64 | 65 | When a Go application is running, each goroutine maintains its own stack area, which can only be used by itself and cannot be used by other goroutines. The initial size of the stack area is 2KB **(much smaller than the default thread stack of 2M under the x86_64 architecture)**. When the goroutine is running, the stack area will grow and shrink as needed. The default value of the maximum memory limit on the 64-bit system is 1GB. 66 | 67 | 1. v1.0 ~ v1.1 — the minimum stack memory space is 4KB 68 | 2. v1.2 — Increased minimum stack memory to 8KB 69 | 3. v1.3 — Use a contiguous stack instead of the previous version's segmented stack 70 | 4. v1.4 — Reduced minimum stack memory to 2KB (Contigous stacks) 71 | 72 | 73 | 74 | ### Memory Allocate 75 | 76 | `TCMalloc `is short for `Thread Cache Malloc` ，Go use the way of TCMalloc. In general, there are two problems to be solved. 77 | 78 | 1. memory fragmentation: With the continuous application and release of memory, there will be a large number of fragments in the memory, reducing the usage of memory 79 | 2. big lock: All threads under the same process share the same memory space. When they apply for memory, they need to be **locked**. If there is no lock, there will be a problem that the same memory is accessed by two threads at the same time. 80 | 81 | Let us start from some context. 82 | 83 | 1. **page**: Memory page, an 8K memory space. The memory application and release between Go and the operating system are all in page units. 84 | 2. **span**: memory block, one or more consecutive pages form a span. 85 | 3. **sizeclass**: space specification, each span has a sizeclass, marking how the pages in the span should be used. 86 | 4. **object**: object, used to store a variable data memory space, a span will be cut into a bunch of objects of equal size when it is initialized. Suppose the size of the object is 16B and the size of the span is 8K, then the page in the span will be initialized 8K / 16B = 512 objects. 87 | 88 | ![memory-span](../pictures/memory-span.png) 89 | 90 | 91 | 92 | #### Less than 16byte memory allocation 93 | 94 | For objects smaller than 16 bytes (and without pointers), the Go language divides them into **tiny** objects. The main purpose of dividing tiny objects is to handle extremely small strings and individual escaped variables. Benchmarks on json show that using tiny objects reduces allocations by 12% and heap size by 20%. The tiny object will be put into a span with class 2. 95 | 96 | 1. First check to see if there is free space in the previously allocated element 97 | 2. If the current size to be allocated is not enough, for example, to allocate a size of 16 bytes, then it is necessary to find the next free element 98 | 99 | The first step of tiny allocation is to try to use the allocated space of the previous element to save memory. 100 | 101 | 102 | 103 | #### Less than 32kb memory allocation 104 | 105 | When a small block of memory below 32kb is requested in the program, Go will allocate memory to the program from a local cache called **mcache**. Such a memory block is called mspan, which is the allocation unit when allocating memory to the program. 106 | 107 | In Go's scheduler model, each thread M is bound to a processor P(As we said before), which can only process and **run at most one goroutine in a single granularity of time**, and each P is bound to a local cache mcache mentioned above. When memory allocation is required, the currently running goroutine looks for available mspans from the mcache. **There is no need to lock when allocating memory from the local mcache**, and this allocation strategy is more efficient. 108 | 109 | If there is no free counterpart sizeclass mspan in **mcachce** when memory is allocated, Go also maintains an **mcentral** for each type of mspan, from which you can try to obtain resources. 110 | 111 | **mcentral** is shared by all worker threads, and there is competition among multiple goroutines. 112 | 113 | When **mcentral** has no free mspan, it will apply to **mheap**. When **mheap** has no resources, it will apply for new memory from the operating system. mheap is mainly used for memory allocation of large objects. 114 | 115 | All **mcentral** collections are stored in **mheap**. The **arena** area in **mheap** is a real heap area, and 8KB is regarded as a page at runtime, and all objects initialized on the heap are stored in these memory pages. The runtime uses a two-dimensional runtime.heapArena array to manage all memory, and each runtime.heapArena will manage 64MB of memory. 116 | 117 | If there is not enough space in the **arena** area, **runtime.mheap.sysAlloc** will be called to request more memory from the operating system. 118 | 119 | 120 | 121 | #### More than 32kb memory allocation 122 | 123 | Go cannot use the local cache mcache of the worker thread and the global central cache **mcentral** to manage memory allocation exceeding 32KB, so for those memory requests exceeding 32KB, the corresponding number of memory pages will be directly allocated from the **heap** (mheap) (per The page size is 8KB) to the program. -------------------------------------------------------------------------------- /problems/go-versions.md: -------------------------------------------------------------------------------- 1 | ### Versions -------------------------------------------------------------------------------- /problems/godep.md: -------------------------------------------------------------------------------- 1 | # Godep -------------------------------------------------------------------------------- /problems/golang-sensitive-problem.md: -------------------------------------------------------------------------------- 1 | # Case Sensitive 2 | 3 | Go uses character case to distinguish between public and private functions. -------------------------------------------------------------------------------- /problems/gopath-and-goroot.md: -------------------------------------------------------------------------------- 1 | # GOPATH and GOROOT 2 | 3 | Unlike other languages, there is no project statement in go, only packages, in which there are two important paths, GOROOT and GOPATH 4 | 5 | The environment variables related to Go development are as follows: 6 | 7 | - GOROOT: GOROOT is the installation directory of Go, (similar to java's JDK) 8 | - GOPATH: GOPATH is our workspace, saving go project code and third-party dependent packages 9 | 10 | Multiple GOPATHs can be set. Among them, the first one will be the default package directory. The package downloaded using go get will be in the src directory in the first path. When using go install, in which GOPATH is the package found? , the executable file will be generated in the bin directory under which GOPATH -------------------------------------------------------------------------------- /problems/how-context-works.md: -------------------------------------------------------------------------------- 1 | # How Context Works 2 | 3 | Always, we have some questions about context. 4 | 5 | 1. ​ how to use context? Is it dangerous? 6 | 2. ​ how context works and change values? 7 | 8 | 9 | 10 | **Context** can control the call of **Goroutine**'s operation, timeout, and cancellation method. 11 | 12 | Is there any other implementation method for these functions.？ 13 | 14 | Of course, there are, controlling the operation of **Goroutine**, can be implemented through the **Select**+**Channel** mechanism, overtime control can also be implemented through the **Timer**, the cancellation method can also send a signal to the **Channel**, and the notification method can be exited. 15 | 16 | Since **Context** can be implemented, there are other ways to achieve it, why do I still have to **Context**? 17 | 18 | In some complicated scenarios, the control is very cumbersome through **Channel** and other methods, and using **Context** can easily implement the above functions. 19 | 20 | 21 | 22 | ### Global Variable 23 | 24 | Let us see how to use **context** as global variable. fn1 create var i and fn2, fn3, fn4 use this. 25 | 26 | ```go 27 | func fn1() { 28 | var i int 29 | 30 | fn2(i) 31 | } 32 | 33 | func fn2(i int) { 34 | fn3(i) 35 | } 36 | 37 | func fn3(i int) { 38 | fn4(i) 39 | } 40 | 41 | func fn4(i int) { 42 | print(i) 43 | } 44 | ``` 45 | 46 | 47 | 48 | The code works fine, it's just a little bit verbose. 49 | But if one day the requirements change, fn4 does not need the "state" of i anymore? 50 | Then all code from fn4 to fn2 needs to remove the i state from the parameter declaration. 51 | This butterfly effect code change is undoubtedly huge. 52 | In order to solve the problem of "although only the parameter list of the last function was changed, all the functions of the whole project were changed in the end". 53 | We can abstract the behavior of "obtaining state" separately, and each function obtains parameters from a unified "parameter center". 54 | This eliminates the need to change function declarations when requirements change. 55 | It can be done like this: 56 | 57 | 1. Create the parameter center object ctx. 58 | 2. Modify the function to declare ctx as a parameter. 59 | 3. Share the "state" i into ctx. 60 | 4. Get status from parameter center. 61 | 62 | 63 | 64 | ```go 65 | type Context map[string]interface{} 66 | 67 | func fn1() { 68 | var ctx = make(Context) 69 | ctx["i"] = 1 70 | 71 | fn2(ctx) 72 | } 73 | 74 | func fn2(ctx Context) { 75 | fn3(ctx) 76 | } 77 | 78 | func fn3(ctx Context) { 79 | fn4(ctx) 80 | } 81 | 82 | func fn4(ctx Context) { 83 | var i = ctx["i"].(int) 84 | print(i) 85 | } 86 | ``` 87 | 88 | （In Java, we can use ThreadLocal.） 89 | 90 | 91 | 92 | The most common is the background HTTP/RPC Server. 93 | 94 | In the Go server, usually every request will start several **goroutines** to work at the same time: some **go** to the database to get data, and some call the downstream interface to get related data, as shown in the following figure: 95 | 96 | ![go-line](../pictures/go-line.png) 97 | 98 | The client generally does not wait indefinitely, and will be requested to set a timeout period, such as 100ms. 99 | 100 | For example, here GoroutineA consumes 80ms, GoroutineB3 consumes 30ms, and it has timed out, so the subsequent GoroutineCDEF does not need to be executed, and the client has returned after a timeout, so even if the server calculates the result, it is meaningless. 101 | 102 | So here you can use Context to transmit timeout signals between multiple Goroutines. 103 | 104 | At the same time, there are two benefits after introducing timeout control: 105 | 106 | 1) The client can return quickly to improve user experience 107 | 2) The server can reduce invalid calculations 108 | 109 | 110 | 111 | ### Context 112 | 113 | In Go services, an independent **goroutine** is often used to process a request, but in this **goroutine**, other **goroutines** may be opened to perform some specific transactions, such as database, RPC, etc. At the same time, this group of **goroutines** may also need Common access to some special values, such as user token, request expiration time, etc. When a request times out, we hope that all **goroutines** related to this request can exit quickly to reclaim system resources. 114 | 115 | Using it, you can easily pass specific values, cancellation signals, and deadlines to all goroutines involved in the request. 116 | 117 | Context was introduced into the standard library in version 1.7 of Go. The main contents can be summarized as follows: 118 | 119 | - 1 interface 120 | - Context 121 | - 4 implementations 122 | - emptyCtx 123 | - cancelCtx 124 | - timerCtx 125 | - valueCtx 126 | - 6 methods 127 | - background 128 | - TODO 129 | - WithCancel 130 | - With Deadline 131 | - With Timeout 132 | - WithValue 133 | 134 | 135 | 136 | 137 | 138 | The core of the context package is the Context interface, whose structure is as follows: 139 | 140 | ```go 141 | // A Context carries a deadline, a cancellation signal, and other values across 142 | // API boundaries. 143 | // 144 | // Context's methods may be called by multiple goroutines simultaneously. 145 | type Context interface { 146 | // Deadline returns the time when work done on behalf of this context 147 | // should be canceled. Deadline returns ok==false when no deadline is 148 | // set. Successive calls to Deadline return the same results. 149 | Deadline() (deadline time.Time, ok bool) 150 | 151 | // Done returns a channel that's closed when work done on behalf of this 152 | // context should be canceled. Done may return nil if this context can 153 | // never be canceled. Successive calls to Done return the same value. 154 | // The close of the Done channel may happen asynchronously, 155 | // after the cancel function returns. 156 | // 157 | // WithCancel arranges for Done to be closed when cancel is called; 158 | // WithDeadline arranges for Done to be closed when the deadline 159 | // expires; WithTimeout arranges for Done to be closed when the timeout 160 | // elapses. 161 | // 162 | // Done is provided for use in select statements: 163 | // 164 | // // Stream generates values with DoSomething and sends them to out 165 | // // until DoSomething returns an error or ctx.Done is closed. 166 | // func Stream(ctx context.Context, out chan<- Value) error { 167 | // for { 168 | // v, err := DoSomething(ctx) 169 | // if err != nil { 170 | // return err 171 | // } 172 | // select { 173 | // case <-ctx.Done(): 174 | // return ctx.Err() 175 | // case out <- v: 176 | // } 177 | // } 178 | // } 179 | // 180 | // See https://blog.golang.org/pipelines for more examples of how to use 181 | // a Done channel for cancellation. 182 | Done() <-chan struct{} 183 | 184 | // If Done is not yet closed, Err returns nil. 185 | // If Done is closed, Err returns a non-nil error explaining why: 186 | // Canceled if the context was canceled 187 | // or DeadlineExceeded if the context's deadline passed. 188 | // After Err returns a non-nil error, successive calls to Err return the same error. 189 | Err() error 190 | 191 | // Value returns the value associated with this context for key, or nil 192 | // if no value is associated with key. Successive calls to Value with 193 | // the same key returns the same result. 194 | // 195 | // Use context values only for request-scoped data that transits 196 | // processes and API boundaries, not for passing optional parameters to 197 | // functions. 198 | // 199 | // A key identifies a specific value in a Context. Functions that wish 200 | // to store values in Context typically allocate a key in a global 201 | // variable then use that key as the argument to context.WithValue and 202 | // Context.Value. A key can be any type that supports equality; 203 | // packages should define keys as an unexported type to avoid 204 | // collisions. 205 | // 206 | // Packages that define a Context key should provide type-safe accessors 207 | // for the values stored using that key: 208 | // 209 | // // Package user defines a User type that's stored in Contexts. 210 | // package user 211 | // 212 | // import "context" 213 | // 214 | // // User is the type of value stored in the Contexts. 215 | // type User struct {...} 216 | // 217 | // // key is an unexported type for keys defined in this package. 218 | // // This prevents collisions with keys defined in other packages. 219 | // type key int 220 | // 221 | // // userKey is the key for user.User values in Contexts. It is 222 | // // unexported; clients use user.NewContext and user.FromContext 223 | // // instead of using this key directly. 224 | // var userKey key 225 | // 226 | // // NewContext returns a new Context that carries value u. 227 | // func NewContext(ctx context.Context, u *User) context.Context { 228 | // return context.WithValue(ctx, userKey, u) 229 | // } 230 | // 231 | // // FromContext returns the User value stored in ctx, if any. 232 | // func FromContext(ctx context.Context) (*User, bool) { 233 | // u, ok := ctx.Value(userKey).(*User) 234 | // return u, ok 235 | // } 236 | Value(key any) any 237 | } 238 | ``` 239 | 240 | 241 | 242 | 243 | 244 | #### 4 implementations 245 | 246 | ##### emptyCtx 247 | 248 | ```go 249 | type emptyCtx int 250 | 251 | func (*emptyCtx) Deadline() (deadline time.Time, ok bool) { 252 | return 253 | } 254 | 255 | func (*emptyCtx) Done() <-chan struct{} { 256 | return nil 257 | } 258 | 259 | func (*emptyCtx) Err() error { 260 | return nil 261 | } 262 | 263 | func (*emptyCtx) Value(key interface{}) interface{} { 264 | return nil 265 | } 266 | 267 | var ( 268 | background = new(emptyCtx) 269 | todo = new(emptyCtx) 270 | ) 271 | 272 | func Background() Context { 273 | return background 274 | } 275 | func TODO() Context { 276 | return todo 277 | } 278 | ``` 279 | 280 | It has no function. Judging from the source code, **context.Background** and **context.TODO** are just aliases for each other, there is not much difference, but slightly different in usage and semantics: 281 | 282 | - **context.Background** is the default value of the context, all other contexts should be derived from it; 283 | - context.TODO should only be used when not sure which context should be used; 284 | 285 | In most cases, if the current function does not have a **context** as an input parameter, we will use **context.Background** as the initial context to pass down. 286 | 287 | 288 | 289 | ##### cancelCtx 290 | 291 | 292 | 293 | ```go 294 | type cancelCtx struct { 295 | // have a field Context 296 | Context 297 | 298 | mu sync.Mutex // protects following fields 299 | done atomic.Value // of chan struct{}, created lazily, closed by first cancel call 300 | children map[canceler]struct{} // set to nil by the first cancel call 301 | err error // set to non-nil by the first cancel call 302 | } 303 | 304 | 305 | type canceler interface { 306 | cancel(removeFromParent bool, err error) 307 | Done() <-chan struct{} 308 | } 309 | 310 | // create cancelCtx 311 | func WithCancel(parent Context) (ctx Context, cancel CancelFunc) { 312 | if parent == nil { 313 | panic("cannot create context from nil parent") 314 | } 315 | c := newCancelCtx(parent) 316 | propagateCancel(parent, &c) 317 | return &c, func() { c.cancel(true, Canceled) } 318 | } 319 | func newCancelCtx(parent Context) cancelCtx { 320 | return cancelCtx{Context: parent} 321 | } 322 | ``` 323 | 324 | 325 | 326 | At the same time, propagateCancel will build the association between parent and child contexts to form a tree structure. When the parent context is canceled, the child context will also be cancelled: 327 | 328 | ```go 329 | func propagateCancel(parent Context, child canceler) { 330 | done := parent.Done() 331 | if done == nil { 332 | return // parent is never canceled 333 | } 334 | select { 335 | case <-done: 336 | // parent is already canceled 337 | child.cancel(false, parent.Err()) 338 | return 339 | default: 340 | } 341 | if p, ok := parentCancelCtx(parent); ok { 342 | p.mu.Lock() 343 | if p.err != nil { 344 | // parent has already been canceled 345 | child.cancel(false, p.err) 346 | } else { 347 | if p.children == nil { 348 | p.children = make(map[canceler]struct{}) 349 | } 350 | p.children[child] = struct{}{} 351 | } 352 | p.mu.Unlock() 353 | } else { 354 | atomic.AddInt32(&goroutines, +1) 355 | go func() { 356 | select { 357 | case <-parent.Done(): 358 | child.cancel(false, parent.Err()) 359 | case <-child.Done(): 360 | } 361 | }() 362 | } 363 | } 364 | 365 | ``` 366 | 367 | In total, the above functions relate to three different cases of the parent **context**: 368 | 369 | 1) When **parent.Done()** == nil, that is, the parent will not trigger the cancellation event, the current function will return directly; 370 | 2) When the child's inheritance chain contains a **context** that can be canceled, it will determine whether the parent has triggered the cancellation signal; 371 | 1) If it has been canceled, the child will be canceled immediately; 372 | 2) If not canceled, the child will be added to the parent's children list, waiting for the parent to release the cancel signal; 373 | 3) When the parent context is a developer-defined type, implements the context.Context interface, and returns a non-empty pipeline in the Done() method; 374 | Run a new Goroutine while listening to the parent.Done() and child.Done() Channels; 375 | call child.cancel to cancel the child context when parent.Done() is closed; 376 | 377 | 378 | 379 | ##### timerCtx 380 | 381 | TimerCtx not only inherits related variables and methods by embedding **cancelCtx**, but also implements the function of timing cancellation through the held timer and deadline: 382 | 383 | ```go 384 | type timerCtx struct { 385 | cancelCtx 386 | timer *time.Timer // Under cancelCtx.mu. 387 | 388 | deadline time.Time 389 | } 390 | 391 | func (c *timerCtx) Deadline() (deadline time.Time, ok bool) { 392 | return c.deadline, true 393 | } 394 | 395 | func (c *timerCtx) cancel(removeFromParent bool, err error) { 396 | c.cancelCtx.cancel(false, err) 397 | if removeFromParent { 398 | removeChild(c.cancelCtx.Context, c) 399 | } 400 | c.mu.Lock() 401 | if c.timer != nil { 402 | c.timer.Stop() 403 | c.timer = nil 404 | } 405 | c.mu.Unlock() 406 | } 407 | 408 | func WithTimeout(parent Context, timeout time.Duration) (Context, CancelFunc) { 409 | return WithDeadline(parent, time.Now().Add(timeout)) 410 | } 411 | 412 | 413 | func WithDeadline(parent Context, d time.Time) (Context, CancelFunc) { 414 | if parent == nil { 415 | panic("cannot create context from nil parent") 416 | } 417 | if cur, ok := parent.Deadline(); ok && cur.Before(d) { 418 | // The current deadline is already sooner than the new one. 419 | return WithCancel(parent) 420 | } 421 | c := &timerCtx{ 422 | cancelCtx: newCancelCtx(parent), 423 | deadline: d, 424 | } 425 | propagateCancel(parent, c) 426 | dur := time.Until(d) 427 | if dur <= 0 { 428 | c.cancel(true, DeadlineExceeded) // deadline has already passed 429 | return c, func() { c.cancel(false, Canceled) } 430 | } 431 | c.mu.Lock() 432 | defer c.mu.Unlock() 433 | if c.err == nil { 434 | c.timer = time.AfterFunc(dur, func() { 435 | c.cancel(true, DeadlineExceeded) 436 | }) 437 | } 438 | return c, func() { c.cancel(true, Canceled) } 439 | } 440 | ``` 441 | 442 | 443 | 444 | ##### valueCtx 445 | 446 | ```go 447 | type valueCtx struct { 448 | Context 449 | key, val interface{} 450 | } 451 | 452 | func WithValue(parent Context, key, val interface{}) Context { 453 | if parent == nil { 454 | panic("cannot create context from nil parent") 455 | } 456 | if key == nil { 457 | panic("nil key") 458 | } 459 | if !reflectlite.TypeOf(key).Comparable() { 460 | panic("key is not comparable") 461 | } 462 | return &valueCtx{parent, key, val} 463 | } 464 | 465 | func (c *valueCtx) Value(key interface{}) interface{} { 466 | if c.key == key { 467 | return c.val 468 | } 469 | return c.Context.Value(key) 470 | } 471 | ``` 472 | 473 | The process of getting the value is actually a recursive search process. If the key is consistent with the value stored in the current ctx, it will be returned directly, if not, it will be searched in the parent. Finally find the root node (usually emptyCtx), directly return a nil. So when using the Value method, it is necessary to judge whether the result is nil. 474 | 475 | ![ctx-kv](../pictures/ctx-kv.png) 476 | 477 | 478 | 479 | 480 | 481 | #### Advice 482 | 483 | 1. Do not store Contexts inside a struct type; instead, pass a Context explicitly to each function that needs it. The Context should be the first parameter, typically named ctx. 484 | 2. Do not pass a nil Context, even if a function permits it. Pass context.TODO if you are unsure about which Context to use. 485 | 3. Use context Values only for request-scoped data that transits processes and APIs, not for passing optional parameters to functions. 486 | 4. The same Context may be passed to functions running in different goroutines; Contexts are safe for simultaneous use by multiple goroutines. 487 | 488 | 489 | 490 | #### Reference 491 | 492 | 1. https://pkg.go.dev/context 493 | -------------------------------------------------------------------------------- /problems/how-goroutine-works.md: -------------------------------------------------------------------------------- 1 | # How Goroutine Works 2 | 3 | Let us start with some questions about Goroutine, 4 | 5 | 1. Single-core CPU, open two Goroutines, one of which has an infinite loop, what will happen? 6 | 2. Processes and threads have IDs, why doesn't Goroutine have an ID? 7 | 3. How much is the right number of Goroutines, will it affect GC and scheduling? 8 | 4. What are the Goroutine leaks? 9 | 5. What can cause a Goroutine to hang? 10 | 11 | 12 | 13 | At first, let us see how Java create threads (In Java 19, they add a new feature called Virtual Threads, this thing just works like Goroutine, so we just compare Go with old version Java). 14 | 15 | 16 | 17 | ### Java Thread 18 | 19 | Java thread just use native thread provided by OS，and it related on OS to scheduler. 20 | 21 | ![java-thread](../pictures/java-thread.png) 22 | 23 | Here is the Java code for 1000 threads. 24 | 25 | ```java 26 | public static void main(String []args) throws InterruptedException { 27 | for (int i = 0 ;i<1000;i++){ 28 | new Thread(()->{ 29 | try { 30 | Thread.sleep(100000000); 31 | } catch (InterruptedException e) { 32 | e.printStackTrace(); 33 | } 34 | }).start(); 35 | } 36 | Thread.sleep(100000000); 37 | } 38 | ``` 39 | 40 | It will actually create 1000 threads (Java need some more threads, like for GC) 41 | 42 | ``` 43 | weiwensnagsang:~$ ps -T 100000 | wc -l1018 44 | 1018 45 | ``` 46 | 47 | 48 | 49 | In the best case, OS thread switching also requires the following operations, 50 | 51 | 1. CPU registers need to be saved and loaded. 52 | 2. The code of the system scheduler needs to be executed. 53 | 3. The CPU pipeline needs to be cleared. 54 | 55 | 56 | 57 | The Java has been trying to optimize this problem. Because if a language wants to support high concurrency, the loss of thread switching is a serious bottleneck, and it will greatly drag down the corresponding time of the system as a whole. 58 | 59 | In some extreme cases, the CPU may only spend 30% of its time on work, and the remaining 70% of its time is spent on creating, maintaining, and switching threads. 60 | 61 | 62 | 63 | ### Goroutine 64 | 65 | 66 | Compared with Java using native threads and relying on the native scheduler of the OS to schedule, goroutine implements its own scheduler and schedules goroutines to execute between fixed threads by itself: 67 | 68 | ![go-thread](../pictures/go-thread.png) 69 | 70 | 71 | 72 | Here is the Go code for 1000 threads. 73 | 74 | ```go 75 | func doSomething() { 76 | time.Sleep(10 * time.Minute) 77 | } 78 | 79 | func main() { 80 | for i := 0; i < 100000; i++ { 81 | go doSomething() 82 | } 83 | 84 | time.Sleep(10 * time.Minute) 85 | 86 | } 87 | ``` 88 | 89 | 90 | Thread will switch to another thread approximately every 10ms of executing a goroutine. And the priority order of thread picking goroutine is 91 | 92 | 1. goroutines in each thread's own queue 93 | 2. goroutines in the global queue 94 | 3. Stealing from other thread's queue (work-stealing) 95 | 96 | 97 | 98 | How many threads for this Go code? 99 | 100 | ``` 101 | 5 102 | ``` 103 | 104 | 105 | So far so good, but what if the thread gets stuck with a blocking system call (ex. reading a large file)? For example, in the figure below, there are three goroutines reading large files through the io system call. At this time, all goroutines will only rely on one thread for execution, which greatly reduces the utilization rate of the core. 106 | 107 | 108 | ![block](../pictures/block.png) 109 | 110 | In order to solve this problem, golang separates another layer of process between thread and goroutine as follows: 111 | 112 | 113 | ![GPM](../pictures/GPM.png) 114 | 115 | When a thread is blocked by the system call block, golang will create a new thread to take over the work of the processor, while the original thread will continue to execute the system call. 116 | 117 | 118 | 119 | ![gp-thread-blocking](../pictures/gp-thread-blocking.png) 120 | 121 | If we open 1000 goroutines to read a large file test, when Golang frequently opens goroutines to call blocking system calls, the number of threads created will probably be between **100 and 200**, which also shows that in some extreme cases, the concurrency of Go may It will degenerate to Java using native thread. 122 | 123 | This is what we called GPM. 124 | 125 | 126 | 127 | ### GMP 128 | 129 | 1. G: go coroutine (Goroutine), each **go** keyword will create a coroutine. 130 | 131 | 2. M: Machine thread. Kernel-level thread, all G must be placed on M to run. 132 | 133 | 3. P: schedules G to M, and maintains a queue that stores all G that needs to be scheduled. 134 | 135 | 136 | The Goroutine scheduler P and the OS scheduler are combined through M, each M represents a kernel thread, and the OS scheduler is responsible for assigning the kernel thread to the core of the CPU for execution 137 | 138 | 139 | 140 | ### Main function 141 | 142 | We can start with a very speical Goroutine: g0. G0 is for main function so it means main is also a goroutine. 143 | 144 | ```go 145 | package main 146 | 147 | import "fmt" 148 | 149 | func main() { 150 | fmt.Println("Hello world") 151 | } 152 | ``` 153 | 154 | In the code, GMP works like this. 155 | 156 | 1. The runtime creates the initial thread m0 and goroutine g0, and associates the two. 157 | 2. Scheduler initialization: Initialize m0, stack, garbage collection, and create and initialize a P-list of GOMAXPROCS P's. 158 | 3. The main function in the sample code is main.main, and there is also a main function in the runtime——runtime.main. After the code is compiled, runtime.main will call main.main. When the program starts, a goroutine will be created for runtime.main, called Let it be the main goroutine, and then add the main goroutine to the local queue of P. 159 | 4. Start m0, m0 has been bound to P, will get G from P's local queue, and get the main goroutine. 160 | 5. G owns the stack, and M sets the operating environment according to the stack information and scheduling information in G 161 | 6. M run G 162 | 7. G exits, returns to M again to obtain a runnable G, and repeats until main.main exits, runtime.main executes Defer and Panic processing, or calls runtime.exit to exit the program. 163 | 164 | 165 | 166 | The starter code for g0 and m0 is in this file: /go/src/runtime/asm_amd64.s. 167 | 168 | ```assembly 169 | TEXT runtime·rt0_go(SB),NOSPLIT|NOFRAME|TOPFRAME,$0 170 | // copy arguments forward on an even stack 171 | // Code to handle command line arguments 172 | MOVQ DI, AX // argc 173 | MOVQ SI, BX // argv 174 | SUBQ $(5*8), SP // 3args 2auto // Expand the stack by 40 bytes, 175 | ANDQ $~15, SP // adjust to 16 byte alignment 176 | MOVQ AX, 24(SP) //argc is placed at SP + 24 bytes 177 | MOVQ BX, 32(SP) //argc is placed at SP + 32 bytes 178 | 179 | //Start to initialize g0, runtime·g0 is a global variable, the global variable will be saved in the data area of the process memory space, the following will introduce the code data and code in the elf binary file global variable method 180 | // The stack of g0 is allocated from the process stack memory area, and g0 occupies about 64k in size. 181 | // create istack out of the given (operating system) stack. 182 | // Start to initialize the three fields stackguard0, stackguard1, stack of the g0 object 183 | MOVQ $runtime·g0(SB), DI 184 | LEAQ (-64*1024)(SP), BX 185 | MOVQ BX, g_stackguard0(DI) 186 | MOVQ BX, g_stackguard1(DI) 187 | MOVQ BX, (g_stack+stack_lo)(DI) 188 | MOVQ SP, (g_stack+stack_hi)(DI) 189 | 190 | ... 191 | 192 | // Code for how m0 and g0 connected. 193 | LEAQ runtime·m0+m_tls(SB), DI 194 | CALL runtime·settls(SB) 195 | 196 | // store through it, to make sure it works 197 | get_tls(BX) //tls: Thread Local Storage 198 | MOVQ $0x123, g(BX) 199 | MOVQ runtime·m0+m_tls(SB), AX 200 | CMPQ AX, $0x123 201 | JEQ 2(PC) 202 | CALL runtime·abort(SB) 203 | ok: 204 | // set the per-goroutine and per-mach "registers" 205 | get_tls(BX) 206 | LEAQ runtime·g0(SB), CX 207 | MOVQ CX, g(BX) 208 | LEAQ runtime·m0(SB), AX 209 | 210 | // save m->g0 = g0 211 | MOVQ CX, m_g0(AX) 212 | // save m0 to g0->m 213 | MOVQ AX, g_m(CX) 214 | ``` 215 | 216 | 217 | 218 | ### The keyword go 219 | 220 | Another way to create Goroutine is use **go** 221 | 222 | If we generate the SSA to see how golang implememt go, we can see this, 223 | 224 | ``` 225 | CALL runtime.newproc(SB) 226 | ``` 227 | 228 | In runtime.newproc, a new stack space will be created, the 12 bytes of the stack parameter will be copied to the new stack space and the stack pointer will point to the parameter. 229 | 230 | The thread state at this time is a bit like when the CPU is deprived by the scheduler, the registers PC and SP will be saved in a structure struct G similar to the process control block. f is stored in the entry field of struct G, and the scheduler resumes the running of goroutine later, and the new thread will start to execute from f. The difference from the implementation in C mentioned above is that no auxiliary structure is used, and runtime.newproc is actually a help function. 231 | 232 | so we can say this, 233 | 234 | ``` 235 | go f(args) = runtime.newproc(siz int32, fn *funcval) 236 | ``` 237 | 238 | 239 | 240 | ### Answers 241 | 242 | 1. **Single-core CPU, open two Goroutines, one of which has an infinite loop, what will happen?** 243 | 244 | - In this case, one of the Goroutines has an infinite loop, which means it will never yield the CPU to the other Goroutine. As a result, the other Goroutine may not be able to execute until the infinite loop is interrupted. This can lead to a situation where the infinite loop dominates the CPU and the other Goroutine may effectively be "starved" of CPU time. 245 | 246 | However, the actual behavior will depend on a number of factors, including the specific code being executed, the workload on the CPU, and the scheduling algorithm used by the Go runtime. In general, it's not recommended to have Goroutines with infinite loops, as they can lead to unpredictable behavior and can be difficult to debug. It's usually better to use channels, timeouts, or other mechanisms to control the behavior of Goroutines and ensure that they don't monopolize the CPU. 247 | 248 | 2. **Processes and threads have IDs, why doesn't Goroutine have an ID?** 249 | 250 | - While it is true that goroutines have unique identifiers, the Go runtime chooses not to expose them to the user in order to encourage a more high-level and declarative approach to concurrency. 251 | 252 | 3. **How much is the right number of Goroutines, will it affect GC and scheduling?** 253 | 254 | - Similarly, if too many Goroutines are created and the Go scheduler needs to frequently switch between them, this can increase overhead and negatively impact performance. In general, it is recommended to use a smaller number of long-running Goroutines rather than a larger number of short-lived ones, as this can reduce the overhead of context switching. 255 | 256 | Ultimately, the right number of Goroutines will depend on the specific requirements of the program, and it may require some experimentation and tuning to find the optimal balance between parallelism, memory usage, and scheduling overhead. 257 | 258 | 4. **What are the Goroutine leaks?** 259 | 260 | - Goroutine leaks can be caused by a number of different factors, including: 261 | 262 | 1. Unbounded channel operations: If a Goroutine is waiting on a channel that is never closed or has no buffer, it will block indefinitely, causing a Goroutine leak. 263 | 2. Forgotten timers or tickers: If a Goroutine is created to handle a timer or ticker event but the timer or ticker is not stopped or the Goroutine is not properly cleaned up, it can result in a Goroutine leak. 264 | 3. Failure to use WaitGroups: If a Goroutine is launched as part of a group of Goroutines but is not properly added to a WaitGroup, it can continue to run even after the main program has exited. 265 | 4. Not using the defer keyword: If the Goroutine is created inside a function and the function returns before the Goroutine completes, it will be orphaned and continue to run indefinitely. 266 | 267 | To avoid Goroutine leaks, it is important to ensure that all Goroutines are properly cleaned up after they complete their tasks. This can be achieved by using mechanisms such as channels, timers, WaitGroups, and the defer keyword to properly manage the lifecycle of Goroutines. 268 | 269 | 5. **What can cause a Goroutine to hang?** 270 | 271 | - A Goroutine can hang or become deadlocked if it gets into a state where it is waiting for a resource that is not available or if there is a circular dependency between two or more Goroutines. Here are some common causes of Goroutine hang: 272 | 273 | 1. Channel blocking: A Goroutine that is waiting for data on an unbuffered channel can become deadlocked if no other Goroutine is sending data on that channel, or if the sending Goroutine is also blocked waiting for the receiver to read the data. 274 | 2. Mutex contention: If multiple Goroutines are waiting to acquire the same mutex lock, they can become deadlocked if the lock is not released by the currently holding Goroutine. 275 | 3. Circular dependencies: If two or more Goroutines are waiting for each other to complete before continuing their own execution, they can become deadlocked. 276 | 4. Incorrect use of synchronization primitives: Incorrect use of synchronization primitives like WaitGroups, Mutex, RWMutex, or Cond can cause deadlocks. 277 | 5. Incomplete error handling: If a Goroutine fails to handle an error condition, it can hang or become deadlocked while waiting for a resource that will never be available. 278 | 279 | To avoid Goroutine hang, it is important to use proper synchronization primitives and ensure that all Goroutines handle errors and resources correctly. Also, the use of tools like `go vet`, `go race detector`, and `go test` can help to identify potential deadlocks in a Go program. -------------------------------------------------------------------------------- /problems/how-to-use-channel.md: -------------------------------------------------------------------------------- 1 | # How to use channel 2 | 3 | Don't communicate by sharing memory, but share memory by communicating. 4 | 5 | In many good programming languages, the way multiple threads transfer data is generally shared memory. 6 | 7 | In order to solve thread competition, we need to limit the number of threads that can read and write these variables at the same time. 8 | 9 | However, this is not the same as the design encouraged by the Go language. 10 | 11 | Although we can also use shared memory with mutex for communication in Go language, Go language provides a different concurrency model, namely Communicating sequential processes (CSP). Goroutine and Channel respectively correspond to the entity in CSP and the medium for transmitting information, and Goroutine will transmit data through Channel. 12 | 13 | 14 | 15 | ### Use it 16 | 17 | Here is the most simple way to use channel. 18 | 19 | ```go 20 | package main 21 | import "fmt" 22 | 23 | func main() { 24 | c := make(chan int) 25 | go func() { 26 | // send 1 to channel 27 | c <- 1 28 | }() 29 | // get data from channel 30 | x := <- c 31 | close(c) 32 | fmt.Println(x) 33 | } 34 | 35 | func makechan(t *chantype, size int) *hchan {} //make(chan int)对应 36 | func chansend1(c *hchan, elem unsafe.Pointer) {} //ch <- 1 37 | func chanrecv1(c *hchan, elem unsafe.Pointer) {} //x := <- c 38 | func closechan(c *hchan) {} //close(c) 39 | ``` 40 | 41 | 42 | 43 | The next code is to create an unbuffered **channel**. Once a **goroutine** sends data to the **channel**, the current **goroutine** will be blocked until other **goroutines** consume the data in the **channel** before continuing to run. 44 | 45 | ```go 46 | ch := make(chan int) 47 | ``` 48 | 49 | There is another buffered **channel**, which is created like this: 50 | 51 | ```text 52 | ch := make(chan int, 2) 53 | ``` 54 | 55 | 56 | 57 | The second parameter indicates the capacity of the **channel** to buffer data. As long as the total number of elements in the current **channel** is not greater than the bufferable capacity, the current **goroutine** will not be blocked. 58 | 59 | 60 | 61 | ### **select** 62 | 63 | When we want to communicate with multiple **goroutines**, we will use the **select** to manage the communication data of multiple **channel: 64 | 65 | 66 | 67 | ```go 68 | ch1 := make(chan struct{}) 69 | ch2 := make(chan struct{}) 70 | 71 | // ch1, ch2 发送数据 72 | go sendCh1(ch1) 73 | go sendCh1(ch2) 74 | 75 | // channel 数据接受处理 76 | for { 77 | select { 78 | case <-ch1: 79 | doSomething1() 80 | case <-ch2: 81 | doSomething2() 82 | } 83 | } 84 | ``` 85 | 86 | 87 | 88 | ### runtime.hchan 89 | 90 | Channel in Go language is represented by runtime.hchan structure at runtime. When we create a new Channel in Go language, what we actually create is the following structure: 91 | 92 | ```go 93 | type hchan struct { 94 | qcount uint // total data in the queue 95 | dataqsiz uint // size of the circular queue 96 | buf unsafe.Pointer // points to an array of dataqsiz elements 97 | elemsize uint16 98 | closed uint32 99 | elemtype *_type // element type 100 | sendx uint // send index 101 | recvx uint // receive index 102 | recvq waitq // list of recv waiters, read more in the type waitq 103 | sendq waitq // list of send waiters 104 | 105 | // lock protects all fields in hchan, as well as several 106 | // fields in sudogs blocked on this channel. 107 | // 108 | // Do not change another G's status while holding this lock 109 | // (in particular, do not ready a G), as this can deadlock 110 | // with stack shrinking. 111 | lock mutex 112 | } 113 | 114 | type waitq struct { 115 | first *sudog // read more in the type sudog 116 | last *sudog 117 | } 118 | 119 | // sudog represents a g in a wait list, such as for sending/receiving 120 | // on a channel. 121 | // 122 | // sudog is necessary because the g ↔ synchronization object relation 123 | // is many-to-many. A g can be on many wait lists, so there may be 124 | // many sudogs for one g; and many gs may be waiting on the same 125 | // synchronization object, so there may be many sudogs for one object. 126 | // 127 | // sudogs are allocated from a special pool. Use acquireSudog and 128 | // releaseSudog to allocate and free them. 129 | type sudog struct { 130 | // The following fields are protected by the hchan.lock of the 131 | // channel this sudog is blocking on. shrinkstack depends on 132 | // this for sudogs involved in channel ops. 133 | 134 | g *g 135 | 136 | next *sudog 137 | prev *sudog 138 | elem unsafe.Pointer // data element (may point to stack) 139 | 140 | // The following fields are never accessed concurrently. 141 | // For channels, waitlink is only accessed by g. 142 | // For semaphores, all fields (including the ones above) 143 | // are only accessed when holding a semaRoot lock. 144 | 145 | acquiretime int64 146 | releasetime int64 147 | ticket uint32 148 | 149 | // isSelect indicates g is participating in a select, so 150 | // g.selectDone must be CAS'd to win the wake-up race. 151 | isSelect bool 152 | 153 | // success indicates whether communication over channel c 154 | // succeeded. It is true if the goroutine was awoken because a 155 | // value was delivered over channel c, and false if awoken 156 | // because c was closed. 157 | success bool 158 | 159 | parent *sudog // semaRoot binary tree 160 | waitlink *sudog // g.waiting list or semaRoot 161 | waittail *sudog // semaRoot 162 | c *hchan // channel 163 | } 164 | ``` 165 | 166 | 167 | As we said before, All Channel creation in Go language will use the make keyword. The compiler will convert the make(chan int, 10) expression to a node of type OMAKE, and converts a node of type OMAKE to type OMAKECHAN during the type checking phase: 168 | 169 | ```go 170 | // tcMake typechecks an OMAKE node. 171 | func tcMake(n *ir.CallExpr) ir.Node { 172 | ... 173 | switch t.Kind() { 174 | default: 175 | base.Errorf("cannot make type %v", t) 176 | n.SetType(nil) 177 | return n 178 | 179 | ... 180 | case types.TCHAN: 181 | l = nil 182 | if i < len(args) { 183 | l = args[i] 184 | i++ 185 | l = Expr(l) 186 | l = DefaultLit(l, types.Types[types.TINT]) 187 | if l.Type() == nil { 188 | n.SetType(nil) 189 | return n 190 | } 191 | if !checkmake(t, "buffer", &l) { 192 | n.SetType(nil) 193 | return n 194 | } 195 | } else { 196 | l = ir.NewInt(0) 197 | } 198 | nn = ir.NewMakeExpr(n.Pos(), ir.OMAKECHAN, l, nil) // change to op OMAKECHAN 199 | } 200 | 201 | ... 202 | } 203 | 204 | // walkMakeChan walks an OMAKECHAN node. 205 | func walkMakeChan(n *ir.MakeExpr, init *ir.Nodes) ir.Node { 206 | // When size fits into int, use makechan instead of 207 | // makechan64, which is faster and shorter on 32 bit platforms. 208 | size := n.Len 209 | fnname := "makechan64" 210 | argtype := types.Types[types.TINT64] 211 | 212 | // Type checking guarantees that TIDEAL size is positive and fits in an int. 213 | // The case of size overflow when converting TUINT or TUINTPTR to TINT 214 | // will be handled by the negative range checks in makechan during runtime. 215 | if size.Type().IsKind(types.TIDEAL) || size.Type().Size() <= types.Types[types.TUINT].Size() { 216 | fnname = "makechan" // use makechan function 217 | argtype = types.Types[types.TINT] 218 | } 219 | 220 | return mkcall1(chanfn(fnname, 1, n.Type()), n.Type(), init, reflectdata.MakeChanRType(base.Pos, n), typecheck.Conv(size, argtype)) 221 | } 222 | ``` 223 | 224 | Finally, we can find the code for make a channel, we use hchan. 225 | 226 | ```go 227 | func makechan(t *chantype, size int) *hchan { 228 | elem := t.elem 229 | 230 | ... 231 | 232 | // Hchan does not contain pointers interesting for GC when elements stored in buf do not contain pointers. 233 | // buf points into the same allocation, elemtype is persistent. 234 | // SudoG's are referenced from their owning thread so they can't be collected. 235 | // TODO(dvyukov,rlh): Rethink when collector can move allocated objects. 236 | var c *hchan 237 | switch { 238 | case mem == 0: 239 | // Queue or element size is zero. 240 | c = (*hchan)(mallocgc(hchanSize, nil, true)) 241 | // Race detector uses this location for synchronization. 242 | c.buf = c.raceaddr() 243 | case elem.ptrdata == 0: 244 | // Elements do not contain pointers. 245 | // Allocate hchan and buf in one call. 246 | c = (*hchan)(mallocgc(hchanSize+mem, nil, true)) 247 | c.buf = add(unsafe.Pointer(c), hchanSize) 248 | default: 249 | // Elements contain pointers. 250 | c = new(hchan) 251 | c.buf = mallocgc(mem, elem, true) 252 | } 253 | 254 | c.elemsize = uint16(elem.size) 255 | c.elemtype = elem 256 | c.dataqsiz = uint(size) 257 | ... 258 | } 259 | ``` 260 | 261 | 262 | 263 | ### How channel work? 264 | 265 | #### no-buffer channel 266 | 267 | ##### write before reading 268 | 269 | There are 2 **goroutines** using **channel** communication, in the order of writing first and then reading, the specific process is as follows: 270 | 271 | 272 | 273 | channel 274 | 275 | | buf | 276 | 277 | | lock | 278 | 279 | | ... | 280 | 281 | | sendq | <=== G1 282 | 283 | | recvq | 284 | 285 | 286 | 287 | It can be seen that because the **channel** is unbuffered, G1 is temporarily suspended in the sendq queue, and then G1 calls gopark to sleep. 288 | 289 | Then, another **goroutine** comes to the **channel** to read data: 290 | 291 | 292 | 293 | channel 294 | 295 | | buf | 296 | 297 | | lock | 298 | 299 | | ... | goready 300 | 301 | | sendq | ------ G1 <=======> G2 302 | 303 | | recvq | data 304 | 305 | 306 | 307 | At this time, G2 finds that there are **goroutines** in the sendq waiting queue, so it directly copies the data from G1, and sets the goready function for G1, so that when the next scheduling occurs, G1 can continue to run and will be removed from the waiting queue 308 | 309 | ##### read before writing 310 | 311 | G1 want to read but get nothing, so this **goroutine** should sleep in the recvq 312 | 313 | 314 | 315 | channel 316 | 317 | | buf | 318 | 319 | | lock | 320 | 321 | | ... | 322 | 323 | | sendq | 324 | 325 | | recvq | <=== G1 326 | 327 | 328 | 329 | When G2 is writing data, it finds that there are **goroutines** in the recvq queue, so it directly sends the data to G1. At the same time, set the G1 goready function and wait for the next scheduled operation. 330 | 331 | 332 | 333 | channel 334 | 335 | | buf | 336 | 337 | | lock | 338 | 339 | | ... | 340 | 341 | | sendq | goready and ata 342 | 343 | | recvq | ------ G1 <======= G2 344 | 345 | 346 | 347 | #### channel with buffer 348 | 349 | ##### write before reading 350 | 351 | 352 | 353 | channel data 354 | 355 | | buf | <======= G1 (if G1 can write data into buf,) 356 | 357 | | lock | 358 | 359 | | ... | 360 | 361 | | sendq | <======= G1 (if G1 can not write data into buf, it means buf is full of data,,) 362 | 363 | | recvq | 364 | 365 | 366 | 367 | When G2 wants to read data, it will read from the buffer data area first, and after reading, it will check the sendq queue. If the **goroutine** has a waiting queue, it will add the data on it to the buffer data area, and Also set the goready function on it. 368 | 369 | 370 | 371 | channel 1. read data 372 | 373 | | buf | =========> G2 374 | 375 | | lock | 376 | 377 | | ... | 2. scan sendq and push G1 to buf 378 | 379 | | sendq | ------ G1 ========> to buf 380 | 381 | | recvq | 382 | 383 | 384 | 385 | #### channel read first, then write 386 | 387 | This situation is the same process as unbuffered first read and then write, and will not be repeated here. 388 | -------------------------------------------------------------------------------- /problems/map-in-go.md: -------------------------------------------------------------------------------- 1 | # Map In Go 2 | 3 | Go build a much more simple Map, simple but effective. 4 | 5 | 1. How map works? 6 | 2. Why is Go map not thread-safe? 7 | 2. Why is the load factor of Go map 6.5? 8 | 3. Why does the key of the map have to be of a comparable type? 9 | 5. How can we copy a map in Go? 10 | 6. Can we override the hashcode()? 11 | 12 | 13 | 14 | The code for map is in src/runtime/map.go. The realization principle is a variant of the zipper method. To be honest, there is nothing new, so we simply analyze it from the four behaviors of init, get, set, and expansion. 15 | 16 | 17 | 18 | ```go 19 | // A header for a Go map. 20 | type hmap struct { 21 | // Note: the format of the hmap is also encoded in cmd/compile/internal/gc/reflect.go. 22 | // Make sure this stays in sync with the compiler's definition. 23 | count int // # live cells == size of map. Must be first (used by len() builtin) 24 | flags uint8 25 | B uint8 // log_2 of # of buckets (can hold up to loadFactor * 2^B items) 26 | noverflow uint16 // approximate number of overflow buckets; see incrnoverflow for details 27 | hash0 uint32 // hash seed 28 | 29 | buckets unsafe.Pointer // array of 2^B Buckets. may be nil if count==0. 30 | oldbuckets unsafe.Pointer // previous bucket array of half the size, non-nil only when growing 31 | nevacuate uintptr // progress counter for evacuation (buckets less than this have been evacuated) 32 | 33 | extra *mapextra // optional fields 34 | } 35 | 36 | type bmap struct { 37 | topbits [8]uint8 38 | keys [8]keytype 39 | values [8]valuetype 40 | pad uintptr 41 | overflow uintptr 42 | } 43 | ``` 44 | 45 | 46 | 47 | ### Make a map 48 | 49 | 50 | 51 | ```go 52 | // makemap implements Go map creation for make(map[k]v, hint). 53 | // If the compiler has determined that the map or the first bucket 54 | // can be created on the stack, h and/or bucket may be non-nil. 55 | // If h != nil, the map can be created directly in h. 56 | // If h.buckets != nil, bucket pointed to can be used as the first bucket. 57 | 58 | func makemap(t *maptype, hint int, h *hmap) *hmap { 59 | mem, overflow := math.MulUintptr(uintptr(hint), t.bucket.size) 60 | if overflow || mem > maxAlloc { 61 | hint = 0 62 | } 63 | 64 | // initialize Hmap 65 | if h == nil { 66 | h = new(hmap) 67 | } 68 | h.hash0 = fastrand() 69 | 70 | // Find the size parameter B which will hold the requested # of elements. 71 | // For hint < 0 overLoadFactor returns false since hint < bucketCnt. 72 | 73 | B := uint8(0) 74 | for overLoadFactor(hint, B) { 75 | B++ 76 | } 77 | h.B = B 78 | 79 | // allocate initial hash table 80 | // if B == 0, the buckets field is allocated lazily later (in mapassign) 81 | // If hint is large zeroing this memory could take a while. 82 | 83 | if h.B != 0 { 84 | var nextOverflow *bmap 85 | h.buckets, nextOverflow = makeBucketArray(t, h.B, nil) // memory for Buckets 86 | if nextOverflow != nil { 87 | h.extra = new(mapextra) 88 | h.extra.nextOverflow = nextOverflow 89 | } 90 | } 91 | 92 | return h 93 | } 94 | ``` 95 | 96 | 97 | 98 | ### Get 99 | 100 | 101 | 102 | ![map-get](../pictures/map-get.png) 103 | 104 | Steps are as follows: 105 | 106 | - Taking the 64-bit operating system as an example, the original key is mapped into a 64-bit binary through the Hash function. 107 | - The last B bit corresponds to the position of the **bmap**, and the corresponding **bmap** is found from **[]bmap**. 108 | - The first 8 digits correspond to the tophash of the key, and the search starts from the **bmap** located in step 2. First, it will compare whether the top-level tophash of the **bmap** is the same as that of the original key. If they are not the same, skip the comparison to the next one; if they are the same, further compare whether the key is the same. 109 | - If the current **bmap** is compared and the target key is not matched, and **overflow** is not empty, then continue to compare from the next **bmap** pointed to by **overflow**. 110 | 111 | 112 | 113 | ### Set 114 | 115 | Without considering the **expansion**, the basic logic of map **get** and map **set** is consistent. 116 | 117 | 1. First, follow the map get method to locate the bmap through the last B bit, and quickly compare the tophash through the first 8 bits. 118 | 2. When the key does not exist in the map, the first free tophash in the **bmap** will be recorded and the key will be inserted. 119 | 3. When the key exists in the map, the value of the key will be updated. 120 | 121 | 122 | 123 | 124 | 125 | ### Expansion 126 | 127 | 128 | 129 | There are two ways for map to trigger expansion: 130 | 131 | 1. If the load factor (the ratio of length to capacity) of the map exceeds the threshold, the map is considered unable to bear more keys at this time, and needs to be expanded by 2 times. 132 | 2. There is a situation in the map that some bmaps contain too many overflows. At this time, the map will think that the local bmaps can be densely arranged by tophash, so that the number of overflows is less. 133 | 134 | 135 | 136 | ![load](../pictures/load.png) 137 | 138 | ![overflow](../pictures/overflow.png) 139 | 140 | 141 | 142 | ### One more thing: 143 | 144 | If you check the source code of Go: hashmap_fast.go#L118, you will see that the **hashWriting** flag will be checked when reading. If there is this flag, a concurrency error will be reported. 145 | 146 | The error message is: **fatal error: concurrent map read and map write.** 147 | 148 | This flag is set when writing: hashmap.go#L542 149 | 150 | ```go 151 | h.flags |= hashWriting 152 | This mark will be canceled after hashmap.go#L628 is set. 153 | ``` 154 | 155 | Of course, there are several concurrent read and write checks in the code, such as checking whether there is concurrent writing when writing, similar writing when deleting keys, concurrent reading and writing when traversing, etc. 156 | 157 | I think this is because running concurrent operations on **map** will produce many unbelievable situations. Go has learned this lesson and decided to prohibit concurrent operations on map. 158 | 159 | In the Java language, concurrent reads of hashmaps can generate infinite loops in some versions. -------------------------------------------------------------------------------- /problems/more-sync.md: -------------------------------------------------------------------------------- 1 | fixme: not start -------------------------------------------------------------------------------- /problems/mutex.md: -------------------------------------------------------------------------------- 1 | # Mutex 2 | 3 | 4 | 5 | If you want to know the **mutex**, you should konw **CAS** first. 6 | 7 | ### CAS 8 | 9 | **CAS** means compare and swap. 10 | Take the atomic.CompareAndSwapInt32 method in the go standard library as an example 11 | 12 | ```go 13 | // CompareAndSwapInt32 executes the compare-and-swap operation for an int32 value. 14 | func CompareAndSwapInt32(addr *int32, old, new int32) (swapped bool) 15 | ``` 16 | 17 | It receives three input parameters, a pointer addr, an old value old, and a new value new 18 | What this method does is divided into three steps: 19 | 20 | 1. First take the value pointed to by addr 21 | 2. Then compare with old, if not equal, return false, the operation failed. 22 | 3. Finally, change the value pointed to by addr to new 23 | 24 | The key point is that the atomicity of these three steps is guaranteed at the machine level, that is, the execution process will not be interrupted by other coroutines. Ordinary code may be interrupted suddenly during execution, and the cpu turns to execute other coroutines. 25 | In addition to **CAS**, go's atomic package also provides other atomic methods, which can solve data competition problems in some concurrent scenarios. 26 | 27 | Let us build a mutex v0! 28 | 29 | 30 | 31 | ### Mutex V0 32 | 33 | 34 | 35 | ```go 36 | package v0 37 | 38 | import "sync/atomic" 39 | 40 | type Mutex struct { 41 | key int32 42 | } 43 | 44 | func (m *Mutex) Lock(){ 45 | for { 46 | if atomic.CompareAndSwapInt32(&m.key,0,1) { 47 | return 48 | } 49 | } 50 | } 51 | 52 | func (m *Mutex) Unlock() { 53 | atomic.CompareAndSwapInt32(&m.key,1,0) 54 | } 55 | ``` 56 | 57 | 58 | 59 | It is a 100% **mutex**. **Mutex** has a key to record the lock status. How to lock? 60 | 61 | Suppose there are two goroutines g1 and g2, competing for the same lock by the infinite-for loop with CAS operations. if g1 is a little fast, 62 | 63 | 1. g1 executes first and gets the lock. When operating critical resources, g2 comes to get the lock 64 | 2. At this time, the Lock method of g2 will try to execute **CAS**, but in the second step of **CAS**, it will fail because g1 has changed the key to 1. g2 will continue to try **CAS**, so g2 blocks here. 65 | 3. g1 has completed the operation of critical resources, called the Unlock method, and changed the key value back to 0 through **CAS**. 66 | 4. Only then the Lock method of g2 can succeed. 67 | 68 | - 69 | 70 | But, g2 has nothing to do but stay in th CPU, it is awful. 71 | 72 | 73 | 74 | ### Mutex V1 75 | 76 | ```go 77 | type Mutex struct { 78 | key int32 //0 means the lock is not held, 1 means the lock is already held, n means the lock is held and there are n-1 lock waiters 79 | sema int32 /*Represents a semaphore 80 | It is used for sleep and wake-up. In the sample code, the following pseudo-code is used to express the function, but there are corresponding methods in the go runtime. 81 | semacquire(sema) 82 | Hibernate the current goroutine 83 | semrelease(sema) 84 | Wake up to take the first coroutine from the coroutine queue that sleeps using the sema semaphore 85 | */ 86 | } 87 | 88 | func xadd(val *int32, delta int32) (new int32) { // +-1 89 | for { 90 | v := *val 91 | // val address 92 | // v old value 93 | // v+delta new value 94 | if atomic.CompareAndSwapInt32(val,v,v+delta) { 95 | return v + delta 96 | } 97 | } 98 | panic("unreached") 99 | } 100 | 101 | func (m *Mutex) Lock() { 102 | // CAS + 1 103 | if xadd(&m.key,1) == 1 { 104 | return 105 | } 106 | // stop goroutine 107 | semacquire(&m.sema) 108 | return 109 | } 110 | 111 | func (m *Mutex) Unlock() { 112 | 113 | if xadd(&m.key,-1) == 0 { 114 | return 115 | } 116 | // wake up goroutine 117 | semrelease(&m.sema) 118 | return 119 | } 120 | ``` 121 | 122 | The Lock method and the Unlock method do not replace the key between 0 and 1, but use ±1 operation, so that not only the lock holding status can be recorded, but also the number of queued goroutines can be recorded 123 | 124 | Still have some problems, 125 | 126 | - If we have lots of **gorroutines** try to get mutes, only one goroutine can get it, others will sleep. 127 | - Sleep and wake up, means goroutine change context, it is just fine. 128 | - Can we do this? if one goroutine Unlock, find the active goroutine and just give the lock? More fast. 129 | 130 | 131 | 132 | ### Mutex V2 133 | 134 | 135 | 136 | ```go 137 | type Mutex struct { 138 | state int32 139 | sema uint32 // same as V1 140 | } 141 | 142 | 143 | const ( 144 | mutexLocked = 1 << iota // mutex is locked // 1 145 | mutexWoken //2 146 | mutexWaiterShift = iota //2 147 | ) 148 | 149 | 150 | ``` 151 | 152 | 153 | 154 | The 32bit state is a composite field, which can be divided into three parts: mutexWaiters | mutexWoken | mutexLocked 155 | 156 | 1. mutexWaiters indicates the number of lock waiters 157 | 2. mutexWoken indicates whether the lock has a wake-up goroutine 158 | 3. mutexLocked indicates whether the lock is held 159 | 160 | Its calculation method is bit operation, which can be understood as combining three fields into one field, reducing storage space. 161 | 162 | 163 | 164 | ### Lock 165 | 166 | 167 | 168 | ```go 169 | func (m *Mutex) Lock() { 170 | // Fast path: lucky case，just get lock 171 | if atomic.CompareAndSwapInt32(&m.state, 0, mutexLocked) { 172 | return 173 | } 174 | 175 | awoke := false 176 | for { 177 | old := m.state 178 | new := old | mutexLocked // add lock 179 | if old&mutexLocked != 0 { 180 | new = old + 1<>mutexWaiterShift != 0 && 226 | atomic.CompareAndSwapInt32(&m.state, old, old|mutexWoken) { 227 | awoke = true 228 | } 229 | runtime_doSpin() 230 | iter++ 231 | continue / 232 | } 233 | new = old + 1<>mutexWaiterShift != 0 && 293 | atomic.CompareAndSwapInt32(&m.state, old, old|mutexWoken) { 294 | awoke = true 295 | } 296 | runtime_doSpin() 297 | iter++ 298 | old = m.state 299 | continue 300 | } 301 | new := old 302 | if old&mutexStarving == 0 { 303 | new |= mutexLocked 304 | } 305 | if old&(mutexLocked|mutexStarving) != 0 { 306 | new += 1 << mutexWaiterShift 307 | } 308 | if starving && old&mutexLocked != 0 { 309 | new |= mutexStarving // set mutex Starving 310 | } 311 | if awoke { 312 | if new&mutexWoken == 0 { 313 | throw("sync: inconsistent mutex state") 314 | } 315 | new &^= mutexWoken 316 | } 317 | 318 | if atomic.CompareAndSwapInt32(&m.state, old, new) { 319 | 320 | if old&(mutexLocked|mutexStarving) == 0 { 321 | break // locked the mutex with CAS 322 | } 323 | 324 | 325 | // remove to the head of queue 326 | queueLifo := waitStartTime != 0 327 | if waitStartTime == 0 { 328 | waitStartTime = runtime_nanotime() 329 | } 330 | 331 | runtime_SemacquireMutex(&m.sema, queueLifo, 1) 332 | 333 | starving = starving || runtime_nanotime()-waitStartTime > starvationThresholdNs 334 | old = m.state 335 | // in straving 336 | if old&mutexStarving != 0 { 337 | if old&(mutexLocked|mutexWoken) != 0 || old>>mutexWaiterShift == 0 { 338 | throw("sync: inconsistent mutex state") 339 | } 340 | 341 | delta := int32(mutexLocked - 1<>mutexWaiterShift == 1 { 343 | delta -= mutexStarving 344 | } 345 | atomic.AddInt32(&m.state, delta) 346 | break 347 | } 348 | awoke = true 349 | iter = 0 350 | } else { 351 | old = m.state 352 | } 353 | } 354 | } 355 | 356 | ``` -------------------------------------------------------------------------------- /problems/new-one-slice.md: -------------------------------------------------------------------------------- 1 | # Slice 2 | 3 | 4 | 5 | ```go 6 | type slice struct { 7 | array unsafe.Pointer 8 | len int 9 | cap int 10 | } 11 | ``` 12 | 13 | The **array** of **slice** is a **pointer** to a piece of contiguous memory. 14 | 15 | Look at the growslice function again, this is the function called by append. 16 | 17 | ```go 18 | func growslice(et *_type, old slice, cap int) slice { 19 | if raceenabled { 20 | callerpc := getcallerpc(unsafe.Pointer(&et)) 21 | racereadrangepc(old.array, uintptr(old.len*int(et.size)), callerpc, funcPC(growslice)) 22 | } 23 | if msanenabled { 24 | msanread(old.array, uintptr(old.len*int(et.size))) 25 | } 26 | 27 | if et.size == 0 { 28 | if cap < old.cap { 29 | panic(errorString("growslice: cap out of range")) 30 | } 31 | // append should not create a slice with nil pointer but non-zero len. 32 | // We assume that append doesn't need to preserve old.array in this case. 33 | return slice{unsafe.Pointer(&zerobase), old.len, cap} 34 | } 35 | 36 | newcap := old.cap 37 | doublecap := newcap + newcap 38 | if cap > doublecap { 39 | newcap = cap 40 | } else { 41 | if old.len < 1024 { 42 | newcap = doublecap 43 | } else { 44 | for newcap < cap { 45 | newcap += newcap / 4 46 | } 47 | } 48 | } 49 | 50 | var lenmem, newlenmem, capmem uintptr 51 | const ptrSize = unsafe.Sizeof((*byte)(nil)) 52 | switch et.size { 53 | case 1: 54 | lenmem = uintptr(old.len) 55 | newlenmem = uintptr(cap) 56 | capmem = roundupsize(uintptr(newcap)) 57 | newcap = int(capmem) 58 | case ptrSize: 59 | lenmem = uintptr(old.len) * ptrSize 60 | newlenmem = uintptr(cap) * ptrSize 61 | capmem = roundupsize(uintptr(newcap) * ptrSize) 62 | newcap = int(capmem / ptrSize) 63 | default: 64 | lenmem = uintptr(old.len) * et.size 65 | newlenmem = uintptr(cap) * et.size 66 | capmem = roundupsize(uintptr(newcap) * et.size) 67 | newcap = int(capmem / et.size) 68 | } 69 | 70 | if cap < old.cap || uintptr(newcap) > maxSliceCap(et.size) { 71 | panic(errorString("growslice: cap out of range")) 72 | } 73 | 74 | var p unsafe.Pointer 75 | if et.kind&kindNoPointers != 0 { 76 | p = mallocgc(capmem, nil, false) 77 | memmove(p, old.array, lenmem) 78 | // The append() that calls growslice is going to overwrite from old.len to cap (which will be the new length). 79 | // Only clear the part that will not be overwritten. 80 | memclrNoHeapPointers(add(p, newlenmem), capmem-newlenmem) 81 | } else { 82 | // Note: can't use rawmem (which avoids zeroing of memory), because then GC can scan uninitialized memory. 83 | p = mallocgc(capmem, et, true) 84 | if !writeBarrier.enabled { 85 | memmove(p, old.array, lenmem) 86 | } else { 87 | for i := uintptr(0); i < lenmem; i += et.size { 88 | typedmemmove(et, add(p, i), add(old.array, i)) 89 | } 90 | } 91 | } 92 | 93 | return slice{p, old.len, newcap} 94 | } 95 | ``` 96 | 97 | 98 | 99 | **Cap** is the number of elements that can be accommodated in the pre-allocated memory of the current **slice**, if new elements are continuously added to the **slice**. If the **cap** of the current **slice** is exceeded, new memory needs to be allocated, and all elements of the current **slice** are copied to the new memory block. -------------------------------------------------------------------------------- /problems/package.md: -------------------------------------------------------------------------------- 1 | # Packages 2 | 3 | Go language uses **packages** to organize source code and implement namespace management. Any Go language program must belong to a **package**, that is, package must be written at the beginning of each go program. 4 | 5 | Go language **packages** generally meet the following three conditions: 6 | 7 | 1. All go files at the same level in the same directory should belong to a **package**; 8 | 2. The name of the **package** can be different from the name of the directory, but the same name is recommended; 9 | 3. A Go language program has one and only one main function, which is the entry function of the Go language program and must belong to the main **package**. If there is no or more than one, an error will be reported when the Go language program is compiled; 10 | 11 | 12 | 13 | 14 | 15 | ### Import 16 | 17 | When organizing Go code structure and referencing packages, a concept called **GOPATH** will inevitably be involved, so what exactly is it? 18 | 19 | **GOPATH** is an environment variable used by the GO language. It uses an absolute path to provide the working directory of the project. It is suitable for processing complex projects composed of a large number of Go language source codes and multiple packages. In actual use, you can first check the current GOPATH value through the command **go env**, and then decide whether it needs to be reset. 20 | 21 | 22 | 23 | ``` 24 | GOARCH="amd64" 25 | GOBIN="" 26 | GOCACHE="/Users/zhen/Library/Caches/go-build" 27 | GOENV="/Users/zhen/Library/Application Support/go/env" 28 | GOEXE="" 29 | GOEXPERIMENT="" 30 | GOFLAGS="" 31 | GOHOSTARCH="amd64" 32 | GOHOSTOS="darwin" 33 | GOINSECURE="" 34 | GOMODCACHE="/Users/zhen/go/pkg/mod" 35 | GONOPROXY="" 36 | GONOSUMDB="" 37 | GOOS="darwin" 38 | GOPATH="/Users/zhen/go" 39 | ``` 40 | 41 | 42 | 43 | #### Import Format 44 | 45 | There are four common import formats for Go packages. The following uses the fmt package as an example to illustrate: 46 | 47 | 1. The standard quotes import fmt 48 | 49 | 2. Set the alias reference import format_go fmt to omit the reference import . fmt. This kind of reference is equivalent to merging the namespace of the package fmt into the namespace of the current program, so it can be directly referenced without adding the prefix fmt. 50 | 3. Execute only the package initialization function import_fmt 51 | 52 | In addition, when a Go program needs to import multiple packages, you can use single-line import or multi-line import. The two import forms are listed below: 53 | 54 | single line import 55 | 56 | ```go 57 | import "package1" 58 | import "package2` 59 | ``` 60 | 61 | multiline import 62 | 63 | ```go 64 | import ( 65 | "package01" 66 | "package02" 67 | ) 68 | ``` 69 | 70 | In addition, the imported package must be used, otherwise an error will be reported when the program is compiled. 71 | 72 | 73 | 74 | #### Go package reference initialization process 75 | 76 | Assuming that the order of importing packages of a go language program is: main-> package A -> package B -> package C, and packages A, B, and C all implement the init() function, then the order of the entire initialization process is: C .init -> B.init -> A.init -> main -> package A -> package B -> package C. 77 | 78 | #### Export of identifiers in packages 79 | 80 | If a package wants to refer to another package's identifier (such as structure, variable, constant, function, etc.), it must be exported first. The specific method is to ensure that the first letter is capitalized (the first letter is lowercase) when defining these identifiers. identifiers can only be referenced within the package). In addition, in the exported structure or interface, only the fields and methods with **capital letters** can be accessed outside the package. -------------------------------------------------------------------------------- /problems/panic-and-recover.md: -------------------------------------------------------------------------------- 1 | # Panic and Recover 2 | 3 | The **panic** keyword in the Go language is mainly used to actively throw exceptions, similar to the **throw** keyword in languages such as java. 4 | 5 | Panic can change the control flow of the program. After calling panic, it will immediately stop executing the remaining code of the current function, and recursively execute the caller's **defer** in the current Goroutine; 6 | 7 | The **recover** keyword in the Go language is mainly used to **catch** exceptions and return the program to a normal state, similar to **try ... catch** in languages such as java. **recover** can abort the program crash caused by **panic**. It is a function that can only work in **defer**, and calling it in other scopes will not work; 8 | 9 | **recover** can only be used in defer. **This** has been clearly written in the comments of the standard library. We can take a look: 10 | 11 | ```go 12 | // The recover built-in function allows a program to manage behavior of a 13 | // panicking goroutine. Executing a call to recover inside a deferred 14 | // function (but not any function called by it) stops the panicking sequence 15 | // by restoring normal execution and retrieves the error value passed to the 16 | // call of panic. If recover is called outside the deferred function it will 17 | // not stop a panicking sequence. In this case, or when the goroutine is not 18 | // panicking, or if the argument supplied to panic was nil, recover returns 19 | // nil. Thus the return value from recover reports whether the goroutine is 20 | // panicking. 21 | func recover() interface{} 22 | ``` 23 | 24 | To summarize the process of program crash and recovery: 25 | 26 | 1. The compiler will be responsible for doing the work of converting keywords; 27 | 28 | 1. Convert **panic** and **recover** to **runtime.gopanic** and **runtime.gorecover** respectively; 29 | 2. Convert **defer** to **runtime.deferproc** function; 30 | 3. Call the **runtime.deferreturn** function at the end of the function that calls **defer**; 31 | 32 | 2. When the r**untime.gopanic** method is encountered during the running process, the **runtime._defer** structure will be sequentially taken out from the **Goroutine** linked list and executed; 33 | 34 | 3. **If runtime.gorecover is encountered when calling the delayed execution function, it will mark _panic.recovered as true and return the parameter of panic;** (This is why you should add recover in defer) 35 | 36 | 1. After this call ends, **runtime.gopanic** will take out the program counter **pc** and stack pointer **sp** from the **runtime._defer** structure and call the **runtime.recovery** function to restore the program; 37 | 2. **runtime.recovery** will jump back to **runtime.deferproc** according to the incoming **pc** and **sp**; 38 | 3. The code automatically generated by the compiler will find that the return value of **runtime.deferproc** is not 0, then it will jump back to **runtime.deferreturn** and return to the normal execution flow; 39 | 40 | 4. If **runtime.gorecover** is not encountered, it will traverse al**l runtime._defer** in turn, and finally call **runtime.fatalpanic** to abort the program, print the parameters of panic and return error code 2; 41 | 42 | 43 | 44 | ### Panic 45 | 46 | Here we first introduce the principle of analyzing the **panic** function to terminate the program. The compiler will convert the keyword **panic** into **runtime.gopanic**, and the execution of this function includes the following steps: 47 | 48 | 1. Create a new **runtime._panic** and add it to the front of the **Goroutine's** **_panic** list; 49 | 2. In the loop, continuously obtain **runtime._defer** from the linked list in the **_defer** of the current **Goroutine** and call **runtime.reflectcall** to run the delayed call function; 50 | 3. Call **runtime.fatalpanic** to abort the entire program; 51 | 52 | 53 | 54 | ```go 55 | func gopanic(e interface{}) { 56 | gp := getg() 57 | ... 58 | var p _panic 59 | p.arg = e 60 | p.link = gp._panic 61 | gp._panic = (*_panic)(noescape(unsafe.Pointer(&p))) 62 | 63 | for { 64 | d := gp._defer 65 | if d == nil { 66 | break 67 | } 68 | 69 | d._panic = (*_panic)(noescape(unsafe.Pointer(&p))) 70 | 71 | reflectcall(nil, unsafe.Pointer(d.fn), deferArgs(d), uint32(d.siz), uint32(d.siz)) 72 | 73 | d._panic = nil 74 | d.fn = nil 75 | gp._defer = d.link 76 | 77 | freedefer(d) 78 | if p.recovered { 79 | ... 80 | } 81 | } 82 | 83 | fatalpanic(gp._panic) 84 | *(*int)(nil) = 0 85 | } 86 | ``` 87 | 88 | 89 | 90 | ### Recover 91 | 92 | ```go 93 | func gorecover(argp uintptr) interface{} { 94 | gp := getg() 95 | p := gp._panic 96 | if p != nil && !p.recovered && argp == uintptr(p.argp) { 97 | p.recovered = true 98 | return p.arg 99 | } 100 | return nil 101 | } 102 | ``` 103 | 104 | -------------------------------------------------------------------------------- /problems/return-multiple-values.md: -------------------------------------------------------------------------------- 1 | # Return Multiple Values 2 | 3 | 4 | 5 | The golang function call process realizes parameter passing and return value through **fp**+**offset**, unlike C/C++, which realizes parameter passing and return value through **registers**. 6 | 7 | 8 | 9 | This means that the return list of go can be a continuous area with an indefinite length, so it can naturally return different numbers of values. -------------------------------------------------------------------------------- /problems/slice-in-go.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 1. ​ 为什么 slice 是非线程安全的？ 4 | 2. ​ Slice 的扩容机制， range, slice 5 | 3. ​ slice 和 array 的区别 6 | 4. ​ How can we copy a slice -------------------------------------------------------------------------------- /problems/string.md: -------------------------------------------------------------------------------- 1 | # String literals, Byte and Rune 2 | 3 | String is immutable. 4 | 5 | Like the string in the C++ language, it has its own memory space, and modifying the string is supported. But in the implementation of Go, string does not contain memory space, only a memory pointer. The advantage of this is that string becomes very lightweight and can be passed easily without worrying about memory copying. 6 | 7 | ```go 8 | type stringStruct struct { 9 | str unsafe.Pointer // only a pointer 10 | len int 11 | } 12 | ``` 13 | 14 | 15 | 16 | ### [] byte to string 17 | 18 | A byte slice can be easily converted to a string, as follows: 19 | 20 | ```go 21 | func GetStringBySlice(s []byte) string { 22 | return string(s) 23 | } 24 | ``` 25 | 26 | 27 | 28 | ### rune 29 | 30 | The rune type is a special numeric type of the Go language. In the builtin/builtin.go file, its definition: 31 | 32 | ``` 33 | type rune = int32; 34 | ``` 35 | 36 | The official explanation for it is: rune is an alias of type int32, which is equivalent to it in all respects, and is used to distinguish character values from integer values. Defined with single quotes, returns a Unicode code point encoded in UTF-8. The Go language handles Chinese through runes and supports international multilingualism. 37 | 38 | Strings are composed of characters, the bottom layer of characters is composed of bytes, and the bottom layer representation of a string is a sequence of bytes. In the Go language, characters can be divided into two types: for English characters that occupy 1 byte, you can use byte (or unit8); for other characters that occupy 1 to 4 bytes, you can use rune (or int32), such as Chinese, special symbols, etc. 39 | 40 | 41 | 42 | The Go language divides characters into two types: 43 | 44 | 1. byte 45 | 2. rune 46 | 47 | byte is an alias of type unit8, which is used to store ASCII characters occupying 1 byte, such as English characters, and returns the original byte of the character. rune is an alias of type int32, which is used to store multi-byte characters, such as **Chinese characters** occupying 3 bytes, and returns the Unicode code point value of the character. -------------------------------------------------------------------------------- /problems/struct-interface-pointer.md: -------------------------------------------------------------------------------- 1 | # Struct, Interface, Pointer 2 | 3 | We should talk about this 4 thing in this doc since go is an object-oriented language. 4 | 5 | 6 | 7 | ### Pointer 8 | 9 | Pointers are an addressing mechanism supported in Go. Pointers have two operators called addressing (&) and taking the value of the variable pointed to by the pointer (*). 10 | 11 | ```go 12 | package main 13 | import ( 14 | "fmt" 15 | ) 16 | func main() { 17 | a := 42 18 | //prints out the value of a 19 | fmt.Println("value of a: ", a) 20 | 21 | //prints out the memory address that pointing to "a" variable 22 | fmt.Println("memory address of a: ", &a) 23 | 24 | } 25 | ``` 26 | 27 | output: 28 | 29 | ``` 30 | value of a: 42 31 | memory address of a: 0xc000014080 32 | ``` 33 | 34 | 35 | 36 | Using pointers, a variable can be addressed to another variable. For example, the b variable can address another variable (in this code, the variable is called a). 37 | 38 | ```go 39 | package main 40 | import ( 41 | "fmt" 42 | ) 43 | func main() { 44 | a := 42 45 | //prints out the value of a 46 | fmt.Println("value of a: ", a) 47 | 48 | //prints out the memory address that pointing to "a" variable 49 | fmt.Println("memory address of a: ", &a) 50 | 51 | b := &a 52 | //prints out the memory address of "b" variable 53 | fmt.Println("value of b:", b) 54 | 55 | //to prints out the value of b, use * (dereference) operator 56 | fmt.Println("value of b using * operator: ", *b) 57 | 58 | } 59 | ``` 60 | 61 | output 62 | 63 | ``` 64 | value of a: 42 65 | memory address of a: 0xc000014080 66 | value of b: 0xc000014080 67 | value of b using * operator: 42 68 | ``` 69 | 70 | 71 | 72 | ### Structure 73 | 74 | Classes are an important part of OOP. In Go, there is a similar concept called struct. 75 | 76 | ```go 77 | // struct declaration 78 | type structName struct { 79 | //field name type 80 | member int 81 | member2 string 82 | member3 [] string 83 | } 84 | ``` 85 | 86 | Example usage of struct. Illustrates a person with the ability to speak. 87 | 88 | ```go 89 | //declare a struct called person 90 | type person struct { 91 | name string 92 | age int 93 | } 94 | 95 | //declare a method say() with type person as receiver 96 | func (p person) say() { 97 | fmt.Println("Hello, my name is: ", p.name) 98 | } 99 | To use the defined struct, assign it to a variable, as follows: 100 | 101 | package main 102 | 103 | import "fmt" 104 | 105 | //declare a struct called person 106 | type person struct { 107 | name string 108 | age int 109 | } 110 | 111 | //declare a method say() with type person as receiver 112 | func (p person) say() { 113 | fmt.Println("Hello, my name is: ", p.name) 114 | } 115 | 116 | func main() { 117 | //Instantiate person and assign value 118 | p1 := person{name: "Enki Gilbert", age: 42} 119 | //call a method say() 120 | p1. say() 121 | } 122 | 123 | // output 124 | // Hello, my name is: Enki Gilbert 125 | Go also provides anonymous struct. 126 | 127 | s1 := struct{ 128 | //declare some fields 129 | field1 int 130 | field2[]string 131 | }{ 132 | //instantiate directly 133 | field1: 12, 134 | field2: []string{"hi","mate"}, 135 | } 136 | ``` 137 | 138 | 139 | 140 | ### Interface 141 | 142 | Interface is another powerful concept in programming. Interface is similar to struct, but only contains some abstract methods. In Go, Interface defines an abstraction for common behavior. 143 | 144 | ```go 145 | package main 146 | import ( 147 | "fmt" 148 | ) 149 | 150 | //declare a rectangle struct 151 | type rectangle struct { 152 | length int 153 | width int 154 | } 155 | 156 | //declare an interface with area() as a member 157 | type shape interface { 158 | area() int 159 | } 160 | 161 | //declare a method area() 162 | //the rectangle struct implements area() method in shape interface 163 | func (r rectangle) area() int { 164 | return r.length * r.width 165 | } 166 | 167 | //declare a method with type shape as a parameter 168 | func info(s shape) { 169 | fmt. Println("the area: ", s. area()) 170 | } 171 | 172 | func main() { 173 | r1 := rectangle{12, 12} 174 | //r1 is a rectangle type. rectangle implements all methods in shape interface. 175 | info(r1) 176 | } 177 | 178 | // output 179 | // the area: 144 180 | ``` 181 | 182 | Following the example, we declare a struct for rectangles and an interface for shapes. Rectangle implements area() in the shape interface. info() takes a shape type as an argument. The struct that implements all the methods in the shape interface can be used as the parameter of info(). 183 | 184 | If there is another struct called square. The info() method is also available, since squares also implement all the methods in the shape interface. 185 | 186 | ```go 187 | package main 188 | 189 | import ( 190 | "fmt" 191 | ) 192 | 193 | //declare a rectangle struct 194 | type rectangle struct { 195 | length int 196 | width int 197 | } 198 | 199 | //declare a square struct 200 | type square struct { 201 | side int 202 | } 203 | 204 | //declare an interface with area() as a member 205 | type shape interface { 206 | area() int 207 | } 208 | 209 | //declare a method area() 210 | //the rectangle struct implements area() method in shape interface 211 | func (r rectangle) area() int { 212 | return r.length * r.width 213 | } 214 | 215 | //the square struct implements area() method in shape interface 216 | func (s square) area() int { 217 | return s.side * s.side 218 | } 219 | 220 | //declare a method with type shape as a parameter 221 | /** 222 | anything that implements all methods in shape interface is considered as a shape in general. 223 | for this case the rectangle and square is a shape because implements all methods in shape interface 224 | **/ 225 | func info(s shape) { 226 | fmt. Println("the area: ", s. area()) 227 | } 228 | 229 | func main() { 230 | r1 := rectangle{12, 12} 231 | info(r1) 232 | 233 | s1 := square{25} 234 | info(s1) 235 | 236 | } 237 | 238 | // output 239 | // the area: 144 240 | // the area: 625 241 | ``` 242 | 243 | It is also worth noting that interfaces can also be combined. 244 | -------------------------------------------------------------------------------- /problems/struct.md: -------------------------------------------------------------------------------- 1 | Go 结构体和结构体指针调用有什么区别吗？ 2 | 3 | ​ What are the uses of an empty struct? 4 | 5 | 6 | 7 | 1. ​ Go 结构体和结构体指针调用有什么区别吗？ 8 | 2. ​ Go 结构体是否可以比较，为什么？ 9 | 10 | ​ What are Golang pointers? 11 | 12 | 13 | 14 | 1. ​ types of Assigning a variable 15 | 2. ​ How can I get the type of object?, 16 | 3. 17 | 4. ​ how you can compare 2 objects in GO?, 18 | 5. ​ Go 是传值还是传引用？ 19 | 20 | ​ check the type of a variable at runtime in Go 21 | 22 | 23 | 24 | ​ Shadowing in Go? 25 | 26 | 27 | 28 | variadic functions in Go -------------------------------------------------------------------------------- /problems/swap-the-values-of-two-variables.md: -------------------------------------------------------------------------------- 1 | # How to Swap the Values of Two Variables 2 | 3 | 4 | 5 | Just this. 6 | 7 | ```go 8 | a := 1 9 | b := 2 10 | a, b, a = b, a, b 11 | ``` 12 | 13 | 14 | 15 | #### Why? 16 | 17 | See how code actually works, 18 | 19 | ```HTML 20 | main.go:5 0x454b9b 48c744241801000000 mov qword ptr [rsp+0x18], 0x1 // a:=1 21 | main.go:6 0x454ba4 48c744241002000000 mov qword ptr [rsp+0x10], 0x2 // b:=2 22 | main.go:7 0x454bad 488b442418 mov rax, qword ptr [rsp+0x18] 23 | main.go:7 0x454bb2 4889442428 mov qword ptr [rsp+0x28], rax // aTemp := a 24 | main.go:7 0x454bb7 488b442410 mov rax, qword ptr [rsp+0x10] 25 | main.go:7 0x454bbc 4889442420 mov qword ptr [rsp+0x20], rax // bTemp := b 26 | main.go:7 0x454bc1 488b442410 mov rax, qword ptr [rsp+0x10] 27 | main.go:7 0x454bc6 4889442418 mov qword ptr [rsp+0x18], rax // a = b 28 | main.go:7 0x454bcb 488b442428 mov rax, qword ptr [rsp+0x28] 29 | main.go:7 0x454bd0 4889442410 mov qword ptr [rsp+0x10], rax // b = aTemp 30 | main.go:7 0x454bd5 488b442420 mov rax, qword ptr [rsp+0x20] 31 | main.go:7 0x454bda 4889442418 mov qword ptr [rsp+0x18], rax // a = bTemp 32 | ``` 33 | 34 | 35 | This code create 2 more variables [rsp+0x28] as aTemp, [rsp+0x20] as bTemp. 36 | 37 | The principle of the two value exchange operations here is to store the values of the two assigned variables in **temporary variables**, and then use the temporary variables to assign values. So the order of assignment in this example has no effect on the result, and the result is still a = 2, b = 1. 38 | 39 | There is no need to write the swap function and then inline it like C language, which is equivalent to leaving the dirty work to the compiler. 40 | 41 | The code actually like this. 42 | 43 | ``` 44 | a := 1 45 | b := 2 46 | FIND a, b, a = b, a, b // Need use a and b to set values. 47 | aTemp = a 48 | bTemp = b 49 | a, b, a = bTemp, aTemp, bTemp 50 | ``` 51 | 52 | 53 | 54 | #### What's more 55 | 56 | If we want to dig out more about how this code works. We need know some context about how Go Compiler works. 57 | 58 | Complier Steps, 59 | 60 | 1. Parsing 61 | 62 | 2. Type checking 63 | 64 | 3. IR construction (intermediate representation, "noding") 65 | 66 | 4. Middle end 67 | 68 | 5. Walk 69 | 70 | 6. Generic SSA (Static Single Assignment) 71 | 72 | 7. Generating machine code 73 | 74 | 75 | 76 | In simple terms, before the code written in Go language is generated into machine code, it needs to first generate intermediate code (IR) with SSA characteristics. The most important feature of this form of IR code is that variables are always defined before they are used and each variable is assigned only once. 77 | 78 | IR is a crucial step because the optimization of Go language takes place there, including some syntax transformations. 79 | 80 | For example, 81 | 82 | ```go 83 | package main 84 | 85 | func main() { 86 | a := 1 87 | b := 2 88 | use(a, b) 89 | } 90 | 91 | func use(a int, b int) (int, int) { 92 | a, b = b, a 93 | return a, b 94 | } 95 | ``` 96 | 97 | Use this commad to generate SSA and show how go compiler works. 98 | 99 | ``` 100 | GOSSAFUNC="use" go build main.go 101 | ``` 102 | 103 | We can see the result, The compiler insert a CX as a temp values. 104 | 105 | ![ssa](../pictures/ssa.png) 106 | 107 | #### Reference 108 | 109 | 1. https://github.com/golang/go/tree/master/src/cmd/compile 110 | -------------------------------------------------------------------------------- /problems/sync-map.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Sync.map 4 | 5 | 6 | 7 | Go's built-in **map** does not support concurrent write operations. The reason is that **map** write operations are not concurrently safe. When you try to operate the same **map** with multiple **Goroutines**, an error will be reported: 8 | 9 | ``` 10 | fatal error: concurrent map writes. 11 | ``` 12 | 13 | Therefore, the official introduced **sync.Map** to meet the application in concurrent programming. 14 | The implementation principle of **sync.Map** can be summarized as: 15 | 16 | 1. Read and write are separated by two fields: **read** and **dirty**. The **read** data is stored in the read-only field **read**, and the latest written data is stored in the **dirty** field. 17 | 2. When reading, it will first query **read**, and then query **dirty** if it does not exist, and only write **dirty** when writing 18 | Reading **read** does not require locking, but reading or writing **dirty** requires locking 19 | 3. In addition, there is a **misses** field to count the number of **read** penetrations (penetration refers to the need to read **dirty**), and if the number exceeds a certain number, the **dirty** data will be synchronized to the **read** 20 | 4. For deleting **data**, directly mark it to delay deletion 21 | 22 | The data structure is this, 23 | 24 | ![sync-map](..\pictures\sync-map.png) 25 | 26 | 27 | 28 | ```go 29 | type Map struct { 30 | // mutex to protect dirty 31 | mu Mutex 32 | // readOnly 33 | read atomic.Value 34 | // save new data 35 | dirty map[interface{}]*entry 36 | // counter for read dirty 37 | misses int 38 | } 39 | 40 | type readOnly struct { 41 | // inner map 42 | m map[interface{}]*entry 43 | // if should read dirty 44 | amended bool 45 | } 46 | 47 | type entry struct { 48 | p unsafe.Pointer // *interface{} 49 | } 50 | ``` 51 | 52 | 53 | 54 | ### Load 55 | 56 | A get function. 57 | 58 | 59 | 60 | ```go 61 | func (m *Map) Load(key interface{}) (value interface{}, ok bool) { 62 | // try to get read first readOnly 63 | read, _ := m.read.Load().(readOnly) 64 | e, ok := read.m[key] 65 | 66 | // try to get key from dirty 67 | if !ok && read.amended { 68 | m.mu.Lock() 69 | read, _ = m.read.Load().(readOnly) 70 | e, ok = read.m[key] 71 | 72 | // try get data from dirty 73 | if !ok && read.amended { 74 | e, ok = m.dirty[key] 75 | // call miss 76 | m.missLocked() 77 | } 78 | m.mu.Unlock() 79 | } 80 | 81 | if !ok { 82 | return nil, false 83 | } 84 | 85 | return e.load() 86 | } 87 | 88 | func (m *Map) missLocked() { 89 | m.misses++ 90 | if m.misses < len(m.dirty) { 91 | return 92 | } 93 | // if too much miss save dirty to read 94 | m.read.Store(readOnly{m: m.dirty}) 95 | m.dirty = nil 96 | m.misses = 0 97 | } 98 | ``` 99 | 100 | 101 | 102 | ### Store 103 | 104 | ```go 105 | func (m *Map) Store(key, value interface{}) { 106 | read, _ := m.read.Load().(readOnly) 107 | // if in read save to entry 108 | if e, ok := read.m[key]; ok && e.tryStore(&value) { 109 | return 110 | } 111 | 112 | m.mu.Lock() 113 | read, _ = m.read.Load().(readOnly) 114 | 115 | if e, ok := read.m[key]; ok { 116 | 117 | if e.unexpungeLocked() { 118 | 119 | m.dirty[key] = e 120 | } 121 | 122 | e.storeLocked(&value) 123 | } else if e, ok := m.dirty[key]; ok { 124 | 125 | e.storeLocked(&value) 126 | } else { 127 | 128 | if !read.amended { 129 | 130 | m.dirtyLocked() 131 | 132 | m.read.Store(readOnly{m: read.m, amended: true}) 133 | } 134 | 135 | m.dirty[key] = newEntry(value) 136 | } 137 | m.mu.Unlock() 138 | } 139 | 140 | func (e *entry) tryStore(i *interface{}) bool { 141 | for { 142 | p := atomic.LoadPointer(&e.p) 143 | if p == expunged { 144 | return false 145 | } 146 | if atomic.CompareAndSwapPointer(&e.p, p, unsafe.Pointer(i)) { 147 | return true 148 | } 149 | } 150 | } 151 | 152 | func (e *entry) unexpungeLocked() (wasExpunged bool) { 153 | return atomic.CompareAndSwapPointer(&e.p, expunged, nil) 154 | } 155 | 156 | func (e *entry) storeLocked(i *interface{}) { 157 | atomic.StorePointer(&e.p, unsafe.Pointer(i)) 158 | } 159 | 160 | func (m *Map) dirtyLocked() { 161 | if m.dirty != nil { 162 | return 163 | } 164 | 165 | read, _ := m.read.Load().(readOnly) 166 | m.dirty = make(map[interface{}]*entry, len(read.m)) 167 | for k, e := range read.m { 168 | 169 | if !e.tryExpungeLocked() { 170 | m.dirty[k] = e 171 | } 172 | } 173 | } 174 | 175 | func (e *entry) tryExpungeLocked() (isExpunged bool) { 176 | p := atomic.LoadPointer(&e.p) 177 | for p == nil { 178 | 179 | if atomic.CompareAndSwapPointer(&e.p, nil, expunged) { 180 | return true 181 | } 182 | p = atomic.LoadPointer(&e.p) 183 | } 184 | return p == expunged 185 | } 186 | ``` 187 | 188 | 189 | 190 | It can be seen that through this design of read-write separation, the write security in the concurrent situation is solved, and the read speed can be close to the built-in map in most cases, which is very suitable for the situation of more reads and less writes. 191 | -------------------------------------------------------------------------------- /problems/timer.md: -------------------------------------------------------------------------------- 1 | # TImer 2 | 3 | 4 | 5 | In Go, `time.Timer` is a commonly used timer that can be used to execute an operation after a specified time interval. Its source code implementation is based on Go's coroutines and the underlying operating system's scheduler, and can trigger an operation precisely after the specified time. 6 | 7 | The following is the main source code implementation principle of `time.Timer`: 8 | 9 | 1. When a `time.Timer` object is created, the `runtime_startTimer` function is called to start a new timer. This function calculates the corresponding absolute time based on the timer's expiration time and adds the timer to the scheduler's timer queue. 10 | 2. When the timer expires, the scheduler adds the timer to the run queue and waits for the Go coroutine to execute. When the scheduler detects a timer in the run queue, it removes the timer from the queue and adds the coroutine corresponding to the timer to the run queue. 11 | 3. The coroutine corresponding to the timer is awakened by the scheduler and executes the corresponding operation. If the timer is a looping timer, the coroutine restarts the timer after executing the operation; otherwise, the timer is destroyed. 12 | 13 | Overall, the implementation principle of `time.Timer` is based on the scheduler's timer queue and run queue, using the underlying operating system's timer mechanism to achieve efficient and accurate timer functionality. 14 | 15 | 16 | 17 | Whether it is business development or infrastructure development, **timers** cannot be bypassed, which shows the importance of **timers**. 18 | 19 | Regardless of whether we use NewTimer, timer.After, or timer.AfterFun to initialize a **timer**, this timer **will** eventually be added to a global **timer** heap, which is managed uniformly by the Go runtime. 20 | 21 | Timer is the most complex and difficult data structure in Go 22 | 23 | 24 | 25 | ### Quad Heap 26 | 27 | The global heap of the timer is a quadruple heap, and each P maintains a quadruple heap, which reduces the concurrency problem between **Goroutines** and improves the performance of the **timer**. 28 | 29 | A quad heap is actually a quad tree. How does the Go timer maintain the quad heap? 30 | 31 | When the Go runtime schedules **timers**, the **timers** whose trigger time is earlier should reduce the number of queries and be triggered as soon as possible. Therefore, the trigger time of the parent node of the quadtree must be less than that of the child node. 32 | As the name implies, the quadtree has up to four child nodes. In order to take into account the speed of insertion, deletion, and rearrangement of the quadtree, the four sibling nodes are not required to be sorted by triggering sooner or later. 33 | 34 | ![quad-heap](..\pictures\quad-heap.gif)\ 35 | 36 | -------------------------------------------------------------------------------- /problems/type-assertion.md: -------------------------------------------------------------------------------- 1 | # Assertion 2 | 3 | Type assertion (assertion) is an operation for interface value, the syntax is x.(T), x is an expression of interface type, and T is assertd type, the type being asserted. 4 | 5 | 6 | There are two main scenarios for the use of assertions: If the asserted type is a concrete type, an instance class type, the assertion will check whether the dynamic type of x is the same as T, if they are the same, the result of the assertion is the dynamic value of x, of course the dynamic value type is T. In other words, the assertion on the concrete type actually fetches the dynamic value of x. 7 | 8 | 9 | If the asserted type is an interface type, the purpose of the assertion is to detect whether the dynamic type of x satisfies T, and if so, the result of the assertion is an expression that satisfies T, but its dynamic type and dynamic value are the same as x. In other words, an assertion on an interface type actually changes the type of x, usually the interface type of a larger method set, but preserves the original dynamic type and dynamic value. 10 | 11 | -------------------------------------------------------------------------------- /problems/use-new-or-make.md: -------------------------------------------------------------------------------- 1 | # Use New or Make? 2 | 3 | New is this from source code. 4 | 5 | ``` 6 | // The new built-in function allocates memory. The first argument is a type, 7 | // not a value, and the value returned is a pointer to a newly 8 | // allocated zero value of that type. 9 | func new(Type) *Type 10 | ``` 11 | 12 | 13 | 14 | Make is this from source code. 15 | 16 | ``` 17 | // The make built-in function allocates and initializes an object of type 18 | // slice, map, or chan (only). Like new, the first argument is a type, not a 19 | // value. Unlike new, make's return type is the same as the type of its 20 | // argument, not a pointer to it. The specification of the result depends on 21 | // the type: 22 | // Slice: The size specifies the length. The capacity of the slice is 23 | // equal to its length. A second integer argument may be provided to 24 | // specify a different capacity; it must be no smaller than the 25 | // length, so make([]int, 0, 10) allocates a slice of length 0 and 26 | // capacity 10. 27 | // Map: An empty map is allocated with enough space to hold the 28 | // specified number of elements. The size may be omitted, in which case 29 | // a small starting size is allocated. 30 | // Channel: The channel's buffer is initialized with the specified 31 | // buffer capacity. If zero, or the size is omitted, the channel is 32 | // unbuffered. 33 | func make(t Type, size ...IntegerType) Type 34 | ``` 35 | 36 | 37 | 38 | In short, there are some distinct differences between them: 39 | 40 | 1. Different purposes: `make` is used to create instances of built-in types (such as slices, maps, and channels), while `new` is used to create instances of any type. 41 | 2. Different parameters: `make` takes a type and an initial size as parameters, while `new` only takes a type as a parameter. 42 | 3. Different return values: `make` returns an initialized object, while `new` returns a pointer to an initialized object. 43 | 4. Different initialization: The object returned by `make` has already been initialized to its zero value, while the object returned by `new` still needs to be initialized. 44 | 45 | 46 | 47 | #### Why? 48 | 49 | From src/cmd/compile/internal/noder/transform.go , we can see this. 50 | 51 | ```go 52 | // Corresponds to Builtin part of tcCall. 53 | func transformBuiltin(n *ir.CallExpr) ir.Node { 54 | // n.Type() can be nil for builtins with no return value 55 | assert(n.Typecheck() == 1) 56 | fun := n.X.(*ir.Name) 57 | op := fun.BuiltinOp 58 | 59 | switch op { 60 | ... 61 | case ir.OMAKE: 62 | return transformMake(n) // MAKE will trans to another KeyWords. 63 | ... 64 | } 65 | ... 66 | } 67 | 68 | // Corresponds to typecheck.tcMake. 69 | func transformMake(n *ir.CallExpr) ir.Node { 70 | args := n.Args 71 | 72 | n.Args = nil 73 | l := args[0] 74 | t := l.Type() 75 | assert(t != nil) 76 | 77 | i := 1 78 | var nn ir.Node 79 | switch t.Kind() { 80 | case types.TSLICE: 81 | l = args[i] 82 | i++ 83 | var r ir.Node 84 | if i < len(args) { 85 | r = args[i] 86 | i++ 87 | } 88 | nn = ir.NewMakeExpr(n.Pos(), ir.OMAKESLICE, l, r) 89 | 90 | case types.TMAP: 91 | if i < len(args) { 92 | l = args[i] 93 | i++ 94 | } else { 95 | l = ir.NewInt(0) 96 | } 97 | nn = ir.NewMakeExpr(n.Pos(), ir.OMAKEMAP, l, nil) 98 | nn.SetEsc(n.Esc()) 99 | 100 | case types.TCHAN: 101 | l = nil 102 | if i < len(args) { 103 | l = args[i] 104 | i++ 105 | } else { 106 | l = ir.NewInt(0) 107 | } 108 | nn = ir.NewMakeExpr(n.Pos(), ir.OMAKECHAN, l, nil) 109 | default: 110 | panic(fmt.Sprintf("transformMake: unexpected type %v", t)) 111 | } 112 | 113 | assert(i == len(args)) 114 | typed(n.Type(), nn) 115 | return nn 116 | } 117 | ``` 118 | 119 | transformMake() Only allow 3 types of MAKE values, 120 | 121 | 1. TSLICE 122 | 2. TMAP 123 | 3. TCHAN 124 | 125 | This is how make implements. 126 | 127 | 128 | 129 | From src/cmd/compile/internal/ir/fmt.go, we can see this. 130 | 131 | ```go 132 | ONEW: "new", 133 | 134 | 135 | case ir.OCLOSE, ir.ONEW, ir.OUNSAFESTRINGDATA, ir.OUNSAFESLICEDATA: 136 | // nothing more to do 137 | return u1 138 | ``` 139 | 140 | ONEW is the operation ENUM for new, and this keywords will not change. The IR will be SSA, just like this, 141 | 142 | ``` 143 | case ir.ONEW: 144 | n := n.(*ir.UnaryExpr) 145 | var rtype *ssa.Value 146 | if x, ok := n.X.(*ir.DynamicType); ok && x.Op() == ir.ODYNAMICTYPE { 147 | rtype = s.expr(x.RType) 148 | } 149 | return s.newObject(n.Type().Elem(), rtype) 150 | ``` 151 | 152 | 153 | 154 | Implementation principles: 155 | 156 | - `make` function: It is a built-in function, a special memory allocation function used to create instances of **built-in types**. It implements the underlying memory allocation function at the bottom and also includes specific initialization steps. 157 | - `new` function: It is also a built-in function used to allocate an unnamed piece of memory and return a pointer to that memory. On the underlying implementation, it calls the memory allocation function to allocate memory for the object, but does not perform any initialization. -------------------------------------------------------------------------------- /problems/waitgroup.md: -------------------------------------------------------------------------------- 1 | # Waitgroup 2 | 3 | 4 | 5 | **WaitGroup** is a synchronization primitive in the Go programming language that allows you to wait for a group of **goroutines** to finish. To use **WaitGroup**, you first initialize its counter to a non-zero value, and then you increment the counter before each **goroutine** starts executing. Each **goroutine** should then call the Done method to signal that it has finished executing and decrement the counter. Finally, you call the Wait method to wait until the counter is zero. 6 | 7 | ```go 8 | package main 9 | 10 | import ( 11 | "fmt" 12 | "sync" 13 | ) 14 | 15 | func worker(id int, wg *sync.WaitGroup) { 16 | defer wg.Done() 17 | 18 | fmt.Printf("Worker %d starting\n", id) 19 | 20 | for i := 0; i < 100000000; i++ { 21 | _ = i * 2 22 | } 23 | fmt.Printf("Worker %d done\n", id) 24 | } 25 | 26 | func main() { 27 | var wg sync.WaitGroup 28 | 29 | // 3 worker goroutine 30 | for i := 1; i <= 3; i++ { 31 | wg.Add(1) 32 | go worker(i, &wg) 33 | } 34 | 35 | 36 | wg.Wait() 37 | 38 | fmt.Println("All workers done") 39 | } 40 | 41 | ``` 42 | 43 | 44 | 45 | 46 | 47 | Here's an example that shows how to use **WaitGroup** in Go. First, you create a **WaitGroup** object and start several worker **goroutines**. Each worker **goroutine** increments the **WaitGroup** counter before it starts executing and calls the Done method when it finishes. The main **goroutine** calls the Wait method to wait until all the worker **goroutines** have finished before proceeding. 48 | 49 | 50 | 51 | The underlying structure of **WaitGroup** looks simple, but **WaitGroup.state1** actually represents three fields: counter, waiter, sema. 52 | 53 | - counter : It can be understood as a counter, which calculates the value after wg.Add(N), wg.Done(). 54 | - waiter : The number of waiters currently waiting for the WaitGroup task to end. In fact, it is the number of calls to wg.Wait(), so usually this value is 1. 55 | - sema: semaphore, used to wake up the Wait() function. 56 | 57 | ```go 58 | type WaitGroup struct { 59 | noCopy noCopy 60 | state1 [3]uint32 61 | } 62 | ``` 63 | 64 | The callers who use **WaitGroup** are generally concurrent operations. If the counter and waiter are not obtained at the same time, the obtained counter and waiter may not match, resulting in program deadlock or premature termination of the program. So put counter and waiter together. -------------------------------------------------------------------------------- /problems/what-is-gpm-and-why-need-p.md: -------------------------------------------------------------------------------- 1 | # What is GPM and Why need P? --------------------------------------------------------------------------------