├── .gitignore ├── AnalysisApi ├── ArgOverwritten │ ├── .gitignore │ ├── LICENSE │ ├── README.md │ ├── cmd │ │ └── argoverwritten │ │ │ └── main.go │ ├── go.mod │ ├── go.sum │ └── passes │ │ └── ArgOverwritten │ │ ├── analyzer.go │ │ ├── analyzer_test.go │ │ └── testdata │ │ ├── AnonymousFunction │ │ └── main.go │ │ ├── AssigningParamToAVariableFirst │ │ └── main.go │ │ ├── DecrementOperator │ │ └── main.go │ │ ├── EmptyBodyFunction │ │ ├── add_amd64.s │ │ └── main.go │ │ ├── MultipleParamsOfSameType │ │ └── main.go │ │ ├── NoWarnings │ │ └── main.go │ │ ├── OverwritingParamFromOuterScope │ │ └── main.go │ │ ├── ShadowingVariable │ │ └── main.go │ │ └── SimpleOverwriting │ │ └── main.go └── README.md ├── CompilerFrontEndASTInGo ├── ArgsOverwriteAnalyzer.go ├── ArgsOverwriteAnalyzer_test.go ├── CodeExamples │ ├── BadExpr.txt │ ├── DifferentSpecsAndDecls.go │ └── SimpleForLoop.go ├── README.md ├── result │ └── ArgsOverwriteAnalyzer.go └── testdata │ ├── AnonymousFunction │ └── main.go │ ├── AssigningParamToAVariableFirst │ └── main.go │ ├── DecrementOperator │ └── main.go │ ├── EmptyBodyFunction │ ├── add_amd64.s │ └── main.go │ ├── LhsMultipleValues │ └── main.go │ ├── MultipleParamsOfSameType │ └── main.go │ ├── NoWarnings │ └── main.go │ ├── OverwritingParamFromOuterScope │ └── main.go │ ├── ShadowingVariable │ └── main.go │ └── SimpleOverwriting │ └── main.go ├── CompilerMiddleEndSSAInGo ├── CodeExamples │ ├── ElseIf │ │ └── ElseIf.go │ └── Map │ │ └── Map.go ├── README.md ├── TerminationInsideGoroutine.go ├── TerminationInsideGoroutine_test.go └── testdata │ ├── FatalInsideGoroutineAnonymousFlag │ └── main.go │ ├── FatalInsideGoroutineSimpleFlag │ └── main.go │ ├── FatalInsideRegularFunctionNoFlag │ └── main.go │ └── SkipInsideGoroutineSimpleFlag │ └── main.go ├── Conclusion └── README.md ├── Intro └── README.md ├── README.md ├── go.mod └── go.sum /.gitignore: -------------------------------------------------------------------------------- 1 | *.o 2 | *.a 3 | *.so 4 | .idea 5 | .vscode 6 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/.gitignore: -------------------------------------------------------------------------------- 1 | *.o 2 | *.a 3 | *.so 4 | .idea 5 | .vscode 6 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Amit Davidson 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/README.md: -------------------------------------------------------------------------------- 1 | # ArgOverwritten 2 | 3 | [![made-with-Go](https://github.com/go-critic/go-critic/workflows/Go/badge.svg)](http://golang.org) 4 | [![made-with-Go](https://img.shields.io/badge/Made%20with-Go-1f425f.svg)](http://golang.org) 5 | [![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](https://lbesson.mit-license.org/) 6 | [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com) 7 | 8 | ArgOverwritten finds function arguments being overwritten 9 | 10 | ## Quick Start: 11 | 12 | Download the package 13 | 14 | ``` bash 15 | go get github.com/amit-davidson/ArgOverwritten/cmd/argoverwritten 16 | ``` 17 | 18 | Pass the entry point 19 | 20 | ``` bash 21 | go vet -vettool=$(which argoverwritten) ${path_to_file} 22 | ``` 23 | 24 | ## Example: 25 | ``` go 26 | package testdata 27 | 28 | func body(a int) { 29 | _ = func() { 30 | a = 5 31 | } 32 | } 33 | 34 | func main() { 35 | body(5) 36 | } 37 | ``` 38 | ``` bash 39 | go vet -vettool=$(which argoverwritten) /testdata/OverwritingParamFromOuterScope 40 | ``` 41 | 42 | Output 43 | ``` 44 | /testdata/OverwritingParamFromOuterScope/main.go:5:3: "a" overwrites func parameter 45 | ``` -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/cmd/argoverwritten/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "github.com/amit-davidson/ArgOverwritten/passes/ArgOverwritten" 5 | "golang.org/x/tools/go/analysis/singlechecker" 6 | ) 7 | 8 | func main() { 9 | singlechecker.Main(ArgOverwritten.Analyzer) 10 | } 11 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/go.mod: -------------------------------------------------------------------------------- 1 | module github.com/amit-davidson/ArgOverwritten 2 | 3 | go 1.15 4 | 5 | require golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e 6 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/go.sum: -------------------------------------------------------------------------------- 1 | github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 2 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 3 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 4 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 5 | golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4= 6 | golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 7 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 8 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 9 | golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= 10 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 11 | golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 12 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 13 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 14 | golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 15 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 16 | golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 17 | golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 18 | golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 19 | golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e h1:4nW4NLDYnU28ojHaHO8OVxFHk/aQ33U01a9cjED+pzE= 20 | golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= 21 | golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 22 | golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 23 | golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= 24 | golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 25 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/analyzer.go: -------------------------------------------------------------------------------- 1 | package ArgOverwritten 2 | 3 | import ( 4 | "fmt" 5 | "go/ast" 6 | "golang.org/x/tools/go/analysis" 7 | ) 8 | 9 | var Analyzer = &analysis.Analyzer{ 10 | Name: "ArgOverwritten", 11 | Doc: doc, 12 | Run: run, 13 | } 14 | 15 | const ( 16 | doc = "ArgOverwritten finds function arguments being overwritten" 17 | ) 18 | 19 | func report(pass *analysis.Pass, ident *ast.Ident) { 20 | message := fmt.Sprintf("\"%s\" overwrites func parameter", ident.Name) 21 | pass.Report(analysis.Diagnostic{ 22 | Pos: ident.Pos(), 23 | Message: message, 24 | }) 25 | } 26 | 27 | func run(pass *analysis.Pass) (interface{}, error) { 28 | visitor := func(node ast.Node) bool { 29 | var typ *ast.FuncType 30 | var body *ast.BlockStmt 31 | switch fn := node.(type) { 32 | case *ast.FuncDecl: // Regular function 33 | typ = fn.Type 34 | body = fn.Body 35 | case *ast.FuncLit: // Anonymous function 36 | typ = fn.Type 37 | body = fn.Body 38 | } 39 | if typ == nil || body == nil { // Exclude other types but also external functions with missing body 40 | return true 41 | } 42 | if len(typ.Params.List) == 0 { 43 | return true 44 | } 45 | 46 | for _, field := range typ.Params.List { 47 | for _, arg := range field.Names { 48 | obj := pass.TypesInfo.ObjectOf(arg) 49 | ast.Inspect(body, func(node ast.Node) bool { 50 | switch stmt := node.(type) { 51 | case *ast.AssignStmt: 52 | for _, lhs := range stmt.Lhs { 53 | ident, ok := lhs.(*ast.Ident) 54 | if ok && pass.TypesInfo.ObjectOf(ident) == obj { 55 | report(pass, ident) 56 | } 57 | } 58 | case *ast.IncDecStmt: 59 | ident, ok := stmt.X.(*ast.Ident) 60 | if ok && pass.TypesInfo.ObjectOf(ident) == obj { 61 | report(pass, ident) 62 | } 63 | } 64 | return true 65 | }) 66 | } 67 | } 68 | return true 69 | } 70 | for _, f := range pass.Files { 71 | ast.Inspect(f, visitor) 72 | } 73 | return nil, nil 74 | } 75 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/analyzer_test.go: -------------------------------------------------------------------------------- 1 | package ArgOverwritten 2 | 3 | import ( 4 | "fmt" 5 | "golang.org/x/tools/go/analysis/analysistest" 6 | "os" 7 | "testing" 8 | ) 9 | 10 | func TestAnalyzer(t *testing.T) { 11 | var testCases = []struct { 12 | name string 13 | }{ 14 | {name: "SimpleOverwriting"}, 15 | {name: "AnonymousFunction"}, 16 | {name: "OverwritingParamFromOuterScope"}, 17 | {name: "AssigningParamToAVariableFirst"}, 18 | {name: "MultipleParamsOfSameType"}, 19 | {name: "ShadowingVariable"}, 20 | {name: "EmptyBodyFunction"}, 21 | {name: "NoWarnings"}, 22 | {name: "DecrementOperator"}, 23 | } 24 | for _, tc := range testCases { 25 | t.Run(tc.name, func(t *testing.T) { 26 | analysistest.Run(t, fmt.Sprintf("%s%s%s", analysistest.TestData(), string(os.PathSeparator), tc.name), Analyzer) 27 | }) 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/testdata/AnonymousFunction/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func main() { 4 | _ = func(a int) { 5 | a = 5 // want `"a" overwrites func parameter` 6 | } 7 | } -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/testdata/AssigningParamToAVariableFirst/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func body(a int) { 4 | b := a 5 | b = 5 6 | _ = b 7 | } 8 | 9 | func main() { 10 | body(5) 11 | } -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/testdata/DecrementOperator/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | func retry(fn func() error, retries int) error { 4 | for { 5 | if err := fn(); err != nil { 6 | if retries < 1 { 7 | return err 8 | } 9 | retries-- // want `"retries" overwrites func parameter` 10 | continue 11 | } 12 | return nil 13 | } 14 | } 15 | func main() { 16 | _ = retry(func() error { 17 | return nil 18 | }, 5) 19 | } 20 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/testdata/EmptyBodyFunction/add_amd64.s: -------------------------------------------------------------------------------- 1 | TEXT ·Add(SB),$0-24 2 | MOVQ a+0(FP), AX 3 | ADDQ b+8(FP), AX 4 | MOVQ AX, ret+16(FP) 5 | RET 6 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/testdata/EmptyBodyFunction/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | // Add returns the sum of a and b. 4 | func Add(a int64, b int64) int64 5 | 6 | func body(a int) { 7 | b := a 8 | b = 5 9 | _ = b 10 | } 11 | 12 | func main() { 13 | body(5) 14 | _ = Add(3, 4) 15 | } 16 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/testdata/MultipleParamsOfSameType/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func body(a, b int, c int) { 4 | a = 5 // want `"a" overwrites func parameter` 5 | _ = b 6 | c = 3 // want `"c" overwrites func parameter` 7 | } 8 | 9 | func main() { 10 | body(1, 2, 3) 11 | } 12 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/testdata/NoWarnings/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | import "fmt" 4 | 5 | func closeBody(body int) { 6 | fmt.Print(body) 7 | } 8 | 9 | func main() { 10 | closeBody(1) 11 | } 12 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/testdata/OverwritingParamFromOuterScope/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func body(a int) { 4 | _ = func() { 5 | a = 5 // want `"a" overwrites func parameter` 6 | } 7 | } 8 | 9 | func main() { 10 | body(5) 11 | } -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/testdata/ShadowingVariable/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func body(a, b int, c int) { 4 | f := func(a int, b int) { 5 | a = 5 // want `"a" overwrites func parameter` 6 | } 7 | _ = f 8 | } 9 | 10 | func main() { 11 | body(1, 2, 3) 12 | } 13 | -------------------------------------------------------------------------------- /AnalysisApi/ArgOverwritten/passes/ArgOverwritten/testdata/SimpleOverwriting/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func closeBody(body int) { 4 | body = 1 // want `"body" overwrites func parameter` 5 | } 6 | 7 | func main() { 8 | closeBody(1) 9 | } 10 | -------------------------------------------------------------------------------- /AnalysisApi/README.md: -------------------------------------------------------------------------------- 1 | ## 4 Analysis API: 2 | ### 4.1 (tools/go/analysis]((https://pkg.go.dev/golang.org/x/tools/go/analysis)) 3 | The package defines an [API](https://pkg.go.dev/golang.org/x/tools/go/analysis) for modular static analysis tools. In other words, it's a common interface for all static 4 | code analyzers. 5 | 6 | Why should you care? What's the difference between using the analysis API, and the way we wrote so far? 7 | The analysis API makes writing analyses easier by taking care of: 8 | - file parsing 9 | - testing 10 | - integration with go vet 11 | - many more 12 | 13 | Also, it enforces a single pattern for all the static analysis tools such as how analysis are structured and how 14 | warnings are reported 15 | 16 | ### 4.2 Analysis API members 17 | The primary type in the API is [`analysis.Analyzer`](https://pkg.go.dev/golang.org/x/tools/go/analysis#hdr-Analyzer). 18 | It describes an analysis function: its name, documentation, flags, relationship to other analyzers, and of course, it's logic. 19 | 20 | ``` go 21 | type Analyzer struct { 22 | Name string 23 | Doc string 24 | Requires []*Analyzer 25 | Run func(*Pass) (interface{}, error) 26 | 27 | ... 28 | } 29 | ``` 30 | 31 | The `Name` and `Doc` are obvious. They are used to describe what the tool does. 32 | 33 | Another interesting is the `Requires` field. It specifies a list of analyses upon which this one depends and whose 34 | results it may access, and it constrains the order in which a driver may run analyses. 35 | 36 | > To use SSA in the analysis api, we would have to require the [SSA builder](https://pkg.go.dev/golang.org/x/tools/go/analysis/passes/buildssa) 37 | > `Requires: []*analysis.Analyzer { 38 | buildssa.Analyzer 39 | }` 40 | 41 | 42 | The most important one is the `Run` function. It contains the logic that should is executed upon a single package. 43 | It takes as an argument [`*analysis.Pass`](https://pkg.go.dev/golang.org/x/tools/go/analysis#hdr-Pass) and returns a 44 | value to be used by other analyzers and an error. 45 | 46 | ``` go 47 | type Pass struct { 48 | Fset *token.FileSet 49 | Files []*ast.File 50 | Pkg *types.Package 51 | TypesInfo *types.Info 52 | Report func(Diagnostic) 53 | 54 | ... 55 | } 56 | ``` 57 | 58 | A `Pass` describes a single unit of work: the application of a particular `Analyzer` to a particular package of Go code. The `Pass` provides information to the Analyzer's `Run` function about the analyzed package and provides operations to the `Run` function for reporting diagnostics and other information back to the driver. It also provides `Fset`, `Files`, `Pkg`, and `TypesInfo` that we know from earlier, so we don't have to take care of ourselves. 59 | 60 | The [`Report`](https://pkg.go.dev/golang.org/x/tools/go/analysis#Pass.Reportf) function emits a diagnostic, a message associated with a source position. For most analyses, diagnostics are their primary result. For convenience, `Pass` provides a helper method, `Reportf`, to report a new diagnostic by formatting a string. Diagnostic is defined as: 61 | 62 | ``` go 63 | type Diagnostic struct { 64 | Pos token.Pos 65 | Category string // optional 66 | Message string 67 | } 68 | ``` 69 | 70 | ### 4.3 Project structure 71 | First let's define the project structure: 72 | ``` 73 | │── README.md 74 | │── cmd 75 | │ └── analyzerName 76 | │ └── main.go 77 | │── go.mod 78 | │── go.sum 79 | └── passes 80 | └── passName 81 | │── pass.go 82 | │── pass_test.go 83 | └── testdata 84 | ``` 85 | 86 | We create a directory where all of our passes reside in named `passes`. Each pass lives in its package, including its logic and tests. 87 | Then we define the usual `cmd` for our executables that contains all the analyzers the module has. 88 | 89 | Regarding our analyzers we wrote previously, we had to handle both the logic of the analyzer and the instrumentation around it. 90 | When converting our code to the analysis API, the AST traversal part (the logic) will sit under `passes` and loading the code 91 | part will be taken care of by the analysis API so we can ignore it. 92 | 93 | Next , we need a way to run the Analyzer and to test it. 94 | 95 | ### 4.4 Running our code 96 | inside `main.go`, we'll add the following code. 97 | 98 | ``` go 99 | package main 100 | 101 | import ( 102 | "path/to/our/pass" 103 | "golang.org/x/tools/go/analysis/singlechecker" 104 | ) 105 | 106 | func main() { singlechecker.Main(passName.Analyzer) } 107 | ``` 108 | Analyzers are provided in the form of packages that a driver program is expected to import. 109 | The [`singlechecker`](https://pkg.go.dev/golang.org/x/tools/go/analysis/singlechecker) package provides the `main` function for a command that runs one Analyzer. By convention, each Analyzer should be accompanied by a singlechecker-based command defined in its entirety as: This code calls our Analyzer. 110 | If we wanted our command to run multiple analyzers, we would have to use [`multichecker`](https://pkg.go.dev/golang.org/x/tools/go/analysis/multichecker). 111 | 112 | Now we can run it using 113 | ``` bash 114 | go install path/to/analyzer 115 | go vet -vettool=$(which analyzername) path/to/files 116 | ``` 117 | 118 | 119 | ### 4.5 Testing our code 120 | The [`analysistest`](https://godoc.org/golang.org/x/tools/go/analysis/analysistes) subpackage provides utilities for testing an Analyzer. Using `analysistest.Run`, it is possible to run an analyzer on a package of `testdata` files and check that it reported all the expected diagnostics. 121 | Expectations are expressed using "// want ..." comments in the input code, such as the following: 122 | 123 | ``` go 124 | package testdata 125 | 126 | func main() { 127 | _ = func(a int) { 128 | a = 5 // want `"a" overwrites func parameter` 129 | } 130 | } 131 | ``` 132 | 133 | ### 4.6 Overviewing an analyzer! 134 | In this section, we'll convert our [`ArgsOverwrite`](https://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/blob/master/CompilerFrontEndASTInGo/result/ArgsOverwriteAnalyzer.gohttps://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/blob/master/CompilerFrontEndASTInGo/result/ArgsOverwriteAnalyzer.go) 135 | Analyzer from earlier to the analysis API 136 | 137 | ### 4.7 Congratulations 138 | You have a good understanding of what the analysis API is and how to use it help us in writing analyses in the future. 139 | 140 | In the [next section](https://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/blob/master/Conclusion/README.md) 141 | we'll conclude this workshop by touching a point regarding static code analyzers in general and take a look at other code 142 | analyzers written by the Go community. -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/ArgsOverwriteAnalyzer.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "go/ast" 6 | "go/importer" 7 | "go/parser" 8 | "go/token" 9 | "go/types" 10 | "log" 11 | "os" 12 | ) 13 | 14 | func main() { 15 | path := os.Args[1] 16 | fset, pkgs, err := loadProgram(path) 17 | if err != nil { 18 | log.Fatalf("Failed to parse dir %s: %s", path, err) 19 | } 20 | 21 | for _, pkg := range pkgs { 22 | outputs := analyzePackage(pkg, fset) 23 | for _, message := range outputs { 24 | fmt.Println(message) 25 | } 26 | } 27 | } 28 | 29 | func loadProgram(path string) (*token.FileSet, map[string]*ast.Package, error) { 30 | fset := token.NewFileSet() 31 | pkgs, err := parser.ParseDir(fset, path, nil, 0) 32 | if err != nil { 33 | return nil, nil, err 34 | } 35 | return fset, pkgs, nil 36 | } 37 | 38 | func populateTypes(conf types.Config, fset *token.FileSet, f *ast.File) (*types.Info, error) { 39 | info := &types.Info{ 40 | Defs: make(map[*ast.Ident]types.Object), 41 | Uses: make(map[*ast.Ident]types.Object), 42 | } 43 | _, err := conf.Check("", fset, []*ast.File{f}, info) 44 | if err != nil { 45 | return nil, err 46 | } 47 | return info, nil 48 | } 49 | 50 | func analyzePackage(p *ast.Package, fset *token.FileSet) []string { 51 | var info *types.Info 52 | var err error 53 | outputs := make([]string, 0) 54 | 55 | visitor := func(node ast.Node) bool { 56 | var typ *ast.FuncType 57 | var body *ast.BlockStmt 58 | switch fn := node.(type) { 59 | case *ast.FuncDecl: // Regular function 60 | typ = fn.Type 61 | body = fn.Body 62 | case *ast.FuncLit: // Anonymous function 63 | typ = fn.Type 64 | body = fn.Body 65 | } 66 | if typ == nil || body == nil { // Exclude other types but also external functions with missing body 67 | return true 68 | } 69 | if len(typ.Params.List) == 0 { 70 | return true 71 | } 72 | 73 | for _, field := range typ.Params.List { 74 | for _, arg := range field.Names { 75 | obj := info.ObjectOf(arg) 76 | ast.Inspect(body, func(node ast.Node) bool { 77 | switch stmt := node.(type) { 78 | case *ast.AssignStmt: 79 | var ident *ast.Ident 80 | // ------------------------------------------------------------------------- 81 | // info.ObjectOf requires *ast.ident. How can I extract *ast.ident from *ast.AssignStmt.Lhs? 82 | // The flow should be similar to how *ast.Ident is taken in the *ast.IncDecStmt case 83 | // ------------------------------------------------------------------------- 84 | if info.ObjectOf(ident) == obj { 85 | outputs = append(outputs, output(ident, fset.Position(ident.Pos()))) 86 | } 87 | 88 | case *ast.IncDecStmt: 89 | ident, ok := stmt.X.(*ast.Ident) 90 | if ok && info.ObjectOf(ident) == obj { 91 | outputs = append(outputs, output(ident, fset.Position(ident.Pos()))) 92 | } 93 | } 94 | return true 95 | }) 96 | } 97 | } 98 | return true 99 | } 100 | 101 | conf := types.Config{Importer: importer.Default()} 102 | for _, f := range p.Files { 103 | info, err = populateTypes(conf, fset, f) 104 | if err != nil { 105 | log.Fatal(err) 106 | } 107 | ast.Inspect(f, visitor) 108 | } 109 | return outputs 110 | } 111 | 112 | func output(ident *ast.Ident, pos token.Position) string { 113 | return fmt.Sprintf("\"%s\" overwrites func parameter in pos: %s:%d:%d", ident.Name, pos.Filename, pos.Line, pos.Column) 114 | } 115 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/ArgsOverwriteAnalyzer_test.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "github.com/stretchr/testify/assert" 5 | "github.com/stretchr/testify/require" 6 | "go/ast" 7 | "go/parser" 8 | "go/token" 9 | "path/filepath" 10 | "testing" 11 | ) 12 | 13 | func Test_analyzePackage(t *testing.T) { 14 | var testCases = []struct { 15 | name string 16 | result []string 17 | }{ 18 | {name: "SimpleOverwriting", result: []string{"\"body\" overwrites func parameter in pos"}}, 19 | {name: "AnonymousFunction", result: []string{"\"a\" overwrites func parameter in pos"}}, 20 | {name: "OverwritingParamFromOuterScope", result: []string{"\"a\" overwrites func parameter in pos"}}, 21 | {name: "AssigningParamToAVariableFirst", result: []string{}}, 22 | {name: "MultipleParamsOfSameType", result: []string{"\"a\" overwrites func parameter in pos", "\"c\" overwrites func parameter in pos"}}, 23 | {name: "ShadowingVariable", result: []string{"\"a\" overwrites func parameter in pos"}}, 24 | {name: "EmptyBodyFunction", result: []string{}}, 25 | {name: "NoWarnings", result: []string{}}, 26 | {name: "DecrementOperator", result: []string{"\"retries\" overwrites func parameter"}}, 27 | {name: "LhsMultipleValues", result: []string{"\"a\" overwrites func parameter", "\"b\" overwrites func parameter"}}, 28 | } 29 | for _, tc := range testCases { 30 | t.Run(tc.name, func(t *testing.T) { 31 | path := filepath.Join(".", "testdata", tc.name) 32 | fset := token.NewFileSet() 33 | pkgs, err := parser.ParseDir(fset, path, nil, 0) 34 | require.NoError(t, err) 35 | require.Len(t, pkgs, 1) 36 | var testPkg *ast.Package 37 | for _, pkg := range pkgs { 38 | testPkg = pkg 39 | } 40 | 41 | outputs := analyzePackage(testPkg, fset) 42 | assert.Len(t, outputs, len(tc.result)) 43 | for i := range outputs { 44 | assert.Contains(t, outputs[i], tc.result[i]) 45 | } 46 | }) 47 | } 48 | } 49 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/CodeExamples/BadExpr.txt: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | func main() { 4 | if { 5 | println("Hello, World!") 6 | } 7 | } 8 | 9 | // focus on the missing condition if and how it's represented in the AST -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/CodeExamples/DifferentSpecsAndDecls.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | const ( 4 | a = iota 5 | b 6 | c = 3 7 | ) 8 | 9 | type d int 10 | 11 | var ( 12 | e = 7 13 | ) 14 | 15 | func main() { 16 | const name = 8 17 | name2 = 7 18 | } 19 | 20 | // 1. See how each declaration is represented in the AST and the difference between them. Specifically look which type 21 | // implements each interface (spec or decl) by looking at the docs for go/ast: https://golang.org/pkg/go/ast/. 22 | 23 | // You can also look at the code, it gives a good overview: https://golang.org/src/go/ast/ast.go?s=26852:26891#L851 24 | // and here:https://golang.org/src/go/ast/ast.go?s=29093:29170#L929 25 | 26 | //2. Look at the difference between the "name" and "name2". 27 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/CodeExamples/SimpleForLoop.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | func main() { 4 | for n := 0; n <= 5; n++ { 5 | if n%2 == 0 { 6 | continue 7 | } 8 | } 9 | } 10 | 11 | //1. See the properties of the for loop 12 | 13 | //2. See the properties of the if condition 14 | 15 | //3. See how continue is represented in the code 16 | 17 | // You can go to https://golang.org/pkg/go/ast/ for the types and their documentation. It'll give you an explanation 18 | // about the properties of each type. 19 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/README.md: -------------------------------------------------------------------------------- 1 | ## Compiler front end, AST, and analysis introduction 2 | ### 2.1 Go packages overview 3 | There are six relevant packages regarding the compiler front end when talking about static analysis: 4 | 5 | - [token](https://golang.org/pkg/go/token/) - Package `token` defines constants representing the lexical tokens of Go 6 | - [scanner](https://golang.org/pkg/go/scanner/) - Package `scanner` implements a scanner for Go source text. It takes a `[]byte` as the source, which can then be tokenized through repeated calls to the `Scan` method. 7 | - [parser](https://golang.org/pkg/go/parser/) - Package `parser` implements a parser for Go source files. The output is an abstract syntax tree (AST) representing the Go source 8 | - [AST](https://golang.org/pkg/go/ast/) - Package `AST` declares the types used to represent syntax trees for Go packages. 9 | - [constant](https://golang.org/pkg/go/constant/) - Package `constant` implements Values representing untyped Go constants and their corresponding operations. 10 | - [types](https://golang.org/pkg/go/types/) - Package `types` declares the data types and implements the algorithms for type-checking of Go packages 11 | 12 | The `scanner` package is fed with `[]byte` representing the source code. Its output is a list of tokens defined by the 13 | `token` package, and the parser package uses them to create the `AST` tree. After the tree is constructed, 14 | the parser runs type-checking algorithms run over the tree, validates its correctness, and evaluates constants. 15 | 16 | 17 | 18 | ### 2.2 AST in Go 19 | An abstract syntax tree (AST) is a way of representing the syntax of a programming language as a hierarchical tree-like structure. Let's take a look at the following program for an explanation. 20 | 21 | ``` go 22 | package main 23 | import "fmt" 24 | 25 | func main() { 26 | fmt.Println("hello world") 27 | } 28 | ``` 29 | 30 | We can use this [AST visualizer](http://goast.yuroyoro.net/) to view it's AST. 31 | ``` 32 | 0 *ast.File { 33 | 1 . Package: 1:1 34 | 2 . Name: *ast.Ident { 35 | 3 . . NamePos: 1:9 36 | 4 . . Name: "main" 37 | 5 . } 38 | 6 . Decls: []ast.Decl (len = 2) { 39 | 7 . . 0: *ast.GenDecl { 40 | 8 . . . TokPos: 3:1 41 | 9 . . . Tok: import 42 | 10 . . . Lparen: - 43 | 11 . . . Specs: []ast.Spec (len = 1) { 44 | 12 . . . . 0: *ast.ImportSpec { 45 | 13 . . . . . Path: *ast.BasicLit { 46 | 14 . . . . . . ValuePos: 3:8 47 | 15 . . . . . . Kind: STRING 48 | 16 . . . . . . Value: "\"fmt\"" 49 | 17 . . . . . } 50 | 18 . . . . . EndPos: - 51 | 19 . . . . } 52 | 20 . . . } 53 | 21 . . . Rparen: - 54 | 22 . . } 55 | 23 . . 1: *ast.FuncDecl { 56 | 24 . . . Name: *ast.Ident { 57 | 25 . . . . NamePos: 5:6 58 | 26 . . . . Name: "main" 59 | 27 . . . . Obj: *ast.Object { 60 | 28 . . . . . Kind: func 61 | 29 . . . . . Name: "main" 62 | 30 . . . . . Decl: *(obj @ 23) 63 | 31 . . . . } 64 | 32 . . . } 65 | 33 . . . Type: *ast.FuncType { 66 | 34 . . . . Func: 5:1 67 | 35 . . . . Params: *ast.FieldList { 68 | 36 . . . . . Opening: 5:10 69 | 37 . . . . . Closing: 5:11 70 | 38 . . . . } 71 | 39 . . . } 72 | 40 . . . Body: *ast.BlockStmt { 73 | 41 . . . . Lbrace: 5:13 74 | 42 . . . . List: []ast.Stmt (len = 1) { 75 | 43 . . . . . 0: *ast.ExprStmt { 76 | 44 . . . . . . X: *ast.CallExpr { 77 | 45 . . . . . . . Fun: *ast.SelectorExpr { 78 | 46 . . . . . . . . X: *ast.Ident { 79 | 47 . . . . . . . . . NamePos: 6:3 80 | 48 . . . . . . . . . Name: "fmt" 81 | 49 . . . . . . . . } 82 | 50 . . . . . . . . Sel: *ast.Ident { 83 | 51 . . . . . . . . . NamePos: 6:7 84 | 52 . . . . . . . . . Name: "Println" 85 | 53 . . . . . . . . } 86 | 54 . . . . . . . } 87 | 55 . . . . . . . Lparen: 6:14 88 | 56 . . . . . . . Args: []ast.Expr (len = 1) { 89 | 57 . . . . . . . . 0: *ast.BasicLit { 90 | 58 . . . . . . . . . ValuePos: 6:15 91 | 59 . . . . . . . . . Kind: STRING 92 | 60 . . . . . . . . . Value: "\"hello world\"" 93 | 61 . . . . . . . . } 94 | 62 . . . . . . . } 95 | 63 . . . . . . . Ellipsis: - 96 | 64 . . . . . . . Rparen: 6:28 97 | 65 . . . . . . } 98 | 66 . . . . . } 99 | 67 . . . . } 100 | 68 . . . . Rbrace: 7:1 101 | 69 . . . } 102 | 70 . . } 103 | 71 . } 104 | 72 . Scope: *ast.Scope { 105 | 73 . . Objects: map[string]*ast.Object (len = 1) { 106 | 74 . . . "main": *(obj @ 27) 107 | 75 . . } 108 | 76 . } 109 | 77 . Imports: []*ast.ImportSpec (len = 1) { 110 | 78 . . 0: *(obj @ 12) 111 | 79 . } 112 | 80 . Unresolved: []*ast.Ident (len = 1) { 113 | 81 . . 0: *(obj @ 46) 114 | 82 . } 115 | 83 } 116 | ``` 117 | 118 | Let's focus on the JSON under [`*ast.File`](https://golang.org/pkg/go/ast/#File) representing a Go source file. The file is the root node, and it contains all 119 | the top-level declarations in the file - the import and the main function. Under `mains'` body, we have a 120 | [`*ast.blockStmt`](https://golang.org/pkg/go/ast/#BlockStmt) containing a list of the function statements. Similar to HTML, the dependency of the nodes create a 121 | tree-like structure. 122 | 123 | The syntax is "abstract" in the sense that it does not represent every detail appearing in the real syntax, but rather 124 | just the structural or content-related details. For instance, grouping parentheses are implicit in the tree structure, 125 | so these are not represented as separate nodes. 126 | 127 | ### 2.3 go/ast Package members 128 | The AST package contains the types used to represent syntax trees in Go. We can divide the members into three categories: 129 | Interfaces, concrete types, and others. 130 | 131 | - Concrete Types: The full list is [long](https://golang.org/pkg/go/ast/#ArrayType). Those are the tree nodes, and they contain values such as: `FuncDecl`, `IncDecStmt`, `Ident`, `Comment`, and so on. 132 | - Interfaces: `Node`, `Decl`, `Spec`, `Stmt`, `Expr` 133 | - Others: `Package`, `File`, `Scope`, `Object` 134 | 135 | Well take a look at `AssignStmt` 136 | ```go 137 | type AssignStmt struct { 138 | Lhs []Expr // All the variables to left side of assign operator 139 | TokPos token.Pos // position of operator 140 | Tok token.Token // assignment token. `=`, `:=`, `+=`, `<<=` and so on... 141 | Rhs []Expr // Expressions to right of the assignment operator 142 | } 143 | ``` 144 | For example, in the expression: `a := 5`, 145 | - Lhs is [a] 146 | - Rhs is [5] 147 | - TokPos [3] (the position of the ":" character) 148 | - Tok [:=] 149 | 150 | Pretty straight forward. Now we'll look at `Expr` which `AssignStmt` implements. `Expr` is common interface for 151 | everything that returns a value. As you can see, it only contains the node interface (which is implemented by all the nodes on the AST graph). 152 | 153 | ```go 154 | type Expr interface { 155 | Node 156 | // contains filtered or unexported methods 157 | } 158 | ``` 159 | 160 | From the other's group we'll look at `File`. 161 | ```go 162 | type File struct { 163 | Doc *CommentGroup // associated documentation; or nil 164 | Package token.Pos // position of "package" keyword 165 | Name *Ident // package name 166 | Decls []Decl // top-level declarations; or nil 167 | Scope *Scope // package scope (this file only) 168 | Imports []*ImportSpec // imports in this file 169 | Unresolved []*Ident // unresolved identifiers in this file 170 | Comments []*CommentGroup // list of all comments in the source file 171 | } 172 | 173 | ``` 174 | 175 | An `ast.File` is the root node of each of the files we analyze. When analyzing a program, we'll iterate over the files 176 | and for each we'll pass it to the `ast.Inspect` function for iteration. 177 | 178 | It's worth mentioning again that `ast` package contains only the "abstract" parts so it ignores parentheses, colon, etc... 179 | 180 | ### 2.4 Exercise: 181 | In the folder [`CompilerFrontEndASTInGo/CodeExamples`](https://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/tree/master/CompilerFrontEndASTInGo/CodeExamples) 182 | there are some interesting programs (well... AST-wise). Using our [AST visualizer](http://goast.yuroyoro.net/) 183 | from earlier, take each of the program and look at their AST. I added comments explaining the important points. 184 | 185 | You should start with [`SimpleForLoop`](https://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/blob/master/CompilerFrontEndASTInGo/CodeExamples/SimpleForLoop.go), 186 | move to [`BadExpr`](https://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/blob/master/CompilerFrontEndASTInGo/CodeExamples/BadExpr.txt) 187 | and finish with [`DifferentSpecsAndDecls`](https://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/blob/master/CompilerFrontEndASTInGo/CodeExamples/DifferentSpecsAndDecls.go) 188 | 189 | ### 2.5 Loading a program using the parser 190 | To load the program, we need to parse it first 191 | ``` go 192 | package main 193 | 194 | import ( 195 | "fmt" 196 | "go/ast" 197 | "go/parser" 198 | "go/token" 199 | ) 200 | 201 | func main() { 202 | 203 | src := `package main 204 | 205 | import "fmt" 206 | 207 | func main() { 208 | fmt.Println("hello world")} 209 | ` 210 | 211 | fset := token.NewFileSet() 212 | f, err := parser.ParseFile(fset, "", src, 0) 213 | if err != nil { 214 | fmt.Println(err) 215 | return 216 | } 217 | visitor := func(node ast.Node) bool { 218 | strLit, ok := (node).(*ast.BasicLit) 219 | if ok && strLit.Value == "\"hello world\"" { 220 | pos := fset.Position(strLit.Pos()) 221 | fmt.Printf("We found hello world in pos:%d:%d", pos.Line, pos.Column) 222 | return false 223 | } 224 | return true 225 | } 226 | ast.Inspect(f, visitor) 227 | } 228 | ``` 229 | 230 | 231 | We first create a [`fileSet`](https://golang.org/pkg/go/token/#FileSet), representing a set of source files. `FileSet` has the properties `files` and `base` 232 | recording the used files and all the files' total size. Using the size property of `token.File`, we can easily determine 233 | in which file a statement is, given its position. 234 | 235 | ```go 236 | fset := token.NewFileSet() 237 | ``` 238 | 239 | 240 | 241 | Then, we call the [`parser.ParseFile`](https://golang.org/pkg/go/parser/#ParseFile) function, providing it our `fileSet` to populate it, an empty path, a string as the 242 | source so the parser will use it instead of loading from a file, and a build mode - 0. In this example, we used 0 to 243 | fully load the program, but any other [mode](https://golang.org/pkg/go/parser/#Mode) can be used. 244 | 245 | ```go 246 | f, err := parser.ParseFile(fset, "", src, 0) 247 | if err != nil { 248 | fmt.Println(err) 249 | return 250 | } 251 | ``` 252 | > Tip: Instead of iterating file by file, you can load an entire directory using [`parser.ParseDir`](https://golang.org/pkg/go/parser/#ParseDir) 253 | 254 | Finally, we define a visitor function that will be called with each node inside the AST. We pass our function to 255 | [`ast.Inspect`](https://golang.org/pkg/go/ast/#Inspect) to iterate over all the nodes in depth-first order and print a message when we reach the 256 | `Hello World` string literal with it's position in the code. We return `true` each iteration to keep traversing the tree until we found the desired 257 | node. Then, we print our message and return false to indicate we're done searching and to exit the traverse function. 258 | 259 | ```go 260 | visitor := func(node ast.Node) bool { 261 | strLit, ok := (node).(*ast.BasicLit) 262 | if ok && strLit.Value == "\"hello world\"" { 263 | pos := fset.Position(strLit.Pos()) 264 | fmt.Printf("We found hello world in pos:%d:%d", pos.Line, pos.Column) 265 | return false 266 | } 267 | return true 268 | } 269 | ast.Inspect(f, visitor) 270 | ``` 271 | 272 | ### 2.6 Exercise 2! 273 | In the file [`CompilerFrontEndASTInGo/ArgsOverwriteAnalyzer.go`](https://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/blob/master/CompilerFrontEndASTInGo/ArgsOverwriteAnalyzer.go) 274 | we have an analyzer that checks if function arguments were modified as in the example below. The problem is that there 275 | are some parts missing from it. There are comments in the places where you should add your code according to the comment. 276 | You can run the tests [`CompilerFrontEndASTInGo/ArgsOverwriteAnalyzer_test.go`](https://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/blob/master/CompilerFrontEndASTInGo/ArgsOverwriteAnalyzer_test.go) 277 | to make sure your tests pass. You can also debug using the test to inspect the AST graph of this program. 278 | 279 | If you give up, you can see the result in [`CompilerFrontEndASTInGo/result/ArgsOverwriteAnalyzer.go`](https://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/blob/master/CompilerFrontEndASTInGo/result/ArgsOverwriteAnalyzer.go) :) 280 | 281 | 282 | ### 2.7 Congratulations 283 | You have a good understanding of what AST is, the different Go packages used to create static code analyzers that 284 | interact with it and how to write such analyzers. 285 | 286 | In the [next section](https://github.com/amit-davidson/GopherCon2021IsraelStaticAnalysisWorkshop/blob/master/CompilerMiddleEndSSAInGo) 287 | we'll focus on the middle end level, and see how analyzer "operating" in this level work. 288 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/result/ArgsOverwriteAnalyzer.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "go/ast" 6 | "go/importer" 7 | "go/parser" 8 | "go/token" 9 | "go/types" 10 | "log" 11 | "os" 12 | ) 13 | 14 | func main() { 15 | path := os.Args[1] 16 | fset, pkgs, err := loadProgram(path) 17 | if err != nil { 18 | log.Fatalf("Failed to parse dir %s: %s", path, err) 19 | } 20 | 21 | for _, pkg := range pkgs { 22 | outputs := analyzePackage(pkg, fset) 23 | for _, message := range outputs { 24 | fmt.Println(message) 25 | } 26 | } 27 | } 28 | 29 | func loadProgram(path string) (*token.FileSet, map[string]*ast.Package, error) { 30 | fset := token.NewFileSet() 31 | pkgs, err := parser.ParseDir(fset, path, nil, 0) 32 | if err != nil { 33 | return nil, nil, err 34 | } 35 | return fset, pkgs, nil 36 | } 37 | 38 | func populateTypes(conf types.Config, fset *token.FileSet, f *ast.File) (*types.Info, error) { 39 | info := &types.Info{ 40 | Defs: make(map[*ast.Ident]types.Object), 41 | Uses: make(map[*ast.Ident]types.Object), 42 | } 43 | _, err := conf.Check("", fset, []*ast.File{f}, info) 44 | if err != nil { 45 | return nil, err 46 | } 47 | return info, nil 48 | } 49 | 50 | func analyzePackage(p *ast.Package, fset *token.FileSet) []string { 51 | var info *types.Info 52 | var err error 53 | outputs := make([]string, 0) 54 | 55 | visitor := func(node ast.Node) bool { 56 | var typ *ast.FuncType 57 | var body *ast.BlockStmt 58 | switch fn := node.(type) { 59 | case *ast.FuncDecl: // Regular function 60 | typ = fn.Type 61 | body = fn.Body 62 | case *ast.FuncLit: // Anonymous function 63 | typ = fn.Type 64 | body = fn.Body 65 | } 66 | if typ == nil || body == nil { // Exclude other types but also external functions with missing body 67 | return true 68 | } 69 | if len(typ.Params.List) == 0 { 70 | return true 71 | } 72 | 73 | for _, field := range typ.Params.List { 74 | for _, arg := range field.Names { 75 | obj := info.ObjectOf(arg) 76 | ast.Inspect(body, func(node ast.Node) bool { 77 | switch stmt := node.(type) { 78 | case *ast.AssignStmt: 79 | for _, lhs := range stmt.Lhs { 80 | ident, ok := lhs.(*ast.Ident) 81 | if ok && info.ObjectOf(ident) == obj { 82 | outputs = append(outputs, output(ident, fset.Position(ident.Pos()))) 83 | } 84 | } 85 | case *ast.IncDecStmt: 86 | ident, ok := stmt.X.(*ast.Ident) 87 | if ok && info.ObjectOf(ident) == obj { 88 | outputs = append(outputs, output(ident, fset.Position(ident.Pos()))) 89 | } 90 | } 91 | return true 92 | }) 93 | } 94 | } 95 | return true 96 | } 97 | 98 | conf := types.Config{Importer: importer.Default()} 99 | for _, f := range p.Files { 100 | info, err = populateTypes(conf, fset, f) 101 | if err != nil { 102 | log.Fatal(err) 103 | } 104 | ast.Inspect(f, visitor) 105 | } 106 | return outputs 107 | } 108 | 109 | func output(ident *ast.Ident, pos token.Position) string { 110 | return fmt.Sprintf("\"%s\" overwrites func parameter in pos: %s:%d:%d", ident.Name, pos.Filename, pos.Line, pos.Column) 111 | } 112 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/AnonymousFunction/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func main() { 4 | _ = func(a int) { 5 | a = 5 6 | } 7 | } -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/AssigningParamToAVariableFirst/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func body(a int) { 4 | b := a 5 | b = 5 6 | _ = b 7 | } 8 | 9 | func main() { 10 | body(5) 11 | } -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/DecrementOperator/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | func retry(fn func() error, retries int) error { 4 | for { 5 | if err := fn(); err != nil { 6 | if retries < 1 { 7 | return err 8 | } 9 | retries-- 10 | continue 11 | } 12 | return nil 13 | } 14 | } 15 | 16 | func main() { 17 | _ = retry(func() error { 18 | return nil 19 | }, 5) 20 | } 21 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/EmptyBodyFunction/add_amd64.s: -------------------------------------------------------------------------------- 1 | TEXT ·Add(SB),$0-24 2 | MOVQ a+0(FP), AX 3 | ADDQ b+8(FP), AX 4 | MOVQ AX, ret+16(FP) 5 | RET 6 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/EmptyBodyFunction/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | // Add returns the sum of a and b. 4 | func Add(a int64, b int64) int64 5 | 6 | func body(a int) { 7 | b := a 8 | b = 5 9 | _ = b 10 | } 11 | 12 | func main() { 13 | body(5) 14 | _ = Add(3, 4) 15 | } 16 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/LhsMultipleValues/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func main() { 4 | _ = func(a, b int) { 5 | a, b = 5, 7 6 | } 7 | } 8 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/MultipleParamsOfSameType/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func body(a, b int, c int) { 4 | a = 5 5 | _ = b 6 | c = 3 7 | } 8 | 9 | func main() { 10 | body(1, 2, 3) 11 | } 12 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/NoWarnings/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | import "fmt" 4 | 5 | func closeBody(body int) { 6 | fmt.Print(body) 7 | } 8 | 9 | func main() { 10 | closeBody(1) 11 | } 12 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/OverwritingParamFromOuterScope/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func body(a int) { 4 | _ = func() { 5 | a = 5 6 | } 7 | } 8 | 9 | func main() { 10 | body(5) 11 | } -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/ShadowingVariable/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func body(a, b int, c int) { 4 | f := func(a int, b int) { 5 | a = 5 6 | } 7 | _ = f 8 | } 9 | 10 | func main() { 11 | body(1, 2, 3) 12 | } 13 | -------------------------------------------------------------------------------- /CompilerFrontEndASTInGo/testdata/SimpleOverwriting/main.go: -------------------------------------------------------------------------------- 1 | package testdata 2 | 3 | func closeBody(body int) { 4 | body = 1 5 | } 6 | 7 | func main() { 8 | closeBody(1) 9 | } 10 | -------------------------------------------------------------------------------- /CompilerMiddleEndSSAInGo/CodeExamples/ElseIf/ElseIf.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "os" 5 | "strconv" 6 | ) 7 | 8 | var a int 9 | 10 | func main() { 11 | a, _ = strconv.Atoi(os.Args[1]) 12 | if a > 0 { 13 | a = 3 14 | } else if a == 0 { 15 | a = 4 16 | } else { 17 | a = 5 18 | } 19 | } 20 | 21 | // 1. In the source code, the condition can go to the if, the else if or the else blocks. Here, each block can have up to 2 22 | // successors. Notice that the first block can only jump to 1 (a=3) or 3 (a==0) and 3 evaluates the condition again and can go to 23 | // 4 (a=4) or 5(a=5) 24 | 25 | // 2. `a` is declared in the global scope (and not under the function scope). Because it's in the global scope, it might 26 | // be used in other places, so the compiler isn't sure if it could have side effects, (not considered dead) so it can't 27 | // optimize away it using dead code elimination. 28 | -------------------------------------------------------------------------------- /CompilerMiddleEndSSAInGo/CodeExamples/Map/Map.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | func main() { 4 | 5 | m := make(map[string]int) 6 | 7 | m["k1"] = 7 8 | m["k2"] = 13 9 | 10 | v1 := m["k1"] 11 | _ = v1 12 | 13 | _, prs := m["k2"] 14 | _ = prs 15 | 16 | delete(m, "k2") 17 | } 18 | 19 | // See how different map operations are carried out in SSA: 20 | // 1. Creating a map 21 | // 2. Assigning a value to a map 22 | // 3. Reading from a map 23 | // 4. Reading from a map into multiple variables 24 | // 5. Deleting a key from a map 25 | -------------------------------------------------------------------------------- /CompilerMiddleEndSSAInGo/README.md: -------------------------------------------------------------------------------- 1 | ## 3. Compiler middle end and static analysis with SSA In Go 2 | ### 3.1 What is SSA? 3 | [SSA](https://en.wikipedia.org/wiki/Static_single_assignment_form) stands for static single assignment. It's a property of an IR **that requires each variable to be assigned exactly once**, and every variable be defined before it is used. 4 | The primary usefulness of SSA comes from how it simplifies the properties of variables and improves compilers optimizations. 5 | 6 | For example, consider this piece of code: 7 | ``` 8 | y := 1 9 | y := 2 10 | x := y 11 | ``` 12 | 13 | In an SSA form, it'll be translated to: 14 | ``` 15 | y1 := 1 16 | y2 := 2 17 | x1 := y2 18 | ``` 19 | Humans can see that the first assignment is unnecessary and that the value of `y` used in the third line comes from the 20 | second assignment of `y`. In SSA form, both of these are immediate 21 | 22 | ### 3.2 go/ssa package members 23 | The package [`tools/go/ssa`](https://pkg.go.dev/golang.org/x/tools/go/ssa) defines the representation of elements of Go programs in SSA format. 24 | The key types form a hierarchical structure. 25 | 26 | #### [Program](https://pkg.go.dev/golang.org/x/tools/go/ssa#Program) 27 | A Program is a partial or complete Go program converted to an SSA form. 28 | 29 | 30 | 31 | #### [Package](https://pkg.go.dev/golang.org/x/tools/go/ssa#Package) 32 | A Package is a single analyzed Go package containing Members for all package-level functions, variables, constants, and types it declares. 33 | 34 | 35 | 36 | #### [Function](https://pkg.go.dev/golang.org/x/tools/go/ssa#Function) 37 | Function represents the parameters, results, and code of a function or method. 38 | 39 | 40 | 41 | #### [Basic Block](https://pkg.go.dev/golang.org/x/tools/go/ssa#BasicBlock) 42 | BasicBlock represents an SSA basic block. A set of instructions that are executed and can't jump somewhere else. Basic blocks are connected using conditions and goto statements. 43 | 44 | 45 | 46 | Control Flow Graph (CFG) - In a control-flow graph, each node in the graph represents a basic block. 47 | Together, they compose all paths that might be traversed through a program during its execution. 48 | 49 | 50 | 51 | #### [Instruction](https://pkg.go.dev/golang.org/x/tools/go/ssa#Instruction) 52 | a statement that consumes values and performs computation. For example, `Call`, `Return`, `TypeAssert`, etc 53 | 54 | 55 | 56 | #### [Value](https://pkg.go.dev/golang.org/x/tools/go/ssa#Value) 57 | an expression that yields a value. For example, function calls are both `Instruction` and `Value` since they both consume values and yield a value. 58 | 59 | 60 | 61 | And when combined: 62 | 63 | 64 | 65 | The package contains other [types](https://pkg.go.dev/golang.org/x/tools/go/ssa#pkg-overview) - Include language keywords such as `Defer`, `If` but also lower level primitives like `MakeChan` and `Alloc`. 66 | 67 | 68 | ### 3.3 Overviewing an analyzer! 69 | In this section we'll look over an analyzer that warns when `t.Fatal` is used inside a goroutine as described here: 70 | https://github.com/ipfs/go-ipfs/issues/2043 71 | 72 | ### 3.4 SSA vs AST 73 | The most important difference is that AST reasons about the structure of the code, where SSA reasons about how data 74 | flows in the code. Why do need both? Each "level" suits for a different problem. You can think of it as satellite vs 75 | terrain modes on maps. They both represent the same source map, but each mode solves a different problem. 76 | 77 | We can summarize the differences using the following table: 78 | | | SSA | AST | 79 | |----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 80 | | Why to Choose? |