├── .travis.yml ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── cmdtest.go ├── cmdtest_test.go ├── go.mod ├── go.sum ├── tempFileNonWindows.go ├── tempFileWindows.go └── testdata ├── bad ├── bad-fail-1.ct ├── bad-fail-2.ct ├── bad-fail-3.ct ├── bad-fail-4.ct ├── bad-fail-5.ct ├── bad-fail-6.ct ├── bad-fail-7.ct ├── bad-fail-8.ct └── bad-output.ct ├── echo-stdin.go ├── good-without-output └── good-without-output.ct ├── good └── good.ct ├── parallel ├── par1.ct └── par2.ct ├── read └── read.ct └── update ├── update.ct └── update.golden /.travis.yml: -------------------------------------------------------------------------------- 1 | language: go 2 | 3 | go: 4 | - 1.12.x 5 | 6 | env: 7 | global: 8 | - GOPROXY=https://proxy.golang.org 9 | - GO111MODULE=on 10 | 11 | script: 12 | - go test 13 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # How to Contribute 2 | 3 | We'd love to accept your patches and contributions to this project. There are 4 | just a few small guidelines you need to follow. 5 | 6 | ## Contributor License Agreement 7 | 8 | Contributions to this project must be accompanied by a Contributor License 9 | Agreement. You (or your employer) retain the copyright to your contribution; 10 | this simply gives us permission to use and redistribute your contributions as 11 | part of the project. Head over to to see 12 | your current agreements on file or to sign a new one. 13 | 14 | You generally only need to submit a CLA once, so if you've already submitted one 15 | (even if it was for a different project), you probably don't need to do it 16 | again. 17 | 18 | ## Code reviews 19 | 20 | All submissions, including submissions by project members, require review. We 21 | use GitHub pull requests for this purpose. Consult 22 | [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more 23 | information on using pull requests. 24 | 25 | ## Community Guidelines 26 | 27 | This project follows [Google's Open Source Community 28 | Guidelines](https://opensource.google.com/conduct/). 29 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Build Status](https://travis-ci.org/google/go-cmdtest.svg?branch=master)](https://travis-ci.org/google/go-cmdtest) 2 | [![godoc](https://godoc.org/github.com/google/go-cmdtest?status.svg)](https://godoc.org/github.com/google/go-cmdtest) 3 | [![Go Report Card](https://goreportcard.com/badge/github.com/google/go-cmdtest)](https://goreportcard.com/report/github.com/google/go-cmdtest) 4 | 5 | # Testing your CLI 6 | 7 | The cmdtest package simplifies testing of command-line interfaces. It provides a 8 | simple, cross-platform, shell-like language to express command execution. It can 9 | compare actual output with the expected output, and can also update a file with 10 | new "golden" output that is deemed correct. 11 | 12 | ## Test files 13 | 14 | Start using cmdtest by writing a test file with the extension `.ct`. The test 15 | file will consist of commands prefixed by `$` and expected output following each 16 | command. Lines starting with `#` are comments. Example: 17 | 18 | ``` 19 | # Testing for my-cli. 20 | 21 | # The "help" command. 22 | $ my-cli help 23 | my-cli is a CLI, and this is its help. 24 | 25 | # Verify that an invalid command fails and prints a useful error. 26 | $ my-cli invalidcmd --> FAIL 27 | Error: unknown command "invalidcmd". 28 | ``` 29 | 30 | You can leave the expected output out and let `cmdtest` fill it in for you using 31 | `update` mode (see below). 32 | 33 | More details on test file format: 34 | 35 | * Before the first line starting with a `$`, empty lines and lines beginning 36 | with "#" are ignored. 37 | * A sequence of consecutive lines starting with `$` begin a test case. These 38 | lines are commands to execute. See below for the valid commands. 39 | * Lines following the `$` lines are command output (merged stdout and stderr). 40 | Output is always treated literally. 41 | * After the command output there should be a blank line. Between that blank 42 | line and the next `$` line, empty lines and lines beginning with `#` are 43 | ignored. (Because of these rules, cmdtest cannot distinguish trailing blank 44 | lines in the output.) 45 | * Syntax of a line beginning with `$`: 46 | * A sequence of space-separated words (no quoting is supported). The first 47 | word is the command, the rest are its args. If the next-to-last word is 48 | `<`, the last word is interpreted as a file and becomes the standard 49 | input to the command. None of the built-in commands (see below) support 50 | input redirection, but commands defined with Program do. 51 | * By default, commands are expected to succeed, and the test will fail 52 | otherwise. However, commands that are expected to fail can be marked with a 53 | `--> FAIL` suffix. 54 | 55 | All test files in the same directory make up a test suite. See the TestSuite 56 | documentation for the syntax of test files, and the `testdata/` directory for 57 | examples. 58 | 59 | ## Commands 60 | 61 | `cmdtest` comes with the following built-in commands: 62 | 63 | * cd DIR 64 | * cat FILE 65 | * mkdir DIR 66 | * setenv VAR VALUE 67 | * echo ARG1 ARG2 ... 68 | * fecho FILE ARG1 ARG2 ... 69 | 70 | These all have their usual Unix shell meaning, except for `fecho`, which writes 71 | its arguments to a file (output redirection is not supported). All file and 72 | directory arguments must refer to the current directory; that is, they cannot 73 | contain slashes. 74 | 75 | You can add your own custom commands by adding them to the `TestSuite.Commands` 76 | map; keep reading for an example. 77 | 78 | ## Variable substitution 79 | 80 | `cmdtest` does its own environment variable substitution, using the syntax 81 | `${VAR}`. Test execution inherits the full environment of the test binary caller 82 | (typically, your shell). The environment variable `ROOTDIR` is set to the 83 | temporary directory created to run the test file (except in parallel mode; see 84 | below). 85 | 86 | ## Running the tests 87 | 88 | To test, first read the suite: 89 | 90 | ```go 91 | ts, err := cmdtest.Read("testdata") 92 | ``` 93 | 94 | Next, configure the resulting `TestSuite` by adding a `Setup` function and/or 95 | adding commands to the `Commands` map. In particular, you will want to add a 96 | command for your CLI. There are two ways to do this: you can run your CLI binary 97 | directly from from inside the test binary process, or you can build the CLI 98 | binary and have the test binary run it as a sub-process. 99 | 100 | ### Invoking your CLI in-process 101 | 102 | To run your CLI from inside the test binary, you will have to prevent it from 103 | calling `os.Exit`. You may be able to refactor your `main` function like this: 104 | 105 | ```go 106 | func main() { 107 | os.Exit(run()) 108 | } 109 | 110 | func run() int { 111 | // Your previous main here, returning 0 for success. 112 | } 113 | ``` 114 | 115 | Then, add the command for your CLI to the `TestSuite`: 116 | 117 | ```go 118 | ts.Commands["my-cli"] = cmdtest.InProcessProgram("my-cli", run) 119 | ``` 120 | 121 | ### Invoking your CLI out-of-process 122 | 123 | You can also run your CLI as an ordinary program, if you build it first. 124 | You can do this outside of your test, or inside with code like 125 | 126 | ```go 127 | if err := exec.Command("go", "build", ".").Run(); err != nil { 128 | t.Fatal(err) 129 | } 130 | defer os.Remove("my-cli") 131 | ``` 132 | 133 | Then add the command for your CLI to the `TestSuite`: 134 | 135 | ```go 136 | ts.Commands["my-cli"] = cmdtest.Program("my-cli") 137 | ``` 138 | 139 | ## Running the test 140 | 141 | Finally, call `TestSuite.Run` with `false` to compare the expected output to the 142 | actual output, or `true` to update the expected output. Typically, this boolean 143 | will be the value of a flag. So, your final test code will look something like: 144 | 145 | ```go 146 | var update = flag.Bool("update", false, "update test files with results") 147 | 148 | func TestCLI(t *testing.T) { 149 | ts, err := cmdtest.Read("testdata") 150 | if err != nil { 151 | t.Fatal(err) 152 | } 153 | ts.Commands["my-cli"] = cmdtest.InProcessProgram("my-cli", run) 154 | ts.Run(t, *update) 155 | } 156 | ``` 157 | 158 | ## Parallel mode 159 | 160 | If you call `ts.RunParallel` instead of `ts.Run`, each file in the suite is run 161 | in parallel with the others. (The cases in a single file are still run 162 | sequentially, however.) In this mode, no temporary directories are created 163 | and `ROOTDIR` is not set. 164 | -------------------------------------------------------------------------------- /cmdtest.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 The Go Cloud Development Kit Authors 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // https://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // The cmdtest package simplifies testing of command-line interfaces. It 16 | // provides a simple, cross-platform, shell-like language to express command 17 | // execution. It can compare actual output with the expected output, and can 18 | // also update a file with new "golden" output that is deemed correct. 19 | // 20 | // Start using cmdtest by writing a test file with commands and expected output, 21 | // giving it the extension ".ct". All test files in the same directory make up a 22 | // test suite. See the TestSuite documentation for the syntax of test files. 23 | // 24 | // To test, first read the suite: 25 | // 26 | // ts, err := cmdtest.Read("testdata") 27 | // 28 | // Then configure the resulting TestSuite by adding commands or enabling 29 | // debugging features. Lastly, call TestSuite.Run with false to compare 30 | // or true to update. Typically, this boolean will be the value of a flag: 31 | // 32 | // var update = flag.Bool("update", false, "update test files with results") 33 | // ... 34 | // ts.Run(t, *update) 35 | package cmdtest 36 | 37 | import ( 38 | "bufio" 39 | "bytes" 40 | "errors" 41 | "fmt" 42 | "io" 43 | "io/ioutil" 44 | "os" 45 | "os/exec" 46 | "path/filepath" 47 | "regexp" 48 | "strconv" 49 | "strings" 50 | "syscall" 51 | "testing" 52 | 53 | "github.com/google/go-cmp/cmp" 54 | ) 55 | 56 | // A TestSuite contains a set of test files, each of which may contain multiple 57 | // test cases. Use Read to build a TestSuite from all the test files in a 58 | // directory. Then configure it and call Run. 59 | // 60 | // Format of a test file: 61 | // 62 | // Before the first line starting with a '$', empty lines and lines beginning with 63 | // "#" are ignored. 64 | // 65 | // A sequence of consecutive lines starting with '$' begins a test case. These lines 66 | // are commands to execute. See below for the valid commands. 67 | // 68 | // Lines following the '$' lines are command output (merged stdout and stderr). 69 | // Output is always treated literally. After the command output there should be a 70 | // blank line. Between that blank line and the next '$' line, empty lines and lines 71 | // beginning with '#' are ignored. (Because of these rules, cmdtest cannot 72 | // distinguish trailing blank lines in the output.) 73 | // 74 | // Syntax of a line beginning with '$': A sequence of space-separated words (no 75 | // quoting is supported). The first word is the command, the rest are its args. 76 | // If the next-to-last word is '<', the last word is interpreted as a file and 77 | // becomes the standard input to the command. None of the built-in commands (see 78 | // below) support input redirection, but commands defined with Program do. 79 | // 80 | // By default, commands are expected to succeed, and the test will fail 81 | // otherwise. However, commands that are expected to fail can be marked 82 | // with a " --> FAIL" suffix. The word FAIL may optionally be followed 83 | // by a non-zero integer specifying the expected exit code. 84 | // 85 | // The cases of a test file are executed in order, starting in a freshly created 86 | // temporary directory. Execution of a file stops with the first case that 87 | // doesn't behave as expected, but other files in the suite will still run. 88 | // 89 | // The built-in commands (initial contents of the Commands map) are: 90 | // 91 | // cd DIR 92 | // cat FILE 93 | // mkdir DIR 94 | // setenv VAR VALUE 95 | // echo ARG1 ARG2 ... 96 | // fecho FILE ARG1 ARG2 ... 97 | // 98 | // These all have their usual Unix shell meaning, except for fecho, which writes its 99 | // arguments to a file (output redirection is not supported). All file and directory 100 | // arguments must refer to the current directory; that is, they cannot contain 101 | // slashes. 102 | // 103 | // cmdtest does its own environment variable substitution, using the syntax 104 | // "${VAR}". Test execution inherits the full environment of the test binary 105 | // caller (typically, your shell). The environment variable ROOTDIR is set to 106 | // the temporary directory created to run the test file. 107 | type TestSuite struct { 108 | // If non-nil, this function is called for each test. It is passed the root 109 | // directory after it has been made the current directory. 110 | Setup func(string) error 111 | 112 | // The commands that can be executed (that is, whose names can occur as the 113 | // first word of a command line). 114 | Commands map[string]CommandFunc 115 | 116 | // If true, don't delete the temporary root directories for each test file, 117 | // and print out their names for debugging. 118 | KeepRootDirs bool 119 | 120 | // If true, don't log while comparing. 121 | DisableLogging bool 122 | 123 | files []*testFile 124 | } 125 | 126 | type testFile struct { 127 | suite *TestSuite 128 | filename string // full filename of the test file 129 | cases []*testCase 130 | suffix []string // non-output lines after last case 131 | } 132 | 133 | type testCase struct { 134 | before []string // lines before the commands 135 | startLine int // line of first command 136 | // The list of commands to execute. 137 | commands []string 138 | 139 | // The stdout and stderr, merged and split into lines. 140 | gotOutput []string // from execution 141 | wantOutput []string // from file 142 | } 143 | 144 | // CommandFunc is the signature of a command function. The function takes the 145 | // subsequent words on the command line (so that arg[0] is the first argument), 146 | // as well as the name of a file to use for input redirection. It returns the 147 | // command's output. 148 | type CommandFunc func(args []string, inputFile string) ([]byte, error) 149 | 150 | // ExitCodeErr is an error that a CommandFunc can return to provide an exit 151 | // code. Tests can check the code by writing the desired value after "--> FAIL". 152 | // 153 | // ExitCodeErr is only necessary when writing commands that don't return errors 154 | // that come from the OS. Commands that return the error from os/exec.Cmd.Run 155 | // or functions in the os package like Chdir and Mkdir don't need to use this, 156 | // because those errors already contain error codes. 157 | type ExitCodeErr struct { 158 | Msg string 159 | Code int 160 | } 161 | 162 | func (e *ExitCodeErr) Error() string { 163 | return fmt.Sprintf("%s (code %d)", e.Msg, e.Code) 164 | } 165 | 166 | // Read reads all the files in dir with extension ".ct" and returns a TestSuite 167 | // containing them. See the TestSuite documentation for syntax. 168 | func Read(dir string) (*TestSuite, error) { 169 | filenames, err := filepath.Glob(filepath.Join(dir, "*.ct")) 170 | if err != nil { 171 | return nil, err 172 | } 173 | ts := &TestSuite{ 174 | Commands: map[string]CommandFunc{ 175 | "cat": fixedArgBuiltin(1, catCmd), 176 | "cd": fixedArgBuiltin(1, cdCmd), 177 | "echo": echoCmd, 178 | "fecho": fechoCmd, 179 | "mkdir": fixedArgBuiltin(1, mkdirCmd), 180 | "setenv": fixedArgBuiltin(2, setenvCmd), 181 | }, 182 | } 183 | for _, fn := range filenames { 184 | tf, err := readFile(fn) 185 | if err != nil { 186 | return nil, err 187 | } 188 | tf.suite = ts 189 | ts.files = append(ts.files, tf) 190 | } 191 | return ts, nil 192 | } 193 | 194 | func readFile(filename string) (*testFile, error) { 195 | // parse states 196 | const ( 197 | beforeFirstCommand = iota 198 | inCommands 199 | inOutput 200 | ) 201 | 202 | tf := &testFile{ 203 | filename: filename, 204 | } 205 | f, err := os.Open(filename) 206 | if err != nil { 207 | return nil, err 208 | } 209 | defer f.Close() 210 | 211 | scanner := bufio.NewScanner(f) 212 | var tc *testCase 213 | lineno := 0 214 | var prefix []string 215 | state := beforeFirstCommand 216 | for scanner.Scan() { 217 | lineno++ 218 | line := scanner.Text() 219 | isCommand := strings.HasPrefix(line, "$") 220 | switch state { 221 | case beforeFirstCommand: 222 | if isCommand { 223 | tc = &testCase{startLine: lineno, before: prefix} 224 | tc.addCommandLine(line) 225 | state = inCommands 226 | } else { 227 | line = strings.TrimSpace(line) 228 | if line == "" || line[0] == '#' { 229 | prefix = append(prefix, line) 230 | } else { 231 | return nil, fmt.Errorf("%s:%d: bad line %q (should begin with '#')", filename, lineno, line) 232 | } 233 | } 234 | 235 | case inCommands: 236 | if isCommand { 237 | tc.addCommandLine(line) 238 | } else { // End of commands marks the start of the output. 239 | tc.wantOutput = append(tc.wantOutput, line) 240 | state = inOutput 241 | } 242 | 243 | case inOutput: 244 | if isCommand { // A command marks the end of the output. 245 | prefix = tf.addCase(tc) 246 | tc = &testCase{startLine: lineno, before: prefix} 247 | tc.addCommandLine(line) 248 | state = inCommands 249 | } else { 250 | tc.wantOutput = append(tc.wantOutput, line) 251 | } 252 | default: 253 | panic("bad state") 254 | } 255 | } 256 | if err := scanner.Err(); err != nil { 257 | return nil, err 258 | } 259 | if tc != nil { 260 | tf.suffix = tf.addCase(tc) 261 | } 262 | return tf, nil 263 | } 264 | 265 | func (tc *testCase) addCommandLine(line string) { 266 | tc.commands = append(tc.commands, strings.TrimSpace(line[1:])) 267 | } 268 | 269 | // addCase first splits the collected output for tc into the actual command 270 | // output, and a suffix consisting of blank lines and comments. It then adds tc 271 | // to the cases of tf, and returns the suffix. 272 | func (tf *testFile) addCase(tc *testCase) []string { 273 | // Trim the suffix of output that consists solely of blank lines and comments, 274 | // and return it. 275 | var i int 276 | for i = len(tc.wantOutput) - 1; i >= 0; i-- { 277 | if tc.wantOutput[i] != "" && tc.wantOutput[i][0] != '#' { 278 | break 279 | } 280 | } 281 | i++ 282 | // i is the index of the first line to ignore. 283 | keep, suffix := tc.wantOutput[:i], tc.wantOutput[i:] 284 | if len(keep) == 0 { 285 | keep = nil 286 | } 287 | tc.wantOutput = keep 288 | tf.cases = append(tf.cases, tc) 289 | return suffix 290 | } 291 | 292 | // Run runs the commands in each file in the test suite. Each file runs in a 293 | // separate subtest. 294 | // 295 | // If update is false, it compares their output with the output in the file, 296 | // line by line. 297 | // 298 | // If update is true, it writes the output back to the file, overwriting the 299 | // previous output. 300 | // 301 | // Before comparing/updating, occurrences of the root directory in the output 302 | // are replaced by ${ROOTDIR}. 303 | func (ts *TestSuite) Run(t *testing.T, update bool) { 304 | ts.run(t, update, false) 305 | } 306 | 307 | // RunParallel is like Run, but runs the tests in parallel using t.Parallel. 308 | // 309 | // Unlike Run, tests are not run in temporary directories, and ROOTDIR is 310 | // neither set nor replaced. 311 | func (ts *TestSuite) RunParallel(t *testing.T, update bool) { 312 | ts.run(t, update, true) 313 | } 314 | 315 | func (ts *TestSuite) run(t *testing.T, update, parallel bool) { 316 | if update { 317 | ts.update(t, parallel) 318 | } else { 319 | ts.compare(t, parallel) 320 | } 321 | } 322 | 323 | // compare runs a subtest for each file in the test suite. See Run. 324 | func (ts *TestSuite) compare(t *testing.T, parallel bool) { 325 | log := t.Logf 326 | if ts.DisableLogging { 327 | log = noopLogger 328 | } 329 | for _, tf := range ts.files { 330 | tf := tf 331 | t.Run(strings.TrimSuffix(tf.filename, ".ct"), func(t *testing.T) { 332 | if parallel { 333 | t.Parallel() 334 | } 335 | if s := tf.compare(log, parallel); s != "" { 336 | t.Error(s) 337 | } 338 | }) 339 | } 340 | } 341 | 342 | var noopLogger = func(_ string, _ ...interface{}) {} 343 | 344 | func (tf *testFile) compare(log func(string, ...interface{}), parallel bool) string { 345 | if err := tf.execute(log, parallel); err != nil { 346 | return fmt.Sprintf("%v", err) 347 | } 348 | buf := new(bytes.Buffer) 349 | for _, c := range tf.cases { 350 | if diff := cmp.Diff(c.wantOutput, c.gotOutput); diff != "" { 351 | fmt.Fprintf(buf, "%s:%d: want=-, got=+\n", tf.filename, c.startLine) 352 | c.writeCommands(buf) 353 | fmt.Fprintf(buf, "%s\n", diff) 354 | } 355 | } 356 | return buf.String() 357 | } 358 | 359 | // update runs a subtest for each file in the test suite, updating their output. 360 | // See Run. 361 | func (ts *TestSuite) update(t *testing.T, parallel bool) { 362 | for _, tf := range ts.files { 363 | t.Run(strings.TrimSuffix(tf.filename, ".ct"), func(t *testing.T) { 364 | if parallel { 365 | t.Parallel() 366 | } 367 | tmpfile, err := tf.updateToTemp(parallel) 368 | if tmpfile != nil { 369 | defer func() { 370 | if err := tmpfile.Cleanup(); err != nil { 371 | t.Fatal(err) 372 | } 373 | }() 374 | } 375 | if err != nil { 376 | t.Fatal(err) 377 | } 378 | if err := tmpfile.CloseAtomicallyReplace(); err != nil { 379 | t.Fatal(err) 380 | } 381 | }) 382 | } 383 | } 384 | 385 | // updateToTemp executes tf and writes the output to a temporary file. 386 | // It returns the temporary file. 387 | func (tf *testFile) updateToTemp(parallel bool) (f tempFile, err error) { 388 | if err := tf.execute(noopLogger, parallel); err != nil { 389 | return nil, err 390 | } 391 | if f, err = createTempFile(tf.filename); err != nil { 392 | return nil, err 393 | } 394 | if err := tf.write(f); err != nil { 395 | // Return f in order to clean it up outside this function. 396 | return f, err 397 | } 398 | return f, nil 399 | } 400 | 401 | func (tf *testFile) execute(log func(string, ...interface{}), parallel bool) error { 402 | var rootDir string 403 | if !parallel { 404 | var err error 405 | rootDir, err = ioutil.TempDir("", "cmdtest") 406 | if err != nil { 407 | return fmt.Errorf("%s: %v", tf.filename, err) 408 | } 409 | if tf.suite.KeepRootDirs { 410 | fmt.Printf("%s: test root directory: %s\n", tf.filename, rootDir) 411 | } else { 412 | defer os.RemoveAll(rootDir) 413 | } 414 | 415 | if err := os.Setenv("ROOTDIR", rootDir); err != nil { 416 | return fmt.Errorf("%s: %v", tf.filename, err) 417 | } 418 | defer os.Unsetenv("ROOTDIR") 419 | cwd, err := os.Getwd() 420 | if err != nil { 421 | return fmt.Errorf("%s: %v", tf.filename, err) 422 | } 423 | 424 | if err := os.Chdir(rootDir); err != nil { 425 | return fmt.Errorf("%s: %v", tf.filename, err) 426 | } 427 | defer func() { _ = os.Chdir(cwd) }() 428 | } 429 | 430 | if tf.suite.Setup != nil { 431 | if err := tf.suite.Setup(rootDir); err != nil { 432 | return fmt.Errorf("%s: calling Setup: %v", tf.filename, err) 433 | } 434 | } 435 | for _, tc := range tf.cases { 436 | if err := tc.execute(tf.suite, log, parallel); err != nil { 437 | return fmt.Errorf("%s:%v", tf.filename, err) // no space after :, for line number 438 | } 439 | } 440 | return nil 441 | } 442 | 443 | // Run the test case by executing the commands. The concatenated output from all commands 444 | // is saved in tc.gotOutput. 445 | // An error is returned if any of the following occur: 446 | // - A command that should succeed instead failed. 447 | // - A command that should fail instead succeeded. 448 | // - A command that should fail with a particular error code instead failed 449 | // with a different one. 450 | // - A built-in command was called incorrectly. 451 | func (tc *testCase) execute(ts *TestSuite, log func(string, ...interface{}), parallel bool) error { 452 | tc.gotOutput = nil 453 | var allout []byte 454 | for i, cmd := range tc.commands { 455 | cmd, wantFail, wantExitCode, err := parseCommand(cmd) 456 | if err != nil { 457 | return err 458 | } 459 | _ = wantExitCode 460 | args := strings.Fields(cmd) 461 | for i := range args { 462 | args[i], err = expandVariables(args[i], os.LookupEnv) 463 | if err != nil { 464 | return err 465 | } 466 | } 467 | log("$ %s", strings.Join(args, " ")) 468 | name := args[0] 469 | args = args[1:] 470 | var infile string 471 | if len(args) >= 2 && args[len(args)-2] == "<" { 472 | infile = args[len(args)-1] 473 | args = args[:len(args)-2] 474 | } 475 | f := ts.Commands[name] 476 | if f == nil { 477 | return fmt.Errorf("%d: no such command %q", tc.startLine+i, name) 478 | } 479 | out, err := f(args, infile) 480 | log("%s\n", string(out)) 481 | allout = append(allout, out...) 482 | line := tc.startLine + i 483 | if err == nil && wantFail { 484 | return fmt.Errorf("%d: %q succeeded, but it was expected to fail", line, cmd) 485 | } 486 | if err != nil && !wantFail { 487 | return fmt.Errorf("%d: %q failed with %v", line, cmd, err) 488 | } 489 | if err != nil && wantFail && wantExitCode != 0 { 490 | gotExitCode, ok := extractExitCode(err) 491 | if !ok { 492 | return fmt.Errorf("%d: %q failed without an exit code, but one was expected", line, cmd) 493 | } 494 | if gotExitCode != wantExitCode { 495 | return fmt.Errorf("%d: %q failed with exit code %d, but %d was expected", 496 | line, cmd, gotExitCode, wantExitCode) 497 | } 498 | } 499 | } 500 | if len(allout) > 0 { 501 | if !parallel { 502 | allout = scrub(os.Getenv("ROOTDIR"), allout) // use Getenv because Setup could change ROOTDIR 503 | } 504 | // Remove final whitespace. 505 | s := strings.TrimRight(string(allout), " \t\n") 506 | tc.gotOutput = strings.Split(s, "\n") 507 | } 508 | return nil 509 | } 510 | 511 | func parseCommand(cmdline string) (cmd string, wantFail bool, wantExitCode int, err error) { 512 | const failMarker = " --> FAIL" 513 | i := strings.LastIndex(cmdline, failMarker) 514 | if i < 0 { 515 | return cmdline, false, 0, nil 516 | } 517 | cmd = cmdline[:i] 518 | wantFail = true 519 | rest := strings.TrimSpace(cmdline[i+len(failMarker):]) 520 | if len(rest) > 0 { 521 | wantExitCode, err = strconv.Atoi(rest) 522 | if err != nil { 523 | return "", false, 0, err 524 | } 525 | if wantExitCode == 0 { 526 | return "", false, 0, errors.New("cannot use 0 as a FAIL exit code") 527 | } 528 | } 529 | return cmd, wantFail, wantExitCode, nil 530 | } 531 | 532 | // extractExitCode extracts an exit code from err and returns it and true. 533 | // If one can't be found, the second return value is false. 534 | func extractExitCode(err error) (code int, ok bool) { 535 | var ( 536 | errno syscall.Errno 537 | ee *exec.ExitError 538 | ece *ExitCodeErr 539 | ) 540 | switch { 541 | case errors.As(err, &errno): 542 | return int(errno), true 543 | case errors.As(err, &ee): 544 | return ee.ExitCode(), true 545 | case errors.As(err, &ece): 546 | return ece.Code, true 547 | default: 548 | return 0, false 549 | } 550 | } 551 | 552 | // Program defines a command function that will run the executable at path using 553 | // the exec.Command package and return its combined output. If path is relative, 554 | // it is converted to an absolute path using the current directory at the time 555 | // Program is called. 556 | // 557 | // In the unlikely event that Program cannot obtain the current directory, it 558 | // panics. 559 | func Program(path string) CommandFunc { 560 | abspath, err := filepath.Abs(path) 561 | if err != nil { 562 | panic(fmt.Sprintf("Program(%q): %v", path, err)) 563 | } 564 | return func(args []string, inputFile string) ([]byte, error) { 565 | return execute(abspath, args, inputFile) 566 | } 567 | } 568 | 569 | // InProcessProgram defines a command function that will invoke f, which must 570 | // behave like an actual main function except that it returns an error code 571 | // instead of calling os.Exit. 572 | // Before calling f: 573 | // 574 | // - os.Args is set to the concatenation of name and args. 575 | // - If inputFile is non-empty, it is redirected to standard input. 576 | // - Standard output and standard error are redirected to a buffer, which is 577 | // returned. 578 | func InProcessProgram(name string, f func() int) CommandFunc { 579 | return func(args []string, inputFile string) ([]byte, error) { 580 | origArgs := os.Args 581 | origOut := os.Stdout 582 | origErr := os.Stderr 583 | defer func() { 584 | os.Args = origArgs 585 | os.Stdout = origOut 586 | os.Stderr = origErr 587 | }() 588 | os.Args = append([]string{name}, args...) 589 | // Redirect stdout and stderr to a pipe. 590 | pr, pw, err := os.Pipe() 591 | if err != nil { 592 | return nil, err 593 | } 594 | os.Stdout = pw 595 | os.Stderr = pw 596 | // Copy both stdout and stderr to the same buffer. 597 | buf := &bytes.Buffer{} 598 | errc := make(chan error, 1) 599 | go func() { 600 | _, err := io.Copy(buf, pr) 601 | errc <- err 602 | }() 603 | 604 | // Redirect stdin if needed. 605 | if inputFile != "" { 606 | f, err := os.Open(inputFile) 607 | if err != nil { 608 | return nil, err 609 | } 610 | defer f.Close() 611 | origIn := os.Stdin 612 | defer func() { os.Stdin = origIn }() 613 | os.Stdin = f 614 | } 615 | 616 | res := f() 617 | if err := pw.Close(); err != nil { 618 | return nil, err 619 | } 620 | // Wait for pipe copying to finish. 621 | if err := <-errc; err != nil { 622 | return nil, err 623 | } 624 | if res != 0 { 625 | err = &ExitCodeErr{ 626 | Msg: fmt.Sprintf("%s failed", name), 627 | Code: res, 628 | } 629 | } 630 | return buf.Bytes(), err 631 | } 632 | } 633 | 634 | // execute uses exec.Command to run the named program with the given args. The 635 | // combined output is captured and returned. If infile is not empty, its contents 636 | // become the command's standard input. 637 | func execute(name string, args []string, infile string) ([]byte, error) { 638 | ecmd := exec.Command(name, args...) 639 | if infile != "" { 640 | f, err := os.Open(infile) 641 | if err != nil { 642 | return nil, err 643 | } 644 | defer f.Close() 645 | ecmd.Stdin = f 646 | } 647 | out, err := ecmd.CombinedOutput() 648 | if err != nil { 649 | return out, err 650 | } 651 | return out, nil 652 | } 653 | 654 | var varRegexp = regexp.MustCompile(`\$\{([^${}]+)\}`) 655 | 656 | // expandVariables replaces variable references in s with their values. A reference 657 | // to a variable V looks like "${V}". 658 | // lookup is called on a variable's name to find its value. Its second return value 659 | // is false if the variable doesn't exist. 660 | // expandVariables fails if s contains a reference to a non-existent variable. 661 | // 662 | // This function differs from os.Expand in two ways. First, it does not expand $var, 663 | // only ${var}. The former is fragile. Second, an undefined variable results in an error, 664 | // rather than expanding to some string. We want to fail if a variable is undefined. 665 | func expandVariables(s string, lookup func(string) (string, bool)) (string, error) { 666 | var sb strings.Builder 667 | for { 668 | ixs := varRegexp.FindStringSubmatchIndex(s) 669 | if ixs == nil { 670 | sb.WriteString(s) 671 | return sb.String(), nil 672 | } 673 | varName := s[ixs[2]:ixs[3]] 674 | varVal, ok := lookup(varName) 675 | if !ok { 676 | return "", fmt.Errorf("variable %q not found", varName) 677 | } 678 | sb.WriteString(s[:ixs[0]]) 679 | sb.WriteString(varVal) 680 | s = s[ixs[1]:] 681 | } 682 | } 683 | 684 | // scrub removes dynamic content from output. 685 | func scrub(rootDir string, b []byte) []byte { 686 | const scrubbedRootDir = "${ROOTDIR}" 687 | const sep = string(filepath.Separator) 688 | rootDirWithSeparator := rootDir + sep 689 | scrubbedRootDirWithSeparator := scrubbedRootDir + sep 690 | b = bytes.Replace(b, []byte(rootDirWithSeparator), []byte(scrubbedRootDirWithSeparator), -1) 691 | b = bytes.Replace(b, []byte(rootDir), []byte(scrubbedRootDir), -1) 692 | return b 693 | } 694 | 695 | func (tf *testFile) write(w io.Writer) error { 696 | for _, c := range tf.cases { 697 | if err := c.write(w); err != nil { 698 | return err 699 | } 700 | } 701 | return writeLines(w, tf.suffix) 702 | } 703 | 704 | func (tc *testCase) write(w io.Writer) error { 705 | if err := writeLines(w, tc.before); err != nil { 706 | return err 707 | } 708 | if err := tc.writeCommands(w); err != nil { 709 | return err 710 | } 711 | out := tc.gotOutput 712 | if out == nil { 713 | out = tc.wantOutput 714 | } 715 | return writeLines(w, out) 716 | } 717 | 718 | func (tc *testCase) writeCommands(w io.Writer) error { 719 | for _, c := range tc.commands { 720 | if _, err := fmt.Fprintf(w, "$ %s\n", c); err != nil { 721 | return err 722 | } 723 | } 724 | return nil 725 | } 726 | 727 | func writeLines(w io.Writer, lines []string) error { 728 | for _, l := range lines { 729 | if _, err := io.WriteString(w, l); err != nil { 730 | return err 731 | } 732 | if _, err := w.Write([]byte{'\n'}); err != nil { 733 | return err 734 | } 735 | } 736 | return nil 737 | } 738 | 739 | func fixedArgBuiltin(nargs int, f func([]string) ([]byte, error)) CommandFunc { 740 | return func(args []string, inputFile string) ([]byte, error) { 741 | if len(args) != nargs { 742 | return nil, fmt.Errorf("need exactly %d arguments", nargs) 743 | } 744 | if inputFile != "" { 745 | return nil, errors.New("input redirection not supported") 746 | } 747 | return f(args) 748 | } 749 | } 750 | 751 | // cd DIR 752 | // change directory 753 | func cdCmd(args []string) ([]byte, error) { 754 | if err := checkPath(args[0]); err != nil { 755 | return nil, err 756 | } 757 | cwd, err := os.Getwd() 758 | if err != nil { 759 | return nil, err 760 | } 761 | return nil, os.Chdir(filepath.Join(cwd, args[0])) 762 | } 763 | 764 | // echo ARG1 ARG2 ... 765 | // write args to stdout 766 | // 767 | // \n is added at the end of the input. 768 | // Also, literal "\n" in the input will be replaced by \n. 769 | func echoCmd(args []string, inputFile string) ([]byte, error) { 770 | if inputFile != "" { 771 | return nil, errors.New("input redirection not supported") 772 | } 773 | s := strings.Join(args, " ") 774 | s = strings.Replace(s, "\\n", "\n", -1) 775 | s += "\n" 776 | return []byte(s), nil 777 | } 778 | 779 | // fecho FILE ARG1 ARG2 ... 780 | // write args to FILE 781 | // 782 | // \n is added at the end of the input. 783 | // Also, literal "\n" in the input will be replaced by \n. 784 | func fechoCmd(args []string, inputFile string) ([]byte, error) { 785 | if len(args) < 1 { 786 | return nil, errors.New("need at least 1 argument") 787 | } 788 | if inputFile != "" { 789 | return nil, errors.New("input redirection not supported") 790 | } 791 | if err := checkPath(args[0]); err != nil { 792 | return nil, err 793 | } 794 | s := strings.Join(args[1:], " ") 795 | s = strings.Replace(s, "\\n", "\n", -1) 796 | s += "\n" 797 | return nil, ioutil.WriteFile(args[0], []byte(s), 0600) 798 | } 799 | 800 | // cat FILE 801 | // copy file to stdout 802 | func catCmd(args []string) ([]byte, error) { 803 | if err := checkPath(args[0]); err != nil { 804 | return nil, err 805 | } 806 | f, err := os.Open(args[0]) 807 | if err != nil { 808 | return nil, err 809 | } 810 | defer f.Close() 811 | buf := &bytes.Buffer{} 812 | _, err = io.Copy(buf, f) 813 | if err != nil { 814 | return nil, err 815 | } 816 | return buf.Bytes(), nil 817 | } 818 | 819 | // mkdir DIR 820 | // create directory 821 | func mkdirCmd(args []string) ([]byte, error) { 822 | if err := checkPath(args[0]); err != nil { 823 | return nil, err 824 | } 825 | return nil, os.Mkdir(args[0], 0700) 826 | } 827 | 828 | // setenv VAR VALUE 829 | // set environment variable 830 | func setenvCmd(args []string) ([]byte, error) { 831 | return nil, os.Setenv(args[0], args[1]) 832 | } 833 | 834 | func checkPath(path string) error { 835 | if strings.ContainsRune(path, '/') || strings.ContainsRune(path, '\\') { 836 | return fmt.Errorf("argument must be in the current directory (%q has a '/')", path) 837 | } 838 | return nil 839 | } 840 | 841 | // tempFile represents a temporary file. 842 | type tempFile interface { 843 | io.Writer 844 | Name() string 845 | 846 | // Close and remove the file. 847 | Cleanup() error 848 | 849 | // Close the file and replace the destination file with it. 850 | CloseAtomicallyReplace() error 851 | } 852 | -------------------------------------------------------------------------------- /cmdtest_test.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 The Go Cloud Development Kit Authors 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // https://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | package cmdtest 16 | 17 | import ( 18 | "errors" 19 | "fmt" 20 | "io" 21 | "io/ioutil" 22 | "log" 23 | "os" 24 | "os/exec" 25 | "path/filepath" 26 | "regexp" 27 | "runtime" 28 | "strings" 29 | "sync" 30 | "testing" 31 | 32 | "github.com/google/go-cmp/cmp" 33 | "github.com/google/renameio" 34 | ) 35 | 36 | var once sync.Once 37 | 38 | func setup() { 39 | // Build echo-stdin, the little program needed to test input redirection. 40 | if err := exec.Command("go", "build", "testdata/echo-stdin.go").Run(); err != nil { 41 | log.Fatalf("building echo-stdin: %v", err) 42 | } 43 | } 44 | 45 | // echoStdin contains the same code as the main function of 46 | // testdata/echo-stdin.go, except that it returns the exit code instead of 47 | // calling os.Exit. It is for testing InProcessProgram. 48 | func echoStdin() int { 49 | fmt.Println("Here is stdin:") 50 | _, err := io.Copy(os.Stdout, os.Stdin) 51 | if err != nil { 52 | fmt.Fprintf(os.Stderr, "failed: %v\n", err) 53 | return 1 54 | } 55 | return 0 56 | } 57 | 58 | func TestMain(m *testing.M) { 59 | ret := m.Run() 60 | // Clean up the echo-stdin binary if we can. (No big deal if we can't.) 61 | cwd, err := os.Getwd() 62 | if err == nil { 63 | name := "echo-stdin" 64 | if runtime.GOOS == "windows" { 65 | name += ".exe" 66 | } 67 | _ = os.Remove(filepath.Join(cwd, name)) 68 | } 69 | os.Exit(ret) 70 | } 71 | 72 | func TestRead(t *testing.T) { 73 | got, err := Read("testdata/read") 74 | if err != nil { 75 | t.Fatal(err) 76 | } 77 | got.Commands = nil 78 | got.files[0].suite = nil 79 | want := &TestSuite{ 80 | files: []*testFile{ 81 | { 82 | filename: filepath.Join("testdata", "read", "read.ct"), 83 | cases: []*testCase{ 84 | { 85 | before: []string{ 86 | "# A sample test file.", 87 | "", 88 | "# Prefix stuff.", 89 | "", 90 | }, 91 | startLine: 5, 92 | commands: []string{"command arg1 arg2", "cmd2"}, 93 | wantOutput: []string{"out1", "out2"}, 94 | }, 95 | { 96 | before: []string{"", "# start of the next case"}, 97 | startLine: 11, 98 | commands: []string{"c3"}, 99 | wantOutput: nil, 100 | }, 101 | { 102 | before: []string{"", "# start of the third", ""}, 103 | startLine: 15, 104 | commands: []string{"c4 --> FAIL"}, 105 | wantOutput: []string{"out3"}, 106 | }, 107 | { 108 | before: []string{""}, 109 | startLine: 18, 110 | commands: []string{"c5 --> FAIL 2"}, 111 | wantOutput: []string{"out4"}, 112 | }, 113 | }, 114 | suffix: []string{"", "", "# end"}, 115 | }, 116 | }, 117 | } 118 | if diff := cmp.Diff(want, got, cmp.AllowUnexported(TestSuite{}, testFile{}, testCase{})); diff != "" { 119 | t.Error(diff) 120 | } 121 | 122 | } 123 | 124 | // compareReturningError is similar to compare, but it returns 125 | // errors/differences in an error. 126 | func (ts *TestSuite) compareReturningError(parallel bool) error { 127 | var ss []string 128 | for _, tf := range ts.files { 129 | if s := tf.compare(noopLogger, parallel); s != "" { 130 | ss = append(ss, s) 131 | } 132 | } 133 | if len(ss) > 0 { 134 | return errors.New(strings.Join(ss, "\n")) 135 | } 136 | return nil 137 | } 138 | 139 | func TestCompare(t *testing.T) { 140 | once.Do(setup) 141 | ts := mustReadTestSuite(t, "good") 142 | ts.DisableLogging = true 143 | ts.Commands["echo-stdin"] = Program("echo-stdin") 144 | ts.Commands["echoStdin"] = InProcessProgram("echoStdin", echoStdin) 145 | ts.Run(t, false) 146 | 147 | // Test errors. 148 | // Since the output of cmp.Diff is unstable, we search for regexps we expect 149 | // to find there, rather than checking an exact match. 150 | t.Run("bad", func(t *testing.T) { 151 | ts = mustReadTestSuite(t, "bad") 152 | ts.Commands["echo-stdin"] = Program("echo-stdin") 153 | ts.Commands["code17"] = func([]string, string) ([]byte, error) { 154 | return nil, fmt.Errorf("wrapping: %w", &ExitCodeErr{Msg: "failed", Code: 17}) 155 | } 156 | ts.Commands["inprocess99"] = InProcessProgram("inprocess99", func() int { return 99 }) 157 | 158 | err := ts.compareReturningError(false) 159 | if err == nil { 160 | t.Fatal("got nil, want error") 161 | } 162 | got := err.Error() 163 | wants := []string{ 164 | `testdata.bad.bad-output\.ct:\d: want=-, got=+`, 165 | `testdata.bad.bad-output\.ct:\d: want=-, got=+`, 166 | `testdata.bad.bad-fail-1\.ct:\d: "echo" succeeded, but it was expected to fail`, 167 | `testdata.bad.bad-fail-2\.ct:\d: "cd foo" failed with chdir`, 168 | `testdata.bad.bad-fail-3\.ct:\d: "cd foo bar" failed with need exactly`, 169 | `testdata.bad.bad-fail-4\.ct:\d: "cd foo bar" failed without an exit code`, 170 | `testdata.bad.bad-fail-5\.ct:\d: "cd foo" failed with exit code 2, but 3 was expected`, 171 | `testdata.bad.bad-fail-6\.ct:\d: "code17" failed with exit code 17, but 4 was expected`, 172 | `testdata.bad.bad-fail-7\.ct:\d: "inprocess99" failed with exit code 99, but 5 was expected`, 173 | `testdata.bad.bad-fail-8\.ct:\d: "echo-stdin -exit 1" failed with exit code 1, but 6 was expected`, 174 | } 175 | failed := false 176 | _ = failed 177 | for _, w := range wants { 178 | match, err := regexp.MatchString(w, got) 179 | if err != nil { 180 | t.Fatal(err) 181 | } 182 | if !match { 183 | t.Errorf(`output does not match "%s"`, w) 184 | failed = true 185 | } 186 | } 187 | const shouldNotAppear = "should not appear" 188 | if strings.Contains(got, shouldNotAppear) { 189 | t.Errorf("saw %q", shouldNotAppear) 190 | failed = true 191 | } 192 | if failed { 193 | // Log full output to aid debugging. 194 | t.Logf("output:\n%s", got) 195 | } 196 | }) 197 | } 198 | 199 | func TestExpandVariables(t *testing.T) { 200 | lookup := func(name string) (string, bool) { 201 | switch name { 202 | case "A": 203 | return "1", true 204 | case "B_C": 205 | return "234", true 206 | default: 207 | return "", false 208 | } 209 | } 210 | for _, test := range []struct { 211 | in, want string 212 | }{ 213 | {"", ""}, 214 | {"${A}", "1"}, 215 | {"${A}${B_C}", "1234"}, 216 | {" x${A}y ${B_C}z ", " x1y 234z "}, 217 | {" ${A${B_C}", " ${A234"}, 218 | } { 219 | got, err := expandVariables(test.in, lookup) 220 | if err != nil { 221 | t.Errorf("%q: %v", test.in, err) 222 | continue 223 | } 224 | if got != test.want { 225 | t.Errorf("%q: got %q, want %q", test.in, got, test.want) 226 | } 227 | } 228 | 229 | // Unknown variable is an error. 230 | if _, err := expandVariables("x${C}y", lookup); err == nil { 231 | t.Error("got nil, want error") 232 | } 233 | } 234 | 235 | func TestUpdateToTemp(t *testing.T) { 236 | once.Do(setup) 237 | for _, dir := range []string{"good", "good-without-output"} { 238 | ts := mustReadTestSuite(t, dir) 239 | ts.Commands["echo-stdin"] = Program("echo-stdin") 240 | ts.Commands["echoStdin"] = InProcessProgram("echoStdin", echoStdin) 241 | f, err := ts.files[0].updateToTemp(false) 242 | defer f.Cleanup() 243 | if err != nil { 244 | t.Fatal(err) 245 | } 246 | if diff := diffFiles(t, "testdata/good/good.ct", f.Name()); diff != "" { 247 | t.Errorf("%s: %s", dir, diff) 248 | } 249 | } 250 | } 251 | 252 | func TestUpdate(t *testing.T) { 253 | ct := "testdata/update/update.ct" 254 | original, err := ioutil.ReadFile(ct) 255 | if err != nil { 256 | t.Fatal(err) 257 | } 258 | defer func() { 259 | // Restore original file content. 260 | if err := renameio.WriteFile(ct, original, 0644); err != nil { 261 | t.Fatal(err) 262 | } 263 | }() 264 | ts := mustReadTestSuite(t, "update") 265 | ts.update(t, false) 266 | if diff := diffFiles(t, ct, "testdata/update/update.golden"); diff != "" { 267 | t.Errorf(diff) 268 | } 269 | } 270 | 271 | func TestParseCommand(t *testing.T) { 272 | for _, test := range []struct { 273 | cmdline string 274 | wantCmd string 275 | wantFail bool 276 | wantCode int 277 | wantErr bool 278 | }{ 279 | { 280 | cmdline: "ls", 281 | wantCmd: "ls", 282 | }, 283 | { 284 | cmdline: "a b c --> FAIL ", 285 | wantCmd: "a b c", 286 | wantFail: true, 287 | }, 288 | { 289 | cmdline: "a b c --> fail", 290 | wantCmd: "a b c --> fail", 291 | }, 292 | { 293 | cmdline: "a b c --> FAIL 23", 294 | wantCmd: "a b c", 295 | wantFail: true, 296 | wantCode: 23, 297 | }, 298 | { 299 | cmdline: "a b c --> FAIL 23a", 300 | wantErr: true, 301 | }, 302 | { 303 | cmdline: "a b c --> FAIL 0", 304 | wantErr: true, 305 | }, 306 | } { 307 | gotCmd, gotFail, gotCode, err := parseCommand(test.cmdline) 308 | if gotCmd != test.wantCmd || gotFail != test.wantFail || gotCode != test.wantCode || (err != nil) != test.wantErr { 309 | t.Errorf("%q:\ngot (%q, %t, %d, %v)\nwant (%q, %t, %d, %t)", 310 | test.cmdline, 311 | gotCmd, gotFail, gotCode, err, 312 | test.wantCmd, test.wantFail, test.wantCode, test.wantErr) 313 | } 314 | } 315 | } 316 | 317 | func TestParallel(t *testing.T) { 318 | ts := mustReadTestSuite(t, "parallel") 319 | ts.RunParallel(t, false) 320 | } 321 | 322 | func diffFiles(t *testing.T, gotFile, wantFile string) string { 323 | got, err := ioutil.ReadFile(gotFile) 324 | if err != nil { 325 | t.Fatal(err) 326 | } 327 | want, err := ioutil.ReadFile(wantFile) 328 | if err != nil { 329 | t.Fatal(err) 330 | } 331 | return cmp.Diff(string(want), string(got)) 332 | } 333 | 334 | func mustReadTestSuite(t *testing.T, dir string) *TestSuite { 335 | t.Helper() 336 | ts, err := Read(filepath.Join("testdata", dir)) 337 | if err != nil { 338 | t.Fatal(err) 339 | } 340 | return ts 341 | } 342 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/google/go-cmdtest 2 | 3 | go 1.12 4 | 5 | require ( 6 | github.com/google/go-cmp v0.3.1 7 | github.com/google/renameio v0.1.0 8 | ) 9 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg= 2 | github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 3 | github.com/google/renameio v0.1.0 h1:GOZbcHa3HfsPKPlmyPyN2KEohoMXOhdMbHrvbpl2QaA= 4 | github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= 5 | -------------------------------------------------------------------------------- /tempFileNonWindows.go: -------------------------------------------------------------------------------- 1 | // Copyright 2020 The Go Cloud Development Kit Authors 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // https://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // Neither windows nor windows_test 16 | //go:build !windows && !windows_test 17 | // +build !windows,!windows_test 18 | 19 | package cmdtest 20 | 21 | import ( 22 | "github.com/google/renameio" 23 | ) 24 | 25 | func createTempFile(filename string) (tempFile, error) { 26 | return renameio.TempFile("", filename) 27 | } 28 | -------------------------------------------------------------------------------- /tempFileWindows.go: -------------------------------------------------------------------------------- 1 | // Copyright 2020 The Go Cloud Development Kit Authors 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // https://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // Either windows or windows_test 16 | //go:build windows || windows_test 17 | // +build windows windows_test 18 | 19 | // Test this code on any machine with 20 | // go test -tags windows_test 21 | 22 | package cmdtest 23 | 24 | import ( 25 | "io/ioutil" 26 | "os" 27 | "path/filepath" 28 | ) 29 | 30 | func createTempFile(filename string) (tempFile, error) { 31 | f, err := ioutil.TempFile("", filepath.Base(filename)) 32 | if err != nil { 33 | return nil, err 34 | } 35 | return &simpleTempFile{File: f, path: filename}, nil 36 | } 37 | 38 | type simpleTempFile struct { 39 | *os.File 40 | path string // rename to this 41 | closed bool // Close was called 42 | done bool // Close and Rename succeeded 43 | } 44 | 45 | // Code is taken from github.com/google/renameio@v0.1.0/tempfile.go. 46 | // Although rename can't properly be done atomically on Windows, 47 | // this is the best we have, and it's better than nothing. 48 | 49 | func (t *simpleTempFile) Cleanup() error { 50 | if t.done { 51 | return nil 52 | } 53 | var closeErr error 54 | if !t.closed { 55 | closeErr = t.Close() 56 | 57 | } 58 | if err := os.Remove(t.Name()); err != nil { 59 | return err 60 | } 61 | return closeErr 62 | } 63 | 64 | func (t *simpleTempFile) CloseAtomicallyReplace() error { 65 | if err := t.Sync(); err != nil { 66 | return err 67 | } 68 | t.closed = true 69 | if err := t.Close(); err != nil { 70 | return err 71 | } 72 | if err := os.Rename(t.Name(), t.path); err != nil { 73 | return err 74 | } 75 | t.done = true 76 | return nil 77 | } 78 | -------------------------------------------------------------------------------- /testdata/bad/bad-fail-1.ct: -------------------------------------------------------------------------------- 1 | # Command should fail, but doesn't. 2 | 3 | $ echo --> FAIL 4 | 5 | # We shouldn't see this output, because the file 6 | # stops after the first error. 7 | 8 | $ echo should not appear 9 | 10 | -------------------------------------------------------------------------------- /testdata/bad/bad-fail-2.ct: -------------------------------------------------------------------------------- 1 | # Command should not fail, but does. 2 | # The failure is from the os.Chdir call. 3 | 4 | $ echo hello 5 | 6 | $ cd foo 7 | 8 | # We shouldn't see this output. 9 | $ echo should not appear 10 | 11 | -------------------------------------------------------------------------------- /testdata/bad/bad-fail-3.ct: -------------------------------------------------------------------------------- 1 | # Command should not fail, but does. 2 | # The failure is from the arg check in cmdtest.go. 3 | 4 | $ cd foo bar 5 | 6 | 7 | # We shouldn't see this output. 8 | $ echo should not appear 9 | -------------------------------------------------------------------------------- /testdata/bad/bad-fail-4.ct: -------------------------------------------------------------------------------- 1 | # Command should fail with an exit code, but there isn't one. 2 | 3 | $ cd foo bar --> FAIL 3 4 | 5 | # We shouldn't see this output. 6 | $ echo should not appear 7 | -------------------------------------------------------------------------------- /testdata/bad/bad-fail-5.ct: -------------------------------------------------------------------------------- 1 | # Command fails, but with the wrong exit code. 2 | 3 | $ cd foo --> FAIL 3 4 | 5 | # We shouldn't see this output. 6 | $ echo should not appear 7 | 8 | -------------------------------------------------------------------------------- /testdata/bad/bad-fail-6.ct: -------------------------------------------------------------------------------- 1 | # Command using ExitCodeErr 2 | 3 | $ code17 --> FAIL 4 4 | 5 | -------------------------------------------------------------------------------- /testdata/bad/bad-fail-7.ct: -------------------------------------------------------------------------------- 1 | # Command using InProcessProgram 2 | 3 | $ inprocess99 --> FAIL 5 4 | -------------------------------------------------------------------------------- /testdata/bad/bad-fail-8.ct: -------------------------------------------------------------------------------- 1 | # Command using Program 2 | 3 | $ echo-stdin -exit 1 --> FAIL 6 4 | -------------------------------------------------------------------------------- /testdata/bad/bad-output.ct: -------------------------------------------------------------------------------- 1 | # Incorrect output. 2 | $ echo hello world 3 | not hello world 4 | 5 | # More incorrect output. 6 | $ echo now 7 | $ echo is 8 | $ echo the time 9 | now 10 | isn't 11 | the time 12 | -------------------------------------------------------------------------------- /testdata/echo-stdin.go: -------------------------------------------------------------------------------- 1 | // Copyright 2019 The Go Cloud Development Kit Authors 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // https://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // This is a small Go binary for testing command redirection. 16 | 17 | package main 18 | 19 | import ( 20 | "flag" 21 | "fmt" 22 | "io" 23 | "os" 24 | ) 25 | 26 | var exit = flag.Int("exit", 0, "exit with this code") 27 | 28 | func main() { 29 | flag.Parse() 30 | if *exit != 0 { 31 | os.Exit(*exit) 32 | } 33 | fmt.Println("Here is stdin:") 34 | _, err := io.Copy(os.Stdout, os.Stdin) 35 | if err != nil { 36 | fmt.Fprintf(os.Stderr, "failed: %v\n", err) 37 | os.Exit(1) 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /testdata/good-without-output/good-without-output.ct: -------------------------------------------------------------------------------- 1 | # Initial comments go here. 2 | 3 | # Fails because there is no subdirectory "foo". 4 | $ cd foo --> FAIL 5 | 6 | # ... and it fails with exit code 2. 7 | $ cd foo --> FAIL 2 8 | 9 | $ echo hello world 10 | 11 | $ echo now 12 | $ echo is 13 | $ echo the time 14 | $ echo when\nwe\ndance 15 | 16 | # Let's make a directory. 17 | 18 | $ mkdir foo 19 | $ cd foo 20 | $ echo the ${ROOTDIR} is what it is 21 | 22 | $ fecho bar line one 23 | $ cat bar 24 | 25 | $ fecho bar line two 26 | $ cat bar 27 | 28 | $ setenv foo bar 29 | $ echo foo equals "${foo}" 30 | 31 | # Input redirection. 32 | $ echo-stdin < bar 33 | 34 | # InProcessProgram with input redirection. 35 | $ echoStdin < bar 36 | 37 | # More stuff here to check that InProcessProgram undoes 38 | # its redirections. 39 | 40 | $ fecho bar line three 41 | $ cat bar 42 | 43 | $ fecho bar line\nfour 44 | $ cat bar 45 | -------------------------------------------------------------------------------- /testdata/good/good.ct: -------------------------------------------------------------------------------- 1 | # Initial comments go here. 2 | 3 | # Fails because there is no subdirectory "foo". 4 | $ cd foo --> FAIL 5 | 6 | # ... and it fails with exit code 2. 7 | $ cd foo --> FAIL 2 8 | 9 | $ echo hello world 10 | hello world 11 | 12 | $ echo now 13 | $ echo is 14 | $ echo the time 15 | $ echo when\nwe\ndance 16 | now 17 | is 18 | the time 19 | when 20 | we 21 | dance 22 | 23 | # Let's make a directory. 24 | 25 | $ mkdir foo 26 | $ cd foo 27 | $ echo the ${ROOTDIR} is what it is 28 | the ${ROOTDIR} is what it is 29 | 30 | $ fecho bar line one 31 | $ cat bar 32 | line one 33 | 34 | $ fecho bar line two 35 | $ cat bar 36 | line two 37 | 38 | $ setenv foo bar 39 | $ echo foo equals "${foo}" 40 | foo equals "bar" 41 | 42 | # Input redirection. 43 | $ echo-stdin < bar 44 | Here is stdin: 45 | line two 46 | 47 | # InProcessProgram with input redirection. 48 | $ echoStdin < bar 49 | Here is stdin: 50 | line two 51 | 52 | # More stuff here to check that InProcessProgram undoes 53 | # its redirections. 54 | 55 | $ fecho bar line three 56 | $ cat bar 57 | line three 58 | 59 | $ fecho bar line\nfour 60 | $ cat bar 61 | line 62 | four 63 | -------------------------------------------------------------------------------- /testdata/parallel/par1.ct: -------------------------------------------------------------------------------- 1 | $ echo hello world 2 | hello world 3 | 4 | # Fails because there is no subdirectory "foo". 5 | $ cd foo --> FAIL 6 | -------------------------------------------------------------------------------- /testdata/parallel/par2.ct: -------------------------------------------------------------------------------- 1 | $ echo hello world 2 2 | hello world 2 3 | 4 | $ cd bar --> FAIL 5 | -------------------------------------------------------------------------------- /testdata/read/read.ct: -------------------------------------------------------------------------------- 1 | # A sample test file. 2 | 3 | # Prefix stuff. 4 | 5 | $ command arg1 arg2 6 | $ cmd2 7 | out1 8 | out2 9 | 10 | # start of the next case 11 | $ c3 12 | 13 | # start of the third 14 | 15 | $ c4 --> FAIL 16 | out3 17 | 18 | $ c5 --> FAIL 2 19 | out4 20 | 21 | 22 | # end 23 | -------------------------------------------------------------------------------- /testdata/update/update.ct: -------------------------------------------------------------------------------- 1 | $ echo update this file 2 | -------------------------------------------------------------------------------- /testdata/update/update.golden: -------------------------------------------------------------------------------- 1 | $ echo update this file 2 | update this file 3 | --------------------------------------------------------------------------------