├── LICENSE ├── README.md ├── vwc_chunk.v └── vwc_parallel.v /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Sebastian Schicho 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # vwc 2 | 3 | Beating C with 100 Lines of [V](https://vlang.io). 4 | A simple wc (word count) clone, designed to be faster than C. 5 | 6 | This is my late addition to a trend from late 2019, about trying to write a simple wc clone in a few lines of code and trying to beat its performance. 7 | 8 | The [original article](https://chrispenner.ca/posts/wc), which started the trend, rewrote it in Haskell and my implementation is based on a program written in Go by [Ajeet D'Souza](https://ajeetdsouza.github.io/blog/posts/beating-c-with-70-lines-of-go/). 9 | 10 | To be exact, as V's syntax is by desgin very close to Go, I mostly just rewrote Ajeet D'Souza's Go code in V. 11 | The original source code can be found [here](https://github.com/ajeetdsouza/blog-wc-go). 12 | 13 | ## Benchmarking & comparison 14 | 15 | I am going to compare the results using GNU time, as done by others in their articles. I am comparing the performance on parsing a 100 MB and 1 GB text file, with ASCII characters only. 16 | 17 | `$ /usr/bin/time -f "%es %MKB" wc test.txt` 18 | 19 | For better comparison with the Go code, I will not rely on the stats given in the original article, but will compile the Go code myself using Go 1.16. 20 | 21 | In [#2](https://github.com/schicho/vwc/issues/2) it was noted that as my implementation only supports ASCII, I should run GNU wc with the ASCII locale `LANG=C` to give a more fair and accurate comparison. 22 | Setting the locale tells WC that it only needs to expect ASCII chars, thus making the program run a bit faster. 23 | The benchmarked times of GNU wc have been updated using `LANG=C time wc text.txt` to set the locale. 24 | 25 | All benchmarks will be run on my system with the following specs: 26 | - Intel Core i5-8265U @ 1.60 GHz @ 4 cores, 8 threads 27 | - 8 GB DDR4 RAM @ 2667 MHz 28 | - 1 TB M.2 SSD 29 | - Ubuntu 20.04 30 | 31 | The V and Go code use a 16 KB buffer for reading input. 32 | 33 | ## The two approaches 34 | 35 | ### Single threaded 36 | 37 | The single threaded code reads into the buffer and then counts the words in that buffer, keeping track of whether we just started a new word previously or not. D'Souza's article goes into this in a lot more detail under the section 'Splitting the input'. The code can be directly transferred from Go to V with minor adjustments. 38 | Only difference is, instead of relying on system calls to get the file size, I decided to count all the bytes manually in the process. 39 | 40 | First we need two structs to organize our data in: 41 | 42 | ```V 43 | struct FileChunk { 44 | mut: 45 | prev_char_is_space bool 46 | buffer []byte 47 | } 48 | 49 | struct Count { 50 | mut: 51 | line_count u32 52 | word_count u32 53 | } 54 | ``` 55 | 56 | These are quite self-explanatory. In a `FileChunk` we store 16KB of our file, and as the last char of the previous chunk might be a space and that would mean we start a new word with the new chunk. 57 | 58 | The `get_count()` function is where the magic happens. Here we simply read every byte and compare it to the ASCII values of different chars. Thus creating the logic of counting words and lines. V's `match` is the perfect candidate here, similar to the `switch()` of many other languages. 59 | 60 | Note that we need to declare all variables as mutable here, and need to initialize them ourselves, as required by V's design. 61 | 62 | ```V 63 | fn get_count(chunk FileChunk) Count { 64 | mut count := Count{0, 0} 65 | mut prev_char_is_space := chunk.prev_char_is_space 66 | 67 | for b in chunk.buffer { 68 | match b { 69 | new_line { 70 | count.line_count++ 71 | prev_char_is_space = true 72 | } 73 | space, tab, carriage_return, vertical_tab, form_feed { 74 | prev_char_is_space = true 75 | } 76 | else { 77 | if prev_char_is_space { 78 | prev_char_is_space = false 79 | count.word_count++ 80 | } 81 | } 82 | } 83 | } 84 | 85 | return count 86 | } 87 | ``` 88 | 89 | I declared all the chars as `const` at the start of the file as they are also used in other functions. 90 | 91 | ```V 92 | const ( 93 | buffer_size = 16 * 1024 94 | new_line = `\n` 95 | space = ` ` 96 | tab = `\t` 97 | carriage_return = `\r` 98 | vertical_tab = `\v` 99 | form_feed = `\f` 100 | ) 101 | ``` 102 | 103 | The only part that's still missing is the main function. 104 | Here you can see that we read the file into the buffer, counting the bytes read per read call, as in the last read we might have reached the end of the file before the buffer is full. 105 | 106 | Error handling in V is similar to Go, but is done via the `or` block. This enforces proper error handling and leaves out the visual noise of checking for `err != nil`. 107 | 108 | Now we just count the words in each chunk separately and then sum up the results to finally print them on the terminal. 109 | 110 | ```V 111 | mut total_count := Count{0, 0} 112 | mut byte_count := 0 113 | mut last_char_is_space := true 114 | 115 | mut buffer := []byte{len: buffer_size} 116 | 117 | for { 118 | nbytes := file.read(mut buffer) or { 119 | match err { 120 | none { // EOF 'error', just break out of the loop. 121 | break 122 | } 123 | else { 124 | println(err) 125 | } 126 | } 127 | exit(1) 128 | } 129 | 130 | count := get_count(FileChunk{last_char_is_space, buffer[..nbytes]}) 131 | last_char_is_space = is_space(buffer[nbytes - 1]) 132 | 133 | total_count.line_count += count.line_count 134 | total_count.word_count += count.word_count 135 | byte_count += nbytes 136 | } 137 | 138 | println('$total_count.line_count $total_count.word_count $byte_count $file_path') 139 | ``` 140 | 141 | Now to the most exciting part! Comparing the results. 142 | 143 | Note: I compiled the Go programs just with `go build main.go`. For the V programs I added the `-prod` flag to get optimized builds: `v -prod vwc_chunk.v`. 144 | Without the production flag, the V compiler is blazingly fast, but the builds are less optimized and the time for parsing the file is actually closer to that of GNU wc than to that of Go. 145 | 146 | | Program | File Size | Time | Memory | 147 | | --- | --- | --- | --- | 148 | | GNU wc | 100 MB | 0.40s | 2268 KB | 149 | | Go wc | 100 MB | 0.29s | 1588 KB | 150 | | V wc | 100 MB | 0.30s | 1424 KB | 151 | | GNU wc | 1 GB | 4.39s | 2264 KB | 152 | | Go wc | 1 GB | 3.26s | 1596 KB | 153 | | V wc | 1 GB | 3.17s | 1476 KB | 154 | 155 | So as we can see, both programs can easily beat C in performance and memory use. For the most part Go and V are very close to each other in performance. 156 | The only difference is binary size, where V can beat Go by a lot. (Not that it really matters, but it's still interesting to see.) 157 | 158 | ``` 159 | C binary: 48 KB 160 | V binary: 108 KB 161 | Go binary: 1.5 MB 162 | ``` 163 | 164 | ### Multithreaded 165 | 166 | As stated by D'Souza: "Admittedly, a parallel wc is overkill, but let's see how far we can go". 167 | 168 | In terms of code, V can again borrow many and almost all lines from the Go code. 169 | 170 | The only difference using V was that the compiler didn't just error, but often crashed using the concurrency features. I guess this is the result of V still being quite a new language and having a very small team of contributors, which are not hired by a company like Go with Google. 171 | Also reference types work slightly differently in V than Go, which resulted in me being stuck with weird compiler errors and crashes. But this is probably also down to me still being new to the language and also me using V's concurrency features for the first time. 172 | 173 | In the end I got it to work with a bit of trying and a bit of luck. 174 | 175 | One thing needs to be said tho: V's concurrency features are not unstable. If it works, it works. But getting it to work in the first place was a bit tough. 176 | 177 | ```V 178 | struct Count { 179 | mut: 180 | line_count u32 181 | word_count u32 182 | byte_count int 183 | } 184 | ``` 185 | 186 | For the multithreaded version, I included the number of bytes into each `Count` struct. This is needed as we now read from multiple threads. 187 | 188 | ```V 189 | struct FileReader { 190 | mut: 191 | file os.File 192 | last_char_is_space bool 193 | mutex sync.Mutex 194 | } 195 | 196 | fn (mut file_reader FileReader) read_chunk(mut buffer []byte) ?FileChunk { 197 | file_reader.mutex.@lock() 198 | defer { 199 | file_reader.mutex.unlock() 200 | } 201 | 202 | nbytes := file_reader.file.read(mut buffer) ? // Propagate error. Either EOF or read error. 203 | chunk := FileChunk{file_reader.last_char_is_space, buffer[..nbytes]} 204 | file_reader.last_char_is_space = is_space(buffer[nbytes - 1]) 205 | return chunk 206 | } 207 | ``` 208 | 209 | The `FileReader` struct is similar to the `FileChunk`, but we now have direct access to the file, and it also includes a mutex, so that multiple reading threads do not get ahead of each other and overwrite the `last_char_is_space`, and we also have consistent results on HDDs, where parallel reads are not directly possible. You can see the mutex being locked and unlocked in the `read_chunk` method. This is also another example of V's error handling, in which a possible error is just propagated to an outer function using `?`. 210 | 211 | ```V 212 | fn file_reader_counter(mut file_reader FileReader, counts chan Count) { 213 | mut buffer := []byte{len: buffer_size} 214 | mut total_count := Count{0, 0, 0} 215 | 216 | for { 217 | chunk := file_reader.read_chunk(mut buffer) or { 218 | match err { 219 | none { 220 | // EOF 'error', just break out of the loop. 221 | break 222 | } 223 | else { 224 | println(err) 225 | } 226 | } 227 | exit(1) 228 | } 229 | 230 | count := get_count(chunk) 231 | 232 | total_count.line_count += count.line_count 233 | total_count.word_count += count.word_count 234 | total_count.byte_count += chunk.buffer.len 235 | } 236 | 237 | counts <- total_count 238 | } 239 | ``` 240 | 241 | The `file_reader_counter` function is very similar to the main function before; the only difference is that this function is now intended to be multithreaded using coroutines. You can see the channel in the funtion header, which is used to send the results. Basically each coroutine reads into its buffer and after reading is finished, the next coroutine can read, while the other coroutine counts the words. 242 | 243 | In the main function the only task left is to start as many coroutines as the CPU has logical cores and then collect the counts from the channel and combine the results. 244 | We create one `FileReader` on the heap via the `&` and create an unbuffered channel of type `Count`. 245 | 246 | Then via the `go` keyword we can start the coroutines just as in Go. 247 | 248 | ```V 249 | mut file_reader := &FileReader{file, true, sync.new_mutex()} 250 | counts := chan Count{} 251 | num_workers := runtime.nr_cpus() 252 | 253 | for i := 0; i < num_workers; i++ { 254 | go file_reader_counter(mut file_reader, counts) 255 | } 256 | 257 | mut total_count := Count{0, 0, 0} 258 | 259 | for i := 0; i < num_workers; i++ { 260 | count := <-counts 261 | total_count.line_count += count.line_count 262 | total_count.word_count += count.word_count 263 | total_count.byte_count += count.byte_count 264 | } 265 | counts.close() 266 | 267 | println('$total_count.line_count $total_count.word_count $total_count.byte_count $file_path') 268 | ``` 269 | 270 | Comparing the different implementations on the same files as before yields astonishing results: 271 | 272 | | Program | File Size | Time | Memory | 273 | | --- | --- | --- | --- | 274 | | GNU wc | 100 MB | 0.40s | 2268 KB | 275 | | GO wc parallel | 100 MB | 0.08s | 1944 KB | 276 | | V wc parallel | 100 MB | 0.09s | 2036 KB | 277 | | GNU wc | 1 GB | 4.39s | 2264 KB | 278 | | GO wc parallel | 1 GB | 0.71s | 1976 KB | 279 | | V wc parallel | 1 GB | 0.88s | 2032 KB | 280 | 281 | We can tell that Go is overall minimally faster than V and also consumes a bit less RAM. The main difference in RAM usage compared to D'Souza's article is probably that my laptop runs the programs on 8 threads, thus allocating more memory for reading the file than the 4 threads in the Go article. Furthermore, I noticed that while Go's memory usage was very consistent on each of the multiple runs I did, V's memory usage varied a bit more significantly on all attempts, almost reaching the same amount of memory usage as C. 282 | 283 | ## Conclusion 284 | 285 | V itself is transpiled to C for optimized builds. You could almost say I compared C to C, but with one of the C implementations being almost like Go. 286 | 287 | Overall it was very interesting to see the comparison, not only between C and V, but also Go and V. Using the same algorithms and functions obviously resulted in very similar results, but smaller differences could still be seen. Like in the other articles, this is not meant to be a "V is better than C" article. I wrote a very simplified version of the complete GNU wc. The only goal was to be faster, but still have the same results as GNU wc on pure ASCII text. 288 | -------------------------------------------------------------------------------- /vwc_chunk.v: -------------------------------------------------------------------------------- 1 | module main 2 | 3 | import os 4 | 5 | // Adapted from: 6 | // https://ajeetdsouza.github.io/blog/posts/beating-c-with-70-lines-of-go/ 7 | // https://github.com/ajeetdsouza/blog-wc-go 8 | 9 | const ( 10 | buffer_size = 16 * 1024 11 | new_line = `\n` 12 | space = ` ` 13 | tab = `\t` 14 | carriage_return = `\r` 15 | vertical_tab = `\v` 16 | form_feed = `\f` 17 | ) 18 | 19 | struct FileChunk { 20 | mut: 21 | prev_char_is_space bool 22 | buffer []byte 23 | } 24 | 25 | struct Count { 26 | mut: 27 | line_count u32 28 | word_count u32 29 | } 30 | 31 | fn get_count(chunk FileChunk) Count { 32 | mut count := Count{0, 0} 33 | mut prev_char_is_space := chunk.prev_char_is_space 34 | 35 | for b in chunk.buffer { 36 | match b { 37 | new_line { 38 | count.line_count++ 39 | prev_char_is_space = true 40 | } 41 | space, tab, carriage_return, vertical_tab, form_feed { 42 | prev_char_is_space = true 43 | } 44 | else { 45 | if prev_char_is_space { 46 | prev_char_is_space = false 47 | count.word_count++ 48 | } 49 | } 50 | } 51 | } 52 | 53 | return count 54 | } 55 | 56 | fn is_space(b byte) bool { 57 | return b == new_line || b == space || b == tab || b == carriage_return || b == vertical_tab 58 | || b == form_feed 59 | } 60 | 61 | fn main() { 62 | if os.args.len < 2 { 63 | println('no file path specified') 64 | exit(1) 65 | } 66 | 67 | file_path := os.args[1] 68 | mut file := os.open_file(file_path, 'rb') or { 69 | println('cannot open file') 70 | exit(1) 71 | } 72 | defer { 73 | file.close() 74 | } 75 | 76 | mut total_count := Count{0, 0} 77 | mut byte_count := 0 78 | mut last_char_is_space := true 79 | 80 | mut buffer := []byte{len: buffer_size} 81 | 82 | for { 83 | nbytes := file.read(mut buffer) or { 84 | match err { 85 | none { 86 | // EOF 'error', just break out of the loop. 87 | break 88 | } 89 | else { 90 | println(err) 91 | } 92 | } 93 | exit(1) 94 | } 95 | 96 | count := get_count(FileChunk{last_char_is_space, buffer[..nbytes]}) 97 | last_char_is_space = is_space(buffer[nbytes - 1]) 98 | 99 | total_count.line_count += count.line_count 100 | total_count.word_count += count.word_count 101 | byte_count += nbytes 102 | } 103 | 104 | println('$total_count.line_count $total_count.word_count $byte_count $file_path') 105 | } 106 | -------------------------------------------------------------------------------- /vwc_parallel.v: -------------------------------------------------------------------------------- 1 | module main 2 | 3 | import os 4 | import sync 5 | import runtime 6 | 7 | // Adapted from: 8 | // https://ajeetdsouza.github.io/blog/posts/beating-c-with-70-lines-of-go/ 9 | // https://github.com/ajeetdsouza/blog-wc-go 10 | 11 | const ( 12 | buffer_size = 16 * 1024 13 | new_line = `\n` 14 | space = ` ` 15 | tab = `\t` 16 | carriage_return = `\r` 17 | vertical_tab = `\v` 18 | form_feed = `\f` 19 | ) 20 | 21 | struct FileChunk { 22 | mut: 23 | prev_char_is_space bool 24 | buffer []byte 25 | } 26 | 27 | struct Count { 28 | mut: 29 | line_count u32 30 | word_count u32 31 | byte_count int 32 | } 33 | 34 | fn get_count(chunk FileChunk) Count { 35 | mut count := Count{0, 0, 0} 36 | mut prev_char_is_space := chunk.prev_char_is_space 37 | 38 | for b in chunk.buffer { 39 | match b { 40 | new_line { 41 | count.line_count++ 42 | prev_char_is_space = true 43 | } 44 | space, tab, carriage_return, vertical_tab, form_feed { 45 | prev_char_is_space = true 46 | } 47 | else { 48 | if prev_char_is_space { 49 | prev_char_is_space = false 50 | count.word_count++ 51 | } 52 | } 53 | } 54 | } 55 | 56 | return count 57 | } 58 | 59 | fn is_space(b byte) bool { 60 | return b == new_line || b == space || b == tab || b == carriage_return || b == vertical_tab 61 | || b == form_feed 62 | } 63 | 64 | struct FileReader { 65 | mut: 66 | file os.File 67 | last_char_is_space bool 68 | mutex sync.Mutex 69 | } 70 | 71 | fn (mut file_reader FileReader) read_chunk(mut buffer []byte) ?FileChunk { 72 | file_reader.mutex.@lock() 73 | defer { 74 | file_reader.mutex.unlock() 75 | } 76 | 77 | nbytes := file_reader.file.read(mut buffer) ? // Propagate error. Either EOF or read error. 78 | chunk := FileChunk{file_reader.last_char_is_space, buffer[..nbytes]} 79 | file_reader.last_char_is_space = is_space(buffer[nbytes - 1]) 80 | return chunk 81 | } 82 | 83 | fn file_reader_counter(mut file_reader FileReader, counts chan Count) { 84 | mut buffer := []byte{len: buffer_size} 85 | mut total_count := Count{0, 0, 0} 86 | 87 | for { 88 | chunk := file_reader.read_chunk(mut buffer) or { 89 | match err { 90 | none { 91 | // EOF 'error', just break out of the loop. 92 | break 93 | } 94 | else { 95 | println(err) 96 | } 97 | } 98 | exit(1) 99 | } 100 | 101 | count := get_count(chunk) 102 | 103 | total_count.line_count += count.line_count 104 | total_count.word_count += count.word_count 105 | total_count.byte_count += chunk.buffer.len 106 | } 107 | 108 | counts <- total_count 109 | } 110 | 111 | fn main() { 112 | if os.args.len < 2 { 113 | println('no file path specified') 114 | exit(1) 115 | } 116 | 117 | file_path := os.args[1] 118 | mut file := os.open_file(file_path, 'rb') or { 119 | println('cannot open file') 120 | exit(1) 121 | } 122 | defer { 123 | file.close() 124 | } 125 | 126 | mut file_reader := &FileReader{file, true, sync.new_mutex()} 127 | counts := chan Count{} 128 | num_workers := runtime.nr_cpus() 129 | 130 | for i := 0; i < num_workers; i++ { 131 | go file_reader_counter(mut file_reader, counts) 132 | } 133 | 134 | mut total_count := Count{0, 0, 0} 135 | 136 | for i := 0; i < num_workers; i++ { 137 | count := <-counts 138 | total_count.line_count += count.line_count 139 | total_count.word_count += count.word_count 140 | total_count.byte_count += count.byte_count 141 | } 142 | counts.close() 143 | 144 | println('$total_count.line_count $total_count.word_count $total_count.byte_count $file_path') 145 | } 146 | --------------------------------------------------------------------------------