├── .clang-tidy ├── .gitignore ├── .gitmodules ├── .nojekyll ├── README.md ├── _config.yml ├── a_small_trick_to_improve_technical_discussions_by_sharing_code.html ├── a_small_trick_to_improve_technical_discussions_by_sharing_code.md ├── a_subtle_data_race_in_go.html ├── a_subtle_data_race_in_go.md ├── addressing_cgo_pains_one_at_a_time.html ├── addressing_cgo_pains_one_at_a_time.md ├── advent_of_code_2018_5.html ├── advent_of_code_2018_5.md ├── advent_of_code_2018_5_revisited.html ├── advent_of_code_2018_5_revisited.md ├── aoc2018_5.asm ├── aoc2020_5.c ├── articles-by-tag.html ├── bench.txt ├── body_of_work.html ├── body_of_work.md ├── build-pie-executables-with-pie.html ├── build-pie-executables-with-pie.md ├── build.sh ├── cnrs.pdf ├── coffee_or_tea.png ├── compile_flags.txt ├── compile_ziglang_from_source_on_alpine_2020_9.html ├── compile_ziglang_from_source_on_alpine_2020_9.md ├── cpu_profile_sha1_sw_debug_asan.svg ├── favicon.ico ├── feed.html ├── feed.md ├── feed.png ├── feed.xml ├── footer.html ├── game-x11-empty-background.png ├── game-x11-first-image.png ├── game-x11-first-scene.png ├── game-x11-freebsd.png ├── game-x11-sprite.png ├── git_web_ui_link.png ├── gnuplot.png ├── gnuplot_lang.html ├── gnuplot_lang.md ├── header.html ├── highlight.min.js ├── how_to_rewrite_a_cpp_codebase_successfully.html ├── how_to_rewrite_a_cpp_codebase_successfully.md ├── image_size_reduction.html ├── image_size_reduction.md ├── index.html ├── index.md ├── kahns_algorithm.html ├── kahns_algorithm.js ├── kahns_algorithm.md ├── kahns_algorithm_1.dot ├── kahns_algorithm_1.svg ├── kahns_algorithm_1_invalid.dot ├── kahns_algorithm_1_invalid.svg ├── kahns_algorithm_2.dot ├── kahns_algorithm_2.svg ├── kahns_algorithm_2_1.dot ├── kahns_algorithm_2_1.svg ├── kahns_algorithm_2_2.dot ├── kahns_algorithm_2_2.svg ├── kahns_algorithm_2_3.dot ├── kahns_algorithm_2_3.svg ├── kahns_algorithm_2_4.dot ├── kahns_algorithm_2_4.svg ├── kahns_algorithm_2_5.dot ├── kahns_algorithm_2_5.svg ├── kahns_algorithm_2_6.dot ├── kahns_algorithm_2_6.svg ├── kahns_algorithm_2_7.dot ├── kahns_algorithm_2_7.svg ├── kahns_algorithm_2_8.dot ├── kahns_algorithm_2_8.svg ├── kahns_algorithm_2_9.dot ├── kahns_algorithm_2_9.svg ├── kahns_algorithm_3.dot ├── kahns_algorithm_3.svg ├── kahns_algorithm_4.dot ├── kahns_algorithm_4.svg ├── lessons_learned_from_a_successful_rust_rewrite.html ├── lessons_learned_from_a_successful_rust_rewrite.md ├── main.c ├── main.css ├── making_my_debug_build_run_100_times_faster.html ├── making_my_debug_build_run_100_times_faster.md ├── making_my_static_blog_generator_11_times_faster.html ├── making_my_static_blog_generator_11_times_faster.md ├── making_my_static_blog_generator_11_times_faster_profile_after.svg ├── making_my_static_blog_generator_11_times_faster_profile_before.svg ├── me.jpeg ├── mem_prof1.png ├── mem_prof2.png ├── mem_prof3.png ├── mem_prof4.png ├── mem_prof_flamegraph.svg ├── mem_profile.c ├── odin_and_musl.html ├── odin_and_musl.md ├── odin_syntax.js ├── perhaps_rust_needs_defer.html ├── perhaps_rust_needs_defer.md ├── roll_your_own_memory_profiling.html ├── roll_your_own_memory_profiling.md ├── rust_c++_interop_trick.html ├── rust_c++_interop_trick.md ├── screencast.mp4 ├── screencast.webm ├── search.js ├── search_index.js ├── simd_talk.png ├── speed_up_your_ci.html ├── speed_up_your_ci.md ├── the_missing_cross_platform_os_api_for_timers.html ├── the_missing_cross_platform_os_api_for_timers.md ├── tip_of_day_1.html ├── tip_of_day_1.md ├── tip_of_day_3.html ├── tip_of_day_3.md ├── tip_of_the_day_2.html ├── tip_of_the_day_2.md ├── tip_of_the_day_4.html ├── tip_of_the_day_4.md ├── tip_of_the_day_5.html ├── tip_of_the_day_5.md ├── tip_of_the_day_6.html ├── tip_of_the_day_6.md ├── todo.md ├── way_too_many_ways_to_wait_for_a_child_process_with_a_timeout.html ├── way_too_many_ways_to_wait_for_a_child_process_with_a_timeout.md ├── wayland-logo.h ├── wayland-screenshot-floating.png ├── wayland-screenshot-red.png ├── wayland-screenshot-tiled.png ├── wayland-screenshot-tiled1.png ├── wayland.c ├── wayland_from_scratch.html ├── wayland_from_scratch.md ├── what_should_your_mutexes_be_named.html ├── what_should_your_mutexes_be_named.md ├── write_a_video_game_from_scratch_like_1987.html ├── write_a_video_game_from_scratch_like_1987.md ├── x11_x64.html ├── x11_x64.md ├── x11_x64_black_window.png ├── x11_x64_final.png ├── x86asm.min.js ├── you_inherited_a_legacy_cpp_codebase_now_what.html ├── you_inherited_a_legacy_cpp_codebase_now_what.md └── zero2.png /.gitignore: -------------------------------------------------------------------------------- 1 | a.out 2 | .project.gf 3 | .zig-cache 4 | zig-out 5 | README.html 6 | src.bin 7 | *.gz 8 | *.bin 9 | perf.data* 10 | *.zst 11 | vgcore.* 12 | main.bin* 13 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "submodules/cstd"] 2 | path = submodules/cstd 3 | url = https://github.com/gaultier/cstd.git 4 | [submodule "submodules/cmark-gfm"] 5 | path = submodules/cmark-gfm 6 | url = https://github.com/github/cmark-gfm.git 7 | -------------------------------------------------------------------------------- /.nojekyll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/.nojekyll -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | https://gaultier.github.io/blog/ 2 | 3 | 4 | ## Quickstart 5 | 6 | *Ensure git submodules are present e.g. `git submodule update --init`.* 7 | 8 | Requirements: 9 | - A C23 compiler 10 | - `git` 11 | 12 | Build this blog (i.e. convert markdown files to HTML): 13 | 14 | ```sh 15 | $ ./build.sh release 16 | # Build once. 17 | $ ./main.bin 18 | # Watch & rebuild. 19 | $ ls *.md | entr -c ./main.bin 20 | ``` 21 | 22 | Serve the files locally: 23 | 24 | ```sh 25 | $ python3 -m http.server -d .. 26 | ``` 27 | 28 | Optimize a PNG (requires `pngquant`): 29 | 30 | ```sh 31 | $ pngquant foo.png -o foo.tmp && mv foo.tmp foo.png 32 | ``` 33 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | title: Philipe Gaultier's technical blog 2 | theme: null 3 | -------------------------------------------------------------------------------- /a_small_trick_to_improve_technical_discussions_by_sharing_code.md: -------------------------------------------------------------------------------- 1 | Title: A small trick to improve technical discussions by sharing code 2 | Tags: Lua, Neovim, Git 3 | --- 4 | 5 | This is a big title for a small trick that I've been using daily for years now, in every place I've worked at. 6 | 7 | Whenever there is a technical discussion, a bug hunt, or any disagreement about the codebase, I think it really helps to look at existing code to anchor the debate in reality and make it concrete. 8 | 9 | Copy pasting code, taking screenshots, or screen sharing may work at times but I have found a low-tech solution that's superior: Sharing a link to a region of code in the codebase. It's shorter, easier, and can be used in chats, documentation and PRs. 10 | It works for any code, be it existing code on the main branch, or experimental code on a branch: 11 | 12 | ![Link in Github's web UI](git_web_ui_link.png) 13 | 14 | Every web UI of every Version Control System (VCS) worth its salt has that feature, let's take Github for example: [https://github.com/gaultier/micro-kotlin/blob/master/class_file.h#L773-L775](https://github.com/gaultier/micro-kotlin/blob/master/class_file.h#L773-L775) 15 | 16 | The hurdle is that every hosting provider has its own URL 'shape' and it's not always documented, so there is a tiny bit of reverse-engineering involved. Compare the previous URL with this one: [https://gitlab.com/philigaultier/jvm-bytecode/-/blob/master/class_file.h?ref_type=heads#L125-127](https://gitlab.com/philigaultier/jvm-bytecode/-/blob/master/class_file.h?ref_type=heads#L125-127). It's slightly different. 17 | 18 | So to make it easy to share a link to code with coworkers, I've written a tiny script to craft the URL for me, inside my editor. I select a few lines, hit a keystroke, and the URL is now in the clipboard for me to paste anywhere. 19 | 20 | Since I use Neovim and Lua, this is what I'll cover, but I'm sure any editor can do that. Now that I think of it, there should be an existing extension for this? Back when I started using this trick I remember searching for one and finding nothing. 21 | 22 | This article could also serve as a gentle introduction to using Lua in Neovim. The code is also directly mappable to Vimscript, Vim9 script or anything really. 23 | 24 | So first thing first we need to create a user command to invoke this functionality and later map it to a keystroke: 25 | 26 | ```lua 27 | vim.api.nvim_create_user_command('GitWebUiUrlCopy', function(arg) 28 | end, 29 | {force=true, range=true, nargs=0, desc='Copy to clipboard a URL to a git webui for the current line'}) 30 | ``` 31 | 32 | - `force=true` overrides any previous definition which is handy when iterating over the implementation 33 | - `range=true` allows for selecting multiple lines and calling this command on the line range, but it also works when not selecting anything (in normal mode) 34 | - `nargs=0` means that no argument is passed to the command 35 | 36 | We pass a callback to `nvim_create_user_command` which will be called when we invoke the command. For now it does nothing but we are going to implement it in a second. 37 | 38 | `arg` is an object containing for our purposes the start and end line numbers: 39 | 40 | ```lua 41 | local line_start = arg.line1 42 | local line_end = arg.line2 43 | ``` 44 | 45 | And we also need to get the absolute path to the current file: 46 | 47 | ```lua 48 | local file_path_abs = vim.fn.expand('%:p') 49 | ``` 50 | 51 | *From this point on explanations are git specific, but I'm sure other VCSes have similar features.* 52 | 53 | Note that since the current directory might be one or several directories deep relative to the root of the git repository, we need to fix this path, because the git web UI expects a path from the root of the git repository. 54 | 55 | The easiest way to do so is using `git ls-files --full-name` to convert the absolute path to the path from the root of the repostory. 56 | 57 | There are many ways in Neovim to call out to a command in a subprocess, here's one of them, to get the output of the command: 58 | 59 | ```lua 60 | local file_path_abs = vim.fn.expand('%:p') 61 | local file_path_rel_cmd = io.popen('git ls-files --full-name "' .. file_path_abs .. '"') 62 | local file_path_relative_to_git_root = file_path_rel_cmd:read('*a') 63 | file_path_rel_cmd.close() 64 | ``` 65 | 66 | We also need to get the git URL of the remote (assuming there is only one, but it's easy to expand the logic to handle multiple): 67 | 68 | ```lua 69 | local cmd_handle = io.popen('git remote get-url origin') 70 | local git_origin = cmd_handle:read('*a') 71 | cmd_handle.close() 72 | git_origin = string.gsub(git_origin, "%s+$", "") 73 | ``` 74 | 75 | And the last bit of information we need is to get the current commit. 76 | In the past, I just used the current branch name, however since this is a moving target, it meant that when opening the link, the code might be completely different than what it was when giving out the link. Using a fixed commit is thus better (assuming no one force pushes and messes with the history): 77 | 78 | ```lua 79 | local cmd_handle = io.popen('git rev-parse HEAD') 80 | local git_commit = cmd_handle:read('*a') 81 | cmd_handle.close() 82 | git_commit = string.gsub(git_commit, "%s+$", "") 83 | ``` 84 | 85 | Now, we can craft the URL by first extracting the interesting parts of the git remote URL and then tacking on at the end all the URL parameters precising the location. 86 | I assume the git remote URL is a `ssh` URL here, again it's easy to tweak to also handle `https` URL. Also note that this is the part that's hosting provider specific. 87 | 88 | Since I am mainly using Azure DevOps (ADO) and Github at the moment this is what I'll show. In ADO, the git remote URL looks like this: 89 | 90 | ``` 91 | git@ssh.:v3/// 92 | ``` 93 | 94 | And the final URL looks like: 95 | 96 | ```lua 97 | https://///_git/? 98 | ``` 99 | 100 | In Github, the git remote URL looks like this: 101 | 102 | ``` 103 | git@github.com:/.git 104 | ``` 105 | 106 | And the final URL looks like this: 107 | 108 | ``` 109 | https://github.com///blob//? 110 | ``` 111 | 112 | We inspect the git remote url to know in which case we are: 113 | 114 | ```lua 115 | local url = '' 116 | if string.match(git_origin, 'github') then 117 | -- Handle Github 118 | elseif string.match(git_origin, 'azure.com') then 119 | -- End is exclusive in that case hence the `+ 1`. 120 | line_end = line_end + 1 121 | 122 | -- Handle ADO 123 | else 124 | print('hosting provider not supported') 125 | end 126 | ``` 127 | 128 | We use a Lua pattern to extract the components from the git remote URL using `string.gmatch`. It weirdly returns an iterator yielding only one result containing our matches, we use a `for` loop to do so (perhaps there is an easier way in Lua?): 129 | 130 | Here's for Github: 131 | 132 | ```lua 133 | for host, user, project in string.gmatch(git_origin, 'git@([^:]+):([^/]+)/([^/]+)%.git') do 134 | url = 'https://' .. host .. '/' .. user .. '/' .. project .. '/blob/' .. git_commit .. '/' .. file_path_relative_to_git_root .. '#l' .. line_start .. '-l' .. line_end 135 | break 136 | end 137 | ``` 138 | 139 | And here's for ADO: 140 | 141 | ```lua 142 | for host, org, dir, project in string.gmatch(git_origin, 'git@ssh%.([^:]+):v3/([^/]+)/([^/]+)/([^\n]+)') do 143 | url = 'https://' .. host .. '/' .. org .. '/' .. dir .. '/_git/' .. project .. '?lineStartColumn=1&lineStyle=plain&_a=contents&version=GC' .. git_commit .. '&path=' .. file_path_relative_to_git_root .. '&line=' .. line_start .. '&lineEnd=' .. line_end 144 | break 145 | end 146 | ``` 147 | 148 | Finally we stick the result in the system clipboard, and we can even open the url in the default browser (I have only tested that logic on Linux but it *should* work on other OSes): 149 | 150 | ```lua 151 | -- Copy to clipboard. 152 | vim.fn.setreg('+', url) 153 | 154 | -- Open URL in the default browser. 155 | local os_name = vim.loop.os_uname().sysname 156 | if os_name == 'Linux' or os_name == 'FreeBSD' or os_name == 'OpenBSD' or os_name == 'NetBSD' then 157 | os.execute('xdg-open "' .. url .. '"') 158 | elseif os_name == 'Darwin' then 159 | os.execute('open "' .. url .. '"') 160 | elseif os_name == 'Windows' then 161 | os.execute('start "' .. url .. '"') 162 | else 163 | print('Unknown os: ' .. os_name) 164 | end 165 | ``` 166 | 167 | We can now map the command to our favorite keystroke, for me space + x, for both normal mode (`n`) and visual mode (`v`): 168 | 169 | ```lua 170 | vim.keymap.set({'v', 'n'}, 'x', ':GitWebUiUrlCopy') 171 | ``` 172 | 173 | And that's it, just 60 lines of Lua, and easy to extend to support even more hosting providers. 174 | 175 | 176 | ## Addendum: the full code 177 | 178 |
179 | The full code 180 | 181 | ```lua 182 | vim.keymap.set({'v', 'n'}, 'x', ':GitWebUiUrlCopy') 183 | vim.api.nvim_create_user_command('GitWebUiUrlCopy', function(arg) 184 | local file_path_abs = vim.fn.expand('%:p') 185 | local file_path_rel_cmd = io.popen('git ls-files --full-name "' .. file_path_abs .. '"') 186 | local file_path_relative_to_git_root = file_path_rel_cmd:read('*a') 187 | file_path_rel_cmd.close() 188 | 189 | local line_start = arg.line1 190 | local line_end = arg.line2 191 | 192 | local cmd_handle = io.popen('git remote get-url origin') 193 | local git_origin = cmd_handle:read('*a') 194 | cmd_handle.close() 195 | git_origin = string.gsub(git_origin, "%s+$", "") 196 | 197 | local cmd_handle = io.popen('git rev-parse HEAD') 198 | local git_commit = cmd_handle:read('*a') 199 | cmd_handle.close() 200 | git_commit = string.gsub(git_commit, "%s+$", "") 201 | 202 | local url = '' 203 | if string.match(git_origin, 'github') then 204 | for host, user, project in string.gmatch(git_origin, 'git@([^:]+):([^/]+)/([^/]+)%.git') do 205 | url = 'https://' .. host .. '/' .. user .. '/' .. project .. '/blob/' .. git_commit .. '/' .. file_path_relative_to_git_root .. '#L' .. line_start .. '-L' .. line_end 206 | break 207 | end 208 | elseif string.match(git_origin, 'azure.com') then 209 | -- End is exclusive in that case hence the `+ 1`. 210 | line_end = line_end + 1 211 | 212 | for host, org, dir, project in string.gmatch(git_origin, 'git@ssh%.([^:]+):v3/([^/]+)/([^/]+)/([^\n]+)') do 213 | url = 'https://' .. host .. '/' .. org .. '/' .. dir .. '/_git/' .. project .. '?lineStartColumn=1&lineStyle=plain&_a=contents&version=GC' .. git_commit .. '&path=' .. file_path_relative_to_git_root .. '&line=' .. line_start .. '&lineEnd=' .. line_end 214 | break 215 | end 216 | else 217 | print('Hosting provider not supported') 218 | end 219 | 220 | -- Copy to clipboard. 221 | vim.fn.setreg('+', url) 222 | 223 | -- Open URL in the default browser. 224 | local os_name = vim.loop.os_uname().sysname 225 | if os_name == 'Linux' or os_name == 'FreeBSD' or os_name == 'OpenBSD' or os_name == 'NetBSD' then 226 | os.execute('xdg-open "' .. url .. '"') 227 | elseif os_name == 'Darwin' then 228 | os.execute('open "' .. url .. '"') 229 | elseif os_name == 'Windows' then 230 | os.execute('start "' .. url .. '"') 231 | else 232 | print('Unknown os: ' .. os_name) 233 | end 234 | end, 235 | {force=true, range=true, nargs=0, desc='Copy to clipboard a URL to a git webui for the current line'}) 236 | ``` 237 | 238 |
239 | -------------------------------------------------------------------------------- /bench.txt: -------------------------------------------------------------------------------- 1 | $ hyperfine --warmup 2 './src-main.bin' './src.bin' 2 | Benchmark 1: ./src-main.bin 3 | Time (mean ± σ): 1.773 s ± 0.022 s [User: 1.267 s, System: 0.472 s] 4 | Range (min … max): 1.748 s … 1.816 s 10 runs 5 | 6 | Benchmark 2: ./src.bin 7 | Time (mean ± σ): 158.7 ms ± 6.6 ms [User: 128.4 ms, System: 133.7 ms] 8 | Range (min … max): 151.7 ms … 175.6 ms 18 runs 9 | 10 | Summary 11 | ./src.bin ran 12 | 11.17 ± 0.48 times faster than ./src-main.bin 13 | -------------------------------------------------------------------------------- /build-pie-executables-with-pie.md: -------------------------------------------------------------------------------- 1 | Title: Build PIE executables in Go: I got nerd-sniped 2 | Tags: Go, PIE, Linux, Musl, Security 3 | --- 4 | 5 | ## Context 6 | 7 | Lately I have been hardening the build at work of our Go services and one (seemingly) low-hanging fruit was [PIE](https://en.wikipedia.org/wiki/Position-independent_code). This is code built to be relocatable, meaning it can be loaded by the operating system at any memory address. That complicates the work of an attacker because they typically want to manipulate the return address of the current function (e.g. by overwriting the stack due to a buffer overflow) to jump to a specific function e.g. `system()` from libc to pop a shell. That's easy if `system` is always at the same address. If the target function is loaded at a different address each time, it makes it more difficult for the attacker to find it. 8 | 9 | This approach was already used in the sixties (!) and has been the default for years now in most OSes and toolchains when building system executables. There is practically no downside. Some people report a single digit percent slowdown in rare cases although it's not the rule. Amusingly this randomness can be used in interesting ways for example seeding a random number generator with the address of a function. 10 | 11 | Go supports PIE as well, however this is not the default so we have to opt in. 12 | 13 | PIE is especially desirable when using CGO to call C functions from Go which is my case at work. 14 | But also Go is [not entirely memory safe](https://blog.stalkr.net/2015/04/golang-data-races-to-break-memory-safety.html) so I'd argue having PIE enabled in all cases is preferable. 15 | 16 | So let's look into enabling it. And this is also a good excuse to learn more about how surprisingly complex it is for an OS to just execute a program. 17 | 18 | ## How hard can it be? 19 | 20 | `go help buildmode` states: 21 | 22 | > -buildmode=pie 23 | > Build the listed main packages and everything they import into 24 | > position independent executables (PIE). Packages not named 25 | > main are ignored. 26 | 27 | Easy enough, right? 28 | 29 | Let's build a hello world program (*not* using CGO) with default options: 30 | 31 | ```sh 32 | $ go build main.go 33 | $ file ./main 34 | ./main: ELF 64-bit LSB executable [..] statically linked [..] 35 | ``` 36 | 37 | Now, let's add the `-buildmode=pie` option: 38 | 39 | ```sh 40 | $ go build -buildmode=pie ./main.go 41 | $ file ./main 42 | ./main: ELF 64-bit LSB pie executable [..] dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2 [..] 43 | ``` 44 | 45 | Ok, it worked, but also: why did we go from a statically linked to a dynamically linked executable? PIE should be orthogonal to static/dynamic linking! Or is it? We'll come back to that in a second. 46 | 47 | When we run our freshly built Go executable in a bare-bone Docker image (distroless), we get a nice cryptic error at runtime: 48 | 49 | ``` 50 | exec /home/nonroot/my-service: no such file or directory 51 | ``` 52 | 53 | Oh oh. Let's investigate. 54 | 55 | ### A helpful mental model 56 | 57 | I'd argue that the wording of the tools and the online content is confusing because it conflates two different things. 58 | 59 | For example, invoking `lld` with the above executable prints `statically linked`. But `file` prints `dynamically linked` for the exact same file! So which is it? 60 | 61 | A [helpful mental model](https://www.quora.com/Systems-Programming/What-is-the-exact-difference-between-dynamic-loading-and-dynamic-linking/answer/Jeff-Darcy) is to split *linking* from *loading* and have thus two orthogonal dimensions: 62 | 63 | - static linking, static loading 64 | - static linking, dynamic loading 65 | - dynamic linking, static loading 66 | - dynamic linking, dynamic loading 67 | 68 | The first dimension (static vs dynamic linking) is, from the point of view of the OS trying to launch our program, decided by the field 'Type' in the ELF header (bytes 16-20): if it's `EXEC`, it's a statically linked executable. If it's `DYN`, it's a shared object file or a statically linked PIE executable (note that the same file can be both a shared library and an executable. Pretty cool, no?). 69 | 70 | The second dimension (static vs dynamic loading) is decided by the ELF program headers: if there is one program header of type `INTERP` (which specifies the loader to use), our executable is using dynamic loading meaning it requires a loader (a.k.a interpreter) at runtime. Otherwise it does not. This aspect can be observed with `readelf`: 71 | 72 | ```sh 73 | $ readelf --program-headers ./main 74 | [..] 75 | Elf file type is DYN (Position-Independent Executable file) 76 | [..] 77 | Program Headers: 78 | Type Offset VirtAddr PhysAddr 79 | FileSiz MemSiz Flags Align 80 | [..] 81 | INTERP 0x0000000000000fe4 0x0000000000400fe4 0x0000000000400fe4 82 | 0x000000000000001c 0x000000000000001c R 0x1 83 | [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2] 84 | [..] 85 | ``` 86 | 87 | Our newly built Go executable is in the second category: static linking, dynamic loading. 88 | 89 | 90 | And that's an issue because we deploy it in a distroless Docker container where there is not even a libc available or the loader `ld.so`. 91 | 92 | It means we now need to change our Docker image to include a loader at runtime. 93 | 94 | So not ideal. What can we do? We'd like to be in the first category: static linking, static loading. 95 | 96 | I suppose that folks that ship executables to customer workstations would also have interest in doing that to remove one moving piece (the loader on each target machine). 97 | 98 | Also possibly people who want to obfuscate what their program does at startup and do not want anyone monkeying around with environment variables that impact the loader such as `LD_PRELOAD` (so, perhaps malware or anti-malware programs?). 99 | 100 | *Addendum: I have found at least one [CVE](https://seclists.org/oss-sec/2023/q4/18) in the glibc loader, and it seems it's a frequent occurrence, so in my opinion that's reason enough to remove one moving piece from the equation and prefer static loading!* 101 | 102 | ## Troubleshooting the problem 103 | 104 | 105 | It turns out that PIE was historically designed for executables using dynamic loading. 106 | The loader loads at startup the sections of the executable at different places in memory, fills (eagerly or lazily) in a global table (the [GOT](https://en.wikipedia.org/wiki/Global_Offset_Table)) the locations of symbols. And voila, functions are placed randomly in memory and function calls go through the GOT which is a level of indirection to know at runtime where the function they want to call is located. Blue team, rejoice! Red team, sad. 107 | 108 | So how does it work with a statically linked executable where a loader is not even *present* on the system? Here's a bare-bone C program that uses PIE *and* is statically linked: 109 | 110 | ```c 111 | #include 112 | 113 | int main() { printf("%p %p hello!\n", &main, &printf); } 114 | ``` 115 | 116 | We compile it, create an empty `chroot` with only our executable in it, and run it multiple times, to observe that the functions `main` and `printf` are indeed loaded in different places of memory each time: 117 | 118 | ```sh 119 | $ musl-gcc pie.c -fPIE -static-pie 120 | $ file ./a.out 121 | ./a.out: ELF 64-bit LSB pie executable [..] static-pie linked [..] 122 | $ mkdir /tmp/scratch 123 | $ sudo chroot /tmp/scratch ./a.out 124 | 0x7fcf33688419 0x7fcf336887e0 hello! 125 | $ sudo chroot /tmp/scratch ./a.out 126 | 0x7f2b44f20419 0x7f2b44f207e0 hello! 127 | $ sudo chroot /tmp/scratch ./a.out 128 | 0x7f891d95e419 0x7f891d95e7e0 hello! 129 | ``` 130 | 131 | So... how does it work when no loader is present in the environment? Well, what is the only thing that we link in our bare-bone program? Libc! And what does libc contain? You guessed it, a loader! 132 | 133 | For musl, it's the file `ldso/dlstart.c` and that's the code that runs before our `main`. Effectively libc doubles as a loader. And when statically linked, the loader gets embedded in our application and runs at startup before our code. 134 | 135 | That means that we can have our cake and eat it too: static linking and PIE! No loader required in the environment. 136 | 137 | 138 | So, how can we coerce Go to do the same? 139 | 140 | ## The solution 141 | 142 | The only way I have found is to ask Go to link with an external linker and pass it the flag `-static-pie`. Due to the explanation above that means that CGO gets enabled automatically and we need to link a libc statically: 143 | 144 | ```sh 145 | $ CGO_ENABLED=0 go build -buildmode=pie -ldflags '-linkmode external -extldflags "-static-pie"' main.go 146 | -linkmode requires external (cgo) linking, but cgo is not enabled 147 | ``` 148 | 149 | We use `musl-gcc` again for simplicity but you can also use the Zig build system to automatically build musl from source, or provide your own build of musl, etc: 150 | 151 | ```sh 152 | $ CC=musl-gcc go build -ldflags '-linkmode external -extldflags "--static-pie"' -buildmode=pie main.go 153 | $ file ./main 154 | ./main: ELF 64-bit LSB pie executable [..] static-pie linked 155 | ``` 156 | 157 | Yeah! 158 | 159 | We can check that it works in our empty `chroot` again. Here's is our Go program: 160 | 161 | ```go 162 | package main 163 | 164 | import ( 165 | "fmt" 166 | ) 167 | 168 | func main() { 169 | fmt.Println(main, fmt.Println, "hello") 170 | } 171 | ``` 172 | 173 | And here's how we build and run it: 174 | 175 | ```sh 176 | $ CC=musl-gcc go build -ldflags '-linkmode external -extldflags "--static-pie"' -buildmode=pie main.go 177 | $ cp ./main /tmp/scratch/ 178 | $ sudo chroot /tmp/scratch ./main 179 | 0x7f0701b17220 0x7f0701b122a0 hello 180 | $ sudo chroot /tmp/scratch ./main 181 | 0x7f0f27f3b220 0x7f0f27f362a0 hello 182 | $ sudo chroot /tmp/scratch ./main 183 | 0x7f61f8fd7220 0x7f61f8fd22a0 hello 184 | ``` 185 | 186 | Compare that with the non-PIE default build where the function addresses are fixed: 187 | 188 | ```sh 189 | $ go build main.go 190 | $ cp ./main /tmp/scratch/ 191 | $ sudo chroot /tmp/scratch ./main 192 | 0x48f0e0 0x48a160 hello 193 | $ sudo chroot /tmp/scratch ./main 194 | 0x48f0e0 0x48a160 hello 195 | $ sudo chroot /tmp/scratch ./main 196 | 0x48f0e0 0x48a160 hello 197 | ``` 198 | 199 | ## Conclusion 200 | 201 | 'Static PIE', or statically linked PIE executables, are a relatively new development: OpenBSD [added](https://www.openbsd.org/papers/asiabsdcon2015-pie-slides.pdf) that in 2015 and builds all system executables in that mode, Clang only [added](https://reviews.llvm.org/D58307) the flag in 2019, etc. Apparently the Go linker does not support this yet, I suppose because it does not ship with a loader and so has to rely on the libc loader (if someone knows for sure, I'd be curious to know!). After all, the preferred and default way for Go on Linux is 'static linking, static loading'. 202 | 203 | Still I think it's great to do since we get the best of both worlds, only requiring a little bit of finagling with linker flags. 204 | 205 | Also it would be nice that the Go documentation talks at least a little about this topic. In the meantime, there is this article, which I hope does not contain inaccuracies and helps a bit. 206 | 207 | A further hardening on top of PIE, that I have not yet explored yet, but is on my to do list, is [read-only relocations](https://www.redhat.com/en/blog/hardening-elf-binaries-using-relocation-read-only-relro) which makes the Global Offset Table read-only to prevent an attacker from overwriting the relocation entries there. On Fedora for example, all system executables are built with this mitigation on. 208 | 209 | -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | set -e 3 | set -f # disable globbing. 4 | 5 | CFLAGS="${CFLAGS} -fpie -fno-omit-frame-pointer -gsplit-dwarf -march=native" 6 | if command -v lld > /dev/null 2>&1; then 7 | CFLAGS+=" -fuse-ld=lld" 8 | fi 9 | LDFLAGS="${LDFLAGS} -Wl,--gc-sections -flto" 10 | 11 | CC="${CC:-clang}" 12 | WARNINGS="$(tr -s '\n' ' ' < compile_flags.txt)" 13 | 14 | error() { 15 | printf "ERROR: %s\n" "$1" 16 | exit 1 17 | } 18 | 19 | build() { 20 | case $1 in 21 | debug) 22 | CFLAGS="${CFLAGS} -O0" 23 | ;; 24 | debug_sanitizer) 25 | CFLAGS="${CFLAGS} -O0 -fsanitize=address,undefined -fsanitize-trap=all" 26 | ;; 27 | release) 28 | CFLAGS="${CFLAGS} -O3" 29 | ;; 30 | release_sanitizer) 31 | CFLAGS="${CFLAGS} -O1 -fsanitize=address,undefined -fsanitize-trap=all" 32 | ;; 33 | *) 34 | error "Build mode \"$1\" unsupported!" 35 | ;; 36 | esac 37 | 38 | # shellcheck disable=SC2086 39 | $CC $WARNINGS -g3 main.c -o main.bin $CFLAGS $LDFLAGS 40 | } 41 | 42 | if [ $# -eq 0 ]; then 43 | build debug 44 | elif [ $# -eq 1 ]; then 45 | build "$1" 46 | else 47 | error "Too many arguments!" 48 | fi 49 | -------------------------------------------------------------------------------- /cnrs.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/cnrs.pdf -------------------------------------------------------------------------------- /coffee_or_tea.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/coffee_or_tea.png -------------------------------------------------------------------------------- /compile_flags.txt: -------------------------------------------------------------------------------- 1 | -std=c23 2 | -Weverything 3 | -Wno-gnu-alignof-expression 4 | -Wno-strict-prototypes 5 | -Wno-declaration-after-statement 6 | -Wno-padded 7 | -Wno-reserved-macro-identifier 8 | -Wno-unsafe-buffer-usage 9 | -Wno-reserved-identifier 10 | -Wno-covered-switch-default 11 | -Wno-class-varargs 12 | -Wno-pre-c23-compat 13 | -Wno-pre-c11-compat 14 | -Wno-cast-qual 15 | -Wno-disabled-macro-expansion 16 | -Wno-unknown-warning-option 17 | -Werror 18 | -Wno-used-but-marked-unused 19 | -------------------------------------------------------------------------------- /compile_ziglang_from_source_on_alpine_2020_9.md: -------------------------------------------------------------------------------- 1 | Title: How to compile LLVM, Clang, LLD, and Ziglang from source on Alpine Linux 2 | Tags: LLVM, Zig, Alpine 3 | --- 4 | 5 | *This article is now outdated but remains for historical reasons.* 6 | 7 | [Ziglang](https://ziglang.org), or `Zig` for short, is an ambitious programming language addressing important flaws of mainstream languages such as failing to handle memory allocation failures or forgetting to handle an error condition in general. 8 | 9 | It is also fast moving so for most, the latest (HEAD) version will be needed, and most package managers will not have it, so we will compile it from source. 10 | 11 | Since the official Zig compiler is (currently) written in C++ and using the LLVM libraries at a specific version, we will need them as well, and once again, some package managers will not have the exact version you want (10.0.0). 12 | 13 | I find it more reliable to compile LLVM, Clang, LLD, and Zig from source and that is what we will do here. I have found that the official LLVM and Zig instructions differed somewhat, were presenting too many options, and I wanted to have one place to centralize them for my future self. 14 | 15 | Incidentally, if you are a lost C++ developer trying to compile LLVM from source, without having ever heard of Zig, well you have stumbled on the right page, you can simply skip the final block about Zig. 16 | 17 | Note that those instructions should work just the same on any Unix system. Feel free to pick the directories you want when cloning the git repositories. 18 | 19 | ```sh 20 | # The only Alpine specific bit. build-base mainly installs make and a C++ compiler. Python 3 is required by LLVM for some reason. 21 | $ apk add build-base cmake git python3 22 | 23 | $ git clone https://github.com/llvm/llvm-project.git --branch llvmorg-10.0.0 --depth 1 24 | $ cd llvm-project/ 25 | $ mkdir build 26 | $ cd build/ 27 | # The flag LLVM_ENABLE_PROJECTS is crucial, otherwise only llvm will be built, without clang or lld, 28 | # and we need all three with the exact same version since C++ does not have a stable ABI. 29 | $ cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD="AVR" -DLLVM_ENABLE_LIBXML2=OFF -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_ENABLE_PROJECTS="clang;lld" ../llvm 30 | 31 | # nproc is Linux only but you can set the number of threads manually 32 | $ make -j$(nproc) 33 | $ sudo make install 34 | 35 | $ cd ~ 36 | $ git clone https://github.com/ziglang/zig.git --depth 1 37 | $ cd zig 38 | $ mkdir build 39 | $ cd build 40 | $ cmake .. -DCMAKE_BUILD_TYPE=Release -DZIG_STATIC=ON 41 | # nproc is Linux only but you can set the number of threads manually 42 | $ make -j$(nproc) 43 | $ sudo make install 44 | ``` 45 | 46 | You will now have a `zig` executable in the PATH as well as the zig standard library. You can verify you have now the latest version by doing: 47 | 48 | ```sh 49 | $ zig version 50 | 0.6.0+749417a 51 | ``` 52 | 53 | -------------------------------------------------------------------------------- /favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/favicon.ico -------------------------------------------------------------------------------- /feed.html: -------------------------------------------------------------------------------- 1 | 5 | 6 | 7 | 8 | This blog now has an Atom feed, and yours should probably too 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 70 | 72 |
73 | 74 |
75 |

⏴ Back to all articles

76 | 77 |

Published on 2023-12-15

78 |
79 |
80 |

This blog now has an Atom feed, and yours should probably too

81 |
Feed Atom UUID
82 |
83 | 84 |

Find it here or in the header on the top right-hand corner.

85 |

Imagine a world where you can see the content of each website you like inside the app of your choosing, read the articles offline and save them on disk for later, be notified whenever the website has something new, and all of which is implemented with an open standard. Well that was most of the web some years ago and this blog now does all of that.

86 |

This feed inside the open-source app NewsFlash (https://flathub.org/apps/io.gitlab.news_flash.NewsFlash)

87 |

And it's not hard! The only thing we need is to serve a feed.xml file that lists articles with some metadata such as 'updated at' and a UUID to be able to uniquely identify an article. This XML file is an Atom feed which has a nice RFC.

88 |

I implemented that in under an hour, skimming at the RFC and examples. It's a bit hacky but it works. The script to do so is here. And you can do too! Again, it's not hard. Here goes:

89 |
    90 |
  • We pick a UUID for our feed. I just generated one and stuck it as a constant in the script.
    • 91 |
  • The 'updated at' field for the feed is just time.Now(). It's not exactly accurate, it should probably be the most recent mtime across articles but it's good enough.
    • 92 |
  • For each article (*.html) file in the directory, we add an entry (<entry>) in the XML document with: 93 |
      94 |
    • The link to the article, that's just the filename in my case.
      • 95 |
    • The 'updated at' field, which is just the mtime of the file locally queried from git
      • 96 |
    • The 'published at' field, which is just the ctime of the file locally queried from git
      • 97 |
    • A UUID. Here I went with UUIDv5 which is simply the sha1 of the file name in the UUID format. It's nifty because it means that the script is stateless and idempotent. If the article is later updated, the UUID remains the same (but the updated at will still hint at the update).
      • 98 |
    99 |
    • 100 |
101 |

And...that's it really. Enjoy reading these articles in your favorite app!

102 |

⏴ Back to all articles

103 | 104 | 107 | 108 |
109 |

110 | This blog is open-source! 111 | If you find a problem, please open a Github issue. 112 | The content of this blog as well as the code snippets are under the BSD-3 License which I also usually use for all my personal projects. It's basically free for every use but you have to mention me as the original author. 113 |

114 |
115 | 116 |
117 | 118 | 119 | -------------------------------------------------------------------------------- /feed.md: -------------------------------------------------------------------------------- 1 | Title: This blog now has an Atom feed, and yours should probably too 2 | Tags: Feed, Atom, UUID 3 | --- 4 | 5 | *Find it [here](https://gaultier.github.io/blog/feed.xml) or in the header on the top right-hand corner.* 6 | 7 | Imagine a world where you can see the content of each website you like inside the app of your choosing, read the articles offline and save them on disk for later, be notified whenever the website has something new, and all of which is implemented with an open standard. Well that was most of the web some years ago and this blog now does all of that. 8 | 9 | 10 | ![This feed inside the open-source app NewsFlash (https://flathub.org/apps/io.gitlab.news_flash.NewsFlash)](feed.png) 11 | 12 | And it's not hard! The only thing we need is to serve a `feed.xml` file that lists articles with some metadata such as 'updated at' and a UUID to be able to uniquely identify an article. This XML file is an [Atom feed](https://en.wikipedia.org/wiki/Atom_(web_standard)) which has a nice [RFC](https://datatracker.ietf.org/doc/html/rfc4287). 13 | 14 | I implemented that in under an hour, skimming at the RFC and examples. It's a bit hacky but it works. The script to do so is [here](https://github.com/gaultier/blog/blob/master/feed.go). And you can do too! Again, it's not hard. Here goes: 15 | 16 | - We pick a UUID for our feed. I just generated one and stuck it as a constant in the script. 17 | - The 'updated at' field for the feed is just `time.Now()`. It's not exactly accurate, it should probably be the most recent `mtime` across articles but it's good enough. 18 | - For each article (`*.html`) file in the directory, we add an entry (``) in the XML document with: 19 | * The link to the article, that's just the filename in my case. 20 | * The 'updated at' field, which is ~~just the `mtime` of the file locally~~ queried from git 21 | * The 'published at' field, which is ~~just the `ctime` of the file locally~~ queried from git 22 | * A UUID. Here I went with UUIDv5 which is simply the sha1 of the file name in the UUID format. It's nifty because it means that the script is stateless and idempotent. If the article is later updated, the UUID remains the same (but the `updated at` will still hint at the update). 23 | 24 | And...that's it really. Enjoy reading these articles in your favorite app! 25 | 26 | -------------------------------------------------------------------------------- /feed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/feed.png -------------------------------------------------------------------------------- /footer.html: -------------------------------------------------------------------------------- 1 | 2 | 5 | 6 |
7 |

8 | This blog is open-source! 9 | If you find a problem, please open a Github issue. 10 | The content of this blog as well as the code snippets are under the BSD-3 License which I also usually use for all my personal projects. It's basically free for every use but you have to mention me as the original author. 11 |

12 |
13 | 14 | 15 | 16 | 17 | -------------------------------------------------------------------------------- /game-x11-empty-background.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/game-x11-empty-background.png -------------------------------------------------------------------------------- /game-x11-first-image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/game-x11-first-image.png -------------------------------------------------------------------------------- /game-x11-first-scene.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/game-x11-first-scene.png -------------------------------------------------------------------------------- /game-x11-freebsd.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/game-x11-freebsd.png -------------------------------------------------------------------------------- /game-x11-sprite.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/game-x11-sprite.png -------------------------------------------------------------------------------- /git_web_ui_link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/git_web_ui_link.png -------------------------------------------------------------------------------- /gnuplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/gnuplot.png -------------------------------------------------------------------------------- /gnuplot_lang.md: -------------------------------------------------------------------------------- 1 | Title: Solving a problem with Gnuplot, the programming language (not the plotting software!) 2 | Tags: Advent of Code, Gnuplot 3 | --- 4 | 5 | 6 | *Is it any good? Can you solve real problems with it?* 7 | 8 | Most people know [Gnuplot](https://en.wikipedia.org/wiki/Gnuplot) as a way to plot data. Two lines of code and we can visualize data: 9 | 10 | ```gnuplot 11 | set output "plot.png" 12 | plot "out.dat" with lines 13 | ``` 14 | ![gnuplot.png](gnuplot.png) 15 | 16 | where `out.data` is a text file with a number on each line. 17 | 18 | The software engineering advice that I heard a long time ago and left a mark on me is: **Find a way to visualize your problem.** 19 | 20 | So Gnuplot is definitely worth of a spot in a Software Engineer's toolbox. 21 | 22 | However, few know that Gnuplot is actually also Turing complete programming language. It is similar in syntax to Perl or Awk. So I scratched an itch and solved a [problem](https://adventofcode.com/2023/day/1) with it. 23 | 24 | 25 | ## The problem 26 | 27 | In short, we get a text file where each line contains random ASCII characters. For each line, we must find the first and last digit characters, combine them into an number and at the end output the sum of all these numbers. 28 | 29 | 30 | The way we read the data into a variable is through a shell command: 31 | 32 | ```gnuplot 33 | data = system("cat in.txt") 34 | ``` 35 | 36 | Gnuplot has the `plot` command to turn input data into a plot, but nothing built-in to read input data into a variable, it seems. No matter, `system` which spawns a command in a subshell does the trick. 37 | 38 | Since we need to check whether a character is a string, let's define our own little function for it. Yes, Gnuplot has user defined functions! The unfortunate limitation is that the body has to be an expression: 39 | 40 | ```gnuplot 41 | is_digit(c) = c eq "0" || c eq "1" || c eq "2" || c eq "3" || c eq "4" || c eq "5" || c eq "6" || c eq "7" || c eq "8" || c eq "9" 42 | ``` 43 | 44 | Characters are not a thing; instead we deal with a string of length 1. Comparing strings for equality is done with the operator `eq`. 45 | 46 | Then, we iterate over each line in the data. Gnuplot has a for-each construct we can use for that. 47 | 48 | We then iterate over each character in the line with a for-range loop, isolating the 'character' (remember, it's just a string of length 1) with a slicing syntax that many modern languages have: 49 | 50 | ```gnuplot 51 | sum = 0 52 | 53 | do for [line in data] { 54 | len = strlen(line) 55 | 56 | do for [i = 1:len] { 57 | c = line[i:i] 58 | } 59 | } 60 | ``` 61 | 62 | One thing to note here is that strings are 1-indexed and the slicing syntax is: `foo[start_inclusive:end_inclusive]`. 63 | 64 | We then set `first` to the first digit character we find: 65 | 66 | ```gnuplot 67 | do for [line in data] { 68 | len = strlen(line) 69 | 70 | first= "" 71 | 72 | do for [i = 1:len] { 73 | c = line[i:i] 74 | if (is_digit(c)) { 75 | if (first eq "") { 76 | first = c 77 | break 78 | } 79 | } 80 | } 81 | } 82 | ``` 83 | 84 | We do the same for the last character, iterating in reverse order: 85 | 86 | ```gnuplot 87 | last = "" 88 | 89 | do for [i = len:1:-1] { 90 | c = line[i:i] 91 | if (is_digit(c)) { 92 | if (last eq "") { 93 | last = c 94 | break 95 | } 96 | } 97 | } 98 | ``` 99 | 100 | Finally, we concatenate the two digits (which are still two strings of length 1 at that point) with the `.` operator, convert it to a number with the `+ 0` idiom, and increase our sum: 101 | 102 | ```gnuplot 103 | num = first . last + 0 104 | sum = sum + num 105 | ``` 106 | 107 | We just have to print the sum at the end: 108 | 109 | ```gnuplot 110 | print(sum) 111 | ``` 112 | 113 | ## Closing thoughts 114 | 115 | Pretty straightforward, isn't it? Well, no. The language is weirdly restrictive, for example `sum += num` does not parse. `for` and `while` loops cannot for some reason be used interchangeably due to the weird `do` prefix for for-loops. Very few builtin functions are available. 116 | There does not seem to be basic data structures such as arrays and maps. Every variable is global. And so on. 117 | 118 | It's weird because the language also has very modern constructs that some mainstream languages still do not have, like the slicing syntax. 119 | 120 | Awk, Lua or Perl are honestly better in every way, to pick relatively simple, dynamic languages that people usually reach to for Unixy text transformations. And these will have better tooling, such as a debugger. Heck, even shell scripting is probably easier and more straightforward, and that's a low bar. 121 | 122 | Everything points to the fact that Gnuplot expects it's input data in some prearranged tabular form, and just wants to plot it, not transform it. That means that another (real) programming language is expected to do prior work and Gnuplot is at the end of the data pipeline as a 'dumb' visualization tool. I can also see how the limited language can still be useful for Physicists or Mathematicians to write simple numerical, pure functions e.g. `f(x) = x*2 + 1`. 123 | 124 | 125 | I'll investigate Julia and perhaps R in the future, which are in the same niche of science/data visualization but are full programming languages with plentiful tooling. 126 | 127 | 128 | ## Addendum: The full code 129 | 130 | Run with `gnuplot my_file.dem`. 131 | 132 | ```gnuplot 133 | data = system("cat in.txt") 134 | 135 | is_digit(c) = c eq "0" || c eq "1" || c eq "2" || c eq "3" || c eq "4" || c eq "5" || c eq "6" || c eq "7" || c eq "8" || c eq "9" 136 | 137 | sum = 0 138 | 139 | do for [line in data] { 140 | len = strlen(line) 141 | 142 | first= "" 143 | 144 | do for [i = 1:len] { 145 | c = line[i:i] 146 | if (is_digit(c)) { 147 | if (first eq "") { 148 | first = c 149 | break 150 | } 151 | } 152 | } 153 | 154 | 155 | last = "" 156 | 157 | do for [i = len:1:-1] { 158 | c = line[i:i] 159 | if (is_digit(c)) { 160 | if (last eq "") { 161 | last = c 162 | break 163 | } 164 | } 165 | } 166 | num = first . last + 0 167 | sum = sum + num 168 | } 169 | 170 | print(sum) 171 | ``` 172 | 173 | -------------------------------------------------------------------------------- /header.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 62 | 64 |
65 | -------------------------------------------------------------------------------- /image_size_reduction.md: -------------------------------------------------------------------------------- 1 | Title: Quick and easy PNG image size reduction 2 | Tags: Optimization, PNG 3 | --- 4 | 5 | I seredenpitously noticed that my blog had somewhat big PNG images. But these are just very simple screenshots. There surely must be a way to reduce their size, without affecting their size or legibility? 6 | Well yes, let's quantize them! 7 | What? Quant-what? 8 | 9 | Quoting Wikipedia: 10 | 11 | > Quantization, involved in image processing, is a lossy compression technique achieved by compressing a range of values to a single quantum (discrete) value. When the number of discrete symbols in a given stream is reduced, the stream becomes more compressible. For example, reducing the number of colors required to represent a digital image makes it possible to reduce its file size 12 | 13 | In other words, by picking the right color palette for an image, we can reduce its size without the human eye noticing. For example, an image which has multiple red variants, all very close, are a prime candidate to be converted to the same red color (perhaps the average value) so long as the human eye does not see the difference. Since PNG images use compression, it will compress better. 14 | 15 | At least, that's my layman understanding. 16 | 17 | Fortunately there is an open-source [command line tool](https://github.com/kornelski/pngquant) that is very easy to use and works great. So go give them a star and come back! 18 | 19 | I simply ran the tool on all images to convert them in place in parallel: 20 | 21 | ```sh 22 | $ ls *.png | parallel 'pngquant {} -o {}.tmp && mv {}.tmp {}' 23 | ``` 24 | 25 | It finished instantly, and here is the result: 26 | 27 | ```sh 28 | $ git show 2e126f55a77e75e182ea18b36fb535a0e37793e4 --compact-summary 29 | commit 2e126f55a77e75e182ea18b36fb535a0e37793e4 (HEAD -> master, origin/master, origin/HEAD) 30 | 31 | use pgnquant to shrink images 32 | 33 | feed.png | Bin 167641 -> 63272 bytes 34 | gnuplot.png | Bin 4594 -> 3316 bytes 35 | mem_prof1.png | Bin 157587 -> 59201 bytes 36 | mem_prof2.png | Bin 209046 -> 81028 bytes 37 | mem_prof3.png | Bin 75019 -> 27259 bytes 38 | mem_prof4.png | Bin 50964 -> 21345 bytes 39 | wayland-screenshot-floating.png | Bin 54620 -> 19272 bytes 40 | wayland-screenshot-red.png | Bin 101047 -> 45230 bytes 41 | wayland-screenshot-tiled.png | Bin 188549 -> 107573 bytes 42 | wayland-screenshot-tiled1.png | Bin 505994 -> 170804 bytes 43 | x11_x64_black_window.png | Bin 32977 -> 16898 bytes 44 | x11_x64_final.png | Bin 47985 -> 16650 bytes 45 | 12 files changed, 0 insertions(+), 0 deletions(-) 46 | ``` 47 | 48 | Eye-balling it, every image was on average halved. Not bad, for no visible difference! 49 | 50 | Initially, I wanted to use the new hotness: AVIF. Here's an example using the `avifenc` tool on the original image: 51 | 52 | ```sh 53 | $ avifenc feed.png feed.avif 54 | $ stat -c '%n %s' feed.{png,avif} 55 | feed.png 167641 56 | feed.avif 36034 57 | ``` 58 | 59 | That's almost a x5 reduction in size! However this format is not yet well supported by all browsers. It's recommended to still serve a PNG as fallback, which is a bit too complex for this blog. Still, this format is very promising so I thought I should mention it. 60 | 61 | 62 | So as of now, all PNG images on this blog are much lighter! Not too bad for 10m of work. 63 | 64 | 65 | -------------------------------------------------------------------------------- /index.md: -------------------------------------------------------------------------------- 1 | ## About me 2 | 3 | I am a French/German Senior Software Engineer from France living in Bavaria, Germany. By day, I work for a Fintech company, and by night I write some fun projects in C, Rust, Odin, Zig, and Assembly. I like to work on low-level systems. 4 | 5 | In my free time, I run, lift weights, play music, learn languages, and garden. 6 | 7 | Get in touch, send me an email (link on my Github profile)! 8 | 9 | ## This blog 10 | 11 | This blog is [open-source](https://github.com/gaultier/blog)! If you find a problem, please open a Github issue. 12 | 13 | The content of this blog as well as the code snippets are under the [BSD-3 License](https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22)) which I also usually use for all my personal projects. It's basically free for every use but you have to mention me as the original author. 14 | 15 | I very unregularly post about various technical topics, usually low-level, esoteric and probably not directly applicable to your run-of-the-mill business application. 16 | 17 | This blog does not and will not employ any tracking or cookie of any sort. The only Javascript present is fully optional and only used to do code highlighting and client-side search. 18 | 19 | 20 | ## I am grateful 21 | 22 | I often am frustrated with the state of the software industry; however there is software out there that I am deeply grateful for. 23 | 24 | - Address Sanitizer/Thread Sanitizer. How many projects can revolutionize the way an established industry writes and thinks about native programming languages, and by so doing, impacts the development of new programming languages, such as Rust and Go? And how many projects are massively helpful for beginners and experts alike? Any respectable C or C++ programmer should test every new line of code under these sanitizers, numerous bugs will be caught and your understanding of the standard and undefined behaviour will be greatly improved. Finally, its benefits compound nicely with fuzzing and automated testing. 25 | - Swaywm. A simple and snappy tiling window manager. Using one of these has completely changed the way I interact with a computer. 26 | - C: The OG, simple (for some definition of simple), fast, ubiquitous. It will outlive us all. 27 | - Odin and Zig: Simple and fast. It makes you realize how many features in other programming languages you can live without; they are an exercise in minimalism. 28 | - Vim/Neovim. Another timeless, minimal, fast program that's great for code and prose alike and that respects your computer by requiring very minimal resources. This blog was entirely written inside Neovim! 29 | - ZFS. The last filesystem you'll need. Reliable and is built on the right concepts. It feels like Git for the filesystem. 30 | - Wine/Proton. What a technical feat, and a massive reverse engineering effort, to make Windows applications work on Linux without the application even noticing. 31 | - Rust. Because it catches all my mistakes at compile time and I wish all compilers would too. Although its complexity is a beast. 32 | - Zig the toolchain. Because straightforward cross-compiling should be what everyone does on a daily basis without having to setup Docker/VMs/a complicated CI. 33 | 34 | 35 | ## Portfolio 36 | 37 | *In chronological order, which is also roughly the ascending order of technical difficulty:* 38 | 39 | [Prototype using the Oculus Rift](https://github.com/gaultier/Simulation_Stars_OpenGL): This was my second internship and the first time I owned a project from start to finish. This was a blast and possibly the most fun I had in all my career. This is back in 2014 and the first version of the Oculus Rift was all the hype back then. The goal of the project in the astronomy/CERN lab I was at was to explore how to teach kids about the solar system and astronomy by having them put the Oculus Rift on and experiment it for themselves, flying through the stars. It was great! I got to learn about OpenGL, SDL and multiplying specific 4x4 matrices together to slightly move the camera for each eye so that the VR effect happens and your brain is tricked. 40 | In terms of implementation this is pretty subpar C++ code in retrospect - I went all in on OOP and trying to use every C++ feature under the sun. But it worked, kind of, each star was a cube because I ran out of time. 41 | Part of this project was also to add a VR mode to an existing 3D application written in C; that was quite a big codebase and I did indeed add this mode; however it never worked quite right, there was some constant stuttering which might have been due to loss of precision in floating point calculations. 42 | Overall a ton of fun. 43 | 44 |
45 | 46 | [lox-ocaml](https://github.com/gaultier/lox-ocaml) is the first compiler and interpreter I wrote while following along the excellent [Crafting Interpreters](http://craftinginterpreters.com/the-lox-language.html). The first part of the book is a Java implementation which I wanted to avoid and thus went with OCaml which I anyway wanted to dig deep on. It fit because it remains a garbage collected language like Java and has support for imperative code if I needed to follow closely the official code. It went really well, I had a blast and OCaml is pretty nice; however I will never understand why it pushes the developer to use linked list and it is so hard to use dynamic arrays instead. The ecosystem and tooling is a bit split in half but overall it was a great experience. 47 | 48 |
49 | 50 | [kotlin-rs](https://github.com/gaultier/kotlin-rs) is a Kotlin compiler written in Rust and my first attempt at it. It is not finished but can still compile Kotlin functions, expressions and statements to JVM bytecode and run that. 51 | This was my first compiler project on my own, I chose Kotlin because I was working with Kotlin at the time and was suffering from the excruciatingly slow official compiler at the time (it improved somewhat since). It allowed me to skip the language design part and focus on the implementation. 52 | I think it still holds up to this day although I would definitely change how the AST is modeled and avoid having each node being a `Box` (see below on another take on the subject). One big topic I did not tackle was generating Stack Map Frames which are a JVM concept to verify bytecode in linear time before running it. 53 | 54 |
55 | 56 | [microkt](https://github.com/gaultier/microkt) is my second take on a Kotlin compiler, this time written in C and targeting x86_64 (no JVM). I think it is nice to have the luxury to revisit a past project in a different language and reevaluating past choices. I got about equally as far as the `kotlin-rs` in terms of language support and stopped while implementing the Garbage Collector which is an can of worms by itself. The reason I stopped is that I noticed I was being regularly stopped in my tracks by bugs in the generated x86_64 assembly code, notably due to 'move with zero extend' issues and register allocation. I thus decided to dig deep on this subject before returning to the compiler which I never did because I got two kids right after. 57 | The implementation still holds, however I would definitely revist the assembly generation part as mentioned and use a structured representation instead of a textual one (no one wants to do string programming). 58 | 59 |
60 | 61 | [My C monorepo](https://github.com/gaultier/c) is a big mix a small and big programs all written in C. The most noteworthy are: 62 | 63 | - `torrent`: A bittorrent client. It works well but only handles one file and is probably a nest of security vulnerabilities now that I think of it 64 | - `crash-reporter`: A crash reporter for macOS x86_64. It gives a full stacktrace at any point in the program. It does so by parsing the DWARF information embedded in the mach-o executable format. 65 | - `clone-gitlab-api`: A small CLI that downloads all projects from a Gitlab instance. Especially useful since my employer at the time had all projects hosted on a private Gitlab instance and there was no search feature. So I figured that the easiest way is to fetch the list of all projects, clone them locally and use grep on them. I worked great! This is conceptually `curl` + `parallel` + `git clone/pull`. I additionally wrote an equivalent [Rust version](https://github.com/gaultier/gitlab-clone-all). I also later wrote [something similar](https://github.com/gaultier/gitlab-events) to poll Gitlab for notifications to watch repository changes. Yeah Gitlab is/was pretty limited in functionality. 66 | 67 |
68 | 69 | [micro-kotlin](https://github.com/gaultier/micro-kotlin): The third (and last?) take on a Kotlin compiler, in C. It goes much further than the two previous attempts both in terms of language support and in terms of implementation: 70 | 71 | - Expressions, statements, control flow, and functions (including recursion) are implemented 72 | - It implements Stack Map Frames contrary the [kotlin-rs](https://github.com/gaultier/kotlin-rs) so the latest JVM versions work seamlessly with it and bytecode generated by the compiler is verified at load time by the JVM 73 | - It implements type constraints in the type checker to infer which function the user intended to call (and the rules in Kotlin for that are so very complex). That's one of the thorniest topics in Kotlin due to the language trying to have every programming language feature under the sun, namely: type inference, function overloading, default parameters, generics, implicit callers (`it` and such), variadic functions, object inheritance with method overriding, etc. 74 | - It explores the class path so that external libraries can be use including the Java and Kotlin standard libraries 75 | - Some Java/Kotlin interop is supported (mostly, calling Java functions from Kotlin) 76 | - It parses and understands .jmod, .class, .jar files, even with zlib compression 77 | - Out-of-order definitions of functions and variables are supported 78 | - Some support for function inlining is supported (inlining the body of a called function) 79 | - All allocations are done with an arena allocator and there is support for a memory dump with stacktraces 80 | - It's only 10k lines of code, it ony needs ~10 MiB of memory to compile real programs, and the final stripped executable for the compiler is ~180 Kib! 81 | 82 | 83 | I think my most noteworthy projects are compilers both because I tend to be attracted to that domain and also because small CLI tools are less interesting. Compilers for real programming languages are hard! 84 | 85 | ## Resume 86 | 87 | You can find my resume [online](https://gaultier.github.io/resume/resume) 88 | or download it as [PDF](https://github.com/gaultier/resume/raw/master/Philippe_Gaultier_resume_en.pdf). 89 | -------------------------------------------------------------------------------- /kahns_algorithm.js: -------------------------------------------------------------------------------- 1 | const adjacencyMatrix = [ 2 | [0, 0, 1, 0, 0, 0], 3 | [1, 0, 0, 0, 0, 0], 4 | [0, 0, 0, 0, 0, 0], 5 | [0, 0, 1, 0, 0, 0], 6 | [1, 0, 0, 0, 0, 0], 7 | [0, 0, 0, 1, 0, 0], 8 | ]; 9 | 10 | const nodes = ["Angela", "Bella", "Ellen", "Jane", "Miranda", "Zoe"]; 11 | 12 | function hasNodeNoIncomingEdge(adjacencyMatrix, nodeIndex) { 13 | const column = nodeIndex; 14 | 15 | for (let row = 0; row < adjacencyMatrix.length; row += 1) { 16 | const cell = adjacencyMatrix[row][column]; 17 | 18 | if (cell != 0) { 19 | return false; 20 | } 21 | } 22 | 23 | return true; 24 | } 25 | 26 | function getNodesWithNoIncomingEdge(adjacencyMatrix, nodes) { 27 | return nodes.filter((_, i) => hasNodeNoIncomingEdge(adjacencyMatrix, i)); 28 | } 29 | 30 | function graphHasEdges(adjacencyMatrix) { 31 | for (let row = 0; row < adjacencyMatrix.length; row += 1) { 32 | for (let column = 0; column < adjacencyMatrix.length; column += 1) { 33 | if (adjacencyMatrix[row][column] == 1) return true; 34 | } 35 | } 36 | 37 | return false; 38 | } 39 | 40 | function topologicalSort(adjacencyMatrix) { 41 | const L = []; 42 | const S = getNodesWithNoIncomingEdge(adjacencyMatrix, nodes); 43 | 44 | while (S.length > 0) { 45 | const node = S.pop(); 46 | L.push(node); 47 | const nodeIndex = nodes.indexOf(node); 48 | 49 | for (let mIndex = 0; mIndex < nodes.length; mIndex += 1) { 50 | const hasEdgeFromNtoM = adjacencyMatrix[nodeIndex][mIndex]; 51 | if (!hasEdgeFromNtoM) continue; 52 | 53 | adjacencyMatrix[nodeIndex][mIndex] = 0; 54 | 55 | if (hasNodeNoIncomingEdge(adjacencyMatrix, mIndex)) { 56 | const m = nodes[mIndex]; 57 | S.push(m); 58 | } 59 | } 60 | } 61 | 62 | if (graphHasEdges(adjacencyMatrix)) { 63 | throw new Error("Graph has at least one cycle"); 64 | } 65 | 66 | return L; 67 | } 68 | 69 | function hasMultipleRoots(adjacencyMatrix) { 70 | let countOfRowsWithOnlyZeroes = 0; 71 | 72 | for (let row = 0; row < adjacencyMatrix.length; row += 1) { 73 | let rowHasOnlyZeroes = true; 74 | for (let column = 0; column < adjacencyMatrix.length; column += 1) { 75 | if (adjacencyMatrix[row][column] != 0) { 76 | rowHasOnlyZeroes = false; 77 | break; 78 | } 79 | } 80 | if (rowHasOnlyZeroes) countOfRowsWithOnlyZeroes += 1; 81 | } 82 | 83 | return countOfRowsWithOnlyZeroes > 1; 84 | } 85 | 86 | console.log(hasMultipleRoots(adjacencyMatrix)); 87 | const employeesTopologicallySorted = topologicalSort(structuredClone(adjacencyMatrix), nodes); 88 | console.log(employeesTopologicallySorted); 89 | 90 | const root = employeesTopologicallySorted[employeesTopologicallySorted.length - 1]; 91 | console.log(`INSERT INTO people VALUES("${root}", NULL)`); 92 | 93 | for (let i = employeesTopologicallySorted.length - 2; i >= 0; i -= 1) { 94 | const employee = employeesTopologicallySorted[i]; 95 | const employeeIndex = nodes.indexOf(employee); 96 | 97 | const managerIndex = adjacencyMatrix[employeeIndex].indexOf(1); 98 | const manager = nodes[managerIndex]; 99 | console.log( 100 | `INSERT INTO people SELECT "${employee}", rowid FROM people WHERE name = "${manager}" LIMIT 1;`, 101 | ); 102 | } 103 | -------------------------------------------------------------------------------- /kahns_algorithm_1.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO"]; 5 | engineer_1[label="Bella, Software Engineer"]; 6 | engineer_2[label="Miranda, Software Engineer"]; 7 | accountant_1[label="Zoe, Accountant"]; 8 | 9 | 10 | cto-> ceo; 11 | cfo -> ceo; 12 | engineer_1 -> cto; 13 | engineer_2 -> cto; 14 | accountant_1 -> cfo; 15 | } 16 | -------------------------------------------------------------------------------- /kahns_algorithm_1.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | cto 33 | 34 | Angela, CTO 35 | 36 | 37 | 38 | cto->ceo 39 | 40 | 41 | 42 | 43 | 44 | engineer_1 45 | 46 | Bella, Software Engineer 47 | 48 | 49 | 50 | engineer_1->cto 51 | 52 | 53 | 54 | 55 | 56 | engineer_2 57 | 58 | Miranda, Software Engineer 59 | 60 | 61 | 62 | engineer_2->cto 63 | 64 | 65 | 66 | 67 | 68 | accountant_1 69 | 70 | Zoe, Accountant 71 | 72 | 73 | 74 | accountant_1->cfo 75 | 76 | 77 | 78 | 79 | 80 | -------------------------------------------------------------------------------- /kahns_algorithm_1_invalid.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO"]; 5 | engineer_1[label="Bella, Software Engineer"]; 6 | engineer_2[label="Miranda, Software Engineer"]; 7 | accountant_1[label="Zoe, Accountant"]; 8 | 9 | 10 | cto-> ceo; 11 | cfo -> ceo; 12 | engineer_1 -> cto; 13 | engineer_2 -> cto; 14 | accountant_1 -> cfo; 15 | accountant_1 -> cto[label="Forbidden", fontcolor="red", color="red"]; 16 | } 17 | -------------------------------------------------------------------------------- /kahns_algorithm_1_invalid.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | cto 33 | 34 | Angela, CTO 35 | 36 | 37 | 38 | cto->ceo 39 | 40 | 41 | 42 | 43 | 44 | engineer_1 45 | 46 | Bella, Software Engineer 47 | 48 | 49 | 50 | engineer_1->cto 51 | 52 | 53 | 54 | 55 | 56 | engineer_2 57 | 58 | Miranda, Software Engineer 59 | 60 | 61 | 62 | engineer_2->cto 63 | 64 | 65 | 66 | 67 | 68 | accountant_1 69 | 70 | Zoe, Accountant 71 | 72 | 73 | 74 | accountant_1->cfo 75 | 76 | 77 | 78 | 79 | 80 | accountant_1->cto 81 | 82 | 83 | Forbidden 84 | 85 | 86 | 87 | -------------------------------------------------------------------------------- /kahns_algorithm_2.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO"]; 5 | engineer_1[label="Bella, Software Engineer"]; 6 | engineer_2[label="Miranda, Software Engineer"]; 7 | accountant_1[label="Zoe, Accountant", fillcolor="turquoise", style="filled"]; 8 | 9 | 10 | cto-> ceo; 11 | cfo -> ceo; 12 | engineer_1 -> cto; 13 | engineer_2 -> cto; 14 | accountant_1 -> cfo[style="dotted", label="Removed", color="gray", fontcolor="grey"]; 15 | accountant_1 -> cto; 16 | } 17 | -------------------------------------------------------------------------------- /kahns_algorithm_2.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | cto 33 | 34 | Angela, CTO 35 | 36 | 37 | 38 | cto->ceo 39 | 40 | 41 | 42 | 43 | 44 | engineer_1 45 | 46 | Bella, Software Engineer 47 | 48 | 49 | 50 | engineer_1->cto 51 | 52 | 53 | 54 | 55 | 56 | engineer_2 57 | 58 | Miranda, Software Engineer 59 | 60 | 61 | 62 | engineer_2->cto 63 | 64 | 65 | 66 | 67 | 68 | accountant_1 69 | 70 | Zoe, Accountant 71 | 72 | 73 | 74 | accountant_1->cfo 75 | 76 | 77 | Removed 78 | 79 | 80 | 81 | accountant_1->cto 82 | 83 | 84 | 85 | 86 | 87 | -------------------------------------------------------------------------------- /kahns_algorithm_2_1.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO"]; 5 | engineer_1[label="Bella, Software Engineer"]; 6 | engineer_2[label="Miranda, Software Engineer"]; 7 | accountant_1[label="Zoe, Accountant", fillcolor="turquoise", style="filled"]; 8 | 9 | 10 | cto-> ceo; 11 | cfo -> ceo; 12 | engineer_1 -> cto; 13 | engineer_2 -> cto; 14 | accountant_1 -> cfo[style="dotted", label="Removed", color="gray", fontcolor="grey"]; 15 | accountant_1 -> cto[style="dotted", label="Removed", color="gray", fontcolor="grey"]; 16 | } 17 | -------------------------------------------------------------------------------- /kahns_algorithm_2_1.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | cto 33 | 34 | Angela, CTO 35 | 36 | 37 | 38 | cto->ceo 39 | 40 | 41 | 42 | 43 | 44 | engineer_1 45 | 46 | Bella, Software Engineer 47 | 48 | 49 | 50 | engineer_1->cto 51 | 52 | 53 | 54 | 55 | 56 | engineer_2 57 | 58 | Miranda, Software Engineer 59 | 60 | 61 | 62 | engineer_2->cto 63 | 64 | 65 | 66 | 67 | 68 | accountant_1 69 | 70 | Zoe, Accountant 71 | 72 | 73 | 74 | accountant_1->cfo 75 | 76 | 77 | Removed 78 | 79 | 80 | 81 | accountant_1->cto 82 | 83 | 84 | Removed 85 | 86 | 87 | 88 | -------------------------------------------------------------------------------- /kahns_algorithm_2_2.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO"]; 5 | engineer_1[label="Bella, Software Engineer", fillcolor="turquoise", style="filled"]; 6 | engineer_2[label="Miranda, Software Engineer"]; 7 | 8 | 9 | cto-> ceo; 10 | cfo -> ceo; 11 | engineer_1 -> cto; 12 | engineer_2 -> cto; 13 | } 14 | -------------------------------------------------------------------------------- /kahns_algorithm_2_2.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | cto 33 | 34 | Angela, CTO 35 | 36 | 37 | 38 | cto->ceo 39 | 40 | 41 | 42 | 43 | 44 | engineer_1 45 | 46 | Bella, Software Engineer 47 | 48 | 49 | 50 | engineer_1->cto 51 | 52 | 53 | 54 | 55 | 56 | engineer_2 57 | 58 | Miranda, Software Engineer 59 | 60 | 61 | 62 | engineer_2->cto 63 | 64 | 65 | 66 | 67 | 68 | -------------------------------------------------------------------------------- /kahns_algorithm_2_3.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO"]; 5 | engineer_1[label="Bella, Software Engineer", fillcolor="turquoise", style="filled"]; 6 | engineer_2[label="Miranda, Software Engineer"]; 7 | 8 | 9 | cto-> ceo; 10 | cfo -> ceo; 11 | engineer_1 -> cto[style="dotted", label="Removed", color="gray", fontcolor="grey"]; 12 | engineer_2 -> cto; 13 | } 14 | -------------------------------------------------------------------------------- /kahns_algorithm_2_3.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | cto 33 | 34 | Angela, CTO 35 | 36 | 37 | 38 | cto->ceo 39 | 40 | 41 | 42 | 43 | 44 | engineer_1 45 | 46 | Bella, Software Engineer 47 | 48 | 49 | 50 | engineer_1->cto 51 | 52 | 53 | Removed 54 | 55 | 56 | 57 | engineer_2 58 | 59 | Miranda, Software Engineer 60 | 61 | 62 | 63 | engineer_2->cto 64 | 65 | 66 | 67 | 68 | 69 | -------------------------------------------------------------------------------- /kahns_algorithm_2_4.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO"]; 5 | engineer_2[label="Miranda, Software Engineer", fillcolor="turquoise", style="filled"]; 6 | 7 | 8 | cto-> ceo; 9 | cfo -> ceo; 10 | engineer_2 -> cto; 11 | } 12 | -------------------------------------------------------------------------------- /kahns_algorithm_2_4.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | cto 33 | 34 | Angela, CTO 35 | 36 | 37 | 38 | cto->ceo 39 | 40 | 41 | 42 | 43 | 44 | engineer_2 45 | 46 | Miranda, Software Engineer 47 | 48 | 49 | 50 | engineer_2->cto 51 | 52 | 53 | 54 | 55 | 56 | -------------------------------------------------------------------------------- /kahns_algorithm_2_5.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO"]; 5 | engineer_2[label="Miranda, Software Engineer", fillcolor="turquoise", style="filled"]; 6 | 7 | 8 | cto-> ceo; 9 | cfo -> ceo; 10 | engineer_2 -> cto[style="dotted", label="Removed", color="gray", fontcolor="grey"]; 11 | } 12 | -------------------------------------------------------------------------------- /kahns_algorithm_2_5.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | cto 33 | 34 | Angela, CTO 35 | 36 | 37 | 38 | cto->ceo 39 | 40 | 41 | 42 | 43 | 44 | engineer_2 45 | 46 | Miranda, Software Engineer 47 | 48 | 49 | 50 | engineer_2->cto 51 | 52 | 53 | Removed 54 | 55 | 56 | 57 | -------------------------------------------------------------------------------- /kahns_algorithm_2_6.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO", fillcolor="turquoise", style="filled"]; 5 | 6 | 7 | cto-> ceo; 8 | cfo -> ceo; 9 | } 10 | -------------------------------------------------------------------------------- /kahns_algorithm_2_6.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | cto 33 | 34 | Angela, CTO 35 | 36 | 37 | 38 | cto->ceo 39 | 40 | 41 | 42 | 43 | 44 | -------------------------------------------------------------------------------- /kahns_algorithm_2_7.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO", fillcolor="turquoise", style="filled"]; 5 | 6 | 7 | cto-> ceo[style="dotted", label="Removed", color="gray", fontcolor="grey"]; 8 | cfo -> ceo; 9 | } 10 | -------------------------------------------------------------------------------- /kahns_algorithm_2_7.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | cto 33 | 34 | Angela, CTO 35 | 36 | 37 | 38 | cto->ceo 39 | 40 | 41 | Removed 42 | 43 | 44 | 45 | -------------------------------------------------------------------------------- /kahns_algorithm_2_8.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO", fillcolor="turquoise", style="filled"]; 4 | 5 | 6 | cfo -> ceo; 7 | } 8 | -------------------------------------------------------------------------------- /kahns_algorithm_2_8.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | 30 | 31 | 32 | -------------------------------------------------------------------------------- /kahns_algorithm_2_9.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO", fillcolor="turquoise", style="filled"]; 4 | 5 | 6 | cfo -> ceo[style="dotted", label="Removed", color="gray", fontcolor="grey"]; 7 | } 8 | -------------------------------------------------------------------------------- /kahns_algorithm_2_9.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | cfo 21 | 22 | Jane, CFO 23 | 24 | 25 | 26 | cfo->ceo 27 | 28 | 29 | Removed 30 | 31 | 32 | 33 | -------------------------------------------------------------------------------- /kahns_algorithm_3.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | ceo_2[label="Kelly, CEO", color="red", fontcolor="red"]; 4 | cfo[label="Jane, CFO"]; 5 | cto[label="Angela, CTO"]; 6 | engineer_1[label="Bella, Software Engineer"]; 7 | engineer_2[label="Miranda, Software Engineer"]; 8 | accountant_1[label="Zoe, Accountant"]; 9 | 10 | 11 | cto-> ceo; 12 | cfo -> ceo; 13 | cto-> ceo_2[label="Forbidden", color="red", fontcolor="red"]; 14 | engineer_1 -> cto; 15 | engineer_2 -> cto; 16 | accountant_1 -> cfo; 17 | } 18 | -------------------------------------------------------------------------------- /kahns_algorithm_3.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | ceo_2 21 | 22 | Kelly, CEO 23 | 24 | 25 | 26 | cfo 27 | 28 | Jane, CFO 29 | 30 | 31 | 32 | cfo->ceo 33 | 34 | 35 | 36 | 37 | 38 | cto 39 | 40 | Angela, CTO 41 | 42 | 43 | 44 | cto->ceo 45 | 46 | 47 | 48 | 49 | 50 | cto->ceo_2 51 | 52 | 53 | Forbidden 54 | 55 | 56 | 57 | engineer_1 58 | 59 | Bella, Software Engineer 60 | 61 | 62 | 63 | engineer_1->cto 64 | 65 | 66 | 67 | 68 | 69 | engineer_2 70 | 71 | Miranda, Software Engineer 72 | 73 | 74 | 75 | engineer_2->cto 76 | 77 | 78 | 79 | 80 | 81 | accountant_1 82 | 83 | Zoe, Accountant 84 | 85 | 86 | 87 | accountant_1->cfo 88 | 89 | 90 | 91 | 92 | 93 | -------------------------------------------------------------------------------- /kahns_algorithm_4.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | ceo[label="Ellen, CEO"]; 3 | cfo[label="Jane, CFO"]; 4 | cto[label="Angela, CTO"]; 5 | engineer_1[label="Bella, Software Engineer"]; 6 | engineer_2[label="Miranda, Software Engineer"]; 7 | accountant_1[label="Zoe, Accountant"]; 8 | 9 | 10 | cto-> ceo; 11 | cfo -> ceo; 12 | engineer_1 -> cto; 13 | engineer_2 -> cto; 14 | accountant_1 -> cfo; 15 | ceo -> accountant_1[label="Forbidden", fontcolor="red", color="red"]; 16 | } 17 | -------------------------------------------------------------------------------- /kahns_algorithm_4.svg: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | %3 11 | 12 | 13 | 14 | ceo 15 | 16 | Ellen, CEO 17 | 18 | 19 | 20 | accountant_1 21 | 22 | Zoe, Accountant 23 | 24 | 25 | 26 | ceo->accountant_1 27 | 28 | 29 | Forbidden 30 | 31 | 32 | 33 | cfo 34 | 35 | Jane, CFO 36 | 37 | 38 | 39 | cfo->ceo 40 | 41 | 42 | 43 | 44 | 45 | cto 46 | 47 | Angela, CTO 48 | 49 | 50 | 51 | cto->ceo 52 | 53 | 54 | 55 | 56 | 57 | engineer_1 58 | 59 | Bella, Software Engineer 60 | 61 | 62 | 63 | engineer_1->cto 64 | 65 | 66 | 67 | 68 | 69 | engineer_2 70 | 71 | Miranda, Software Engineer 72 | 73 | 74 | 75 | engineer_2->cto 76 | 77 | 78 | 79 | 80 | 81 | accountant_1->cfo 82 | 83 | 84 | 85 | 86 | 87 | -------------------------------------------------------------------------------- /main.css: -------------------------------------------------------------------------------- 1 | input { 2 | padding: .3rem; 3 | margin: auto; 4 | } 5 | 6 | .publication-date { 7 | font-style: italic; 8 | color: gray; 9 | } 10 | 11 | .article-title { 12 | display: inline-flex; 13 | justify-content: space-between; 14 | width: 100%; 15 | align-items: center; 16 | } 17 | 18 | .article-prelude { 19 | display: inline-flex; 20 | justify-content: space-between; 21 | width: 100%; 22 | } 23 | 24 | p { 25 | padding: .5rem; 26 | } 27 | 28 | #banner { 29 | background-color: #FB8C00; 30 | display: inline-flex; 31 | width: 100%; 32 | justify-content: space-between; 33 | align-items: center; 34 | flex-wrap: wrap; 35 | } 36 | 37 | #banner > ul { 38 | list-style: none; 39 | padding: 0 .5rem; 40 | display: inline-flex; 41 | flex-wrap: wrap; 42 | justify-content: space-evenly; 43 | gap: .5rem; 44 | } 45 | 46 | #banner > ul > li { 47 | display: inline-flex; 48 | align-items: center; 49 | } 50 | 51 | #banner > ul > li > a:hover { 52 | background-color: #FFA726; 53 | } 54 | 55 | #banner > ul > li > a { 56 | text-decoration: none; 57 | border-radius: 8px; 58 | border: 15px; 59 | padding: 0; 60 | display: inline-block; 61 | padding: .2rem; 62 | } 63 | 64 | #banner > ul > li > a > svg { 65 | width: 2rem; 66 | height: 2rem; 67 | } 68 | 69 | #me { 70 | border-radius: 231px; 71 | border: 15px; 72 | padding: .5rem; 73 | max-width: 6rem; 74 | margin: 0; 75 | margin-right: 1rem; 76 | } 77 | 78 | #pseudo-body, #search-matches { 79 | padding: .7rem; 80 | } 81 | 82 | .articles > ul { 83 | list-style: none; 84 | width: 100%; 85 | padding: 0; 86 | } 87 | 88 | .articles > ul > li { 89 | display: inline-flex; 90 | align-items: center; 91 | width: 100%; 92 | } 93 | 94 | .home-link { 95 | margin-right: 1rem; 96 | } 97 | 98 | .home-link > a { 99 | text-decoration: none; 100 | color: #E65100; 101 | font-weight: bold; 102 | } 103 | 104 | .tag { 105 | border-radius: 10px; 106 | background-color: #BDBDBD; 107 | padding: 0.1rem 0.5rem; 108 | color: white; 109 | text-decoration: none; 110 | margin: .2rem; 111 | } 112 | 113 | .tags { 114 | display: inline-flex; 115 | flex-wrap: wrap; 116 | flex-shrink: 5; 117 | } 118 | 119 | .date { 120 | color: #9E9E9E; 121 | } 122 | 123 | #name { 124 | font-weight: bolder; 125 | display: inline-flex; 126 | flex-wrap: wrap; 127 | align-items: center; 128 | margin: 0; 129 | color: black; 130 | border-radius: 8px; 131 | border: 15px; 132 | } 133 | 134 | figure { 135 | display: flex; 136 | justify-content: center; 137 | flex-wrap: wrap; 138 | } 139 | 140 | figure > figcaption { 141 | width: 100%; 142 | text-align: center; 143 | } 144 | 145 | img, video { 146 | max-width: 90%; 147 | padding: 1rem; 148 | display: block; 149 | margin: auto; 150 | } 151 | 152 | hr { 153 | max-width: 30%; 154 | margin-top: 3rem; 155 | margin-bottom: 3rem; 156 | } 157 | 158 | body { 159 | margin: 0; 160 | font-family: monospace, sans-serif; 161 | margin-bottom: 5rem; 162 | } 163 | 164 | li { 165 | margin: 1rem; 166 | } 167 | 168 | blockquote { 169 | padding: 1rem; 170 | border-bottom: 1px solid gray; 171 | border-top: 1px solid gray; 172 | margin: auto; 173 | margin-top: 3rem; 174 | margin-bottom: 3rem; 175 | max-width: 80%; 176 | color: #4c566a; 177 | padding: .1rem; 178 | text-align: center; 179 | } 180 | 181 | pre { 182 | background: #FFF3E0 !important; 183 | white-space: pre-wrap; 184 | border: 1px solid grey; 185 | padding: .3rem !important; 186 | } 187 | 188 | code { 189 | font-size: .9rem; 190 | background: #FFF3E0 !important; 191 | background: #FFF3E0 !important; 192 | word-wrap: break-word; 193 | padding: 0 !important; 194 | } 195 | 196 | object { 197 | padding: 1rem .5rem; 198 | max-width: 95%; 199 | min-width: 60%; 200 | display: block; 201 | margin: auto; 202 | } 203 | 204 | pre > code.code-no-line-numbers { 205 | padding: .5rem !important; 206 | } 207 | 208 | code.code-no-line-numbers { 209 | padding: .1rem !important; 210 | border-radius: .2rem; 211 | } 212 | 213 | 214 | table { 215 | border-collapse: collapse; 216 | word-break: break-word; 217 | /* Center. */ 218 | margin: auto; 219 | } 220 | 221 | figcaption { 222 | font-style: italic; 223 | font-weight: lighter; 224 | opacity: .8; 225 | } 226 | 227 | td, th { 228 | border: 1px solid black; 229 | padding: 0.5rem; 230 | text-align: center; 231 | } 232 | 233 | th { 234 | background: rgb(255,250,240); 235 | } 236 | 237 | h1 { 238 | margin: 3rem 2rem 3rem 0; 239 | flex-basis: 90%; 240 | } 241 | 242 | h2, h3, h4, h5, h6 { 243 | margin: 3rem 2rem 1rem 0; 244 | } 245 | 246 | a.title { 247 | text-decoration: none !important; 248 | color: initial; 249 | } 250 | 251 | details { 252 | display: inline-flex; 253 | } 254 | 255 | summary:hover { 256 | cursor: pointer; 257 | } 258 | 259 | a.title:hover { 260 | cursor: pointer; 261 | color: blue; 262 | text-decoration: underline !important; 263 | } 264 | 265 | .hash-anchor { 266 | opacity: 0; 267 | color: blue; 268 | } 269 | 270 | .title:hover + .hash-anchor { 271 | opacity: 1; 272 | } 273 | 274 | .hash-anchor:hover { 275 | opacity: 1; 276 | } 277 | 278 | .hash-anchor::before { 279 | margin-left: .5rem; 280 | content: '#'; 281 | } 282 | 283 | p:has(img):has(em) { 284 | padding-bottom: 2.5rem; 285 | } 286 | 287 | #donate { 288 | margin-top: 8rem; 289 | text-align: center; 290 | } 291 | 292 | .line-number { 293 | width: 2rem; 294 | display: inline-block; 295 | text-align: right; 296 | padding: .2rem; 297 | margin-right: 1.2rem; 298 | } 299 | 300 | @media only screen and (max-width: 450px) { 301 | .line-number { display: none; } 302 | code { padding-left: 0.3rem !important; } 303 | } 304 | 305 | details { 306 | width: fit-content; 307 | border-radius: .5rem; 308 | background-color: #BDBDBD6B; 309 | margin: 0 auto; 310 | padding: .5rem; 311 | display: block; 312 | } 313 | 314 | summary { 315 | width: 100%; 316 | text-align: center; 317 | font-weight: bold; 318 | } 319 | 320 | .footnotes::before { 321 | content: "Footnotes:"; 322 | } 323 | 324 | .footnotes { 325 | border-top: 1px solid grey; 326 | border-bottom: 1px solid grey; 327 | padding: 1rem; 328 | margin: 2rem 1rem; 329 | } 330 | -------------------------------------------------------------------------------- /me.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/me.jpeg -------------------------------------------------------------------------------- /mem_prof1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/mem_prof1.png -------------------------------------------------------------------------------- /mem_prof2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/mem_prof2.png -------------------------------------------------------------------------------- /mem_prof3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/mem_prof3.png -------------------------------------------------------------------------------- /mem_prof4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/mem_prof4.png -------------------------------------------------------------------------------- /mem_profile.c: -------------------------------------------------------------------------------- 1 | #define _GNU_SOURCE 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | 11 | typedef struct { 12 | uint64_t in_use_space, in_use_objects, alloc_space, alloc_objects; 13 | uint64_t *call_stack; 14 | uint64_t call_stack_len; 15 | } mem_record_t; 16 | 17 | typedef struct mem_profile mem_profile_t; 18 | typedef struct { 19 | uint8_t *start; 20 | uint8_t *end; 21 | mem_profile_t *profile; 22 | } arena_t; 23 | 24 | struct mem_profile { 25 | mem_record_t *records; 26 | uint64_t records_len; 27 | uint64_t records_cap; 28 | uint64_t in_use_space, in_use_objects, alloc_space, alloc_objects; 29 | arena_t arena; 30 | }; 31 | 32 | static void *arena_alloc(arena_t *a, size_t size, size_t align, size_t count); 33 | 34 | static uint8_t record_call_stack(uint64_t *dst, uint64_t cap) { 35 | uintptr_t *rbp = __builtin_frame_address(0); 36 | 37 | uint64_t len = 0; 38 | 39 | while (rbp != 0 && ((uint64_t)rbp & 7) == 0 && *rbp != 0) { 40 | const uintptr_t rip = *(rbp + 1); 41 | rbp = (uintptr_t *)*rbp; 42 | 43 | // `rip` points to the return instruction in the caller, once this call is 44 | // done. But: We want the location of the call i.e. the `call xxx` 45 | // instruction, so we subtract one byte to point inside it, which is not 46 | // quite 'at' it, but good enough. 47 | dst[len++] = rip - 1; 48 | 49 | if (len >= cap) 50 | return len; 51 | } 52 | return len; 53 | } 54 | static void mem_profile_record_alloc(mem_profile_t *profile, 55 | uint64_t objects_count, 56 | uint64_t bytes_count) { 57 | // Record the call stack by stack walking. 58 | uint64_t call_stack[64] = {0}; 59 | uint64_t call_stack_len = 60 | record_call_stack(call_stack, sizeof(call_stack) / sizeof(call_stack[0])); 61 | 62 | // Update the sums. 63 | profile->alloc_objects += objects_count; 64 | profile->alloc_space += bytes_count; 65 | profile->in_use_objects += objects_count; 66 | profile->in_use_space += bytes_count; 67 | 68 | // Upsert the record. 69 | for (uint64_t i = 0; i < profile->records_len; i++) { 70 | mem_record_t *r = &profile->records[i]; 71 | 72 | if (r->call_stack_len == call_stack_len && 73 | memcmp(r->call_stack, call_stack, call_stack_len * sizeof(uint64_t)) == 74 | 0) { 75 | // Found an existing record, update it. 76 | r->alloc_objects += objects_count; 77 | r->alloc_space += bytes_count; 78 | r->in_use_objects += objects_count; 79 | r->in_use_space += bytes_count; 80 | return; 81 | } 82 | } 83 | 84 | // Not found, insert a new record. 85 | mem_record_t record = { 86 | .alloc_objects = objects_count, 87 | .alloc_space = bytes_count, 88 | .in_use_objects = objects_count, 89 | .in_use_space = bytes_count, 90 | }; 91 | record.call_stack = arena_alloc(&profile->arena, sizeof(uint64_t), 92 | _Alignof(uint64_t), call_stack_len); 93 | memcpy(record.call_stack, call_stack, call_stack_len * sizeof(uint64_t)); 94 | record.call_stack_len = call_stack_len; 95 | 96 | if (profile->records_len >= profile->records_cap) { 97 | uint64_t new_cap = profile->records_cap * 2; 98 | // Grow the array. 99 | mem_record_t *new_records = arena_alloc( 100 | &profile->arena, sizeof(mem_record_t), _Alignof(mem_record_t), new_cap); 101 | memcpy(new_records, profile->records, 102 | profile->records_len * sizeof(mem_record_t)); 103 | profile->records_cap = new_cap; 104 | profile->records = new_records; 105 | } 106 | profile->records[profile->records_len++] = record; 107 | } 108 | 109 | static void mem_profile_write(mem_profile_t *profile, FILE *out) { 110 | fprintf(out, "heap profile: %lu: %lu [ %lu: %lu] @ heapprofile\n", 111 | profile->in_use_objects, profile->in_use_space, 112 | profile->alloc_objects, profile->alloc_space); 113 | 114 | for (uint64_t i = 0; i < profile->records_len; i++) { 115 | mem_record_t r = profile->records[i]; 116 | 117 | fprintf(out, "%lu: %lu [%lu: %lu] @ ", r.in_use_objects, r.in_use_space, 118 | r.alloc_objects, r.alloc_space); 119 | 120 | for (uint64_t j = 0; j < r.call_stack_len; j++) { 121 | fprintf(out, "%#lx ", r.call_stack[j]); 122 | } 123 | fputc('\n', out); 124 | } 125 | 126 | fputs("\nMAPPED_LIBRARIES:\n", out); 127 | 128 | static uint8_t mem[4096] = {0}; 129 | int fd = open("/proc/self/maps", O_RDONLY); 130 | assert(fd != -1); 131 | ssize_t read_bytes = read(fd, mem, sizeof(mem)); 132 | assert(read_bytes != -1); 133 | close(fd); 134 | 135 | fwrite(mem, 1, read_bytes, out); 136 | 137 | fflush(out); 138 | } 139 | 140 | static void *arena_alloc(arena_t *a, size_t size, size_t align, size_t count) { 141 | size_t available = a->end - a->start; 142 | size_t padding = -(size_t)a->start & (align - 1); 143 | 144 | size_t offset = padding + size * count; 145 | if (available < offset) { 146 | fprintf(stderr, 147 | "Out of memory: available=%lu " 148 | "allocation_size=%lu\n", 149 | available, offset); 150 | abort(); 151 | } 152 | 153 | uint8_t *res = a->start + padding; 154 | 155 | a->start += offset; 156 | 157 | if (a->profile) { 158 | mem_profile_record_alloc(a->profile, count, offset); 159 | } 160 | 161 | return (void *)res; 162 | } 163 | 164 | static arena_t arena_new(uint64_t cap, mem_profile_t *profile) { 165 | uint8_t *mem = mmap(NULL, cap, PROT_READ | PROT_WRITE, 166 | MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); 167 | 168 | arena_t arena = { 169 | .profile = profile, 170 | .start = mem, 171 | .end = mem + cap, 172 | }; 173 | return arena; 174 | } 175 | 176 | void b(int n, arena_t *arena) { 177 | arena_alloc(arena, sizeof(int), _Alignof(int), n); 178 | } 179 | 180 | void a(int n, arena_t *arena) { 181 | arena_alloc(arena, sizeof(int), _Alignof(int), n); 182 | b(n, arena); 183 | } 184 | 185 | int main() { 186 | arena_t mem_profile_arena = arena_new(1 << 16, NULL); 187 | mem_profile_t mem_profile = { 188 | .arena = mem_profile_arena, 189 | .records = arena_alloc(&mem_profile_arena, sizeof(mem_record_t), 190 | _Alignof(mem_record_t), 16), 191 | .records_cap = 16, 192 | }; 193 | 194 | arena_t arena = arena_new(1 << 28, &mem_profile); 195 | 196 | for (int i = 0; i < 2; i++) 197 | a(2 * 1024 * 1024, &arena); 198 | 199 | b(3 * 1024 * 1024, &arena); 200 | 201 | mem_profile_write(&mem_profile, stderr); 202 | } 203 | -------------------------------------------------------------------------------- /odin_and_musl.md: -------------------------------------------------------------------------------- 1 | Title: Odin and musl: Cross-compiling Odin programs for the Raspberry Pi Zero 2 | Tags: Odin, Musl, ARM, Cross-compilation 3 | --- 4 | 5 | [Odin programming language](https://odin-lang.org/) is becoming my favorite tool as a Software Engineer. It's a fantastic programming language, mostly because it is dead simple. 6 | 7 | I have purchased some time ago a Raspberry Pi Zero 2, and I found myself wanting to write command-line Odin programs for it. Here it is in all its beauty: 8 | 9 | ![Raspberry Pi Zero 2](zero2.png) 10 | 11 | Here's the story of how I did it. If you do not work with Odin but do work a lot with cross-compilation, like I do at work, all of these techniques will be, I believe, very valuable anyway. 12 | 13 | *Note: ARM64 is sometimes also called AARCH64 interchangeably.* 14 | *Note 2: The Rapsberry Pi Zero 1 is based on ARM (32 bits). The Raspberry Pi Zero 2 is based on ARM64 (64 bits). If you have a Raspberry Pi Zero 1, this article still applies, just adjust the target when cross-compiling.* 15 | 16 | ## Inciting incident 17 | 18 | The thing is, I work on an Intel Linux laptop and the Zero is a Linux ARM 64 bits piece of hardware. It's also a relatively cheap component with only 512 MiB of RAM and a slow CPU (compared to a modern developer workstation), and based on a very slow SD card, so it's not fast to install the required tools and to build source code on it. Cross-compilation is much easier and faster. 19 | 20 | Odin can cross-compile to it with `-target=linux_arm64`, so that's great, let's try it: 21 | 22 | ```sh 23 | $ odin build src -target=linux_arm64 24 | [...] 25 | /usr/bin/ld: /home/pg/my-code/odin-music-chords-placements/src.o: error adding symbols: file in wrong format 26 | clang: error: linker command failed with exit code 1 (use -v to see invocation) 27 | ``` 28 | 29 | Oh no...The key part is: `file in wrong format`. 30 | 31 | That's because behind the scenes, the Odin compiler builds our code into an ARM64 object file, which is great. But then it tries to link this object file with libc, which on this computer is a x86_64 library, and that won't work. 32 | 33 | We can confirm this theory by asking Odin to print the linking command: 34 | 35 | ```sh 36 | $ odin build src -target=linux_arm64 -print-linker-flags 37 | clang -Wno-unused-command-line-argument [...] -lm -lc -L/ -no-pie 38 | ``` 39 | And we see it links libc with `-lc`, meaning it links our program with the local libc it finds on my machine which is a different architecture than our target. 40 | 41 | ## Confrontation 42 | 43 | What we want is to link our object file with the correct libc, meaning one that has been built for ARM64. Moreover, we'd like to build our program statically with libc so that we can simply copy the one executable to the Raspberry Pi Zero and it is fully self-contained. We completely side-step issues of different glibc versions not being compatible with each other. 44 | 45 | Enter musl, a C library for Linux that supports many platforms including ARM64, and static compilation. That's exactly what we need! 46 | 47 | A big difference between Odin and Zig is that Zig is a full cross-compilation toolchain: it comes with the source code of `musl`, and has put in a ton of work to cross-compile it to the target the user desires. 48 | 49 | So to make our use-case work with Odin, without Odin the toolchain supporting what Zig supports, what we need to do is cross-compile our code to an ARM64 object file but without linking it yet. Then we link it manually to musl libc that has been built for ARM64. We could download this musl artifact from the internet but it's both more educational, empowering, and secure, to build it ourselves. So let's do this, it's not too much work. 50 | 51 | To build musl, we can either use clang since it is a cross-compiler by default, or a GCC toolchain that has been made to target ARM64. Most Linux distributions provide such a compiler as a package typically called `gcc-aarch64-xxx` e.g. `sudo apt-get install gcc-aarch64-linux-gnu` or `sudo dnf install gcc-aarch64-linux-gnu`. 52 | 53 | So let's now build a static musl for ARM64, following the official instructions. We just need to this once: 54 | 55 | ```sh 56 | $ git clone --recurse --depth 1 https://git.musl-libc.org/git/musl 57 | $ cd musl 58 | 59 | # With Clang: 60 | $ CFLAGS='--target=aarch64-unknown-linux-musl' RANLIB=llvm-ranlib AR=llvm-ar CC=clang ./configure --target=aarch64 --disable-shared 61 | # Or with GCC: 62 | $ RANLIB=/usr/bin/aarch64-linux-gnu-gcc-ranlib AR=/usr/bin/aarch64-linux-gnu-gcc-ar CC=/usr/bin/aarch64-linux-gnu-gcc ./configure --target=aarch64 --disable-shared 63 | 64 | # Either way (Clang/GCC), the build command itself is the same. 65 | $ make 66 | ``` 67 | 68 | We now have the two artifacts we want: `crt1.o` and `libc.a`. We can confirm that they have been correctly built for our target: 69 | 70 | ```sh 71 | $ file lib/crt1.o 72 | lib/crt1.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), not stripped 73 | $ readelf -h lib/libc.a | grep '^\s*Machine:' 74 | Machine: AArch64 75 | Machine: AArch64 76 | Machine: AArch64 77 | [...] 78 | ``` 79 | 80 | ## Resolution 81 | 82 | Now we can finally put all the pieces together. We can use any linker, I am using LLD (the LLVM linker) here, but the GNU LD linker would also work as long as it knows to target ARM64 e.g. using the one coming with the right GCC toolchain would work. 83 | 84 | ```sh 85 | $ odin build src -target=linux_arm64 -build-mode=object 86 | $ file src.o 87 | src.o: ELF 64-bit LSB relocatable, ARM aarch64, version 1 (SYSV), not stripped 88 | $ ld.lld main.o ~/not-my-code/musl/lib/libc.a ~/not-my-code/musl/lib/crt1.o 89 | $ file a.out 90 | a.out: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped 91 | ``` 92 | 93 | Alternatively, we can decide to stick with the Odin compiler through and through, and we pass it the (lengthy) required build options: 94 | 95 | ```sh 96 | $ odin build src -target=linux_arm64 -extra-linker-flags:'-L ~/not-my-code/musl/lib/ -nostdlib -fuse-ld=lld --target=linux-aarch64 ~/not-my-code/musl/lib/crt1.o -static' 97 | ``` 98 | 99 | We can even verify it works by running it inside a ARM64 Linux system using `qemu`: 100 | 101 | ```sh 102 | $ qemu-aarch64-static a.out 103 | # It runs! 104 | ``` 105 | 106 | Cherry on the cake, the resulting program is small: 107 | 108 | ```sh 109 | $ llvm-strip a.out 110 | $ du -h a.out 111 | 288K a.out 112 | ``` 113 | 114 | So it's a breeze to `scp` or `rsync` our small executable over to the Raspberry Pi Zero while hacking on it. 115 | 116 | Perhaps Odin will have built-in support for musl in the future like Zig does. In the meantime, this article shows it's absolutely possible to do that ourselves! 117 | 118 | By the way, this technique can be used to cross-compile any C library that's a dependency of our project, assuming the library did not do anything silly that would prevent cross-compilation. 119 | 120 | ## Appendix: Maybe you don't even need a libc 121 | 122 | Odin comes with batteries included with a rich standard library. So why do we even need libc? Let's inspect which functions we really use from libc, i.e. are undefined symbols in the object file built from our source code: 123 | 124 | ```sh 125 | $ nm -u src.o 126 | U calloc 127 | U free 128 | U malloc 129 | U memcpy 130 | U memmove 131 | U memset 132 | U realloc 133 | ``` 134 | 135 | Ok, so basically: heap allocation and some functions to copy/set memory. 136 | 137 | Heap allocation functions are not actually required if our program does not do heap allocations (Odin provides the option `-default-to-nil-allocator` for this case), or if we implement these ourselves, for example with a naive `mmap` implementation, or by setting in our program the default allocator to be an arena. Odin has first class support for custom allocators! 138 | 139 | The functions to manipulate memory are required even if we do not call them directly because typically, the compiler will replace some code patterns, e.g. `struct` or array initialization, with these functions behind the scene. 140 | 141 | These `memxxx` functions could potentially be implemented by us, likely incurring a performance cost compared to the hand-optimized libc versions. But Odin can provide them for us! We can just use the `-no-crt` option. 142 | 143 | Note that not all targets will be equally supported for this use-case. ARM64 is not yet supported, so I will demonstrate targeting AMD64 (i.e. Intel/AMD 64 bits). 144 | 145 | I also had to install `nasm` to make it work because Odin ships with some assembly files which are then built for the target with `nasm`, but Odin does not ship with `nasm` itself. 146 | 147 | Let's try with a 'hello world' example: 148 | 149 | ```odin 150 | package main 151 | 152 | import "core:fmt" 153 | 154 | main :: proc() { 155 | fmt.println("Hello") 156 | } 157 | ``` 158 | 159 | We can build it as outlined like this: 160 | 161 | ```sh 162 | $ odin build hello.odin -file -target=linux_amd64 -default-to-nil-allocator -no-crt 163 | $ file hello 164 | hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=ef8dfc9dc297295808f80ec66e92763358a598d1, not stripped 165 | ``` 166 | 167 | And we can see the `malloc` symbol is not present since we do opted out of it, and that Odin provided with these assembly files the correct implementation for `memset`: 168 | 169 | ```sh 170 | $ nm hello | grep malloc 171 | # Nothing 172 | $ nm hello | grep memset 173 | 00000000004042c0 T memset 174 | ``` 175 | 176 | 177 | I'll soon write about what programs I made for the Raspberry Pi Zero, so check back soon! 178 | -------------------------------------------------------------------------------- /odin_syntax.js: -------------------------------------------------------------------------------- 1 | hljs.registerLanguage("odin", function(e) { 2 | return { 3 | aliases: ["odin", "odinlang", "odin-lang"], 4 | keywords: { 5 | keyword: "auto_cast bit_field bit_set break case cast context continue defer distinct do dynamic else enum fallthrough for foreign if import in map matrix not_in or_else or_return package proc return struct switch transmute type_of typeid union using when where", 6 | literal: "true false nil", 7 | built_in: "abs align_of cap clamp complex conj expand_to_tuple imag jmag kmag len max min offset_of quaternion real size_of soa_unzip soa_zip swizzle type_info_of type_of typeid_of" 8 | }, 9 | illegal: " { 45 | if (0 == el.classList.length || el.classList.contains('language-sh') || el.classList.contains('language-shell') || el.classList.contains('language-bash')){ 46 | el.classList.add('code-no-line-numbers'); 47 | return; 48 | } 49 | 50 | var lines = el.innerHTML.trimEnd().split('\n'); 51 | var out = []; 52 | lines.forEach(function(l, i){ 53 | out.push('' + (i+1).toString() + ' ' + l); 54 | }); 55 | el.innerHTML = out.join('\n'); 56 | }); 57 | -------------------------------------------------------------------------------- /screencast.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/screencast.mp4 -------------------------------------------------------------------------------- /screencast.webm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/screencast.webm -------------------------------------------------------------------------------- /search.js: -------------------------------------------------------------------------------- 1 | import search from './search_index.js'; 2 | const {raw_index}=search; 3 | 4 | const excerpt_len_around = 100; 5 | 6 | function search_text(needle) { 7 | console.time("search_text"); 8 | 9 | const matches=[]; 10 | for (const [doc_i, doc] of raw_index.documents.entries()){ 11 | let start = 0; 12 | for (let _i =0; _i < doc.text.length; _i++){ 13 | const idx = doc.text.slice(start).indexOf(needle); 14 | if (-1==idx) { break; } 15 | 16 | matches.push([doc_i | 0, start+idx]); 17 | start += idx + 1; 18 | } 19 | } 20 | console.timeEnd("search_text"); 21 | 22 | const res = []; 23 | for (const [doc_i, idx] of matches) { 24 | const doc = raw_index.documents[doc_i]; 25 | 26 | let title = undefined; 27 | for (const t of doc.titles){ 28 | if (idx < t.offset) { 29 | break; 30 | } 31 | title = t; 32 | } 33 | 34 | const link = title ? doc.html_file_name + '#' + title.hash + '-' + title.content_html_id : doc.html_file_name; 35 | 36 | res.push({ 37 | index: idx, 38 | title: title, 39 | link: link, 40 | document_index: doc_i, 41 | }); 42 | } 43 | 44 | return res; 45 | } 46 | 47 | window.onload = function() { 48 | const dom_search_matches = window.document.getElementById('search-matches'); 49 | const dom_input = window.document.getElementById('search'); 50 | const dom_pseudo_body = window.document.getElementById('pseudo-body'); 51 | 52 | function search_and_display_results(_event) { 53 | const needle = dom_input.value; 54 | if (needle.length < 3) { 55 | dom_pseudo_body.hidden = false; 56 | dom_search_matches.hidden = true; 57 | return; 58 | } 59 | 60 | dom_pseudo_body.hidden = true; 61 | dom_search_matches.hidden = false; 62 | dom_search_matches.innerHTML = ''; 63 | 64 | const matches = search_text(dom_input.value); 65 | 66 | const dom_search_title = window.document.createElement('h2'); 67 | dom_search_title.innerText = `Search results (${matches.length}):`; 68 | dom_search_matches.append(dom_search_title); 69 | 70 | const match_entries = matches.entries(); 71 | for (const [i, match] of match_entries) { 72 | const doc = raw_index.documents[match.document_index]; 73 | const dom_match = window.document.createElement('p'); 74 | 75 | const dom_doc = window.document.createElement('h3'); 76 | dom_doc.innerHTML = `${doc.title}` 77 | if (match.title) { 78 | dom_doc.innerHTML += `: ${match.title.title}`; 79 | } 80 | dom_match.append(dom_doc); 81 | 82 | const dom_excerpt = window.document.createElement('p'); 83 | const excerpt_idx_start = match.index - excerpt_len_around < 0 ? 0 : match.index - excerpt_len_around; 84 | const excerpt_idx_end = match.index + needle.length + excerpt_len_around; 85 | dom_excerpt.innerHTML = '...' + 86 | doc.text.slice(excerpt_idx_start, match.index) + 87 | '' + 88 | doc.text.slice(match.index, match.index + needle.length) + 89 | '' + 90 | doc.text.slice(match.index + needle.length, excerpt_idx_end) + 91 | '...'; 92 | dom_match.append(dom_excerpt); 93 | 94 | if (i + 1 < matches.length){ 95 | const dom_hr = window.document.createElement('hr'); 96 | dom_match.append(dom_hr); 97 | } 98 | 99 | dom_search_matches.append(dom_match); 100 | } 101 | }; 102 | dom_input.oninput = search_and_display_results; 103 | dom_input.onfocus = search_and_display_results; 104 | }; 105 | -------------------------------------------------------------------------------- /simd_talk.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/simd_talk.png -------------------------------------------------------------------------------- /tip_of_day_1.md: -------------------------------------------------------------------------------- 1 | Title: Tip of the day #1: Count lines of Rust code, ignoring tests 2 | Tags: Rust, Tip of the day, Awk 3 | --- 4 | 5 | I have a Rust codebase at work. The other day, I was wondering how many lines of code were in there. Whether you use `wc -l ***.rs` or a more fancy tool like `tokei`, there is an issue: this will count the source code *as well as* tests. 6 | 7 | That's because in Rust and in some other languages, people write their tests in the same files as the implementation. Typically it looks like that: 8 | 9 | ```rust 10 | // src/foo.rs 11 | 12 | fn foo() { 13 | ... 14 | } 15 | 16 | #[cfg(test)] 17 | mod tests { 18 | fn test_foo(){ 19 | ... 20 | } 21 | 22 | ... 23 | } 24 | ``` 25 | 26 | But I only want to know how big is the implementation. I don't care about the tests. And `wc` or `tokei` will not show me that. 27 | 28 | So I resorted to my trusty `awk`. Let's first count all lines, like `wc` does: 29 | 30 | ```sh 31 | $ awk '{count += 1} END{print(count)}' src/***.rs 32 | # Equivalent to: 33 | $ wc -l src/***/.rs 34 | ``` 35 | 36 | On my open-source Rust [project](https://github.com/gaultier/kotlin-rs), this prints `11485`. 37 | 38 | Alright, now let's exclude the tests. When we encounter the line `mod tests`, we stop counting. Note that this name is just a convention, but that's one that followed pretty much universally in Rust code, and there is usually no more code after this section. Tweak the name if needed: 39 | 40 | ```sh 41 | $ awk '/mod tests/{skip[FILENAME]=1} !skip[FILENAME]{count += 1} END{print(count)}' src/***.rs 42 | ``` 43 | 44 | And this prints in the same project: `10057`. 45 | 46 | Let's unpack it: 47 | 48 | - We maintain a hashtable called `skip` which is a mapping of the file name to whether or not we should skip the rest of this file. In AWK we do not need to initialize variables, we can use them right away and they are zero initialized. AWK also automatically stores the name of the current file in the global builtin variable `FILENAME`. 49 | - `/mod tests/`: this pattern matches the line containing `mod tests`. The action for this line is to flag this file as 'skipped', by setting the value in the map for this file to `1` (i.e. `true`). 50 | - `!skip[FILENAME]{count += 1}`: If this line for the current file is not flagged as 'skipped', we increment for each line, the global counter. Most people think that AWK can only use patterns as clauses before the action, but in fact it also supports boolean conditions, and both can be use together, e.g.: `/foo/ && !skip[FILENAME] {print("hello")}` 51 | - `END{print(count)}`: we print the count at the very end. 52 | 53 | And that's it. AWK is always very nifty. 54 | 55 | ## Addendum: exit 56 | 57 | Originally I implemented it wrongly, like this: 58 | 59 | 60 | ```sh 61 | $ awk '/mod tests/{exit 0} {count += 1} END{print(count)}' src/***.rs 62 | ``` 63 | 64 | If we encounter tests, stop processing the file altogether, with the builtin statement `exit` ([docs](https://www.gnu.org/software/gawk/manual/html_node/Exit-Statement.html)). 65 | 66 | Running this on the same Rust codebase prints: `1038` which is obviously wrong. 67 | 68 | Why is it wrong then? 69 | 70 | Well, as I understand it, AWK processes all inputs files one by one, as if it was one big sequential file (it will still fill the builtin constant `FILENAME` though, that's why the solution above works). Since there is no isolation between the processing each file (AWK does not spawn a subprocess for each file), it means we simply stop altogether at the first encountered test in any file. 71 | -------------------------------------------------------------------------------- /tip_of_day_3.md: -------------------------------------------------------------------------------- 1 | Title: Tip of the day #3: Convert a CSV to a markdown or HTML table 2 | Tags: Markdown, Csv, Awk, Tip of the day 3 | --- 4 | 5 | The other day at work, I found myself having to produce a human-readable table of all the direct dependencies in the project, for auditing purposes. 6 | 7 | There is a [tool](https://github.com/onur/cargo-license) for Rust projects that outputs a TSV (meaning: a CSV where the separator is the tab character) of this data. That's great, but not really fit for consumption by a non-technical person. 8 | 9 | I just need to convert that to a human readable table in markdown or HTML, and voila! 10 | 11 | 12 | Here's the output of this tool in my open-source Rust [project](https://github.com/gaultier/kotlin-rs): 13 | 14 | ```sh 15 | $ cargo license --all-features --avoid-build-deps --avoid-dev-deps --direct-deps-only --tsv 16 | name version authors repository license license_file description 17 | clap 2.33.0 Kevin K. https://github.com/clap-rs/clap MIT A simple to use, efficient, and full-featured Command Line Argument Parser 18 | heck 0.3.1 Without Boats https://github.com/withoutboats/heck Apache-2.0 OR MIT heck is a case conversion library. 19 | kotlin 0.1.0 Philippe Gaultier 20 | log 0.4.8 The Rust Project Developers https://github.com/rust-lang/log Apache-2.0 OR MIT A lightweight logging facade for Rust 21 | pretty_env_logger 0.3.1 Sean McArthur https://github.com/seanmonstar/pretty-env-logger Apache-2.0 OR MIT a visually pretty env_logger 22 | termcolor 1.1.0 Andrew Gallant https://github.com/BurntSushi/termcolor MIT OR Unlicense A simple cross platform library for writing colored text to a terminal. 23 | ``` 24 | 25 | Not really readable. We need to transform this data into a [markdown table](https://github.github.com/gfm/#tables-extension-), something like that: 26 | 27 | ```markdown 28 | | First Header | Second Header | 29 | | ------------- | ------------- | 30 | | Content Cell | Content Cell | 31 | | Content Cell | Content Cell | 32 | ``` 33 | 34 | Technically, markdown tables are an extension to standard markdown (if there is such a thing), but they are very common and supported by all the major platforms e.g. Github, Azure, etc. So how do we do that? 35 | 36 | Once again, I turn to the trusty AWK. It's always been there for me. And it's present on every UNIX system out of the box. 37 | 38 | AWK neatly handles all the 'decoding' of the CSV format for us, we just need to output the right thing: 39 | 40 | - Given a line (which AWK calls 'record'): output each field interleaved with the `|` character 41 | - Output a delimiting line between the table headers and rows. The markdown table spec states that this delimiter should be at least 3 `-` characters in each cell. 42 | - Alignment is not a goal, it does not matter for a markdown parser. If you want to produce a pretty markdown table, it's easy to achieve, it simply makes the implementation a bit bigger 43 | 44 | Here's the full implementation (don't forget to mark the file executable). The shebang line instructs AWK to use the tab character `\t` as the delimiter between fields: 45 | 46 | ```awk 47 | #!/usr/bin/env -S awk -F '\t' -f 48 | 49 | { 50 | printf("|"); 51 | for (i = 1; i <= NF; i++) { 52 | # Note: if a field contains the character `|`, 53 | # it will mess up the table. 54 | # In this case, we should replace this character 55 | # by something else e.g. `,`: 56 | gsub(/\|/, ",", $i); 57 | printf(" %s |", $i); 58 | } 59 | printf("\n"); 60 | } 61 | 62 | NR==1 { # Output the delimiting line 63 | printf("|"); 64 | for(i = 1; i <= NF; i++) { 65 | printf(" --- | "); 66 | } 67 | printf("\n"); 68 | } 69 | ``` 70 | 71 | The first clause will execute for each line of the input. 72 | The for loop then iterates over each field and outputs the right thing. 73 | 74 | The second clause will execute only for the first line (`NR` is the line number). 75 | 76 | The same line can trigger multiple clauses, here, the first line of the input will trigger both clauses, whilst the remaining lines will only trigger the first clause. 77 | 78 | 79 | So let's run it! 80 | 81 | ```sh 82 | $ cargo license --all-features --avoid-build-deps --avoid-dev-deps --direct-deps-only --tsv | ./md-table.awk 83 | | name | version | authors | repository | license | license_file | description | 84 | | --- | --- | --- | --- | --- | --- | --- | 85 | | clap | 2.33.0 | Kevin K. | https://github.com/clap-rs/clap | MIT | | A simple to use, efficient, and full-featured Command Line Argument Parser | 86 | | heck | 0.3.1 | Without Boats | https://github.com/withoutboats/heck | Apache-2.0 OR MIT | | heck is a case conversion library. | 87 | | kotlin | 0.1.0 | Philippe Gaultier | | | | | 88 | | log | 0.4.8 | The Rust Project Developers | https://github.com/rust-lang/log | Apache-2.0 OR MIT | | A lightweight logging facade for Rust | 89 | | pretty_env_logger | 0.3.1 | Sean McArthur | https://github.com/seanmonstar/pretty-env-logger | Apache-2.0 OR MIT | | a visually pretty env_logger | 90 | | termcolor | 1.1.0 | Andrew Gallant | https://github.com/BurntSushi/termcolor | MIT OR Unlicense | | A simple cross platform library for writing colored text to a terminal. | 91 | ``` 92 | 93 | Ok, it's hard to really know if that's correct or not. Let's pipe it into [cmark-gfm](https://github.com/github/cmark-gfm) to render this markdown table as HTML: 94 | 95 | ```sh 96 | $ cargo license --all-features --avoid-build-deps --avoid-dev-deps --direct-deps-only --tsv | ./md-table.awk | cmark-gfm -e table 97 | ``` 98 | 99 | And voila: 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 |
nameversionauthorsrepositorylicenselicense_filedescription
clap2.33.0Kevin K. kbknapp@gmail.comhttps://github.com/clap-rs/clapMITA simple to use, efficient, and full-featured Command Line Argument Parser
heck0.3.1Without Boats woboats@gmail.comhttps://github.com/withoutboats/heckApache-2.0 OR MITheck is a case conversion library.
kotlin0.1.0Philippe Gaultier philigaultier@gmail.com
log0.4.8The Rust Project Developershttps://github.com/rust-lang/logApache-2.0 OR MITA lightweight logging facade for Rust
pretty_env_logger0.3.1Sean McArthur sean@seanmonstarhttps://github.com/seanmonstar/pretty-env-loggerApache-2.0 OR MITa visually pretty env_logger
termcolor1.1.0Andrew Gallant jamslam@gmail.comhttps://github.com/BurntSushi/termcolorMIT OR UnlicenseA simple cross platform library for writing colored text to a terminal.
170 | 171 | 172 | All in all, very little code. I have a feeling that I will use this approach a lot in the future for reporting or even inspecting data easily, for example from a database dump. 173 | -------------------------------------------------------------------------------- /tip_of_the_day_4.md: -------------------------------------------------------------------------------- 1 | Title: Tip of the day #4: Type annotations on Rust match patterns 2 | Tags: Rust, Tip of the day 3 | --- 4 | 5 | *Discussions: [/r/rust](https://old.reddit.com/r/rust/comments/1in1wyr/tip_of_the_day_4_type_annotations_on_rust_match/)* 6 | 7 | Today at work I was adding error logs to our Rust codebase and I hit an interesting case. I had a match pattern, and the compiler asked me to add type annotations to a branch of the pattern, because it could not infer by itself the right type. 8 | 9 | ```rust 10 | fn decode_foo(input: &[u8]) -> Result<(&[u8], [u8; 33]), Error> { 11 | if input.len() < 33 { 12 | return Err(Error::InvalidData); 13 | } 14 | 15 | let (left, right) = input.split_at(33); 16 | let value = match left.try_into() { 17 | Ok(v) => v, 18 | Err(err) => { 19 | let err_s = err.to_string(); // <= Here is the compilation error. 20 | eprintln!("failed to decode data, wrong length: {}", err_s); 21 | return Err(Error::InvalidData); 22 | } 23 | }; 24 | 25 | Ok((right, value)) 26 | } 27 | ``` 28 | 29 | ``` 30 | error[E0282]: type annotations needed 31 | --> src/main.rs:14:25 32 | | 33 | 14 | let err_s = err.to_string(); 34 | | ^^^ cannot infer type 35 | ``` 36 | 37 | This function parses a slice of bytes, and on error, logs the error. The real code is of course more complex but I could reduce the error to this minimal code. 38 | 39 | --- 40 | 41 | So I tried to add type annotations the usual Rust way: 42 | 43 | ```rust 44 | let value = match left.try_into() { 45 | Ok(v) => v, 46 | Err(err: TryFromSliceError) => { 47 | // [...] 48 | } 49 | ``` 50 | 51 | Which leads to this nice error: 52 | 53 | ``` 54 | error: expected one of `)`, `,`, `@`, or `|`, found `:` 55 | --> src/main.rs:15:16 56 | | 57 | 15 | Err(err: TryFromSliceError) => { 58 | | ^ expected one of `)`, `,`, `@`, or `|` 59 | ``` 60 | 61 | If you're feeling smart, thinking, 'Well that's because you did not use `inspect_err` or `map_err`!'. Well they suffer from the exact same problem: a type annotation is needed. However, since they use a lambda, the intuitive type annotation, like the one I tried, works. But not for `match`. 62 | 63 | --- 64 | 65 | Alright, so after some [searching around](https://users.rust-lang.org/t/type-annotation-on-match-pattern/49180/10), I came up with this mouthful of a syntax: 66 | 67 | ```rust 68 | let value = match left.try_into() { 69 | Ok(v) => v, 70 | Err::<_, TryFromSliceError>(err) => { 71 | // [...] 72 | } 73 | ``` 74 | 75 | Which works! And the same syntax can be applied to the `Ok` branch (per the link above) if needed. Note that this is a partial type annotation: we only care about the `Err` part of the `Result` type. 76 | 77 | That was a TIL for me. It's a bit of a weird syntax here. It's usually the syntax for type annotations on methods (more on that in a second). 78 | 79 | You can be even more verbose by mentioning the whole type, if you want to: 80 | 81 | ```rust 82 | let value = match left.try_into() { 83 | Ok(v) => v, 84 | Result::<_, TryFromSliceError>::Err(err) => { 85 | ``` 86 | 87 | --- 88 | 89 | Anyways, there's a much better way to solve this issue. We can simply annotate the resulting variable outside of the whole match pattern, so that `rustc` knows which `try_into` method we are using: 90 | 91 | ```rust 92 | let value: [u8; 33] = match left.try_into() { 93 | Ok(v) => v, 94 | Err(err) => { 95 | // [...] 96 | } 97 | ``` 98 | 99 | --- 100 | 101 | Or alternatively, as pointed out by a perceptive reader, annotate the `err` variable inside the body for the `Err` branch: 102 | 103 | ```rust 104 | let value = match left.try_into() { 105 | Ok(v) => v, 106 | Err(err) => { 107 | let err: TryFromSliceError = err; // <= Here. 108 | let err_s = err.to_string(); 109 | eprintln!("failed to decode data, wrong length: {}", err_s); 110 | return Err(Error::InvalidData); 111 | } 112 | }; 113 | ``` 114 | 115 | --- 116 | 117 | Another reader had a different idea: use a match binding, which mentions the error type explicitly (that only works if the error type is a struct): 118 | 119 | ```rust 120 | let value = match left.try_into() { 121 | Ok(v) => v, 122 | Err(err @ TryFromSliceError { .. }) => { 123 | ``` 124 | 125 | Pretty succinct! This reader mentions this [PR](https://github.com/rust-lang/rfcs/pull/3753) to expand this to all types, but that the general feedback is that the 'intuitive' syntax `Err(err: Bar) => {` should be possible instead. 126 | 127 | --- 128 | 129 | Yet another approach that works is to annotate the `try_into()` function with the type, but I find it even noisier than annotating the `Err` branch: 130 | 131 | ```rust 132 | let value = match TryInto::<[u8; 33]>::try_into(left) { 133 | Ok(v) => v, 134 | Err(err) => { 135 | // [...] 136 | } 137 | ``` 138 | 139 | --- 140 | 141 | Astute readers will think at this point that all of this is unnecessary: let's just have the *magic traits(tm)* do their wizardry. We do not convert the error to a string, we simply let `eprintln!` call `err.fmt()` under the hood, since `TryFromSliceError` implements the `Display` trait (which is why we could convert it to a `String` with `.to_string()`): 142 | 143 | ```rust 144 | let value = match left.try_into() { 145 | Ok(v) => v, 146 | Err(err) => { 147 | eprintln!("failed to decode data, wrong length: {}", err); 148 | return Err(Error::InvalidData); 149 | } 150 | }; 151 | ``` 152 | 153 | That works but in my case I really needed to convert the error to a `String`, to be able to pass it to C, which does not know anything about fancy traits. 154 | 155 | 156 | I find this issue interesting because it encapsulates well the joy and pain of writing Rust: match patterns are really handy, but they sometimes lead to weird syntax not found elsewhere in the Rust language (maybe due to the OCaml heritage?). Type inference is nice but sometimes the compiler/language server fails at inferring things you'd think they should really be able to infer. Traits and `into/try_into` are found everywhere in Rust code, but it's hard to know what type is being converted to what, especially when these are chained several times without any type annotation whatsoever. 157 | 158 | --- 159 | 160 | By the way, here's a tip I heard some time ago: if you want to know the real type of a variable that's obscured by type inference, just add a type annotation that's obviously wrong, and the compiler will show the correct type. That's how I pinpointed the `TryFromSliceError` type. Let's add a bogus `bool` type annotation: 161 | 162 | ```rust 163 | let value = match left.try_into() { 164 | Ok(v) => v, 165 | Err::<_, bool>(err) => { 166 | // [...] 167 | } 168 | ``` 169 | 170 | And the compiler helpfully gives us the type: 171 | 172 | ``` 173 | error[E0271]: type mismatch resolving `<[u8; 33] as TryFrom<&[u8]>>::Error == bool` 174 | --> src/main.rs:11:28 175 | | 176 | 11 | let value = match left.try_into() { 177 | | ^^^^^^^^ expected `bool`, found `TryFromSliceError` 178 | ``` 179 | 180 | So...it *does* actually know the type of `err`... You naughty compiler, playing games with me! It reminds me of this picture: 181 | 182 | Coffee or tea?Coffee or tea? 183 | -------------------------------------------------------------------------------- /tip_of_the_day_5.html: -------------------------------------------------------------------------------- 1 | 5 | 6 | 7 | 8 | Tip of the day #5: Install Go tools with a specific version 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 70 | 72 |
73 | 74 |
75 |

⏴ Back to all articles

76 | 77 |

Published on 2025-02-18

78 |
79 |
80 |

Tip of the day #5: Install Go tools with a specific version

81 |
Go Tip of the day
82 |
83 | 84 |

I had an issue with Go tools around versioning, and here's how I solved it. It could be useful to others. This was the error:

85 | 
$ staticcheck  ./...
 86 | -: module requires at least go1.23.6, but Staticcheck was built with go1.23.1 (compile)
 87 | $ go version
 88 | go version go1.23.6 linux/amd64
 89 |
90 |

Indeed the project was specifying go 1.23.6 in go.mod.

91 |

Even after removing the staticcheck binary and re-installing it I still had the same issue:

92 | 
$ which staticcheck
 93 | /home/pg/go/bin/staticcheck
 94 | $ rm /home/pg/go/bin/staticcheck
 95 | $ which staticcheck
 96 | which: no staticcheck 
 97 | $ go install honnef.co/go/tools/cmd/staticcheck@v0.5.1
 98 | $ staticcheck  ./...
 99 | -: module requires at least go1.23.6, but Staticcheck was built with go1.23.1 (compile)
100 |
101 |

I even tried the -a flag for go install to force a clean build (since go install fetches the sources and builds them) to no avail.

102 |

Solution: following https://go.dev/doc/manage-install, I installed the specific version of Go I needed and used that to install the tool:

103 | 
$ go install golang.org/dl/go1.23.6@latest
104 | $ go1.23.6 download
105 | $ go1.23.6 install honnef.co/go/tools/cmd/staticcheck@v0.5.1
106 | $ staticcheck  -tags=integration_tests ./... # Works!
107 |
108 |

That was a TIL for me.

109 |

Note that Go 1.24 supports the project listing tools directly in go.mod which would probably solve this issue directly.

110 |

⏴ Back to all articles

111 | 112 | 115 | 116 |
117 |

118 | This blog is open-source! 119 | If you find a problem, please open a Github issue. 120 | The content of this blog as well as the code snippets are under the BSD-3 License which I also usually use for all my personal projects. It's basically free for every use but you have to mention me as the original author. 121 |

122 |
123 | 124 |
125 | 126 | 127 | -------------------------------------------------------------------------------- /tip_of_the_day_5.md: -------------------------------------------------------------------------------- 1 | Title: Tip of the day #5: Install Go tools with a specific version 2 | Tags: Go, Tip of the day 3 | --- 4 | 5 | I had an issue with Go tools around versioning, and here's how I solved it. It could be useful to others. This was the error: 6 | 7 | ```sh 8 | $ staticcheck ./... 9 | -: module requires at least go1.23.6, but Staticcheck was built with go1.23.1 (compile) 10 | $ go version 11 | go version go1.23.6 linux/amd64 12 | ``` 13 | 14 | Indeed the project was specifying `go 1.23.6` in `go.mod`. 15 | 16 | Even after removing the staticcheck binary and re-installing it I still had the same issue: 17 | 18 | ```sh 19 | $ which staticcheck 20 | /home/pg/go/bin/staticcheck 21 | $ rm /home/pg/go/bin/staticcheck 22 | $ which staticcheck 23 | which: no staticcheck 24 | $ go install honnef.co/go/tools/cmd/staticcheck@v0.5.1 25 | $ staticcheck ./... 26 | -: module requires at least go1.23.6, but Staticcheck was built with go1.23.1 (compile) 27 | ``` 28 | 29 | I even tried the `-a` flag for `go install` to force a clean build (since `go install` fetches the sources and builds them) to no avail. 30 | 31 | **Solution:** following [https://go.dev/doc/manage-install](https://go.dev/doc/manage-install), I installed the specific version of Go I needed and used that to install the tool: 32 | 33 | 34 | ```sh 35 | $ go install golang.org/dl/go1.23.6@latest 36 | $ go1.23.6 download 37 | $ go1.23.6 install honnef.co/go/tools/cmd/staticcheck@v0.5.1 38 | $ staticcheck -tags=integration_tests ./... # Works! 39 | ``` 40 | 41 | That was a TIL for me. 42 | 43 | Note that Go 1.24 supports the project listing tools directly in `go.mod` which would probably solve this issue directly. 44 | -------------------------------------------------------------------------------- /tip_of_the_day_6.md: -------------------------------------------------------------------------------- 1 | Title: Tip of the day #6: Use Bpftrace to estimate how much memory an in-memory cache will use 2 | Tags: Go, Tip of the day, Bpftrace, C 3 | --- 4 | 5 | ## Context 6 | 7 | I have a Go service that has an in-memory LRU (Least Recently Used) cache to speed up some things. 8 | Here I am, writing documentation for this service, and it happens that you can specify in its configuration the maximum number of cache entries. 9 | That's useful to limit the overall memory usage. Obviously this value is directly related to the Kubernetes memory limit for this deployment. 10 | 11 | But then I am wondering: what value should the docs recommend for this configuration field? A 1000 entries, 10 000? One factor is how many distinct entries do we expect, but another is: *How big is a cache entry*? 12 | 13 | An entry in the cache in this case is a slice of bytes (a blob) so it's not statically possible to determine, just looking at the code, how much memory it will consume. 14 | 15 | This distribution of entry sizes is however easy to uncover: all entries in the cache are inserted by one callback. It happens to be a Go function that is passed to a C library (via CGO) but this trick works with any language. This function takes as argument a slice of bytes to be inserted in the cache. So, add a log in this callback, print the slice length, process all the relevant logs, compute some statistics, and done? Or, add a custom Prometheus metric, deploy, done? 16 | 17 | Well... why modify the source code when we don't have too? Let's use [bpftrace](https://github.com/bpftrace/bpftrace) to determine the distribution of entry sizes *at runtime* on the unmodified program! In the past I have used [dtrace](https://illumos.org/books/dtrace/preface.html#preface) on macOS/FreeBSD which is similar and the direct inspiration for `bpftrace`. I find `dtrace` more powerful in some regards - although `bpftrace` has support for loops whereas `dtrace` does not. Point being, the `bpftrace` incantation can be adapted for `dtrace` pretty easily. Both of these tools are essential workhorses of exploratory programming and troubleshooting. 18 | 19 | ## Bpftrace 20 | 21 | So, the plan is: I run the tests under `bpftrace`, collect a histogram of the slice of bytes to be inserted in the cache, and voila! 22 | 23 | We can also run the real service with a load test to generate traffic, or simply wait for real traffic to come - all of that works, and `dtrace`/`bpftrace` are designed to inspect production programs without the risk of crashing them, or adversely impacting the system. The `bpftrace` incantation will be the same in all of these cases, only the binary (or process id) will change. 24 | 25 | Here, my function to insert a slice of bytes in the cache is called `cache_insert`, the executable is called `itest.test`, and the length of the slice of bytes happens to be passed as the third function argument. Arguments are zero-indexed so that means `arg2`: 26 | 27 | ```sh 28 | $ sudo bpftrace -e 'uprobe:./itest.test:cache_insert {@bytes=lhist(arg2, 0 , 16384, 128)}' -c './itest.test -test.count=1' 29 | ``` 30 | 31 | `lhist` creates a linear histogram with the minimum value here being `0`, the maximum value `16384` and the bucket size `128`. I used the `hist` function initially which uses a power-of-two bucket size but my values were all in one big bucket so that was a bit imprecise. Still a good first approximation. But we can get a better estimate by using a small bucket size with `lhist`. 32 | 33 | `bpftrace` prints the histogram by default at the end: 34 | 35 | ``` 36 | @bytes: 37 | [512, 640) 96 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| 38 | ``` 39 | 40 | So all slices of bytes have their length between `512` and `640` in this case, all in one bucket. 41 | 42 | --- 43 | 44 | Alternatively, we can point `bpftrace` at the Go function instead of the C function: 45 | 46 | ``` 47 | func (c Cache) Insert(ctx context.Context, key [32]byte, value []byte, expiryDate time.Time) error { [...] } 48 | ``` 49 | 50 | We are interested in `len(value)` which happens to be accessible in `arg5`: 51 | 52 | ```sh 53 | $ sudo bpftrace -e 'uprobe:./itest.test:/path/to/my/pkg/Cache.Insert {@bytes=lhist(arg5, 0 , 16384, 128)}' -c './itest.test -test.count=1' 54 | ``` 55 | 56 | and we get the same output. 57 | 58 | --- 59 | 60 | Note that we are doing very basic runtime inspection in this case, but we could also for example look at the hit rate of cache lookups, how much time inserting a new entry takes, etc. `bpftrace` and `dtrace` are really designed to be lightweight swiss-army knives. 61 | 62 | ## Addendum: Function arguments in bpftrace 63 | 64 | `bpftrace` reads neither debug information nor C headers by default so all function arguments are register sized, i.e. 64 bits on x86_64. `bpftrace` does not even know how many arguments the function accepts! 65 | 66 | My function signature is (simplified): 67 | 68 | ```c 69 | struct ByteSliceView { 70 | uint8_t* data; 71 | size_t len; 72 | } 73 | 74 | void cache_insert(const uint8_t *key, struct ByteSliceView value, [...]); 75 | ``` 76 | 77 | The value of interest is `value.len`. So initially I tried to access it in `bpftrace` using `arg1.len`, however it did not work. Here is an excerpt from the documentation: 78 | 79 | > Function arguments are available through the argN for register args. Arguments passed on stack are available using the stack pointer, e.g. $stack_arg0 = (int64)reg("sp") + 16. Whether arguments passed on stack or in a register depends on the architecture and the number or arguments used, e.g. on x86_64 the first 6 non-floating point arguments are passed in registers and all following arguments are passed on the stack. Note that floating point arguments are typically passed in special registers which don’t count as argN arguments which can cause confusion 80 | 81 | So, it's a mess ... 82 | 83 | I fired up `gdb` and printed registers directly when the `cache_insert` function is entered. I discovered by doing `info registers` that (on my machine, with this compiler and build flags, yada yada yada), the `rdx` register contains `value.len`. I.e. the compiler unpacks `value` which is a struct of two fields, into `arg1` (i.e. the `rsi` register) and `arg2` (i.e. the `rdx` register). 84 | 85 | Thus, this call: `cache_insert(foo, bar)` gets transformed by the compiler into `cache_insert(foo, bar.data, bar.len)`, and the third function argument (aka `arg2`) is our length. 86 | 87 | 88 | -------------------------------------------------------------------------------- /todo.md: -------------------------------------------------------------------------------- 1 | ## Ideas for articles 2 | 3 | - [ ] HTTP server using arenas 4 | - [ ] ~~Migration from mbedtls to aws-lc~~ 5 | - [ ] ~~ASM crypto~~ 6 | - [ ] How this blog is made. Line numbers in code snippets. 7 | - [ ] Banjo chords 8 | - [ ] Dtrace VM 9 | - [ ] From BIOS to bootloader to kernel to userspace 10 | - [ ] A faster HTTP parser than NodeJS's one 11 | - [ ] SHA1 multi-block hash 12 | - [ ] How CGO calls are implemented in assembly 13 | - [ ] How I develop Windows applications from the comfort of Linux 14 | - [ ] ~~Add a JS/Wasm live running version of the programs described in each article. As a fallback: Video/Gif. 15 | + [ ] Kahn's algorithm~~ 16 | - [ ] Blog search implementation 17 | - [ ] C++ web server undefined behavior bug 18 | - [ ] Weird and surprising things about x64 assembly 19 | + non symetric mnemonics (`cmp 1, rax` vs `cmp rax, 1`) 20 | + some different mnemonics encode to the same bytes (`jne`, `jz`) 21 | + diffent calling convention for functions & system calls in the SysV ABI (4th argument) 22 | + no (to my knowledge) mnemonic accepts 2 immediates or effective addresses as operands e.g. `cmp 1, 0` 23 | - [ ] How to get the current SQL schema when all you have is lots of migrations (deltas) 24 | - [ ] 'About' page 25 | - [ ] Search and replace fish function 26 | - [ ] Go+Dtrace 27 | ``` 28 | pid$target::*DispatchMessage:entry { 29 | stack_offset =656; 30 | this->data=copyin(uregs[R_SP] + stack_offset, 16); 31 | tracemem(this->data, 16); 32 | 33 | this->body_len = *((ssize_t*)this->data+1); 34 | 35 | this->body_ptr = (uint8_t**)this->data; 36 | 37 | this->s = copyinstr((user_addr_t)*this->body_ptr, this->body_len); 38 | printf("msg.Body: %s\n", this->s); 39 | } 40 | ``` 41 | ``` 42 | 43 | 10 2968 github.com/ory/kratos/courier.(*courier).DispatchMessage:entry 44 | 0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef 45 | 0: 80 ae c7 01 40 01 00 00 1e 00 00 00 00 00 00 00 ....@........... 46 | msg.Body: Your recovery code is: 707144 47 | ``` 48 | 49 | ## Blog implementation 50 | 51 | - [x] Add autogenerated html comment in rendered html files 52 | - [x] Consider post-processing HTML instead of markdown to simplify e.g. to add title ids 53 | - [x] Support markdown syntax in article title in metadata 54 | - [x] Use libcmark to simplify parsing 55 | - [ ] Link to related articles at the end (requires post-processing after all articles have been generated) 56 | - [x] Search 57 | + May require proper markdown parsing (e.g. with libcmark) to avoid having html/markdown elements in search results, to get the (approximate) location of each results (e.g. parent title), and show a rendered excerpt in search results. Perhaps simply post-process html? 58 | + Results are shown inline 59 | + Results show (some) surrounding text 60 | + Results have a link to the page and surrounding html section 61 | + Implementation: trigrams in js/wasm? Fuzzy search? See https://pkg.odin-lang.org/search.js 62 | + Search code/data are only fetched lazily 63 | + Cache invalidation of the search blob e.g. when an article is created/modified? => split search code (rarely changes) and data (changes a lot) 64 | + Code snippets included in search results? 65 | - [ ] Articles excerpt on the home page? 66 | - [ ] Dark mode 67 | - [ ] Browser live reload 68 | + Send the F5 key to the browser window (does not work in Wayland) 69 | + Somehow send an IPC message to the right browser process to reload the page? 70 | + HTTP server & HTML page communicate via websocket to reload content (complex) 71 | + HTTP2 force push to force the browser to get the new page version? 72 | + Server sent event to reload the page 73 | + Websocket 74 | - [ ] Built-in http server 75 | - [ ] Built-in file watch 76 | - [ ] Syntax highlighting done statically 77 | - [ ] Highlight only the changed lines in a code snippet 78 | - [ ] Copy-paste button for code snippets 79 | -------------------------------------------------------------------------------- /wayland-screenshot-floating.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/wayland-screenshot-floating.png -------------------------------------------------------------------------------- /wayland-screenshot-red.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/wayland-screenshot-red.png -------------------------------------------------------------------------------- /wayland-screenshot-tiled.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/wayland-screenshot-tiled.png -------------------------------------------------------------------------------- /wayland-screenshot-tiled1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/wayland-screenshot-tiled1.png -------------------------------------------------------------------------------- /what_should_your_mutexes_be_named.md: -------------------------------------------------------------------------------- 1 | Title: What should your mutexes be named? 2 | Tags: Go, Awk, Search 3 | --- 4 | 5 | *Discussions: [/r/golang](https://www.reddit.com/r/golang/comments/1l7h9cw/what_should_your_mutexes_be_named/), [HN](https://news.ycombinator.com/item?id=44232897).* 6 | 7 | I have started a new job recently and the main challenge, I find, is to ingest and adapt to a big codebase that does things slightly differently than what I am used to. 8 | 9 | This article explores ways to make this phase go smoother. 10 | 11 | --- 12 | 13 | The other day a Pull Request popped up at work, which made me think for a bit. It looked like this (Go, simplified): 14 | 15 | ```go 16 | type Foo struct { 17 | bar int 18 | barMux sync.Mutex 19 | } 20 | ``` 21 | 22 | It's a typical approach with concurrent code: a mutex protects the variable it is named after. So using the variable `bar` looks like this: 23 | 24 | ```go 25 | barMux.Lock() 26 | bar += 1 27 | barMux.Unlock() 28 | ``` 29 | 30 | *Yes, in this simplistic case an atomic would likely be used instead of a mutex but that's just to illustrate.* 31 | 32 | But I paused for a second: What should the mutex be named? I usually use the `xxxMtx` convention, so I'd have named it `barMtx`. 33 | 34 | To avoid a sterile 'you vs me' debate, I thought: What do other people do? What naming convention is in use in the project, if any? I'll demonstrate this method with the code of the Go standard library. 35 | 36 | And more generally, what is the best way to find out what naming conventions or code patterns are used in a project you don't know? I need a good tool to find these answers quickly. 37 | 38 | ## Structural search 39 | 40 | I use `ripgrep` and `awk` all the time when developing, probably at least once a minute, and these tools can give us the answers... kind of, since they operate on raw text. Complex code constructs or contextual searches e.g. 'A function call whose third argument is a number and the function name starts with `get`' may be impossible to find correctly. 41 | 42 | What I often actually need to do is *search the code structurally*, meaning search the Abstract Syntax Tree (AST). And the good news is, there are tools nowadays than can do that! I never took the time to learn one, even though this came up a few times, so I felt this is finally the occasion. 43 | 44 | Enter [ast-grep](https://github.com/ast-grep/ast-grep). Surprisingly, the main way to search with it is to write a rule file in YAML. A command line search also exists but seems much more limited. 45 | 46 | Let's thus search for 'structure fields' whose type is a 'mutex': 47 | 48 | ```yaml 49 | id: find-mtx-fields 50 | message: Mutex fields found 51 | severity: info 52 | language: go 53 | rule: 54 | kind: field_declaration 55 | all: 56 | - has: 57 | field: name 58 | # Ignore nameless fields. 59 | regex: ".+" 60 | - has: 61 | field: type 62 | regex: "Mutex" 63 | ``` 64 | 65 | A potential match must pass all conditions under the `all` section to be a result. There are other ways to write this rule, but this works. Note that the regexp is loose enough to match different kinds of mutex types such as `sync.Mutex` and `sync.RWMutex`. 66 | 67 | 68 | When we run the tool on the Go standard library, we get something like this (excerpt): 69 | 70 | ```sh 71 | $ ast-grep scan --rule ~/scratch/mtx.yaml 72 | [...] 73 | 74 | note[find-mtx-fields]: Mutex fields found 75 | ┌─ net/textproto/pipeline.go:29:2 76 | │ 77 | 29 │ mu sync.Mutex 78 | │ ^^^^^^^^^^^^^^^^^^^ 79 | 80 | 81 | note[find-mtx-fields]: Mutex fields found 82 | ┌─ net/rpc/server.go:191:2 83 | │ 84 | 191 │ reqLock sync.Mutex // protects freeReq 85 | │ ^^^^^^^^^^^^^^^^^^^^^ 86 | 87 | note[find-mtx-fields]: Mutex fields found 88 | ┌─ net/rpc/jsonrpc/server.go:31:2 89 | │ 90 | 31 │ mutex sync.Mutex // protects seq, pending 91 | │ ^^^^^^^^^^^^^^^^^^ 92 | 93 | [...] 94 | ``` 95 | 96 | Very useful. The tool can do much more, but that's enough for us to discover that there isn't *one* naming convention in use. Also, the mutex is not always named after the variable it protects (e.g.: `mutex` protects `seq`). 97 | 98 | 99 | So, is there at least a convention used in the majority of cases? How can we aggregate the results? 100 | 101 | ## A naming convention to rule them all 102 | 103 | Unfortunately I did not find a built-in way to post-process the results from `ast-grep`, so I resorted to outputting the matches as JSON, extracting the matched text with `jq`, and finally aggregating the results with good old AWK: 104 | 105 | ```sh 106 | $ ast-grep scan --rule ~/scratch/mtx.yaml --json | jq '.[].text' -r | awk -f ~/scratch/ast-grep-post.awk 107 | ``` 108 | 109 | And this is the ad-hoc AWK script: 110 | 111 | ```awk 112 | # ~/scratch/ast-grep-post.awk 113 | 114 | # `$1` is the variable name. 115 | { 116 | if ($1 ~ /(m|M)u$/) { 117 | stats["mu"] += 1 118 | } 119 | else if ($1 ~ /(m|M)ux$/) { 120 | stats["mux"] += 1 121 | } 122 | else if ($1 ~ /(m|M)tx$/) { 123 | stats["mtx"] += 1 124 | } 125 | else if ($1 ~ /(m|M)utex$/) { 126 | stats["mutex"] += 1 127 | } 128 | else if ($1 ~ /(l|L)ock$/) { 129 | stats["lock"] += 1 130 | } else { 131 | stats["other"] += 1 132 | } 133 | } 134 | 135 | END { 136 | for (k in stats) { 137 | print k, stats[k] 138 | } 139 | } 140 | ``` 141 | 142 | And here are the statistics (commit `7800f4f`, 2025-06-08): 143 | 144 | |Variable name suffix|Count| 145 | |--------------------|-----| 146 | | `mu` | 131 | 147 | | `mutex` | 11 | 148 | | something else | 11 | 149 | | `lock` | 6 | 150 | | `mux` | 0 | 151 | 152 | So according to these statistics: if you want to follow the same naming convention as the Go project, use `xxxMu` as a name for your mutexes. 153 | 154 | More importantly, I would add, and this is subjective: **name the mutex after the variable it protects for clarity, e.g.: `bar` and `barMu`**. In nearly every case in the Go project where this rule of thumb was not followed, a code comment was present to explain which variable the mutex protects. We might as well have this information in the mutex variable name. 155 | 156 | Even for cases where the mutex protects multiple variables, the Go developers often picked one of the variables and named the mutex after it: 157 | 158 | ```go 159 | type Link struct { 160 | hashmu sync.Mutex // protects hash, funchash 161 | hash map[string]*LSym 162 | funchash map[string]*LSym 163 | 164 | [...] 165 | } 166 | ``` 167 | 168 | ## Low-tech alternatives 169 | 170 | A quick and dirty way to achieve the same with a regexp is: 171 | 172 | ``` 173 | $ rg -t go '^\s+\w+\s+sync\.Mutex$' 174 | ``` 175 | 176 | This works since Go is a language with only one way to define a `struct` field, but some languages would be more difficult. 177 | 178 | A slightly smarter way, to only find field declarations, would be to use AWK to remember whether or not we are inside a `struct` definition: 179 | 180 | ```awk 181 | /\s+struct\s+/ { in_struct = 1 } 182 | 183 | in_struct && /\s+\w+\s+sync\.Mutex/ { print } 184 | 185 | in_struct && /^}$/ { in_struct = 0 } 186 | ``` 187 | 188 | But this might not work, or at least need to be adapted, to cover complex constructs such as defining a `struct` within a `struct`: 189 | 190 | ```go 191 | type Foo struct { 192 | bar struct { 193 | x int 194 | y int 195 | } 196 | barMtx sync.Mutex 197 | } 198 | ``` 199 | 200 | These approaches are not bullet-proof, but they will find most relevant code locations, which is enough. 201 | 202 | ## Limitations of structural search tools 203 | 204 | - Most if not all structural search tools only work on a valid AST, and not every language is supported by every tool. 205 | - The rule syntax is arcane and in parts language specific (see the addendum for details). 206 | - Speed can be an issue: `ast-grep` is relatively fast but still slower than `ripgrep` and it states that it is one of the fastest in its category. On my (admittedly very old) laptop: 207 | + `ast-grep` takes ~10s to scan ~2 millions LOC. Which is pretty good! 208 | + `find + awk` takes ~1.5s. 209 | + `ripgrep` takes ~100ms (I'm impressed). 210 | 211 | 212 | 213 | ## Conclusion 214 | 215 | I think having one structural search program in your toolbox is a good idea, especially if you intend to use it as a linter and mass refactoring tool. 216 | 217 | If all you want to do is a one-time search, text search programs such as `ripgrep` and `awk` should probably be your first stab at it. 218 | 219 | Also, I think I would prefer using a SQL-like syntax to define rules, over writing YAML with pseudo-code constructs. 220 | 221 | 222 | ## Addendum: What were mutexes named in the C implementation of the Go compiler and runtime? 223 | 224 | I wondered if the way mutexes are named in the Go project actually comes from the time were the Go compiler and much of the runtime were implemented in C. 225 | We can easily check this out with the same approach. This illustrates that `ast-grep` works for different languages, and also the slight differences. 226 | 227 | `1.4` was the last major version to use the C compiler and runtime apparently, so we checkout this commit: 228 | 229 | ```sh 230 | $ git checkout go1.4beta1 231 | ``` 232 | 233 | I initially simply changed the `language: go` field in the rule to `language: c` but was surprised nothing turned up. After toying with the [treesitter playground](https://tree-sitter.github.io/tree-sitter/7-playground.html) (`treesitter` is used under the covers), I realized that for C, the AST is structured differently and the nodes have different names. Here is the rule working for `struct` fields: 234 | 235 | 236 | ```yaml 237 | id: find-mtx-fields-struct-fields 238 | message: Mutex fields found 239 | severity: info 240 | language: c 241 | rule: 242 | kind: field_declaration 243 | all: 244 | - has: 245 | field: declarator 246 | regex: ".+" 247 | - has: 248 | field: type 249 | regex: "(M|m)utex" 250 | ``` 251 | 252 | Here are the results of this rule (excerpt): 253 | 254 | ``` 255 | note[find-mtx-fields]: Mutex fields found 256 | ┌─ runtime/malloc.h:430:2 257 | │ 258 | 430 │ Mutex specialLock; // guards specials list 259 | │ ^^^^^^^^^^^^^^^^^^^^ 260 | 261 | note[find-mtx-fields]: Mutex fields found 262 | ┌─ runtime/malloc.h:451:2 263 | │ 264 | 451 │ Mutex lock; 265 | │ ^^^^^^^^^^^^ 266 | ``` 267 | 268 | --- 269 | 270 | Right after, I realized that some mutexes are defined as global variables, so here is an additional rule file for that: 271 | 272 | ```yaml 273 | id: find-mtx-fields-vars 274 | message: Mutex variables found 275 | severity: info 276 | language: c 277 | rule: 278 | kind: declaration 279 | all: 280 | - has: 281 | field: declarator 282 | regex: ".+" 283 | - has: 284 | field: type 285 | regex: "(M|m)utex" 286 | ``` 287 | 288 | And here are the results (excerpt): 289 | 290 | ``` 291 | note[find-mtx-fields-vars]: Mutex variables found 292 | ┌─ runtime/panic.c:18:1 293 | │ 294 | 18 │ static Mutex paniclk; 295 | │ ^^^^^^^^^^^^^^^^^^^^^ 296 | 297 | note[find-mtx-fields-vars]: Mutex variables found 298 | ┌─ runtime/panic.c:162:3 299 | │ 300 | 162 │ static Mutex deadlock; 301 | │ ^^^^^^^^^^^^^^^^^^^^^^ 302 | 303 | note[find-mtx-fields-vars]: Mutex variables found 304 | ┌─ runtime/mem_plan9.c:15:1 305 | │ 306 | 15 │ static Mutex memlock; 307 | │ ^^^^^^^^^^^^^^^^^^^^^ 308 | 309 | note[find-mtx-fields-vars]: Mutex variables found 310 | ┌─ runtime/os_windows.c:586:2 311 | │ 312 | 586 │ static Mutex lock; 313 | ``` 314 | 315 | --- 316 | 317 | Funnily, it seems that: 1) the C code has much less variability in naming than the Go code: it's mostly `xxxlock`, and 2) The naming in the Go code does not stem from the C code since it's completely different. 318 | -------------------------------------------------------------------------------- /x11_x64_black_window.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/x11_x64_black_window.png -------------------------------------------------------------------------------- /x11_x64_final.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/x11_x64_final.png -------------------------------------------------------------------------------- /zero2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaultier/blog/4e2c31924b4dbf0fa726f9025d0a988a22f13e46/zero2.png --------------------------------------------------------------------------------