├── .gitignore ├── .python-version ├── README.md ├── algorithms └── 2024-06-30-Box-blur.ipynb ├── compilers ├── 2025-05-04-Single-pass-code-generation.md ├── 2025-05-18-popcnt-and-lzcnt.md └── 2025-05-25-binaryen.js.md ├── concepts ├── 2024-02-07-Pareto-frontier.md ├── 2024-02-17-Data-race.md ├── 2024-04-28-Simplex.md ├── 2024-05-10-Corner-plot.md └── 2025-01-15-Why-silicon-wafers-are-round.md ├── emacs └── 2025-02-21-Castlemacs-and-Magit.md ├── geometry └── 2024-05-23-Curvature-combs.ipynb ├── git └── 2025-03-16-Managing-dotfiles-with-a-bare-Git-repo.md ├── gpu ├── 2024-09-13-Shader-performance-profiling.md └── 2024-09-23-Debugging-shaders.md ├── images ├── Matisse-Small.ppm ├── box-blur-kernel.svg ├── chrome-framework-ignore-list.png ├── corner-plot.png ├── curvature-combs.png ├── curvature-kappa.png ├── deopt-explorer.png ├── gmail-add-scope-2.png ├── gmail-add-scope.png ├── gmail-create-client-2.png ├── gmail-create-client.png ├── gmail-download-json.png ├── gmail-oauth-consent.png ├── imgui-squigglies.png ├── instruments-choose-process.png ├── instruments-drill-down.png ├── instruments-profiling-template.png ├── metal-buffer-layout.png ├── metal-inspect-buffer.png ├── pareto-frontier.png ├── playwright-inspector.png ├── pyscript-platform.png ├── pyscript.com.png ├── scratch-interleaving.png ├── silicon-ingot.jpg ├── simplexes.jpg ├── wafer-scale.png ├── wasm-code-generation.png ├── xcode-capture-gpu-workload.png ├── xcode-debug-pixel.png ├── xcode-debug-shader.png ├── xcode-group-by-pipeline-state.png └── xcode-shader-trace.png ├── js ├── 2023-11-14-Debug-options.md ├── 2023-11-23-Finding-deoptimizations.md ├── 2024-01-25-Source-maps-in-Vite.md ├── 2024-02-03-Bun-dev-server.md ├── 2024-03-07-esbuild-dev-server.md ├── 2024-11-22-Longer-Node-stack-traces.md └── 2025-01-18-Lightweight-multitenancy.md ├── legal └── 2025-03-08-Cookie-banners.md ├── llms └── 2025-03-20-EU-hosted-llms-for-coding.md ├── macOS └── 2025-02-14-Get-image-dimensions-at-the-command-line.md ├── package-lock.json ├── package.json ├── pyproject.toml ├── python ├── 2024-03-14-The-symtable-module.md ├── 2024-05-16-Rye-and-Poetry.md ├── 2024-05-22-JAX.md ├── 2024-07-01-PyScript.md ├── 2024-07-17-installing-Python-type-stubs.md └── 2025-01-07-scripting-GMail.md ├── requirements-dev.lock ├── requirements.lock ├── scratch └── 2024-02-07-Scratch-looping-semantics.md ├── scripts └── generateIndex.js ├── sysadmin ├── 2025-03-23-Migrating-from-GMail-to-Soverin.md ├── 2025-03-30-Gmail-backups-with-imapsync.md ├── 2025-04-06-Managing-a-Bunny-CDN-config-with-Terraform.md ├── 2025-04-13-Two-way-sync-with-Unison.md ├── 2025-04-20-Backups-with-borg-and-borgmatic.md └── 2025-04-27-Logrotate.md ├── wasm └── 2024-02-22-Run-time-code-generation.md ├── webdev ├── 2024-02-16-Authentication-in-Playwright-scripts.md ├── 2025-02-14-Webp-is-awesome.md └── 2025-02-26-Styling-for-print.md └── zig ├── 2025-01-22-Getting-started-with-Zig.md └── 2025-06-08-Zig-shadowing-and-builtins.md /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | node_modules 3 | -------------------------------------------------------------------------------- /.python-version: -------------------------------------------------------------------------------- 1 | 3.12.3 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # til 2 | 3 | Short notes on useful things I've learned. Inspired by [@simonw](https://github.com/simonw/til) and [@jbranchaud](https://github.com/jbranchaud/til). 4 | 5 | --- 6 | 7 | ## By date 8 | 9 | - [Zig shadowing and builtins](./zig/2025-06-08-Zig-shadowing-and-builtins.md) - 2025-06-08 10 | - [binaryen.js](./compilers/2025-05-25-binaryen.js.md) - 2025-05-25 11 | - [The `popcnt` and `lzcnt` instructions](./compilers/2025-05-18-popcnt-and-lzcnt.md) - 2025-05-18 12 | - [Single-pass code generation](./compilers/2025-05-04-Single-pass-code-generation.md) - 2025-05-04 13 | - [logrotate](./sysadmin/2025-04-27-Logrotate.md) - 2025-04-27 14 | - [Backups with Borg and borgmatic](./sysadmin/2025-04-20-Backups-with-borg-and-borgmatic.md) - 2025-04-20 15 | - [Two-way sync with Unison](./sysadmin/2025-04-13-Two-way-sync-with-Unison.md) - 2025-04-13 16 | - [Managing a Bunny CDN config with Terraform](./sysadmin/2025-04-06-Managing-a-Bunny-CDN-config-with-Terraform.md) - 2025-04-06 17 | - [Gmail backups with imapsync](./sysadmin/2025-03-30-Gmail-backups-with-imapsync.md) - 2025-03-30 18 | - [Migrating from GMail to Soverin](./sysadmin/2025-03-23-Migrating-from-GMail-to-Soverin.md) - 2025-03-23 19 | - [EU-hosted LLMs for coding](./llms/2025-03-20-EU-hosted-llms-for-coding.md) - 2025-03-20 20 | - [Managing dotfiles with a bare Git repo](./git/2025-03-16-Managing-dotfiles-with-a-bare-Git-repo.md) - 2025-03-16 21 | - [Cookie banners](./legal/2025-03-08-Cookie-banners.md) - 2025-03-08 22 | - [Styling for print](./webdev/2025-02-26-Styling-for-print.md) - 2025-02-26 23 | - [Castlemacs and Magit](./emacs/2025-02-21-Castlemacs-and-Magit.md) - 2025-02-21 24 | - [WebP is awesome](./webdev/2025-02-14-Webp-is-awesome.md) - 2025-02-14 25 | - [Get image dimensions at the command line](./macOS/2025-02-14-Get-image-dimensions-at-the-command-line.md) - 2025-02-14 26 | - [Getting started with Zig](./zig/2025-01-22-Getting-started-with-Zig.md) - 2025-01-22 27 | - [Lightweight multitenancy for server-side JS](./js/2025-01-18-Lightweight-multitenancy.md) - 2025-01-18 28 | - [Why silicon wafers are round](./concepts/2025-01-15-Why-silicon-wafers-are-round.md) - 2025-01-15 29 | - [Scripting GMail with Python](./python/2025-01-07-scripting-GMail.md) - 2025-01-07 30 | - [Longer V8 stack traces](./js/2024-11-22-Longer-Node-stack-traces.md) - 2024-11-22 31 | - [Debugging shaders](./gpu/2024-09-23-Debugging-shaders.md) - 2024-09-23 32 | - [Shader performance profiling on macOS](./gpu/2024-09-13-Shader-performance-profiling.md) - 2024-09-13 33 | - [Installing Python type stubs](./python/2024-07-17-installing-Python-type-stubs.md) - 2024-07-17 34 | - [PyScript](./python/2024-07-01-PyScript.md) - 2024-07-01 35 | - [Box blur](./algorithms/2024-06-30-Box-blur.ipynb) - 2024-06-30 36 | - [Curvature combs](./geometry/2024-05-23-Curvature-combs.ipynb) - 2024-05-23 37 | - [JAX](./python/2024-05-22-JAX.md) - 2024-05-22 38 | - [Rye and Poetry](./python/2024-05-16-Rye-and-Poetry.md) - 2024-05-16 39 | - [Corner plot](./concepts/2024-05-10-Corner-plot.md) - 2024-05-10 40 | - [Simplex](./concepts/2024-04-28-Simplex.md) - 2024-04-28 41 | - [The symtable module in Python](./python/2024-03-14-The-symtable-module.md) - 2024-03-14 42 | - [esbuild dev server](./js/2024-03-07-esbuild-dev-server.md) - 2024-03-07 43 | - [Run-time code generation in WebAssembly](./wasm/2024-02-22-Run-time-code-generation.md) - 2024-02-22 44 | - [Data race](./concepts/2024-02-17-Data-race.md) - 2024-02-17 45 | - [Authentication in Playwright scripts](./webdev/2024-02-16-Authentication-in-Playwright-scripts.md) - 2024-02-16 46 | - [Scratch's semantics](./scratch/2024-02-07-Scratch-looping-semantics.md) - 2024-02-07 47 | - [Pareto frontier](./concepts/2024-02-07-Pareto-frontier.md) - 2024-02-07 48 | - [Bun dev server](./js/2024-02-03-Bun-dev-server.md) - 2024-02-03 49 | - [Source maps in Vite](./js/2024-01-25-Source-maps-in-Vite.md) - 2024-01-25 50 | - [Finding deoptimizations](./js/2023-11-23-Finding-deoptimizations.md) - 2023-11-23 51 | - [Debug options for JS libraries](./js/2023-11-14-Debug-options.md) - 2023-11-14 52 | 53 | ## By topic 54 | 55 | ### Algorithms 56 | 57 | - [Box blur](./algorithms/2024-06-30-Box-blur.ipynb) 58 | 59 | ### Compilers 60 | 61 | - [binaryen.js](./compilers/2025-05-25-binaryen.js.md) 62 | - [The `popcnt` and `lzcnt` instructions](./compilers/2025-05-18-popcnt-and-lzcnt.md) 63 | - [Single-pass code generation](./compilers/2025-05-04-Single-pass-code-generation.md) 64 | 65 | ### Concepts 66 | 67 | - [Why silicon wafers are round](./concepts/2025-01-15-Why-silicon-wafers-are-round.md) 68 | - [Corner plot](./concepts/2024-05-10-Corner-plot.md) 69 | - [Simplex](./concepts/2024-04-28-Simplex.md) 70 | - [Data race](./concepts/2024-02-17-Data-race.md) 71 | - [Pareto frontier](./concepts/2024-02-07-Pareto-frontier.md) 72 | 73 | ### Emacs 74 | 75 | - [Castlemacs and Magit](./emacs/2025-02-21-Castlemacs-and-Magit.md) 76 | 77 | ### Geometry 78 | 79 | - [Curvature combs](./geometry/2024-05-23-Curvature-combs.ipynb) 80 | 81 | ### Git 82 | 83 | - [Managing dotfiles with a bare Git repo](./git/2025-03-16-Managing-dotfiles-with-a-bare-Git-repo.md) 84 | 85 | ### Gpu 86 | 87 | - [Debugging shaders](./gpu/2024-09-23-Debugging-shaders.md) 88 | - [Shader performance profiling on macOS](./gpu/2024-09-13-Shader-performance-profiling.md) 89 | 90 | ### JavaScript 91 | 92 | - [Lightweight multitenancy for server-side JS](./js/2025-01-18-Lightweight-multitenancy.md) 93 | - [Longer V8 stack traces](./js/2024-11-22-Longer-Node-stack-traces.md) 94 | - [esbuild dev server](./js/2024-03-07-esbuild-dev-server.md) 95 | - [Bun dev server](./js/2024-02-03-Bun-dev-server.md) 96 | - [Source maps in Vite](./js/2024-01-25-Source-maps-in-Vite.md) 97 | - [Finding deoptimizations](./js/2023-11-23-Finding-deoptimizations.md) 98 | - [Debug options for JS libraries](./js/2023-11-14-Debug-options.md) 99 | 100 | ### Legal 101 | 102 | - [Cookie banners](./legal/2025-03-08-Cookie-banners.md) 103 | 104 | ### Llms 105 | 106 | - [EU-hosted LLMs for coding](./llms/2025-03-20-EU-hosted-llms-for-coding.md) 107 | 108 | ### MacOS 109 | 110 | - [Get image dimensions at the command line](./macOS/2025-02-14-Get-image-dimensions-at-the-command-line.md) 111 | 112 | ### Python 113 | 114 | - [Scripting GMail with Python](./python/2025-01-07-scripting-GMail.md) 115 | - [Installing Python type stubs](./python/2024-07-17-installing-Python-type-stubs.md) 116 | - [PyScript](./python/2024-07-01-PyScript.md) 117 | - [JAX](./python/2024-05-22-JAX.md) 118 | - [Rye and Poetry](./python/2024-05-16-Rye-and-Poetry.md) 119 | - [The symtable module in Python](./python/2024-03-14-The-symtable-module.md) 120 | 121 | ### Scratch 122 | 123 | - [Scratch's semantics](./scratch/2024-02-07-Scratch-looping-semantics.md) 124 | 125 | ### Sysadmin 126 | 127 | - [logrotate](./sysadmin/2025-04-27-Logrotate.md) 128 | - [Backups with Borg and borgmatic](./sysadmin/2025-04-20-Backups-with-borg-and-borgmatic.md) 129 | - [Two-way sync with Unison](./sysadmin/2025-04-13-Two-way-sync-with-Unison.md) 130 | - [Managing a Bunny CDN config with Terraform](./sysadmin/2025-04-06-Managing-a-Bunny-CDN-config-with-Terraform.md) 131 | - [Gmail backups with imapsync](./sysadmin/2025-03-30-Gmail-backups-with-imapsync.md) 132 | - [Migrating from GMail to Soverin](./sysadmin/2025-03-23-Migrating-from-GMail-to-Soverin.md) 133 | 134 | ### WebAssembly 135 | 136 | - [Run-time code generation in WebAssembly](./wasm/2024-02-22-Run-time-code-generation.md) 137 | 138 | ### Web dev 139 | 140 | - [Styling for print](./webdev/2025-02-26-Styling-for-print.md) 141 | - [WebP is awesome](./webdev/2025-02-14-Webp-is-awesome.md) 142 | - [Authentication in Playwright scripts](./webdev/2024-02-16-Authentication-in-Playwright-scripts.md) 143 | 144 | ### Zig 145 | 146 | - [Zig shadowing and builtins](./zig/2025-06-08-Zig-shadowing-and-builtins.md) 147 | - [Getting started with Zig](./zig/2025-01-22-Getting-started-with-Zig.md) 148 | -------------------------------------------------------------------------------- /compilers/2025-05-04-Single-pass-code-generation.md: -------------------------------------------------------------------------------- 1 | # Single-pass code generation 2 | 3 | Most of the compilers I've written are quite simple. Sometimes they're explicitly toys (e.g. for [Two little interpreters](https://dubroy.com/blog/two-little-interpreters/)), and other times, the situation just didn't call for anything complex. 4 | 5 | Anyways, I've recently been working a compiler that targets WebAssembly, and I was looking for ways to improve the generated code without adding too much complexity. I found a couple of interesting approaches. 6 | 7 | ## Delayed code generation 8 | 9 | The first thing I ran across was Ian Piumarta's thesis, [Delayed Code Generation in a 10 | Smalltalk-80 Compiler](https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=7e11d58c3e0797f5f899fa207557cd82138bfc29). 11 | 12 | > This thesis argues and demonstrates that it is possible to compile Smalltalk-80 directly into machine code for stock hardware, and to do this efficiently in terms of both compiler performance (the compiler must be small and fast) and generated code performance. The techniques developed for ‘delayed code generation’ in single-pass recursive descent compilation (or code generation by walking a parse tree) are applicable to almost any language. 13 | 14 | I also found that Niklaus Wirth had used a similar approach. Mössenböck's [Compiler Construction: The Art of Niklaus Wirth](http://pascal.hansotten.com/uploads/books/art3.pdf) describes it like this: 15 | 16 | > The spine of Wirth's code generators is what I call the _Item_ technique. It is described in Wirth's compiler book although it is not given a specific name. The general idea is that every value that turns up during code generation is described by a structure that is called an item. Items can describe constants, variables, intermediate results, or condition codes resulting from compare instructions. Every parsing procedure processes a language construct and returns an item that describes the value of this construct. Items are thus the attributes of the imaginary syntax tree. While the parsing procedures move top-down, the resulting items are propagated bottom-up thus creating more complex items and generating code alongside. 17 | 18 | The way I think about it is that you use a temporary and local intermediate representation, rather than a persistent global one. 19 | 20 | ## Destination-driven code generation 21 | 22 | This is another approach I learned about. The best description I found from Max Bernstein's [A quick look at destination-driven code generation](https://bernsteinbear.com/blog/ddcg/): 23 | 24 | > The key insight is that in a recursive code generator, the caller of the recursive compilation step knows where it wants the result of the callee to go — so we should not be copying everything around through some result register like RAX. We should instead pass the destination we want as a parameter. 25 | 26 | So, while delayed code generation involves information flowing _up_ the tree — from operands to operators — DDCG involves passing information _down_. 27 | 28 | ## Peephole optimization 29 | 30 | Another trick is to do peephole optimization in the `emit` methods. In other words, the optimization is done incrementally, rather than as a separate pass. 31 | 32 | E.g., here's [an example from the Lua bytecode compiler](https://github.com/lua/lua/blob/3dbb1a4b894c0744a331d4319d8d1704dc4ad943/lcode.c#L122C1-L144C2): 33 | 34 | ```c 35 | /* 36 | ** Create a OP_LOADNIL instruction, but try to optimize: if the previous 37 | ** instruction is also OP_LOADNIL and ranges are compatible, adjust 38 | ** range of previous instruction instead of emitting a new one. (For 39 | ** instance, 'local a; local b' will generate a single opcode.) 40 | */ 41 | void luaK_nil (FuncState *fs, int from, int n) { 42 | int l = from + n - 1; /* last register to set nil */ 43 | Instruction *previous = previousinstruction(fs); 44 | if (GET_OPCODE(*previous) == OP_LOADNIL) { /* previous is LOADNIL? */ 45 | int pfrom = GETARG_A(*previous); /* get previous range */ 46 | int pl = pfrom + GETARG_B(*previous); 47 | if ((pfrom <= from && from <= pl + 1) || 48 | (from <= pfrom && pfrom <= l + 1)) { /* can connect both? */ 49 | if (pfrom < from) from = pfrom; /* from = min(from, pfrom) */ 50 | if (pl > l) l = pl; /* l = max(l, pl) */ 51 | SETARG_A(*previous, from); 52 | SETARG_B(*previous, l - from); 53 | return; 54 | } /* else go through */ 55 | } 56 | luaK_codeABC(fs, OP_LOADNIL, from, n - 1, 0); /* else no optimization */ 57 | } 58 | ``` 59 | -------------------------------------------------------------------------------- /compilers/2025-05-18-popcnt-and-lzcnt.md: -------------------------------------------------------------------------------- 1 | # The `popcnt` and `lzcnt` instructions 2 | 3 | This week I was working on [some WebAssembly code involving math on powers of two](https://bsky.app/profile/dubroy.com/post/3lpcbstxi6g2a), and I discovered Wasm's `popcnt` and `clz` instructions. And then I was surprised to learn that these map to equivalent x86 instructions, which I had never heard of. 4 | 5 | ## What they do 6 | 7 | - [`popcnt`](https://www.felixcloutier.com/x86/popcnt) (short for "population count") counts the number of non-zero bits in the binary representation of a number. Wasm has `i32.popcnt`, `i64.popcnt`, and the SIMD instruction `i8x16.popcnt`. 8 | - [`lzcnt`](https://www.felixcloutier.com/x86/lzcnt) counts the number of leading zero bits (_leading_ meaning "most signficant"). Wasm has `i32.clz` and `i64.clz`. 9 | 10 | ## Trivia 11 | 12 | I wondered why I'd never encountered the x86 instructions before. Turns out that they are somewhat recent (depending on your definition of recent anyways) — `popcnt` was first supported by Intel in 2008, and `lzcnt` in 2013. 13 | 14 | Vaibhav Sagar's [You Won’t Believe This One Weird CPU Instruction!](https://vaibhavsagar.com/blog/2019/09/08/popcount/) has some interesting trivia about `popcnt` — apparently it's known as "the NSA instruction"? 15 | -------------------------------------------------------------------------------- /compilers/2025-05-25-binaryen.js.md: -------------------------------------------------------------------------------- 1 | # binaryen.js 2 | 3 | Binaryen is a compiler and toolchain infrastructure library for WebAssembly, written in C++. One of the main use cases is as a Wasm-to-Wasm optimizer: if you have a compiler that produces WebAssembly, you can use Binaryen to optimize the module for size and performance. 4 | 5 | What I didn't know is that there's also a [JS API](https://github.com/WebAssembly/binaryen/wiki/binaryen.js-API) and an [npm package](https://www.npmjs.com/package/binaryen). It's used by AssemblyScript, among others. 6 | 7 | Here's an example of how to use it: 8 | 9 | ```js 10 | import binaryen from "binaryen"; 11 | 12 | const mod = new binaryen.Module(); 13 | 14 | mod.addFunction( 15 | "add", 16 | binaryen.createType([binaryen.i32, binaryen.i32]), 17 | binaryen.i32, 18 | [binaryen.i32], 19 | mod.block(null, [ 20 | mod.local.set( 21 | 2, 22 | mod.i32.add( 23 | mod.local.get(0, binaryen.i32), 24 | mod.local.get(1, binaryen.i32), 25 | ), 26 | ), 27 | mod.return(mod.local.get(2, binaryen.i32)), 28 | ]), 29 | ); 30 | mod.addFunctionExport("add", "add"); 31 | mod.optimize(); 32 | 33 | const wasmData = mod.emitBinary(); 34 | ``` 35 | 36 | Unfortunately, the full package is quite big — the bundle that rollup produced for me was 9MB. Fine for ahead-of-time compilers, but maybe less useful if you're generating Wasm in the browser. 37 | -------------------------------------------------------------------------------- /concepts/2024-02-07-Pareto-frontier.md: -------------------------------------------------------------------------------- 1 | # Pareto frontier 2 | 3 | I learned about the concept of a _Pareto frontier_ from the paper [Copy-and-Patch Compilation](https://sillycross.github.io/assets/copy-and-patch.pdf): 4 | 5 | > We evaluate our algorithm by evaluating the copy-and-patch-based compilers we built for WebAssembly and our high-level language. [...] Our results show 6 | that our algorithm replaces all prior baseline compilers on the Pareto frontier and moves first-tier compilation closer to the performance of optimizing compilers. 7 | 8 | It's named after the Italian engineer and economist [Vilfredo Pareto](https://en.wikipedia.org/wiki/Vilfredo_Pareto) (who also gave us the _Pareto Principle_, aka the 80-20 rule). The concept applies to any problem where you're evaluating solutions based on multiple objectives. JIT compilers are a good example: you care about how long compilation takes, as well as how fast the resulting code is. 9 | 10 | I find it easiest to understand visually. You've got a number of solutions, which you can plot based on their performance on the metrics you care about: 11 | 12 | ![A scatter plot showing a number of points, with a subset of points labeled as the Pareto frontier.](../images/pareto-frontier.png) 13 | 14 | Assuming you've chosen an initial solution, a _Pareto improvement_ would be a solution that is better on both axes. In the example above, moving from C to either A or B is a Pareto improvement. For JIT compilers, that would mean a compiler that's both faster, and which produces faster code. 15 | 16 | The Pareto frontier is the set of solutions where there's no Pareto improvement — meaning, you can't improve on one axis without getting worse on the other axis. 17 | -------------------------------------------------------------------------------- /concepts/2024-02-17-Data-race.md: -------------------------------------------------------------------------------- 1 | # Data race 2 | 3 | Yesterday I was talking with [David Albert](https://twitter.com/davidbalbert) about data race safety in Swift 6, and it occurred to me that I didn't actually know what the precise definition of a _data race_ was. 4 | 5 | According to [Wikipedia](https://en.wikipedia.org/wiki/Race_condition#Data_race), the precise definition can differ depending on the language and the formal concurrency model. But, the high level definitions are pretty similar: 6 | 7 | > when multiple threads access the same memory without synchronization, and at least one access is a write. 8 | 9 | (From the [Apple Developer documentation on data races](https://developer.apple.com/documentation/xcode/data-races).) 10 | 11 | Here are the definitions for some other languages: 12 | 13 | [Java](https://docs.oracle.com/cd/E19205-01/820-0619/geojs/index.html): 14 | 15 | > A data race occurs when: 16 | > 17 | > - two or more threads in a single process access the same memory location concurrently, and 18 | > 19 | > - at least one of the accesses is for writing, and 20 | > 21 | > - the threads are not using any exclusive locks to control their accesses to that memory. 22 | 23 | [Go](https://go.dev/ref/mem#overview): 24 | 25 | > A data race is defined as a write to a memory location happening concurrently with another read or write to that same location, unless all the accesses involved are atomic data accesses as provided by the `sync/atomic` package. 26 | 27 | [Rust](https://doc.rust-lang.org/nomicon/races.html): 28 | 29 | > data races, which are defined as: 30 | > 31 | > - two or more threads concurrently accessing a location of memory 32 | > - one or more of them is a write 33 | > - one or more of them is unsynchronized 34 | -------------------------------------------------------------------------------- /concepts/2024-04-28-Simplex.md: -------------------------------------------------------------------------------- 1 | # Simplex 2 | 3 | This week I was playing with some Rust optimization libraries, and was investigating derivative-free optimization methods. Two of the algorithms I looked at — [Nelder-Mead][] and [COBYLA][] — work by modeling the candidate solution as a _simplex_. 4 | 5 | I'm sure I learned this at some point, but — a simplex is the generalization of a triangle to arbitrary dimensions. Specifically, it's the simplest possible polytope (object with flat sides) in a given dimension. So a 2D simplex (aka a _2-simplex_) is a triangle, and a 3D simplex (_3-simplex_) is a tetrahedron — a triangular pyramid. 6 | 7 | [Nelder-Mead]: https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method 8 | [COBYLA]: https://handwiki.org/wiki/COBYLA 9 | 10 |
11 | The four simplexes which can be fully represented in 3D: point, line, triangle, tetrahedron. 12 |
The four simplexes which can be fully represented in 3D — by Hjhornbeck on Wikipedia.
13 |
14 | -------------------------------------------------------------------------------- /concepts/2024-05-10-Corner-plot.md: -------------------------------------------------------------------------------- 1 | # Corner plot 2 | 3 | For a project that I'm working on, I wanted to understand the shape of the cost function for a simple optimization problem. Since there were only two parameters, I used a heat map with the parameters on the x/y axes and the color showing the cost function at each point. 4 | 5 | But I wondered what I would do with a higher-dimensional problem. And then I learned about [corner plots](https://www.star.bris.ac.uk/~mbt/topcat/sun253/MatrixPlotWindow.html): 6 | 7 | > The Corner Plot represents the relationships between multiple quantities by drawing **a scatter-like plot of every pair of coordinates, and/or a histogram-like plot of every single coordinate, and placing these on (half or all of) a square grid.** The horizontal coordinates of all the plots on each column, and the vertical coordinates of all the plots on each row, are aligned. The plots are all linked, so you can make graphical selections or click on a point to activate it in one of the panels, and the other panels will immediately reflect that action. Single-coordinate (histogram-like) plots appear on the diagonal, and coordinate-pair (scatter plot-like) plots appear off diagonal. By default only the diagonal and sub-diagonal part of the resulting plot matrix is shown, since the plots above the diagonal are equivalent to those below it, but this is configurable. This representation is variously known as a corner plot, scatter plot matrix, or pairs plot. 8 | 9 | ![A corner plot with 10 panels showing the relationships between 4 variables](../images/corner-plot.png) 10 | -------------------------------------------------------------------------------- /concepts/2025-01-15-Why-silicon-wafers-are-round.md: -------------------------------------------------------------------------------- 1 | # Why silicon wafers are round 2 | 3 | I was reading [100x Defect Tolerance: How Cerebras Solved the Yield Problem](https://cerebras.ai/blog/100x-defect-tolerance-how-cerebras-solved-the-yield-problem): 4 | 5 | > Conventional wisdom in semiconductor manufacturing has long held that bigger chips mean worse yields. Yet at Cerebras, we’ve successfully built and commercialized a chip 50x larger than the largest computer chips – and achieved comparable yields. This seeming paradox is one of our most frequently asked questions: how do we achieve a usable yield with a wafer-scale processor? 6 | 7 | And the article illustrated _wafer scale_ with the following image: 8 | 9 | ![A comparison of two diagrams showing semiconductor wafer layouts. The left diagram titled "GPUs per Wafer" shows a circular wafer divided into a grid pattern, with blue squares representing individual GPUs arranged within the circular boundary, surrounded by gray unusable areas. The right diagram titled "Wafer Scale Engine per Wafer" shows a circular wafer with a single large orange square representing one wafer-scale chip taking up most of the usable area of the wafer.](../images/wafer-scale.png) 10 | 11 | And I thought, "Huh, so why are silicon wafers always round?" 12 | 13 | Turns out it's because the Czochralski method, which [produces a cylindrical rod of monocrystalline silicon](https://en.wikipedia.org/wiki/Monocrystalline_silicon#Production): 14 | 15 | > The most common production technique is the Czochralski method, which dips a precisely oriented rod-mounted seed crystal into the molten silicon. The rod is then slowly pulled upwards and rotated simultaneously, allowing the pulled material to solidify into a monocrystalline cylindrical ingot up to 2 meters in length and weighing several hundred kilograms 16 | 17 | The cylinder is then sliced into discs before being polished. And that's why silicon wafers are round! 18 | 19 | ![A silicon ingot (or boule) for the production of wafers](../images/silicon-ingot.jpg) 20 | 21 | CC BY-SA 3.0, Link 22 | -------------------------------------------------------------------------------- /emacs/2025-02-21-Castlemacs-and-Magit.md: -------------------------------------------------------------------------------- 1 | # Castlemacs and Magit 2 | 3 | I've never been someone to get too obsessed over my editor configuration. A long, long time ago I decided to learn Emacs, via the built-in tutorial. Since then, it's been the main editor that I used on Linux, but on macOS I've always used some kind of more modern editor (Sublime, VS Code, and now Zed). 4 | 5 | At this point it's been at least five years since I used Emacs regularly. But I decided I need a better Git client in my daily workflow, and I've heard good things about Magit, so I decided to find a good emacs setup for macOS. 6 | 7 | ## Installing Emacs 8 | 9 | One of the first decisions is how to install Emacs. There are a bunch of different "ports" to choose from: 10 | 11 | - The [Emacs for Mac OS X](https://emacsformacosx.com/) binary download. 12 | - From Homebrew: 13 | * `brew install emacs` (terminal-only I think?) 14 | * `brew install --cask emacs` 15 | * `brew install emacs-plus` 16 | * `brew install emacs-mac` (by railwaycat) 17 | 18 | 🫠 19 | 20 | I recently heard about [Castlemacs](https://github.com/freetonik/castlemacs): "a simple, modern and minimalist Emacs setup tailored to macOS". They suggest the `emacs-mac` Homebrew formula so I went with that: 21 | 22 | ``` 23 | brew tap railwaycat/emacsmacport 24 | brew install emacs-mac --with-natural-title-bar --with-starter --with-modules --with-emacs-big-sur-icon 25 | ``` 26 | 27 | **NOTE:** The Castlemacs README suggests `brew install emacs-mac --with-natural-title-bar` but when I tried that, and ran `emacs` in my terminal, the GUI window wouldn't take keyboard input! I think the `--with-starter` option fixes that; it installs a helper script to `/opt/homebrew/bin/emacs`. 28 | 29 | ## Setting up Castlemacs 30 | 31 | Castlemacs is just an Emacs _configuration_, and the setup is straightforward: 32 | 33 | ``` 34 | brew install ripgrep aspell gnutls # Install dependencies 35 | mv ~/.emacs.d ~/.emacs.d.bak 36 | git clone https://github.com/freetonik/castlemacs ~/.emacs.d 37 | ``` 38 | 39 | If you had a pre-existing Emacs configuration, you'd want to move it into ~/.emacs.d/private.el. 40 | 41 | So far Castlemacs seems nice! I appreciate having a config where platform-native shortcuts (like ⌘-z, ⌘-c, etc.) "just work". In the past I always found it difficult to keep two completely separate lists of shortcuts in my brain. 42 | 43 | ## Starter scripts 44 | 45 | Starter scripts are another big rabbit hole. How do you want to launch Emacs, and what happens when you type `emacs` in the terminal? 46 | 47 | [Railwaycat's note on Emacs start helpers](https://github.com/railwaycat/homebrew-emacsmacport/blob/master/docs/emacs-start-helpers.md) seems useful, as does [this blog post by Aidan Scannell](https://www.aidanscannell.com/post/setting-up-an-emacs-playground-on-mac/). But I just did the following: 48 | 49 | - I put `(server start)` in my ~/.emacs.d/private.el. 50 | - Added the following function to my fish config: 51 | ``` 52 | function e 53 | emacsclient -n -a '' $argv; and open -a Emacs 54 | end 55 | ``` 56 | 57 | AFAIK the `open -a Emacs` is required to bring the window to the front. 58 | 59 | ## Magit 60 | 61 | I also wanted a shortcut to launch Magit from the command line. Here's what I came up with: 62 | 63 | ``` 64 | function magit 65 | set -l dir $argv[1] 66 | if test -z "$dir" 67 | set dir $PWD 68 | end 69 | emacsclient -n -a '' -e "(magit-status \"$dir\")" 70 | open -a Emacs # Bring window to the front 71 | end 72 | ``` 73 | 74 | I also wanted Magit to take up the full window (or _frame_ in Emacs-speak), so I added the following to my ~/.emacs.d/private.el: 75 | 76 | ``` 77 | ;; Always make magit take up the full frame 78 | (setq magit-display-buffer-function 'magit-display-buffer-fullframe-status-v1) 79 | ``` 80 | -------------------------------------------------------------------------------- /git/2025-03-16-Managing-dotfiles-with-a-bare-Git-repo.md: -------------------------------------------------------------------------------- 1 | # Managing dotfiles with a bare Git repo 2 | 3 | I've never had a good system for keeping my dotfiles (config files) under version control. I've tried a `dotfiles` repo with symlinks, but it never seemed all that convenient. 4 | 5 | I recently discovered a new approach that I like: a bare Git repo and a shell alias. It comes from an [HN comment by StreakyCobra](https://news.ycombinator.com/item?id=11070797): 6 | 7 | > I use: 8 | > ``` 9 | > git init --bare $HOME/.myconf 10 | > alias config='/usr/bin/git --git-dir=$HOME/.myconf/ --work-tree=$HOME' 11 | > config config status.showUntrackedFiles no 12 | > ``` 13 | > where my ~/.myconf directory is a git bare repository. Then any file within the home folder can be versioned with normal commands like: 14 | > ``` 15 | > config status 16 | > config add .vimrc 17 | > config commit -m "Add vimrc" 18 | > config add .config/redshift.conf 19 | > config commit -m "Add redshift config" 20 | > config push 21 | > ``` 22 | > And so on… 23 | > No extra tooling, no symlinks, files are tracked on a version control system, you can use different branches for different computers, you can replicate you configuration easily on new installation. 24 | 25 | My setup is based on this, but with slightly different names: my alias is `cfg`, and the repo is `~/.cfg`. Here's my full setup — note that I use the [fish shell](https://fishshell.com/): 26 | 27 | ``` 28 | # https://www.atlassian.com/git/tutorials/dotfiles 29 | function cfg 30 | command git --git-dir=$HOME/.cfg/ --work-tree=$HOME $argv 31 | end 32 | 33 | # Prevent default completions for the `cfg` command. It will 34 | # recurse into directories and basically hang forever. 35 | complete -e -c cfg 36 | complete -c cfg -f 37 | ``` 38 | 39 | The nice thing is, I can easily add _anything_ under my home directory to the repo with a simple `cfg add ~/path-to-the-file`. 40 | 41 | Who knows if I'll stick with us, but it seems nice so far. 42 | -------------------------------------------------------------------------------- /gpu/2024-09-13-Shader-performance-profiling.md: -------------------------------------------------------------------------------- 1 | # Shader performance profiling on macOS 2 | 3 | I'm writing a GLSL fragment shader, and wanted to better understand its performance. I started this page to collect my notes on how you can do this on macOS with Instruments. 4 | 5 | ## Basic profiling 6 | 7 | 1. Run your app (mine's written in Python, using [moderngl][]). 8 | 2. Start Instruments. 9 | 3. Choose "Game Performance" as the profiling template. 10 | 11 | 12 | 4. In the top left, click on "All Processes", and find your application process. 13 | 14 | 15 | 5. Click the "record" button in the top left, do stuff in your app, then stop the recording. 16 | 17 | To look at the execution time for your fragment shader: 18 | 19 | 20 | 21 | [moderngl]: https://github.com/moderngl/moderngl 22 | 23 | Unfortunately I don't know how if/how you can go into more depth. In the Metal Shading Language there are [GPU counters](https://developer.apple.com/documentation/metal/gpu_counters_and_counter_sample_buffers), but I don't think there's any way to use those in GLSL. 24 | -------------------------------------------------------------------------------- /gpu/2024-09-23-Debugging-shaders.md: -------------------------------------------------------------------------------- 1 | # Debugging shaders 2 | 3 | Some context: I'm working on a project involving 2D SDFs, and we're experimenting with moving computation to the GPU (including rendering, but not limited to that). Most of us working on the project primarily work on macOS, so CUDA didn't seem like the best choice. 4 | 5 | I started prototyping with an OpenGL fragment shader (written in GLSL). But before going further, I wanted to figure out a good workflow for debugging, either on macOS or Linux. And we're not really tied to OpenGL or GLSL, so I also looked at Metal, Vulkan, and WebGPU. 6 | 7 | ## What I tried 8 | 9 | For macOS/Metal, I only tried XCode. I'm not sure if there are any other credible options for debugging Metal shaders. 10 | 11 | For Linux (OpenGL/Vulkan), I started with the [list of debugging tools on the OpenGL wiki](https://www.khronos.org/opengl/wiki/Debugging_Tools), and narrowed it down to [RenderDoc][], [GLSL-Debugger][] aka glslDevil, and [NVIDIA Nsight Graphics][Nsight] as the most promising options. 12 | 13 | [RenderDoc]: https://renderdoc.org/ 14 | [GLSL-Debugger]: https://glsl-debugger.github.io/ 15 | [Nsight]: https://developer.nvidia.com/nsight-graphics 16 | 17 | AFAIK, the three tools above all work in basically the same way: they use [the LD_PRELOAD trick](https://www.baeldung.com/linux/ld_preload-trick-what-is) to intercept your program's OpenGL/Vulkan calls and insert instrumentation. 18 | 19 | The original code I was trying to debug is written in Python using moderngl, with a fragment shader written in GLSL. For Vulkan, I used the [triangle example](https://github.com/SaschaWillems/Vulkan/tree/master/examples/triangle) from [Sascha Willems' Vulkan examples](https://github.com/SaschaWillems/Vulkan/). 20 | 21 | ## XCode (Metal) 22 | 23 | XCode seemed like the best option by far. I can get full traces of my shader execution, including the ability to inspect all intermediate values (e.g. local variables). For some reason, I still can't get breakpoints to work, but that doesn't seem necessary when I have full traces. 24 | 25 | I wasn't able to debug shaders at all with XCode 14 / macOS Ventura, so I bit the bullet and upgraded to Sonoma and XCode 15. 26 | 27 | To capture a GPU trace: 28 | 29 | 1. Run the app in XCode. 30 | 2. Click on the Metal button in the debug toolbar. (If you can't find the debug toolbar for some reason, try setting a breakpoint somewhere in the main app code.) Then, click "Capture". 31 | 32 | 33 | 34 | To debug — 35 | 36 | 1. Click the "Debug shader" button in the debug toolbar. 37 | 38 | 39 | 40 | 2. Hover over the pixel you want to debug. Right click -> Debug pixel. 41 | 42 | 43 | 44 | 3. You should now see a trace of all the values in your shader program. If there are loops, you can even pick which iteration of the loop to view values from. 45 | 46 | 47 | 48 | ## RenderDoc 49 | 50 | I read lots of good things about RenderDoc, and I found it pretty easy to install and get started with. It turns out it only supports shader debugging for Vulkan, not OpenGL. 51 | 52 | It only partially worked with my Python (OpenGL) app. It was able to launch the app, and I could see that the interception was successful, but I wasn't able to capture a trace properly — not sure why. 53 | 54 | With a Vulkan program, I was able to step through the SPIR-V disassembly for my shader. In theory, you can compile your shaders to include source-level debug info, and then you'd be able to step through the original GLSL code. But, I didn't invest the time to figure this out. 55 | 56 | ## GLSL-Debugger 57 | 58 | I had to build this from source to get it working on my Ubuntu box (the instructions on the project page were sufficient, if a bit confusing). 59 | 60 | The basic setup is similar to RenderDoc. However, I couldn't get it working with my Python app. It 61 | 62 | ## NVIDIA Nsight 63 | 64 | Like RenderDoc, Nsight can only debug shaders in Vulkan apps. I didn't end up trying that, since it didn't seem to offer anything beyond what RenderDoc did. 65 | -------------------------------------------------------------------------------- /images/Matisse-Small.ppm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/Matisse-Small.ppm -------------------------------------------------------------------------------- /images/box-blur-kernel.svg: -------------------------------------------------------------------------------- 1 | 2 | {\displaystyle {\frac {1}{9}}{\begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}}.} 3 | 16 | 57 | -------------------------------------------------------------------------------- /images/chrome-framework-ignore-list.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/chrome-framework-ignore-list.png -------------------------------------------------------------------------------- /images/corner-plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/corner-plot.png -------------------------------------------------------------------------------- /images/curvature-combs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/curvature-combs.png -------------------------------------------------------------------------------- /images/curvature-kappa.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/curvature-kappa.png -------------------------------------------------------------------------------- /images/deopt-explorer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/deopt-explorer.png -------------------------------------------------------------------------------- /images/gmail-add-scope-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/gmail-add-scope-2.png -------------------------------------------------------------------------------- /images/gmail-add-scope.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/gmail-add-scope.png -------------------------------------------------------------------------------- /images/gmail-create-client-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/gmail-create-client-2.png -------------------------------------------------------------------------------- /images/gmail-create-client.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/gmail-create-client.png -------------------------------------------------------------------------------- /images/gmail-download-json.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/gmail-download-json.png -------------------------------------------------------------------------------- /images/gmail-oauth-consent.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/gmail-oauth-consent.png -------------------------------------------------------------------------------- /images/imgui-squigglies.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/imgui-squigglies.png -------------------------------------------------------------------------------- /images/instruments-choose-process.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/instruments-choose-process.png -------------------------------------------------------------------------------- /images/instruments-drill-down.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/instruments-drill-down.png -------------------------------------------------------------------------------- /images/instruments-profiling-template.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/instruments-profiling-template.png -------------------------------------------------------------------------------- /images/metal-buffer-layout.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/metal-buffer-layout.png -------------------------------------------------------------------------------- /images/metal-inspect-buffer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/metal-inspect-buffer.png -------------------------------------------------------------------------------- /images/pareto-frontier.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/pareto-frontier.png -------------------------------------------------------------------------------- /images/playwright-inspector.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/playwright-inspector.png -------------------------------------------------------------------------------- /images/pyscript-platform.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/pyscript-platform.png -------------------------------------------------------------------------------- /images/pyscript.com.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/pyscript.com.png -------------------------------------------------------------------------------- /images/scratch-interleaving.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/scratch-interleaving.png -------------------------------------------------------------------------------- /images/silicon-ingot.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/silicon-ingot.jpg -------------------------------------------------------------------------------- /images/simplexes.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/simplexes.jpg -------------------------------------------------------------------------------- /images/wafer-scale.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/wafer-scale.png -------------------------------------------------------------------------------- /images/wasm-code-generation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/wasm-code-generation.png -------------------------------------------------------------------------------- /images/xcode-capture-gpu-workload.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/xcode-capture-gpu-workload.png -------------------------------------------------------------------------------- /images/xcode-debug-pixel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/xcode-debug-pixel.png -------------------------------------------------------------------------------- /images/xcode-debug-shader.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/xcode-debug-shader.png -------------------------------------------------------------------------------- /images/xcode-group-by-pipeline-state.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/xcode-group-by-pipeline-state.png -------------------------------------------------------------------------------- /images/xcode-shader-trace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pdubroy/til/86873aa224a4639d5a0199f4c7d924c72cf749d3/images/xcode-shader-trace.png -------------------------------------------------------------------------------- /js/2023-11-14-Debug-options.md: -------------------------------------------------------------------------------- 1 | # Debug options for JS libraries 2 | 3 | This is a follow up to a question I asked [on Twitter](https://twitter.com/dubroy/status/1724509780369088542): 4 | 5 | > What are some good patterns for permanently including debugging code in a JS lib? 6 | > 7 | > Eg: verbose logging, saving intermediate artifacts to disk, etc. But minimal / no perf impact in prod build. 8 | > 9 | > I've done this in different ways, but never settled on patterns I'm really happy with. 10 | 11 | ## The challenge 12 | 13 | It's often useful to have debugging code that's permanently part of your codebase. This could be logging/tracing code, or something more complex (e.g., [Node's `--prof` option](https://nodejs.org/en/docs/guides/simple-profiling). 14 | 15 | In a CLI tool, it's easy to add options to enable debugging. For libraries, it's common to use environment variables. For example, GTK has [`GTK_DEBUG`](https://developer-old.gnome.org/gtk4/stable/gtk-running.html). 16 | 17 | In the past, I've struggled to come up with a good way of doing this for JavaScript libraries. One of the challenges is that your library can be consumed in many different ways — for example: 18 | 19 | - Directly imported by a Node script. 20 | - Compiled (e.g. by [ts-node][] or [vitest][]) and run on Node. 21 | - Bundled (e.g. with Webpack) and then run in the browser. 22 | 23 | [ts-node]: https://www.npmjs.com/package/ts-node 24 | [vitest]: https://vitest.dev/ 25 | [Webpack]: https://webpack.js.org/ 26 | 27 | Another challenge is that some of the debugging options might want to write files to disk. In my current project, I'm generating WebAsssembly at runtime, and need a debug option to save a binary file with the contents of the Wasm module. This should work when running on Node, and gracefully fail in the browser. 28 | 29 | ## The solution (for Vite) 30 | 31 | Here's the solution I came up with — 32 | 33 | 1. Create a separate module for debug code. Something like this: 34 | 35 | ``` 36 | import * as fs from "node:fs" 37 | 38 | export const DEBUG = process.env.FIZZBUZZ_DEBUG !== "" 39 | ``` 40 | 41 | For the pure Node.js case, nothing more is required. You could run your tests with debugging turned on with something like `FIZZBUZZ_DEBUG=true npm test`. 42 | 43 | 2. For code that's built with Vite or vitest, there two aspects to deal with: the environment variable, and the import of `"node:fs"`. Here's the config file I used: 44 | 45 | ``` 46 | import { defineConfig, PluginOption, UserConfig } from "vite" 47 | 48 | function useShims(): PluginOption { 49 | const shims = { 50 | // Return stubs, rather than empty modules, to avoid errors from 51 | // `vite build`, e.g. '"writeFileSync" is not exported by "node:fs"'. 52 | "node:fs": "export function writeFileSync() {}", 53 | } 54 | 55 | return { 56 | name: "use-shim", 57 | enforce: "pre", 58 | resolveId: (id) => (id in shims ? id : undefined), 59 | load: (id) => (id in shims ? shims[id] : undefined), 60 | } 61 | } 62 | 63 | export default defineConfig(({ mode }) => { 64 | const devConfig: UserConfig = { 65 | build: { 66 | rollupOptions: { 67 | external: ["node:fs", "node:process"], 68 | }, 69 | }, 70 | define: { 71 | "process.env.FIZZBUZZ_DEBUG": JSON.stringify(process.env.FIZZBUZZ_DEBUG ?? ""), 72 | }, 73 | } 74 | 75 | const prodConfig: UserConfig = { 76 | plugins: [useShims()], 77 | define: { 78 | "process.env.FIZZBUZZ_DEBUG": JSON.stringify(""), 79 | }, 80 | } 81 | return mode === "production" ? prodConfig : devConfig 82 | }) 83 | ``` 84 | 85 | ### For the environment variable 86 | 87 | In the development build, use the `define` option to statically replace all instances of the string `"process.env.FIZZBUZZ_DEBUG"` with **the current value of the variable at build time**. 88 | 89 | In the production build, always set it to `""`. 90 | 91 | ### For the import of `"node:fs"`: 92 | 93 | In development, the `external` option tells Vite not to include the contents of the module into my bundle, but instead try to load it at runtime. When the code runs on Node, it will successfully import the module. 94 | 95 | In the browser, it seems that Vite returns an empty module. I should investigate more to better understand exactly what's happening. 96 | 97 | For production, the `useShims` bit replaces `"node:fs"` with a stub module. 98 | 99 | One catch is that any debugging code that uses `writeFileSync` needs to handle three different conditions: 100 | 101 | 1. `DEBUG=false` 102 | 2. `DEBUG=true` (and `"node:fs"` available). 103 | 3. `DEBUG=true` but `"node:fs"` NOT available. 104 | 105 | You can handle these by guarding the code with something like `if (!DEBUG || !fs.writeFileSync) return`. 106 | 107 | ## Other considerations 108 | 109 | It may not be necessary to handle production builds specially. You can simplify the config by treating development and production identically. 110 | 111 | For larger amounts of debugging code, you may want to take advantage of Vite's dead code elimination. See [Removing assertions from production build with Vite](https://blog.battlefy.com/removing-assertions-from-production-build-with-vite). 112 | -------------------------------------------------------------------------------- /js/2023-11-23-Finding-deoptimizations.md: -------------------------------------------------------------------------------- 1 | # Finding deoptimizations 2 | 3 | If your JavaScript code is running slower than you might expect, it might be due to _deoptimizations_. 4 | 5 | When V8 compiles a JavaScript into optimized machine code, it makes certain assumptions about how the code will behave. The compiled code contains checks to detect when the assumptions are violated — and when they are, it throws out the optimized version of the method, and falls back to the interpreter. 6 | 7 | So, if you want to improve the performance of your JS code, you want to find out where deoptimization is happening. 8 | 9 | ## Deopt Explorer 10 | 11 | [Deopt Explorer](https://github.com/microsoft/deoptexplorer-vscode) is a VS Code extension for visualizing V8 trace log information. It can show you deoptimization points, along with other useful profiling data. 12 | 13 | The instructions in their [How to Use](https://github.com/microsoft/deoptexplorer-vscode?tab=readme-ov-file#how-to-use) section worked well for me. 14 | 15 |
16 | A screenshot showing Deopt Explorer in use. 17 |
Example of an eager bailout in Deopt Explorer.
18 |
19 | -------------------------------------------------------------------------------- /js/2024-01-25-Source-maps-in-Vite.md: -------------------------------------------------------------------------------- 1 | # Source maps in Vite 2 | 3 | I'm working on an app written in TypeScript, which is built with [Vite][]. The app depends on a library that's also written in TypeScript, and which ships with source maps. 4 | 5 | When I'm debugging the app code, I want to be able to step into a function from the library, and see the original TypeScript source code. 6 | 7 | [Vite]: https://vitejs.dev/ 8 | 9 | ## Problem 10 | 11 | This wasn't working for me out of the box. Stepping into one of the library functions would take me to some random, unrelated line in some other file. Sometimes it was a content script from one of my extensions, another time it was some generated WebAssembly that's used in the app. 12 | 13 | ## Solution 14 | 15 | The Chrome dev tools has an _Ignore list_ setting, and [by default, the ignore list includes `node_modules`](https://developer.chrome.com/docs/devtools/settings/ignore-list/#skip-third-party). 16 | 17 | The solution is to turn off this setting, or to remove `node_modules` from the ignore list. 18 | 19 |
20 | A screenshot of the Chrome dev tools Framework Ignore List panel, showing how to add a custom pattern to the Ignore List. 21 |
The Framework Ignore List setting in the Chrome dev tools.
22 |
23 | 24 | ## Why does this happen? 25 | 26 | At first, I was confused about why I've never encountered a problem like this before. It turns out that it's somewhat specific to Vite. 27 | 28 | [Vite doesn't bundle in dev mode](https://vitejs.dev/guide/why.html#slow-server-start). Instead, it relies on the browser-native support for ES modules. So if your app contains the following code: 29 | 30 | ```js 31 | import * as ohm from "ohm-js"; 32 | ``` 33 | 34 | …then in dev mode, Vite will transpile it to something like this: 35 | 36 | ```js 37 | import * as ohm from "/node_modules/.vite/deps/ohm-js.js?v=7d82ab79"; 38 | ``` 39 | 40 | So, you won't encounter this problem with many bundlers, because they're not serving scripts directly from `node_modules`. 41 | -------------------------------------------------------------------------------- /js/2024-02-03-Bun-dev-server.md: -------------------------------------------------------------------------------- 1 | # Bun dev server 2 | 3 | A couple years ago, I decided to stop using TypeScript for personal projects, because of how much complexity it added to the build process. So I've been excited to see [Bun](https://bun.sh/) emerge — and I've finally started using TypeScript for personal stuff again. 4 | 5 | And, while Bun can directly execute[^1] TypeScript, many of my projects target the browser. So I still need some kind of development server to compile and bundle my TypeScript source code for the browser. 6 | 7 | [^1]: Technically, Bun internally translates TypeScript into JavaScript before execution. 8 | 9 | [Vite](https://vitejs.dev/) does a great job of this out of the box, but it adds a lot of complexity, since it's a wrapper around two separate bundlers: [Rollup](https://rollupjs.org/) and [esbuild](https://esbuild.github.io/). 10 | 11 | Ideally, I wanted to find a solution using Bun's built-in bundler. 12 | 13 | ## The solution 14 | 15 | First, I set up a Bun build script, which I use to bundle my library for release, in `scripts/build.ts`: 16 | 17 | ```ts 18 | import * as url from 'node:url'; 19 | 20 | import type { BuildConfig } from "bun"; 21 | 22 | export const demoConfig: BuildConfig = { 23 | entrypoints: ["demo/app.ts"], 24 | minify: false, 25 | sourcemap: "external", 26 | }; 27 | 28 | export const config: BuildConfig = { 29 | entrypoints: ["index.ts"], 30 | minify: true, 31 | sourcemap: "external", 32 | }; 33 | 34 | if (process.argv[1] === url.fileURLToPath(import.meta.url)) { 35 | Bun.build({ 36 | ...config, 37 | outdir: "dist", 38 | }); 39 | } 40 | ``` 41 | 42 | Then, I added a script to start a development web server in `scripts/serve.ts`. It uses imports `demoConfig` from the build script, and starts a server on port 3000. 43 | 44 | ```ts 45 | import { demoConfig } from "./build"; 46 | 47 | const port = process.env.PORT || 3000; 48 | 49 | try { 50 | Bun.serve({ 51 | async fetch(req: Request): Promise { 52 | const url = new URL(req.url); 53 | switch (url.pathname) { 54 | case "/": 55 | return new Response(Bun.file("demo/index.html")); 56 | case "/bundle": 57 | const build = await Bun.build(demoConfig); 58 | if (!build.success) { 59 | console.error(build.logs); 60 | return new Response("Build failed", { status: 500 }); 61 | } 62 | return new Response(build.outputs[0]); 63 | default: 64 | const f = Bun.file("demo" + url.pathname); 65 | return (await f.exists()) 66 | ? new Response(f) 67 | : new Response("404!", { status: 404 }); 68 | } 69 | }, 70 | port, 71 | }); 72 | console.log("Server started at http://localhost:3000/"); 73 | } catch (e: any) { 74 | if ("code" in e && e.code === "EADDRINUSE") { 75 | console.log( 76 | `Port ${port} already in use. Try setting $PORT to another value.`, 77 | ); 78 | process.exit(1); 79 | } 80 | throw e; 81 | } 82 | ``` 83 | 84 | The key part is this — on requests to the `/bundle` endpoint, it builds and bundles the TypeScript sources using the `demoConfig`. 85 | 86 | ```ts 87 | switch (url.pathname) { 88 | // ... 89 | case "/bundle": 90 | const build = await Bun.build(demoConfig); 91 | return new Response(build.outputs[0]); 92 | // ... 93 | } 94 | ``` 95 | 96 | There are certainly other ways to do this, but this is working well for me. Bun's bundler is super fast, and my projects are generally small, so re-running on the bundler on every request hasn't been a problem. 97 | 98 | **Edit 2024-03-07:** Fixed a bug where `Bun.build` was being called with the configuration was imported from `scripts/serve.ts`. 99 | -------------------------------------------------------------------------------- /js/2024-03-07-esbuild-dev-server.md: -------------------------------------------------------------------------------- 1 | # esbuild dev server 2 | 3 | What I wanted was set of scripts just like I described in [Bun dev server](js/2024-02-03-Bun-dev-server.md), but for [esbuild](https://esbuild.github.io/) rather than Bun: 4 | 5 | > So I still need some kind of development server to compile and bundle my TypeScript source code for the browser. 6 | > 7 | > Vite does a great job of this out of the box, but it adds a lot of complexity, since it's a wrapper around two separate bundlers: Rollup and esbuild. 8 | > 9 | > Ideally, I wanted to find a solution using Bun's built-in bundler. 10 | 11 | ## The solution 12 | 13 | Here's `scripts/build.ts`: 14 | 15 | ```ts 16 | import * as url from 'node:url'; 17 | 18 | import * as esbuild from 'esbuild' 19 | 20 | export const mainConfig: esbuild.BuildOptions = { 21 | entryPoints: ['bootstrap.js'], 22 | outdir: 'build', 23 | bundle: true, 24 | format: 'esm' 25 | }; 26 | 27 | export const demoConfig: esbuild.BuildOptions = { 28 | ...mainConfig, 29 | entryPoints: ['index.js'], 30 | minify: false, 31 | }; 32 | 33 | if (process.argv[1] === url.fileURLToPath(import.meta.url)) { 34 | await esbuild.build(mainConfig) 35 | } 36 | ``` 37 | 38 | And `scripts/server.ts`: 39 | 40 | ```ts 41 | import * as esbuild from 'esbuild' 42 | 43 | import { demoConfig } from './build' 44 | 45 | const ctx = await esbuild.context(demoConfig) 46 | const { port } = await ctx.serve({ 47 | servedir: '.', 48 | }) 49 | console.log(`Server started at http://localhost:${port}/`); 50 | ``` 51 | 52 | It's also possible to do [live reloading](https://esbuild.github.io/api/#live-reload) but I haven't tried that yet. 53 | -------------------------------------------------------------------------------- /js/2024-11-22-Longer-Node-stack-traces.md: -------------------------------------------------------------------------------- 1 | # Longer V8 stack traces 2 | 3 | I didn't learn this one today. But, I haven't done it in a while and had to look it up today, so it seemed worth writing up. 4 | 5 | By default, V8's stack traces are limited to 10 entries. From the [V8 stack trace API docs](https://v8.dev/docs/stack-trace-api): 6 | 7 | > We collect 10 frames because it is usually enough to be useful but not so many that it has a noticeable negative performance impact 8 | 9 | There are a few different ways to change it: 10 | 11 | ## `Error.stackTraceLimit` 12 | 13 | One way to change the limit is to assign a value to `Error.stackTraceLimit` before the error occurs. For example: 14 | 15 | ```js 16 | Error.stackTraceLimit = Infinity; 17 | ``` 18 | 19 | ## `NODE_OPTIONS` 20 | 21 | ```sh 22 | $ NODE_OPTIONS="--stack-trace-limit=100" npm test 23 | ``` 24 | 25 | ## Node CLI 26 | 27 | ```sh 28 | $ node --stack-trace-limit 100 my-script.js 29 | ``` 30 | -------------------------------------------------------------------------------- /js/2025-01-18-Lightweight-multitenancy.md: -------------------------------------------------------------------------------- 1 | # Lightweight multitenancy for server-side JS 2 | 3 | Cloudflare, in [Cloud Computing without Containers](https://blog.cloudflare.com/cloud-computing-without-containers/) (2018): 4 | 5 | > Cloudflare has a cloud computing platform called Workers. Unlike essentially every other cloud computing platform I know of, it doesn’t use containers or virtual machines. We believe that is the future of Serverless and cloud computing in general. 6 | 7 | This week I had two conversations with teams exploring a deployment model like this. I decided to do a bit of research to understand this model better: what the pros and cons are, and how you can do something like this if you're _not_ Cloudflare. 8 | 9 | ## What are isolates? 10 | 11 | From the Cloudflare post: 12 | 13 | > Isolates are lightweight contexts that group variables with the code allowed to mutate them. Most importantly, a single process can run hundreds or thousands of Isolates, seamlessly switching between them. They make it possible to run untrusted code from many different customers within a single operating system process. They’re designed to start very quickly…and to not allow one Isolate to access the memory of another. 14 | 15 | The V8 docs on [Getting started with embedding V8](https://v8.dev/docs/embed) is also a good resource. 16 | 17 | ## Who else is doing this? 18 | 19 | ### Deno Deploy 20 | 21 | From [The Anatomy of an Isolate Cloud](https://deno.com/blog/anatomy-isolate-cloud) (2022): 22 | 23 | > Designing Deno Deploy was an opportunity to re-imagine how we want the developer experience of deploying an app to be, and we wanted to focus on speed and security. Instead of deploying VMs or containers, we decided on V8 isolates, which allows us to securely run untrusted code with significantly less overhead. 24 | 25 | However, unlike Cloudflare, Deno uses a **process per isolate**, using namespaces and cgroups for additional isolation. [How security and tenant isolation allows Deno Subhosting to run untrusted code securely](https://deno.com/blog/subhosting-security-run-untrusted-code) has more details. _h/t [@macwright.com](https://bsky.app/profile/macwright.com)_ 26 | 27 | Deno Deploy powers [Netlify serverless functions](https://docs.netlify.com/functions/overview/). 28 | 29 | ### Supabase 30 | 31 | [Supabase Edge Functions](https://supabase.com/edge-functions) used to be on Deno Deploy, but [according to Supabase's CEO](https://news.ycombinator.com/item?id=38623676): 32 | 33 | > We self-host the Edge Runtime now: https://github.com/supabase/edge-runtime 34 | 35 | The blog post [Supabase Edge Runtime: Self-hosted Deno Functions](https://supabase.com/blog/edge-runtime-self-hosted-deno-functions) gives some more details: 36 | 37 | > User Workers are separate JS contexts (V8 isolates) that can run a given Edge Function. They have a restricted API (for example, they don’t get access to the host machine’s environment variables). You can also control the memory and duration a User Worker can run. 38 | 39 | So it sounds like they are running multiple isolates per process, like Cloudflare. 40 | 41 | ### Vercel 42 | 43 | [Vercel Edge Runtime](https://vercel.com/docs/functions/runtimes/edge-runtime) is supposedly hosted on Cloudflare ([source](https://news.ycombinator.com/item?id=42080178)). 44 | 45 | ## Security considerations 46 | 47 | There's a great [HN discussion](https://news.ycombinator.com/item?id=31759170) between Kurt Mackey (CEO of [Fly.io](https://news.ycombinator.com/item?id=31740885)), Kenton Varda (Tech lead for Cloudflare Workers), and tptacek (a well-known HN user and security researcher who works at Fly.io). Some highlights: 48 | 49 | - mkurt: 50 | > The downside of v8 isolates is: you have to reinvent a whole bunch of stuff to get good isolation (both security and of resources). 51 | Here's an example. Under no circumstances should CloudFlare or anyone else be running multiple isolates in the same OS process. They need to be sandboxed in isolated processes. Chrome sandboxes them in isolated processes. 52 | > 53 | > Process isolation is slightly heavier weight (though forking is wicked fast) but more secure. Processes give you the advantage of using cgroups to restrict resources, namespaces to limit network access, etc. 54 | > 55 | > My understanding is that this is exactly what Deno Deploy does (https://deno.com/deploy). 56 | > 57 | > Once you've forked a process, though, you're not far off from just running something like Firecracker. This is both true and intense bias on my part. I work on https://fly.io, we use Firecracker. We started with v8 and decided it was wrong. So obviously I would be saying this. 58 | - kentonv: 59 | > The future of compute is fine-grained. Cloudflare Workers is all about fine-grained compute, that is, splitting compute into small chunks -- a single HTTP request, rather than a single server instance. This is what allows us to run every customer's code (no matter how little traffic they get) in hundreds of locations around the world, at a price accessible to everyone. 60 | > 61 | > The finer-grained your compute gets, the higher the overhead of strict process isolation gets. At the point Cloudflare is operating at, we've measured that imposing strict process isolation would mean an order of magnitude more overhead, in terms of CPU and memory usage. It depends a bit on the workload of course, but it's big. Yes, this is with all the tricks, zygote processes, etc. 62 | 63 | Some more details about the Cloudflare Workers architecture can be found in the post [Mitigating Spectre and Other Security Threats: The Cloudflare Workers Security Model](https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/). 64 | 65 | ## Can we do this ourselves? 66 | 67 | The [Supabase Edge Runtime](https://github.com/supabase/edge-runtime) is one option. 68 | 69 | Another option would be the [isolated-vm](https://www.npmjs.com/package/isolated-vm) NPM package: 70 | 71 | > `isolated-vm` is a library for nodejs which gives you access to v8's `Isolate` interface. This allows you to create JavaScript environments which are completely isolated from each other. 72 | 73 | But, they give the following caveats: 74 | 75 | > 1. Multi-process architecture. v8 is not resilient to out of memory conditions and is unable to gracefully unwind from these errors. Therefore it is possible, and even common, to crash a process with poorly-written or hostile software. I implemented a band-aid for this with the onCatastrophicError callback which quarantines a corrupted isolate, but it is not reliable. 76 | > 77 | > 2. Bundled v8 version. nodejs uses a patched version of v8 which makes development of this module more difficult than it needs to be. For some reason they're also allowed to change the v8 ABI in semver minor releases as well, which causes issues for users while upgrading nodejs. Also, some Linux distributions strip "internal" symbols from their nodejs binaries which makes usage of this module impossible. I think the way to go is to compile and link against our own version of v8. 78 | 79 | ## What about WebAssembly? 80 | 81 | WebAssembly offers a model that's conceptually similar to isolates. You can have multiple Wasm modules in the same process, and the runtime ensures that they are isolated from each other. 82 | 83 | Fastly apparently does this in their Compute@Edge platform, [according to Pat Hickey](https://news.ycombinator.com/item?id=32744291) (Principal Engineer at Fastly): 84 | 85 | > The security properties are ultimately why we invested in WebAssembly. We (Fastly; the author is my colleague) run very large numbers of WebAssembly modules, all in the same process where the plaintext HTTP requests and responses from very large numbers of our customers reside, without needing to trust the authors of those modules. 86 | 87 | More about Fastly's approach: 88 | - The WebAssembly runtime they use is [Wasmtime](https://wasmtime.dev/). 89 | - For JavaScript code, [they use SpiderMonkey.wasm](https://news.ycombinator.com/item?id=32916019) and [weval](https://github.com/bytecodealliance/weval). More details in [a talk by Chris Fallin](https://www.youtube.com/watch?v=_T3s6-C38JI) 90 | > We require fully ahead-of-time compilation…in our distributed system, this is in a separate control plane. […] We do not give the user any primitives to generate new code at runtime.” 91 | - See also Max Bernstein's [Compilers for free with weval](https://bernsteinbear.com/blog/weval/). 92 | 93 | You could debate whether this is more secure than isolates. From an [HN comment by tptacek](https://news.ycombinator.com/item?id=32990417): 94 | 95 | > I'm a little fuzzy on the multitenant security promise of WebAssembly. I haven't dug deeply into it. It seems though that it can be asymptotically as secure as the host system wrapper you build around it: that is, the bugs won't be in the WebAssembly but in the bridge between WebAssembly and the host OS. This is approximately the same situation as with v8 isolates, except that we have reasons to believe that WASM has a more trustworthy and coherent design than v8 isolates, so we're not worried about things like memory corruption breaking boundaries between cotenant WASM programs running in the same process. 96 | 97 | The main argument is that: 98 | 99 | - The surface area of WebAssembly (both the spec and the implementations) is much smaller than JavaScript and V8. 100 | - The WebAssembly spec was designed with formal verification in mind, and there's [a machine-verified version of the formalization and soundness proof](https://dl.acm.org/doi/10.1145/3167082), proving that: 101 | > no computation defined by instantiation or invocation of a valid module can "crash" or otherwise (mis)behave in ways not covered by the execution semantics given in [the WebAssembly Core Specification](https://www.w3.org/TR/wasm-core-1/). 102 | 103 | Other notes: 104 | - Wasmer's [WinterJS](https://github.com/wasmerio/winterjs) also uses Spidermonkey. 105 | - Some other providers using a Wasm-based model use QuickJS ([some HN discussion](https://news.ycombinator.com/item?id=32916019)) 106 | 107 | ## More resources 108 | 109 | - Charlie Marsh has a good writeup: [Isolates, microVMs, and WebAssembly](https://notes.crmarsh.com/isolates-microvms-and-webassembly). 110 | - [Fine-Grained Sandboxing with V8 Isolates](https://www.infoq.com/presentations/cloudflare-v8/), a 2019 talk by Kenton Varda about the Cloudflare model. 111 | - https://wasmer.io/posts/announcing-winterjs-service-workers 112 | - Tom MacWright's post [The first four Val Town runtimes](https://blog.val.town/blog/first-four-val-town-runtimes/) 113 | -------------------------------------------------------------------------------- /legal/2025-03-08-Cookie-banners.md: -------------------------------------------------------------------------------- 1 | # Cookie banners 2 | 3 | I was interested in better understanding the EU "cookie law" (officially, the _ePrivacy Directive_). Here's what I found. 4 | 5 | From [Cookies, the GDPR, and the ePrivacy Directive](https://gdpr.eu/cookies/) 6 | 7 | > To comply with the regulations governing cookies under the GDPR and the ePrivacy Directive you must: 8 | > 9 | > - Receive users’ consent before you use any cookies except strictly necessary cookies. 10 | > - Provide accurate and specific information about the data each cookie tracks and its purpose in plain language before consent is received. 11 | > - Document and store consent received from users. 12 | > - Allow users to access your service even if they refuse to allow the use of certain cookies 13 | > - Make it as easy for users to withdraw their consent as it was for them to give their consent in the first place 14 | 15 | A couple interesting aspects: 16 | 17 | 1. You don't need a cookie banner, you need _informed consent_. 18 | 2. You don't need any consent if the cookies are essential! So, what is considered _essential_? 19 | 20 | From [Opinion 04/2012 on Cookie Consent Exemption](https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/files/2012/wp194_en.pdf): 21 | 22 | > A cookie matching CRITERION B would need to pass the following tests: 23 | > 24 | > 1. A cookie is necessary to provide a specific functionality to the user (or subscriber): if cookies are disabled, the functionality will not be available. 25 | > 2. This functionality has been explicitly requested by the user (or subscriber). 26 | 27 | Session cookies used for authentication do not require consent: 28 | 29 | > When a user logs in, he explicitly requests access to the content or functionality to which he is authorized. Without the use of an authentication token stored in a cookie the user would have 30 | to provide a username/password on each page request. Therefore this authentication functionality is an essential part of the information society service he is explicitly requesting. As such these cookies are exempted under CRITERION B. 31 | 32 | Here's what it says about persistent login cookies: 33 | 34 | > Persistent login cookies which store an authentication token across browser sessions are not exempted under CRITERION B. 35 | 36 | But: 37 | 38 | > The commonly seen method of using a checkbox and a simple information note such as “remember me (uses cookies)” next to the submit form would be an appropriate means of gaining consent therefore negating the need to apply an exemption in this case. 39 | -------------------------------------------------------------------------------- /llms/2025-03-20-EU-hosted-llms-for-coding.md: -------------------------------------------------------------------------------- 1 | # EU-hosted LLMs for coding 2 | 3 | When it comes to AI, I'm neither a "believer" nor a "hater". I have a lot of concerns about generative AI _in general_, but I do find LLMs quite useful for coding. 4 | 5 | Up to now, I've been using Claude and Copilot in Zed, and for the past few weeks, Claude Code. But I recently canceled my Claude subscription and went shopping for EU-based alternatives. 6 | 7 | ## [Mistral](https://mistral.ai/) 8 | 9 | Mistral's [Le Chat](https://mistral.ai/products/le-chat) is pretty much a drop-in replacement for Claude. It's been handy for quick, one-off questions. It's quite fast, which is nice, but the model doesn't seem to be quite as good as even Claude 3.5 Sonnet. 10 | 11 | Zed supports Mistral out of the box: 12 | 13 | - Open the assistant panel (⌘-?) 14 | - Click on ⋮ -> Configure 15 | - Enter your API key under "Mistral" 16 | 17 | For the chat ("Assistant Panel") and inline assistant (Ctrl-Enter), I think you want to use `mistral-large-latest`. AFAIK there's no easy way (yet) to use [Codestral](https://mistral.ai/news/codestral-2501) as an edit prediction provider. 18 | 19 | ## Aider with DeepSeek V3 & R1 on [Nebius](https://nebius.com/) 20 | 21 | I've been using [Aider](https://aider.chat/) as a replacement for Claude Code. 22 | 23 | Nebius is a Dutch company (although they're listed on Nasdaq?) and they offer an API for DeepSeek V3 and R1 (supposedly comparable to OpenAI's 4o and o1, respectively) via their [AI Studio](https://nebius.com/ai-studio) offering. 24 | 25 | I was able to use these models with Aider by setting the following environment variables: 26 | 27 | ``` 28 | set OPENAI_API_BASE="https://api.studio.nebius.com/v1/" 29 | set OPENAI_API_KEY="abcdefg123456" # <- Use your actual Nebius API key here 30 | ``` 31 | 32 | and then running Aider with: 33 | 34 | ``` 35 | aider --model openai/deepseek-ai/DeepSeek-V3 36 | ``` 37 | or 38 | ``` 39 | aider --model openai/deepseek-ai/DeepSeek-R1 40 | ``` 41 | -------------------------------------------------------------------------------- /macOS/2025-02-14-Get-image-dimensions-at-the-command-line.md: -------------------------------------------------------------------------------- 1 | # Get image dimensions at the command line 2 | 3 | Say you have a directory with a bunch of images: 4 | 5 | ``` 6 | ~/d/til (main) [0|0|1]▸ find images -name \*.png | head -3 7 | images/corner-plot.png 8 | images/metal-buffer-layout.png 9 | images/instruments-choose-process.png 10 | ``` 11 | 12 | On macOS, you can use `sips -g pixelWidth -g pixelHeight` to get the dimensions: 13 | 14 | ``` 15 | ~/d/til (main)▸ find images -name \*.png | head -3 | xargs sips -g pixelWidth -g pixelHeight 16 | /Users/pdubroy/dev/til/images/corner-plot.png 17 | pixelWidth: 551 18 | pixelHeight: 900 19 | /Users/pdubroy/dev/til/images/metal-buffer-layout.png 20 | pixelWidth: 1517 21 | pixelHeight: 926 22 | /Users/pdubroy/dev/til/images/instruments-choose-process.png 23 | pixelWidth: 1072 24 | pixelHeight: 602 25 | ``` 26 | 27 | Very useful! 28 | 29 | You can find more documentation on the [man page](https://ss64.com/mac/sips.html). See also Simon Willison's TIL: [sips: Scriptable image processing system](https://til.simonwillison.net/macos/sips). 30 | -------------------------------------------------------------------------------- /package-lock.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "til", 3 | "lockfileVersion": 3, 4 | "requires": true, 5 | "packages": { 6 | "": { 7 | "devDependencies": { 8 | "prettier": "^3.2.5" 9 | } 10 | }, 11 | "node_modules/prettier": { 12 | "version": "3.2.5", 13 | "resolved": "https://registry.npmjs.org/prettier/-/prettier-3.2.5.tgz", 14 | "integrity": "sha512-3/GWa9aOC0YeD7LUfvOG2NiDyhOWRvt1k+rcKhOuYnMY24iiCphgneUfJDyFXd6rZCAnuLBv6UeAULtrhT/F4A==", 15 | "dev": true, 16 | "bin": { 17 | "prettier": "bin/prettier.cjs" 18 | }, 19 | "engines": { 20 | "node": ">=14" 21 | }, 22 | "funding": { 23 | "url": "https://github.com/prettier/prettier?sponsor=1" 24 | } 25 | } 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "private": true, 3 | "type": "module", 4 | "scripts": { 5 | "format": "npx prettier --write '**/*.js'", 6 | "generate": "node scripts/generateIndex.js" 7 | }, 8 | "devDependencies": { 9 | "prettier": "^3.2.5" 10 | } 11 | } 12 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [project] 2 | name = "til" 3 | version = "0.1.0" 4 | description = "Add your description here" 5 | authors = [ 6 | { name = "Patrick Dubroy", email = "pdubroy@gmail.com" } 7 | ] 8 | dependencies = [ 9 | "jax[cpu]>=0.4.28", 10 | "matplotlib>=3.9.0", 11 | "ipykernel>=6.29.4", 12 | ] 13 | readme = "README.md" 14 | requires-python = ">= 3.8" 15 | 16 | [tool.rye] 17 | managed = true 18 | virtual = true 19 | dev-dependencies = [] 20 | -------------------------------------------------------------------------------- /python/2024-03-14-The-symtable-module.md: -------------------------------------------------------------------------------- 1 | # The symtable module in Python 2 | 3 | This week, I've been reworking some of the code for [WebAssembly from the Ground Up](https://wasmgroundup.com), the book that [Mariano](https://marianoguerra.github.io/) and I are writing together. 4 | 5 | The code in question was related to the creation and manipulation of symbol tables. I was thinking about the API I wanted to present, and — as I often do — I wanted to look at some other implementations for reference. 6 | 7 | Symbol tables are usually just an implementation detail of a compiler, so I thought I'd have to dig to find some examples. But, then I learned about Python's [`symtable` module](https://docs.python.org/3/library/symtable.html). 8 | 9 | Here's a small example of how you can use it: 10 | 11 | ```python 12 | import symtable 13 | 14 | src = """ 15 | def foo(): 16 | global y 17 | x = 3 18 | y = 4 19 | """ 20 | 21 | syms = symtable.symtable(src, "foo.py", 'exec') 22 | assert syms.lookup('foo').is_namespace() 23 | foo_ns = syms.lookup('foo').get_namespace() 24 | assert foo_ns.lookup('x').get_name() == 'x' 25 | assert foo_ns.lookup('y').is_global() 26 | ``` 27 | -------------------------------------------------------------------------------- /python/2024-05-16-Rye-and-Poetry.md: -------------------------------------------------------------------------------- 1 | # Rye and Poetry 2 | 3 | I've been using Python for a little over 20 years, and it feels like the packaging and dependency management story has always been in flux. First there was easy_install, then it was all about pip and virtualenv. 4 | 5 | But I haven't been paying attention to the Python ecosystem over the past 5 years or so, and of course there are a few new tools in the mix: [Rye](https://rye-up.com/) and [Poetry](https://python-poetry.org/). They seem to be quite similar, and enable a workflow similar to tools like cargo, npm, etc. 6 | 7 | ## Rye 8 | 9 | Rye is the newest of the two. It was created by Armin Ronacher (the creator of Flask) and is now maintained by [Astral](https://astral.sh/), creators of Ruff ("an extremely fast Python linter") and uv ("an extremely fast Python package installer and resolver"). 10 | 11 | ### Basic usage 12 | 13 | ```bash 14 | rye init my-project 15 | cd my-project 16 | rye add numpy 17 | rye sync 18 | ``` 19 | 20 | ### Running scripts 21 | 22 | You can either run scripts via rye: 23 | 24 | ``` 25 | rye run python my_script.py 26 | ``` 27 | 28 | You can also rely on the shims: 29 | 30 | > After installation Rye places two shims on your PATH: python and python3. These shims have specific behavior that changes depending on if they are used within a Rye managed project or outside. 31 | > 32 | > Inside a Rye managed project they resolve to the Python interpreter of the virtualenv. This means that even if you do not enable the virtualenv, you can just run python in a shell, and it will automatically operate in the right environment. 33 | > 34 | > Outside a Rye managed project it typically resolves to your system Python, though you can also opt to have it resolve to a Rye managed Python installation for you. 35 | 36 | To enable the shims, I added the following to .zprofile: 37 | 38 | ``` 39 | source "$HOME/.rye/env" 40 | ``` 41 | 42 | ### Non-package mode 43 | 44 | To use Rye in "non-package" mode — i.e. only for managing dependencies, and not for publishing a package — you can add the following to pyproject.toml: 45 | 46 | ``` 47 | [tool.rye] 48 | virtual = true 49 | ``` 50 | 51 | I haven't done a whole lot with Rye yet, but so far my experiences have been positive. And it's certainly fast! 52 | 53 | ## Poetry 54 | 55 | Poetry is a bit more established than Rye, having been around since 2018. 56 | 57 | ### Basic usage 58 | 59 | Basic usage is almost identical to Rye: 60 | 61 | ```bash 62 | poetry new my-project 63 | cd my-project 64 | poetry add numpy 65 | poetry install 66 | ``` 67 | 68 | ### Running scripts 69 | 70 | ``` 71 | poetry run python ./my_script.py 72 | ``` 73 | 74 | Or you can activate the virtual environment in a subshell: 75 | 76 | ``` 77 | poetry shell 78 | ``` 79 | 80 | ### Non-package mode 81 | 82 | ``` 83 | [tool.poetry] 84 | package-mode = false 85 | ``` 86 | -------------------------------------------------------------------------------- /python/2024-05-22-JAX.md: -------------------------------------------------------------------------------- 1 | # JAX 2 | 3 | I was recently introduced to [JAX](https://github.com/google/jax): 4 | 5 | > JAX is a Python library for accelerator-oriented array computation and program transformation, designed for high-performance numerical computing and large-scale machine learning. 6 | 7 | My own use cases lean more towards numerical computing; in particular, I'm interested JAX's support for automatic differentiation and non-linear optimization. Below is a short example. 8 | 9 | ### Automatic differentiation 10 | 11 | First, we'll define a couple functions. We'll use the equation for a parabola, since that's a nice, smooth function whose minimum we can easily compute ourselves. 12 | 13 | ```python 14 | import jax.numpy as jnp 15 | from jax import grad 16 | from jax.scipy.optimize import minimize 17 | 18 | def parabola(h, k, x): 19 | """Parabola with minimum at (h, k).""" 20 | return (x - h) ** 2 + k 21 | 22 | def my_parabola(x): 23 | """Parabola with minimum at (2, 3).""" 24 | return parabola(2., 3., x) 25 | ``` 26 | 27 | We can use `grad` to use JAX's autodiff support to find the gradient of `my_parabola`. If we evaluate it at x=0, we should get a negative value, indicating that the function is decreasing at that point: 28 | 29 | ```python 30 | my_parabola_grad = grad(my_parabola) 31 | print(my_parabola_grad(0.)) # Prints -4.0 32 | ``` 33 | 34 | And if we evaluate it at x=3, we should get a positive value: 35 | 36 | ```python 37 | print(my_parabola_grad(3.)) # Prints 2.0 38 | ``` 39 | 40 | We can also specify which parameters the function should be differentiated with respect to. For example, we can use this to differentiate `parabola` directly, and pass the values for `h` and `k` when we evaluate the gradient: 41 | 42 | ```python 43 | parabola_grad = grad(parabola, argnums=2) 44 | print(parabola_grad(2., 3., 0.)) # Prints 2.0 45 | ``` 46 | 47 | ### Optimization 48 | 49 | We can use `minimize`, from the `jax.scipy.optimize` package to find the minimum of the function. Note that parameters of the optimization problem are passed in a JAX array as the first argument to the function. Since our function has a different interface, we need to adapt it: 50 | 51 | ```python 52 | f = lambda args_arr: my_parabola(args_arr[0]) 53 | results = minimize(f, jnp.array([0.]), method="BFGS") 54 | print(results.x) # Prints [2.] 55 | print(results.fun) # Prints 3.0 (the final return value of the function). 56 | ``` 57 | 58 | [BFGS][] is the only supported optimization method. For more methods, see [Optax][], which is a gradient processing and optimization library for JAX. 59 | 60 | [BFGS]: https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm 61 | [Optax]: https://github.com/google-deepmind/optax 62 | -------------------------------------------------------------------------------- /python/2024-07-01-PyScript.md: -------------------------------------------------------------------------------- 1 | # PyScript 2 | 3 | [PyScript][] is a WebAssembly-based platform for Python in the browser. I first heard about it via Chris Laffra, who's using it in [PySheets][] (an online, Python-based spreadsheet). 4 | 5 | PyScript supports two different versions of Python (both are compiled to WebAssembly): 6 | 7 | - [Pyodide][] is a version of the standard CPython interpreter, patched to compile to WASM and work in the browser. 8 | - [MicroPython][] is an implementation of Python 3 (including a small subset of the Python standard library) which is optimised to run on microcontrollers. 9 | 10 | A diagram showing the architecture of a web application using Python and WASM. From top to bottom: (1) A large red block labeled 'User code', (2) Two smaller light green blocks side by side, both labeled 'Framework', (3) A dark gray block with '<py-script>' written in white text, (4) A blue block at the bottom labeled 'Python interpreters compiled to WASM'. The blocks are stacked on top of each other, with the two 'Framework' blocks sitting side by side within the 'User code' block. The entire diagram is surrounded by a thin white border. 11 | 12 | [Pyscript.com][] is a web-based IDE for PyScript, à la StackBlitz, Repl.it, etc. Seems like a great way to try out PyScript without having to install anything. 13 | 14 | Screenshot of an online Python coding environment. The interface is divided into three main sections: a file explorer on the left showing files like 'index.html', 'main.py', and 'pyscript.toml'; a code editor in the center displaying a single line of Python code `print('Hello, World!')`; and a preview panel on the right showing the output 'Hello, World!' on a black background 15 | 16 | [PyScript]: https://pyscript.net 17 | [PySheets]: https://www.pysheets.app/ 18 | [Pyodide]: https://pyodide.org/ 19 | [MicroPython]: https://micropython.org/ 20 | [Pyscript.com]: https://pyscript.com/ 21 | -------------------------------------------------------------------------------- /python/2024-07-17-installing-Python-type-stubs.md: -------------------------------------------------------------------------------- 1 | # Installing Python type stubs 2 | 3 | If you're using a third-party library that doesn't ship with type hints, how can you add them yourself? 4 | 5 | We're using [pyimgui](https://pyimgui.readthedocs.io/) on a project that I'm working on, and I was getting sick of seeing the red squigglies under every imgui function call. 6 | 7 | ![Screnshot of some Python source code, with a wavy red underline under the code `imgui.selectable(name, is_selected)`](../images/imgui-squigglies.png) 8 | 9 | ## Typeshed 10 | 11 | First I learned about [Typeshed](https://github.com/python/typeshed), which contains third-party stubs for lots of popular libraries (similar to the [DefinitinlyTyped](https://github.com/DefinitelyTyped/DefinitelyTyped) project for TypeScript.) However, they don't have any stubs for pyimgui. 12 | 13 | ## Installing type stubs locally 14 | 15 | So I search a bit more and found [denballakh/pyimgui-stubs](https://github.com/denballakh/pyimgui-stubs). But, I didn't know how to install these in my project! 16 | 17 | I'm using [Zed](https://zed.dev/), which uses [Pyright](https://github.com/microsoft/pyright) as the Python language server. Here's how I installed the stubs: 18 | 19 | - I created a `typings` directory in my project root, with a `imgui` subdirectory. 20 | - According to the [Pyright documentation](https://github.com/microsoft/pyright/blob/main/docs/configuration.md), "each package's type stub file(s) are expected to be in its own subdirectory". 21 | - I put the stubs (from [imgui.pyi](https://github.com/denballakh/pyimgui-stubs/blob/master/imgui.pyi)) into a file named `__init__.pyi` in the `imgui` directory. 22 | 23 | That's it. No more red squigglies! 24 | 25 | ## Notes 26 | 27 | - Our project uses a monorepo structure, managed by [rye](https://rye.astral.sh/). I originally tried to put the stubs in a `typings` directory of the package that uses imgui, but that didn't work. 28 | - It seems that it's also possible to put the stubs in a directory with a different name, if you specify the `stubPath` option in the Pyright config. But `typings` in the default path, so I went with that. 29 | -------------------------------------------------------------------------------- /python/2025-01-07-scripting-GMail.md: -------------------------------------------------------------------------------- 1 | # Scripting GMail with Python 2 | 3 | As a freelancer, I periodically need to collect up receipts for all my business expenses. Many of those receipts are in my GMail account. I've always wanted to automate this, but was never willing to dedicate the time required to get a reliable solution working. 4 | 5 | Today I decided to take another crack at this with the help of Claude 3.5 Sonnet. 6 | 7 | ## The script 8 | 9 | Here was my prompt: 10 | 11 | > please give me a python script that uses the GMail API. Given a query string, it should search for all messages that match the query string, and then download the attachments from those messages to the current working dir. Make it a standalone script that uses the uv tool, with inline dependencies if necessary. 12 | 13 | Surprisingly, it had trouble with the inline dependencies. But after I gave it an example, it produced the following script: 14 | 15 | ```python 16 | # /// script 17 | # requires-python = ">=3.11" 18 | # dependencies = [ 19 | # "google-auth-oauthlib", 20 | # "google-api-python-client", 21 | # ] 22 | # /// 23 | 24 | import os.path 25 | from google.oauth2.credentials import Credentials 26 | from google_auth_oauthlib.flow import InstalledAppFlow 27 | from google.auth.transport.requests import Request 28 | from googleapiclient.discovery import build 29 | import pickle 30 | 31 | SCOPES = ['https://www.googleapis.com/auth/gmail.readonly'] 32 | 33 | def get_credentials(): 34 | creds = None 35 | if os.path.exists('token.pickle'): 36 | with open('token.pickle', 'rb') as token: 37 | creds = pickle.load(token) 38 | 39 | if not creds or not creds.valid: 40 | if creds and creds.expired and creds.refresh_token: 41 | creds.refresh(Request()) 42 | else: 43 | flow = InstalledAppFlow.from_client_secrets_file('credentials.json', SCOPES) 44 | creds = flow.run_local_server(port=0) 45 | with open('token.pickle', 'wb') as token: 46 | pickle.dump(creds, token) 47 | return creds 48 | 49 | def main(): 50 | query = input("Enter Gmail search query: ") 51 | 52 | creds = get_credentials() 53 | service = build('gmail', 'v1', credentials=creds) 54 | 55 | # Search for messages 56 | results = service.users().messages().list(userId='me', q=query).execute() 57 | messages = results.get('messages', []) 58 | 59 | if not messages: 60 | print("No messages found.") 61 | return 62 | 63 | # Get and display subject lines 64 | for message in messages: 65 | msg = service.users().messages().get(userId='me', id=message['id'], format='metadata', 66 | metadataHeaders=['subject']).execute() 67 | headers = msg['payload']['headers'] 68 | subject = next((header['value'] for header in headers if header['name'] == 'Subject'), '(no subject)') 69 | print(f"Subject: {subject}") 70 | if __name__ == '__main__': 71 | main() 72 | ``` 73 | 74 | ## OAuth credentials 75 | 76 | For this script to work, you'll need valid OAuth credentials in `credentials.json`. To set that up, I went to [https://console.cloud.google.com/apis/dashboard](https://console.cloud.google.com/apis/dashboard). 77 | 78 | The UI is quite confusing, and I got bounced back and forth between the "new experience" and the old one (I guess). But I think this is what I did: 79 | 80 | 1. Created a new project. 81 | 2. Created an OAuth consent screen. The email account I want to access is a Google Workspace account, so I created used the _Internal_ user type, which means that it can only be used within my organization. 82 | 83 | OAuth Consent Screen 84 | 85 | 3. Under _Clients_, clicked _Create client_. For _Application type_, I chose _Desktop app_, and gave it the name "Pat's GMail bot". 86 | 87 | 88 | 89 | 90 | 4. Under _Data access_, clicked _Add or remove scopes_, and then under _Manually add scopes_, I entered `https://www.googleapis.com/auth/gmail.modify`. 91 | 92 | 93 | 94 | 95 | 5. Finally, downloaded the OAuth client credentials as JSON. 96 | 97 | 98 | 99 | ## Run the script 100 | 101 | - `uv run gmail_search.py`. 102 | - It will open up a window for the OAuth flow. When that was complete, my script hung. So I Ctrl-C'd and then re-ran, and everything worked. 103 | -------------------------------------------------------------------------------- /requirements-dev.lock: -------------------------------------------------------------------------------- 1 | # generated by rye 2 | # use `rye lock` or `rye sync` to update this lockfile 3 | # 4 | # last locked with the following flags: 5 | # pre: false 6 | # features: [] 7 | # all-features: false 8 | # with-sources: false 9 | 10 | appnope==0.1.4 11 | # via ipykernel 12 | asttokens==2.4.1 13 | # via stack-data 14 | comm==0.2.2 15 | # via ipykernel 16 | contourpy==1.2.1 17 | # via matplotlib 18 | cycler==0.12.1 19 | # via matplotlib 20 | debugpy==1.8.1 21 | # via ipykernel 22 | decorator==5.1.1 23 | # via ipython 24 | executing==2.0.1 25 | # via stack-data 26 | fonttools==4.51.0 27 | # via matplotlib 28 | ipykernel==6.29.4 29 | ipython==8.24.0 30 | # via ipykernel 31 | jax==0.4.28 32 | jaxlib==0.4.28 33 | # via jax 34 | jedi==0.19.1 35 | # via ipython 36 | jupyter-client==8.6.2 37 | # via ipykernel 38 | jupyter-core==5.7.2 39 | # via ipykernel 40 | # via jupyter-client 41 | kiwisolver==1.4.5 42 | # via matplotlib 43 | matplotlib==3.9.0 44 | matplotlib-inline==0.1.7 45 | # via ipykernel 46 | # via ipython 47 | ml-dtypes==0.4.0 48 | # via jax 49 | # via jaxlib 50 | nest-asyncio==1.6.0 51 | # via ipykernel 52 | numpy==1.26.4 53 | # via contourpy 54 | # via jax 55 | # via jaxlib 56 | # via matplotlib 57 | # via ml-dtypes 58 | # via opt-einsum 59 | # via scipy 60 | opt-einsum==3.3.0 61 | # via jax 62 | packaging==24.0 63 | # via ipykernel 64 | # via matplotlib 65 | parso==0.8.4 66 | # via jedi 67 | pexpect==4.9.0 68 | # via ipython 69 | pillow==10.3.0 70 | # via matplotlib 71 | platformdirs==4.2.2 72 | # via jupyter-core 73 | prompt-toolkit==3.0.43 74 | # via ipython 75 | psutil==5.9.8 76 | # via ipykernel 77 | ptyprocess==0.7.0 78 | # via pexpect 79 | pure-eval==0.2.2 80 | # via stack-data 81 | pygments==2.18.0 82 | # via ipython 83 | pyparsing==3.1.2 84 | # via matplotlib 85 | python-dateutil==2.9.0.post0 86 | # via jupyter-client 87 | # via matplotlib 88 | pyzmq==26.0.3 89 | # via ipykernel 90 | # via jupyter-client 91 | scipy==1.13.0 92 | # via jax 93 | # via jaxlib 94 | six==1.16.0 95 | # via asttokens 96 | # via python-dateutil 97 | stack-data==0.6.3 98 | # via ipython 99 | tornado==6.4 100 | # via ipykernel 101 | # via jupyter-client 102 | traitlets==5.14.3 103 | # via comm 104 | # via ipykernel 105 | # via ipython 106 | # via jupyter-client 107 | # via jupyter-core 108 | # via matplotlib-inline 109 | wcwidth==0.2.13 110 | # via prompt-toolkit 111 | -------------------------------------------------------------------------------- /requirements.lock: -------------------------------------------------------------------------------- 1 | # generated by rye 2 | # use `rye lock` or `rye sync` to update this lockfile 3 | # 4 | # last locked with the following flags: 5 | # pre: false 6 | # features: [] 7 | # all-features: false 8 | # with-sources: false 9 | 10 | appnope==0.1.4 11 | # via ipykernel 12 | asttokens==2.4.1 13 | # via stack-data 14 | comm==0.2.2 15 | # via ipykernel 16 | contourpy==1.2.1 17 | # via matplotlib 18 | cycler==0.12.1 19 | # via matplotlib 20 | debugpy==1.8.1 21 | # via ipykernel 22 | decorator==5.1.1 23 | # via ipython 24 | executing==2.0.1 25 | # via stack-data 26 | fonttools==4.51.0 27 | # via matplotlib 28 | ipykernel==6.29.4 29 | ipython==8.24.0 30 | # via ipykernel 31 | jax==0.4.28 32 | jaxlib==0.4.28 33 | # via jax 34 | jedi==0.19.1 35 | # via ipython 36 | jupyter-client==8.6.2 37 | # via ipykernel 38 | jupyter-core==5.7.2 39 | # via ipykernel 40 | # via jupyter-client 41 | kiwisolver==1.4.5 42 | # via matplotlib 43 | matplotlib==3.9.0 44 | matplotlib-inline==0.1.7 45 | # via ipykernel 46 | # via ipython 47 | ml-dtypes==0.4.0 48 | # via jax 49 | # via jaxlib 50 | nest-asyncio==1.6.0 51 | # via ipykernel 52 | numpy==1.26.4 53 | # via contourpy 54 | # via jax 55 | # via jaxlib 56 | # via matplotlib 57 | # via ml-dtypes 58 | # via opt-einsum 59 | # via scipy 60 | opt-einsum==3.3.0 61 | # via jax 62 | packaging==24.0 63 | # via ipykernel 64 | # via matplotlib 65 | parso==0.8.4 66 | # via jedi 67 | pexpect==4.9.0 68 | # via ipython 69 | pillow==10.3.0 70 | # via matplotlib 71 | platformdirs==4.2.2 72 | # via jupyter-core 73 | prompt-toolkit==3.0.43 74 | # via ipython 75 | psutil==5.9.8 76 | # via ipykernel 77 | ptyprocess==0.7.0 78 | # via pexpect 79 | pure-eval==0.2.2 80 | # via stack-data 81 | pygments==2.18.0 82 | # via ipython 83 | pyparsing==3.1.2 84 | # via matplotlib 85 | python-dateutil==2.9.0.post0 86 | # via jupyter-client 87 | # via matplotlib 88 | pyzmq==26.0.3 89 | # via ipykernel 90 | # via jupyter-client 91 | scipy==1.13.0 92 | # via jax 93 | # via jaxlib 94 | six==1.16.0 95 | # via asttokens 96 | # via python-dateutil 97 | stack-data==0.6.3 98 | # via ipython 99 | tornado==6.4 100 | # via ipykernel 101 | # via jupyter-client 102 | traitlets==5.14.3 103 | # via comm 104 | # via ipykernel 105 | # via ipython 106 | # via jupyter-client 107 | # via jupyter-core 108 | # via matplotlib-inline 109 | wcwidth==0.2.13 110 | # via prompt-toolkit 111 | -------------------------------------------------------------------------------- /scratch/2024-02-07-Scratch-looping-semantics.md: -------------------------------------------------------------------------------- 1 | # Scratch's semantics 2 | 3 | Some things I learned about the semantics of `repeat` blocks in [Scratch](https://scratch.mit.edu/). 4 | 5 | ## Delays 6 | 7 | According to the [official Scratch wiki](https://en.scratch-wiki.info/wiki/Repeat_Until_()_(block)): 8 | 9 | > After each iteration of the loop, a delay of 1/30 of a second (or one frame) is added before the next iteration continues. This allows for animations to be run smoother. 10 | 11 | But, this is not quite true. If you update a variable every iteration of the loop, [you'll see the variable update instantaneously](https://twitter.com/dubroy/status/1753435097674534965). 12 | 13 | A [discussion on the Scratch forum](https://scratch.mit.edu/discuss/topic/313368/?page=1#post-3230950) provides a helpful answer: 14 | 15 | > If it's not a non-refresh script loop then Scratch will go through each of the currently running scripts and execute its blocks up until it reaches a ‘yield point’. Once it reaches such a point, that script will ‘yield’ (stop executing) and Scratch will move on to the next script and do the same thing. A yield point is either the end of a loop (forever, repeat, repeat until), or some kind of wait block (broadcast and wait, ask and wait, wait N secs, wait until), or the end of the script. 16 | > 17 | > While Scratch is going through those currently running scripts, it keeps a note of when one of them executed a block that can (potentially) change something on-screen. This includes such things as motion blocks (e.g: move N steps), looks blocks (e.g. switch costume), pen down, etc. It will often count as a change even if nothing actually changed (e.g. if the costume switched to the costume it's already on, or if it moved to the position it already has) – although it does take into account if the sprite is hidden and the pen is not down (the change won't be seen, so it doesn't register as a change). 18 | > 19 | > If it did NOT execute such a block, OR if it is in turbo mode, then Scratch will NOT (usually) wait for the next screen refresh, but will go straight back to executing another pass through all running scripts. That is, unless it is coming up to time for the next screen refresh (i.e. it has been executing scripts like this for nearly 1/30th second since the last refresh). 20 | > 21 | > Conversely, if it DID execute such a block, and it's NOT in turbo mode, or if it has been nearly 1/30th sec since the last refresh, then Scratch will commit all pen drawing to the pen canvas (which can take some time if there has been a lot of drawing), and wait for the next screen refresh. 22 | 23 | ## Interleaving 24 | 25 | What about interleaving? If there are two scripts with repeat blocks, and one of them causes a redraw and the other doesn't, what happens? 26 | 27 | ![](../images/scratch-interleaving.png) 28 | 29 | The setup is as follows: 30 | 31 | - Two scripts with the same trigger. 32 | - Script #1 contains a repeat that, reads a variable, and does something that causes redraw. 33 | - Script #2 contains a repeat that *doesn't* cause redraw, but updates the variable. 34 | 35 | At the end, the `vals` list contains the values 1 through 10. This seems to indicate that both scripts yield between each iteration of the repeat block. 36 | 37 | Naively, I would have expected Script #2 to complete in a single frame, since it doesn't cause a redraw. 38 | -------------------------------------------------------------------------------- /scripts/generateIndex.js: -------------------------------------------------------------------------------- 1 | import fs from "fs"; 2 | import path from "path"; 3 | 4 | const byDate = []; 5 | const byTopic = {}; 6 | 7 | const topicTitleByDir = { 8 | js: "JavaScript", 9 | wasm: "WebAssembly", 10 | webdev: "Web dev", 11 | }; 12 | 13 | const topicTitle = (dir) => 14 | topicTitleByDir[dir] ?? dir.charAt(0).toUpperCase() + dir.slice(1); 15 | 16 | function traverseDir(dir) { 17 | const files = fs.readdirSync(dir); 18 | files.forEach((file) => { 19 | const filePath = path.join(dir, file); 20 | const fileStat = fs.statSync(filePath); 21 | 22 | if (fileStat.isDirectory()) { 23 | if (file[0] !== ".") traverseDir(filePath); 24 | } else if ( 25 | [".md", ".ipynb"].includes(path.extname(file)) && 26 | file !== "README.md" 27 | ) { 28 | const cat = path.basename(dir); 29 | const date = file.match(/(\d{4}-\d{2}-\d{2})/)[0]; 30 | let title = file 31 | .slice(date.length + 1) 32 | .replace("-", " ") 33 | .replace(".ipynb", ""); 34 | if (path.extname(file) === ".md") { 35 | title = fs 36 | .readFileSync(filePath, "utf-8") 37 | .split("\n")[0] 38 | .replace("# ", ""); 39 | } 40 | 41 | byDate.push({ date, entry: `- [${title}](./${filePath}) - ${date}` }); 42 | 43 | if (!byTopic.hasOwnProperty(cat)) { 44 | byTopic[cat] = []; 45 | } 46 | byTopic[cat].push({ date, entry: `- [${title}](./${filePath})` }); 47 | } 48 | }); 49 | } 50 | 51 | traverseDir("."); 52 | byDate.sort((a, b) => (a.date < b.date ? 1 : -1)); 53 | 54 | const topicKeys = Object.keys(byTopic).sort((a, b) => (a < b ? -1 : 1)); 55 | Object.values(byTopic).forEach((entries) => 56 | entries.sort((a, b) => (a.date < b.date ? 1 : -1)), 57 | ); 58 | 59 | const readmeContent = `# til 60 | 61 | Short notes on useful things I've learned. Inspired by [@simonw](https://github.com/simonw/til) and [@jbranchaud](https://github.com/jbranchaud/til). 62 | 63 | --- 64 | 65 | ## By date 66 | 67 | ${byDate.map(({ entry }) => entry).join("\n")} 68 | 69 | ## By topic 70 | 71 | ${topicKeys.map((topic) => `### ${topicTitle(topic)}\n\n${byTopic[topic].map(({ entry }) => entry).join("\n")}`).join("\n\n")} 72 | `; 73 | 74 | fs.writeFileSync("README.md", readmeContent); 75 | -------------------------------------------------------------------------------- /sysadmin/2025-03-23-Migrating-from-GMail-to-Soverin.md: -------------------------------------------------------------------------------- 1 | # Migrating from GMail to Soverin 2 | 3 | I registered my domain almost exactly 25 years ago, in large part because I didn't want to have to change my email address ever again. After self-hosting for a few years, I started using "Google Apps for your Domain" (now known as Google Workspace) in 2006 or so, and have been using it every since. 4 | 5 | This weekend I migrated my email to [Soverin](https://soverin.com/). For €3.25 a month, you can get email + CalDAV hosting with 25GB of storage from an EU-based company. Not bad! 6 | 7 | I ran into a few issues with my setup and thought I'd write them up here. 8 | 9 | ## Importing from GMail 10 | 11 | After signing up, I wanted to import my email from GMail before changing the DNS records. I created a new app password especially for Soverin and started a new import (under _My mailbox_ > _Import email_). But it kept showing the status "Verification failed". 12 | 13 | I eventually realized that it was failing because I hadn't yet verified my _domain_. (I'm not sure why that's required before importing email.) Fixing this was easy ([How to connect your existing custom domain](https://soverin.com/help/domain-custom-add)) and afterwards my GMail import ran successfully. 14 | 15 | ## Problems with the Cloudflare SPF record 16 | 17 | After that, I was able to connect to the Soverin IMAP server with Apple Mail. Then, I updated all my DNS records so that my mail would be delivered to Soverin rather than GMail. This all worked fine. 18 | 19 | What didn't work was _sending_ email from Apple Mail. DNS for my domain is handled by Cloudflare, and I had replaced my old SPF record with the new one for Soverin. However, Apple Mail was giving an SPF error when I tried to send an email from my account. 20 | 21 | I tried the [EasyDMARC SPF Checker](https://easydmarc.com/tools/spf-lookup), and it gave an error related to the quotation marks. It turns out that [the Cloudflare UI automatically adds quotation marks around TXT records](https://community.cloudflare.com/t/cant-remove-quotes-from-txt-record/737786). And apparently this was causing problems with Soverin's SPF check. 22 | 23 | The workaround I found is to create the record via the API. I created a new API token, and [found the zone ID](https://developers.cloudflare.com/fundamentals/setup/find-account-and-zone-ids/). I put both of these into environment variables and then ran the following: 24 | 25 | ```bash 26 | curl -X POST "https://api.cloudflare.com/client/v4/zones/$CLOUDFLARE_ZONE_ID_DUBROY_COM/dns_records" \ 27 | -H "Authorization: Bearer $CLOUDFLARE_DNS_API_TOKEN" \ 28 | -H "Content-Type: application/json" \ 29 | --data '{ 30 | "type": "TXT", 31 | "name": "@", 32 | "content": "v=spf1 include:soverin.net ~all", 33 | "ttl": 60 34 | }' 35 | ``` 36 | 37 | Everything worked after that. 🙌 38 | -------------------------------------------------------------------------------- /sysadmin/2025-03-30-Gmail-backups-with-imapsync.md: -------------------------------------------------------------------------------- 1 | # Gmail backups with imapsync 2 | 3 | I recently realized that I had ~18 years of email stored in Gmail, with no backup. This week I fixed that, using [imapsync][] to backup up the entire contents of my Gmail (~10GB) to [mailbox.org](https://mailbox.org/). 4 | 5 | [imapsync]: https://imapsync.lamiral.info/ 6 | 7 | The imapsync documentation is great, but it took me a few tries to find all the right flags for my use case. Here's what I ended up with: 8 | 9 | ``` 10 | #!/usr/bin/env bash 11 | imapsync \ 12 | --user1 xyz@gmail.com \ 13 | --password1 "${GMAIL_APP_PASSWORD}" \ 14 | --host2 imap.mailbox.org \ 15 | --user2 abc@mailbox.org \ 16 | --password2 "${MAILBOX_ORG_PASSWORD}" \ 17 | --gmail1 \ 18 | --folderfirst ohm \ 19 | --folderfirst GitHub \ 20 | --exclude "^\[Gmail\]/Important$" \ 21 | --exclude "^\[Gmail\]/Spam$" \ 22 | --exclude "^\[Gmail\]/Starred$" \ 23 | --exclude "^\[Gmail\]/Trash$" \ 24 | --f1f2 "[Gmail]/Spam"="Spam" \ 25 | --f1f2 "[Gmail]/All Mail"="Archive" \ 26 | --delete2folders \ 27 | --delete2foldersbutnot "/\d{4}/" \ 28 | --nofoldersizes \ 29 | --nofoldersizesatend \ 30 | "$@" 31 | ``` 32 | 33 | A quick explanation: 34 | - `--folderfirst`: Gmail's _labels_ are exposed as IMAP folders, which means that the same email might appear in multiple folders. By using `--folderfirst`, imapsync will sync emails with that label to a folder of the same name, and all other labels will be dropped. 35 | - I don't use `Important` or `Starred` in Gmail, so I didn't sync those. `Spam` and `Trash` I didn't care about syncing. 36 | - `--delete2folders` removes any folders on the dest that don't appear in Gmail, and `--delete2foldersbutnot` lets you specify a regex for folders that _shouldn't_ be deleted. 37 | - The `$@` means that any arguments to the script will be passed in to imapsync. So I can call my script like this: `bin/gmail-sync.sh --dry` and it will pass the `--dry` arg to imapsync. 38 | -------------------------------------------------------------------------------- /sysadmin/2025-04-06-Managing-a-Bunny-CDN-config-with-Terraform.md: -------------------------------------------------------------------------------- 1 | # Managing a Bunny CDN config with Terraform 2 | 3 | I'm moving my web site over to [Bunny](https://bunny.net/), and have a bunch of redirect rules in my nginx config that I wanted to turn into Edge Rules. The thought of doing them by hand in the GUI made me sad, but Bunny has an [official Terraform provider](https://registry.terraform.io/providers/BunnyWay/bunnynet/latest/docs), so I decided to give that a try. 4 | 5 | _Caveat: I'm not a Terraform expert, so take this with a grain of salt!_ 6 | 7 | ## Getting started 8 | 9 | I installed Terraform via Homebrew: `brew tap hashicorp/tap && brew install hashicorp/tap/terraform`. Then, my first goal was to come up with a "no-op" Terraform config — something where running `terraform apply` wouldn't make any changes. Here's my config (in main.tf): 10 | 11 | ```tf 12 | terraform { 13 | required_providers { 14 | bunnynet = { 15 | source = "BunnyWay/bunnynet" 16 | } 17 | } 18 | } 19 | 20 | variable "bunnynet_api_key" { 21 | type = string 22 | } 23 | 24 | provider "bunnynet" { 25 | api_key = var.bunnynet_api_key 26 | } 27 | 28 | import { 29 | to = bunnynet_storage_zone.dubroy 30 | id = 1234567 31 | } 32 | 33 | resource "bunnynet_storage_zone" "dubroy" { 34 | name = "dubroy" 35 | region = "DE" 36 | zone_tier = "Edge" 37 | replication_regions = [ 38 | "BR", 39 | "NY", 40 | "SG", 41 | "WA", 42 | ] 43 | } 44 | 45 | import { 46 | to = bunnynet_pullzone.dubroy 47 | id = 7654321 48 | } 49 | 50 | resource "bunnynet_pullzone" "dubroy" { 51 | name = "dubroy" 52 | 53 | origin { 54 | type = "StorageZone" 55 | storagezone = bunnynet_storage_zone.dubroy.id 56 | host_header = "dubroy.com" 57 | } 58 | 59 | routing { 60 | tier = "Standard" 61 | zones = [ 62 | "EU", 63 | "US", 64 | "ASIA" 65 | ] 66 | } 67 | } 68 | 69 | import { 70 | to = bunnynet_pullzone_hostname.dubroycom 71 | id = "7654321|dubroy.com" 72 | } 73 | 74 | resource "bunnynet_pullzone_hostname" "dubroycom" { 75 | pullzone = bunnynet_pullzone.dubroy.id 76 | name = "dubroy.com" 77 | tls_enabled = true 78 | force_ssl = true 79 | } 80 | ``` 81 | 82 | Notes: 83 | - This assumes your Bunny API key is in an environment variable named `TF_VAR_bunnynet_api_key`. 84 | - You'll need to replace `1234567` and `7654321` with proper ID of your storage zone and pull zone, respectively. You can find those in the Bunny control panel. 85 | - If you don't have an existing pull zone / storage zone, you can remove the `import` blocks altogether. 86 | 87 | ## Next steps 88 | 89 | - `terraform init` to initialize 90 | - `terraform plan` to see what actions Terraform would take. I iterated on this until there were 0 changes, only imports: 91 | ``` 92 | Plan: 5 to import, 5 to add, 0 to change, 0 to destroy. 93 | ``` 94 | - `terraform apply` 95 | 96 | I committed these two files: 97 | - `main.tf` 98 | - `.terraform.lock.hcl` 99 | 100 | And updated my .gitignore to ignore the others: 101 | 102 | ``` 103 | .terraform 104 | terraform.tfstate* 105 | ``` 106 | 107 | ## Edge rules 108 | 109 | Here's an example of defining an edge rule, based one of Gabriel Garrido's [Bunny CDN Edge Rules for HTML canonical URLs](https://garrido.io/notes/bunny-edge-rules-for-html-canonical-urls/): 110 | 111 | ```tf 112 | resource "bunnynet_pullzone_edgerule" "strip_index" { 113 | enabled = true 114 | pullzone = bunnynet_pullzone.dubroy.id 115 | description = "Strip index.html" 116 | 117 | actions = [ 118 | { 119 | type = "Redirect" 120 | parameter1 = "https://%%{Url.Hostname}%%{Url.Directory}" 121 | parameter2 = "301" 122 | parameter3 = null 123 | } 124 | ] 125 | 126 | match_type = "MatchAll" 127 | triggers = [ 128 | { 129 | type = "Url" 130 | match_type = "MatchAny" 131 | patterns = [ 132 | "*/index.html*", 133 | ] 134 | parameter1 = null 135 | parameter2 = null 136 | }, 137 | { 138 | match_type = "MatchNone" 139 | patterns = [ 140 | "?*=*", 141 | ] 142 | type = "UrlQueryString" 143 | parameter1 = null 144 | parameter2 = null 145 | }, 146 | ] 147 | } 148 | ``` 149 | 150 | Notes: 151 | - You can import an existing edge rule like so: 152 | ``` 153 | import { 154 | to = bunnynet_pullzone_edgerule.strip_index 155 | id = "7654321|40235ee2-4468-4023-5383-8aeee29073ac" 156 | } 157 | ``` 158 | …where `7654321` is your pull zone ID, and the rest is the edge rule GUID, which you can get from the URL if you edit the rule in the Bunny control panel. 159 | - Bunny appears to support a maximum of 5 patterns per trigger, and 5 triggers per rule. 160 | -------------------------------------------------------------------------------- /sysadmin/2025-04-13-Two-way-sync-with-Unison.md: -------------------------------------------------------------------------------- 1 | # Two-way sync with Unison 2 | 3 | Usually when I want to sync things between machines (e.g., for backup) I'd use `rsync`. But, despite the name, what `rsync` does isn't really _syncing_; it's better described as one-way _mirroring_. 4 | 5 | [Unison](https://github.com/bcpierce00/unison), otoh, is designed for proper two-way sync. I'd first heard of it years ago, but never took a close look at it since I didn't have a need for it. But now I do: I wanted to be able to sync a directory on my laptop with my Hetzner VPS, and be able to propagate changes in both directions. 6 | 7 | ## Installation 8 | 9 | On my Mac, I installed it with Homebrew: 10 | 11 | ``` 12 | brew install unison 13 | ``` 14 | 15 | On the server: 16 | 17 | ``` 18 | sudo apt install unison 19 | ``` 20 | 21 | ## Syncing 22 | 23 | You don't have to do it this way, but a handy way to use Unison is to create a _profile_. Here's my profile (in `~/.unison/hetzner.prf`): 24 | 25 | ``` 26 | root = /Users/pdubroy/Docs 27 | root = ssh://pdubroy@123.123.123.123//mnt/data/dav 28 | 29 | ignore = Name .DS_Store 30 | ignore = Name ._* 31 | perms = 0 32 | maxsizethreshold = 102400 33 | ``` 34 | 35 | Then I can sync via `unison hetzner`. A nice feature of Unison is that by default, it prompts you for all changes and lets you manually choose a resolution. You can use `-batch` to skip the manual resolution for non-conflicting changes, and use the `-prefer` to automatically resolve conflicts by picking a particular side. For example, I use: 36 | 37 | ``` 38 | unison hetzner -batch -prefer /Users/pdubroy/Docs 39 | ``` 40 | 41 | ## Trivia 42 | 43 | Unison was designed by Benjamin C. Pierce — yeah, _that_ Benjamin C. Pierce (of [TAPL](https://www.cis.upenn.edu/~bcpierce/tapl/) fame). And it's written in OCaml. 😎 44 | -------------------------------------------------------------------------------- /sysadmin/2025-04-20-Backups-with-borg-and-borgmatic.md: -------------------------------------------------------------------------------- 1 | # Backups with Borg and borgmatic 2 | 3 | I recently set up an old Mac Mini with Debian, mainly to use as a file server for my family. I decided to set up off-site backups to a [Hetzner Storage Box](https://www.hetzner.com/storage/storage-box/). 4 | 5 | If you do a bit of research, you'll find a lot of recommendations for [Borg](https://www.borgbackup.org/) and [Restic](https://restic.net/). Ultimately they're pretty similar — they're both open source, deduplicating backup tools that support compression and encryption at rest. I decided to go with Borg. 6 | 7 | ## borgmatic 8 | 9 | On its own, Borg is pretty flexible, but relatively complex to configure. borgmatic is a Python-based wrapper script around Borg that's supposed to be easier to use. 10 | 11 | The stuff I wanted to back up is in my home directory, so I followed the instructions for a non-root install: 12 | 13 | ``` 14 | sudo apt install pipx 15 | pipx ensurepath 16 | pipx install borgmatic 17 | ``` 18 | 19 | (I could have used the Debian-packaged version, but I'm running Debian stable, which has a [relatively old version of borgmatic](https://packages.debian.org/bookworm/borgmatic).) 20 | 21 | I set up the following config in ~/.config/borgmatic/config.yaml (sensitive stuff redacted of course): 22 | 23 | ```yaml 24 | source_directories: 25 | - /home/pdubroy/██████ 26 | - /home/pdubroy/█████ 27 | - /home/pdubroy/███████ 28 | 29 | encryption_passphrase: "██████████████████" 30 | repositories: 31 | - path: ssh://u██████-sub2@u██████.your-storagebox.de:23/./pats-server.borg 32 | label: storagebox 33 | encryption: repokey-blake2 34 | # append_only: true 35 | 36 | # List of checks to run to validate your backups. 37 | checks: 38 | - name: repository 39 | - name: archives 40 | frequency: 2 weeks 41 | 42 | keep_daily: 7 43 | keep_weekly: 4 44 | keep_monthly: 6 45 | ``` 46 | 47 | Some notes: 48 | 49 | - I'm using a sub-account to connect to the Storage Box with key-based authentication. It's possible to restrict the key to using borg only in append-only mode, as described in [How I Use Borg for Ransomware-Resilient Backups](https://artemis.sh/2022/06/22/how-i-use-borg.html) — but I haven't done that yet. 50 | - I considered other ways of storing the passphrase, but anyone who can read the file also has access to all my data, so I went for the simple approach. 51 | 52 | I run the actual backups via a cron job on my user account. I ran `crontab -e` and created the following entry: 53 | 54 | ```cron 55 | 0 3 * * * /home/pdubroy/.local/bin/borgmatic --verbosity 1 --log-file ~/logs/borgmatic/daily.log 56 | ``` 57 | -------------------------------------------------------------------------------- /sysadmin/2025-04-27-Logrotate.md: -------------------------------------------------------------------------------- 1 | # logrotate 2 | 3 | I recently set up a Debian on an old Mac Mini, and it's been about 20 years since I actively ran a Linux server, so I'm relearning some things. :-D 4 | 5 | This week, it's `logrotate`: 6 | 7 | > **logrotate** is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large. 8 | 9 | In my last TIL, I wrote about [my daily backups with borgmatic](./2025-04-20-Backups-with-borg-and-borgmatic.md). That runs as a user (non-root) cron job, and logs to ~/logs/borgmatic/daily.log. I wanted those logs to be handled just like logs for system services; turns out that's pretty easy with `logrotate`. 10 | 11 | I added the following config in ~/.logrotate.d/borgmatic: 12 | 13 | ``` 14 | /home/pdubroy/logs/borgmatic/daily.log { 15 | rotate 14 16 | weekly 17 | missingok 18 | notifempty 19 | create 640 pdubroy pdubroy 20 | } 21 | ``` 22 | 23 | Notes: 24 | - `rotate 14` means "keep 14 rotated log files before deleting the oldest ones" 25 | - `weekly` -> Rotate the log file once per week 26 | - `missingok` -> Don't throw an error if the log file is missing 27 | - `notifempty` -> Don't rotate the log file if it's empty 28 | - `create 640 pdubroy pdubroy` -> Permissions and owner/group for new files 29 | 30 | Then I run logrotate daily with another cron job: 31 | 32 | ``` 33 | 30 2 * * * /sbin/logrotate --state $HOME/.logrotate.state ~/.logrotate.d/borgmatic 34 | ``` 35 | 36 | There may be better setups but this is working for me so far! 37 | -------------------------------------------------------------------------------- /wasm/2024-02-22-Run-time-code-generation.md: -------------------------------------------------------------------------------- 1 | # Run-time code generation in WebAssembly 2 | 3 | I first read about this sometime last year, but I decided to write it up since I've been asked about it twice in the past few weeks. 4 | 5 | As it stands today, WebAssembly is a great compilation target for languages like C++ and Rust. For people who are interested in dynamic languages, a common question is: does WebAssembly support just-in-time (JIT) code generation? 6 | 7 | In his blog post [Just-in-time code generation within WebAssembly](https://wingolog.org/archives/2022/08/18/just-in-time-code-generation-within-webassembly), Andy Wingo explains why it's not so easy: 8 | 9 | > In a von Neumman machine, like the ones that you are probably reading this on, code and data share an address space. There's only one kind of pointer, and it can point to anything: the bytes that implement the sin function, the number 42, the characters in "biscuits", or anything at all. WebAssembly is different in that its code is not addressable at run-time. Functions in a WebAssembly module are numbered sequentially from 0, and the WebAssembly call instruction takes the callee as an immediate parameter. 10 | 11 | This means that, unlike on a conventional architecture, you can't just patch the call site directly. But, he goes on to explain how late linking of WebAssembly module _is_ in fact possible: 12 | 13 | > The key idea here is that to add code, the main program should generate a new WebAssembly module containing that code. Then we run a linking phase to actually bring that new code to life and make it available. 14 | 15 | > The generated module will also import the indirect function table from the main module. [...] When the main module makes the generated module, it also embeds a special `patch` function in the generated module. This function will add the new functions to the main module's indirect function table, and perform any relocations onto the main module's memory. All references from the main module to generated functions are installed via the `patch` function. 16 | 17 | Here's a diagram to clarify: 18 | 19 | A diagram illustrating the scenario described above 20 | 21 | [WAForth](https://github.com/remko/waforth) uses a technique similar to this. According to the [design doc](https://github.com/remko/waforth/blob/master/doc/Design.md): 22 | 23 | > While in compile mode for a word, the compiler generates WebAssembly instructions in binary format (since there is no assembler infrastructure in the browser). Since WebAssembly doesn't support JIT compilation yet, a finished word is bundled into a separate binary WebAssembly module, and sent to the loader, which dynamically loads it and registers it with a shared function table at the next offset, which in turn is recorded in the word dictionary. 24 | > 25 | > Because words reside in different modules, all calls to and from the words need to happen as indirect call_indirect calls through the shared function table. This of course introduces some overhead, although it appears limited. 26 | -------------------------------------------------------------------------------- /webdev/2024-02-16-Authentication-in-Playwright-scripts.md: -------------------------------------------------------------------------------- 1 | # Authentication in Playwright scripts 2 | 3 | Every time I use Playwright, I'm impressed by the quality and usefulness of the tooling they built. So much useful stuff that seems to just work. And it's well-documented, too. 4 | 5 | ## Problem 6 | 7 | I have a Playwright script that requires the user to be logged in. Ideally, I'd like to be able to do the following: 8 | 9 | - Detect if the user is logged in. 10 | - If not, open up a browser window to the login page. 11 | - After login, save the cookies to avoid having to log in every time the script is run. 12 | 13 | ## Solution 14 | 15 | Turns out this is quite easy to do in Playwright. There's an [example in the Playwright documentation](https://playwright.dev/docs/auth), which got me most of the way there, but it's a little bit different than what I was looking for. 16 | 17 | Here's the script I ended up with: 18 | 19 | ```ts 20 | const { test, chromium } = require("@playwright/test"); 21 | import fs from "node:fs"; 22 | 23 | const authFile = "playwright/.auth/user.json"; 24 | 25 | // Use existing storage state if it exists. 26 | test.use({ 27 | storageState: fs.existsSync(authFile) ? authFile : undefined, 28 | }); 29 | 30 | test("download Buttondown invoices", async ({ page }) => { 31 | await page.goto("https://buttondown.email/emails"); 32 | 33 | const loginRequired = (await page.getByLabel("Username").count()) > 0; 34 | if (loginRequired) { 35 | // Close the headless context 36 | await page.context().close(); 37 | 38 | // Start a new headful context 39 | const browser = await chromium.launch({ headless: false }); 40 | const context = await browser.newContext(); 41 | page = await context.newPage(); 42 | await page.goto("https://buttondown.email/emails"); 43 | 44 | await page.pause(); // Await manual login 45 | 46 | // Write the storage state to a file. 47 | await page.context().storageState({ path: authFile }); 48 | } 49 | // Proceed in logged in state 50 | // … 51 | }); 52 | ``` 53 | 54 | Note that the call to `page.pause()` will open the [Playwright inspector](https://playwright.dev/docs/debug#playwright-inspector), and you have to click the _Resume_ button in the inspector to continue the script after logging in: 55 | 56 | Screenshot of the Playwright inspector window, showing the 'Resume' button at the top 57 | 58 | To avoid having to manually continue the script, you could instead use 59 | [`waitFor`](https://playwright.dev/docs/api/class-locator#locator-wait-for) or something similar. 60 | -------------------------------------------------------------------------------- /webdev/2025-02-14-Webp-is-awesome.md: -------------------------------------------------------------------------------- 1 | # WebP is awesome 2 | 3 | On the web site for [our book](https://wasmgroundup.com), we had a couple large screenshots, stored as PNGs. I was looking for ways to reduce the file size, so I tried running them through [ImageOptim][]. That helped somewhat, but didn't make a huge difference. 4 | 5 | I asked Claude what to do, and it suggested WebP. My first thoughts were: 6 | 7 | - Is it well supported in browsers other than Chrome? 8 | - Will it make much of a difference? 9 | 10 | Some quick research gave me the answers: yes and yes! 11 | 12 | [caniuse](https://caniuse.com/webp) says: 13 | 14 | > Since September 2020, this feature works across the latest devices and major browser versions. 15 | 16 | As for the file size, I was pretty impressed. Here are a few files I tried it on: 17 | 18 | ``` 19 | 376K macOS-specific-bits-wide.png 20 | 75K macOS-specific-bits-wide.webp 21 | 487K macOS-specific-bits.png 22 | 83K macOS-specific-bits.webp 23 | 92K native-binary-vs-wasm-binary.png 24 | 44K native-binary-vs-wasm-binary.webp 25 | 230K nodejs-downloads.png 26 | 30K nodejs-downloads.webp 27 | 83K og-image-large.png 28 | 42K og-image-large.webp 29 | ``` 30 | 31 | So: 20%, 17%, 48%, 13%, 51%. Not bad! 32 | 33 | To convert the PNGs to WebP, I used [cwebp](https://developers.google.com/speed/webp/docs/cwebp). On macOS, you can install it with `brew install cwebp`. 34 | 35 | Here's the command line I used to convert all images in a given directory: 36 | 37 | ```sh 38 | find . -name "*.png" -exec sh -c 'cwebp "$1" -o "${1%.png}.webp"' _ {} \; 39 | ``` 40 | 41 | [ImageOptim]: https://imageoptim.com/mac 42 | -------------------------------------------------------------------------------- /webdev/2025-02-26-Styling-for-print.md: -------------------------------------------------------------------------------- 1 | # Styling for print 2 | 3 | I'm working on official PDFs for [our WebAssembly book](https://wasmgroundup.com), and it's the first time I've ever had to care about print styles in CSS. 4 | 5 | ## Allow or prevent page breaks with `break-inside` 6 | 7 | One issue I ran into was page breaks in awkward places. In fixing that, I learned about these three CSS properties: 8 | 9 | - [`break-inside`](https://developer.mozilla.org/en-US/docs/Web/CSS/break-inside) 10 | - [`break-before`](https://developer.mozilla.org/en-US/docs/Web/CSS/break-before) 11 | - [`break-after`](https://developer.mozilla.org/en-US/docs/Web/CSS/break-after) 12 | 13 | ## The `beforeprint` / `afterprint` events 14 | 15 | Another thing learned is that certain style properties (e.g. CSS transforms, absolute positioning?) don't play well with the page break logic. My solution was to make some changes to the DOM before printing to deal with those on a case-by-case basis. 16 | 17 | You can do that with the [`beforeprint` event](https://developer.mozilla.org/en-US/docs/Web/API/Window/beforeprint_event). 18 | -------------------------------------------------------------------------------- /zig/2025-01-22-Getting-started-with-Zig.md: -------------------------------------------------------------------------------- 1 | # Getting started with Zig 2 | 3 | ## Installation 4 | 5 | I used [mise](https://mise.jdx.dev/): 6 | 7 | ```sh 8 | mise install zig && mise use -g zig@latest 9 | ``` 10 | 11 | ## Setting up a new project 12 | 13 | I ran the following to generate the boilerplate for a new Zig project: 14 | 15 | ```sh 16 | zig init 17 | ``` 18 | 19 | ## Misc notes 20 | 21 | - The build & test configuration is in build.zig. That's what determines what subcommands you can use with `zig build`. 22 | - Seems like it's standard to support the following: 23 | * `zig build` 24 | * `zig build run` to run the executable. 25 | * `zig run test` to run the tests. 26 | - **NOTE:** by default, Zig's test runner only produces output when the tests fail! This tripped me up for a few minutes. 27 | - The build.zig.zon file is similar to package.json, and should be under source control. 28 | - There's no built-in command to rebuild / re-run the tests whenever a file changes, but `mise watch test` works well with the following in .mise.toml: 29 | ``` 30 | [tasks] 31 | build = "zig build" 32 | test = "zig build test" 33 | ``` 34 | -------------------------------------------------------------------------------- /zig/2025-06-08-Zig-shadowing-and-builtins.md: -------------------------------------------------------------------------------- 1 | # Zig shadowing and builtins 2 | 3 | A couple things I learned when I was playing with Zig earlier this year, but forgot to write up… 4 | 5 | ## No shadowing 6 | 7 | Zig is the only language I know of that disallows "variable shadowing". As the docs say: 8 | 9 | > Identifiers are never allowed to "hide" other identifiers by using the same name. 10 | 11 | In other words, the following code is not allowed: 12 | 13 | ```zig 14 | const pi = 3.14; 15 | 16 | test "inside test block" { 17 | // Let's even go inside another block 18 | { 19 | var pi: i32 = 1234; 20 | } 21 | } 22 | ``` 23 | 24 | I find this to be a really interesting design choice; I used to wonder why no languages (that I knew of) did this, especially educational languages. 25 | 26 | ## Builtins begin with `@` 27 | 28 | As a consequence of the "no shadowing" rule, every builtin function would prevent user code from using that name. But in Zig, builtins are prefixed with `@`: `@addWithOverflow`, `@atomicLoad`, etc. 29 | 30 | We have a similar problem in [Ohm](https://ohmjs.org). Ohm does allow shadowing, but it requires different syntax when you are overriding an existing rule (`:=`) vs. declaring a new rule (`=`). But, that means that it's a breaking change every time we introduce a new built-in rule, because user grammars that used to work will no longer be accepted. Not a huge deal, but I wonder if it would be better to do something like Zig. 31 | --------------------------------------------------------------------------------