├── dotopo.lisp └── README.adoc /dotopo.lisp: -------------------------------------------------------------------------------- 1 | ;2009-03-14 2 | ;Dorai Sitaram, ds26gte at yahoo dot com 3 | ;last change 2009-03-17 4 | 5 | (defmacro do-in-topological-order (fn &rest deps) 6 | (let ((all-items (remove-duplicates (apply #'append deps))) 7 | (f (gensym)) 8 | (callf (gensym))) 9 | `(let ((,f ,fn) 10 | ,@(mapcar (lambda (x) 11 | `(,x (vector ',x nil nil))) 12 | all-items)) 13 | ,@(mapcar 14 | (lambda (dep) 15 | (destructuring-bind (x &rest befores) dep 16 | `(setf (svref ,x 1) (list ,@befores)))) 17 | deps) 18 | (labels ((,callf (x) 19 | (unless (svref x 2) 20 | (setf (svref x 2) t) 21 | (mapc #',callf (svref x 1)) 22 | (funcall ,f (svref x 0))))) 23 | ,@(mapcar 24 | (lambda (x) `(,callf ,x)) 25 | all-items))))) 26 | -------------------------------------------------------------------------------- /README.adoc: -------------------------------------------------------------------------------- 1 | = Hassle-free delivery of a multi-file Lisp program 2 | 3 | Assume the end-user has an ANSI-compliant Common Lisp, and downloads your 4 | program, which you distribute as a tarball named, say, 5 | `coffee.tar.bz2`, that unpacks to a directory called `coffee`. What is 6 | the easiest way for them to use your program? 7 | 8 | == ☕ The scenario 9 | 10 | If your distribution contains the single Lisp file `coffee.lisp`, 11 | the user can 12 | load it into their CL, thus: 13 | 14 | ---- 15 | (load "coffee") 16 | ---- 17 | 18 | and use whatever functions, macros, and variables you provided as the 19 | external interface. They can even do 20 | 21 | ---- 22 | (compile-file "coffee") 23 | ---- 24 | 25 | and then 26 | 27 | ---- 28 | (load "coffee") 29 | ---- 30 | 31 | CL 32 | is now smart enough to pick up the faster compiled file — which has the same 33 | basename as the source file, but with the file extension changed from 34 | `.lisp` to something like `.fasl` (or `.fas`, `.dx64fsl`, or 35 | `.sse2f`, depending on the user’s CL implementation). 36 | 37 | Typically, though, your distribution contains several Lisp files spread 38 | across 39 | several directories, to ease your development process, i.e., to 40 | keep things neat and maintainable. `coffee.lisp` may load `caf.lisp` and 41 | `decaf.lisp`; 42 | `caf.lisp` may then load `cappuccino.lisp`, `espresso.lisp`, and `mocha.lisp`; 43 | and `decaf.lisp` may load a completely different 44 | `cappuccino.lisp` and `mocha.lisp` 45 | (in some other directory of course). The various 46 | calls to `load` use relative pathnames, viz., 47 | 48 | ---- 49 | (load (merge-pathnames ... *load-pathname*)) 50 | ---- 51 | 52 | So the user can still load 53 | the “main” file `coffee.lisp`, as before: 54 | 55 | ---- 56 | (load "coffee") 57 | ---- 58 | 59 | and expect things to work. Just to consolidate our example situation, 60 | let’s say the directory structure of the files in the distribution is as 61 | follows: 62 | 63 | ---- 64 | coffee 65 | ├── coffee.lisp 66 | ├── type 67 | │ ├── caf.lisp 68 | │ └── decaf.lisp 69 | ├── caf 70 | │ ├── cappuccino.lisp 71 | │ ├── espresso.lisp 72 | │ └── mocha.lisp 73 | └── decaf 74 | ├── cappuccino.lisp 75 | └── mocha.lisp 76 | ---- 77 | 78 | Since all these several coffee-related files must be introducing symbols 79 | galore, you of course want to define a package `coffee`, so that the 80 | `coffee` symbols 81 | don’t clash with any symbols the user has, either of their own, or from 82 | other packages. So you put 83 | 84 | ---- 85 | (defpackage :coffee 86 | (:use :cl) 87 | (:export :WHATEVER ...)) 88 | 89 | (in-package :coffee) 90 | ---- 91 | 92 | at the head of `coffee.lisp`. This should be enough, because when 93 | `coffee.lisp` loads other files, and these other files load still other 94 | files, the `load` function fluidly sets the special variable 95 | `+*package*+` to the prevailing value of `+*package*+`, so 96 | all the files will inherit `coffee` as their `+*package*+`. However, for 97 | the purpose of documentation, or maybe even to play nice with any slimy 98 | text-editor setup you may have, you may still want to put 99 | 100 | ---- 101 | (in-package :coffee) 102 | ---- 103 | 104 | at the head of each constituent file. It doesn’t hurt. 105 | 106 | Let’s say the user likes your program, but finds it somewhat on 107 | the slow side, and wants to speed things 108 | up with 109 | 110 | ---- 111 | (compile-file "coffee") 112 | ---- 113 | 114 | What can you, as program developer, do to facilitate this? 115 | 116 | == A solution 117 | 118 | This is when most program deliverers reach for 119 | http://common-lisp.net/project/asdf[ASDF]. ASDF, however requires 120 | some non-trivial effort on the 121 | part of the user (and of course, much more on the part of the developer too). 122 | The user has to download ASDF, they have to create a 123 | registry directory, and for every program they download, they have to 124 | put the `.asd` file in the registry directory, and they have to follow a 125 | new syntax to load the program. I wondered if it was possible to have 126 | something really simple instead, where the user approach remains the 127 | same as for a program consisting of a single Lisp file. I.e., 128 | the user calls `load` on the main file, and they can optionally 129 | `compile-file` that main file. 130 | 131 | Turns out there is a protocol — and quite an easy one — for securing this. Again, as 132 | with adding `defpackage` and `in-package`, the little bit of 133 | extra code goes just in 134 | the main `coffee.lisp` file. We place one requirement: all the 135 | names of the 136 | Lisp files and subdirectories they are in can be identified with Lisp 137 | symbols without any escaping mechanism. Thus, the file and directory 138 | names are lowercase and are allowed to use the same characters that a 139 | Lisp symbol can, provided they don’t interfere with the OS’s own 140 | convention for unadorned file naming. (E.g., the name can’t be a number 141 | even though the OS doesn’t mind, and it can’t contain a slash even though Lisp 142 | doesn’t mind.) 143 | 144 | Recall that `coffee.lisp` loads `caf.lisp` and 145 | `decaf.lisp`. First, make sure that you don’t specify the extension 146 | `.lisp` in the calls to `load`, as we want the compiled versions 147 | of the files to be picked up whenever possible. 148 | 149 | Now add an expression to `coffee.lisp` that compiles these subfiles whenever 150 | `coffee.lisp` itself is compiled. We could try 151 | 152 | ---- 153 | (eval-when (:compile-toplevel) 154 | (compile-file (merge-pathnames "type/caf" *compile-file-pathname*)) 155 | (compile-file (merge-pathnames "type/decaf" *compile-file-pathname*))) 156 | ---- 157 | 158 | But we don’t just want to 159 | compile 160 | `type/caf.lisp` and `type/decaf.lisp`; 161 | we also want to compile the other files 162 | that these files load, viz., `caf/espresso.lisp`, 163 | `caf/mocha.lisp`, 164 | and 165 | `caf/cappuccino.lisp`; and `decaf/mocha.lisp` and 166 | `decaf/cappuccino.lisp`. And we want them compiled in the right 167 | order, because these files depend on each other in a certain order. 168 | Instead of explicitly writing the various ``compile-file``s in the 169 | right order, we can simply specify the dependencies in a succinct 170 | makefile-like way, 171 | and have a Lisp macro take care of calling `compile-file` 172 | on the files in the correct topological order. For this we use the macro 173 | `do-in-topological-order`, defined in the file 174 | link:dotopo.lisp[]. 175 | 176 | `do-in-topological-order`’s first argument is a function, and its 177 | second argument is an expression of the dependencies among the symbols 178 | representing the various files. It topologically arranges the symbols, and 179 | then calls the function on them, ensuring that a file is 180 | treated before any file depending on it. Here’s a more maintainable way 181 | to add the ``compile-file``s to `coffee.lisp`, 182 | assuming `dotopo.lisp` is in the same directory as 183 | `coffee.lisp`: 184 | 185 | ---- 186 | (eval-when (:compile-toplevel) 187 | (load (merge-pathnames "dotopo" *compile-file-pathname*))) 188 | 189 | (eval-when (:compile-toplevel) 190 | (do-in-topological-order 191 | (lambda (f) 192 | (compile-file (merge-pathnames (string-downcase (symbol-name f)) 193 | *compile-file-pathname*))) 194 | (coffee type/caf type/decaf) 195 | (type/caf caf/cappuccino caf/espresso caf/mocha) 196 | (type/decaf decaf/cappuccino decaf/mocha))) 197 | ---- 198 | 199 | This is almost right, except for two issues, which we will now 200 | address. 201 | 202 | == Preventing repeated compilation of main file 203 | 204 | First, one of the files that 205 | `do-in-topological-order`’s first argument will attempt to 206 | compile is `coffee.lisp`, the file that we are already in the 207 | process of compiling 208 | when this expression is encountered! We could add a conditional 209 | disallowing the loop in 210 | `do-in-topological-order`’s first argument: 211 | 212 | ---- 213 | (eval-when (:compile-toplevel) 214 | (do-in-topological-order 215 | (lambda (f) 216 | (unless (eq f 'coffee) ; <1> 217 | (compile-file (merge-pathnames (string-downcase (symbol-name f)) 218 | *compile-file-pathname*)))) 219 | ...)) 220 | ---- 221 | 222 | But there is an easier way: We could simply leave out (or comment 223 | out) the 224 | dependency line for `coffee.lisp`. Thus: 225 | 226 | ---- 227 | (eval-when (:compile-toplevel) 228 | (do-in-topological-order 229 | (lambda (f) 230 | (compile-file (merge-pathnames (string-downcase (symbol-name f)) 231 | *compile-file-pathname*))) 232 | ;(coffee type/caf type/decaf) ; <2> 233 | (type/caf caf/cappuccino caf/espresso caf/mocha) 234 | (type/decaf decaf/cappuccino decaf/mocha))) 235 | ---- 236 | 237 | This works, but only because the files that `coffee.lisp` depends 238 | on, viz., `type/caf.lisp` and `type/decaf.lisp`, are mentioned 239 | elsewhere in the dependency list. If `coffee.lisp` depended on a 240 | file `type/water.lisp` that nothing else depended on and that 241 | didn’t depend on anything else, then the 242 | compiler wouldn’t recognize that `type/water.lisp` needed to be 243 | compiled at all. To accommodate this, we can add a dependency line for 244 | `type/water.lisp` that listed no dependencies: 245 | 246 | ---- 247 | ;(coffee type/caf type/decaf) 248 | (type/water ) ; <3> 249 | (type/caf caf/cappuccino caf/espresso caf/mocha) 250 | (type/decaf decaf/cappuccino decaf/mocha) 251 | ---- 252 | 253 | We of course need only do this for files that the main file 254 | depends on. A file like `type/espresso.lisp`, which also doesn’t 255 | depend on anything else, doesn’t need a line of its own because 256 | it’s already mentioned in the dependency line for 257 | `type/caf.lisp`. But it doesn’t hurt to add such a line anyway. 258 | 259 | == Making special variables visible to dependent files 260 | 261 | The second issue has to do with a file B using a special variable 262 | introduced in another file A that B depends on. In some 263 | implementations, `compile-file` may 264 | issue a warning that file B has an “undeclared free variable”. 265 | To avoid this annoyance, make the introduction of the special 266 | variable in file A visible to `compile-file`, e.g., 267 | 268 | ---- 269 | (eval-when (:compile-toplevel :load-toplevel :execute) ; <4> 270 | (defvar *bean-type*)) 271 | ---- 272 | 273 | N.B. 274 | http://ccl.clozure.com[Clozure] and http://ecls.sourceforge.net[ECL] issue 275 | this warning, but it seems benign. http://abcl.org[ABCL] and 276 | http://sbcl.org[SBCL] are properly 277 | silent. 278 | 279 | == Summary 280 | 281 | That’s all there is to it. 282 | 283 | To summarize: 284 | 285 | 1. you include 286 | `dotopo.lisp` in your distribution alongside the main file; 287 | 288 | 2. include the above changes to the content of that main file; and 289 | 290 | 3. make sure 291 | that all special variables used outside their files are made visible to 292 | the compiler. 293 | 294 | The user who unpacks your tarball then simply ``load``s your main 295 | file, either as source, or after `compile-file`-ing it. They do 296 | not have to worry about dealing with the other files at all, so 297 | long as the latter’s relative path to the main file is not 298 | altered. 299 | 300 | ❧❧❧ 301 | 302 | —Dorai Sitaram 303 | --------------------------------------------------------------------------------