├── LICENSE ├── README.md ├── list-comprehensions.lisp ├── list-comprehensions.lisp.md ├── list-comprehensions.lisp.pdf └── makedoc.sh /LICENSE: -------------------------------------------------------------------------------- 1 | Creative Commons Legal Code 2 | 3 | Attribution-ShareAlike 3.0 Unported 4 | 5 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE 6 | LEGAL SERVICES. DISTRIBUTION OF THIS LICENSE DOES NOT CREATE AN 7 | ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS 8 | INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES 9 | REGARDING THE INFORMATION PROVIDED, AND DISCLAIMS LIABILITY FOR 10 | DAMAGES RESULTING FROM ITS USE. 11 | 12 | License 13 | 14 | THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE 15 | COMMONS PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY 16 | COPYRIGHT AND/OR OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS 17 | AUTHORIZED UNDER THIS LICENSE OR COPYRIGHT LAW IS PROHIBITED. 18 | 19 | BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE 20 | TO BE BOUND BY THE TERMS OF THIS LICENSE. TO THE EXTENT THIS LICENSE MAY 21 | BE CONSIDERED TO BE A CONTRACT, THE LICENSOR GRANTS YOU THE RIGHTS 22 | CONTAINED HERE IN CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND 23 | CONDITIONS. 24 | 25 | 1. Definitions 26 | 27 | a. "Adaptation" means a work based upon the Work, or upon the Work and 28 | other pre-existing works, such as a translation, adaptation, 29 | derivative work, arrangement of music or other alterations of a 30 | literary or artistic work, or phonogram or performance and includes 31 | cinematographic adaptations or any other form in which the Work may be 32 | recast, transformed, or adapted including in any form recognizably 33 | derived from the original, except that a work that constitutes a 34 | Collection will not be considered an Adaptation for the purpose of 35 | this License. For the avoidance of doubt, where the Work is a musical 36 | work, performance or phonogram, the synchronization of the Work in 37 | timed-relation with a moving image ("synching") will be considered an 38 | Adaptation for the purpose of this License. 39 | b. "Collection" means a collection of literary or artistic works, such as 40 | encyclopedias and anthologies, or performances, phonograms or 41 | broadcasts, or other works or subject matter other than works listed 42 | in Section 1(f) below, which, by reason of the selection and 43 | arrangement of their contents, constitute intellectual creations, in 44 | which the Work is included in its entirety in unmodified form along 45 | with one or more other contributions, each constituting separate and 46 | independent works in themselves, which together are assembled into a 47 | collective whole. A work that constitutes a Collection will not be 48 | considered an Adaptation (as defined below) for the purposes of this 49 | License. 50 | c. "Creative Commons Compatible License" means a license that is listed 51 | at https://creativecommons.org/compatiblelicenses that has been 52 | approved by Creative Commons as being essentially equivalent to this 53 | License, including, at a minimum, because that license: (i) contains 54 | terms that have the same purpose, meaning and effect as the License 55 | Elements of this License; and, (ii) explicitly permits the relicensing 56 | of adaptations of works made available under that license under this 57 | License or a Creative Commons jurisdiction license with the same 58 | License Elements as this License. 59 | d. "Distribute" means to make available to the public the original and 60 | copies of the Work or Adaptation, as appropriate, through sale or 61 | other transfer of ownership. 62 | e. "License Elements" means the following high-level license attributes 63 | as selected by Licensor and indicated in the title of this License: 64 | Attribution, ShareAlike. 65 | f. "Licensor" means the individual, individuals, entity or entities that 66 | offer(s) the Work under the terms of this License. 67 | g. "Original Author" means, in the case of a literary or artistic work, 68 | the individual, individuals, entity or entities who created the Work 69 | or if no individual or entity can be identified, the publisher; and in 70 | addition (i) in the case of a performance the actors, singers, 71 | musicians, dancers, and other persons who act, sing, deliver, declaim, 72 | play in, interpret or otherwise perform literary or artistic works or 73 | expressions of folklore; (ii) in the case of a phonogram the producer 74 | being the person or legal entity who first fixes the sounds of a 75 | performance or other sounds; and, (iii) in the case of broadcasts, the 76 | organization that transmits the broadcast. 77 | h. "Work" means the literary and/or artistic work offered under the terms 78 | of this License including without limitation any production in the 79 | literary, scientific and artistic domain, whatever may be the mode or 80 | form of its expression including digital form, such as a book, 81 | pamphlet and other writing; a lecture, address, sermon or other work 82 | of the same nature; a dramatic or dramatico-musical work; a 83 | choreographic work or entertainment in dumb show; a musical 84 | composition with or without words; a cinematographic work to which are 85 | assimilated works expressed by a process analogous to cinematography; 86 | a work of drawing, painting, architecture, sculpture, engraving or 87 | lithography; a photographic work to which are assimilated works 88 | expressed by a process analogous to photography; a work of applied 89 | art; an illustration, map, plan, sketch or three-dimensional work 90 | relative to geography, topography, architecture or science; a 91 | performance; a broadcast; a phonogram; a compilation of data to the 92 | extent it is protected as a copyrightable work; or a work performed by 93 | a variety or circus performer to the extent it is not otherwise 94 | considered a literary or artistic work. 95 | i. "You" means an individual or entity exercising rights under this 96 | License who has not previously violated the terms of this License with 97 | respect to the Work, or who has received express permission from the 98 | Licensor to exercise rights under this License despite a previous 99 | violation. 100 | j. "Publicly Perform" means to perform public recitations of the Work and 101 | to communicate to the public those public recitations, by any means or 102 | process, including by wire or wireless means or public digital 103 | performances; to make available to the public Works in such a way that 104 | members of the public may access these Works from a place and at a 105 | place individually chosen by them; to perform the Work to the public 106 | by any means or process and the communication to the public of the 107 | performances of the Work, including by public digital performance; to 108 | broadcast and rebroadcast the Work by any means including signs, 109 | sounds or images. 110 | k. "Reproduce" means to make copies of the Work by any means including 111 | without limitation by sound or visual recordings and the right of 112 | fixation and reproducing fixations of the Work, including storage of a 113 | protected performance or phonogram in digital form or other electronic 114 | medium. 115 | 116 | 2. Fair Dealing Rights. Nothing in this License is intended to reduce, 117 | limit, or restrict any uses free from copyright or rights arising from 118 | limitations or exceptions that are provided for in connection with the 119 | copyright protection under copyright law or other applicable laws. 120 | 121 | 3. License Grant. Subject to the terms and conditions of this License, 122 | Licensor hereby grants You a worldwide, royalty-free, non-exclusive, 123 | perpetual (for the duration of the applicable copyright) license to 124 | exercise the rights in the Work as stated below: 125 | 126 | a. to Reproduce the Work, to incorporate the Work into one or more 127 | Collections, and to Reproduce the Work as incorporated in the 128 | Collections; 129 | b. to create and Reproduce Adaptations provided that any such Adaptation, 130 | including any translation in any medium, takes reasonable steps to 131 | clearly label, demarcate or otherwise identify that changes were made 132 | to the original Work. For example, a translation could be marked "The 133 | original work was translated from English to Spanish," or a 134 | modification could indicate "The original work has been modified."; 135 | c. to Distribute and Publicly Perform the Work including as incorporated 136 | in Collections; and, 137 | d. to Distribute and Publicly Perform Adaptations. 138 | e. For the avoidance of doubt: 139 | 140 | i. Non-waivable Compulsory License Schemes. In those jurisdictions in 141 | which the right to collect royalties through any statutory or 142 | compulsory licensing scheme cannot be waived, the Licensor 143 | reserves the exclusive right to collect such royalties for any 144 | exercise by You of the rights granted under this License; 145 | ii. Waivable Compulsory License Schemes. In those jurisdictions in 146 | which the right to collect royalties through any statutory or 147 | compulsory licensing scheme can be waived, the Licensor waives the 148 | exclusive right to collect such royalties for any exercise by You 149 | of the rights granted under this License; and, 150 | iii. Voluntary License Schemes. The Licensor waives the right to 151 | collect royalties, whether individually or, in the event that the 152 | Licensor is a member of a collecting society that administers 153 | voluntary licensing schemes, via that society, from any exercise 154 | by You of the rights granted under this License. 155 | 156 | The above rights may be exercised in all media and formats whether now 157 | known or hereafter devised. The above rights include the right to make 158 | such modifications as are technically necessary to exercise the rights in 159 | other media and formats. Subject to Section 8(f), all rights not expressly 160 | granted by Licensor are hereby reserved. 161 | 162 | 4. Restrictions. The license granted in Section 3 above is expressly made 163 | subject to and limited by the following restrictions: 164 | 165 | a. You may Distribute or Publicly Perform the Work only under the terms 166 | of this License. You must include a copy of, or the Uniform Resource 167 | Identifier (URI) for, this License with every copy of the Work You 168 | Distribute or Publicly Perform. You may not offer or impose any terms 169 | on the Work that restrict the terms of this License or the ability of 170 | the recipient of the Work to exercise the rights granted to that 171 | recipient under the terms of the License. You may not sublicense the 172 | Work. You must keep intact all notices that refer to this License and 173 | to the disclaimer of warranties with every copy of the Work You 174 | Distribute or Publicly Perform. When You Distribute or Publicly 175 | Perform the Work, You may not impose any effective technological 176 | measures on the Work that restrict the ability of a recipient of the 177 | Work from You to exercise the rights granted to that recipient under 178 | the terms of the License. This Section 4(a) applies to the Work as 179 | incorporated in a Collection, but this does not require the Collection 180 | apart from the Work itself to be made subject to the terms of this 181 | License. If You create a Collection, upon notice from any Licensor You 182 | must, to the extent practicable, remove from the Collection any credit 183 | as required by Section 4(c), as requested. If You create an 184 | Adaptation, upon notice from any Licensor You must, to the extent 185 | practicable, remove from the Adaptation any credit as required by 186 | Section 4(c), as requested. 187 | b. You may Distribute or Publicly Perform an Adaptation only under the 188 | terms of: (i) this License; (ii) a later version of this License with 189 | the same License Elements as this License; (iii) a Creative Commons 190 | jurisdiction license (either this or a later license version) that 191 | contains the same License Elements as this License (e.g., 192 | Attribution-ShareAlike 3.0 US)); (iv) a Creative Commons Compatible 193 | License. If you license the Adaptation under one of the licenses 194 | mentioned in (iv), you must comply with the terms of that license. If 195 | you license the Adaptation under the terms of any of the licenses 196 | mentioned in (i), (ii) or (iii) (the "Applicable License"), you must 197 | comply with the terms of the Applicable License generally and the 198 | following provisions: (I) You must include a copy of, or the URI for, 199 | the Applicable License with every copy of each Adaptation You 200 | Distribute or Publicly Perform; (II) You may not offer or impose any 201 | terms on the Adaptation that restrict the terms of the Applicable 202 | License or the ability of the recipient of the Adaptation to exercise 203 | the rights granted to that recipient under the terms of the Applicable 204 | License; (III) You must keep intact all notices that refer to the 205 | Applicable License and to the disclaimer of warranties with every copy 206 | of the Work as included in the Adaptation You Distribute or Publicly 207 | Perform; (IV) when You Distribute or Publicly Perform the Adaptation, 208 | You may not impose any effective technological measures on the 209 | Adaptation that restrict the ability of a recipient of the Adaptation 210 | from You to exercise the rights granted to that recipient under the 211 | terms of the Applicable License. This Section 4(b) applies to the 212 | Adaptation as incorporated in a Collection, but this does not require 213 | the Collection apart from the Adaptation itself to be made subject to 214 | the terms of the Applicable License. 215 | c. If You Distribute, or Publicly Perform the Work or any Adaptations or 216 | Collections, You must, unless a request has been made pursuant to 217 | Section 4(a), keep intact all copyright notices for the Work and 218 | provide, reasonable to the medium or means You are utilizing: (i) the 219 | name of the Original Author (or pseudonym, if applicable) if supplied, 220 | and/or if the Original Author and/or Licensor designate another party 221 | or parties (e.g., a sponsor institute, publishing entity, journal) for 222 | attribution ("Attribution Parties") in Licensor's copyright notice, 223 | terms of service or by other reasonable means, the name of such party 224 | or parties; (ii) the title of the Work if supplied; (iii) to the 225 | extent reasonably practicable, the URI, if any, that Licensor 226 | specifies to be associated with the Work, unless such URI does not 227 | refer to the copyright notice or licensing information for the Work; 228 | and (iv) , consistent with Ssection 3(b), in the case of an 229 | Adaptation, a credit identifying the use of the Work in the Adaptation 230 | (e.g., "French translation of the Work by Original Author," or 231 | "Screenplay based on original Work by Original Author"). The credit 232 | required by this Section 4(c) may be implemented in any reasonable 233 | manner; provided, however, that in the case of a Adaptation or 234 | Collection, at a minimum such credit will appear, if a credit for all 235 | contributing authors of the Adaptation or Collection appears, then as 236 | part of these credits and in a manner at least as prominent as the 237 | credits for the other contributing authors. For the avoidance of 238 | doubt, You may only use the credit required by this Section for the 239 | purpose of attribution in the manner set out above and, by exercising 240 | Your rights under this License, You may not implicitly or explicitly 241 | assert or imply any connection with, sponsorship or endorsement by the 242 | Original Author, Licensor and/or Attribution Parties, as appropriate, 243 | of You or Your use of the Work, without the separate, express prior 244 | written permission of the Original Author, Licensor and/or Attribution 245 | Parties. 246 | d. Except as otherwise agreed in writing by the Licensor or as may be 247 | otherwise permitted by applicable law, if You Reproduce, Distribute or 248 | Publicly Perform the Work either by itself or as part of any 249 | Adaptations or Collections, You must not distort, mutilate, modify or 250 | take other derogatory action in relation to the Work which would be 251 | prejudicial to the Original Author's honor or reputation. Licensor 252 | agrees that in those jurisdictions (e.g. Japan), in which any exercise 253 | of the right granted in Section 3(b) of this License (the right to 254 | make Adaptations) would be deemed to be a distortion, mutilation, 255 | modification or other derogatory action prejudicial to the Original 256 | Author's honor and reputation, the Licensor will waive or not assert, 257 | as appropriate, this Section, to the fullest extent permitted by the 258 | applicable national law, to enable You to reasonably exercise Your 259 | right under Section 3(b) of this License (right to make Adaptations) 260 | but not otherwise. 261 | 262 | 5. Representations, Warranties and Disclaimer 263 | 264 | UNLESS OTHERWISE MUTUALLY AGREED TO BY THE PARTIES IN WRITING, LICENSOR 265 | OFFERS THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY 266 | KIND CONCERNING THE WORK, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, 267 | INCLUDING, WITHOUT LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, 268 | FITNESS FOR A PARTICULAR PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF 269 | LATENT OR OTHER DEFECTS, ACCURACY, OR THE PRESENCE OF ABSENCE OF ERRORS, 270 | WHETHER OR NOT DISCOVERABLE. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION 271 | OF IMPLIED WARRANTIES, SO SUCH EXCLUSION MAY NOT APPLY TO YOU. 272 | 273 | 6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE 274 | LAW, IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR 275 | ANY SPECIAL, INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES 276 | ARISING OUT OF THIS LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS 277 | BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 278 | 279 | 7. Termination 280 | 281 | a. This License and the rights granted hereunder will terminate 282 | automatically upon any breach by You of the terms of this License. 283 | Individuals or entities who have received Adaptations or Collections 284 | from You under this License, however, will not have their licenses 285 | terminated provided such individuals or entities remain in full 286 | compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will 287 | survive any termination of this License. 288 | b. Subject to the above terms and conditions, the license granted here is 289 | perpetual (for the duration of the applicable copyright in the Work). 290 | Notwithstanding the above, Licensor reserves the right to release the 291 | Work under different license terms or to stop distributing the Work at 292 | any time; provided, however that any such election will not serve to 293 | withdraw this License (or any other license that has been, or is 294 | required to be, granted under the terms of this License), and this 295 | License will continue in full force and effect unless terminated as 296 | stated above. 297 | 298 | 8. Miscellaneous 299 | 300 | a. Each time You Distribute or Publicly Perform the Work or a Collection, 301 | the Licensor offers to the recipient a license to the Work on the same 302 | terms and conditions as the license granted to You under this License. 303 | b. Each time You Distribute or Publicly Perform an Adaptation, Licensor 304 | offers to the recipient a license to the original Work on the same 305 | terms and conditions as the license granted to You under this License. 306 | c. If any provision of this License is invalid or unenforceable under 307 | applicable law, it shall not affect the validity or enforceability of 308 | the remainder of the terms of this License, and without further action 309 | by the parties to this agreement, such provision shall be reformed to 310 | the minimum extent necessary to make such provision valid and 311 | enforceable. 312 | d. No term or provision of this License shall be deemed waived and no 313 | breach consented to unless such waiver or consent shall be in writing 314 | and signed by the party to be charged with such waiver or consent. 315 | e. This License constitutes the entire agreement between the parties with 316 | respect to the Work licensed here. There are no understandings, 317 | agreements or representations with respect to the Work not specified 318 | here. Licensor shall not be bound by any additional provisions that 319 | may appear in any communication from You. This License may not be 320 | modified without the mutual written agreement of the Licensor and You. 321 | f. The rights granted under, and the subject matter referenced, in this 322 | License were drafted utilizing the terminology of the Berne Convention 323 | for the Protection of Literary and Artistic Works (as amended on 324 | September 28, 1979), the Rome Convention of 1961, the WIPO Copyright 325 | Treaty of 1996, the WIPO Performances and Phonograms Treaty of 1996 326 | and the Universal Copyright Convention (as revised on July 24, 1971). 327 | These rights and subject matter take effect in the relevant 328 | jurisdiction in which the License terms are sought to be enforced 329 | according to the corresponding provisions of the implementation of 330 | those treaty provisions in the applicable national law. If the 331 | standard suite of rights granted under applicable copyright law 332 | includes additional rights not granted under this License, such 333 | additional rights are deemed to be included in the License; this 334 | License is not intended to restrict the license of any rights under 335 | applicable law. 336 | 337 | 338 | Creative Commons Notice 339 | 340 | Creative Commons is not a party to this License, and makes no warranty 341 | whatsoever in connection with the Work. Creative Commons will not be 342 | liable to You or any party on any legal theory for any damages 343 | whatsoever, including without limitation any general, special, 344 | incidental or consequential damages arising in connection to this 345 | license. Notwithstanding the foregoing two (2) sentences, if Creative 346 | Commons has expressly identified itself as the Licensor hereunder, it 347 | shall have all rights and obligations of Licensor. 348 | 349 | Except for the limited purpose of indicating to the public that the 350 | Work is licensed under the CCPL, Creative Commons does not authorize 351 | the use by either party of the trademark "Creative Commons" or any 352 | related trademark or logo of Creative Commons without the prior 353 | written consent of Creative Commons. Any permitted use will be in 354 | compliance with Creative Commons' then-current trademark usage 355 | guidelines, as may be published on its website or otherwise made 356 | available upon request from time to time. For the avoidance of doubt, 357 | this trademark restriction does not form part of the License. 358 | 359 | Creative Commons may be contacted at https://creativecommons.org/. 360 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # List comprehensions in Lisp (tutorial) 3 | 4 | ```lisp 5 | (list-of (cons i j) 6 | for i in '(1 2 3 4 5 6 7 8) 7 | for j in '(A B) 8 | when (evenp i)) 9 | ``` 10 | 11 | => ((2 . A) (2 . B) (4 . A) (4 . B) (6 . A) (6 . B) (8 . A) (8 . B)) 12 | 13 | 14 | ---- 15 | 16 | (C) 2015 Frederic Peschanski - CC BY SA 3.0 17 | -------------------------------------------------------------------------------- /list-comprehensions.lisp: -------------------------------------------------------------------------------- 1 | 2 | #| 3 | 4 | # List comprehensions in Common Lisp 5 | 6 | > This document is (C) Frédéric Peschanski CC-BY-SA 3.0 7 | 8 | My favorite programming languages are undoubtedly *Lisp*s, originally Scheme but todays I enjoy a lot coding in Common Lisp and Clojure. I am however quite a polyglot programmer: I taught (and thus used, a lot) Java for several years. And at work I program a lot in Ocaml and Python (but that's *not* for fun). 9 | 10 | There's no programming language that I really hate (except VB) or love (except Lisp). Let's take for example Python: not efficient, full of what I think are ill-designed constructs (e.g. `lambda`, also the lexical bindings) but clearly usable. One feature I use a lot in Python is the *comprehensions*. There are comprehension expressions for building lists, sets, dictionaries and (perhaps most interestingly) generators. Today I will only discuss the case of *list comprehensions* because ... I think that's an interesting topic to begin with. 11 | 12 | A simple list comprehension expression is written as follows in Python: 13 | 14 | ``` 15 | [ for in ] 16 | ``` 17 | 18 | This reads: "build the list of `` in which the occurrences of `` are replaced in turn by 19 | the successive elements of the source ``." 20 | 21 | Note that the source of the comprehension can be something else than a list (it can be an *iterable* or a generator) but we will mostly consider the case for lists (although we will make some remarks about this aspect at the end). 22 | 23 | **Remark**: the functional programming language Haskell has a similar syntactic construct, and so does Clojure (cf. the `for` macro). 24 | 25 | Let's see some examples of Python comprehensions. 26 | 27 | ```python 28 | >>> [i*i for i in [1, 2, 3, 4, 5]] 29 | [1, 4, 9, 16, 25] 30 | ``` 31 | 32 | Comprehensions can also nest, performing a kind of *cartesian product*, e.g.: 33 | 34 | ```python 35 | >>> [(i, j) for i in [1, 2, 3, 4] 36 | for j in ['A', 'B']] 37 | [(1, 'A'), (1, 'B'), (2, 'A'), (2, 'B'), (3, 'A'), (3, 'B'), (4, 'A'), (4, 'B')] 38 | ``` 39 | 40 | It is also possible to filter the elements: 41 | 42 | ```python 43 | >>> [i for i in [1, 2, 3, 4, 5, 6, 7, 8] if i % 2 == 0] 44 | [2, 4, 6, 8] 45 | ``` 46 | You know that already, but Common Lisp does not provide *list comprehension expressions*. However, the programmable programming language is never short of macro-based solutions for "missing" constructs! 47 | 48 | So, our objective will be to find a few ways to provide list comprehensions to Common Lisp. 49 | 50 | 51 | |# 52 | 53 | #| 54 | 55 | ## The List monad 56 | 57 | No no no! This is not *yet another monad tutorial* (tm)! But please don't hate monads or other interesting design patterns of functional programming languages. There's some beauty in the concept, and I will try to convey this beauty by considering one of the simplest case: the *list monad*. 58 | 59 | The fundamental operator that we are concerned with is a function that does not look *that* useful: `append-map`. 60 | 61 | |# 62 | 63 | (defun append-map (f ll) 64 | (if (endp ll) 65 | '() 66 | (append (funcall f (car ll)) (append-map f (cdr ll))))) 67 | 68 | #| 69 | 70 | Of course, `append` is interesting, and `map`/`mapcar` too, but mixing the two does not look extremely useful (as pointed out by `lispm@reddit/lisp` a similar function is called `mappend` in [alexandria](https://common-lisp.net/project/alexandria/)). Anyway the function works as expected, for example: 71 | 72 | |# 73 | 74 | (append-map (lambda (x) (list x (* 10 x))) '(1 2 3 4 5)) 75 | #| 76 | => (1 10 2 20 3 30 4 40 5 50) 77 | |# 78 | 79 | (append-map (lambda (x) (list (evenp x) (oddp x))) '(1 2 3 4 5)) 80 | #| 81 | => (NIL T T NIL NIL T T NIL NIL T) 82 | |# 83 | 84 | 85 | #| 86 | 87 | The type of this function is something like this: 88 | 89 | ``` 90 | (a -> L b) * L a -> L b 91 | ``` 92 | 93 | where we interpret `L a` as "a `L`ist of `a`'s". Not that I want to force you thinking in types, but we need at least an intuitive understanding of the types of our functions, right ? 94 | 95 | The first and most famous monadic combinator is called `bind` and in the case of the list monad it is almost a synonymous for `append-map`. 96 | 97 | |# 98 | 99 | ;; L a * (a -> L b) -> L b 100 | (defun list-bind (l f) 101 | (append-map f l)) 102 | 103 | #| 104 | 105 | **Remark**: for a monad `M` the type of `bind` is `M a * (a -> M b) -> M b`. It's good to know even if we will only consider the list monad `L` today. 106 | 107 | |# 108 | 109 | #| 110 | 111 | The second fundamental combinator is `return` that is the simplest function we can imagine with a type `a -> L a` (`a -> M a` for a monad `M` in general). 112 | 113 | |# 114 | 115 | ;; a -> L a 116 | (defun list-return (x) 117 | (list x)) 118 | 119 | #| 120 | 121 | And we in fact already have everything we need to write simple comprehensions. 122 | 123 | Let's start slowly: 124 | 125 | |# 126 | 127 | (list-bind '(1 2 3 4 5) 128 | (lambda (x) (list-return (* x x)))) 129 | #| 130 | => (1 4 9 16 25) 131 | |# 132 | 133 | #| 134 | 135 | And now something a little bit more involved. 136 | 137 | |# 138 | 139 | (list-bind 140 | '(1 2 3 4) 141 | (lambda (x) (list-bind 142 | '(A B) 143 | (lambda (y) (list-return (cons x y)))))) 144 | 145 | #| 146 | => ((1 . A) (1 . B) (2 . A) (2 . B) (3 . A) (3 . B) (4 . A) (4 . B)) 147 | |# 148 | 149 | #| 150 | 151 | If we want to add some condition in the comprehension, we need a third combinator called `fail`. In the case of the list monad, failing simply boils down to yielding the empty list. 152 | 153 | |# 154 | 155 | (defun list-fail () 156 | (list)) 157 | 158 | #| 159 | 160 | And now we can write e.g.: 161 | 162 | |# 163 | 164 | (list-bind '(1 2 3 4 5 6 7 8) 165 | (lambda (x) (if (evenp x) (list-return x) (list-fail)))) 166 | 167 | #| 168 | => (2 4 6 8) 169 | |# 170 | 171 | (list-bind '(1 2 3 4 5 6 7 8) 172 | (lambda (x) (if (oddp x) (list-return x) (list-fail)))) 173 | 174 | #| 175 | => (1 3 5 7) 176 | |# 177 | 178 | #| 179 | 180 | ## The do notation 181 | 182 | If we compare to the comprehension expressions of Python, we can probably say 183 | that using directly the combinators `bind`, `return` and `fail` is on the one 184 | hand more explicit but on the other hand less elegant. 185 | 186 | Haskell provides a `do` notation that greatly improves the readability of monadic combinations. In fact, 187 | monads (and their relatives functors and applicatives) are mostly about plumbing, as 188 | we can see with the examples above. 189 | The do notation allows to hide and reorganize some of this plumbing. 190 | 191 | This is a straightforward macro in Common Lisp. 192 | 193 | |# 194 | 195 | ;; a very permissive recognizer for "symbols" 196 | (defun is-sym (kw str) 197 | (and (symbolp kw) 198 | (string= (symbol-name kw) str))) 199 | 200 | (defmacro do-list (&body body) 201 | (cond 202 | ((null body) (progn)) 203 | ((null (cdr body)) (car body)) 204 | ((is-sym (cadr body) "<-") 205 | `(list-bind ,(caddr body) (lambda (,(car body)) (do-list ,@(cdddr body))))) 206 | ((is-sym (car body) "WHEN") 207 | `(if ,(cadr body) (do-list ,@(cddr body)) (list-fail))) 208 | ((is-sym (car body) "YIELD") 209 | `(list-return ,(cadr body))) 210 | (t (error "Not a do-able expression: ~S" `(quote ,body))))) 211 | 212 | #| 213 | 214 | And now we can write: 215 | 216 | |# 217 | 218 | 219 | (do-list 220 | i <- '(1 2 3 4 5) 221 | yield (* i i)) 222 | #| 223 | => (1 4 9 16 25) 224 | |# 225 | 226 | (do-list 227 | i <- '(1 2 3 4) 228 | j <- '(A B) 229 | yield (cons i j)) 230 | #| 231 | => ((1 . A) (1 . B) (2 . A) (2 . B) (3 . A) (3 . B) (4 . A) (4 . B)) 232 | |# 233 | 234 | (do-list 235 | i <- '(1 2 3 4 5 6 7 8) 236 | when (evenp i) 237 | yield i) 238 | #| 239 | => (2 4 6 8) 240 | |# 241 | 242 | (do-list 243 | i <- '(1 2 3 4 5 6 7 8) 244 | when (oddp i) 245 | yield i) 246 | #| 247 | => (1 3 5 7) 248 | |# 249 | 250 | #| 251 | 252 | 253 | All is well, we have now a simple macro for list comprehensions, and as you can see the monadic thinking make things very concise and (if you spend some time to understand) straightforward. 254 | 255 | But is this the simplest and best way to bring list comprehensions to Lisp? 256 | 257 | Let's see another (and please be reassured "un-monadic") way... 258 | 259 | |# 260 | 261 | #| 262 | 263 | ## The Loop way to comprehensions 264 | 265 | Instead of building on basic functions -- the Monad way -- we can 266 | build our list comprehensions on higher-level abstractions. In plain 267 | Common Lisp, the `loop` macro is a 268 | good candidate. For `loop`-*haters* I would recommend porting the code 269 | below to the more *lispy* and undoubtedly (even) more powerful [iterate](https://common-lisp.net/project/iterate/) 270 | macro (that you'll find on [quicklisp](https://www.quicklisp.org/beta/) of course). But let's stick with 271 | `loop` that already provides everything we need (and more !). If you 272 | need a loop refresher, please consult [the excellent Practical Common Lisp](http://www.gigamonkeys.com/book/loop-for-black-belts.html). 273 | 274 | So, how would we write the list of squares in loop ? Easy ! 275 | 276 | |# 277 | 278 | (loop for i in '(1 2 3 4 5) 279 | collect (* i i)) 280 | #| 281 | => (1 4 9 16 25) 282 | |# 283 | 284 | #| 285 | 286 | For nested comprehensions, the `append` clause is our friend. 287 | 288 | |# 289 | 290 | (loop for i in '(1 2 3 4) 291 | append (loop for j in '(A B) 292 | collect (cons i j))) 293 | #| 294 | => ((1 . A) (1 . B) (2 . A) (2 . B) (3 . A) (3 . B) (4 . A) (4 . B)) 295 | |# 296 | 297 | #| 298 | 299 | **Remark**: `Aidenn0@reddit/lisp` tells me quite rightly that instead of the "constructive" `append` we 300 | can use the "destructive" `nconc` since the inner `collect` clause generates a fresh 301 | list. So we have an even better friend since `nconc`-ing is much more efficient. 302 | 303 | |# 304 | 305 | (loop for i in '(1 2 3 4) 306 | nconc (loop for j in '(A B) 307 | collect (cons i j))) 308 | #| 309 | => ((1 . A) (1 . B) (2 . A) (2 . B) (3 . A) (3 . B) (4 . A) (4 . B)) 310 | |# 311 | 312 | #| 313 | 314 | Filtering is also easy with the `when` clause. 315 | 316 | |# 317 | 318 | (loop for i in '(1 2 3 4 5 6 7 8) 319 | when (evenp i) 320 | collect i) 321 | #| 322 | => (2 4 6 8) 323 | |# 324 | 325 | #| 326 | 327 | By looking at these examples, we might say that list comprehension expressions are 328 | not really needed in Common Lisp since the corresponding `loop` expressions are both 329 | readable and efficient (probably more so than using the list monad). 330 | 331 | However, the loop syntax is complex and it is still useful to provide a simpler 332 | abstraction *exclusively* for list comprehensions. Moreover, we can exploit various loop features 333 | to enrich our comprehension framework. 334 | 335 | |# 336 | 337 | #| 338 | 339 | ## The `list-of` macro for list comprehensions 340 | 341 | We now reach the final stage of our exploration of list comprehensions. We will build 342 | a (relatively) simple macro named `list-of` that will macroexpand to `loop` expressions. 343 | 344 | The basic comprehensions will be of the form: 345 | 346 | ``` 347 | (list-of for in ) 348 | ``` 349 | 350 | More generally, all `list-of` expressions will be of the form `(list-of for ...)`. 351 | Hence, the first clause will always be a `loop`-like `for` clause. 352 | 353 | The following transformer manages to extract this first clause from a `list-of` expression. 354 | 355 | |# 356 | 357 | (defun transform-first-clause (expr) 358 | (cond 359 | ((null expr) (error "Empty comprehension: missing first 'for' clause")) 360 | ((is-sym (car expr) "FOR") 361 | (values `(for ,(cadr expr) ,(caddr expr) ,(cadddr expr)) (cddddr expr))) 362 | (t (error "First 'for' clause missing in comprehension.")))) 363 | 364 | #| 365 | 366 | For example: 367 | 368 | |# 369 | 370 | (transform-first-clause 371 | '(for i in '(1 2 3 4 5) for j in '(A B) for k across "Hello")) 372 | #| 373 | => (FOR I IN '(1 2 3 4 5)) ; the extracted clause as a first value 374 | => (FOR J IN '(A B) FOR K ACROSS "Hello") ; the remaining clauses as a second value 375 | |# 376 | 377 | #| 378 | 379 | Now we write a second transformer for all the remaining clauses. The supported clauses 380 | are the following ones: 381 | 382 | - the `for` clause for nested comprehensions (using an inner `loop` nested within an `append` clause) 383 | 384 | - the `and` clause for parallel comprehensions (similarly to multiple `for` clauses in a single `loop`) 385 | 386 | - the `with` clause to fix a temporary variable (`with = ` corresponds to `loop`'s 387 | `for = `) 388 | 389 | - other intresting `loop` clauses: `when` for filtering and `until` for early terminations (do you see something else ?). 390 | 391 | Note that this transformer is not very robust (with many cadd...r's), 392 | but adding all the error cases would make the code less readable I guess. For the real thing I would 393 | strongly recommand using the `optima` pattern matcher. 394 | 395 | |# 396 | 397 | 398 | (defun transform-clause (expr) 399 | (if (null expr) 400 | (values :END '() '()) 401 | (cond 402 | ((is-sym (car expr) "AND") 403 | (values :AND `(for ,(cadr expr) ,(caddr expr) ,(cadddr expr)) (cddddr expr))) 404 | ((is-sym (car expr) "WHEN") 405 | (values :WHEN `(when ,(cadr expr)) (cddr expr))) 406 | ((is-sym (car expr) "UNTIL") 407 | (values :UNTIL `(until ,(cadr expr)) (cddr expr))) 408 | ((is-sym (car expr) "WITH") 409 | (values :WITH `(for ,(cadr expr) ,(caddr expr) ,(cadddr expr)) (cddddr expr))) 410 | ((is-sym (car expr) "FOR") 411 | (values :FOR `(for ,(cadr expr) ,(caddr expr) ,(cadddr expr)) (cddddr expr))) 412 | (t (error "Expecting 'and', 'for', 'when' or 'until' in comprehension."))))) 413 | 414 | #| 415 | 416 | A few examples: 417 | 418 | |# 419 | 420 | (transform-clause '()) 421 | #| 422 | => :END , NIL, NIL 423 | |# 424 | 425 | (transform-clause '(and i in '(1 2 3 4) for j in '(A B) collect i)) 426 | #| 427 | => :AND ; the kind of clause as a first value 428 | => (FOR I IN '(1 2 3 4)) ; the clause content as a second value 429 | => (FOR J IN '(A B) COLLECT I) ; the remaining clauses as a last value 430 | |# 431 | 432 | #| 433 | 434 | Now we can write the main transformer, that simply recurse 435 | on the clause transformer above. The case for the nested 436 | comprehensions injects the *`loop`-within-`nconc`* expression. 437 | But everything is otherwise rather straightforward. 438 | 439 | |# 440 | 441 | (defun list-of-transformer (what expr) 442 | (multiple-value-bind (kind next rexpr) 443 | (transform-clause expr) 444 | (case kind 445 | (:END `(collect ,what)) 446 | ((:AND :WHEN :UNTIL :WITH) (append next (list-of-transformer what rexpr))) 447 | (:FOR `(nconc (loop ,@next ,@(list-of-transformer what rexpr))))))) 448 | 449 | (list-of-transformer '(cons i j) '(with k = i and i in '(1 2 3 4) for j in '(A B))) 450 | #| 451 | => (FOR K = I FOR I IN '(1 2 3 4) APPEND 452 | (LOOP FOR J IN '(A B) 453 | COLLECT (CONS I J))) 454 | |# 455 | 456 | #| 457 | 458 | The macro itself is simply the construction of a `loop` expression with 459 | a body generated by our two transformers `transform-first-clause` and 460 | `list-of-transformer`. 461 | 462 | |# 463 | 464 | (defmacro list-of (what &body body) 465 | (multiple-value-bind (first-clause rest) 466 | (transform-first-clause body) 467 | `(loop ,@first-clause ,@(list-of-transformer what rest)))) 468 | 469 | #| 470 | 471 | Now let's see somme expansions... we'll also check that things 472 | are working (at least seemingly) well. 473 | 474 | |# 475 | 476 | (macroexpand-1 '(list-of (* i i) for i in '(1 2 3 4 5))) 477 | #| 478 | => (LOOP FOR I IN '(1 2 3 4 5) 479 | COLLECT (* I I)) 480 | |# 481 | 482 | (list-of (* i i) for i in '(1 2 3 4 5)) 483 | #| 484 | => (1 4 9 16 25) 485 | |# 486 | 487 | #| 488 | 489 | Let's try the `with` clause (and see that it is expanded to 490 | a `for` clause in the loop). 491 | 492 | |# 493 | 494 | (macroexpand-1 '(list-of k 495 | for i in '(1 2 3 4 5) 496 | and j in '(1 2 3 4 5) with k = (+ i j))) 497 | #| 498 | => (LOOP FOR I IN '(1 2 3 4 5) 499 | FOR J IN '(1 2 3 4 5) 500 | FOR K = (+ I J) 501 | COLLECT K) 502 | |# 503 | 504 | (list-of k for i in '(1 2 3 4 5) 505 | and j in '(1 2 3 4 5) 506 | with k = (+ i j)) 507 | #| 508 | => (2 4 6 8 10) 509 | |# 510 | 511 | 512 | #| 513 | 514 | Filtering with `when` of course works. 515 | 516 | |# 517 | 518 | (list-of i 519 | for i in '(1 2 3 4 5 6 7 8) 520 | when (evenp i)) 521 | #| 522 | => (2 4 6 8) 523 | |# 524 | 525 | 526 | (list-of i 527 | for i in '(1 2 3 4 5 6 7 8) 528 | when (oddp i)) 529 | #| 530 | => (1 3 5 7) 531 | |# 532 | 533 | #| 534 | 535 | The nesting of comprehensions yields nested loops as expected. 536 | 537 | |# 538 | 539 | (macroexpand-1 '(list-of (cons i j) 540 | for i in '(1 2 3 4) 541 | for j in '(A B))) 542 | #| 543 | => (LOOP FOR I IN '(1 2 3 4) 544 | NCONC (LOOP FOR J IN '(A B) 545 | COLLECT (CONS I J))) 546 | |# 547 | 548 | (list-of (cons i j) 549 | for i in '(1 2 3 4) 550 | for j in '(A B)) 551 | #| 552 | => ((1 . A) (1 . B) (2 . A) (2 . B) (3 . A) (3 . B) (4 . A) (4 . B)) 553 | |# 554 | 555 | #| 556 | 557 | To illustrate the `until` clause let's stop a little bit earlier. 558 | 559 | |# 560 | 561 | (list-of (cons i j) 562 | for i in '(1 2 3 4) 563 | until (= i 3) 564 | for j in '(A B)) 565 | #| 566 | => ((1 . A) (1 . B) (2 . A) (2 . B)) 567 | |# 568 | 569 | 570 | #| 571 | 572 | It is to be compared with parallel comprehensions, as in the following example. 573 | 574 | |# 575 | 576 | (macroexpand-1 '(list-of (cons i j) 577 | for i in '(1 2 3 4) 578 | and j in '(A B))) 579 | #| 580 | => (LOOP FOR I IN '(1 2 3 4) 581 | FOR J IN '(A B) 582 | COLLECT (CONS I J)) 583 | |# 584 | 585 | 586 | (list-of (cons i j) 587 | for i in '(1 2 3 4) 588 | and j in '(A B)) 589 | #| 590 | => ((1 . A) (2 . B)) 591 | |# 592 | 593 | 594 | #| 595 | 596 | And the last one is left as en excercise. 597 | 598 | |# 599 | 600 | (list-of (list i j k) 601 | for i in '(1 2 3 4 5) 602 | and j in '(A B C D E) 603 | when (oddp i) 604 | for k in '(1 2 3 4) 605 | until (= i 5) 606 | when (evenp k)) 607 | 608 | #| 609 | 610 | ## Conclusion 611 | 612 | > So what did we achieve ? 613 | 614 | First, we found two different ways to provide *list comprehensions* in Common Lisp. Whereas 615 | in many programming languages this is a matter of hope or faith, a Lisp programmer can take 616 | his destiny at hand and introduce the feature he wants *the way* he wants it ! 617 | 618 | Another take away is that monadic thinking is useful, even if you must 619 | be able to go beyond the type-based plumbing they propose... In a way 620 | monads are a kind of well-typed-although-limited macros. Simply relying on the powerful 621 | `loop` mini-language allowed us to go beyond simple comprehensions (especially with the `until` clause). 622 | 623 | If compared to comprehension expressions found in other languages, it is not a bad start. 624 | Thanks to `loop` we can take different sources: lists (with `in`), strings (with `across`), etc. 625 | There would be some (simple) work to support more collections such as hashtables. Perhaps a better 626 | thing would be to base our macro on `iterate` since the latter is extensible. 627 | 628 | Finally, Python also has set, dictionnary comprehensions as well as generator expressions (actually 629 | closer to Clojure's sequence comprehensions). This goes beyond our `list-of` macro but it is 630 | a topic I do intend to further study. 631 | 632 | And that's it for today... 633 | 634 | ## Discussion 635 | 636 | cf. the [reddit/lisp comments](https://www.reddit.com/r/lisp/comments/3j7zyt/list_comprehensions_in_common_lisp_tutorial/) 637 | 638 | |# 639 | -------------------------------------------------------------------------------- /list-comprehensions.lisp.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | # List comprehensions in Common Lisp 5 | 6 | > This document is (C) Frédéric Peschanski CC-BY-SA 3.0 7 | 8 | My favorite programming languages are undoubtedly *Lisp*s, originally Scheme but todays I enjoy a lot coding in Common Lisp and Clojure. I am however quite a polyglot programmer: I taught (and thus used, a lot) Java for several years. And at work I program a lot in Ocaml and Python (but that's *not* for fun). 9 | 10 | There's no programming language that I really hate (except VB) or love (except Lisp). Let's take for example Python: not efficient, full of what I think are ill-designed constructs (e.g. `lambda`, also the lexical bindings) but clearly usable. One feature I use a lot in Python is the *comprehensions*. There are comprehension expressions for building lists, sets, dictionaries and (perhaps most interestingly) generators. Today I will only discuss the case of *list comprehensions* because ... I think that's an interesting topic to begin with. 11 | 12 | A simple list comprehension expression is written as follows in Python: 13 | 14 | ``` 15 | [ for in ] 16 | ``` 17 | 18 | This reads: "build the list of `` in which the occurrences of `` are replaced in turn by 19 | the successive elements of the source ``." 20 | 21 | Note that the source of the comprehension can be something else than a list (it can be an *iterable* or a generator) but we will mostly consider the case for lists (although we will make some remarks about this aspect at the end). 22 | 23 | **Remark**: the functional programming language Haskell has a similar syntactic construct, and so does Clojure (cf. the `for` macro). 24 | 25 | Let's see some examples of Python comprehensions. 26 | 27 | ```python 28 | >>> [i*i for i in [1, 2, 3, 4, 5]] 29 | [1, 4, 9, 16, 25] 30 | ``` 31 | 32 | Comprehensions can also nest, performing a kind of *cartesian product*, e.g.: 33 | 34 | ```python 35 | >>> [(i, j) for i in [1, 2, 3, 4] 36 | for j in ['A', 'B']] 37 | [(1, 'A'), (1, 'B'), (2, 'A'), (2, 'B'), (3, 'A'), (3, 'B'), (4, 'A'), (4, 'B')] 38 | ``` 39 | 40 | It is also possible to filter the elements: 41 | 42 | ```python 43 | >>> [i for i in [1, 2, 3, 4, 5, 6, 7, 8] if i % 2 == 0] 44 | [2, 4, 6, 8] 45 | ``` 46 | You know that already, but Common Lisp does not provide *list comprehension expressions*. However, the programmable programming language is never short of macro-based solutions for "missing" constructs! 47 | 48 | So, our objective will be to find a few ways to provide list comprehensions to Common Lisp. 49 | 50 | 51 | 52 | 53 | 54 | 55 | ## The List monad 56 | 57 | No no no! This is not *yet another monad tutorial* (tm)! But please don't hate monads or other interesting design patterns of functional programming languages. There's some beauty in the concept, and I will try to convey this beauty by considering one of the simplest case: the *list monad*. 58 | 59 | The fundamental operator that we are concerned with is a function that does not look *that* useful: `append-map`. 60 | 61 | 62 | 63 | ```lisp 64 | (defun append-map (f ll) 65 | (if (endp ll) 66 | '() 67 | (append (funcall f (car ll)) (append-map f (cdr ll))))) 68 | 69 | ``` 70 | 71 | 72 | Of course, `append` is interesting, and `map`/`mapcar` too, but mixing the two does not look extremely useful (as pointed out by `lispm@reddit/lisp` a similar function is called `mappend` in [alexandria](https://common-lisp.net/project/alexandria/)). Anyway the function works as expected, for example: 73 | 74 | 75 | 76 | ```lisp 77 | (append-map (lambda (x) (list x (* 10 x))) '(1 2 3 4 5)) 78 | ``` 79 | 80 | => (1 10 2 20 3 30 4 40 5 50) 81 | 82 | 83 | ```lisp 84 | (append-map (lambda (x) (list (evenp x) (oddp x))) '(1 2 3 4 5)) 85 | ``` 86 | 87 | => (NIL T T NIL NIL T T NIL NIL T) 88 | 89 | 90 | 91 | 92 | 93 | The type of this function is something like this: 94 | 95 | ``` 96 | (a -> L b) * L a -> L b 97 | ``` 98 | 99 | where we interpret `L a` as "a `L`ist of `a`'s". Not that I want to force you thinking in types, but we need at least an intuitive understanding of the types of our functions, right ? 100 | 101 | The first and most famous monadic combinator is called `bind` and in the case of the list monad it is almost a synonymous for `append-map`. 102 | 103 | 104 | 105 | ```lisp 106 | ;; L a * (a -> L b) -> L b 107 | (defun list-bind (l f) 108 | (append-map f l)) 109 | 110 | ``` 111 | 112 | 113 | **Remark**: for a monad `M` the type of `bind` is `M a * (a -> M b) -> M b`. It's good to know even if we will only consider the list monad `L` today. 114 | 115 | 116 | 117 | 118 | 119 | The second fundamental combinator is `return` that is the simplest function we can imagine with a type `a -> L a` (`a -> M a` for a monad `M` in general). 120 | 121 | 122 | 123 | ```lisp 124 | ;; a -> L a 125 | (defun list-return (x) 126 | (list x)) 127 | 128 | ``` 129 | 130 | 131 | And we in fact already have everything we need to write simple comprehensions. 132 | 133 | Let's start slowly: 134 | 135 | 136 | 137 | ```lisp 138 | (list-bind '(1 2 3 4 5) 139 | (lambda (x) (list-return (* x x)))) 140 | ``` 141 | 142 | => (1 4 9 16 25) 143 | 144 | 145 | 146 | 147 | And now something a little bit more involved. 148 | 149 | 150 | 151 | ```lisp 152 | (list-bind 153 | '(1 2 3 4) 154 | (lambda (x) (list-bind 155 | '(A B) 156 | (lambda (y) (list-return (cons x y)))))) 157 | 158 | ``` 159 | 160 | => ((1 . A) (1 . B) (2 . A) (2 . B) (3 . A) (3 . B) (4 . A) (4 . B)) 161 | 162 | 163 | 164 | 165 | If we want to add some condition in the comprehension, we need a third combinator called `fail`. In the case of the list monad, failing simply boils down to yielding the empty list. 166 | 167 | 168 | 169 | ```lisp 170 | (defun list-fail () 171 | (list)) 172 | 173 | ``` 174 | 175 | 176 | And now we can write e.g.: 177 | 178 | 179 | 180 | ```lisp 181 | (list-bind '(1 2 3 4 5 6 7 8) 182 | (lambda (x) (if (evenp x) (list-return x) (list-fail)))) 183 | 184 | ``` 185 | 186 | => (2 4 6 8) 187 | 188 | 189 | ```lisp 190 | (list-bind '(1 2 3 4 5 6 7 8) 191 | (lambda (x) (if (oddp x) (list-return x) (list-fail)))) 192 | 193 | ``` 194 | 195 | => (1 3 5 7) 196 | 197 | 198 | 199 | 200 | ## The do notation 201 | 202 | If we compare to the comprehension expressions of Python, we can probably say 203 | that using directly the combinators `bind`, `return` and `fail` is on the one 204 | hand more explicit but on the other hand less elegant. 205 | 206 | Haskell provides a `do` notation that greatly improves the readability of monadic combinations. In fact, 207 | monads (and their relatives functors and applicatives) are mostly about plumbing, as 208 | we can see with the examples above. 209 | The do notation allows to hide and reorganize some of this plumbing. 210 | 211 | This is a straightforward macro in Common Lisp. 212 | 213 | 214 | 215 | ```lisp 216 | ;; a very permissive recognizer for "symbols" 217 | (defun is-sym (kw str) 218 | (and (symbolp kw) 219 | (string= (symbol-name kw) str))) 220 | 221 | (defmacro do-list (&body body) 222 | (cond 223 | ((null body) (progn)) 224 | ((null (cdr body)) (car body)) 225 | ((is-sym (cadr body) "<-") 226 | `(list-bind ,(caddr body) (lambda (,(car body)) (do-list ,@(cdddr body))))) 227 | ((is-sym (car body) "WHEN") 228 | `(if ,(cadr body) (do-list ,@(cddr body)) (list-fail))) 229 | ((is-sym (car body) "YIELD") 230 | `(list-return ,(cadr body))) 231 | (t (error "Not a do-able expression: ~S" `(quote ,body))))) 232 | 233 | ``` 234 | 235 | 236 | And now we can write: 237 | 238 | 239 | 240 | 241 | ```lisp 242 | (do-list 243 | i <- '(1 2 3 4 5) 244 | yield (* i i)) 245 | ``` 246 | 247 | => (1 4 9 16 25) 248 | 249 | 250 | ```lisp 251 | (do-list 252 | i <- '(1 2 3 4) 253 | j <- '(A B) 254 | yield (cons i j)) 255 | ``` 256 | 257 | => ((1 . A) (1 . B) (2 . A) (2 . B) (3 . A) (3 . B) (4 . A) (4 . B)) 258 | 259 | 260 | ```lisp 261 | (do-list 262 | i <- '(1 2 3 4 5 6 7 8) 263 | when (evenp i) 264 | yield i) 265 | ``` 266 | 267 | => (2 4 6 8) 268 | 269 | 270 | ```lisp 271 | (do-list 272 | i <- '(1 2 3 4 5 6 7 8) 273 | when (oddp i) 274 | yield i) 275 | ``` 276 | 277 | => (1 3 5 7) 278 | 279 | 280 | 281 | 282 | 283 | All is well, we have now a simple macro for list comprehensions, and as you can see the monadic thinking make things very concise and (if you spend some time to understand) straightforward. 284 | 285 | But is this the simplest and best way to bring list comprehensions to Lisp? 286 | 287 | Let's see another (and please be reassured "un-monadic") way... 288 | 289 | 290 | 291 | 292 | 293 | ## The Loop way to comprehensions 294 | 295 | Instead of building on basic functions -- the Monad way -- we can 296 | build our list comprehensions on higher-level abstractions. In plain 297 | Common Lisp, the `loop` macro is a 298 | good candidate. For `loop`-*haters* I would recommend porting the code 299 | below to the more *lispy* and undoubtedly (even) more powerful [iterate](https://common-lisp.net/project/iterate/) 300 | macro (that you'll find on [quicklisp](https://www.quicklisp.org/beta/) of course). But let's stick with 301 | `loop` that already provides everything we need (and more !). If you 302 | need a loop refresher, please consult [the excellent Practical Common Lisp](http://www.gigamonkeys.com/book/loop-for-black-belts.html). 303 | 304 | So, how would we write the list of squares in loop ? Easy ! 305 | 306 | 307 | 308 | ```lisp 309 | (loop for i in '(1 2 3 4 5) 310 | collect (* i i)) 311 | ``` 312 | 313 | => (1 4 9 16 25) 314 | 315 | 316 | 317 | 318 | For nested comprehensions, the `append` clause is our friend. 319 | 320 | 321 | 322 | ```lisp 323 | (loop for i in '(1 2 3 4) 324 | append (loop for j in '(A B) 325 | collect (cons i j))) 326 | ``` 327 | 328 | => ((1 . A) (1 . B) (2 . A) (2 . B) (3 . A) (3 . B) (4 . A) (4 . B)) 329 | 330 | 331 | 332 | 333 | **Remark**: `Aidenn0@reddit/lisp` tells me quite rightly that instead of the "constructive" `append` we 334 | can use the "destructive" `nconc` since the inner `collect` clause generates a fresh 335 | list. So we have an even better friend since `nconc`-ing is much more efficient. 336 | 337 | 338 | 339 | ```lisp 340 | (loop for i in '(1 2 3 4) 341 | nconc (loop for j in '(A B) 342 | collect (cons i j))) 343 | ``` 344 | 345 | => ((1 . A) (1 . B) (2 . A) (2 . B) (3 . A) (3 . B) (4 . A) (4 . B)) 346 | 347 | 348 | 349 | 350 | Filtering is also easy with the `when` clause. 351 | 352 | 353 | 354 | ```lisp 355 | (loop for i in '(1 2 3 4 5 6 7 8) 356 | when (evenp i) 357 | collect i) 358 | ``` 359 | 360 | => (2 4 6 8) 361 | 362 | 363 | 364 | 365 | By looking at these examples, we might say that list comprehension expressions are 366 | not really needed in Common Lisp since the corresponding `loop` expressions are both 367 | readable and efficient (probably more so than using the list monad). 368 | 369 | However, the loop syntax is complex and it is still useful to provide a simpler 370 | abstraction *exclusively* for list comprehensions. Moreover, we can exploit various loop features 371 | to enrich our comprehension framework. 372 | 373 | 374 | 375 | 376 | 377 | ## The `list-of` macro for list comprehensions 378 | 379 | We now reach the final stage of our exploration of list comprehensions. We will build 380 | a (relatively) simple macro named `list-of` that will macroexpand to `loop` expressions. 381 | 382 | The basic comprehensions will be of the form: 383 | 384 | ``` 385 | (list-of for in ) 386 | ``` 387 | 388 | More generally, all `list-of` expressions will be of the form `(list-of for ...)`. 389 | Hence, the first clause will always be a `loop`-like `for` clause. 390 | 391 | The following transformer manages to extract this first clause from a `list-of` expression. 392 | 393 | 394 | 395 | ```lisp 396 | (defun transform-first-clause (expr) 397 | (cond 398 | ((null expr) (error "Empty comprehension: missing first 'for' clause")) 399 | ((is-sym (car expr) "FOR") 400 | (values `(for ,(cadr expr) ,(caddr expr) ,(cadddr expr)) (cddddr expr))) 401 | (t (error "First 'for' clause missing in comprehension.")))) 402 | 403 | ``` 404 | 405 | 406 | For example: 407 | 408 | 409 | 410 | ```lisp 411 | (transform-first-clause 412 | '(for i in '(1 2 3 4 5) for j in '(A B) for k across "Hello")) 413 | ``` 414 | 415 | => (FOR I IN '(1 2 3 4 5)) ; the extracted clause as a first value 416 | => (FOR J IN '(A B) FOR K ACROSS "Hello") ; the remaining clauses as a second value 417 | 418 | 419 | 420 | 421 | Now we write a second transformer for all the remaining clauses. The supported clauses 422 | are the following ones: 423 | 424 | - the `for` clause for nested comprehensions (using an inner `loop` nested within an `append` clause) 425 | 426 | - the `and` clause for parallel comprehensions (similarly to multiple `for` clauses in a single `loop`) 427 | 428 | - the `with` clause to fix a temporary variable (`with = ` corresponds to `loop`'s 429 | `for = `) 430 | 431 | - other intresting `loop` clauses: `when` for filtering and `until` for early terminations (do you see something else ?). 432 | 433 | Note that this transformer is not very robust (with many cadd...r's), 434 | but adding all the error cases would make the code less readable I guess. For the real thing I would 435 | strongly recommand using the `optima` pattern matcher. 436 | 437 | 438 | 439 | 440 | ```lisp 441 | (defun transform-clause (expr) 442 | (if (null expr) 443 | (values :END '() '()) 444 | (cond 445 | ((is-sym (car expr) "AND") 446 | (values :AND `(for ,(cadr expr) ,(caddr expr) ,(cadddr expr)) (cddddr expr))) 447 | ((is-sym (car expr) "WHEN") 448 | (values :WHEN `(when ,(cadr expr)) (cddr expr))) 449 | ((is-sym (car expr) "UNTIL") 450 | (values :UNTIL `(until ,(cadr expr)) (cddr expr))) 451 | ((is-sym (car expr) "WITH") 452 | (values :WITH `(for ,(cadr expr) ,(caddr expr) ,(cadddr expr)) (cddddr expr))) 453 | ((is-sym (car expr) "FOR") 454 | (values :FOR `(for ,(cadr expr) ,(caddr expr) ,(cadddr expr)) (cddddr expr))) 455 | (t (error "Expecting 'and', 'for', 'when' or 'until' in comprehension."))))) 456 | 457 | ``` 458 | 459 | 460 | A few examples: 461 | 462 | 463 | 464 | ```lisp 465 | (transform-clause '()) 466 | ``` 467 | 468 | => :END , NIL, NIL 469 | 470 | 471 | ```lisp 472 | (transform-clause '(and i in '(1 2 3 4) for j in '(A B) collect i)) 473 | ``` 474 | 475 | => :AND ; the kind of clause as a first value 476 | => (FOR I IN '(1 2 3 4)) ; the clause content as a second value 477 | => (FOR J IN '(A B) COLLECT I) ; the remaining clauses as a last value 478 | 479 | 480 | 481 | 482 | Now we can write the main transformer, that simply recurse 483 | on the clause transformer above. The case for the nested 484 | comprehensions injects the *`loop`-within-`nconc`* expression. 485 | But everything is otherwise rather straightforward. 486 | 487 | 488 | 489 | ```lisp 490 | (defun list-of-transformer (what expr) 491 | (multiple-value-bind (kind next rexpr) 492 | (transform-clause expr) 493 | (case kind 494 | (:END `(collect ,what)) 495 | ((:AND :WHEN :UNTIL :WITH) (append next (list-of-transformer what rexpr))) 496 | (:FOR `(nconc (loop ,@next ,@(list-of-transformer what rexpr))))))) 497 | 498 | (list-of-transformer '(cons i j) '(with k = i and i in '(1 2 3 4) for j in '(A B))) 499 | ``` 500 | 501 | => (FOR K = I FOR I IN '(1 2 3 4) APPEND 502 | (LOOP FOR J IN '(A B) 503 | COLLECT (CONS I J))) 504 | 505 | 506 | 507 | 508 | The macro itself is simply the construction of a `loop` expression with 509 | a body generated by our two transformers `transform-first-clause` and 510 | `list-of-transformer`. 511 | 512 | 513 | 514 | ```lisp 515 | (defmacro list-of (what &body body) 516 | (multiple-value-bind (first-clause rest) 517 | (transform-first-clause body) 518 | `(loop ,@first-clause ,@(list-of-transformer what rest)))) 519 | 520 | ``` 521 | 522 | 523 | Now let's see somme expansions... we'll also check that things 524 | are working (at least seemingly) well. 525 | 526 | 527 | 528 | ```lisp 529 | (macroexpand-1 '(list-of (* i i) for i in '(1 2 3 4 5))) 530 | ``` 531 | 532 | => (LOOP FOR I IN '(1 2 3 4 5) 533 | COLLECT (* I I)) 534 | 535 | 536 | ```lisp 537 | (list-of (* i i) for i in '(1 2 3 4 5)) 538 | ``` 539 | 540 | => (1 4 9 16 25) 541 | 542 | 543 | 544 | 545 | Let's try the `with` clause (and see that it is expanded to 546 | a `for` clause in the loop). 547 | 548 | 549 | 550 | ```lisp 551 | (macroexpand-1 '(list-of k 552 | for i in '(1 2 3 4 5) 553 | and j in '(1 2 3 4 5) with k = (+ i j))) 554 | ``` 555 | 556 | => (LOOP FOR I IN '(1 2 3 4 5) 557 | FOR J IN '(1 2 3 4 5) 558 | FOR K = (+ I J) 559 | COLLECT K) 560 | 561 | 562 | ```lisp 563 | (list-of k for i in '(1 2 3 4 5) 564 | and j in '(1 2 3 4 5) 565 | with k = (+ i j)) 566 | ``` 567 | 568 | => (2 4 6 8 10) 569 | 570 | 571 | 572 | 573 | 574 | Filtering with `when` of course works. 575 | 576 | 577 | 578 | ```lisp 579 | (list-of i 580 | for i in '(1 2 3 4 5 6 7 8) 581 | when (evenp i)) 582 | ``` 583 | 584 | => (2 4 6 8) 585 | 586 | 587 | 588 | ```lisp 589 | (list-of i 590 | for i in '(1 2 3 4 5 6 7 8) 591 | when (oddp i)) 592 | ``` 593 | 594 | => (1 3 5 7) 595 | 596 | 597 | 598 | 599 | The nesting of comprehensions yields nested loops as expected. 600 | 601 | 602 | 603 | ```lisp 604 | (macroexpand-1 '(list-of (cons i j) 605 | for i in '(1 2 3 4) 606 | for j in '(A B))) 607 | ``` 608 | 609 | => (LOOP FOR I IN '(1 2 3 4) 610 | NCONC (LOOP FOR J IN '(A B) 611 | COLLECT (CONS I J))) 612 | 613 | 614 | ```lisp 615 | (list-of (cons i j) 616 | for i in '(1 2 3 4) 617 | for j in '(A B)) 618 | ``` 619 | 620 | => ((1 . A) (1 . B) (2 . A) (2 . B) (3 . A) (3 . B) (4 . A) (4 . B)) 621 | 622 | 623 | 624 | 625 | To illustrate the `until` clause let's stop a little bit earlier. 626 | 627 | 628 | 629 | ```lisp 630 | (list-of (cons i j) 631 | for i in '(1 2 3 4) 632 | until (= i 3) 633 | for j in '(A B)) 634 | ``` 635 | 636 | => ((1 . A) (1 . B) (2 . A) (2 . B)) 637 | 638 | 639 | 640 | 641 | 642 | It is to be compared with parallel comprehensions, as in the following example. 643 | 644 | 645 | 646 | ```lisp 647 | (macroexpand-1 '(list-of (cons i j) 648 | for i in '(1 2 3 4) 649 | and j in '(A B))) 650 | ``` 651 | 652 | => (LOOP FOR I IN '(1 2 3 4) 653 | FOR J IN '(A B) 654 | COLLECT (CONS I J)) 655 | 656 | 657 | 658 | ```lisp 659 | (list-of (cons i j) 660 | for i in '(1 2 3 4) 661 | and j in '(A B)) 662 | ``` 663 | 664 | => ((1 . A) (2 . B)) 665 | 666 | 667 | 668 | 669 | 670 | And the last one is left as en excercise. 671 | 672 | 673 | 674 | ```lisp 675 | (list-of (list i j k) 676 | for i in '(1 2 3 4 5) 677 | and j in '(A B C D E) 678 | when (oddp i) 679 | for k in '(1 2 3 4) 680 | until (= i 5) 681 | when (evenp k)) 682 | 683 | ``` 684 | 685 | 686 | ## Conclusion 687 | 688 | > So what did we achieve ? 689 | 690 | First, we found two different ways to provide *list comprehensions* in Common Lisp. Whereas 691 | in many programming languages this is a matter of hope or faith, a Lisp programmer can take 692 | his destiny at hand and introduce the feature he wants *the way* he wants it ! 693 | 694 | Another take away is that monadic thinking is useful, even if you must 695 | be able to go beyond the type-based plumbing they propose... In a way 696 | monads are a kind of well-typed-although-limited macros. Simply relying on the powerful 697 | `loop` mini-language allowed us to go beyond simple comprehensions (especially with the `until` clause). 698 | 699 | If compared to comprehension expressions found in other languages, it is not a bad start. 700 | Thanks to `loop` we can take different sources: lists (with `in`), strings (with `across`), etc. 701 | There would be some (simple) work to support more collections such as hashtables. Perhaps a better 702 | thing would be to base our macro on `iterate` since the latter is extensible. 703 | 704 | Finally, Python also has set, dictionnary comprehensions as well as generator expressions (actually 705 | closer to Clojure's sequence comprehensions). This goes beyond our `list-of` macro but it is 706 | a topic I do intend to further study. 707 | 708 | And that's it for today... 709 | 710 | ## Discussion 711 | 712 | cf. the [reddit/lisp comments](https://www.reddit.com/r/lisp/comments/3j7zyt/list_comprehensions_in_common_lisp_tutorial/) 713 | 714 | 715 | -------------------------------------------------------------------------------- /list-comprehensions.lisp.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fredokun/lisp-list-comprehensions/bec7c7b6d6c55e1dbe0651e1767cbd1b91448a9c/list-comprehensions.lisp.pdf -------------------------------------------------------------------------------- /makedoc.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | python3 ../markdownize/markdownize.py -i ./list-comprehensions.lisp -o ./list-comprehensions.lisp.md -b '#|' -e '|#' -l "lisp" 4 | 5 | python3 ../markdownize/markdownize.py -i ./list-comprehensions.lisp -o ./list-comprehensions.lisp-pandoc.md -b '#|' -e '|#' -l "commonlisp" 6 | 7 | # "commonlisp" for pandoc ! 8 | pandoc -s --highlight-style=pygments list-comprehensions.lisp-pandoc.md -o list-comprehensions.lisp.pdf 9 | 10 | rm -f list-comprehensions.lisp-pandoc.md 11 | 12 | 13 | --------------------------------------------------------------------------------