└── cognitive_tools.md /cognitive_tools.md: -------------------------------------------------------------------------------- 1 | # Working notes on cognitive tools 2 | 3 | Rough and incomplete working notes on how to create tools that "enable 4 | people to understand and create in unprecedentedly powerful ways" 5 | (Bret Victor). I shall refer to such tools as _cognitive tools_. The 6 | purpose of the notes is to think about the problem, to understand some 7 | of what other people have thought about the problem, and to collect 8 | project ideas. 9 | 10 | **What would it take to get the average person to understand general 11 | relativity?** Could we develop general cognitive tools to make it that 12 | easy? (C.f. Augustine as the "smartest person in the world" due to 13 | being able to read without speaking.) It may be that _motivational 14 | hacks_ are the key. 15 | 16 | 17 | **What can I contribute? Where, if anywhere, is my comparative 18 | advantage to be found?** 19 | 20 | 21 | 22 | **The comparison to self-help:** Self-help gets a bad rap. Many of 23 | these software tools have a similar goal, though, which is to improve 24 | the way we think. Yet a concrete tool like (say) org mode seems to 25 | have significantly more impact than lots of what you see in the 26 | self-help books. It's a type of executable or active self-help. That 27 | shift from passive self-help, which is often vague and unactionable, 28 | to active self-help (which is by its nature very concrete) makes a big 29 | difference. 30 | 31 | # Observations 32 | 33 | **What principles can we use to build cognitive tools?** 34 | 35 | **Cognitive tools often make easy cognitive tasks that were formerly 36 | hard:** It requires careful thought to understand why (and when) 37 | automating a formerly difficult task can make us smarter. Sometimes, 38 | solving a big problem, X, has a bottleneck, Y. Absent a way of 39 | avoiding the bottleneck entirely, we need to find a way of solving Y. 40 | 41 | **Principle: We could all do with having more cues in our life:** I 42 | wonder about having a unique buzzer for each habit I'd like cued? 43 | 44 | # What's going on when we multiply two numbers on paper? 45 | 46 | Most of us find it challenging to multiply two-digit numbers in our 47 | heads. And yet, with a paper and a pencil, it becomes almost trivial 48 | to do. 49 | 50 | This seems unsurprising, because it's so familiar. But it's really 51 | quite a remarkable phenomenon: someone hands you some tree pulp and a 52 | piece of graphite embedded in some wood. And this lets you trivially 53 | do tasks that were impossible before. What, exactly, is going on? 54 | 55 | At some level, it seems almost obvious: multiplying two numbers in our 56 | head taxes our short-term memory. We lose track of intermediate 57 | results, and so have to start the calculation over again. The use of 58 | pencil and paper gives us a way to overcome the limitations of our 59 | short-term memory. 60 | 61 | That's a nice explanation. But it's pretty vague. Let's see if we 62 | can better understand what's going on by working through a detailed 63 | example. Suppose we're trying to multiply 123 by 456. I'll work 64 | through the operations involved. Of course, there's nothing new here 65 | (!) --- I'm going through this simply to give us a concrete reference 66 | point to start thinking about what, exactly, is going on. Here's the 67 | example: 68 | 69 | > 123 70 | > x 456 71 | 72 | Using the grade school multiplication algorithm, I start by 73 | multiplying the rightmost digits of both numbers, i.e., 6 by 3. This 74 | gives me 18, so I write down 8, and "carry the 1": 75 | 76 | > 123 77 | > x 456 78 | > --- 79 | > 8 carry 1 80 | 81 | Incidentally, the way I was taught the multiplication algorithm in 82 | school, it was usual to write the carried digit as a superscript to 83 | the left of the 8. I haven't done that here, because it's 84 | inconvenient. I'll come back to this point a bit later. 85 | 86 | The next step of the procedure is to multiply the rightmost digit of 87 | the bottom number (6) by the second-to-right digit of the top number 88 | (2). That gives us 12, to which I add the carried result (1), to get 89 | a total of 13. So I write down 3, and again carry the 1: 90 | 91 | > 123 92 | > x 456 93 | > --- 94 | > 38 carry 1 95 | 96 | Then I multiply 6 by 1, and add the carried digit, to get 7, with 97 | nothing carried. 98 | 99 | > 123 100 | > x 456 101 | > --- 102 | > 738 103 | 104 | Next I do essentially the same procedure, but for the 5 in 456. It's 105 | very similar, with lots of single-digit multiplication, carries, and 106 | additions. The final result is: 107 | 108 | > 123 109 | > x 456 110 | > --- 111 | > 738 112 | > 615 113 | 114 | And then finally I do it for the 4 in 456: 115 | 116 | > 123 117 | > x 456 118 | > --- 119 | > 738 120 | > 615 121 | > 492 122 | 123 | Then we add the three rows below the "---". I won't explicitly write 124 | out all the operations involved --- they're along much the same lines 125 | as before. But we get a final result: 126 | 127 | > 123 128 | > x 456 129 | > --- 130 | > 738 131 | > 615 132 | > 492 133 | > --- 134 | > 56088 135 | 136 | What operations were involved? The most complicated arithmetic 137 | operations involved were just the multiplication or addition of two 138 | single-digit numbers. About the most complicated thing we did was the 139 | very first step: multiplying 6 by 3 to get 18. 140 | 141 | **Variations:** An interesting thing about the procedure is that there 142 | are some ambiguities in how operations are carried out. For exampe, 143 | in the second step of the procedure I multiplied 6 by 2 to get 12, and 144 | then added the carry (1), to get a total of 13. But I could equally 145 | well have started with the carry (1), and then added 6 by 2 (12), to 146 | get 13. A disadvantage of doing that is that it requires me to keep 147 | more information in working memory. Not a lot more, but enough to be 148 | a little taxing. An advantage is that I wouldn't have needed to write 149 | down the carry digit, since in this approach the only time I use the 150 | carry digit is immediately after learning its value. 151 | 152 | **Why is it easier to remember where things are stored than to 153 | remember what is stored?** The same information is involved. But I 154 | don't need to store it in working memory. It's on the paper. But 155 | there's a problem: how do I find it? Naively, it seems as though I'd 156 | have to remember the paper locations where the information is stored. 157 | Why isn't recalling those locations just as hard as remembering the 158 | information to be stored? In fact, when people are just learning to 159 | multiply for the first time, this really is a problem, and they get 160 | confused about where to look for the information they need. But 161 | experienced multipliers have committed an algorithm to long-term 162 | memory. That algorithm tells them how to organize the information on 163 | paper so that they can find the information they need, when they need 164 | it. In other words, *by committing an algorithm for organizing and 165 | finding information to long-term memory, we can outperform short-term 166 | memory alone*. 167 | 168 | **Reading, learning, and emotion:** Some of the power of reading is 169 | that it lets us learn from others, when we otherwise couldn't. We 170 | have a major cultural deficit, though, which is that conventionally 171 | we strip much of the emotional content. Scientific prose tends to 172 | be dry, not because it must, but because it is forced to be so by 173 | convention. It thus loses much of its power to teach. 174 | 175 | # What can we learn from brain research? 176 | 177 | **Look for bottlenecks in cognition:** What are our limits? Can we 178 | get around them technologically? 179 | 180 | **Are there ways we could overcome the well-known limits of working 181 | memory?** I wonder if there's a way of using visual cues to 182 | effectively outsource working memory? We already do this, for 183 | example, when we multiply two numbers on paper. Might there be a 184 | slick way we can do this more generally? Whiteboards are a way of 185 | doing this. 186 | 187 | **Why not ask some brain scientists what the limits and bottlenecks of 188 | our minds are?** My sense is that one thing we could do with a lot 189 | more of in our lives is the right cues. 190 | 191 | **Augmenting our senses:** Things like telescopes, microscopes, or 192 | infrared cameras are old examples of cognitive tools. 193 | 194 | **I don't understand what makes something memorable:** Why, when I 195 | watch a good movie trailer or listen to a great piece of music, do I 196 | remember it? What makes an explanation memorable? What makes 197 | something *vivid*? 198 | 199 | **The poverty of the verbal:** Many intellectuals love verbal content, 200 | but don't really quite get other types of content: movies, music 201 | videos (say), and so on. (Exceptions are made for "respectable" 202 | artforms, such as classical music, which have a strong intellectual 203 | verbal tradition associated to them.) I've talked to people whose only 204 | opinion about the movie _Avatar_ was that the plot and dialogue 205 | weren't very good. That was to miss the point. If verbal content is 206 | everything then that's a reasonable point of view. But _Avatar_ has 207 | astounding non-verbal content, and those people largely missed that 208 | content, or else got hung up on details. 209 | 210 | # Writing as a cognitive tool 211 | 212 | **What is it about writing that makes writing powerful as a tool for 213 | thought?** We may ask the prior question: is writing a powerful tool 214 | for thought? When we teach someone to write, does that make them 215 | smarter? The obvious answer, I think, is that it only makes them 216 | smarter if they _practice the skill_. Furthermore, writing _is_ a 217 | skill and it's possible to be better or worse at that skill. One way 218 | in which a writer can be skilled is that it makes them much smarter. 219 | 220 | **How can one use writing to make one smarter?** One way is through 221 | purely formal means. If someone asks me to square 123456 I can't 222 | easily do that in my head. But it would be straightforward to do on 223 | paper. 224 | 225 | **What makes reading a powerful tool for thought?** The most obvious 226 | reason is that it allows us to acquire information that would 227 | otherwise be hard to come by. There is, in principle, little that 228 | can't be done in conversation that can be done through print. But 229 | in practice we have limited conversational resources. Writing 230 | scales better than does access to conversation. The flipside to 231 | this is that conversation is in some ways a richer medium than 232 | writing, and as a result there are things we can learn easily in 233 | conversation 234 | 235 | **Reading can make us more creative:** This is a way in which reading 236 | can make us smarter which is not about the acquisition of 237 | information. And that's by stimulating. Indeed, it's possible to 238 | be good at one type of reading (say, creating new ideas), and not so 239 | good at another (say, acquiring information). 240 | 241 | **Is reading a more powerful tool for thought than writing?** 242 | 243 | **Cognitive tools make us think different, not necessarily smarter:** 244 | Every research scientist knows someone who spends too much time 245 | reading the past literature. It's a classic failure mode: the 246 | researcher whose first response to hearing of problem is to go and 247 | start looking up what's already been done, rather than beginning to 248 | think about the problem themself, and develp their own independent 249 | point of view. Of course, ultimately it's often helpful to do both, 250 | but it's rarely a good idea to leap straight into intensive reading. 251 | 252 | # List of cognitive tools 253 | 254 | **Cognitive tools to think about:** reading, writing, conversation, 255 | any algorithm, Mathematica, oblique strategies, the calendar, any 256 | method for monitoring oneself (e.g., lifelogging), money, attention 257 | managers, games, flashcards, libraries, programming languages, 258 | github, Google, table of contents, page numbers, cities. 259 | 260 | **Prompting or checklisting:** A simple, powerful pattern is to be led 261 | through a series of prompts or checklists. See, e.g., Gatawande's 262 | book on checklists for a good case that this can really improve 263 | performance. 264 | 265 | **Choice architecture:** See, e.g., Sunstei and Thaler's book _Nudge_ 266 | for discussion of how the way choices are presented to us can 267 | influence our cognitive capability. 268 | 269 | **Indices:** What makes indices a powerful tool for thought? They 270 | expand the range of information we can find. When I read a book I no 271 | longer need to memorize the details of the contents. Instead, I can 272 | memorize things at a coarser grain, knowing that I can go and look up 273 | details when I need. Basically, an index expands our access to 274 | knowledge. It effectively gives us a bigger database of facts to draw 275 | on. 276 | 277 | **What criteria should we judge a cognitive tool on?** That it expands 278 | the range of cognitive tasks that we can reasonably perform. 279 | 280 | **The pattern: Cognitive technologies break bottlenecks in our 281 | brain:** 282 | 283 | # Ideas about how to make people smarter and more effective 284 | 285 | **What are the best ideas we have about how to make people smarter and 286 | more effective?** I think there are two such ideas: 287 | 288 | 1. Desire: People need to be responsible for, and reap the benefits 289 | of, their own learning. If they are, they will take ferocious control 290 | over their own learning. A huge mistake made by the education system 291 | is to ignore this fact: most 14 year-olds in the classroom have no 292 | desperate personal need to learn, and there is only a little that even 293 | the best-intentioned teachers can do to change that. Indeed, it is a 294 | natural consequence of being in the classroom: the work done there 295 | matters only a little, and smart 14 year olds know it, and respond 296 | with an appropriate attitude. Unfortunately, someone without a strong 297 | desire to learn is probably not a very effective learner. Once 298 | someone develops a very strong desire to become more effective, then 299 | they begin to seek and benefit from learning opportunities at a 300 | tremendous rate. 301 | 302 | 2. Environment trumps everything except desire. Perhaps 100,000 303 | people lived in Florence in the mid 1400s. People born there (or in 304 | the countryside just outside) included Michaelangelo, Botticelli, and 305 | Donatello, as well as many other famous (but not quite as immortal) 306 | artists. Now, you start to do back-of-the-envelope calculations, and 307 | someone born in Florence at that time was perhaps 10,000 times more 308 | likely to be an immortal artist than an aspiring artist today. In 309 | other words, the best way to become a great artist wasn't to be born 310 | very talented, or be inspired by some special muse or whatever. It 311 | was to live in Florence in the late 1400s. That choice of environment 312 | completely swamped "natural talent" and just about every other 313 | variable I can think of, _except_ desire. 314 | 315 | I'd say these conditions are very close to necessary and sufficient 316 | conditions. Conditions 2 is not quite necessary --- one can think of 317 | examples where it has not been met. But as the Florentine example 318 | shows, it is tremendously important, probably far more important than 319 | we think. 320 | 321 | **Much of what we believe about learning is false:** This is a 322 | corollary to the above remarks. It follows from the observation 323 | that schools get something very basic wrong: they don't understand 324 | the primary nature of desire for learning, and how difficult it is 325 | to develop that desire when you treat children in the paternalistic 326 | manner that underlies schooling. This is an example of a false 327 | belief, in this case one that's predicated on a (very widespread) 328 | **false model** of how we learn. Another good example of how we use 329 | false models comes from Tim Gallwey's work on the inner game, 330 | showing how conventional models of coaching fail, since they 331 | actually create internal interference. A consequence of all this is 332 | that I should feel free to **re-examine the basics**, and to 333 | **identify (and then challenge) conventional wisdom**. 334 | 335 | # Dealing with abstraction 336 | 337 | **How to deal with abstraction?** _finding multiple representations_, 338 | _simplifying_, _approximating_, _finding the salient feature_. All 339 | these may be used in combination. One way of building software tools 340 | to help us think is to build software tools that make it easier to do 341 | these things. 342 | 343 | _Multiple representations:_ It helps us dramatically when we can find 344 | ways of representing a new abstraction in terms of something else that 345 | we already have ways of working with. Think of the way we can 346 | understand Hermitian matrices in terms of their eigenvalue 347 | decomposition, for example. 348 | 349 | _Simplifying the abstraction_: Consider the way we can simplify a 350 | topological space down to a number, by counting the number of 351 | connected components. This provides information that can, for 352 | example, be used to prove that two topological spaces are not 353 | homeomorphic. 354 | 355 | _Approximating the abstraction:_ A cow is not perfectly spherical, but 356 | to a good approximation... To work, we need to develop a good 357 | understanding of how to control the impact of the approximation. 358 | 359 | _Finding the salient feature:_ This is often the key to solving 360 | problems. 361 | 362 | # Games 363 | 364 | **What can we learn from games?** 365 | 366 | **Why are so many educational games terrible?** I think the problem is 367 | that many of the theories from the educational literature aren't 368 | very good. 369 | 370 | 371 | **What makes games such a great tool for thought?** 372 | [Watch this guy play Tetris](http://www.youtube.com/watch?feature=player_detailpage&v=jwC544Z37qo#t=171s). 373 | I won't spoil the surprise, but the last minute or so is almost 374 | beyond belief. 375 | 376 | **Are games the only way?** Marshall McLuhan once commented that 377 | "Anyone who tries to make a distinction between education and 378 | entertainment doesn't know the first thing about either." 379 | 380 | 381 | **People get amazingly good at games, very, very quickly. Why is 382 | this? Why do they get good so much faster than they do with other 383 | kinds of learning?** 384 | 385 | **A related question: why do people remember so much more from movie 386 | trailers than they do from 2-3 minutes of other material?** 387 | 388 | **Why do we easily tolerate repetition of some material but not of 389 | others?** 390 | 391 | **How much does it help to add sound?** I'll bet it helps a lot. 392 | 393 | # Active reading and explorable explanations 394 | 395 | This is an idea of Bret Victor's. The idea is to make documents 396 | interactive so that the read can easily modify elements of the 397 | document and, in real time, other elements will change in response. 398 | It's a way of exploring what's going on. 399 | 400 | **Reactive documents:** A synthesis of spreadsheet and document. 401 | Elements of the document can be modified, which in turn updates 402 | other parts of the document. Victor has a Javascript library, 403 | [tangle](https://github.com/worrydream/Tangle), which can be used to 404 | author reactive documents. I could potentially write a reactive 405 | document for my blog. Here's a 406 | [WordPress page](http://codex.wordpress.org/Using_Javascript) about 407 | how to include Javascript in just a single post. 408 | 409 | **The advantage over spreadsheets:** Victor makes the point that while 410 | spreadsheets are great, they don't explain the model that they 411 | implement. Reactive documents could _explain_ the model and 412 | simultaneously allow people to _explore_ the model. Here's Victor: 413 | 414 | > "a spreadsheet is not an explanation. It is merely a dataset and 415 | > model; it cannot be read. An explanation requires an author, to 416 | > interpret the results of the model, and present them to the reader 417 | > via language and graphics. 418 | 419 | **Explorable explanations enable discovery:** You can play with the 420 | parameters, and by looking at the consequences you start to have 421 | suspicions about what is true, and may even be able to empirically 422 | establish some of your suspicions. 423 | 424 | **Explorable explanations stimulate questions** 425 | 426 | **Interactivity is not the point; combining exploration and 427 | explanation is the point:** Here's Victor on this: 428 | 429 | > It's tempting to be impressed by the novelty of an interactive 430 | > widget such as this, but the interactivity itself is not really the 431 | > point. The primary point of this example -- the reason I call it an 432 | > "explorable explanation" -- is the subtlety with which the 433 | > explorable is integrated with the explanation. 434 | > 435 | > Like the proposition example earlier, the filter description works 436 | > as a static explanation -- it can be read like normal text. The 437 | > reader is not forced to interact in order to learn. The reader 438 | > interacts if he wants to go deeper, if he has piqued curiosity or 439 | > unanswered questions. There are no UI elements screaming for 440 | > attention. The reader is not transported off to a separate 441 | > "interactive" context. Instead, the reader simply nudges the 442 | > examples that the author has already presented. 443 | > 444 | > Most interactive widgets dump the user in a sandbox and say "figure 445 | > it out for yourself". Those are not explanations. To me, an 446 | > essential aspect of the "explorable explanation" concept is that the 447 | > author holds up his end of the conversation. The author must guide 448 | > the reader, and provide a structure for the learning 449 | > experience. Only then can the reader respond, by asking and 450 | > answering the questions that the author provokes. 451 | 452 | I guess people often draw a distinction between a model we can play 453 | with (like, for example, a spreadsheet), and an explanation. Victor's 454 | key idea here is to _integrate_ the two, and that we gain something by 455 | doing that integration. 456 | 457 | **What happens when we step outside the model?** This is 458 | 459 | **Use cases:** With something like this I think it's a mistake to ask 460 | the question "So what could it be used for?" Instead, it'd be 461 | better to simply try out lots of use cases, and see what works well, 462 | and what doesn't. 463 | 464 | **Building a simplifed version of tangle:** I took a look at the 465 | library. It's very nice. However, the API is pretty complicated. 466 | It'd be nice to have a templating system that made it a lot easier 467 | to use tangle. Maybe a Wordpress plugin that made it trivially 468 | possible to compile to tangle? 469 | 470 | **What's a neat reactive document that I could write?** Maybe one that 471 | showed off the ideas of relativity? Maybe one that did some kind of 472 | scenario planning along the lines championed by the futurists? 473 | Maybe I could include it in my Clock of the Long Now essay, for the 474 | analysis of the relativistic clock? 475 | 476 | **Explorable examples:** Victor gives a very beautiful example of a 477 | digital filter where you can literally move the parameters of the 478 | filter around, and dynamically see how the frequency response 479 | changes. 480 | 481 | **Could we combine tangle with mathjax?** That'd be a fun project! 482 | 483 | **Text as an environment to think in:** Here's Victor: 484 | 485 | > The goal of Explorable Explanations is to change people's 486 | > relationship with text. People currently think of text as 487 | > information to be consumed. I want text to be an environment to 488 | > think in. 489 | 490 | **Scientific communication as sequential art:** Another of 491 | [Bret Victor's ideas](http://worrydream.com/#!/ScientificCommunicationAsSequentialArt). 492 | I was struck by the following, which seems to sum up much of Victor's 493 | philosophy: 494 | 495 | > When an algorithm is described in prose (or code), we are typically 496 | > given only the rules of the system — we can't see the data or the 497 | > state. In order to understand what the algorithm is doing, we have 498 | > to "play computer" and imagine the state in our head. Illustrating 499 | > the state of an algorithm at each step can make the description 500 | > dramatically easier to follow. 501 | 502 | Furthermore, Victor does this in such a way that's not merely an 503 | illustration, but there is actually 504 | 505 | **Core idea: using multiple representations of concepts, and coupling 506 | those representations, so we can interact with them in different 507 | ways:** We could imagine doing this to illustrate something like the 508 | eigenvalues of a Hermitian matrix. We could have two coupled 509 | descriptions: 510 | 511 | matrix <=> eigenvalues and eigenvectors 512 | 513 | We could interact with either description, and see how it affects 514 | the other. 515 | 516 | **Scientific communication and comics:** More from Victor: 517 | 518 | > Finally, "comic-style" needn't mean characters, dialog, word 519 | > balloons, or sound effects. Let alone superheroes or anthropomorphic 520 | > animals. The comic form is about sequences of tightly-integrated 521 | > words and pictures, together conveying a message more powerfully 522 | > than the sum of their parts. The potential of this form to explain 523 | > difficult concepts is unmatched and underused. 524 | 525 | I am strikingly reminded of Manny Knill's idea of writing a paper that 526 | would just be figures to illustrate the Solovay-Kitaev theorem. 527 | Essentially Manny was proposing exactly what Victor is suggesting 528 | above. 529 | 530 | # Source material 531 | 532 | **Who is working on this?** Many people and organizations are working 533 | on this problem. Looked at a certain way, for example, it's a 534 | problem that many software companies are working on. But often they 535 | don't _think_ they're working on this. They may be primarily 536 | focused on trying to turn a profit. Or if they do think they're 537 | working on this, it's a secondary motivation. Motivation matters. 538 | 539 | **Bret Victor:** One of the people who is most consciously working on 540 | this problem is [Bret Victor](). 541 | 542 | **Bret Victor's principle:** Very roughly speaking, it's to make the 543 | output of code fully visible, in real time. What makes this such a 544 | powerful principle is that it means that we can take something 545 | abstract (a program), and then give it another representation (the 546 | visualisation), and then flip back and forth between the two. 547 | 548 | **Is there a good way of building Victor's visual representation 549 | easily into the program?** 550 | 551 | **Get Victor's principle more correct:** 552 | 553 | **Explore the links on Victor's page as stimulus:** 554 | 555 | # The current status of work on cognitive tools 556 | 557 | Much of the best work in the early days of computing was aimed at 558 | developing cognitive tools. Pioneers such as Douglas Engelbart, Alan 559 | Kay, and Ted Nelson all struggled, in different ways, with the 560 | question: "How can we build better cognitive tools?" For reasons I 561 | don't entirely understand, today few people are squarely addressing 562 | this question. There _is_ good, practical work coming from the many 563 | companies and organizations which are creating products that change 564 | how we think. But that work is oriented toward the development of 565 | monetizable products, not directly about developing a deep general 566 | understanding of cognitive tools. The academic work on the subject 567 | seems, with some notable exceptions, dull and uninspired. 568 | 569 | I find this strange. Many of humanity's greatest inventions are 570 | cognitive tools --- language, the alphabet, mathematics. These tools 571 | enable our minds to pull themselves up by their mental bootstraps. 572 | Tools such as word processors and visualization software push this 573 | tradition further. It's easy to be snide and remark that _Microsoft 574 | Word_ doesn't seem like _quite_ as much of a jump as the invention of 575 | the alphabet. But the invention of the alphabet didn't happen all at 576 | once. It took thousands of years to develop all the ideas used in 577 | modern alphabets. We've had word processors for just a few decades. 578 | 579 | # Project ideas 580 | 581 | **Attention manager:** 582 | 583 | **Essay on how online tools can amplify our collective intelligence:** 584 | 585 | # Programming languages 586 | 587 | **Programming languages are tools for thought** 588 | 589 | **Smalltalk arose from a vision of a programming language as a 590 | learning environment** 591 | 592 | **[Alan Kay on Smalltalk's design](http://www.smalltalk.org/smalltalk/TheEarlyHistoryOfSmalltalk_Introduction.html):** 593 | 594 | > Smalltalk's design--and existence--is due to the insight that 595 | > everything we can describe can be represented by the recursive 596 | > composition of a single kind of behavioral building block that hides 597 | > its combination of state and process inside itself and can be dealt 598 | > with only through the exchange of messages. 599 | 600 | In other words, objects can be decomposed into smaller objects, and so 601 | on, _ad infinitum_. This is taking a metaphor from physics, and 602 | turning it into a way of describing active knowledge. 603 | 604 | **What would force-based programming look like?** Could we use 605 | potentials? What about Einstein's idea in GR? Prolog already uses 606 | something like a Lagrangian point of view. 607 | 608 | **How important are programming languages as cognitive tools?** This 609 | seems like a key question. What is maybe interesting about 610 | programming languages (and, to some extent, media more generally) is 611 | that they can be used to create new types of cognitive tool. In 612 | some sense they're a meta-cognitive tool. And so as you master 613 | programming you can start to build more and more powerful cognitive 614 | tools for yourself, and for others. 615 | 616 | 617 | # All Our Knowledge 618 | 619 | ## Writing 620 | 621 | **Writing principle:** It should make ideas vivid. It should be 622 | coherent. 623 | 624 | ## Community and collaboration 625 | 626 | **It's more important to be polite than to be right:** 627 | 628 | 629 | # Other material 630 | 631 | **Task management:** One of the hardest things we can do. 632 | 633 | **What should I do next?** 634 | 635 | **When do cognitive tools help? When do they hinder? When does 636 | access to the net make us smarter? When does it make us stupider? 637 | What other effects does it have?** 638 | 639 | **What is a cognitive tool?** 640 | 641 | **Something that does for habit what Google does for factual memory** 642 | 643 | **The intangibility of cognitive tools:** I'm having a strange feeling 644 | as I grapple with these notions of cognitive tools. It's that I 645 | can't fully grasp what these tools actually do for us. They have an 646 | intangible quality. Think about how strange it is to not have the 647 | notion of zero, or of negative numbers. It's easy to overlook these 648 | notions. 649 | 650 | **Institutions are some of our best cognitive tools:** SOmetimes 651 | unintentionally, too. People think "Oh, Steve Jobs must be so 652 | smart." But much of his smartness was a _consequence_ of being in 653 | charge of Apple, not the other way round, which is how people tend 654 | to think. 655 | 656 | **Why are all these projects of Victor's just at the prototype 657 | stage?** I think the answer is that there's really two very hard 658 | tasks here. One is developing interesting prototypes (which Victor 659 | has done, as PARC did, and so on). The other is developing those 660 | prototypes into mass market products. Both tasks are tremendously 661 | difficult to do. 662 | 663 | **What would it mean to make mathematics executable?** 664 | 665 | **"The interface fundamentally determines behaviour":** 666 | ([Rory Sutherland](http://www.ted.com/talks/rory_sutherland_life_lessons_from_an_ad_man.html) 667 | This is another perspective on the idea that "the medium is the 668 | message". 669 | 670 | **What can we learn from?** 671 | 672 | **Who is working on this? Which aspects are there too many people 673 | working on? Which aspects are there too few people working on? What 674 | are some ideas I can bring in from left field?** MIT Media Lab. 675 | 676 | **Corporate environments versus academia:** It's interesting to think 677 | about the environments in which one can do this kind of work. 678 | Corporations are usually design to take prototypes and turn them 679 | into products. Academia funnels people toward a few areas deemed 680 | fashionable (often for spurious reasons), and draws the 681 | highest-expertise people out of work at the coalface. This is not a 682 | recipe for success. 683 | 684 | **Why aren't corporations good at the full prototype-product 685 | pipeline?** 686 | 687 | **What is an explanation?** 688 | 689 | **What is a document?** 690 | 691 | # Cognitive tools and super-realist art 692 | 693 | _This material is adapted from an essay I wrote about super-realist 694 | art_ 695 | 696 | Let me show you an example where super-realist art is used to change 697 | how people think: 698 | 699 | 700 | 701 | Listen to the audience cheer and whoop at 5:02. The presenter, Hans 702 | Rosling, has taken four decades of United Nations' data about family 703 | size and life expectancy in different countries, and animated the data 704 | so we can see it change over time. Sounds simple enough. But by 705 | presenting the data in this way, Rosling enables the audience to make 706 | a startling discovery. Most people in the audience, I expect, grew up 707 | believing that there is a sharp distinction between rich and poor 708 | countries, that people in rich countries live much longer, and have 709 | much smaller families. Certainly, I was brought up with this story. 710 | What the video reveals is that while this story was true in the 1960s 711 | and 1970s, it is no longer true. The world has changed dramatically, 712 | and we are challenged to rethink what we believe. Even if we suspect 713 | the video is misleading, the onus is on us to explain why it's wrong, 714 | and what evidence we have for an alternate view of the world. 715 | 716 | Rosling's video is super-realist art, revealing a previously hidden 717 | world. But it also changes how we think, and so it's also a cognitive 718 | tool. You may perhaps worry about whether it's the video itself which 719 | is a cognitive tool, or is the cognitive tool really the software 720 | which Rosling used to analyse the data and create the video? In fact, 721 | the distinction is more subtle than it first appears. The video was 722 | perhaps constructed through dozens of intermediate iterations, 723 | experimenting with many different ways of representing the data. Each 724 | of those iterations informed the thinking that went . And so for the 725 | most part I won't try to make clear separation between cognitive tools 726 | and their outputs. 727 | 728 | Cognitive tools and super-realist art are closely related. They're 729 | not the same --- a calculator is a cognitive tool, but it would be a 730 | most unusual calculator that could be considered (or which produced) 731 | super-realist art. However, consider examples of super-realist art 732 | such as the light-in-a-bottle, _Water in Suspense_, and the Hubble 733 | Extreme Deep Field. Each vividly represents new aspects of reality. 734 | Each suggests many questions. Personally, I've found that they've 735 | each changed how I think about light, about water, and about the 736 | Universe. And so these pieces of super-realist art are also cognitive 737 | tools. 738 | 739 | This is not to say that the _point_ of super-realist art is to create 740 | cognitive tools. A Hollywood movie mogul of old --- some say it was 741 | Samuel Goldwyn --- supposedly rejected a moralizing movie script with 742 | the words: "If you want to send a message, send a telegram". Taking a 743 | similar stance, artists such as Juan Geuer often create art that is 744 | principally for art's sake. The purpose of Geuer's _Water in 745 | Suspense_ is not to provide us with a new tool for thought. Rather, 746 | it is to reveal an unsuspected beauty inside a drop of water. It is 747 | fortuitous that it also stimulates new ways of thinking about water, 748 | but it is not the point. By contrast, Rosling's software is primarily 749 | a cognitive tool, enabling new ways of thinking. It uses beauty as a 750 | vehicle, and so it also produces super-realist art, but this is not 751 | its primary purpose. And so while super-realist art and cognitive 752 | tools overlap, we should analyse them on separate grounds. 753 | 754 | I find the creation of new and better cognitive tools inspiring. It's 755 | something I'd like to contribute to. And so I will conclude this 756 | essay with two heuristics which both help make better cognitive tools 757 | _and_ making better super-realist art. 758 | 759 | **Radical concreteness:** The first concerns a heuristic I call 760 | _radical concreteness_. To explain radical concreteness, let me 761 | describe another cognitive tool and super-realist art piece, the 762 | computer game [http://braid-game.com/](_Braid_). One of the 763 | best-selling computer games of 2008, _Braid_ appears at first glance 764 | to be a standard platform game. It uses many common elements of the 765 | platform genre: there are puzzles to be solved, and six worlds to 766 | explore. But _Braid_ has a beautiful twist. Each world gives the 767 | player the ability to manipulate time in a different way. It might be 768 | a simple ability to rewind time. Or perhaps to slow time down. Or 769 | some more complex manipulation. Furthermore, the puzzles are 770 | ingeniously designed so that their solution requires you to manipulate 771 | time in just the right way. 772 | 773 | The psychological effect of playing _Braid_ is remarkable. My 774 | professional background is as a theoretical physicist, and like many 775 | theoretical physicists I've thought about different models of time. 776 | But _Braid_ pushed my thinking further. It immerses you in concrete, 777 | vividly rendered models of how time can be altered. And success in 778 | the game demands that you internalize those models. I found myself 779 | _dreaming_ about new models of time. Braid was a cognitive tool to 780 | help thinking about time. 781 | 782 | What makes Rosling's animation and _Braid_ so powerful is that they 783 | make vivid and concrete what would otherwise be vague and abstract. 784 | It's tempting to phrase this as a heuristic: "develop representations 785 | which are as vivid and concrete as possible". The problem with this 786 | heuristic is that it's not strong enough. It's an easily-ignored 787 | platitude. What examples like Rosling's statistics and _Braid_ really 788 | show is that we routinely and dramatically underestimate how vivid and 789 | how concrete representations can be made. We look at the UN 790 | statistics, and see only a table, or perhaps a few graphs. We don't 791 | challenge ourselves to find a way of showing the data that would cause 792 | an auditorium full of people to erupt in wild cheering. Or we think 793 | about models of time, and play with them mathematically, but never 794 | imagine that if put into the right form we'd start to see them in our 795 | dreams. It's a failure of the imagination. And so we can say, with 796 | only a little dramatic license: _you can **always** find a way to make 797 | your representation more vivid and more concrete_. This is a doctrine 798 | of _radical concreteness_. 799 | 800 | Let me give you an example where I think this idea of radical 801 | concreteness could be pushed much further. In October of 2012, a team 802 | of researchers at MIT announced that they've developed a 803 | [game](http://gamelab.mit.edu/games/a-slower-speed-of-light/) which 804 | show what it's like to live in a world where the effects of relativity 805 | are obvious. The idea is to reduce the speed of light to the kind of 806 | velocity we're familiar with in the everyday world, so that it's 807 | possible to see effects such as time dilation, length contraction, the 808 | Doppler effect, and so on. By playing the game, you begin to build up 809 | intuition about what it's like to live in a relatavistic world. 810 | 811 | I won't show you the game trailer (see the game link above). Frankly, 812 | the trailer doesn't make it so easy to see the relativistic effects. 813 | Instead, I'll show you an older visualization developed by XXX and 814 | XXX. It shows what it's would be like to drive down a highway, if the 815 | speed of light was much slower. The video is a bit fuzzy, but stick 816 | with it: 817 | 818 | 819 | 820 | In worlds like these you can actually *experience* what it's like for 821 | time to dilate, for space to contract, and so on. And it's easy to 822 | imagine games which, like _Braid_, would push this idea a long way. 823 | 824 | We can go further. In 1994, a scientist named 825 | [Miguel Alcubierre](http://en.wikipedia.org/wiki/Miguel_Alcubierre) 826 | [announced](http://arxiv.org/abs/gr-qc/0009013) to the world that he'd 827 | discovered a _warp drive_. Now, of course, you've no doubt heard that 828 | it's impossible for objects to travel faster than light. This is 829 | based on a result of special relativity, which shows that 830 | faster-than-light objects could be used to violate causality. 831 | However, special relativity is really an approximation to a deeper 832 | theory --- general relativity. Alcubierre was a specialist in 833 | studying simulations of general relativity, and he'd built up a lot of 834 | intuition about how spacetime behaves. He used that intuition to find 835 | a solution to general relativity with the remarkable property that an 836 | object could move faster than light (when viewed in the usual frames 837 | of reference), without any violation of causality occurring. 838 | 839 | I won't try to describe. But after I'd read about both Alcubierre's 840 | drive, and XXX and XXX's simulations of relativity, it started me 841 | wondering. We've seen lots of games that let people design levels or 842 | design dungeons. Why not a game that lets people design their own 843 | spacetimes? It'd be challenging to make the game fun --- there's no 844 | doubt players would need warming up. But I'll bet it could be done! 845 | After all, one of the most popular games of all time was about time 846 | travel. 847 | 848 | 849 | **Multiple representations:** One of the people who is think hard 850 | about cognitive tools is Bret Victor. In a remarkable presentation 851 | made in 2012, he develops a single, simple principle: it should XXXX. 852 | This sounds simple, but has many consequences. Take a look at the 853 | following video from XXX to XXX where he shows what this principle 854 | 855 | XXX 856 | 857 | Like any powerful principle, Victor's principle is generative. It can 858 | be applied to a large number of different . He shows how it can be 859 | applied to XXX. To XXX. And it's easy to think of. I imagine myself 860 | as CEO of PIxar, and asking what would I do? An obvious application 861 | is to wonder: should we buy ourselves enough computing power that 862 | artists can see their designs rendered in near real-time? Yes, it 863 | would cost a lot --- but it would also totally transform the artist's 864 | work experience. Would it be worth it? I don't know. But it's 865 | 866 | I'd like to explore beyond Victor's principle. Lots of things have 867 | multiple representations. And we get power by understanding how those 868 | representations relate. So you can generalize Victor's principle: 869 | 870 | > 871 | 872 | 873 | ## Problem: understanding complex tools: 874 | 875 | How should we understand a model with a trillion parameters? We 876 | *have* these kinds of models. Google's translation service uses 877 | incredibly complicated models of human language. In some ways Google 878 | knows more about human language than any one person. But how can we 879 | make sense of the model? And we're getting lots of such models. My 880 | guess is that we're ultimately going to end with mighty towers of 881 | tools: trillion parameter models which will get re-understood as 882 | million-parameter models, which can then be understood in sixteen 883 | hundred different ways. 884 | 885 | 886 | My instinct here is right. What do we currently have trouble with 887 | doing? Where are the current limits where we'd like to pull our 888 | cognition futher? 889 | 890 | 891 | ## Notes on relevant papers 892 | 893 | David H. Jonassen and Thomas C. Reeves, "Learning With Technology: 894 | Using Computers as Cognitive Tools": Alternate terms include cognitive 895 | technologies, and technologies of mind. What's the difference between 896 | this point of view and computer software as media? It's not so clear 897 | to me that there is one, except in the sense that tools aim to make 898 | certain tasks easy that were formerly hard, while media aims to do 899 | something entirely different (communication). This paper is obviously 900 | within the area of educational research. One perspective that seems 901 | really badly lacking from this area: students as independent entities 902 | with dreams and goals of their own. If they don't align with the 903 | tools, then there's no way the tools can matter. Places a lot of 904 | emphasis on constructivist versus instructivist models of learning; I 905 | think desire-based learning is much more important. A curious fact: 906 | it's not clear from the paper what success would look like. A very 907 | interesting obervation: tools most effective when they involve us in 908 | designing our own representation of knowledge. 909 | 910 | Wouter van Joolingen, "Cognitive tools for discovery learning" (1999). 911 | Another educational article. 912 | 913 | ## To read 914 | 915 | Papert 916 | 917 | # Essay 918 | 919 | Suppose someone asks us to multiply 73 by 36 in our head. If we're an 920 | ordinary person, chances are that it'll be a struggle. But, if we're 921 | given a pencil and paper, all of a sudden we can do it easily. 922 | 923 | Familiarity makes us take this for granted. But if you were viewing 924 | it from the outside, for the first time, it would appear quite 925 | remarkable, as though the pencil and paper had somehow granted the 926 | person doing the multiplication magical powers. 927 | 928 | It's not just the pencil and paper which make it easier. If you give 929 | the pencil and paper to an untrained person, chances are that it will 930 | help only a little with the multiplication. 931 | 932 | No, what makes it easier is pencil, paper, and an \emph{idea} --- the 933 | multiplication process which most of us learn in school. 934 | 935 | In fact, there are many variations on the multiplication process. And 936 | some variations make the multiplication easier, while others make it 937 | harder. In other words, how much easier multiplication becomes 938 | depends on the quality of the idea. 939 | 940 | The question I want to address: what makes a good cognitive tool? 941 | 942 | *A priori* we don't have a good instintive feeling for what makes a 943 | good cognitive tool. 944 | 945 | 946 | What we'd really like is two things: (1) a *predictive theory* which 947 | tells us what effect a particular cognitive tool will have; and (2) 948 | design principles that make it easy to find cognitive tools which have 949 | particular desired impacts. 950 | 951 | We've made a start on these things with (on the academic side) the 952 | study of human-comuter interaction and computer-supported co-operative 953 | work, and (on the commercial side) design and marketing. But we a 954 | long way from a truly predictive theory. 955 | 956 | Here's the 957 | example: 958 | 959 | > 123 960 | > x 456 961 | 962 | Using the grade school multiplication algorithm, I start by 963 | multiplying the rightmost digits of both numbers, i.e., 6 by 3. This 964 | gives me 18, so I write down 8, and "carry the 1": 965 | 966 | > 123 967 | > x 456 968 | > --- 969 | > 8 carry 1 970 | 971 | Incidentally, the way I was taught the multiplication algorithm in 972 | school, it was usual to write the carried digit as a superscript to 973 | the left of the 8. I haven't done that here, because it's 974 | inconvenient. I'll come back to this point a bit later. 975 | 976 | The next step of the procedure is to multiply the rightmost digit of 977 | the bottom number (6) by the second-to-right digit of the top number 978 | (2). That gives us 12, to which I add the carried result (1), to get 979 | a total of 13. So I write down 3, and again carry the 1: 980 | 981 | > 123 982 | > x 456 983 | > --- 984 | > 38 carry 1 985 | 986 | Then I multiply 6 by 1, and add the carried digit, to get 7, with 987 | nothing carried. 988 | 989 | > 123 990 | > x 456 991 | > --- 992 | > 738 993 | 994 | Next I do essentially the same procedure, but for the 5 in 456. It's 995 | very similar, with lots of single-digit multiplication, carries, and 996 | additions. The final result is: 997 | 998 | > 123 999 | > x 456 1000 | > --- 1001 | > 738 1002 | > 615 1003 | 1004 | And then finally I do it for the 4 in 456: 1005 | 1006 | > 123 1007 | > x 456 1008 | > --- 1009 | > 738 1010 | > 615 1011 | > 492 1012 | 1013 | Then we add the three rows below the "---". I won't explicitly write 1014 | out all the operations involved --- they're along much the same lines 1015 | as before. But we get a final result: 1016 | 1017 | > 123 1018 | > x 456 1019 | > --- 1020 | > 738 1021 | > 615 1022 | > 492 1023 | > --- 1024 | > 56088 1025 | 1026 | --------------------------------------------------------------------------------