└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # Towards a pattern language for visualizing AI 2 | 3 | This is an attempt to start cataloging design patterns for visualizing AI. These patterns are introduced in [these talk slides](https://docs.google.com/presentation/d/1h8pubQ4v8cZelU72wvAjvg4566rxa0p93U6eQ9x_OmU/edit#slide=id.p) with screenshots and GIFs. The goal of this document is to add to the language and enrich the patterns with more detailed information and examples. 4 | 5 | Help us collect good examples of these patterns and sharpen their definitions. Feel free to open an issue or pull request! 6 | 7 | ## Why a pattern language? 8 | TODO summarize introduction from talk on design & development challenges 9 | 10 | ## Content 11 | These patterns are concerned with representing concepts that show up repeatedly in AI. 12 | 13 | ### Architecture 14 | TODO 15 | 16 | ### Input data 17 | **Show the input to the model.** 18 | It is often helpful to show example inputs to give a sense for the models behavior. 19 | It is also important for the reader to understand what the input looks like, and any constraints on the data format or validity. 20 | 21 | Examples: 22 | * https://pair.withgoogle.com/explorables/fill-in-the-blank/ 23 | * https://observablehq.com/@krisrs1128/remembrances-of-states-past 24 | * https://anomagram.fastforwardlabs.com/#/ 25 | 26 | ### Activations 27 | **The outputs of the activation function for each neuron.** 28 | TODO further describe & add more examples 29 | 30 | Examples: 31 | * https://distill.pub/2018/building-blocks/ 32 | * https://thilospinner.com/towards-an-interpretable-latent-space/ 33 | 34 | ### Weights 35 | **The strength of connections between neurons.** 36 | TODO further describe & add more examples 37 | 38 | Examples: 39 | * https://distill.pub/2020/circuits/visualizing-weights/ 40 | 41 | ### Attention 42 | **It's all you need.** 43 | TODO further describe & add more examples 44 | 45 | Examples: 46 | * https://distill.pub/2016/augmented-rnns/ 47 | 48 | 49 | ### Parameter spaces 50 | **Show how parameters change together.** 51 | We rarely are interested in a single parameter in isolation. When possible allow exploration of the possibility space created by multiple parameters. 52 | 53 | Examples: 54 | * https://distill.pub/2017/momentum/ 55 | * https://jgoertler.com/visual-exploration-gaussian-processes/ 56 | 57 | ### Graphs 58 | **Nodes and links, objects and connections, entities and relationships.** 59 | Examples: 60 | * https://distill.pub/2021/gnn-intro/ 61 | * http://www.shawnfarris.io/research/graphicalDataRepresentations/ 62 | 63 | 64 | ## Context 65 | These patterns are concerned with clarifying our communication. Keep in mind that all of the patterns in this category work just as well in static media like PDFs. It is still important to call them out explicitly because when they are combined with the patterns in the Choice categories it opens up a whole new world of possibilities. 66 | 67 | ### Annotation 68 | **Point out important elements in figures with textual annotations.** 69 | When making new representations via data visualization help the reader orient by pointing out examples of patterns and important elements. 70 | 71 | Examples: 72 | * https://pair.withgoogle.com/explorables/fill-in-the-blank/ 73 | * https://bert-vs-gpt2.dbvis.de/ 74 | 75 | ### Text integration 76 | **Connect written words with figures directly via color, location and hyperlinks.** 77 | When describing concepts in text, link their representations visually via thoughtful layout and consistent use of color. When describing interactions, hyperlink the text to the figure, making a linked view. 78 | 79 | Examples: 80 | * https://distill.pub/2017/momentum/ 81 | * https://introduction-to-autoencoders.vercel.app/ 82 | * https://nipunbatra.github.io/hmm/ 83 | * https://observablehq.com/@clpuc/analyzing-the-design-space-for-visualizing-neural-attenti 84 | * https://observablehq.com/@observablehq/text-color-annotations-in-markdown 85 | * https://theo-jaunet.github.io/MemoryReduction/ 86 | 87 | ### Aggregation 88 | **Aggregate sparingly.** 89 | The nature of AI work involves large quantities of data and high dimensions. The most practiced method of reducing complexity is via aggregation. Aggregation destroys context quickly by hiding actual data behind invisible computation. 90 | When possible, link views between aggregations and underlying data points. 91 | 92 | Examples: 93 | * https://distill.pub/2020/circuits/visualizing-weights/ 94 | * https://distill.pub/2018/building-blocks/ 95 | 96 | ### Dimensionality reduction 97 | **Reduce dimensions carefully and consistently.** 98 | Dimensionality reduction can be a powerful way to make high dimensional data accessible. 99 | Use with caution, combining with other patterns can help with clarity. 100 | 101 | Examples: 102 | * https://distill.pub/2016/misread-tsne/ 103 | * https://tiga1231.github.io/umap-tour/ 104 | * https://distill.pub/2020/grand-tour/ 105 | 106 | ### Small multiples 107 | **Show many facets of the same data with small multiples.** 108 | A common technique in static figures, small multiples can be even more powerful with linked views. 109 | 110 | Examples: 111 | * https://distill.pub/2020/grand-tour/ 112 | 113 | ### Slow build 114 | **Build up context slowly, introducing concepts and interactions piece by piece.** 115 | Complex games, simulations and interactive visualizations can be overwhelming without proper orientation. 116 | 117 | Examples: 118 | * https://distill.pub/2021/gnn-intro/ 119 | * https://aatishb.com/entropy/ 120 | * https://theo-jaunet.github.io/theo-guesser/ 121 | * https://ncase.me/polygons/ 122 | * https://nan.fyi/how-arrays-work 123 | 124 | 125 | 126 | ## Choice 127 | The differentiating patterns of dynamic media are about giving readers a choice. 128 | 129 | ### Examples 130 | **Allow readers to choose from examples.** 131 | Letting the reader compare and contrast the differences between examples allows them to build their own intuition for how those differences come about. 132 | Use representations and affordances to give a preview of what the user can expect when choosing on an example. 133 | 134 | Examples: 135 | * https://theo-jaunet.github.io/theo-guesser/ 136 | * https://distill.pub/2018/building-blocks/ 137 | 138 | ### Games 139 | **Build intuition by framing choices as a game.** 140 | Games can be a powerful way to form intuition of a system. 141 | By encouraging readers to step into the shoes of the algorithm they can better understand the power and the challenges behind it. 142 | 143 | Examples: 144 | * https://www.pewresearch.org/interactives/how-does-a-computer-see-gender/ 145 | * https://interactive-maml.github.io/ 146 | * https://peopleofthepandemicgame.com/ 147 | 148 | ### Playback 149 | **Allow the reader to control playback when many steps are involved.** 150 | Animating or simulating a complex series of steps can provide a powerful intuition of how a system changes. 151 | Give users control over playback with familiar inputs and affordances. 152 | 153 | Examples: 154 | * stepping: https://distill.pub/2020/grand-tour/ 155 | * scrollytelling: [Beginners guide to dimensionality reduction](https://visxai-dimensionality-reduction-1dbad0a67a092b007c526a45.vercel.app/) 156 | 157 | ### Affordances 158 | **Let readers know where they can interact.** 159 | Many representations and choices are new to readers of dynamic media. It can be helpful to indicate where and how a reader can interact with the article. 160 | Common affordances include signage, wayfinding, demonstrations, hover effects and scrollytelling. 161 | 162 | Examples: 163 | * https://distill.pub/2017/momentum/ 164 | 165 | ### Linked views 166 | **Setup interactions with one figure to change the contents of another figure.** 167 | A powerful way of demonstrating the impacts of certain choices is by linking them to 168 | Linked views are often used with small multiples, dimensionality reduction and aggregations. 169 | 170 | Examples: 171 | * https://distill.pub/2018/building-blocks/ 172 | * https://theo-jaunet.github.io/MemoryReduction/ 173 | * https://www.ruthfong.com/projects/interactive_overlay/ 174 | 175 | ### Inputs 176 | **Allow readers to make choices using familiar input methods.** 177 | Letting the reader control parameters of a model, simulation or game with sliders and select menus can help them build intuition via familiar methods. 178 | 179 | Allowing the reader to provide more custom input to models via text input or hand-drawing samples can also be powerful. Some care should be given to guide custom input. 180 | 181 | Examples: 182 | * https://pair.withgoogle.com/explorables/fill-in-the-blank/ 183 | * https://jgoertler.com/visual-exploration-gaussian-processes/ 184 | 185 | ### Object constancy 186 | **As a figure changes, keep representations consistent over time.** 187 | Treat representations of data points like physical objects, transition their appearance and positions as they change. 188 | Object constancy enables the reader to follow visual transformations to better understand the conceptual transformations. 189 | 190 | Examples: 191 | * https://mlu-explain.github.io/train-test-validation/ 192 | * [Beginners guide to dimensionality reduction](https://visxai-dimensionality-reduction-1dbad0a67a092b007c526a45.vercel.app/) 193 | * https://bost.ocks.org/mike/constancy/ 194 | 195 | 196 | ## Environment 197 | These patterns help us organize our thoughts across different programming and computing environments. 198 | 199 | ### Kernel 200 | 201 | ### Browser 202 | 203 | ### Package management 204 | **Know how to download code other people wrote.** 205 | Leveraging community code is a huge factor in building any software. Python has pip and anaconda, JavaScript has npm and yarn. 206 | Knowing where packages get installed and how to resolve version conflicts can save hours of your life. 207 | 208 | ### Data serialization 209 | 210 | ### File management 211 | **Know where your data is stored.** 212 | In order to consume data one must first be able to find it. Files and databases can live on local machines, behind firewalls in the cloud or in memory. Being deliberate about where data lives and how it is accessed will improve all aspects of building. 213 | Flat files are better than fancy hierarchies. Tidy data is better than nested data. 214 | 215 | Examples: 216 | * https://microscope.openai.com/models/inceptionv1/mixed5b_3x3_0/98 217 | 218 | 219 | ### APIs 220 | **Design APIs (functions) by considering their context, choices and environment.** 221 | 222 | What **context** does this function need? 223 | What **choices** does the caller have? 224 | What **environment** does it run in? 225 | 226 | TODO: examples 227 | 228 | --- 229 | TODO: replace URLs with their article titles 230 | --------------------------------------------------------------------------------