├── 01-Introduction ├── Fitts_Law.ipynb ├── fitts_task.png └── getting_started.ipynb ├── 02-DeepRL ├── Gaze_Based_Interaction │ ├── .ipynb_checkpoints │ │ └── gaze_based_interaction-checkpoint.ipynb │ ├── gaze_based_interaction.ipynb │ ├── gazetools.py │ ├── image │ │ ├── cog_arch.png │ │ ├── cognitive_POMDP.png │ │ ├── gaze_task.png │ │ ├── internal_env.png │ │ ├── sub_movements.png │ │ ├── time_v_target_size.png │ │ └── visual_acuity.png │ └── output │ │ ├── behaviour_trace.csv │ │ ├── monitor.csv │ │ └── policy.zip ├── bayesian_state_estimation.ipynb └── foveated_vision.ipynb ├── 03-Modelbuilding ├── Go_Nogo.ipynb ├── animate_trace.py ├── corati_model.png ├── driver_agent_physics.py ├── go_nogo.py ├── go_nogo_task.png └── physics_env.py └── README.md /01-Introduction/Fitts_Law.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "id": "3e0c5f5d", 7 | "metadata": {}, 8 | "source": [ 9 | "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/01-Introduction/Fitts_Law.ipynb)\n" 10 | ] 11 | }, 12 | { 13 | "attachments": {}, 14 | "cell_type": "markdown", 15 | "id": "b1800c71-8555-4ffa-b993-249b80cb0234", 16 | "metadata": {}, 17 | "source": [ 18 | "# Fitts' Law\n", 19 | "\n", 20 | "Fitts' law provides a quantitative relationship between movement time in human pointing and the design of the pointing task. This model is instrumental in HCI for evaluating and designing user interfaces by optimizing the time and effort needed for common tasks. In this notebook, we will explore the mathematical underpinnings of Fitts' Law, demonstrating its application in analyzing and enhancing UI efficiency. The learning outcomes of this notebook are:\n", 21 | "\n", 22 | "* understanding of how mathematical formulas can describe psychological phenomena; and\n", 23 | "* being able to apply Fitts' law to make predictions of various UI designs.\n", 24 | "\n", 25 | "## Origins of Fitts' Law\n", 26 | "\n", 27 | "Fitts' Law was formulated in 1954 by psychologist Paul Fitts, originating from his empirical research on human motor skills. His work, deeply rooted in the principles of information theory and cybernetics, sought to understand the control aspects of human movement, particularly in the context of aiming or pointing tasks. Fitts' experiments led to the realization that a logarithmic relationship exists between the speed and accuracy of movements, encapsulated as an equation (the *Fitts' Law*). This discovery was pivotal in linking human cognitive processes with quantifiable metrics, laying a foundation for subsequent research in ergonomics, human factors, and HCI. Fitts' Law stands as a landmark in the history of cognitive psychology and its application to real-world problems.\n", 28 | "\n", 29 | "The original experiment conducted by Paul Fitts involved a simple setup to study the relationship between movement speed and target accuracy. Participants were asked to perform a series of rapid, repetitive pointing tasks using a hand-held stylus. The task required them to move the stylus back and forth between two flat, rectangular targets as quickly and accurately as possible. These targets varied in width (W) and distance (D) from each other. By manipulating these two parameters, Fitts observed how the speed of movement was influenced by the difficulty of the task, which he quantified as the ratio of the distance to the width of the target. This setup provided empirical data that led to the formulation of Fitts' Law, highlighting the logarithmic relationship between the time taken to complete the movement and the difficulty of the task.\n", 30 | "\n", 31 | "\n", 32 | "Figure 1: Illustration of the original Fitts' experiment. Two target rectangles of width W are placed distance D apart, and the participant is tasked with moving a stylus between the targets." 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "id": "65536e12-2ed3-4ea6-ad97-297adcbcdfb8", 38 | "metadata": {}, 39 | "source": [ 40 | "# Definition\n", 41 | "\n", 42 | "Fitts' Law is typically represented by the equation\n", 43 | "\n", 44 | "$\n", 45 | "MT = a + b \\cdot \\log_2\\left(\\frac{D}{W} + 1\\right),\n", 46 | "$\n", 47 | "\n", 48 | "where:\n", 49 | "- $MT$ is the movement time, the time it takes for a user to move the pointer to the target;\n", 50 | "- $a$ and $b$ are empirically determined constants through linear regression;\n", 51 | "- $D$ is the distance from the starting point to the center of the target; and\n", 52 | "- $W$ is the width of the target.\n", 53 | "\n", 54 | "We can define this as a function." 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "id": "20223e8d-9749-489d-aa43-6e6950f0d3dd", 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "import numpy as np\n", 65 | "\n", 66 | "def fitts_law(D, W, a, b):\n", 67 | " \"\"\"\n", 68 | " Calculate the movement time as per Fitts' Law.\n", 69 | "\n", 70 | " Parameters:\n", 71 | " D (float): Distance from the starting point to the center of the target.\n", 72 | " W (float): Width of the target.\n", 73 | " a, b (float): Empirically determined constants.\n", 74 | "\n", 75 | " Returns:\n", 76 | " float: Estimated movement time.\n", 77 | " \"\"\"\n", 78 | " ID = np.log2(D / W + 1) # Index of Difficulty\n", 79 | " MT = a + b * ID\n", 80 | " return MT\n" 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "id": "c1325bd7-66d0-4fbf-b138-197f7ad70785", 86 | "metadata": {}, 87 | "source": [ 88 | "Fitts' law is agnostic to the units of distance, because they cancel out in the equation. With UI design, a standard unit is a pixel. The predicted movement time is in seconds. Let's create a Fitts' law prediction for how a button width impacts movement time, when the distance between the origin of the pointing movement stays constant. You can explore how distance and the constants *a* and *b* impact predictions." 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "id": "dc90c105-12eb-4f0b-a41b-20b87391d811", 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "import numpy as np\n", 99 | "import matplotlib.pyplot as plt\n", 100 | "\n", 101 | "# Fitts' Law function\n", 102 | "def fitts_law(D, W, a, b):\n", 103 | " ID = np.log2(D / W + 1) # Index of Difficulty\n", 104 | " MT = a + b * ID\n", 105 | " return MT\n", 106 | "\n", 107 | "# Constants (example values, should be determined empirically)\n", 108 | "a = 0.1\n", 109 | "b = 0.2\n", 110 | "\n", 111 | "# Fixed distance\n", 112 | "D = 1000 # example distance, you can adjust this\n", 113 | "\n", 114 | "# Generate a range of target widths\n", 115 | "target_widths = np.linspace(10, 600, 100)\n", 116 | "\n", 117 | "# Calculate movement times for each target width\n", 118 | "movement_times = [fitts_law(D, W, a, b) for W in target_widths]\n", 119 | "\n", 120 | "# Create the scatter plot\n", 121 | "plt.scatter(target_widths, movement_times)\n", 122 | "plt.title('Fitts\\' Law: Movement Time vs Target Width')\n", 123 | "plt.xlabel('Target Width (W)')\n", 124 | "plt.ylabel('Predicted Movement Time (s)')\n", 125 | "plt.grid(True)\n", 126 | "plt.show()" 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "id": "dc77c452-7169-4c80-bcbe-c5465bcfe698", 132 | "metadata": {}, 133 | "source": [ 134 | "# Fitts' Law And UI Design\n", 135 | "\n", 136 | "Fitts' law can be directly used to predict movement times between UI elements, and summed to make predictions of trajectories. Let's create a simple code for creating mockup UIs and try out Fitts' law with it." 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": null, 142 | "id": "e1dc68a8-6b42-4fd3-b029-36157ed437d0", 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "%matplotlib inline\n", 147 | "\n", 148 | "import matplotlib.pyplot as plt\n", 149 | "import matplotlib.patches as patches\n", 150 | "from math import sqrt\n", 151 | "\n", 152 | "class element():\n", 153 | " def __init__(self, name, x = 0, y = 0, x_size = 1, y_size = 1, color = \"blue\"):\n", 154 | " self.name = name\n", 155 | " self.x = x\n", 156 | " self.y = y\n", 157 | " self.x_size = x_size\n", 158 | " self.y_size = y_size\n", 159 | " self.color = color\n", 160 | "\n", 161 | " def loc(self):\n", 162 | " return [int(self.x+self.x_size/2), int(self.y+self.y_size/2)]\n", 163 | " \n", 164 | " def get_size(self):\n", 165 | " return min(self.x_size, self.y_size)\n", 166 | "\n", 167 | " def distance_to(self, e):\n", 168 | " loc1 = self.loc()\n", 169 | " loc2 = e.loc()\n", 170 | " return sqrt( (loc1[0] - loc2[0])**2 + (loc1[1] - loc2[1])**2 )\n", 171 | "\n", 172 | "def draw_ui(elements, x_size = 0, y_size = 0, fig_width = 20, fig_height = 20):\n", 173 | " # Prepare plot\n", 174 | " plt.style.use('classic')\n", 175 | " fig, ax = plt.subplots()\n", 176 | " # Figure out size\n", 177 | " size_offset = 10 \n", 178 | " for e in elements:\n", 179 | " x_s = e.x + e.x_size + size_offset\n", 180 | " if x_size < x_s:\n", 181 | " x_size = x_s\n", 182 | " y_s = e.y + e.y_size + size_offset\n", 183 | " if y_size < y_s:\n", 184 | " y_size = y_s\n", 185 | " ax.set_xlim(0, x_size)\n", 186 | " ax.set_ylim(0, y_size)\n", 187 | " ax.invert_yaxis()\n", 188 | " ax.set_aspect('equal', adjustable='box')\n", 189 | " # Draw elements\n", 190 | " for e in elements:\n", 191 | " shape = patches.Rectangle((e.x, e.y), e.x_size, e.y_size, fc=e.color)\n", 192 | " ax.add_patch(shape)\n", 193 | " ax.text(e.x, e.y, str(e.name))\n", 194 | " plt.show()" 195 | ] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "execution_count": null, 200 | "id": "3fdc837b-f527-410e-8d47-4e3994e7d0b2", 201 | "metadata": { 202 | "scrolled": true 203 | }, 204 | "outputs": [], 205 | "source": [ 206 | "# Make a simple UI of four elements.\n", 207 | "elements = [\n", 208 | " element(\"logo\", x = 10, y = 10, x_size = 100, y_size = 100, color = \"blue\"),\n", 209 | " element(\"search\", x = 500, y = 10, x_size = 100, y_size = 20, color = \"grey\"),\n", 210 | " element(\"go\", x = 500, y = 200, x_size = 50, y_size = 50, color = \"red\")\n", 211 | "]\n", 212 | "draw_ui(elements)" 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": null, 218 | "id": "e149604b-c50b-454f-a9b9-2e1e883ae076", 219 | "metadata": {}, 220 | "outputs": [], 221 | "source": [ 222 | "# Create a wrapper for calculating Fitts' law between two elements: from element 1 to element 2.\n", 223 | "def fitts_mt(e1, e2, a = 0.1, b = 0.1):\n", 224 | " D = e1.distance_to(e2)\n", 225 | " W = e2.get_size() \n", 226 | " return fitts_law(D, W, a, b)\n", 227 | "\n", 228 | "def fitts_total_mt(elements, a = 0.1, b = 0.1):\n", 229 | " mt = 0\n", 230 | " for e in range(len(elements) - 1):\n", 231 | " mt += fitts_mt(elements[e], elements[e+1], a, b)\n", 232 | " return mt" 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": null, 238 | "id": "09265c14-383b-48f3-8c3d-08618f05ffbe", 239 | "metadata": {}, 240 | "outputs": [], 241 | "source": [ 242 | "# Calculate total movement time of pointing trajectory, starting from logo, going via search, and ending with go.\n", 243 | "fitts_total_mt(elements)" 244 | ] 245 | }, 246 | { 247 | "cell_type": "markdown", 248 | "id": "11e6fa9f-a3ed-47f3-9d83-c1a8eb6ee0f9", 249 | "metadata": {}, 250 | "source": [ 251 | "With this general formulation of Fitts' law. We can investigate the impact of UI design (in this case, element size) on pointing time." 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": null, 257 | "id": "3bc41bfe-942c-4a58-af37-2cd3639cb522", 258 | "metadata": {}, 259 | "outputs": [], 260 | "source": [ 261 | "# make 'go' very small\n", 262 | "elements = [\n", 263 | " element(\"logo\", x = 10, y = 10, x_size = 100, y_size = 100, color = \"blue\"),\n", 264 | " element(\"search\", x = 500, y = 10, x_size = 1, y_size = 1, color = \"grey\"),\n", 265 | " element(\"go\", x = 10, y = 200, x_size = 1, y_size = 1, color = \"red\")\n", 266 | "]\n", 267 | "draw_ui(elements)\n", 268 | "print(fitts_total_mt(elements))" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "id": "82c21cae-074a-4ed1-b538-c88f717af19a", 275 | "metadata": {}, 276 | "outputs": [], 277 | "source": [ 278 | "# make 'go' very small\n", 279 | "elements = [\n", 280 | " element(\"logo\", x = 10, y = 10, x_size = 100, y_size = 100, color = \"blue\"),\n", 281 | " element(\"search\", x = 200, y = 10, x_size = 100, y_size = 100, color = \"grey\"),\n", 282 | " element(\"go\", x = 200, y = 120, x_size = 100, y_size = 100, color = \"red\")\n", 283 | "]\n", 284 | "draw_ui(elements)\n", 285 | "print(fitts_total_mt(elements))" 286 | ] 287 | }, 288 | { 289 | "cell_type": "markdown", 290 | "id": "c5c6802f-38a9-4e19-9f6d-a78f74baa226", 291 | "metadata": {}, 292 | "source": [ 293 | "We can also use Fitts' law to calculate average time to a target, for instance assuming that pointing approaches the target uniformly from other UI elements." 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "id": "20c7e964-5402-43e3-97c6-7b06c94d301b", 300 | "metadata": {}, 301 | "outputs": [], 302 | "source": [ 303 | "elements = [\n", 304 | " element(\"logo\", x = 10, y = 10, x_size = 100, y_size = 100, color = \"blue\"),\n", 305 | " element(\"header\", x = 150, y = 10, x_size = 300, y_size = 50, color = \"blue\"),\n", 306 | " element(\"search\", x = 500, y = 10, x_size = 100, y_size = 20, color = \"grey\"),\n", 307 | " element(\"go\", x = 500, y = 200, x_size = 50, y_size = 50, color = \"red\")\n", 308 | "]\n", 309 | "draw_ui(elements)\n", 310 | "fitts_total = 0\n", 311 | "for e in elements:\n", 312 | " if e.name != \"go\":\n", 313 | " # elements[-1] is the last element, which is \"go\"\n", 314 | " fitts_total += fitts_mt(e, elements[-1])\n", 315 | "# Print average\n", 316 | "print(fitts_total/(len(elements)-1))" 317 | ] 318 | }, 319 | { 320 | "cell_type": "markdown", 321 | "id": "9f51f177-f956-4b35-bcdf-23d0103275b1", 322 | "metadata": {}, 323 | "source": [ 324 | "# Fitts' Law Parameters\n", 325 | "\n", 326 | "The parameters $a$ and $b$ in Fitts' law are fitted to a particular device and user. The parameter $a$ is the intercept, and it implies that regardless of how difficult a task is (long distance, small target), there is always some overhead associated with pointing, such as motor preparation. The parmaeter $b$ controls the slope: the larger this parameter, the more impactful distance and target width are on movement time.\n", 327 | "\n", 328 | "For instance, we can simulate a user with motor trouble by increasing both parameter values." 329 | ] 330 | }, 331 | { 332 | "cell_type": "code", 333 | "execution_count": null, 334 | "id": "8e914eaf-f5da-4b4a-8eb2-b12239ccbc9d", 335 | "metadata": {}, 336 | "outputs": [], 337 | "source": [ 338 | "print(fitts_total_mt(elements, a = 0.1, b = 0.1))\n", 339 | "print(fitts_total_mt(elements, a = 0.2, b = 0.2))" 340 | ] 341 | }, 342 | { 343 | "cell_type": "markdown", 344 | "id": "3f9d76ab-20c5-4c2e-bb00-3a4ef271374d", 345 | "metadata": {}, 346 | "source": [ 347 | "# Information Processing in Fitts' Law\n", 348 | "\n", 349 | "Fitts' law views motor manipulations as transmission of information between the human decision maker and the interactive system. A human user, equipped with a device such as a mouse, keyboard, joystick, eye movement sensor, or even a some sort of a neural link, transmits information to the system according to an index of performance (IP). The units of IP is bits/s, meaning that the higher the IP, the more information is being transmitted in a second. The task itself has an index of difficuly (ID), which determines how many bits need to be transmitted before the task is completed. Therefore, given movement time (MT), we define $IP = ID/MT$. As index of performance, given a user and a device, stays approximately constant, we can infer that as the task becomes more difficult (IP increases), movement time must equally increase.\n", 350 | "\n", 351 | "We can treat all \"motor\" actions by the user as transmission of information. For instance, perhaps a successful command will require 128 bits of information (this can encode, e.g., a short sentence). The IP computed for the human hand in the original Fitts' experiment is about 10 bits/s. We can solve how long it takes to transmit one sentence with the hand:\n", 352 | "\n", 353 | "$ IP = ID/MT $,\n", 354 | "$ MT = ID/IP $,\n", 355 | "$ MT = 128 bits / 10 bits/s = 12.8s $.\n", 356 | "\n", 357 | "In many time-critical tasks, this is too long. Fortunately, we can immediately cut the transmission time in half by assuming that two fingers use the keyboard to type. If we can make efficient use of ten-finger touch typing, we can achieve a theoretical transmission time of $1.28s$. This is a theoretical lower limit: generally a sentence cannot be optimally divided among 10 fingers.\n" 358 | ] 359 | }, 360 | { 361 | "cell_type": "markdown", 362 | "id": "de171bb9-c313-4630-8d29-54f6ab3894b2", 363 | "metadata": {}, 364 | "source": [ 365 | "# Assignment\n", 366 | "\n", 367 | "Create a mock-up of a UI you are designing. Investigate how Fitts' law predicts movement time to a key element in the UI from various other elements (or all of them on average). Create a scatterplot that shows how the size of the target element impacts movement time, and investigate the shape of the plot: at what size does more size increase really not pay off anymore?" 368 | ] 369 | }, 370 | { 371 | "cell_type": "markdown", 372 | "id": "fd8cf6d7-be8d-4c4c-b8bf-17f5906a7bae", 373 | "metadata": {}, 374 | "source": [ 375 | "# Sources\n", 376 | "\n", 377 | "- A good overview of Fitts' law in HCI: MacKenzie, I. S. (1992). Fitts' law as a research and design tool in human-computer interaction. Human-computer interaction, 7(1), 91-139.\n", 378 | "- The original paper: Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47, 381-391." 379 | ] 380 | }, 381 | { 382 | "cell_type": "code", 383 | "execution_count": null, 384 | "id": "d7d2c89b-814d-4a66-b15a-a5e36fdb035d", 385 | "metadata": {}, 386 | "outputs": [], 387 | "source": [] 388 | } 389 | ], 390 | "metadata": { 391 | "kernelspec": { 392 | "display_name": "Python 3 (ipykernel)", 393 | "language": "python", 394 | "name": "python3" 395 | }, 396 | "language_info": { 397 | "codemirror_mode": { 398 | "name": "ipython", 399 | "version": 3 400 | }, 401 | "file_extension": ".py", 402 | "mimetype": "text/x-python", 403 | "name": "python", 404 | "nbconvert_exporter": "python", 405 | "pygments_lexer": "ipython3", 406 | "version": "3.12.3" 407 | } 408 | }, 409 | "nbformat": 4, 410 | "nbformat_minor": 5 411 | } 412 | -------------------------------------------------------------------------------- /01-Introduction/fitts_task.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/01-Introduction/fitts_task.png -------------------------------------------------------------------------------- /01-Introduction/getting_started.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "cardiovascular-breed", 6 | "metadata": {}, 7 | "source": [ 8 | "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/01-Introduction/getting_started.ipynb)\n" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "id": "operational-lloyd", 14 | "metadata": {}, 15 | "source": [ 16 | "# C28: Cognitive Modelling: From GOMS to Deep Reinforcement Learning\n", 17 | "\n", 18 | "\n", 19 | "\n", 20 | "\n", 21 | "Organizers:\n", 22 | "\n", 23 | "Jussi P. P. Jokinen, University of Jyväskylä
\n", 24 | "Antti Oulasvirta, Aalto University
\n", 25 | "Andrew Howes, University of Exeter\n", 26 | "\n", 27 | "Contact email: jussi.p.p.jokinen@jyu.fi\n", 28 | "\n", 29 | "This course introduces computational cognitive modeling for researchers and practitioners in the field of HCI. Cognitive models use computer programs to model how users perceive, think, and act in interactive tasks. They offer a powerful approach for understanding interaction and improving user interfaces. The course starts with a review of classic models such as Fitts's Law, GOMS and ACT-R. It builds on this review to provide an introduction to modern modelling approaches powered by machine learning methods, in particular deep learning, reinforcement learning (RL), and deep RL. The course is built around hands-on Python programming using Colab notebooks.\n", 30 | "\n", 31 | "A basic understanding of programming concepts is a prerequisite for the course and some familiarity with Python and Google Colab Notebooks (similar to Jupyter notebooks) is likely to be useful. Hopefully, you are reading this text having uploaded it to your private Google Colab -- if not then click on the \"Open in Colab\" link at the top of this page. This is a good start and it means that the document you are reading is not a static web page but, instead, an interactive environment. The key property of this environment is that it lets you write and execute code interactively. \n", 32 | "\n", 33 | "We will illustrate interactive code execution later in this notebook. We will also briefly review some of the historically important ideas in cognitive modelling for HCI. However, before we do that, we first state the learning objectives of the course and also propose a set of scientific objectives for a modern approach to cognitive modeling -- an approach that takes full advantage of recent advances in machine learning. The scientific objectives will be cruical to understanding the modeling approaches and techniques introduced later in the course. They will also be used as a basis for critiquing the historically important approaches from GOMS to ACT-R" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "id": "plastic-ghost", 39 | "metadata": {}, 40 | "source": [ 41 | "## Learning Objectives\n", 42 | "\n", 43 | "\n", 44 | "The learning objectives for the course are of two types, either \"understand...\" or \"be able to...\", where the latter is a practical skill. The objectives are grouped according to the three main modules of the course.\n", 45 | "\n", 46 | "* Understand the strengths and weaknesses of a cognitive model specified as a Partially Observable Markov Decision Process (POMDP) with foveated vision and Kalman state estimation.\n", 47 | "* Be able to use a Python reinforcement learning library -- running on a Colab server -- to find a solution to this POMDP.\n", 48 | "* Be able to test the above model under a range of conditions and generate results for comparison to human data.\n", 49 | "* Understand the HCI cognitive modelling problems for which RL is useful.\n", 50 | "\n", 51 | "\n", 52 | "* Understand the strengths and weaknesses of a cognitive model specified as an Artificial Neural Network (ANN).\n", 53 | "* Be able to make use of well-known pre-trained ANNs by incorporating Python libraries in your models.\n", 54 | "* Understand the HCI cognitive modelling problems for which ANNs are useful.\n", 55 | "\n", 56 | "\n", 57 | "* Understand modeling workflow, with an ability to provide a detailed case example.\n", 58 | "* Be able to use a reinforcement learning model of multitasking while driving.\n", 59 | "* Be able to use Bayesian likelihood free inference to fit model parameters.\n", 60 | "* Understand how parameterised simulation models can be used to explore design candidates." 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "id": "medium-robinson", 66 | "metadata": {}, 67 | "source": [ 68 | "## Criteria for a modern approach to cognitive modeling\n", 69 | "\n", 70 | "While the tutorial seeks to cover a wide range of modeling techniques it is unashamedly committed to an emergent approach in cognitive science that is both routed in the history of information processing psychology and which seeks to take advantage of the recent convergence of cognitive science and HCI. Accordingly, the tutorial works towards introducing the student to modelling techniques which meet the following criteria:\n", 71 | "\n", 72 | "* Be firmly grounded in the information processing view of cognition (Miller, 1956).\n", 73 | "* Explain the adaptation of human behaviour to the environment (including the designed environment) (Brunswick, 1943). Brunswik proposed that humans be explained in terms of statistical descriptions of the structure of the environment. In HCI, models ought to be tested on tasks that are representative of the environment that computer users inhabit.\n", 74 | "* Build on Simon's (1955) insight that adaptation is bounded by two sources of constraint; not only those on on the environment, but also constraints on the mind, including perceptual/motor and memory constraints.\n", 75 | "* Clearly distinguish the explanatory role of invariant bounds on the mind, such as memory limits and visual acuity, from the strategies that are learned through experience, for example rehersal or knowing where to look for important information when a web page is opened. While people are bounded, much of what computer users do is strategic (Oulasvirta, et al., 2022).\n", 76 | "* Formally define theories of cognition as optimisation problems that incorporate bounds on both the environment and cognition (Lewis, Howes, Singh, 2014; Lieder, Griffiths, 2020). Human-like strategies can then be predicted by solving such optimisation problems.\n", 77 | "* Make use of the convergence between machine learning and cognitive science (Gershman, Tenenbaum and Horvitz, 2015), both to specify theories of the bounds on cognition and to derive the strategies that are a response to these bounds. Machine learning has the potential to add both rigour and convenience to the process of cognitive modeling in HCI.\n", 78 | "* Test theories of cognition using inverse modeling (Kangasrääsiö, et al., 2019).\n", 79 | "\n", 80 | "These are a challenging set of criteria that are not met in their entirety by any single cognitive model. However, they point in a direction for the future and can be used as a basis of understanding the relative stengths and weaknesses of each existing approach." 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "id": "catholic-guess", 86 | "metadata": {}, 87 | "source": [ 88 | "## Background\n", 89 | "\n", 90 | "This section shows you how to interact with the notebooks.\n", 91 | "\n", 92 | "**Fitts's Law**\n", 93 | "\n", 94 | "Below find a short Python script -- in a code cell -- that computes a predicted movement time MT given the distance D and width W of a button on a screen." 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "id": "roman-norfolk", 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "from math import *\n", 105 | "\n", 106 | "a = 1.03\n", 107 | "b = 0.096\n", 108 | "D = 8\n", 109 | "W = 1.25\n", 110 | "\n", 111 | "MT = a + b * log2(0.5+D/W)\n", 112 | "MT" 113 | ] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "id": "three-pastor", 118 | "metadata": {}, 119 | "source": [ 120 | "You can execute the code in the above cell by selecting it with a click and then pressing the play button to the left of the code, or by using the keyboard shortcut 'Command/Ctrl+Enter'. \n", 121 | "\n", 122 | "To edit the code, click the cell and start typing. For example, you can predict MT for a different distance by changing the value of D and re-running the cell. Try it now.\n", 123 | "\n", 124 | "Some readers may recognise the above formula as Fitts's Law, but whether you recognise it or not, you have have now used Python and Colab to execute your first model of a user. Congratulations, you are on the path to mastery!\n", 125 | "\n", 126 | "Fitts's Law illustrates one key property of models of users in HCI which is that they make predictions. If parameters 'a' and 'b' are known then Fitts's Law can be used to predict movement time given the distance and width of a target. For now, lets continue to learn about Colab's notebooks and Python. One important thing to know is that variables that you define in one cell can be used in other cells. For example, the value of MT is available in the code cell below:" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "id": "african-queens", 133 | "metadata": {}, 134 | "outputs": [], 135 | "source": [ 136 | "MT" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "id": "standing-harvest", 142 | "metadata": {}, 143 | "source": [ 144 | "Code cells can also be used to define Python functions." 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "id": "local-stamp", 151 | "metadata": {}, 152 | "outputs": [], 153 | "source": [ 154 | "def fitts_mt(D,W):\n", 155 | " a = 1.03\n", 156 | " b = 0.096\n", 157 | " MT = a + b * log2(0.5+D/W)\n", 158 | " return MT" 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "id": "lesbian-investigation", 164 | "metadata": {}, 165 | "source": [ 166 | "Run the code cell above to define the function. Once the function is defined, we can call it from other code cells." 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "id": "flying-structure", 173 | "metadata": {}, 174 | "outputs": [], 175 | "source": [ 176 | "fitts_mt(2,1.25)" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "id": "instant-practice", 182 | "metadata": {}, 183 | "source": [ 184 | "Another important feature of Colab notebooks is that they can be used to visualise data. In the code cell below our fitts_mt() function is used to visualise the relationship between MT and distance. " 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": null, 190 | "id": "crude-lesson", 191 | "metadata": { 192 | "scrolled": true 193 | }, 194 | "outputs": [], 195 | "source": [ 196 | "import numpy as np\n", 197 | "from matplotlib import pyplot as plt\n", 198 | "\n", 199 | "xs = [x for x in range(1,17)]\n", 200 | "ys = [fitts_mt(x,1.25) for x in xs]\n", 201 | "\n", 202 | "plt.plot(xs, ys, '-')\n", 203 | "plt.xlabel(\"Distance D\")\n", 204 | "plt.ylabel(\"Movement time MT\")\n", 205 | "plt.title(\"Fitts's Law\")\n", 206 | "plt.show()" 207 | ] 208 | }, 209 | { 210 | "cell_type": "markdown", 211 | "id": "developmental-stupid", 212 | "metadata": {}, 213 | "source": [ 214 | "You may have noticed the use of the Python import statement in the above code cells. As their name suggests these import code from libraries. There are many basic libraries in python that are used for maths and plotting, for example. More exciting are the libraries that support machine learning. We will see as the tutorial progresses that these libraries can be very useful for modeling cognition." 215 | ] 216 | }, 217 | { 218 | "cell_type": "markdown", 219 | "id": "dutch-authorization", 220 | "metadata": {}, 221 | "source": [ 222 | "

\n", 223 | "

\n", 224 | "\n", 225 | "

\n", 226 | "Reflection

\n", 227 | "

Fitts's Law has proved highly influential in HCI since it was used by Card, English and Burr (1978) to compare various input devices. There are many research papers that describe more recent developments, the application of Fitts's Law, and a number of limitations. One limitation is the fact that the law says nothing about speed/accuracy trade-offs, nor anything about errors; thereby failing to meet the demand for explaining adaptation in our above criteria for modelling. When a person moves a pointer to a target, they can choose whether to move quickly or slowly. If they choose to move quickly then the chance that they will make an error will increase. This is a topic that we will return to later in the tutorial when we look at reinforcement learning models of cognition. In these models, a reward function determines the desired speed/accuracy trade-off function and predictions of movement time emerge through learning an optimal policy for the reward function. The ability to model speed/accuracy trade-offs is critical to the scientific understanding of HCI because of the adaptive nature of human cognition.\n", 228 | "

\n", 229 | "

" 230 | ] 231 | }, 232 | { 233 | "cell_type": "markdown", 234 | "id": "opposite-calculator", 235 | "metadata": {}, 236 | "source": [ 237 | "**GOMS**\n", 238 | "\n", 239 | "GOMS was an approach to cognitive modeling in Human-Computer Interaction in which human skill for computing systems was represented in terms of goals, operators, methods and selection rules. Many other methods for modeling task knowledge were based on similar concepts and GOMS gave rise to many variants through the 1980s and into the 2000s. Though their meanings have been refined, some GOMS concepts remain useful today, and it is worth briefly reviewing them. \n", 240 | "\n", 241 | "Goals are what the user has to accomplish and represent the end towards which the user is directed. Example goals might include making a set of changes to a text document or replying to an email. Goals typically have subgoals giving rise to the hierarchical structure of skill. For example, the goal of correcting the typos in a text might have as subgoals to \"delete text\" and \"insert word\". \n", 242 | "\n", 243 | "Operators model what a user can do in service of a goal. Operators may be theories of how a user makes changes to the external environment or they may be theories of mental operators that, for example, update memory. The very lowest level of operator could be indiviudal muscle axon innervations but cognitive models typically define operators at a much higher level than this. For example, a cognitive model of form filling might define operators at the word entry level, whereas a cognitive model of typing might define operators at a character level. \n", 244 | "\n", 245 | "Methods are sequences of subgoals and operators that achieve specific goals. For example, the goal of entering a name into a form might be accomplished with methods that we can represent with Python functions." 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "id": "precious-cliff", 252 | "metadata": {}, 253 | "outputs": [], 254 | "source": [ 255 | "def enter_name():\n", 256 | " move_caret_to_location()\n", 257 | " print(\"ENTER NAME\")\n", 258 | "\n", 259 | "def move_caret_to_location():\n", 260 | " print(\"DETECT LOCATION\")\n", 261 | " print(\"MOVE MOUSE UNTIL POINTER AT LOCATION\")\n", 262 | " print(\"CLICK MOUSE\")\n", 263 | "\n", 264 | "enter_name()" 265 | ] 266 | }, 267 | { 268 | "cell_type": "markdown", 269 | "id": "gentle-stanford", 270 | "metadata": {}, 271 | "source": [ 272 | "When the code cell above is executed, the GOMS methods for the goal ENTER NAME are expanded resulting in the required operator sequence being printed.\n", 273 | "\n", 274 | "Selection Rules choose between different methods for achieving a goal. For example, an alternative method for ENTER NAME might use TAB to move the caret from an earlier location rather than using the mouse. The selection rule might prefer one method or the other depending on the sequentiality of the entry process.\n", 275 | "\n", 276 | "The difference between a goal and an operator in GOMS is that operators are not broken down any further; they are at the bottom of a hierarchy of subgoals. Card, Moran and Newell demonstrated GOMS models at 4 levels of analysis. At one level the operators represented tasks taking about 30 seconds, at another the operators represented single keystrokes. As with Fitts's Law, the idea was that GOMS could be used to predict human performance time by adding up the durations of the sequence of operators required to achieve a goal. \n", 277 | "\n", 278 | "

\n", 279 | "

\n", 280 | "\n", 281 | "

\n", 282 | "Reflection

\n", 283 | "

GOMS embodied a number of concepts tha remain important to modeling today. These include the idea that human task knowledge is hierarchical and that goals are decomposed into subgoals and eventually operators. This ideas echoes in modern ideas concerning hierarchical reinforcement learning, for example.\n", 284 | "\n", 285 | "

But, there are a number of concepts missing in GOMS that are crucial to modern analyses of task knowledge. One of these concepts is that of utility (or reward). Another is the extent to which task knowledge is now known to be conditioned on state; that is on bottom-up information and not just on top-down hierarchical control. Cognition is embodied and interactive. GOMS therefore fails to meet the demand of our criteria; cognitive models must be sensitive to the statistical structure of the task environment; in other words the ecology.\n", 286 | " \n", 287 | "

In addition, GOMS models are difficult to build because they demand that the analyst hand-code a set of rules that represent a user's task knowledge -- which can require many interlated rules and be difficult to determine from human behaviour. GOMS, therefore, fails to meet our demand that cognitive modeling approaches should take full advantage of the recent convergence between ML and cognitive science. This convergence provides methods for learning task knowledge that, in for some tasks, results in human-level performance.\n", 288 | "

\n", 289 | "

\n" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "id": "unexpected-productivity", 295 | "metadata": {}, 296 | "source": [ 297 | "**Cognitive architectures**\n", 298 | "\n", 299 | "Cognitive architectures, particuarly ACT-R (Anderson, 2007) and EPIC (Kieras and Meyer, 1997), have played a crucial role in cognitive modeling for HCI. \n", 300 | "\n", 301 | "ACT-R consists of a number of integrated modules for modeling cognition. The modules include those for audition, vision, manual movement, declarative memory, and temporal memory. These modules are motivated by neuroscientific and behavioural evidence and empirically validated by many hundreds of human experiments. The modules are coordinated by a central production module which is responsible for matching, selecting and executing production rules. These rules make changes to the state of the other modules by, for example, issuing motor commands, or updating memories. ACT-R also provides theories of learning. In ACT-R's theory of procedural learning, skills are acquired by recruiting past problem solutions while solving new problems, thereby providing a theory of learning by doing and learning by example (Anderson and Schunn, 2000). ACT-R also models activation learning in declarative memory. Two factors influence this learning: how many times an item in memory was needed in the past, and how recently it was needed. The activation learning rule is derived from Bayesian statistics. ACT-R uses activation values of items in memory to order the chunks in the matching process.\n", 302 | "\n", 303 | "\n", 304 | "

\n", 305 | "

\n", 306 | "\n", 307 | "

\n", 308 | "Reflection

\n", 309 | "

One of the most significant theoretical contributions to HCI, information foraging theory, was grounded in ACT-R theory (Pirolli, 1997). EPIC has also played a significant role, for example, in models of visual search (Kieras and Hornoff, 2014). Arguably, ACT-R represents the state of the art in computational models of human memory.\n", 310 | "\n", 311 | "

ACT-R and EPIC meet many of the criteria for cognitive modeling that we set at the beginning of this notebook. They are both bounded information processing theories of cognition and they embrace the notion of adaptation. In ACT-R's case adaptation is modelled through various learning mechanisms built into the architecture. These can both be used to generate multiple strategies and to select between them, meeting another criteria. However, ACT-R and EPIC do not permit the definition of theories of cognition as optimisation problems in response to which modern machine learning algorithms can be used to automatically find human-like strategies. Instead, even with learning researchers are still required to program some production rules, considerably increasing the burden on the modeler, and also increasing the parametric flexibility of the model leading to difficulties associated with fitting ACT-R models to data.\n", 312 | " \n", 313 | "

\n", 314 | "

\n" 315 | ] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "id": "divided-congress", 320 | "metadata": {}, 321 | "source": [ 322 | "## Discussion\n", 323 | "\n", 324 | "You are now at the end of the \"getting started\" module of the CHI'2023 course on cognitive modeling. Well done, for getting this far. \n", 325 | "\n", 326 | "We have attempted to cover over 40 years of cognitive modeling in HCI in a few hundred words; as a consequence we are aware of many many omissions. However, our main purpose was to illustrate the contrast between the well-known approaches in HCI and the modern -- machine learning fuelled -- approach that has been emerging over the past few years. This new approach, sometimes known as computational rationality (or resource rationality), begins to meet all of the criteria for cognitive modeling that we articulated at the beginnig of this document. Accordingly, the main body of the tutorial will pick up on these criteria to motivate the use deep learning and deep reinforcement learning in cognitive models for HCI.\n", 327 | "\n", 328 | "If you have time to find out more then recommended, and highly recommended papers, are starred and double starred in the bibliogrpahy below. To find out more about Colab notebooks, see Overview of Colab. Colab notebooks are Jupyter notebooks that are hosted by Colab. To find out more about the Jupyter project, see jupyter.org.\n", 329 | "\n", 330 | "If you are a novice Python programmer then you should find out more about lists, dictionaries and about classes/objects.\n", 331 | "\n", 332 | "Further modules will be provided on the day of the tutorial. We look forward to seeing you there.\n", 333 | "\n", 334 | "Jussi Jokinen
\n", 335 | "Antti Oulasvirta
\n", 336 | "Andrew Howes\n" 337 | ] 338 | }, 339 | { 340 | "cell_type": "markdown", 341 | "id": "brief-vehicle", 342 | "metadata": {}, 343 | "source": [ 344 | "**References**\n", 345 | "\n", 346 | "Anderson, J. R. (2007). How Can the Human Mind Exist in the Physical Universe? Oxford University Press.\n", 347 | "\n", 348 | "Card, S. M., & Newell, T. (1983). A.(1983)“The Psychology of Human-Computer Interaction.”.\n", 349 | "\n", 350 | "Card, S. K., English, W. K., Burr, B. J., 1978. Evaluation of mouse, rate controlled isometric joystick, step keys and text keys for text selection on a CRT. Ergonomics 21, 601-613.\n", 351 | "\n", 352 | "\\* Gershman, S. J., Horvitz, E. J., & Tenenbaum, J. B. (2015). Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science, 349(6245), 273-278.\n", 353 | "\n", 354 | "John, B. E., & Kieras, D. E. (1996). The GOMS family of user interface analysis techniques: Comparison and contrast. ACM Transactions on Computer-Human Interaction (TOCHI), 3(4), 320-351.\n", 355 | "\n", 356 | "\\* Kangasrääsiö, A., Jokinen, J. P., Oulasvirta, A., Howes, A., & Kaski, S. (2019). Parameter inference for computational cognitive models with Approximate Bayesian Computation. Cognitive science, 43(6), e12738.\n", 357 | "\n", 358 | "Kieras, D. E., & Meyer, D. E. (1997). An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Human–Computer Interaction, 12(4), 391-438.\n", 359 | "\n", 360 | "Kieras, D. E., & Hornof, A. J. (2014, April). Towards accurate and practical predictive models of active-vision-based visual search. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 3875-3884).\n", 361 | "\n", 362 | "** Lewis, R. L., Howes, A., & Singh, S. (2014). Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in cognitive science, 6(2), 279-311.\n", 363 | "\n", 364 | "** Lieder, F., & Griffiths, T. L. (2020). Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences, 43.\n", 365 | "\n", 366 | "Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2), 81.\n", 367 | "\n", 368 | "\\* Pirolli, P. (1997). Computational models of information scent-following in a very large browsable text collection. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (pp. 3-10).\n", 369 | "\n", 370 | "Simon, H. A. (1955). A behavioral model of rational choice. The quarterly journal of economics, 69(1), 99-118.\n" 371 | ] 372 | } 373 | ], 374 | "metadata": { 375 | "kernelspec": { 376 | "display_name": "Python 3 (ipykernel)", 377 | "language": "python", 378 | "name": "python3" 379 | }, 380 | "language_info": { 381 | "codemirror_mode": { 382 | "name": "ipython", 383 | "version": 3 384 | }, 385 | "file_extension": ".py", 386 | "mimetype": "text/x-python", 387 | "name": "python", 388 | "nbconvert_exporter": "python", 389 | "pygments_lexer": "ipython3", 390 | "version": "3.9.5" 391 | } 392 | }, 393 | "nbformat": 4, 394 | "nbformat_minor": 5 395 | } 396 | -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/.ipynb_checkpoints/gaze_based_interaction-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "id": "3e0c5f5d", 7 | "metadata": {}, 8 | "source": [ 9 | "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/02-DeepRL/Gaze_Based_Interaction/gaze_based_interaction.ipynb)\n" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "id": "fitted-component", 15 | "metadata": { 16 | "id": "fitted-component" 17 | }, 18 | "source": [ 19 | "# A cognitive model of gaze-based interaction\n", 20 | "\n", 21 | "Some cognitive models describe behaviour and others predict it. Models that predict behaviour must be capable of generating output without that output being part of the input. In this notebook we demonstrate this property for a model of eye movements. The model is a reinforcement learner that is trained by performing hundreds of thousands of simulated eye movements in search of a target of varying size and distance. The model predicts how many eye movements people will tend to make to find a target of a given size and distance and, in addition, that the first eye movement to a target will tend to undershoot, rather than overshoot.\n", 22 | "\n", 23 | "### The task: Gaze-based interaction\n", 24 | "\n", 25 | "Gaze-based interaction is a mode of interaction in which users, including users with a range of movement disabilities, are able to indicate which display item they wish to select by fixating their eyes on it. Confirmation of selection is then made with one of a number of methods, including with a key press, or by holding the fixation for an extended duration of time. The model performs this task for targets with randomly selected location and size. \n", 26 | "\n", 27 | "<>\n", 28 | "\n", 29 | "### Model architecture\n", 30 | "\n", 31 | "The model has a simple architecture that you have previously seen in the introduction. The figure is reproduced here:\n", 32 | "\n", 33 | "\"Corati\n", 34 | "\n", 35 | "- The **control** module makes decisions about where to move the eyes with the oculomotor system. Decisions are conditioned on the current belief about the location of the target.\n", 36 | "- The **motor** module implements decisions but it is bounded by Gaussian noise, which models noise in the human motor system.\n", 37 | "- The **environment** models the physics of the world and the task (the location of the target). Given a response from the motor system, a saccade is made to the aim point of the eyes, and a fixation is initiated.\n", 38 | "- The **perception** module simulates the human capacity to localize a target with foveated vision. The accuracy of the location estimate generated by perception is negatively affected by the eccentricity of the target from the current fixation location.\n", 39 | "- The **Memory** module stores a representation of the current state. Over the course of an episode a sequence of location estimates will be made. Humans are known to integrate these estimates into a single integrated representation of the location. People are known to do this optimally using a process that can be modelled with Bayesian state estimation. The state estimation constitutes a belief about the location of the target.\n", 40 | "- The **Utility** module calculates a reward signal given the current belief about the enviornment. The reward signal is used to train the controller.\n", 41 | "\n", 42 | "### Prerequisites\n", 43 | "\n", 44 | "Before proceeding with this notebook you should firrst review the notebooks on foveated vision and on Bayesian integration." 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "id": "a427be22", 50 | "metadata": {}, 51 | "source": [ 52 | "### Machine learning\n", 53 | "\n", 54 | "In order to learn how to perform the task, the model uses a set of implementations of reinforcement learning algorithms in PyTorch known as stable-baselines3." 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "id": "vL8yMY6q_Rd-", 61 | "metadata": { 62 | "id": "vL8yMY6q_Rd-" 63 | }, 64 | "outputs": [], 65 | "source": [ 66 | "# Install stable_baselines3\n", 67 | "# This is a well known machine learning library that provides a suite of reinforcement learning methods.\n", 68 | "# Only needs to be run once\n", 69 | "\n", 70 | "!pip install --pre -U stable_baselines3" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "id": "oeJU5wJR7A8V", 77 | "metadata": { 78 | "colab": { 79 | "base_uri": "https://localhost:8080/" 80 | }, 81 | "executionInfo": { 82 | "elapsed": 1652, 83 | "status": "ok", 84 | "timestamp": 1643210006123, 85 | "user": { 86 | "displayName": "Andrew Howes", 87 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64", 88 | "userId": "02694399383679444060" 89 | }, 90 | "user_tz": 0 91 | }, 92 | "id": "oeJU5wJR7A8V", 93 | "outputId": "f8e53d98-820c-4a31-b9f5-e4b5c46bd72b" 94 | }, 95 | "outputs": [], 96 | "source": [ 97 | "# Ignore this cell unless you need to save results to Google drive.\n", 98 | "# Mount Google drive and change directory into the project folder.\n", 99 | "# from google.colab import drive\n", 100 | "# drive.mount('/content/drive')\n", 101 | "# %cd '/content/drive/MyDrive/CHI22CMT/CHI22_CogMod_Tutorial/03-Reinforcement-Learning/034_Gaze_based_Interaction'" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": null, 107 | "id": "P44tljjb8hBC", 108 | "metadata": { 109 | "colab": { 110 | "base_uri": "https://localhost:8080/" 111 | }, 112 | "executionInfo": { 113 | "elapsed": 6305, 114 | "status": "ok", 115 | "timestamp": 1643209694066, 116 | "user": { 117 | "displayName": "Andrew Howes", 118 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64", 119 | "userId": "02694399383679444060" 120 | }, 121 | "user_tz": 0 122 | }, 123 | "id": "P44tljjb8hBC", 124 | "outputId": "d635fb70-1df0-4ab3-f15e-13b3d7b2fdc9" 125 | }, 126 | "outputs": [], 127 | "source": [ 128 | "# Load local modules\n", 129 | "# gazetools is a module that contains functions for modeling gaze-based interaction.\n", 130 | "\n", 131 | "!wget https://raw.githubusercontent.com/howesa/CHI22-CogMod-Tutorial/main/03-Reinforcement-Learning/034_Gaze_based_Interaction/gazetools.py\n", 132 | "\n", 133 | "import gazetools" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "id": "obvious-point", 140 | "metadata": { 141 | "executionInfo": { 142 | "elapsed": 358, 143 | "status": "ok", 144 | "timestamp": 1643209000116, 145 | "user": { 146 | "displayName": "Andrew Howes", 147 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64", 148 | "userId": "02694399383679444060" 149 | }, 150 | "user_tz": 0 151 | }, 152 | "id": "obvious-point" 153 | }, 154 | "outputs": [], 155 | "source": [ 156 | "# load required standard modules and configure matplotlib\n", 157 | "\n", 158 | "import numpy as np\n", 159 | "import math\n", 160 | "import matplotlib.pyplot as plt\n", 161 | "import sys\n", 162 | "\n", 163 | "import gymnasium as gym\n", 164 | "sys.modules[\"gym\"] = gym\n", 165 | "from gym import spaces\n", 166 | "\n", 167 | "import matplotlib as mpl\n", 168 | "%matplotlib inline\n", 169 | "mpl.style.use('ggplot')" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "id": "injured-leadership", 175 | "metadata": { 176 | "id": "injured-leadership" 177 | }, 178 | "source": [ 179 | "### Implementation of the Cognitive Architecture as a Python Class\n", 180 | "\n", 181 | "The first step to formalise the model architecture presented in the above figure. We do this by specifying a class of cognitive theories and will later define instances of this class.\n", 182 | "\n", 183 | "The class has only a single method, which defines a step through the processes defined in the figure. " 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": null, 189 | "id": "monetary-ivory", 190 | "metadata": { 191 | "executionInfo": { 192 | "elapsed": 270, 193 | "status": "ok", 194 | "timestamp": 1643209727593, 195 | "user": { 196 | "displayName": "Andrew Howes", 197 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64", 198 | "userId": "02694399383679444060" 199 | }, 200 | "user_tz": 0 201 | }, 202 | "id": "monetary-ivory" 203 | }, 204 | "outputs": [], 205 | "source": [ 206 | "class CognitivePOMDP():\n", 207 | "\n", 208 | " def __init__(self):\n", 209 | " self.internal_state = {}\n", 210 | " \n", 211 | " def step(self, ext, action):\n", 212 | " ''' Define the cognitive POMDP.'''\n", 213 | " self._update_state_with_action(action)\n", 214 | " response = self._get_response()\n", 215 | " external_state, done = ext.external_env(response)\n", 216 | " stimulus, stimulus_std = self._get_stimulus(ext.external_state)\n", 217 | " self._update_state_with_stimulus(stimulus, stimulus_std)\n", 218 | " obs = self._get_obs()\n", 219 | " reward = self._get_reward()\n", 220 | " return obs, reward, done" 221 | ] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "id": "brazilian-timeline", 226 | "metadata": { 227 | "id": "brazilian-timeline" 228 | }, 229 | "source": [ 230 | "### A theory of gaze-based interaction\n", 231 | "\n", 232 | "Each of the entities in CognitivePOMDP must be defined so as to state our theory of gaze-based interaction. The theory makes the following assumptions:\n", 233 | "\n", 234 | "* Target location stimuli are corrupted by Gaussian noise in human vision.\n", 235 | "* The standard deviation of noise increases linearly with eccentricity from the fovea.\n", 236 | "* Sequences of stimuli are noisily perceived and optimally integrated.\n", 237 | "* Intended eye movements (oculomotor actions) are corrupted by signal dependent Gaussian noise to generate responses.\n", 238 | "\n", 239 | "These assumptions are further described in Chen et al. (2021)." 240 | ] 241 | }, 242 | { 243 | "cell_type": "code", 244 | "execution_count": null, 245 | "id": "accepted-reunion", 246 | "metadata": { 247 | "executionInfo": { 248 | "elapsed": 242, 249 | "status": "ok", 250 | "timestamp": 1643209731182, 251 | "user": { 252 | "displayName": "Andrew Howes", 253 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64", 254 | "userId": "02694399383679444060" 255 | }, 256 | "user_tz": 0 257 | }, 258 | "id": "accepted-reunion" 259 | }, 260 | "outputs": [], 261 | "source": [ 262 | "class GazeTheory(CognitivePOMDP):\n", 263 | "\n", 264 | " def __init__(self):\n", 265 | " ''' Initialise the theoretically motivated parameters.'''\n", 266 | " # weight eye movement noise with distance of saccade\n", 267 | " self.oculamotor_noise_weight = 0.01\n", 268 | " # weight noise with eccentricity\n", 269 | " self.stimulus_noise_weight = 0.09\n", 270 | " # step_cost for the reward function\n", 271 | " self.step_cost = -1\n", 272 | " # super.__init__()\n", 273 | "\n", 274 | " def reset_internal_env(self, external_state):\n", 275 | " ''' The internal state includes the fixation location, the latest estimate of \n", 276 | " the target location and the target uncertainty. Assumes that there is no \n", 277 | " uncertainty in the fixation location.\n", 278 | " Assumes that width is known. All numbers are on scale -1 to 1.\n", 279 | " The target_std represents the strength of the prior.'''\n", 280 | " self.internal_state = {'fixation': np.array([-1,-1]), \n", 281 | " 'target': np.array([0,0]), \n", 282 | " 'target_std': 0.1,\n", 283 | " 'width': external_state['width'],\n", 284 | " 'action': np.array([-1,-1])} \n", 285 | " return self._get_obs() \n", 286 | "\n", 287 | " def _update_state_with_action(self, action):\n", 288 | " self.internal_state['action'] = action\n", 289 | " \n", 290 | " def _get_response(self):\n", 291 | " ''' Take an action and add noise.'''\n", 292 | " # !!!! should take internal_state as parameter\n", 293 | " move_distance = gazetools.get_distance( self.internal_state['fixation'], \n", 294 | " self.internal_state['action'] )\n", 295 | " \n", 296 | " ocularmotor_noise = np.random.normal(0, self.oculamotor_noise_weight * move_distance, \n", 297 | " self.internal_state['action'].shape)\n", 298 | " # response is action plus noise\n", 299 | " response = self.internal_state['action'] + ocularmotor_noise\n", 300 | " \n", 301 | " # update the ocularmotor state (internal)\n", 302 | " self.internal_state['fixation'] = response\n", 303 | " \n", 304 | " # make an adjustment if response is out of range. \n", 305 | " response = np.clip(response,-1,1)\n", 306 | " return response\n", 307 | " \n", 308 | " def _get_stimulus(self, external_state):\n", 309 | " ''' define a psychologically plausible stimulus function in which acuity \n", 310 | " falls off with eccentricity.''' \n", 311 | " eccentricity = gazetools.get_distance( external_state['target'], external_state['fixation'] )\n", 312 | " stm_std = self.stimulus_noise_weight * eccentricity\n", 313 | " stimulus_noise = np.random.normal(0, stm_std, \n", 314 | " external_state['target'].shape)\n", 315 | " # stimulus is the external target location plus noise\n", 316 | " stm = external_state['target'] + stimulus_noise\n", 317 | " return stm, stm_std\n", 318 | "\n", 319 | " \n", 320 | " def _update_state_with_stimulus(self, stimulus, stimulus_std):\n", 321 | " posterior, posterior_std = self.bayes_update(stimulus, \n", 322 | " stimulus_std, \n", 323 | " self.internal_state['target'],\n", 324 | " self.internal_state['target_std'])\n", 325 | " self.internal_state['target'] = posterior\n", 326 | " self.internal_state['target_std'] = posterior_std\n", 327 | "\n", 328 | " def bayes_update(self, stimulus, stimulus_std, belief, belief_std):\n", 329 | " ''' A Bayes optimal function that integrates multiple stimuluss.\n", 330 | " The belief is the prior.'''\n", 331 | " z1, sigma1 = stimulus, stimulus_std\n", 332 | " z2, sigma2 = belief, belief_std\n", 333 | " w1 = sigma2**2 / (sigma1**2 + sigma2**2)\n", 334 | " w2 = sigma1**2 / (sigma1**2 + sigma2**2)\n", 335 | " posterior = w1*z1 + w2*z2\n", 336 | " posterior_std = np.sqrt( (sigma1**2 * sigma2**2)/(sigma1**2 + sigma2**2) )\n", 337 | " return posterior, posterior_std\n", 338 | " \n", 339 | " def _get_obs(self):\n", 340 | " # the Bayesian posterior has already been calculated so just return it.\n", 341 | " # also return the target_std so that the controller knows the uncertainty \n", 342 | " # of the observation.\n", 343 | " #return self.internal_state['target']\n", 344 | " return np.array([self.internal_state['target'][0],\n", 345 | " self.internal_state['target'][1],\n", 346 | " self.internal_state['target_std']])\n", 347 | " \n", 348 | " def _get_reward(self):\n", 349 | " distance = gazetools.get_distance(self.internal_state['fixation'], \n", 350 | " self.internal_state['target'])\n", 351 | " \n", 352 | " if distance < self.internal_state['width'] / 2:\n", 353 | " reward = 0\n", 354 | " else:\n", 355 | " reward = -distance # a much better model of the psychological reward function is possible.\n", 356 | " \n", 357 | " return reward" 358 | ] 359 | }, 360 | { 361 | "cell_type": "markdown", 362 | "id": "boolean-missile", 363 | "metadata": { 364 | "id": "boolean-missile" 365 | }, 366 | "source": [ 367 | "### Task environment\n", 368 | "\n", 369 | "In order to test the theory we need to define the task environment. \n", 370 | "\n", 371 | "The task environment allows the theory to make predictions for a particular task. The theory makes predictions for many more tasks. For example, adaptation to mixed target widths and distances." 372 | ] 373 | }, 374 | { 375 | "cell_type": "code", 376 | "execution_count": null, 377 | "id": "bacterial-archives", 378 | "metadata": { 379 | "executionInfo": { 380 | "elapsed": 472, 381 | "status": "ok", 382 | "timestamp": 1643209742334, 383 | "user": { 384 | "displayName": "Andrew Howes", 385 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64", 386 | "userId": "02694399383679444060" 387 | }, 388 | "user_tz": 0 389 | }, 390 | "id": "bacterial-archives" 391 | }, 392 | "outputs": [], 393 | "source": [ 394 | "class GazeTask():\n", 395 | " \n", 396 | " def __init__(self):\n", 397 | " self.target_width = 0.15\n", 398 | " self.target_loc_std = 0.3\n", 399 | "\n", 400 | " def reset_external_env(self):\n", 401 | " ''' The external_state includes the fixation and target location.\n", 402 | " Choose a new target location and reset to the first fixation location.'''\n", 403 | " \n", 404 | " def _get_new_target():\n", 405 | " x_target =np.clip(np.random.normal(0, self.target_loc_std),-1,1)\n", 406 | " y_target =np.clip(np.random.normal(0, self.target_loc_std),-1,1) \n", 407 | " return np.array( [x_target, y_target] )\n", 408 | " \n", 409 | " fx = np.array([-1,-1])\n", 410 | " tg = _get_new_target()\n", 411 | " self.external_state = {'fixation':fx, 'target':tg, 'width':self.target_width }\n", 412 | " \n", 413 | " def external_env(self, action):\n", 414 | " self.external_state['fixation'] = action\n", 415 | " \n", 416 | " # determine when the goal has been achieved.\n", 417 | " distance = gazetools.get_distance(self.external_state['fixation'], \n", 418 | " self.external_state['target'])\n", 419 | " if distance < self.external_state['width']/2 :\n", 420 | " done = True\n", 421 | " else:\n", 422 | " done = False\n", 423 | " \n", 424 | " return self.external_state, done\n", 425 | " " 426 | ] 427 | }, 428 | { 429 | "cell_type": "markdown", 430 | "id": "combined-alliance", 431 | "metadata": { 432 | "id": "combined-alliance" 433 | }, 434 | "source": [ 435 | "### Gym environment\n", 436 | "\n", 437 | "In order to find an optimal policy we use the theory and external environment to define a machine learning problem, here, making use of the framework defined by one specific library called gym.\n", 438 | "\n", 439 | "For further information see: https://gym.openai.com/\n", 440 | "\n", 441 | "gym.Env is a class provided by this library. Note that Env here refers to all of the components of the, including both internal and external environment, with the exception of the controller." 442 | ] 443 | }, 444 | { 445 | "cell_type": "code", 446 | "execution_count": null, 447 | "id": "applicable-sleeve", 448 | "metadata": { 449 | "executionInfo": { 450 | "elapsed": 231, 451 | "status": "ok", 452 | "timestamp": 1643209749681, 453 | "user": { 454 | "displayName": "Andrew Howes", 455 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64", 456 | "userId": "02694399383679444060" 457 | }, 458 | "user_tz": 0 459 | }, 460 | "id": "applicable-sleeve" 461 | }, 462 | "outputs": [], 463 | "source": [ 464 | "class GazeModel(gym.Env):\n", 465 | " \n", 466 | " def __init__(self):\n", 467 | " \n", 468 | " def default_box(x):\n", 469 | " return spaces.Box(low=-1, high=1, shape=(x, ), dtype=np.float64)\n", 470 | " \n", 471 | " self.GT = GazeTheory()\n", 472 | " self.TX = GazeTask() \n", 473 | " \n", 474 | " # Required by gym. These define the range of each variable.\n", 475 | " # Each action has an x,y coordinate therefore the box size is 2.\n", 476 | " # Each obs has a an x,y and an uncertainty therefore the box size is 3.\n", 477 | " self.action_space = default_box(2)\n", 478 | " self.observation_space = default_box(3)\n", 479 | " \n", 480 | " # max_fixations per episode. Used to curtail exploration early in training.\n", 481 | " self.max_steps = 500\n", 482 | " \n", 483 | " def reset(self):\n", 484 | " self.n_step = 0\n", 485 | " self.TX.reset_external_env()\n", 486 | " self.GT.reset_internal_env(self.TX.external_state)\n", 487 | " obs = self.GT.reset_internal_env( self.TX.external_state )\n", 488 | " return obs, {}\n", 489 | " \n", 490 | " def step(self, action):\n", 491 | " obs, reward, done = self.GT.step( self.TX, action )\n", 492 | " self.n_step+=1\n", 493 | "\n", 494 | " # give up if been looking for too long\n", 495 | " if self.n_step > self.max_steps:\n", 496 | " done = True\n", 497 | " \n", 498 | " info = self.get_info()\n", 499 | " truncated = False\n", 500 | " return obs, reward, done, truncated, info\n", 501 | " \n", 502 | " def get_info(self):\n", 503 | " return {'step': self.n_step,\n", 504 | " 'target_width': self.TX.target_width,\n", 505 | " 'target_x': self.TX.external_state['target'][0],\n", 506 | " 'target_y': self.TX.external_state['target'][1],\n", 507 | " 'fixate_x':self.TX.external_state['fixation'][0],\n", 508 | " 'fixate_y':self.TX.external_state['fixation'][1] }" 509 | ] 510 | }, 511 | { 512 | "cell_type": "markdown", 513 | "id": "complex-covering", 514 | "metadata": { 515 | "id": "complex-covering" 516 | }, 517 | "source": [ 518 | "### Test the model\n", 519 | "\n", 520 | "Step through the untrained model to check for simple bugs. More comprehensive tests needed." 521 | ] 522 | }, 523 | { 524 | "cell_type": "code", 525 | "execution_count": null, 526 | "id": "4ed00048", 527 | "metadata": { 528 | "colab": { 529 | "base_uri": "https://localhost:8080/" 530 | }, 531 | "executionInfo": { 532 | "elapsed": 534, 533 | "status": "ok", 534 | "timestamp": 1643209756680, 535 | "user": { 536 | "displayName": "Andrew Howes", 537 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64", 538 | "userId": "02694399383679444060" 539 | }, 540 | "user_tz": 0 541 | }, 542 | "id": "4ed00048", 543 | "outputId": "95f13fd2-30ad-43d9-8cab-1d127f6359d6" 544 | }, 545 | "outputs": [], 546 | "source": [ 547 | "model = GazeModel()\n", 548 | "\n", 549 | "model.reset()\n", 550 | "\n", 551 | "i=0\n", 552 | "done = False\n", 553 | "while not done:\n", 554 | " # make a step with a randomly sampled action\n", 555 | " obs, reward, done, truncated, info = model.step(model.action_space.sample())\n", 556 | " i+=1\n", 557 | "\n", 558 | "print(i)" 559 | ] 560 | }, 561 | { 562 | "cell_type": "markdown", 563 | "id": "cognitive-stevens", 564 | "metadata": { 565 | "id": "cognitive-stevens" 566 | }, 567 | "source": [ 568 | "### Train the model\n", 569 | "\n", 570 | "We can train the model to generate a controller.\n", 571 | "\n", 572 | "By plotting the learning curve we can see whether the performance improves with training and whether the model approaches an optimum performance. We are interested in approximately optimal performance, so if the training curve is not approaching asymptote then we need to train with more timesteps or revise the model.\n", 573 | "\n", 574 | "We can see that at first the model uses hundreds of fixations to find the target, this is because it has not yet learned to move the gaze in a way that is informed by the observation. As it learns to do this, it takes fewer steps to gaze at the target and its performance improves.\n", 575 | "\n", 576 | "If our problem definition is correct then the model will get more 'human-like' the more that it is trained. In other words, training makes it a better model of interaction.\n", 577 | "\n", 578 | "If we assume that people are computationally rational then the optimal solution to a cognitive problem predicts human behavior." 579 | ] 580 | }, 581 | { 582 | "cell_type": "code", 583 | "execution_count": null, 584 | "id": "7d0910ac", 585 | "metadata": { 586 | "colab": { 587 | "base_uri": "https://localhost:8080/", 588 | "height": 299 589 | }, 590 | "executionInfo": { 591 | "elapsed": 66705, 592 | "status": "ok", 593 | "timestamp": 1643209827031, 594 | "user": { 595 | "displayName": "Andrew Howes", 596 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64", 597 | "userId": "02694399383679444060" 598 | }, 599 | "user_tz": 0 600 | }, 601 | "id": "7d0910ac", 602 | "outputId": "d927873a-e745-4931-d6b8-fb6f3a668245" 603 | }, 604 | "outputs": [], 605 | "source": [ 606 | "timesteps = 200000\n", 607 | "#timesteps = 50000\n", 608 | "\n", 609 | "controller = gazetools.train(model, timesteps)\n", 610 | "gazetools.plot_learning_curve()" 611 | ] 612 | }, 613 | { 614 | "cell_type": "markdown", 615 | "id": "japanese-georgia", 616 | "metadata": { 617 | "id": "japanese-georgia" 618 | }, 619 | "source": [ 620 | "### Run the model for N trials\n", 621 | "Run the trained model and save a trace of each episode to csv file. " 622 | ] 623 | }, 624 | { 625 | "cell_type": "code", 626 | "execution_count": null, 627 | "id": "after-evans", 628 | "metadata": { 629 | "colab": { 630 | "base_uri": "https://localhost:8080/", 631 | "height": 35 632 | }, 633 | "executionInfo": { 634 | "elapsed": 162308, 635 | "status": "ok", 636 | "timestamp": 1643210003598, 637 | "user": { 638 | "displayName": "Andrew Howes", 639 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64", 640 | "userId": "02694399383679444060" 641 | }, 642 | "user_tz": 0 643 | }, 644 | "id": "after-evans", 645 | "outputId": "838f8f2b-9414-454a-8700-acfd39062bb8" 646 | }, 647 | "outputs": [], 648 | "source": [ 649 | "data = gazetools.run_model( model, controller, 100, 'behaviour_trace.csv' )\n", 650 | "\n", 651 | "gazetools.animate_multiple_episodes(data, n=30)\n" 652 | ] 653 | }, 654 | { 655 | "cell_type": "markdown", 656 | "id": "b6c4c96f", 657 | "metadata": {}, 658 | "source": [] 659 | }, 660 | { 661 | "cell_type": "markdown", 662 | "id": "weekly-headquarters", 663 | "metadata": { 664 | "id": "weekly-headquarters" 665 | }, 666 | "source": [ 667 | "### References\n", 668 | "Chen, X., Acharya, A., Oulasvirta, A., & Howes, A. (2021, May). An adaptive model of gaze-based selection. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-11)." 669 | ] 670 | } 671 | ], 672 | "metadata": { 673 | "colab": { 674 | "collapsed_sections": [], 675 | "name": "gaze-based-interaction.ipynb", 676 | "provenance": [] 677 | }, 678 | "kernelspec": { 679 | "display_name": "Python 3 (ipykernel)", 680 | "language": "python", 681 | "name": "python3" 682 | }, 683 | "language_info": { 684 | "codemirror_mode": { 685 | "name": "ipython", 686 | "version": 3 687 | }, 688 | "file_extension": ".py", 689 | "mimetype": "text/x-python", 690 | "name": "python", 691 | "nbconvert_exporter": "python", 692 | "pygments_lexer": "ipython3", 693 | "version": "3.9.5" 694 | } 695 | }, 696 | "nbformat": 4, 697 | "nbformat_minor": 5 698 | } 699 | -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/gaze_based_interaction.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "id": "3e0c5f5d", 7 | "metadata": {}, 8 | "source": [ 9 | "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/02-DeepRL/Gaze_Based_Interaction/gaze_based_interaction.ipynb)\n" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "id": "fitted-component", 15 | "metadata": { 16 | "id": "fitted-component" 17 | }, 18 | "source": [ 19 | "# A cognitive model of gaze-based interaction\n", 20 | "\n", 21 | "Some cognitive models describe behaviour and others predict it. Models that predict behaviour must be capable of generating output without that output being part of the input. In this notebook we demonstrate this property for a model of eye movements. The model is a reinforcement learner that is trained by performing hundreds of thousands of simulated eye movements in search of a target of varying size and distance. The model predicts how many eye movements people will tend to make to find a target of a given size and distance. It predicts inhibition of return. It **predicts** Fitts's Law like behaviour and it predicts that the first eye movement will usually undershoot, rather than overshoot the target.\n", 22 | "\n", 23 | "\"Corati\n", 24 | "\n", 25 | "(source: Meyer, D. E., Abrams, R. A., Kornblum, S., Wright, C. E., & Keith Smith, J. E. (1988). Optimality in human motor performance: ideal control of rapid aimed movements. Psychological review, 95(3), 340.)\n", 26 | "\n", 27 | "Note that what might seem the obvious strategy -- aim for where you believe the target is -- is not necessarily the optimal strategy.\n", 28 | "\n", 29 | "### The task: Gaze-based interaction\n", 30 | "\n", 31 | "Gaze-based interaction is a mode of interaction in which users, including users with a range of movement disabilities, are able to indicate which display item they wish to select by fixating their eyes on it. Confirmation of selection is then made with one of a number of methods, including with a key press, or by holding the fixation for an extended duration of time. The model performs this task for targets with randomly selected location and size.\n", 32 | "\n", 33 | "\"Corati\n", 34 | "\n", 35 | "In the figure, the red lines represent saccades (eye movements). Multiple eye movements are needed to reach the target (the black circle).\n", 36 | "\n", 37 | "### Model architecture\n", 38 | "\n", 39 | "The model has a simple architecture that you have previously seen in the introduction. The figure is reproduced here:\n", 40 | "\n", 41 | "\"Corati\n", 42 | "\n", 43 | "- The **control** module makes decisions about where to move the eyes with the oculomotor system. Decisions are conditioned on the current belief about the location of the target.\n", 44 | "- The **motor** module implements decisions but it is bounded by Gaussian noise, which models noise in the human motor system.\n", 45 | "- The **environment** models the physics of the world and the task (the location of the target). Given a response from the motor system, a saccade is made to the aim point of the eyes, and a fixation is initiated.\n", 46 | "- The **perception** module simulates the human capacity to localize a target with foveated vision. The accuracy of the location estimate generated by perception is negatively affected by the eccentricity of the target from the current fixation location.\n", 47 | "- The **Memory** module stores a representation of the current state. Over the course of an episode a sequence of location estimates will be made. Humans are known to integrate these estimates into a single integrated representation of the location. People are known to do this optimally using a process that can be modelled with Bayesian state estimation. The state estimation constitutes a belief about the location of the target.\n", 48 | "- The **Utility** module calculates a reward signal given the current belief about the enviornment. The reward signal is used to train the controller.\n", 49 | "\n", 50 | "### Prerequisites\n", 51 | "\n", 52 | "Before proceeding with this notebook you should firrst review the notebooks on foveated vision and on Bayesian integration. These explain how the perception and memory modules work.\n", 53 | "\n" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "id": "a427be22", 59 | "metadata": { 60 | "id": "a427be22" 61 | }, 62 | "source": [ 63 | "### Machine learning\n", 64 | "\n", 65 | "In order to learn how to perform the task, the model uses implementations of reinforcement learning algorithms in PyTorch known as stable-baselines3." 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "id": "vL8yMY6q_Rd-", 72 | "metadata": { 73 | "colab": { 74 | "base_uri": "https://localhost:8080/" 75 | }, 76 | "id": "vL8yMY6q_Rd-", 77 | "outputId": "0f80da38-b03b-4b02-f041-58f91d3d7c18" 78 | }, 79 | "outputs": [], 80 | "source": [ 81 | "# Install stable_baselines3 and the gymnasium environment\n", 82 | "# This is a well known machine learning library that provides a suite of reinforcement learning methods.\n", 83 | "# Only needs to be run once\n", 84 | "\n", 85 | "!pip install --pre -U stable_baselines3\n", 86 | "\n", 87 | "import gymnasium as gym\n", 88 | "from gymnasium import spaces" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "id": "P44tljjb8hBC", 95 | "metadata": { 96 | "id": "P44tljjb8hBC" 97 | }, 98 | "outputs": [], 99 | "source": [ 100 | "# Load local modules\n", 101 | "# gazetools is a module that contains support functions for modeling gaze-based interaction.\n", 102 | "# the code below makes use of them but we do not need to understand how they work in this tutorial.\n", 103 | "\n", 104 | "!wget https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/main/02-DeepRL/Gaze_Based_Interaction/gazetools.py\n", 105 | "\n", 106 | "import gazetools" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "id": "obvious-point", 113 | "metadata": { 114 | "id": "obvious-point" 115 | }, 116 | "outputs": [], 117 | "source": [ 118 | "# load required standard modules and configure matplotlib\n", 119 | "\n", 120 | "import numpy as np\n", 121 | "import math\n", 122 | "import matplotlib.pyplot as plt\n", 123 | "import sys\n", 124 | "\n", 125 | "import matplotlib as mpl\n", 126 | "%matplotlib inline\n", 127 | "mpl.style.use('ggplot')" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "id": "injured-leadership", 133 | "metadata": { 134 | "id": "injured-leadership" 135 | }, 136 | "source": [ 137 | "### Implementation of the Cognitive Architecture as a Python Class\n", 138 | "\n", 139 | "The first step to formalise the model architecture presented in the above figure. We do this by specifying a class of cognitive theories and will later define an instance of this class.\n", 140 | "\n", 141 | "The class has only a single method, which defines a cycle through the processes defined in the figure." 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "id": "monetary-ivory", 148 | "metadata": { 149 | "id": "monetary-ivory" 150 | }, 151 | "outputs": [], 152 | "source": [ 153 | "class CognitivePOMDP():\n", 154 | "\n", 155 | " def __init__(self):\n", 156 | " self.internal_state = {}\n", 157 | "\n", 158 | " def step(self, ext, action):\n", 159 | " ''' Define the cognitive architecture.'''\n", 160 | " self._update_state_with_action(action)\n", 161 | " response = self._get_response()\n", 162 | " external_state, done = ext.external_env(response)\n", 163 | " stimulus, stimulus_std = self._get_stimulus(ext.external_state)\n", 164 | " self._update_state_with_stimulus(stimulus, stimulus_std)\n", 165 | " obs = self._get_obs()\n", 166 | " reward = self._get_reward()\n", 167 | " return obs, reward, done" 168 | ] 169 | }, 170 | { 171 | "cell_type": "markdown", 172 | "id": "brazilian-timeline", 173 | "metadata": { 174 | "id": "brazilian-timeline" 175 | }, 176 | "source": [ 177 | "### A theory of gaze-based interaction\n", 178 | "\n", 179 | "Each of the entities in the cognitive architecture must be defined so as to make explicit our theory of gaze-based interaction. The theory makes the following assumptions:\n", 180 | "\n", 181 | "- The perception of the target location is corrupted by Gaussian noise in human vision.\n", 182 | "- The standard deviation of noise increases linearly with eccentricity from the fovea.\n", 183 | "- Sequences of stimuli are noisily perceived and optimally integrated.\n", 184 | "- Intended eye movements (oculomotor actions) are corrupted by signal dependent Gaussian noise to generate responses.\n", 185 | "\n", 186 | "These assumptions are further described in Chen et al. (2021)." 187 | ] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": null, 192 | "id": "accepted-reunion", 193 | "metadata": { 194 | "id": "accepted-reunion" 195 | }, 196 | "outputs": [], 197 | "source": [ 198 | "class GazeTheory(CognitivePOMDP):\n", 199 | "\n", 200 | " def __init__(self):\n", 201 | " ''' Initialise the theoretically motivated parameters.'''\n", 202 | " # weight eye movement noise with distance of saccade\n", 203 | " self.oculamotor_noise_weight = 0.01\n", 204 | " # weight noise with eccentricity\n", 205 | " self.stimulus_noise_weight = 0.09\n", 206 | " # step_cost for the reward function\n", 207 | " self.step_cost = -1\n", 208 | " # super.__init__()\n", 209 | "\n", 210 | " def reset_internal_env(self, external_state):\n", 211 | " ''' The internal state includes the fixation location, the latest estimate of\n", 212 | " the target location and the target uncertainty. Assumes that there is no\n", 213 | " uncertainty in the fixation location.\n", 214 | " Assumes that width is known. All numbers are on scale -1 to 1.\n", 215 | " The target_std represents the strength of the prior.'''\n", 216 | " self.internal_state = {'fixation': np.array([-1,-1]),\n", 217 | " 'target': np.array([0,0]),\n", 218 | " 'target_std': 0.1,\n", 219 | " 'width': external_state['width'],\n", 220 | " 'action': np.array([-1,-1])}\n", 221 | " return self._get_obs()\n", 222 | "\n", 223 | " def _update_state_with_action(self, action):\n", 224 | " self.internal_state['action'] = action\n", 225 | "\n", 226 | " def _get_response(self):\n", 227 | " ''' Take an action and add noise.'''\n", 228 | " # !!!! should take internal_state as parameter\n", 229 | " move_distance = gazetools.get_distance( self.internal_state['fixation'],\n", 230 | " self.internal_state['action'] )\n", 231 | "\n", 232 | " ocularmotor_noise = np.random.normal(0, self.oculamotor_noise_weight * move_distance,\n", 233 | " self.internal_state['action'].shape)\n", 234 | " # response is action plus noise\n", 235 | " response = self.internal_state['action'] + ocularmotor_noise\n", 236 | "\n", 237 | " # update the ocularmotor state (internal)\n", 238 | " self.internal_state['fixation'] = response\n", 239 | "\n", 240 | " # make an adjustment if response is out of range.\n", 241 | " response = np.clip(response,-1,1)\n", 242 | " return response\n", 243 | "\n", 244 | " def _get_stimulus(self, external_state):\n", 245 | " ''' define a psychologically plausible stimulus function in which acuity\n", 246 | " falls off with eccentricity.'''\n", 247 | " eccentricity = gazetools.get_distance( external_state['target'], external_state['fixation'] )\n", 248 | " stm_std = self.stimulus_noise_weight * eccentricity\n", 249 | " stimulus_noise = np.random.normal(0, stm_std,\n", 250 | " external_state['target'].shape)\n", 251 | " # stimulus is the external target location plus noise\n", 252 | " stm = external_state['target'] + stimulus_noise\n", 253 | " return stm, stm_std\n", 254 | "\n", 255 | "\n", 256 | " def _update_state_with_stimulus(self, stimulus, stimulus_std):\n", 257 | " posterior, posterior_std = self.bayes_update(stimulus,\n", 258 | " stimulus_std,\n", 259 | " self.internal_state['target'],\n", 260 | " self.internal_state['target_std'])\n", 261 | " self.internal_state['target'] = posterior\n", 262 | " self.internal_state['target_std'] = posterior_std\n", 263 | "\n", 264 | " def bayes_update(self, stimulus, stimulus_std, belief, belief_std):\n", 265 | " ''' A Bayes optimal function that integrates multiple stimului.\n", 266 | " The belief is the prior.'''\n", 267 | " z1, sigma1 = stimulus, stimulus_std\n", 268 | " z2, sigma2 = belief, belief_std\n", 269 | " w1 = sigma2**2 / (sigma1**2 + sigma2**2)\n", 270 | " w2 = sigma1**2 / (sigma1**2 + sigma2**2)\n", 271 | " posterior = w1*z1 + w2*z2\n", 272 | " posterior_std = np.sqrt( (sigma1**2 * sigma2**2)/(sigma1**2 + sigma2**2) )\n", 273 | " return posterior, posterior_std\n", 274 | "\n", 275 | " def _get_obs(self):\n", 276 | " # the Bayesian posterior has already been calculated so just return it.\n", 277 | " # also return the target_std so that the controller knows the uncertainty\n", 278 | " # of the observation.\n", 279 | " #return self.internal_state['target']\n", 280 | " return np.array([self.internal_state['target'][0],\n", 281 | " self.internal_state['target'][1],\n", 282 | " self.internal_state['target_std']])\n", 283 | "\n", 284 | " def _get_reward(self):\n", 285 | " distance = gazetools.get_distance(self.internal_state['fixation'],\n", 286 | " self.internal_state['target'])\n", 287 | "\n", 288 | " if distance < self.internal_state['width'] / 2:\n", 289 | " reward = 0\n", 290 | " else:\n", 291 | " reward = -distance # a much better model of the psychological reward function is possible.\n", 292 | "\n", 293 | " return reward" 294 | ] 295 | }, 296 | { 297 | "cell_type": "markdown", 298 | "id": "boolean-missile", 299 | "metadata": { 300 | "id": "boolean-missile" 301 | }, 302 | "source": [ 303 | "### Task environment\n", 304 | "\n", 305 | "In order to test the theory we need to define the task environment.\n", 306 | "\n", 307 | "The task environment allows the theory to make predictions for a particular task. The theory makes predictions for many more tasks. For example, adaptation to mixed target widths and distances." 308 | ] 309 | }, 310 | { 311 | "cell_type": "code", 312 | "execution_count": null, 313 | "id": "bacterial-archives", 314 | "metadata": { 315 | "id": "bacterial-archives" 316 | }, 317 | "outputs": [], 318 | "source": [ 319 | "class GazeTask():\n", 320 | "\n", 321 | " def __init__(self):\n", 322 | " self.target_width = 0.15\n", 323 | " self.target_loc_std = 0.3\n", 324 | "\n", 325 | " def reset_external_env(self):\n", 326 | " ''' The external_state includes the fixation and target location.\n", 327 | " Choose a new target location and reset fixation to the first fixation location.'''\n", 328 | "\n", 329 | " def _get_new_target():\n", 330 | " x_target =np.clip(np.random.normal(0, self.target_loc_std),-1,1)\n", 331 | " y_target =np.clip(np.random.normal(0, self.target_loc_std),-1,1)\n", 332 | " return np.array( [x_target, y_target] )\n", 333 | "\n", 334 | " fx = np.array([-1,-1])\n", 335 | " tg = _get_new_target()\n", 336 | " self.external_state = {'fixation':fx, 'target':tg, 'width':self.target_width }\n", 337 | "\n", 338 | " def external_env(self, action):\n", 339 | " self.external_state['fixation'] = action\n", 340 | "\n", 341 | " # determine when the goal has been achieved.\n", 342 | " distance = gazetools.get_distance(self.external_state['fixation'],\n", 343 | " self.external_state['target'])\n", 344 | " if distance < self.external_state['width']/2 :\n", 345 | " done = True\n", 346 | " else:\n", 347 | " done = False\n", 348 | "\n", 349 | " return self.external_state, done\n" 350 | ] 351 | }, 352 | { 353 | "cell_type": "markdown", 354 | "id": "combined-alliance", 355 | "metadata": { 356 | "id": "combined-alliance" 357 | }, 358 | "source": [ 359 | "### Gym (Gymnasium)\n", 360 | "\n", 361 | "In order to find an optimal policy we use the theory and external environment to define a machine learning problem, here, making use of the framework defined by one specific library called gym.\n", 362 | "\n", 363 | "For further information see: https://gymnasium.farama.org/\n", 364 | "\n", 365 | "gym.Env is a class provided by this library. Note that all of the modules of the cognitive architecture are part of gym.env except for the controller." 366 | ] 367 | }, 368 | { 369 | "cell_type": "code", 370 | "execution_count": null, 371 | "id": "applicable-sleeve", 372 | "metadata": { 373 | "colab": { 374 | "base_uri": "https://localhost:8080/" 375 | }, 376 | "id": "applicable-sleeve", 377 | "outputId": "f3e0499f-5ac2-4d2a-dcce-41131c66801b" 378 | }, 379 | "outputs": [], 380 | "source": [ 381 | "class GazeModel(gym.Env):\n", 382 | "\n", 383 | " def __init__(self):\n", 384 | "\n", 385 | " def default_box(x):\n", 386 | " return spaces.Box(low=-1, high=1, shape=(x, ), dtype=np.float64)\n", 387 | "\n", 388 | " self.GT = GazeTheory()\n", 389 | " self.TX = GazeTask()\n", 390 | "\n", 391 | " # Required by gym. These define the range of each variable.\n", 392 | " # Each action has an x,y coordinate therefore the box size is 2.\n", 393 | " # Each obs has a an x,y and an uncertainty therefore the box size is 3.\n", 394 | " self.action_space = default_box(2)\n", 395 | " self.observation_space = default_box(3)\n", 396 | "\n", 397 | " # max_fixations per episode. Used to curtail exploration early in training.\n", 398 | " self.max_steps = 500\n", 399 | "\n", 400 | " def reset(self, seed=None, options=None):\n", 401 | " ''' reset the model.'''\n", 402 | " self.n_step = 0\n", 403 | " self.TX.reset_external_env()\n", 404 | " self.GT.reset_internal_env(self.TX.external_state)\n", 405 | " obs = self.GT.reset_internal_env( self.TX.external_state )\n", 406 | " return obs, {}\n", 407 | "\n", 408 | " def step(self, action):\n", 409 | " ''' Step through one cycle of the model.'''\n", 410 | " obs, reward, done = self.GT.step( self.TX, action )\n", 411 | " self.n_step+=1\n", 412 | "\n", 413 | " # give up if been looking for too long\n", 414 | " if self.n_step > self.max_steps:\n", 415 | " done = True\n", 416 | "\n", 417 | " info = self.get_info()\n", 418 | " truncated = False\n", 419 | " return obs, reward, done, truncated, info\n", 420 | "\n", 421 | " def get_info(self):\n", 422 | " return {'step': self.n_step,\n", 423 | " 'target_width': self.TX.target_width,\n", 424 | " 'target_x': self.TX.external_state['target'][0],\n", 425 | " 'target_y': self.TX.external_state['target'][1],\n", 426 | " 'fixate_x':self.TX.external_state['fixation'][0],\n", 427 | " 'fixate_y':self.TX.external_state['fixation'][1] }" 428 | ] 429 | }, 430 | { 431 | "cell_type": "markdown", 432 | "id": "complex-covering", 433 | "metadata": { 434 | "id": "complex-covering" 435 | }, 436 | "source": [ 437 | "### Test the model\n", 438 | "\n", 439 | "Step through the untrained model to check for simple bugs. More comprehensive tests needed." 440 | ] 441 | }, 442 | { 443 | "cell_type": "code", 444 | "execution_count": null, 445 | "id": "4ed00048", 446 | "metadata": { 447 | "colab": { 448 | "base_uri": "https://localhost:8080/" 449 | }, 450 | "id": "4ed00048", 451 | "outputId": "e4395a7a-aa41-4e5c-9ab1-3e6d77a6876c" 452 | }, 453 | "outputs": [], 454 | "source": [ 455 | "model = GazeModel()\n", 456 | "\n", 457 | "model.reset()\n", 458 | "\n", 459 | "i=0\n", 460 | "done = False\n", 461 | "while not done:\n", 462 | " # make a step with a randomly sampled action\n", 463 | " obs, reward, done, truncated, info = model.step(model.action_space.sample())\n", 464 | " i+=1\n", 465 | "\n", 466 | "print(i)" 467 | ] 468 | }, 469 | { 470 | "cell_type": "markdown", 471 | "id": "cognitive-stevens", 472 | "metadata": { 473 | "id": "cognitive-stevens" 474 | }, 475 | "source": [ 476 | "### Train the model\n", 477 | "\n", 478 | "We can train the model to generate a policy for the controller.\n", 479 | "\n", 480 | "By plotting the learning curve we can see whether the performance improves with training and whether the model approaches an optimum performance. We are interested in approximately optimal performance, so if the training curve is not approaching asymptote then we need to train with more timesteps or revise the model.\n", 481 | "\n", 482 | "We can see that at first the model uses hundreds of fixations to find the target, this is because it has not yet learned to move the gaze in a way that is informed by the observation. As it learns to do this, it takes fewer steps to gaze at the target and its performance improves.\n", 483 | "\n", 484 | "If our problem definition is correct then the model will get more 'human-like' the more that it is trained. In other words, training makes it a better model of interaction.\n", 485 | "\n", 486 | "If we assume that people are computationally rational then the optimal solution to a cognitive problem predicts human behavior." 487 | ] 488 | }, 489 | { 490 | "cell_type": "code", 491 | "execution_count": null, 492 | "id": "7d0910ac", 493 | "metadata": { 494 | "colab": { 495 | "base_uri": "https://localhost:8080/", 496 | "height": 470 497 | }, 498 | "id": "7d0910ac", 499 | "outputId": "e01f85a6-70a8-4fb5-9bf2-a36469669684" 500 | }, 501 | "outputs": [], 502 | "source": [ 503 | "timesteps = 100000\n", 504 | "\n", 505 | "controller = gazetools.train(model, timesteps)" 506 | ] 507 | }, 508 | { 509 | "cell_type": "code", 510 | "execution_count": null, 511 | "id": "dc728251-2e31-4f0f-ae3d-9f92028a48db", 512 | "metadata": {}, 513 | "outputs": [], 514 | "source": [ 515 | "gazetools.plot_learning_curve()" 516 | ] 517 | }, 518 | { 519 | "cell_type": "markdown", 520 | "id": "ccba2f2c", 521 | "metadata": { 522 | "id": "ccba2f2c" 523 | }, 524 | "source": [ 525 | "### Increase timesteps\n", 526 | "\n", 527 | "100,000 timesteps is not enough to train this model. Try doubling the number of timesteps and train again." 528 | ] 529 | }, 530 | { 531 | "cell_type": "markdown", 532 | "id": "japanese-georgia", 533 | "metadata": { 534 | "id": "japanese-georgia" 535 | }, 536 | "source": [ 537 | "### Run the model for N trials\n", 538 | "Run the trained model, save a trace of each episode to csv file, and animate the results." 539 | ] 540 | }, 541 | { 542 | "cell_type": "code", 543 | "execution_count": null, 544 | "id": "after-evans", 545 | "metadata": { 546 | "colab": { 547 | "base_uri": "https://localhost:8080/", 548 | "height": 955 549 | }, 550 | "id": "after-evans", 551 | "outputId": "3c6adc93-94cd-4ec9-915e-0274b306e8b3" 552 | }, 553 | "outputs": [], 554 | "source": [ 555 | "data = gazetools.run_model( model, controller, 100, 'behaviour_trace.csv' )\n", 556 | "\n", 557 | "gazetools.animate_multiple_episodes(data, n=30)" 558 | ] 559 | }, 560 | { 561 | "cell_type": "markdown", 562 | "id": "b6c4c96f", 563 | "metadata": { 564 | "id": "b6c4c96f" 565 | }, 566 | "source": [ 567 | "### Exercises\n", 568 | "\n", 569 | "- Rerun the model with different parameter settings. Start by trying a different target size. What is the impact on the behaviour?\n", 570 | "- Discuss in groups how you would use the methods presented in the notebook to build a cognitive model of another task." 571 | ] 572 | }, 573 | { 574 | "cell_type": "markdown", 575 | "id": "53e77c9c", 576 | "metadata": { 577 | "id": "53e77c9c" 578 | }, 579 | "source": [ 580 | "### Discussion\n", 581 | "\n", 582 | "- The cognitive model that we have described above accurately predicts human gaze-based interaction performance (Chen et al., 2021).\n", 583 | "- It is an example of a **computationally rational** cognitive model. This is because the behaviour is predicted from an approximately optimal policy given hypothesised bounds on cognition.\n", 584 | "- It should be possible to find an approximately optimal policy using any reinforcement learning algorithm. The only difference that the algorithm will make is to the efficiency with which solution is found.\n", 585 | "- The separation of cognitive theory and reinforcement learning algorithm is achieved through the statement of the architecture as what is known as a belief-state Markov Decision Process (a **belief-state MDP**), which is a type of Partially Observable Markov Decision Process (POMDP).\n", 586 | "- Using reinforcement learning (RL) to model cognition with the approximately optimal policy contrast does not model the human learning process. For work that does use RL to model human learning see Daw and Dayan (2008).\n", 587 | "- A number of CHI papers have made use of this architecture. See Oulasvirta et al., 2022 for a review.\n", 588 | "- An important issue concerns how model parameters are fitted to data. See Keuralinen et al., 2023." 589 | ] 590 | }, 591 | { 592 | "cell_type": "markdown", 593 | "id": "weekly-headquarters", 594 | "metadata": { 595 | "id": "weekly-headquarters" 596 | }, 597 | "source": [ 598 | "### References\n", 599 | "Chen, X., Acharya, A., Oulasvirta, A., & Howes, A. (2021, May). An adaptive model of gaze-based selection. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-11).\n", 600 | "\n", 601 | "Chen, H., Chang, H. J., & Howes, A. (2021, May). Apparently Irrational Choice as Optimal Sequential Decision Making. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 1, pp. 792-800).\n", 602 | "\n", 603 | "Dayan, P., & Daw, N. D. (2008). Decision theory, reinforcement learning, and the brain. Cognitive, Affective, & Behavioral Neuroscience, 8(4), 429-453.\n", 604 | "\n", 605 | "Oulasvirta, A., Jokinen, J. P., & Howes, A. (2022, April). Computational rationality as a theory of interaction. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1-14).\n", 606 | "\n", 607 | "Keurulainen, A., Westerlund, I. R., Keurulainen, O., & Howes, A. (2023, April). Amortised Experimental Design and Parameter Estimation for User Models of Pointing. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (pp. 1-17)." 608 | ] 609 | } 610 | ], 611 | "metadata": { 612 | "colab": { 613 | "provenance": [] 614 | }, 615 | "kernelspec": { 616 | "display_name": "Python 3 (ipykernel)", 617 | "language": "python", 618 | "name": "python3" 619 | }, 620 | "language_info": { 621 | "codemirror_mode": { 622 | "name": "ipython", 623 | "version": 3 624 | }, 625 | "file_extension": ".py", 626 | "mimetype": "text/x-python", 627 | "name": "python", 628 | "nbconvert_exporter": "python", 629 | "pygments_lexer": "ipython3", 630 | "version": "3.12.3" 631 | } 632 | }, 633 | "nbformat": 4, 634 | "nbformat_minor": 5 635 | } 636 | -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/gazetools.py: -------------------------------------------------------------------------------- 1 | 2 | import os 3 | import csv 4 | 5 | import gymnasium as gym 6 | import numpy as np 7 | import matplotlib.pyplot as plt 8 | import pandas as pd 9 | import time 10 | from IPython import display 11 | 12 | from stable_baselines3 import PPO 13 | from stable_baselines3.common import results_plotter 14 | from stable_baselines3.common.monitor import Monitor 15 | from stable_baselines3.common.noise import NormalActionNoise 16 | from stable_baselines3.common.callbacks import BaseCallback 17 | from stable_baselines3.common.callbacks import CheckpointCallback 18 | from stable_baselines3.common.results_plotter import load_results, ts2xy, plot_results 19 | 20 | 21 | output_dir = 'output/' 22 | policy_file = 'policy' 23 | 24 | 25 | def train(env, timesteps): 26 | ''' 27 | ''' 28 | env = Monitor(env, output_dir) 29 | controller = PPO('MlpPolicy', env, verbose=0, clip_range=0.15) 30 | controller.learn(total_timesteps=int(timesteps)) 31 | controller.save(f'{output_dir}{policy_file}') 32 | print('Done training.') 33 | return controller 34 | 35 | ''' 36 | Plot learning curve 37 | ''' 38 | 39 | def plot_learning_curve(title='Learning Curve'): 40 | """ 41 | :param output_folder: (str) the save location of the results to plot 42 | :param title: (str) the title of the task to plot 43 | """ 44 | x, y = ts2xy(load_results(output_dir), 'timesteps') 45 | y = moving_average(y, window=100) 46 | # Truncate x 47 | #x = x[len(x) - len(y):] 48 | fig = plt.figure(title) 49 | plt.plot(y) 50 | plt.xlabel('Number of Timesteps') 51 | plt.ylabel('Rewards') 52 | 53 | def moving_average(values, window): 54 | """ 55 | Smooth values by doing a moving average 56 | :param values: (numpy array) 57 | :param window: (int) 58 | :return: (numpy array) 59 | """ 60 | weights = np.repeat(1.0, window) / window 61 | return np.convolve(values, weights, 'valid') 62 | 63 | ''' 64 | Get the Euclidean distance between points p and q 65 | ''' 66 | 67 | def get_distance(p,q): 68 | return np.sqrt(np.sum((p-q)**2)) 69 | 70 | ''' 71 | ''' 72 | 73 | def load_controller(): 74 | controller = PPO.load(f'{output_dir}{policy_file}') 75 | return controller 76 | 77 | ''' 78 | ''' 79 | 80 | def run_model(env, controller, n_episodes, filename): 81 | ''' 82 | run the model for n_episodes and save its behaviour in a csv file. 83 | Note that 'env' is a term used by Gym to describe everything but the controller. 84 | ''' 85 | max_episodes = 900000 86 | if n_episodes > max_episodes: 87 | print(f'We ask that you limit training to a max of {max_episodes} on the School of Computer Science AWS account.') 88 | print(f'If you want to run more training episodes then please do so on a local computer.') 89 | return 90 | 91 | result = [] 92 | # repeat for n episodes 93 | eps = 0 94 | while eps < n_episodes: 95 | done = False 96 | step = 0 97 | obs, _ = env.reset() 98 | # record the initial state 99 | info = env.get_info() 100 | info['episode'] = eps 101 | result.append(info) 102 | 103 | # repeat until the gaze is on the target. 104 | while not done: 105 | step+=1 106 | # get the next prediction action from the controller 107 | action, _ = controller.predict(obs,deterministic = True) 108 | obs, reward, done, _, info = env.step(action) 109 | info['episode'] = eps 110 | result.append(info) 111 | if done: 112 | eps+=1 113 | path = f'{output_dir}{filename}' 114 | df = pd.DataFrame(result) 115 | df.to_csv(path,index=False) 116 | return df 117 | 118 | ''' 119 | ''' 120 | 121 | def plot_gaze(gaze_x,gaze_y): 122 | plt.plot(gaze_x,gaze_y,'r+',markersize=20,linewidth=2) 123 | 124 | ''' 125 | ''' 126 | 127 | def update_display(gap_time): 128 | # update the display with a time step 129 | display.display(plt.gcf()) 130 | display.clear_output(wait=True) 131 | time.sleep(gap_time) 132 | 133 | ''' 134 | ''' 135 | 136 | def set_canvas(): 137 | time.sleep(2) 138 | #set the canvas 139 | plt.close() 140 | fig, ax = plt.subplots(figsize=(7,7)) # note we must use plt.subplots, not plt.subplot 141 | plt.xlim(-1,1) 142 | plt.ylim(-1,1) 143 | plt.gca().set_aspect('equal', adjustable='box') 144 | update_display(gap_time=1) 145 | return(ax) 146 | 147 | ''' 148 | ''' 149 | 150 | def animate_episode(ax, df_eps): 151 | gaze_x,gaze_y=df_eps.loc[0]['fixate_x'],df_eps.loc[0]['fixate_y'] 152 | for t in range(1,len(df_eps)): 153 | # each time step t 154 | 155 | if t==2: 156 | # target 157 | target_x,target_y=df_eps.loc[t]['target_x'],df_eps.loc[t]['target_y'] 158 | target_width=df_eps.loc[t]['target_width'] 159 | circle1 = plt.Circle((target_x,target_y), target_width/2, color='k') 160 | ax.add_patch(circle1) 161 | update_display(gap_time=0.5) 162 | 163 | new_gaze_x,new_gaze_y=df_eps.loc[t]['fixate_x'],df_eps.loc[t]['fixate_y'] 164 | plt.arrow(gaze_x,gaze_y, new_gaze_x-gaze_x,new_gaze_y-gaze_y,head_width=0.05, 165 | length_includes_head=True,linestyle='-',color='r') 166 | plot_gaze(new_gaze_x,new_gaze_y) 167 | update_display(gap_time=0.5) 168 | 169 | # new gaze becomes the current gaze 170 | gaze_x,gaze_y=new_gaze_x,new_gaze_y 171 | 172 | ''' 173 | ''' 174 | 175 | def animate_multiple_episodes(data, n): 176 | for eps in range(n): 177 | # each episode 178 | ax = set_canvas() 179 | 180 | # behaviour data for each episode 181 | df_eps=data.loc[data['episode']==eps] 182 | df_eps.reset_index(drop=True, inplace=True) 183 | 184 | # truncate the length of the episode if it is too long. 185 | if len(df_eps) > 5: 186 | df_eps = df_eps[0:5] 187 | 188 | animate_episode(ax, df_eps) 189 | 190 | -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/image/cog_arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/cog_arch.png -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/image/cognitive_POMDP.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/cognitive_POMDP.png -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/image/gaze_task.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/gaze_task.png -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/image/internal_env.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/internal_env.png -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/image/sub_movements.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/sub_movements.png -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/image/time_v_target_size.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/time_v_target_size.png -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/image/visual_acuity.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/visual_acuity.png -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/output/behaviour_trace.csv: -------------------------------------------------------------------------------- 1 | step,target_width,target_x,target_y,fixate_x,fixate_y,episode 2 | 0.0,0.15,0.06414962203309098,-0.09282954521216029,-1.0,-1.0,0.0 3 | 1.0,0.15,0.06414962203309098,-0.09282954521216029,0.022952957267056928,-0.006057112040038275,0.0 4 | 2.0,0.15,0.06414962203309098,-0.09282954521216029,0.04155284368127787,-0.09377532502789823,0.0 5 | 0.0,0.15,-0.6019876004334083,-0.01653672263142971,-1.0,-1.0,1.0 6 | 1.0,0.15,-0.6019876004334083,-0.01653672263142971,0.0517793655965969,-0.0027747333854521543,1.0 7 | 2.0,0.15,-0.6019876004334083,-0.01653672263142971,-0.46166395072500815,-0.04279898880266997,1.0 8 | 3.0,0.15,-0.6019876004334083,-0.01653672263142971,-0.6167801155204129,-0.01282450722378098,1.0 9 | 0.0,0.15,0.2735984614270531,0.07705367206811643,-1.0,-1.0,2.0 10 | 1.0,0.15,0.2735984614270531,0.07705367206811643,0.06467241975995563,0.01139736623678792,2.0 11 | 2.0,0.15,0.2735984614270531,0.07705367206811643,0.2567464278578047,0.09175435009612219,2.0 12 | 0.0,0.15,-0.4091416057153972,0.19017308200151792,-1.0,-1.0,3.0 13 | 1.0,0.15,-0.4091416057153972,0.19017308200151792,0.04194268164996753,0.01701461828107996,3.0 14 | 2.0,0.15,-0.4091416057153972,0.19017308200151792,-0.3241411211315401,0.2637026101099477,3.0 15 | 3.0,0.15,-0.4091416057153972,0.19017308200151792,-0.44797239496562263,0.251710953880758,3.0 16 | 0.0,0.15,-0.43270166291918166,0.17741346914408665,-1.0,-1.0,4.0 17 | 1.0,0.15,-0.43270166291918166,0.17741346914408665,0.06365889434067198,0.008223384887533575,4.0 18 | 2.0,0.15,-0.43270166291918166,0.17741346914408665,-0.2806227767187054,0.2571779935728619,4.0 19 | 3.0,0.15,-0.43270166291918166,0.17741346914408665,-0.4115424460223932,0.2277943909412183,4.0 20 | 0.0,0.15,-0.23129327102684388,0.4138951414435922,-1.0,-1.0,5.0 21 | 1.0,0.15,-0.23129327102684388,0.4138951414435922,0.0336275538898401,0.009590282598844277,5.0 22 | 2.0,0.15,-0.23129327102684388,0.4138951414435922,-0.15376627437176435,0.3822021336329997,5.0 23 | 3.0,0.15,-0.23129327102684388,0.4138951414435922,-0.2845421921926004,0.4287584537408162,5.0 24 | 0.0,0.15,0.13119399718688762,-0.2760649421438989,-1.0,-1.0,6.0 25 | 1.0,0.15,0.13119399718688762,-0.2760649421438989,0.05725105287474052,-0.018994036110992575,6.0 26 | 2.0,0.15,0.13119399718688762,-0.2760649421438989,0.1561455574500748,-0.28200717375522844,6.0 27 | 0.0,0.15,0.05492472079290004,-0.1091640358530668,-1.0,-1.0,7.0 28 | 1.0,0.15,0.05492472079290004,-0.1091640358530668,0.07426093370549192,0.0007454376495246442,7.0 29 | 2.0,0.15,0.05492472079290004,-0.1091640358530668,0.039504127171326134,-0.1235305008939485,7.0 30 | 0.0,0.15,-0.09426987324585008,0.44388258583777,-1.0,-1.0,8.0 31 | 1.0,0.15,-0.09426987324585008,0.44388258583777,0.02245436246505295,0.0006928245864493396,8.0 32 | 2.0,0.15,-0.09426987324585008,0.44388258583777,-0.1585791253892003,0.46898011871891543,8.0 33 | 0.0,0.15,0.1788326944590339,-0.10607480320702313,-1.0,-1.0,9.0 34 | 1.0,0.15,0.1788326944590339,-0.10607480320702313,0.03230664141133567,0.0164669214868451,9.0 35 | 2.0,0.15,0.1788326944590339,-0.10607480320702313,0.20853109964240588,-0.1549831528301417,9.0 36 | 0.0,0.15,0.297092196113112,0.21844844944428302,-1.0,-1.0,10.0 37 | 1.0,0.15,0.297092196113112,0.21844844944428302,0.040204626309107525,-0.008753829929950116,10.0 38 | 2.0,0.15,0.297092196113112,0.21844844944428302,0.22202757522189964,0.21357752121546333,10.0 39 | 3.0,0.15,0.297092196113112,0.21844844944428302,0.3014779727637586,0.23356455635149298,10.0 40 | 0.0,0.15,-0.0010135714283298935,0.03252154433349142,-1.0,-1.0,11.0 41 | 1.0,0.15,-0.0010135714283298935,0.03252154433349142,0.032673756789997545,0.006636726653141751,11.0 42 | 0.0,0.15,0.14311920421285432,-0.1904122733975137,-1.0,-1.0,12.0 43 | 1.0,0.15,0.14311920421285432,-0.1904122733975137,0.05320920868652877,0.01192599646948581,12.0 44 | 2.0,0.15,0.14311920421285432,-0.1904122733975137,0.17039800284622325,-0.2301220193302062,12.0 45 | 0.0,0.15,0.1304458056773149,-0.17277410465817883,-1.0,-1.0,13.0 46 | 1.0,0.15,0.1304458056773149,-0.17277410465817883,0.07206775310725205,0.0022185344400437314,13.0 47 | 2.0,0.15,0.1304458056773149,-0.17277410465817883,0.15132024072142863,-0.1810170380982514,13.0 48 | 0.0,0.15,-0.2319148262633478,0.11551146203429558,-1.0,-1.0,14.0 49 | 1.0,0.15,-0.2319148262633478,0.11551146203429558,0.06500919418862863,0.02093501024815099,14.0 50 | 2.0,0.15,-0.2319148262633478,0.11551146203429558,-0.25764939483544036,0.11023141618499335,14.0 51 | 0.0,0.15,0.08798296512857404,0.5050904680819654,-1.0,-1.0,15.0 52 | 1.0,0.15,0.08798296512857404,0.5050904680819654,0.03074062566277619,-0.014660190308663086,15.0 53 | 2.0,0.15,0.08798296512857404,0.5050904680819654,0.061462926050982136,0.47385098253255187,15.0 54 | 0.0,0.15,-0.2199571192121488,-0.38194913727322655,-1.0,-1.0,16.0 55 | 1.0,0.15,-0.2199571192121488,-0.38194913727322655,0.04103194201179049,-0.016897207485399093,16.0 56 | 2.0,0.15,-0.2199571192121488,-0.38194913727322655,-0.1824121517619583,-0.35504430366653866,16.0 57 | 0.0,0.15,-0.40502354625131043,0.24221279961199477,-1.0,-1.0,17.0 58 | 1.0,0.15,-0.40502354625131043,0.24221279961199477,0.01366919697760631,-0.011032686269866872,17.0 59 | 2.0,0.15,-0.40502354625131043,0.24221279961199477,-0.31228562332639914,0.1944457467497777,17.0 60 | 3.0,0.15,-0.40502354625131043,0.24221279961199477,-0.43113638296124585,0.2593286077752522,17.0 61 | 0.0,0.15,-0.22727139645640565,0.2946231485409358,-1.0,-1.0,18.0 62 | 1.0,0.15,-0.22727139645640565,0.2946231485409358,0.0520387559850861,0.025569081145251568,18.0 63 | 2.0,0.15,-0.22727139645640565,0.2946231485409358,-0.19906948297016727,0.3292951359416989,18.0 64 | 0.0,0.15,0.38715871532518803,0.20774700755126518,-1.0,-1.0,19.0 65 | 1.0,0.15,0.38715871532518803,0.20774700755126518,0.05716638781815297,0.02865869820594034,19.0 66 | 2.0,0.15,0.38715871532518803,0.20774700755126518,0.34506664507958873,0.2264340116256558,19.0 67 | 0.0,0.15,0.051585093869399294,-0.09859287099233052,-1.0,-1.0,20.0 68 | 1.0,0.15,0.051585093869399294,-0.09859287099233052,0.031677345778908836,-0.011722727661730548,20.0 69 | 2.0,0.15,0.051585093869399294,-0.09859287099233052,0.051921909518558566,-0.11337538849596154,20.0 70 | 0.0,0.15,-0.35042118098373126,-0.08159096141110943,-1.0,-1.0,21.0 71 | 1.0,0.15,-0.35042118098373126,-0.08159096141110943,0.04256702571677419,-0.03724214483816485,21.0 72 | 2.0,0.15,-0.35042118098373126,-0.08159096141110943,-0.36780503506284273,-0.12048313941483238,21.0 73 | 0.0,0.15,-0.05847971133248091,0.007466396681718116,-1.0,-1.0,22.0 74 | 1.0,0.15,-0.05847971133248091,0.007466396681718116,0.05882039862008667,0.009462747740083951,22.0 75 | 2.0,0.15,-0.05847971133248091,0.007466396681718116,-0.0735318983679421,-0.022065207603506876,22.0 76 | 0.0,0.15,-0.7121889593373226,-0.058211251887428206,-1.0,-1.0,23.0 77 | 1.0,0.15,-0.7121889593373226,-0.058211251887428206,0.07833873456267743,-0.0046009149695391705,23.0 78 | 2.0,0.15,-0.7121889593373226,-0.058211251887428206,-0.4888168532941895,-0.01542165319999887,23.0 79 | 3.0,0.15,-0.7121889593373226,-0.058211251887428206,-0.7052765656466627,-0.05884400570490193,23.0 80 | 0.0,0.15,-0.06377558808898039,0.07697389227769467,-1.0,-1.0,24.0 81 | 1.0,0.15,-0.06377558808898039,0.07697389227769467,0.032571280308437985,0.0034459504458742023,24.0 82 | 2.0,0.15,-0.06377558808898039,0.07697389227769467,-0.05672195932675608,0.09931880624900916,24.0 83 | 0.0,0.15,0.7335238138165546,-0.13170960527105272,-1.0,-1.0,25.0 84 | 1.0,0.15,0.7335238138165546,-0.13170960527105272,0.06745996712968717,-0.006104395446774213,25.0 85 | 2.0,0.15,0.7335238138165546,-0.13170960527105272,0.5608453285367376,-0.12081348609434935,25.0 86 | 3.0,0.15,0.7335238138165546,-0.13170960527105272,0.7560028011382774,-0.16552934483847775,25.0 87 | 0.0,0.15,0.21092527942879483,0.05766350974855612,-1.0,-1.0,26.0 88 | 1.0,0.15,0.21092527942879483,0.05766350974855612,0.04813346464752367,0.004511532595652595,26.0 89 | 2.0,0.15,0.21092527942879483,0.05766350974855612,0.15622527683658335,0.10737691450154997,26.0 90 | 0.0,0.15,0.013798287743157896,0.33217787272568056,-1.0,-1.0,27.0 91 | 1.0,0.15,0.013798287743157896,0.33217787272568056,0.03886747129985962,0.024548304971569446,27.0 92 | 2.0,0.15,0.013798287743157896,0.33217787272568056,-0.01732608431268916,0.3563519665124801,27.0 93 | 0.0,0.15,-0.360303984705875,-0.327273325302741,-1.0,-1.0,28.0 94 | 1.0,0.15,-0.360303984705875,-0.327273325302741,0.02726670469832683,0.012457424701257254,28.0 95 | 2.0,0.15,-0.360303984705875,-0.327273325302741,-0.26902971168434164,-0.29556970814055217,28.0 96 | 3.0,0.15,-0.360303984705875,-0.327273325302741,-0.3729469247514502,-0.3427752180041761,28.0 97 | 0.0,0.15,-0.03209989163813316,-0.45575960437966573,-1.0,-1.0,29.0 98 | 1.0,0.15,-0.03209989163813316,-0.45575960437966573,0.011335510217212637,0.030859051994577974,29.0 99 | 2.0,0.15,-0.03209989163813316,-0.45575960437966573,-0.139917793287394,-0.40786515358813574,29.0 100 | 3.0,0.15,-0.03209989163813316,-0.45575960437966573,-0.07065421227561385,-0.4917718377430215,29.0 101 | 0.0,0.15,-0.11114769843081844,-0.25760476017786643,-1.0,-1.0,30.0 102 | 1.0,0.15,-0.11114769843081844,-0.25760476017786643,0.046768882821610624,0.012103487681706472,30.0 103 | 2.0,0.15,-0.11114769843081844,-0.25760476017786643,-0.06716154994302884,-0.2680336906946792,30.0 104 | 0.0,0.15,-0.21572122240217956,0.21900657337141247,-1.0,-1.0,31.0 105 | 1.0,0.15,-0.21572122240217956,0.21900657337141247,0.04701869869948062,-0.005015993217710385,31.0 106 | 2.0,0.15,-0.21572122240217956,0.21900657337141247,-0.17726392496954035,0.28666828358076674,31.0 107 | 3.0,0.15,-0.21572122240217956,0.21900657337141247,-0.26830770770877527,0.2567624560905446,31.0 108 | 0.0,0.15,-0.2557688203408042,-0.3705300094035497,-1.0,-1.0,32.0 109 | 1.0,0.15,-0.2557688203408042,-0.3705300094035497,0.022672953975151525,0.011432524554609176,32.0 110 | 2.0,0.15,-0.2557688203408042,-0.3705300094035497,-0.21830717914722544,-0.39557011550092075,32.0 111 | 0.0,0.15,0.3925558133595611,0.009645602631218825,-1.0,-1.0,33.0 112 | 1.0,0.15,0.3925558133595611,0.009645602631218825,0.033052474580954706,0.002330908544370104,33.0 113 | 2.0,0.15,0.3925558133595611,0.009645602631218825,0.4051381077328279,0.00606755656341242,33.0 114 | 0.0,0.15,-0.5792107364870995,-0.1341705550315782,-1.0,-1.0,34.0 115 | 1.0,0.15,-0.5792107364870995,-0.1341705550315782,0.002772622556956948,-0.00596026714047021,34.0 116 | 2.0,0.15,-0.5792107364870995,-0.1341705550315782,-0.4994376094388785,-0.1406418191286596,34.0 117 | 3.0,0.15,-0.5792107364870995,-0.1341705550315782,-0.5785026175998892,-0.155571987275992,34.0 118 | 0.0,0.15,-0.09905383104177455,-0.16520292858999563,-1.0,-1.0,35.0 119 | 1.0,0.15,-0.09905383104177455,-0.16520292858999563,0.049125101213618225,-0.01537779278047801,35.0 120 | 2.0,0.15,-0.09905383104177455,-0.16520292858999563,-0.10655875053079945,-0.12863714554675929,35.0 121 | 0.0,0.15,0.014167280348627517,0.15857296217323158,-1.0,-1.0,36.0 122 | 1.0,0.15,0.014167280348627517,0.15857296217323158,0.03305959117787621,0.02879766143838659,36.0 123 | 2.0,0.15,0.014167280348627517,0.15857296217323158,0.02258934805537644,0.17435283824891662,36.0 124 | 0.0,0.15,-0.21258332296495472,-0.2738291738166111,-1.0,-1.0,37.0 125 | 1.0,0.15,-0.21258332296495472,-0.2738291738166111,0.05040203028709987,0.0009078370570102637,37.0 126 | 2.0,0.15,-0.21258332296495472,-0.2738291738166111,-0.21231669611041493,-0.2624300954214877,37.0 127 | 0.0,0.15,-0.11325574379967678,0.2606798163870956,-1.0,-1.0,38.0 128 | 1.0,0.15,-0.11325574379967678,0.2606798163870956,0.050771828488389306,-0.029084077030268635,38.0 129 | 2.0,0.15,-0.11325574379967678,0.2606798163870956,-0.15466377415483434,0.2451767228104351,38.0 130 | 0.0,0.15,0.2970797829940354,0.06893641225287236,-1.0,-1.0,39.0 131 | 1.0,0.15,0.2970797829940354,0.06893641225287236,0.05791949596829962,0.024893731115873983,39.0 132 | 2.0,0.15,0.2970797829940354,0.06893641225287236,0.3283271125241888,0.051679102169388466,39.0 133 | 0.0,0.15,-0.2963203554072132,0.12079854442864445,-1.0,-1.0,40.0 134 | 1.0,0.15,-0.2963203554072132,0.12079854442864445,0.06434721731752746,-0.0072914479440132195,40.0 135 | 2.0,0.15,-0.2963203554072132,0.12079854442864445,-0.2630085542712542,0.14108980161523205,40.0 136 | 0.0,0.15,-0.26988059977344403,0.019232442339303834,-1.0,-1.0,41.0 137 | 1.0,0.15,-0.26988059977344403,0.019232442339303834,0.04012862463736115,0.018798609769400062,41.0 138 | 2.0,0.15,-0.26988059977344403,0.019232442339303834,-0.3091295268470044,0.046094511674736195,41.0 139 | 0.0,0.15,0.3758137523899551,0.4781124222105405,-1.0,-1.0,42.0 140 | 1.0,0.15,0.3758137523899551,0.4781124222105405,0.02122804053703171,0.016671632512660284,42.0 141 | 2.0,0.15,0.3758137523899551,0.4781124222105405,0.3552325628681667,0.3914842955693962,42.0 142 | 3.0,0.15,0.3758137523899551,0.4781124222105405,0.36926595423414255,0.5183266431142247,42.0 143 | 0.0,0.15,0.15222996964533295,0.32291035215543207,-1.0,-1.0,43.0 144 | 1.0,0.15,0.15222996964533295,0.32291035215543207,0.037047387645389396,-0.017120727028373754,43.0 145 | 2.0,0.15,0.15222996964533295,0.32291035215543207,0.14624229001337818,0.3831583523828629,43.0 146 | 0.0,0.15,-0.08806591231709889,-0.025347802810632616,-1.0,-1.0,44.0 147 | 1.0,0.15,-0.08806591231709889,-0.025347802810632616,0.041929553923185,-0.005869781327381585,44.0 148 | 2.0,0.15,-0.08806591231709889,-0.025347802810632616,-0.10094349725108763,-0.014614378249899963,44.0 149 | 0.0,0.15,-0.12711103701755513,-0.17693277412880634,-1.0,-1.0,45.0 150 | 1.0,0.15,-0.12711103701755513,-0.17693277412880634,0.0593024206893812,0.004370286214580236,45.0 151 | 2.0,0.15,-0.12711103701755513,-0.17693277412880634,-0.12388865752662613,-0.18843359075306945,45.0 152 | 0.0,0.15,-0.08256924015915106,0.048593197245711324,-1.0,-1.0,46.0 153 | 1.0,0.15,-0.08256924015915106,0.048593197245711324,0.06810056120511035,-0.0024220661633051394,46.0 154 | 2.0,0.15,-0.08256924015915106,0.048593197245711324,-0.11214327627343487,0.09725742079152619,46.0 155 | 0.0,0.15,-0.13262781190697362,0.43200649111674316,-1.0,-1.0,47.0 156 | 1.0,0.15,-0.13262781190697362,0.43200649111674316,0.03844814118539968,-0.0027020399505069364,47.0 157 | 2.0,0.15,-0.13262781190697362,0.43200649111674316,-0.1190194025468621,0.3848054887445577,47.0 158 | 0.0,0.15,-0.11458794491331732,0.23435796026687644,-1.0,-1.0,48.0 159 | 1.0,0.15,-0.11458794491331732,0.23435796026687644,0.049030087262185926,0.011897053674317596,48.0 160 | 2.0,0.15,-0.11458794491331732,0.23435796026687644,-0.16008155223306866,0.260335089112164,48.0 161 | 0.0,0.15,0.6050736122918101,-0.4415878085778648,-1.0,-1.0,49.0 162 | 1.0,0.15,0.6050736122918101,-0.4415878085778648,0.0456558282825393,0.003535315417539779,49.0 163 | 2.0,0.15,0.6050736122918101,-0.4415878085778648,0.45241791523245317,-0.43710174506214855,49.0 164 | 3.0,0.15,0.6050736122918101,-0.4415878085778648,0.5947286574836945,-0.4250176157386889,49.0 165 | 0.0,0.15,0.028534804504455533,-0.5029034689888454,-1.0,-1.0,50.0 166 | 1.0,0.15,0.028534804504455533,-0.5029034689888454,0.04562523035287361,0.017295290789046375,50.0 167 | 2.0,0.15,0.028534804504455533,-0.5029034689888454,0.05679213503125965,-0.4251238137994234,50.0 168 | 3.0,0.15,0.028534804504455533,-0.5029034689888454,0.06277804943206478,-0.5421370560026485,50.0 169 | 0.0,0.15,0.13469486289502433,-0.018689808144753378,-1.0,-1.0,51.0 170 | 1.0,0.15,0.13469486289502433,-0.018689808144753378,0.051618668485832554,0.02309904429515406,51.0 171 | 2.0,0.15,0.13469486289502433,-0.018689808144753378,0.16401229018374777,-0.0032743946462125695,51.0 172 | 0.0,0.15,0.03472171946092767,0.08520911109949268,-1.0,-1.0,52.0 173 | 1.0,0.15,0.03472171946092767,0.08520911109949268,0.052852156548921635,-0.007758706006038601,52.0 174 | 2.0,0.15,0.03472171946092767,0.08520911109949268,0.04807617403923386,0.12117233702376463,52.0 175 | 0.0,0.15,0.07281945112720674,0.24121167965817558,-1.0,-1.0,53.0 176 | 1.0,0.15,0.07281945112720674,0.24121167965817558,0.05093923716587817,0.009334745376458958,53.0 177 | 2.0,0.15,0.07281945112720674,0.24121167965817558,0.07407276337486085,0.28688718218489656,53.0 178 | 0.0,0.15,0.6036099272755281,-0.6510761013245265,-1.0,-1.0,54.0 179 | 1.0,0.15,0.6036099272755281,-0.6510761013245265,0.040313000829106375,-0.005629327786859507,54.0 180 | 2.0,0.15,0.6036099272755281,-0.6510761013245265,0.41624308883742517,-0.43746266644469783,54.0 181 | 3.0,0.15,0.6036099272755281,-0.6510761013245265,0.5084991891979846,-0.6335392718309133,54.0 182 | 4.0,0.15,0.6036099272755281,-0.6510761013245265,0.5986746484057115,-0.6673983535708621,54.0 183 | 0.0,0.15,-0.38249193660182096,0.11278538372898168,-1.0,-1.0,55.0 184 | 1.0,0.15,-0.38249193660182096,0.11278538372898168,0.041975369073639676,-0.005791739047306393,55.0 185 | 2.0,0.15,-0.38249193660182096,0.11278538372898168,-0.372428313960294,0.10727477484414108,55.0 186 | 0.0,0.15,-0.048559227233096615,0.2876997488282835,-1.0,-1.0,56.0 187 | 1.0,0.15,-0.048559227233096615,0.2876997488282835,0.06717131599300977,-0.004576395134306483,56.0 188 | 2.0,0.15,-0.048559227233096615,0.2876997488282835,-0.06676296914803206,0.356423500298756,56.0 189 | 0.0,0.15,0.24699293463725366,-0.08012061281579443,-1.0,-1.0,57.0 190 | 1.0,0.15,0.24699293463725366,-0.08012061281579443,0.03374253273613548,0.01957981216463243,57.0 191 | 2.0,0.15,0.24699293463725366,-0.08012061281579443,0.25313774410125695,-0.09707887478864087,57.0 192 | 0.0,0.15,0.14328171731660966,-0.04604751295962332,-1.0,-1.0,58.0 193 | 1.0,0.15,0.14328171731660966,-0.04604751295962332,0.04398067553636723,0.00343140325174186,58.0 194 | 2.0,0.15,0.14328171731660966,-0.04604751295962332,0.11684359820270622,-0.05904696259540202,58.0 195 | 0.0,0.15,0.2420063453315825,0.09815412226409745,-1.0,-1.0,59.0 196 | 1.0,0.15,0.2420063453315825,0.09815412226409745,0.04364679433184551,0.01562795969513265,59.0 197 | 2.0,0.15,0.2420063453315825,0.09815412226409745,0.23876502648537987,0.10025650797433755,59.0 198 | 0.0,0.15,-0.3054900856834516,-0.4105104403641334,-1.0,-1.0,60.0 199 | 1.0,0.15,-0.3054900856834516,-0.4105104403641334,0.042615296846733854,-0.0036176807661398762,60.0 200 | 2.0,0.15,-0.3054900856834516,-0.4105104403641334,-0.2501658227882057,-0.38577027139134656,60.0 201 | 0.0,0.15,0.2528246121566485,0.4156339727682298,-1.0,-1.0,61.0 202 | 1.0,0.15,0.2528246121566485,0.4156339727682298,0.049234816588684636,-0.031877686718629686,61.0 203 | 2.0,0.15,0.2528246121566485,0.4156339727682298,0.19588534506519198,0.507019792245592,61.0 204 | 3.0,0.15,0.2528246121566485,0.4156339727682298,0.2288160164352217,0.48210417668230565,61.0 205 | 0.0,0.15,0.5339529395844452,0.12095781254025344,-1.0,-1.0,62.0 206 | 1.0,0.15,0.5339529395844452,0.12095781254025344,0.026651969980413203,0.008018272567443741,62.0 207 | 2.0,0.15,0.5339529395844452,0.12095781254025344,0.47177530234383414,0.11531164094678442,62.0 208 | 0.0,0.15,-0.30830156560046995,-0.09624866343194168,-1.0,-1.0,63.0 209 | 1.0,0.15,-0.30830156560046995,-0.09624866343194168,0.04563699167426992,0.002493875851612919,63.0 210 | 2.0,0.15,-0.30830156560046995,-0.09624866343194168,-0.2362059119647002,-0.06291433272060422,63.0 211 | 3.0,0.15,-0.30830156560046995,-0.09624866343194168,-0.3244653765558659,-0.0808664575364132,63.0 212 | 0.0,0.15,0.43446472653753404,0.14515415096782353,-1.0,-1.0,64.0 213 | 1.0,0.15,0.43446472653753404,0.14515415096782353,0.06381318600934156,-0.005277850829486452,64.0 214 | 2.0,0.15,0.43446472653753404,0.14515415096782353,0.44561120727588255,0.1874217070655866,64.0 215 | 0.0,0.15,0.021414007866896772,0.11803432419507363,-1.0,-1.0,65.0 216 | 1.0,0.15,0.021414007866896772,0.11803432419507363,0.01926858492225006,0.025178849131179643,65.0 217 | 2.0,0.15,0.021414007866896772,0.11803432419507363,-0.003380475971317909,0.1274972348273112,65.0 218 | 0.0,0.15,-0.4028512560702591,0.12961671617906234,-1.0,-1.0,66.0 219 | 1.0,0.15,-0.4028512560702591,0.12961671617906234,0.03989866960398037,0.011084014697792199,66.0 220 | 2.0,0.15,-0.4028512560702591,0.12961671617906234,-0.3798878087165995,0.18070576812395367,66.0 221 | 0.0,0.15,0.47926072233611483,-0.4620852377768718,-1.0,-1.0,67.0 222 | 1.0,0.15,0.47926072233611483,-0.4620852377768718,0.047054844119704,0.02043600240537462,67.0 223 | 2.0,0.15,0.47926072233611483,-0.4620852377768718,0.4716500912877814,-0.3519639376054769,67.0 224 | 3.0,0.15,0.47926072233611483,-0.4620852377768718,0.5005765673758799,-0.48503510289990187,67.0 225 | 0.0,0.15,-0.3539999907854146,-0.01494454868179744,-1.0,-1.0,68.0 226 | 1.0,0.15,-0.3539999907854146,-0.01494454868179744,0.028634466653953847,0.0025977557824362397,68.0 227 | 2.0,0.15,-0.3539999907854146,-0.01494454868179744,-0.3107122399766935,-0.05514276781928499,68.0 228 | 0.0,0.15,0.230716284717682,0.13640014049354968,-1.0,-1.0,69.0 229 | 1.0,0.15,0.230716284717682,0.13640014049354968,0.053067792362395824,-0.023428836725051597,69.0 230 | 2.0,0.15,0.230716284717682,0.13640014049354968,0.18790464103588556,0.106833651342968,69.0 231 | 0.0,0.15,0.05690541663401461,-0.35686683304857864,-1.0,-1.0,70.0 232 | 1.0,0.15,0.05690541663401461,-0.35686683304857864,0.05860590345151194,0.020754495498751593,70.0 233 | 2.0,0.15,0.05690541663401461,-0.35686683304857864,-0.012516840782491687,-0.3114224250867912,70.0 234 | 3.0,0.15,0.05690541663401461,-0.35686683304857864,0.0787443081123331,-0.3662274477420743,70.0 235 | 0.0,0.15,-0.1897894059320973,1.0,-1.0,-1.0,71.0 236 | 1.0,0.15,-0.1897894059320973,1.0,0.05493091159721517,-0.0012256752407652834,71.0 237 | 2.0,0.15,-0.1897894059320973,1.0,-0.11494042183996693,0.5839021517669961,71.0 238 | 3.0,0.15,-0.1897894059320973,1.0,-0.12845720778926661,0.9287329591181651,71.0 239 | 4.0,0.15,-0.1897894059320973,1.0,-0.17908269638778357,0.9958940571187267,71.0 240 | 0.0,0.15,0.2894556977298768,-0.19042996524257982,-1.0,-1.0,72.0 241 | 1.0,0.15,0.2894556977298768,-0.19042996524257982,0.06591334129256077,-0.021672374850798422,72.0 242 | 2.0,0.15,0.2894556977298768,-0.19042996524257982,0.26557900013650754,-0.18567349069256536,72.0 243 | 0.0,0.15,0.2289136167829819,0.27547648426779997,-1.0,-1.0,73.0 244 | 1.0,0.15,0.2289136167829819,0.27547648426779997,0.04146521939364157,0.007863348624264105,73.0 245 | 2.0,0.15,0.2289136167829819,0.27547648426779997,0.21140341118622086,0.31235243592848866,73.0 246 | 0.0,0.15,0.29464148446892996,-0.09670023612505375,-1.0,-1.0,74.0 247 | 1.0,0.15,0.29464148446892996,-0.09670023612505375,0.03473276501386732,0.02366963215703282,74.0 248 | 2.0,0.15,0.29464148446892996,-0.09670023612505375,0.29607887783420983,-0.038247222756002,74.0 249 | 0.0,0.15,0.3057012956840737,-0.1806763284059769,-1.0,-1.0,75.0 250 | 1.0,0.15,0.3057012956840737,-0.1806763284059769,0.05035416607246748,-0.00352481742655003,75.0 251 | 2.0,0.15,0.3057012956840737,-0.1806763284059769,0.35233367276661204,-0.17642769307522474,75.0 252 | 0.0,0.15,0.43012519249758574,0.22795864779806024,-1.0,-1.0,76.0 253 | 1.0,0.15,0.43012519249758574,0.22795864779806024,0.04891613582403732,0.004578535300660017,76.0 254 | 2.0,0.15,0.43012519249758574,0.22795864779806024,0.38309366193720434,0.18223495814373736,76.0 255 | 0.0,0.15,-0.13013651525088213,0.20875569325155852,-1.0,-1.0,77.0 256 | 1.0,0.15,-0.13013651525088213,0.20875569325155852,0.0278877339227807,0.010394419962968885,77.0 257 | 2.0,0.15,-0.13013651525088213,0.20875569325155852,-0.12935723776418123,0.19684372102835468,77.0 258 | 0.0,0.15,-0.45622768013490356,0.35583833588891983,-1.0,-1.0,78.0 259 | 1.0,0.15,-0.45622768013490356,0.35583833588891983,0.052425303454858484,0.012134049029832082,78.0 260 | 2.0,0.15,-0.45622768013490356,0.35583833588891983,-0.2880896228012208,0.2902996644808326,78.0 261 | 3.0,0.15,-0.45622768013490356,0.35583833588891983,-0.44996333641647984,0.35804634576498834,78.0 262 | 0.0,0.15,-0.34682336048798984,-0.26268658735084593,-1.0,-1.0,79.0 263 | 1.0,0.15,-0.34682336048798984,-0.26268658735084593,0.07159180011960406,-0.0033328598431632073,79.0 264 | 2.0,0.15,-0.34682336048798984,-0.26268658735084593,-0.2825572537592489,-0.21329305194325823,79.0 265 | 3.0,0.15,-0.34682336048798984,-0.26268658735084593,-0.3688974947478353,-0.2998556158160257,79.0 266 | 0.0,0.15,-0.07870547938499006,-0.3590920891782355,-1.0,-1.0,80.0 267 | 1.0,0.15,-0.07870547938499006,-0.3590920891782355,0.05104582879604966,-0.0044563645709609885,80.0 268 | 2.0,0.15,-0.07870547938499006,-0.3590920891782355,-0.07419677030806847,-0.3343262643939309,80.0 269 | 0.0,0.15,0.2595751172318591,-0.11938729575498783,-1.0,-1.0,81.0 270 | 1.0,0.15,0.2595751172318591,-0.11938729575498783,0.04395797955333259,0.025731629186670928,81.0 271 | 2.0,0.15,0.2595751172318591,-0.11938729575498783,0.24427088238909558,-0.18037443153479354,81.0 272 | 0.0,0.15,-0.4152503203327221,-0.3729230417884008,-1.0,-1.0,82.0 273 | 1.0,0.15,-0.4152503203327221,-0.3729230417884008,0.05659389401944776,0.0015146173653884172,82.0 274 | 2.0,0.15,-0.4152503203327221,-0.3729230417884008,-0.2907187899038383,-0.2575232744444416,82.0 275 | 3.0,0.15,-0.4152503203327221,-0.3729230417884008,-0.43542322006932327,-0.36194061745697864,82.0 276 | 0.0,0.15,0.06833655919544931,-0.3578982170855646,-1.0,-1.0,83.0 277 | 1.0,0.15,0.06833655919544931,-0.3578982170855646,0.062473579940861744,0.015880003671061238,83.0 278 | 2.0,0.15,0.06833655919544931,-0.3578982170855646,0.06477214586412991,-0.3608195398764706,83.0 279 | 0.0,0.15,-0.18782958245868078,-0.2957670484861216,-1.0,-1.0,84.0 280 | 1.0,0.15,-0.18782958245868078,-0.2957670484861216,0.04352090889416995,-0.01346503311007322,84.0 281 | 2.0,0.15,-0.18782958245868078,-0.2957670484861216,-0.14688455263980343,-0.33236018294895964,84.0 282 | 0.0,0.15,-0.4104674464924564,-0.24592297397902857,-1.0,-1.0,85.0 283 | 1.0,0.15,-0.4104674464924564,-0.24592297397902857,0.04062409915979944,0.023775673120202134,85.0 284 | 2.0,0.15,-0.4104674464924564,-0.24592297397902857,-0.3969725589743864,-0.23205898409435435,85.0 285 | 0.0,0.15,-0.0899391373110621,-0.24878143578794137,-1.0,-1.0,86.0 286 | 1.0,0.15,-0.0899391373110621,-0.24878143578794137,0.0384996928531391,0.007605025983544439,86.0 287 | 2.0,0.15,-0.0899391373110621,-0.24878143578794137,-0.10448857847222599,-0.320608550649896,86.0 288 | 0.0,0.15,0.22183678495821674,-0.044633705885785205,-1.0,-1.0,87.0 289 | 1.0,0.15,0.22183678495821674,-0.044633705885785205,0.06138592120157123,0.014220974837514366,87.0 290 | 2.0,0.15,0.22183678495821674,-0.044633705885785205,0.2275532498231114,-0.1003589818149725,87.0 291 | 0.0,0.15,0.17973235230787293,0.1054573166264953,-1.0,-1.0,88.0 292 | 1.0,0.15,0.17973235230787293,0.1054573166264953,0.04390686084806695,0.0068106848190078164,88.0 293 | 2.0,0.15,0.17973235230787293,0.1054573166264953,0.22745039782475637,0.10880135587843613,88.0 294 | 0.0,0.15,0.024232540309781267,0.08823578646862316,-1.0,-1.0,89.0 295 | 1.0,0.15,0.024232540309781267,0.08823578646862316,0.02177577870772335,-0.0023974348772158676,89.0 296 | 2.0,0.15,0.024232540309781267,0.08823578646862316,0.028704555081370504,0.11363150670307684,89.0 297 | 0.0,0.15,0.12510010345250125,0.5049281053802411,-1.0,-1.0,90.0 298 | 1.0,0.15,0.12510010345250125,0.5049281053802411,0.03159606715752105,-0.001350709820192308,90.0 299 | 2.0,0.15,0.12510010345250125,0.5049281053802411,0.16610943367172865,0.5622599427025267,90.0 300 | 0.0,0.15,0.37176927763744083,0.18149997923124098,-1.0,-1.0,91.0 301 | 1.0,0.15,0.37176927763744083,0.18149997923124098,0.030765377046042257,0.009001317167140309,91.0 302 | 2.0,0.15,0.37176927763744083,0.18149997923124098,0.3156905735020419,0.18738362679554607,91.0 303 | 0.0,0.15,0.35827533465789985,0.5231881817476288,-1.0,-1.0,92.0 304 | 1.0,0.15,0.35827533465789985,0.5231881817476288,0.03449691832105268,0.013570796050154075,92.0 305 | 2.0,0.15,0.35827533465789985,0.5231881817476288,0.27456614771874693,0.46944059683556405,92.0 306 | 3.0,0.15,0.35827533465789985,0.5231881817476288,0.3424637532830716,0.5725817368293623,92.0 307 | 0.0,0.15,-0.4749629527354867,0.21436051662154296,-1.0,-1.0,93.0 308 | 1.0,0.15,-0.4749629527354867,0.21436051662154296,0.04581524599597033,0.009300807472284733,93.0 309 | 2.0,0.15,-0.4749629527354867,0.21436051662154296,-0.3904651534498414,0.2106282446625461,93.0 310 | 3.0,0.15,-0.4749629527354867,0.21436051662154296,-0.49296192856726023,0.22788851750009345,93.0 311 | 0.0,0.15,0.2563971181361048,-0.45207121570246067,-1.0,-1.0,94.0 312 | 1.0,0.15,0.2563971181361048,-0.45207121570246067,0.062249544992495576,0.01421147584998032,94.0 313 | 2.0,0.15,0.2563971181361048,-0.45207121570246067,0.28873523938394974,-0.42431173724507887,94.0 314 | 0.0,0.15,-0.11699825099127519,-0.012616547848888224,-1.0,-1.0,95.0 315 | 1.0,0.15,-0.11699825099127519,-0.012616547848888224,0.036235154645072466,0.02240319662878835,95.0 316 | 2.0,0.15,-0.11699825099127519,-0.012616547848888224,-0.13635482594512197,0.023346386003769887,95.0 317 | 0.0,0.15,-0.563067614511279,-0.133422326825027,-1.0,-1.0,96.0 318 | 1.0,0.15,-0.563067614511279,-0.133422326825027,0.05633923033678578,0.01970467595343732,96.0 319 | 2.0,0.15,-0.563067614511279,-0.133422326825027,-0.415691901855508,-0.10306197502599378,96.0 320 | 3.0,0.15,-0.563067614511279,-0.133422326825027,-0.5576120003404873,-0.1320557464260784,96.0 321 | 0.0,0.15,0.09438061162636577,0.14078322003740132,-1.0,-1.0,97.0 322 | 1.0,0.15,0.09438061162636577,0.14078322003740132,0.04914396475444823,0.01293729776848159,97.0 323 | 2.0,0.15,0.09438061162636577,0.14078322003740132,0.11965079624613723,0.18327703015810212,97.0 324 | 0.0,0.15,0.3205021573192244,-0.31866902934058344,-1.0,-1.0,98.0 325 | 1.0,0.15,0.3205021573192244,-0.31866902934058344,0.030143631406051308,-0.014656394195799909,98.0 326 | 2.0,0.15,0.3205021573192244,-0.31866902934058344,0.2224616543168862,-0.2931398805230987,98.0 327 | 3.0,0.15,0.3205021573192244,-0.31866902934058344,0.31822088776472296,-0.3422053725624213,98.0 328 | 0.0,0.15,0.2818323846028262,0.19257608412474328,-1.0,-1.0,99.0 329 | 1.0,0.15,0.2818323846028262,0.19257608412474328,0.06266328374018314,-0.019065860245672105,99.0 330 | 2.0,0.15,0.2818323846028262,0.19257608412474328,0.2746391341886509,0.2463885597684051,99.0 331 | -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/output/monitor.csv: -------------------------------------------------------------------------------- 1 | #{"t_start": 1651051715.872973, "env_id": null} 2 | r,l,t 3 | -270.238114,197,0.126351 4 | -70.45687,41,0.149668 5 | -450.6688,331,0.377506 6 | -40.491495,27,0.401056 7 | -36.059652,31,0.424587 8 | -68.402355,33,0.449067 9 | -651.427712,501,0.731244 10 | -5.560864,4,0.733522 11 | -726.74203,501,0.99392 12 | -210.438871,246,1.139322 13 | -417.107543,259,2.040449 14 | -163.488867,90,2.095777 15 | -413.139406,286,2.253394 16 | -96.259316,102,2.311559 17 | -686.001824,426,2.548265 18 | -36.668556,28,2.563605 19 | -427.858066,318,2.738643 20 | -751.233692,501,3.011576 21 | -119.316984,103,3.067752 22 | -476.773689,328,4.013326 23 | -587.949785,409,4.243478 24 | -64.544827,34,4.261252 25 | -347.592196,233,4.389178 26 | -681.977659,501,4.659089 27 | -39.70289,27,4.674886 28 | -57.45577,33,4.693156 29 | -302.868888,246,4.829842 30 | -435.299536,310,5.777861 31 | -865.254316,501,6.063542 32 | -206.363768,158,6.158182 33 | -696.591703,501,6.446138 34 | -470.246127,261,6.590776 35 | -306.453634,201,6.704873 36 | -337.570682,362,6.912561 37 | -463.76346,468,7.953302 38 | -11.091099,9,7.958137 39 | -515.656413,325,8.143822 40 | -593.924865,343,8.328973 41 | -764.606413,403,8.537441 42 | -212.390134,155,8.617828 43 | -56.783953,48,8.642774 44 | -167.849232,127,8.709011 45 | -923.586156,501,9.723475 46 | -115.381516,88,9.770051 47 | -692.61731,501,10.034998 48 | -25.941193,18,10.044562 49 | -725.809703,482,10.31997 50 | -268.148157,350,10.500654 51 | -683.383136,501,11.511595 52 | -686.847391,501,11.769005 53 | -616.669334,501,12.044885 54 | -186.926686,101,12.098603 55 | -410.428062,242,12.240204 56 | -23.795001,13,12.247405 57 | -117.048887,58,12.278432 58 | -683.828573,501,13.307283 59 | -484.983503,386,13.507191 60 | -525.987955,432,13.73062 61 | -66.91409,50,13.756836 62 | -834.088138,501,14.02622 63 | -628.605142,501,14.300635 64 | -367.270673,344,15.233433 65 | -633.391248,501,15.493408 66 | -471.862407,315,15.658821 67 | -583.168004,501,15.917828 68 | -252.470917,156,16.00025 69 | -788.385194,501,17.104452 70 | -869.031968,501,17.387437 71 | -247.755699,236,17.512748 72 | -554.398024,501,17.775839 73 | -436.929084,501,18.050405 74 | -513.590731,313,19.005381 75 | -644.785843,496,19.274933 76 | -151.517457,161,19.374352 77 | -709.617392,501,19.640415 78 | -97.302595,72,19.677779 79 | -997.274582,501,19.953038 80 | -250.480285,217,20.845903 81 | -103.913595,65,20.879674 82 | -129.062583,105,20.934251 83 | -142.373265,72,20.971557 84 | -139.622501,78,21.012116 85 | -31.304996,34,21.029814 86 | -265.417721,204,21.134568 87 | -267.426301,166,21.249965 88 | -420.489312,313,21.423452 89 | -395.32882,254,21.557129 90 | -3.638601,3,21.558926 91 | -527.730012,385,21.761692 92 | -79.298416,44,21.784667 93 | -853.512807,501,22.79841 94 | -266.821066,247,22.925931 95 | -115.728642,68,22.960809 96 | -27.513952,23,22.973281 97 | -743.726516,501,23.233552 98 | -800.448276,501,23.491189 99 | -589.245595,501,23.750347 100 | -490.507758,308,24.65453 101 | -909.684089,470,24.903285 102 | -137.160766,122,24.966798 103 | -136.215509,124,25.031685 104 | -435.643075,258,25.170146 105 | -64.970162,47,25.19517 106 | -530.445917,344,25.373026 107 | -19.239661,19,25.382997 108 | -900.003598,501,26.395676 109 | -88.191976,53,26.423578 110 | -297.677491,217,26.536716 111 | -23.977437,18,26.546375 112 | -130.462261,97,26.596575 113 | -225.327652,220,26.709932 114 | -413.500658,369,26.900383 115 | -77.250564,58,26.930496 116 | -88.146758,56,26.95946 117 | -536.437296,402,27.167 118 | -5.064053,3,27.168782 119 | -207.727417,157,27.249876 120 | -1017.881826,501,28.237284 121 | -306.921896,258,28.370334 122 | -599.31209,501,28.629156 123 | -685.492016,501,28.888627 124 | -369.789108,225,29.004486 125 | -448.738238,501,29.994172 126 | -24.958855,15,30.002127 127 | -375.297178,284,30.148698 128 | -104.459979,72,30.186112 129 | -11.964718,13,30.192992 130 | -79.506423,46,30.216647 131 | -9.318598,7,30.2204 132 | -238.72573,193,30.31953 133 | -661.494081,501,30.57868 134 | -154.972209,96,30.62846 135 | -209.862287,157,30.709779 136 | -198.934958,244,30.835935 137 | -61.771382,31,30.852112 138 | -292.460423,197,30.954206 139 | -117.788986,91,31.734583 140 | -236.172908,124,31.799227 141 | -28.284777,24,31.811764 142 | -143.644375,114,31.870582 143 | -454.980379,297,32.02876 144 | -201.886001,162,32.11285 145 | -444.249692,219,32.234179 146 | -58.113063,43,32.256626 147 | -635.112842,428,32.477578 148 | -805.219098,501,32.736082 149 | -21.099919,17,32.74503 150 | -5.438016,5,32.747768 151 | -1128.80401,501,33.728304 152 | -294.507243,228,33.846657 153 | -31.774225,19,33.856771 154 | -683.94211,373,34.052945 155 | -85.592797,57,34.083025 156 | -217.012366,205,34.190326 157 | -726.560934,501,34.450016 158 | -417.775387,330,35.395153 159 | -6.336624,4,35.397486 160 | -30.662592,29,35.412612 161 | -245.292014,231,35.5324 162 | -713.621102,445,35.762142 163 | -231.620731,180,35.855667 164 | -222.690429,125,35.920385 165 | -102.781025,98,35.971138 166 | -253.187206,149,36.048243 167 | -94.243206,62,36.080373 168 | -115.859755,116,36.141463 169 | -451.575929,239,36.267143 170 | -20.113289,13,36.274027 171 | -298.460404,174,36.364168 172 | -575.523252,359,37.273887 173 | -147.530859,103,37.327122 174 | -563.0783,501,37.586466 175 | -402.632172,235,37.709309 176 | -514.287164,403,37.922127 177 | -387.283806,259,38.056219 178 | -231.749075,260,38.910821 179 | -456.299809,305,39.068392 180 | -410.415375,322,39.234573 181 | -352.316783,343,39.412163 182 | -75.116848,75,39.45099 183 | -303.791576,179,39.5431 184 | -643.682811,501,39.801513 185 | -443.261806,407,40.745171 186 | -317.704064,255,40.881954 187 | -432.956835,334,41.054438 188 | -189.132677,181,41.147997 189 | -1035.502595,501,41.407192 190 | -805.08485,501,41.665944 191 | -176.415396,132,41.734854 192 | -403.654102,291,42.63318 193 | -423.818276,434,42.856554 194 | -475.336539,349,43.036561 195 | -4.080923,4,43.038828 196 | -150.852508,83,43.081987 197 | -530.340171,501,43.340627 198 | -762.951706,378,43.535331 199 | -51.948013,50,44.289367 200 | -242.1572,166,44.375033 201 | -91.357708,65,44.40934 202 | -5.618529,5,44.412106 203 | -82.927803,47,44.436747 204 | -391.183684,245,44.563635 205 | -354.643619,241,44.688048 206 | -63.018276,38,44.707739 207 | -386.543786,344,44.884556 208 | -298.839204,200,44.988852 209 | -223.440882,166,45.074275 210 | -859.186875,501,45.333198 211 | -327.935273,168,46.140247 212 | -542.751992,501,46.399477 213 | -622.031937,399,46.605784 214 | -1089.394166,501,46.863352 215 | -143.512519,77,46.903026 216 | -285.352008,176,46.99401 217 | -608.80004,326,47.88989 218 | -41.278845,26,47.903778 219 | -573.983885,397,48.10836 220 | -22.793199,13,48.115264 221 | -496.469159,309,48.274795 222 | -737.621739,478,48.521487 223 | -235.202847,121,48.583923 224 | -694.884878,372,48.776056 225 | -377.053268,314,49.673651 226 | -560.061261,403,49.882402 227 | -511.953535,501,50.141807 228 | -65.172665,50,50.167615 229 | -404.84448,245,50.302659 230 | -90.536932,53,50.333036 231 | -488.327428,300,50.491577 232 | -319.27516,265,50.646296 233 | -354.305958,208,51.487087 234 | -297.069953,314,51.649384 235 | -3.936732,2,51.650627 236 | -71.701991,64,51.683701 237 | -9.730047,6,51.686969 238 | -882.081769,501,51.946964 239 | -148.432536,125,52.017169 240 | -173.665166,99,52.068771 241 | -345.257843,189,52.166533 242 | -24.031451,21,52.177608 243 | -480.069364,501,52.443967 244 | -189.52461,124,52.508083 245 | -262.018742,139,53.306486 246 | -182.435698,151,53.384191 247 | -671.736276,501,53.642752 248 | -333.472777,210,53.751138 249 | -683.660386,501,54.014385 250 | -314.596261,192,54.113867 251 | -273.901962,145,54.188488 252 | -168.131956,109,54.245066 253 | -1.828591,1,54.245774 254 | -263.462486,202,55.082513 255 | -210.324617,161,55.165542 256 | -121.74771,100,55.216918 257 | -563.595026,403,55.424777 258 | -145.824319,125,55.489594 259 | -1116.835224,501,55.748927 260 | -539.534915,501,56.007737 261 | -92.306932,83,56.05075 262 | -103.366579,57,56.080233 263 | -50.15664,34,56.097883 264 | -123.519471,73,56.864982 265 | -127.635617,119,56.926585 266 | -477.812803,251,57.056328 267 | -455.613034,334,57.229069 268 | -109.856141,76,57.268671 269 | -3.75146,3,57.270414 270 | -168.679902,110,57.328045 271 | -802.904051,501,57.587415 272 | -351.984541,249,57.716554 273 | -252.957009,292,57.866828 274 | -100.473388,78,57.907035 275 | -619.927454,340,58.811698 276 | -305.392081,171,58.899863 277 | -121.164008,96,58.949751 278 | -221.349855,129,59.016837 279 | -153.556921,119,59.078355 280 | -303.097736,287,59.22675 281 | -524.186145,501,59.4852 282 | -109.468051,58,59.515068 283 | -710.788964,444,60.487546 284 | -149.386425,148,60.564102 285 | -969.725995,501,60.82261 286 | -610.427908,501,61.080847 287 | -543.622891,438,61.30673 288 | -30.993787,28,61.321322 289 | -289.809657,213,61.431841 290 | -90.1746,75,61.470719 291 | -71.945301,56,61.500555 292 | -521.031288,387,62.444912 293 | -548.757956,304,62.60145 294 | -75.179505,52,62.628627 295 | -28.580685,21,62.6396 296 | -201.52732,103,62.692948 297 | -306.087152,205,62.79919 298 | -172.661651,96,62.84883 299 | -25.684064,19,62.858798 300 | -97.018955,44,62.881578 301 | -122.482675,96,62.931249 302 | -476.118654,258,63.064645 303 | -783.644057,433,63.288705 304 | -902.57989,501,64.265511 305 | -494.665819,238,64.388764 306 | -583.948686,468,64.630816 307 | -193.997325,308,64.793686 308 | -996.512255,501,65.052387 309 | -1231.665369,501,66.035455 310 | -363.617374,268,66.174089 311 | -445.838303,223,66.289533 312 | -729.60199,501,66.548133 313 | -473.51601,462,66.78628 314 | -30.895458,22,66.797897 315 | -111.852937,89,66.84375 316 | -558.064493,390,67.774793 317 | -21.871277,17,67.783736 318 | -264.608603,224,67.899299 319 | -655.66551,501,68.157769 320 | -207.217712,106,68.213306 321 | -165.940232,136,68.283434 322 | -143.161632,116,68.343706 323 | -159.489388,164,68.4337 324 | -473.571298,254,68.564858 325 | -142.008557,83,68.607805 326 | -167.248194,106,69.385032 327 | -539.720511,501,69.644215 328 | -40.995992,34,69.662717 329 | -332.566056,262,69.802239 330 | -799.222735,420,70.018402 331 | -106.354926,81,70.060902 332 | -14.220241,13,70.067844 333 | -79.698876,80,70.108954 334 | -84.838758,74,70.147464 335 | -315.054589,183,70.241472 336 | -163.881898,117,70.301802 337 | -483.657473,348,71.210978 338 | -605.93902,470,71.454153 339 | -367.485854,342,71.629674 340 | -179.130894,121,71.692191 341 | -828.842084,501,71.957032 342 | -988.919924,501,72.220826 343 | -542.026966,419,73.167031 344 | -457.748008,319,73.332162 345 | -284.786987,200,73.435587 346 | -519.097219,294,73.587529 347 | -713.859824,496,73.843223 348 | -56.269814,34,73.861081 349 | -123.790402,103,73.914311 350 | -87.996351,103,73.9679 351 | -385.043821,249,74.820072 352 | -657.911323,457,75.057352 353 | -200.138519,124,75.121424 354 | -643.586353,501,75.379185 355 | -68.547897,63,75.411848 356 | -217.525292,126,75.477393 357 | -564.838947,501,75.736238 358 | -97.21376,85,75.780316 359 | -886.64754,439,76.73552 360 | -351.442165,282,76.880911 361 | -133.061046,78,76.921094 362 | -41.755196,31,76.937195 363 | -95.475104,83,76.980063 364 | -139.162686,102,77.032542 365 | -15.608,10,77.037832 366 | -442.081879,407,77.247784 367 | -512.047465,341,77.424088 368 | -751.501455,501,78.432001 369 | -573.584814,367,78.621646 370 | -115.236664,62,78.653747 371 | -648.749659,501,78.912568 372 | -179.929357,89,78.958783 373 | -70.709235,44,78.982129 374 | -36.811894,25,78.99518 375 | -95.76711,64,79.028412 376 | -203.878792,134,79.09777 377 | -88.419949,72,79.134931 378 | -264.77556,121,79.197884 379 | -530.869704,330,79.369119 380 | -60.840846,35,79.387333 381 | -238.134076,160,80.203963 382 | -75.772265,53,80.23132 383 | -12.094617,9,80.236113 384 | -334.654188,246,80.362997 385 | -410.236705,288,80.510792 386 | -397.070975,314,80.673282 387 | -128.643713,90,80.720599 388 | -101.852827,85,80.764327 389 | -73.534807,56,80.793203 390 | -776.267783,353,80.978849 391 | -173.057712,106,81.0337 392 | -65.643663,47,81.058004 393 | -500.696606,286,81.206163 394 | -275.017173,341,82.113731 395 | -242.561784,188,82.211193 396 | -770.569878,501,82.478559 397 | -140.289288,135,82.548226 398 | -571.57828,501,82.806174 399 | -313.211247,227,82.922984 400 | -355.094128,220,83.957002 401 | -563.463271,410,84.2157 402 | -11.916761,10,84.22705 403 | -109.314511,99,84.278551 404 | -248.962878,186,84.374854 405 | -411.976699,319,84.54087 406 | -125.023701,90,84.587692 407 | -753.489905,501,84.847509 408 | -326.282063,229,84.965594 409 | -164.118769,84,85.009422 410 | -152.589359,107,85.803168 411 | -17.788048,11,85.809021 412 | -51.933488,40,85.829858 413 | -281.263543,180,85.923587 414 | -422.514302,279,86.068558 415 | -104.951833,84,86.112193 416 | -89.88128,87,86.157333 417 | -276.453867,205,86.264256 418 | -196.749055,145,86.339292 419 | -1128.657978,501,86.599219 420 | -49.378145,24,86.611773 421 | -520.488802,389,86.812664 422 | -321.047723,209,87.739039 423 | -362.532505,255,87.879349 424 | -286.463145,249,88.008787 425 | -578.860104,501,88.268366 426 | -589.305299,501,88.528816 427 | -106.932005,62,88.561038 428 | -365.061244,226,88.677537 429 | -------------------------------------------------------------------------------- /02-DeepRL/Gaze_Based_Interaction/output/policy.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/output/policy.zip -------------------------------------------------------------------------------- /02-DeepRL/bayesian_state_estimation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "id": "77bab68c", 7 | "metadata": {}, 8 | "source": [ 9 | "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/02-DeepRL/bayesian_state_estimation.ipynb)\n" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "id": "15dc1fb8", 15 | "metadata": {}, 16 | "source": [ 17 | "# Bayesian estimation\n", 18 | "\n", 19 | "Andrew Howes\\\n", 20 | "School of Computer Science\\\n", 21 | "University of Birmingham\n", 22 | "\n", 23 | "What do people do with a sequence of observations (visual or otherwise)? Should they just use the most recent observation? Or perhaps the \"best\" observation? In fact, people do neither. Multiple sources of evidence suggests that people optimally integrate observations to generate a posterior estimate given all of the available information. One way to model this process is with Bayesian inference.\n", 24 | "\n", 25 | "This notebook gives one simple worked example of Bayesian inference for a human estimating the visual location of a target from a sequence of fixations at a fixed location. \n" 26 | ] 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "id": "29ad7b7e", 31 | "metadata": {}, 32 | "source": [ 33 | "Given a prior location distribution $z_1$ with uncertainty $\\sigma^2_{z1}$, the user makes a visual observation $z_2$ with uncertainty $\\sigma^2_{z2}$. \n", 34 | "\n", 35 | "The user combines their prior and observation optimally using Bayesian estimation.\n", 36 | "\n", 37 | "The best estimate, given the prior and observation, is $\\mu$ with an associated error variance $ \\sigma^2$ as defined below. \n", 38 | "\n", 39 | "$$ \\mu =[\\sigma^2_{z_2}/(\\sigma^2_{z_1}+\\sigma^2_{z_2})] z_1 +[\\sigma^2_{z_1}/(\\sigma^2_{z_1}+\\sigma^2_{z_2})] z_2 $$\n", 40 | "\n", 41 | "\n", 42 | "$$1/ \\sigma^2=1/ \\sigma^2_{z_1}+1/ \\sigma^2_{z_2} $$" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "id": "cbf6a373", 48 | "metadata": {}, 49 | "source": [ 50 | "$\\sigma$ is less than either $\\sigma_{z_1}$ or $\\sigma_{z_2}$ , which is to say that the uncertainty in the user's estimate of location has been decreased by combining the two pieces of information (the prior and the observation). \n", 51 | "\n", 52 | "If $\\sigma_{z_1}$ were equal to $\\sigma_{z_2}$, which is to say that the prior and observation are of equal precision, then the equation says the optimal estimate of position is simply the average of the two measurements, as would be expected. On the other hand, if $\\sigma_{z_1}$ were larger than $\\sigma_{z_2}$, which is to say that the uncertainty in the prior $z_1$ is greater than that of the observation $z_2$ , then the equation dictates “weighting” $z2$ more heavily than $z1$. Finally, the variance of the estimate is less than $\\sigma_{z_1}$ , even if $\\sigma_{z_2}$ is very large: even poor quality data provide some information, and should thus increase the precision of the user's estimate." 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "id": "40464312", 58 | "metadata": {}, 59 | "source": [ 60 | "The above equations can be reformulated.\n", 61 | "\n", 62 | "We have a Guassian prior $p(x)$, and a noisy observation $o$.\n", 63 | "\n", 64 | "The optimal location estimate $\\hat{X}$, that is the maximum of the posterior is:\n", 65 | "\n", 66 | "$$\\hat{X}=\\alpha o +(1- \\alpha) \\hat{\\mu}$$\n", 67 | "\n", 68 | "Where,\n", 69 | "\n", 70 | "$$\\alpha=\\dfrac{\\sigma^2_{p}}{\\sigma^2_{p}+\\sigma^2_{o}}$$\n" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "id": "nuclear-rachel", 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "# This cell and the following are only if you are running on Google Colab.\n", 81 | "\n", 82 | "from google.colab import drive\n", 83 | "drive.mount('/content/drive')" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "id": "cac37a84", 90 | "metadata": {}, 91 | "outputs": [], 92 | "source": [ 93 | "import matplotlib.pyplot as plt\n", 94 | "#%matplotlib inline\n", 95 | "import matplotlib as mpl\n", 96 | "\n", 97 | "import scipy.stats\n", 98 | "import numpy as np\n", 99 | "\n", 100 | "mpl.style.use('fivethirtyeight')\n", 101 | "\n", 102 | "\n", 103 | "def combine_two_guassian(m1,sigma1,m2,sigma2):\n", 104 | " '''\n", 105 | " Optimally combine two gaussians\n", 106 | " Return combine mean and std\n", 107 | " '''\n", 108 | " w1=sigma2**2/(sigma1**2+sigma2**2)\n", 109 | " w2=sigma1**2/(sigma1**2+sigma2**2)\n", 110 | "\n", 111 | " m=w1*m1+w2*m2\n", 112 | " sigma=np.sqrt( (sigma1**2 * sigma2**2)/(sigma1**2 + sigma2**2))\n", 113 | "\n", 114 | " return m,sigma\n", 115 | "\n", 116 | "def plot_gaussian(mean,sigma,fmt,label):\n", 117 | " '''\n", 118 | " plot the guassian pdf\n", 119 | " '''\n", 120 | " x_min = mean-3*sigma\n", 121 | " x_max = mean+3*sigma\n", 122 | " x = np.linspace(x_min, x_max, 100)\n", 123 | " y = scipy.stats.norm.pdf(x,mean,sigma)\n", 124 | " plt.xlim(-1,80)\n", 125 | " plt.ylim(0,0.06)\n", 126 | " plt.plot(x,y,fmt,label=label)\n" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "id": "f7f06dae", 133 | "metadata": {}, 134 | "outputs": [], 135 | "source": [ 136 | "fixation=0\n", 137 | "target=50\n", 138 | "m1,sigma1=40,20\n", 139 | "m2,sigma2=47,10\n", 140 | "\n", 141 | "plt.figure(figsize=(10,6))\n", 142 | "# obs 1\n", 143 | "\n", 144 | "plot_gaussian(m1,sigma1,'g:','prior')\n", 145 | "\n", 146 | "# plot the target line and fixation line\n", 147 | "plt.axvline(x = target, color = 'b', label = 'target')\n", 148 | "plt.axvline(x = fixation, color = 'k')\n", 149 | "\n", 150 | "plt.legend(loc='best')\n", 151 | "plt.xlabel('Eccentricity')\n" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "id": "c372d75b", 158 | "metadata": {}, 159 | "outputs": [], 160 | "source": [ 161 | "\n", 162 | "\n", 163 | "plt.figure(figsize=(10,6))\n", 164 | "# obs 1\n", 165 | "\n", 166 | "plot_gaussian(m1,sigma1,'g:','prior')\n", 167 | "\n", 168 | "# obs 2\n", 169 | "\n", 170 | "plot_gaussian(m2,sigma2,'r--','observation 1')\n", 171 | "\n", 172 | "\n", 173 | "# plot the target line and fixation line\n", 174 | "plt.axvline(x = target, color = 'b', label = 'target')\n", 175 | "plt.axvline(x = fixation, color = 'k')\n", 176 | "\n", 177 | "plt.legend(loc='best')\n", 178 | "plt.xlabel('Eccentricity')\n", 179 | "\n" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "id": "08f31602", 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "fixation=0\n", 190 | "target=50\n", 191 | "\n", 192 | "plt.figure(figsize=(10,6))\n", 193 | "# obs 1\n", 194 | "\n", 195 | "plot_gaussian(m1,sigma1,'g:','prior')\n", 196 | "\n", 197 | "# obs 2\n", 198 | "\n", 199 | "plot_gaussian(m2,sigma2,'r--','observation 1')\n", 200 | "\n", 201 | "# combine obs1 and obs2\n", 202 | "m3,sigma3=combine_two_guassian(m1,sigma1,m2,sigma2)\n", 203 | "plot_gaussian(m3,sigma3,'y-','posterior')\n", 204 | "\n", 205 | "# plot the target line and fixation line\n", 206 | "plt.axvline(x = target, color = 'b', label = 'target')\n", 207 | "plt.axvline(x = fixation, color = 'k')\n", 208 | "\n", 209 | "plt.legend(loc='best')\n", 210 | "plt.xlabel('Eccentricity')\n", 211 | "\n", 212 | "print(m3)\n", 213 | "print(sigma3)\n" 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "id": "6e11290c", 219 | "metadata": {}, 220 | "source": [ 221 | "## Exercises\n", 222 | "\n", 223 | "1. Try different values of the mean and variance of the distributions. Satisfy yourself that the posterior estimate is always more accurate than the prior and the observation.\n", 224 | "\n", 225 | "2. Given a prior with standard deviation of 20 and mean 40, imagine three observations each with different standard deviation (5,10,15) but the same location (70). Illustrate the effect of each observation on the posterior.\n", 226 | "\n", 227 | "Advanced\n", 228 | "\n", 229 | "3. Write a function that takes as input a sequences of noisy observations, possibly from foveated vision(!), of arbitrary length and generates a posterior estimate of the target location. This will be your first simple model of human vision.\n" 230 | ] 231 | }, 232 | { 233 | "cell_type": "code", 234 | "execution_count": null, 235 | "id": "b88774f7", 236 | "metadata": {}, 237 | "outputs": [], 238 | "source": [] 239 | } 240 | ], 241 | "metadata": { 242 | "kernelspec": { 243 | "display_name": "Python 3", 244 | "language": "python", 245 | "name": "python3" 246 | }, 247 | "language_info": { 248 | "codemirror_mode": { 249 | "name": "ipython", 250 | "version": 3 251 | }, 252 | "file_extension": ".py", 253 | "mimetype": "text/x-python", 254 | "name": "python", 255 | "nbconvert_exporter": "python", 256 | "pygments_lexer": "ipython3", 257 | "version": "3.8.8" 258 | } 259 | }, 260 | "nbformat": 4, 261 | "nbformat_minor": 5 262 | } 263 | -------------------------------------------------------------------------------- /02-DeepRL/foveated_vision.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "id": "2847b331", 7 | "metadata": {}, 8 | "source": [ 9 | "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/02-DeepRL/foveated_vision.ipynb)\n" 10 | ] 11 | }, 12 | { 13 | "attachments": {}, 14 | "cell_type": "markdown", 15 | "id": "recovered-bottle", 16 | "metadata": {}, 17 | "source": [ 18 | "# Foveated vision\n", 19 | "\n", 20 | "Andrew Howes\n", 21 | "\n", 22 | "This notebook illustrates the effect of retinal eccentricity on spatial resolution and, therefore, on visual acuity in the human eye.\n", 23 | "\n", 24 | "\"Corati\n", 25 | "\n", 26 | "(source: Geisler, W. S. (2011). Contributions of ideal observer theory to vision research. Vision research, 51(7), 771-781.)\n", 27 | "\n", 28 | "The retina is the layer of photoreceptors at the back of the eye that captures photons and transmits information to the brain. The fovea is a small depression in the middle of the retina with a particularly high density of photoreceptors. It is where visual acuity is highest. People use eye movements to bring the fovea to bear on locations about which they require more information.\n", 29 | "\n", 30 | "As retinal eccentricity from the fovea increases, visual acuity falls exponentially. By just 2.5 degrees of retinal eccentricity, acuity has fallen by 50%.\n", 31 | "\n", 32 | "As a consequence, our ability to estimate the location of an item on a display decreases exponentially with eccentricity from the fovea. The less accurate a target location estimate then the more diffcult it is to move the eyes to the target.\n", 33 | "\n", 34 | "In order to model this 'bound' on cognition, it is assumed that noise in location estimates increases with eccentricity and that the noise is Gaussian distributed.\n", 35 | "\n", 36 | "In order to start modeling let us first define some parameters." 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "id": "homeless-young", 43 | "metadata": {}, 44 | "outputs": [], 45 | "source": [ 46 | "# fixed_noise is the noise at the fovea.\n", 47 | "fixed_noise = 2\n", 48 | "\n", 49 | "# the noise parameter determines how much noise increases with eccentricity\n", 50 | "noise_parameter = 0.25" 51 | ] 52 | }, 53 | { 54 | "attachments": {}, 55 | "cell_type": "markdown", 56 | "id": "0afe8d0b", 57 | "metadata": {}, 58 | "source": [ 59 | "### Import libraries\n", 60 | "\n", 61 | "Next we can import standard libraries for maths, statistics and plotting. " 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "id": "growing-alarm", 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "import matplotlib.pyplot as plt\n", 72 | "import matplotlib as mpl\n", 73 | "import scipy.stats\n", 74 | "import numpy as np\n", 75 | "\n", 76 | "# set a style for the plots.\n", 77 | "\n", 78 | "mpl.style.use('fivethirtyeight')" 79 | ] 80 | }, 81 | { 82 | "attachments": {}, 83 | "cell_type": "markdown", 84 | "id": "f04c5865", 85 | "metadata": {}, 86 | "source": [ 87 | "### Plot\n", 88 | "\n", 89 | "Now we define a function that plots a Gaussian distribution. \n", 90 | "\n", 91 | "Below, we will use this function to represent the distribution of probable target locations given a noisy observation." 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": null, 97 | "id": "78d162ae", 98 | "metadata": {}, 99 | "outputs": [], 100 | "source": [ 101 | "def plot_gaussian(mean,sigma,fmt,label):\n", 102 | " # plot a Gaussian distributed at 'mean' with standard deviation sigma.\n", 103 | " # fmt provides a string of line parameters (colour etc.) and 'label' is a label for the plotted line.\n", 104 | " x_min = mean-3*sigma\n", 105 | " x_max = mean+3*sigma\n", 106 | " x = np.linspace(x_min, x_max, 100)\n", 107 | " y = scipy.stats.norm.pdf(x,mean,abs(sigma))\n", 108 | " plt.xlim(-80,80)\n", 109 | " plt.ylim(0,0.2)\n", 110 | " plt.plot(x,y,fmt,label=label)\n" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "id": "d1119022", 117 | "metadata": {}, 118 | "outputs": [], 119 | "source": [ 120 | "\n", 121 | "eccentricity = np.arange(-60,70,20)\n", 122 | "\n", 123 | "plt.figure(figsize=(10,6))\n", 124 | "\n", 125 | "for i in eccentricity:\n", 126 | " plot_gaussian(i, fixed_noise+abs(noise_parameter*i), 'g:',f'{i}')\n", 127 | "\n", 128 | "x = plt.xlabel('Eccentricity')" 129 | ] 130 | }, 131 | { 132 | "attachments": {}, 133 | "cell_type": "markdown", 134 | "id": "portable-accommodation", 135 | "metadata": {}, 136 | "source": [ 137 | "In the figure above it is assumed that the fovea (the gaze location) is at 0 eccentricity. Each distribution then represents the perceived target location probability given that the actual target location is at the centre of the distribution. \n", 138 | "\n", 139 | "So, for example, if the target is at eccentricity 60 then the probability of perceiving it at 60 is about 0.025, whereas if the target is at 20 then the probability of perceiving it at 20 is over 0.05.\n", 140 | "\n", 141 | "Parameter values do not represent the actual human acuity function and are for illustration only." 142 | ] 143 | }, 144 | { 145 | "attachments": {}, 146 | "cell_type": "markdown", 147 | "id": "metallic-gather", 148 | "metadata": {}, 149 | "source": [ 150 | "### Exercise\n", 151 | "\n", 152 | "- Build a Python model of human vision which returns the (stochastic) perceived location of a target given the true location.\n", 153 | "\n", 154 | "- Hint: \n", 155 | " 1. Use the numpy function random.uniform() to generate a target location between some lower and upper bound of eccentricity. \n", 156 | " 2. Use the eccentricity and the function random.normal() to generate an estimate of the target location.\n", 157 | "\n", 158 | "### Advanced exercises\n", 159 | "\n", 160 | "- Assume that the eyes are moved to the perceived location and a new observation is made of the target (which has not moved). Show, through simulated trials, how the error in the perceived location reduces as each successive observation is made.\n", 161 | "\n", 162 | "- Explain why the accuracy increases." 163 | ] 164 | } 165 | ], 166 | "metadata": { 167 | "kernelspec": { 168 | "display_name": "Python 3", 169 | "language": "python", 170 | "name": "python3" 171 | }, 172 | "language_info": { 173 | "codemirror_mode": { 174 | "name": "ipython", 175 | "version": 3 176 | }, 177 | "file_extension": ".py", 178 | "mimetype": "text/x-python", 179 | "name": "python", 180 | "nbconvert_exporter": "python", 181 | "pygments_lexer": "ipython3", 182 | "version": "3.8.8" 183 | } 184 | }, 185 | "nbformat": 4, 186 | "nbformat_minor": 5 187 | } 188 | -------------------------------------------------------------------------------- /03-Modelbuilding/Go_Nogo.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "id": "3e0c5f5d", 7 | "metadata": {}, 8 | "source": [ 9 | "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/03-Modelbuilding/Go_Nogo.ipynb)\n" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 1, 15 | "id": "20e1ba45", 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "# Install the files to local.\n", 20 | "# Please note that the go / no go model we are using here is still being developed,\n", 21 | "# and should not yet be adapted for other work! The development is in close collaboration\n", 22 | "# with the Commotions project at Leeds, led by Gustav Markkula, and collaborated with Aravinda Srinivasan.\n", 23 | "! wget https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/main/03-Modelbuilding/animate_trace.py\n", 24 | "! wget https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/main/03-Modelbuilding/driver_agent_physics.py\n", 25 | "! wget https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/main/03-Modelbuilding/go_nogo.py\n", 26 | "! wget https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/main/03-Modelbuilding/physics_env.py" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": null, 32 | "id": "02706422", 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "# Install the required library.\n", 37 | "! pip install stable_baselines3" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "id": "50888742", 43 | "metadata": {}, 44 | "source": [ 45 | "# Module 3: Building a Model\n", 46 | "\n", 47 | "In this module, we take a step-by-step walktrough of how to create a computational rational (CR) model using deep reinforcement learning (RL). This notebook does not cover the full workflow of CR modeling, which is long and detailed. It can be found here, make sure to follow it when creating your own models. For the purpose of this notebook, the simplified workflow looks like this:\n", 48 | "\n", 49 | "1. Define the goals.\n", 50 | "\n", 51 | "2. Define the environment.\n", 52 | "\n", 53 | "3. Define the cognitive limitations.\n", 54 | "\n", 55 | "4. Derive the optimal behavior.\n", 56 | "\n", 57 | "5. Inspect model validity.\n", 58 | "\n", 59 | "The model will be defined according to the standard CR flow of information.\n", 60 | "\n", 61 | "\"Corati\n" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "id": "ac368997", 67 | "metadata": {}, 68 | "source": [ 69 | "# 1. Define the agent's goals\n", 70 | "\n", 71 | "The task explored in this notebook is a fairly simple one: in a junction, when turning against the oncoming traffic, the driver needs to decide if they can go, or if they need to wait for an oncoming car before they can cross. In left-handed traffic, this means that the agent driver is turning right. Here is the illustration:\n", 72 | "\n", 73 | "\"Go\n", 74 | "\n", 75 | "The agent (the yellow car) has two main goals:\n", 76 | "\n", 77 | "1. **Proceed** to the destination by turning left.\n", 78 | "\n", 79 | "2. Drive **safely**, avoiding collisions.\n", 80 | "\n", 81 | "First, we need to analyze these goals. First, the agent wants to drive efficiently and not get stuck on the road for too long. They probably want to get to their destination, and also they would be blocking the traffic behind them if they wait unreasonably long. So we can analyse these goals into a reward function.\n", 82 | "\n", 83 | "1. When the agent **turns successfully**, there is a positive reward.\n", 84 | "2. For this positive reward, there is a penalty of **time spent** waiting.\n", 85 | "3. For a **collision**, there is a negative reward." 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "id": "03fe18e6", 91 | "metadata": {}, 92 | "source": [ 93 | "**Task 1**. Discuss alternative goals that the driver may have. Can you specify them in terms of the scalar reward function?" 94 | ] 95 | }, 96 | { 97 | "cell_type": "markdown", 98 | "id": "3a685485", 99 | "metadata": {}, 100 | "source": [ 101 | "# 2. Define the task environment\n", 102 | "\n", 103 | "For simplicity, the environment has only two cars, in a 2-dimensional environment: the agent's car and the oncoming car. The cars are particles that have their individual `(x,y)` coordinates. The agent's car is stationary until the agent decides to turn left. The oncoming car has a variable position, which moves along the y-axis of the environment according to its velocity, which is fixed constant. A collision is detected if the vehicles are too close to each other regardless of their velocities." 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "id": "c7e8921b", 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "# Let's define an environment and an agent and see how it works.\n", 114 | "import physics_env\n", 115 | "import driver_agent_physics\n", 116 | "\n", 117 | "e = physics_env.physics_env()\n", 118 | "agent = driver_agent_physics.driver_agent_physics(e, observation_var = 0) # a full observer: no noise\n", 119 | "agent.reset()" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "id": "24a692d5", 126 | "metadata": {}, 127 | "outputs": [], 128 | "source": [ 129 | "# Let's wait for 20 ticks, then go.\n", 130 | "from IPython.display import HTML, display\n", 131 | "import animate_trace\n", 132 | "\n", 133 | "agent.reset(y_start = -30)\n", 134 | "agent.env.save_trace = True\n", 135 | "for i in range(20):\n", 136 | " agent.step(0)\n", 137 | "agent.step(1)\n", 138 | "agent.env.save_trace = False\n", 139 | "\n", 140 | "HTML(animate_trace.animate_trace(agent.env.trace, get_anim = True, x_lim = [-50,50], y_lim = [10,-80]).to_jshtml())" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "id": "ff573d60", 146 | "metadata": {}, 147 | "source": [ 148 | "**Task 2.** Change the starting distance of the incoming car. Try to collide the cars! (Note that the y-coordinate needs to be negative.)" 149 | ] 150 | }, 151 | { 152 | "cell_type": "markdown", 153 | "id": "3ed88fdb", 154 | "metadata": {}, 155 | "source": [ 156 | "# 3. Define the relevant cognitive bounds of the agent\n", 157 | "\n", 158 | "The agent must make a decision to go (turn left) or wait. It bases this decision on a noisy observation of the oncoming car's distance to the agent's car. If the distance is long enough, the agent can go and save time. If the car is too close, the agent must wait for it to pass to avoid a collision.\n", 159 | "\n", 160 | "For modeling noisy observation, we will be using the formula from Markkula, et al. (2022). Explaining human interactions on the road requires large-scale integration of psychological theory. https://psyarxiv.com/hdxbs/\n", 161 | "\n", 162 | "$ \\hat{D} = D_{oth} \\cdot \\left(1 - \\frac{h}{D \\cdot \\tan \\left(\\arctan \\left(\\frac{h}{D}\\right) + \\sigma \\right)}\\right) $,\n", 163 | "\n", 164 | "where $ \\hat{D} $ is a noisy observation of distance, $D_{oth}$ is the oncoming car's longitudinal distance to the crossing point, and $h$ is the observer's eye height over ground (1.5m). The important parameter here is $\\sigma$, which describes how noisy the observation is.\n", 165 | "\n", 166 | "The noisy observation is not used directly, but via Bayesian filtering. This considers prior belief about the distance and integrates the new observation with it. Additionally, we represent the uncertainty associated with the belief.\n", 167 | "\n", 168 | "Before deriving the optimal behavior, we want to establish what hypotheses our model in fact makes, so that we can then assess the plausibility of model predictions against them. While modeling is virtually always interactive and the hypotheses might develop during iteration, but it is important to have a strong initial assumption about how our model will behave. Here are initial proposals, which would then be turned into exact, testable hypotheses:\n", 169 | "\n", 170 | "P1. For a non-noisy (full) observer, there is a precise distance: if the other car is farther than this distance, the agent decides to go. If the other car is closer, the agent chooses to wait. If the car has passed, the agent always goes.\n", 171 | "\n", 172 | "P2. For a noisy (partial) observer, the decision to go is more conservative and uncertain: the distance that determines the go / no go decision is larger, and due to noisy estimates, the choice is probabilistic given distance." 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": null, 178 | "id": "b960cbd8", 179 | "metadata": {}, 180 | "outputs": [], 181 | "source": [ 182 | "# Here is a theoretical illustration of our proposals.\n", 183 | "import numpy as np\n", 184 | "import matplotlib.pyplot as plt\n", 185 | "import seaborn as sns\n", 186 | "sns.set_style('whitegrid')\n", 187 | "x1 = np.linspace(-1, 0, 100)\n", 188 | "x2 = np.linspace(-1, 0, 100)\n", 189 | "y1 = np.zeros(100)\n", 190 | "y1[50:] = 1\n", 191 | "y2 = 1 / (1 + np.exp(-10*(x2+0.5)))\n", 192 | "plt.plot(x1, y1, label='Full Observer', linewidth = 2)\n", 193 | "plt.plot(x2, y2, label='Noisy Observer', linewidth = 2)\n", 194 | "plt.ylim(0, 1.01)\n", 195 | "plt.xlabel('Distance', fontsize=14)\n", 196 | "plt.ylabel('Probability of wait', fontsize=14)\n", 197 | "plt.legend(fontsize=14)\n", 198 | "plt.show()" 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "id": "1eebb341", 204 | "metadata": {}, 205 | "source": [ 206 | "**Task 3.** Discuss what the hypothesis actually means. What do the lines tell us?" 207 | ] 208 | }, 209 | { 210 | "cell_type": "markdown", 211 | "id": "8610046c", 212 | "metadata": {}, 213 | "source": [ 214 | "## How does the noisy observation happen?\n", 215 | "\n", 216 | "The observation of the approaching vehicle distance is noisy. In practice, we simulate noisy observation by taking the true observation and then adding noise from a normal distribution, according to some parameter sigma. However, instead of using the returned noisy value as such, we assume that the human visual system exploits the fact that each noisy value is based on the true value, and the noise comes from a known distribution. We can then use a Kalman filter to approximate the true state. With more observations, the approximation becomes better. In our simulation, this is confounded by the formula above, which states that observations get better as the distance shortens. Let's investigate this behavior." 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": null, 222 | "id": "994429e4", 223 | "metadata": {}, 224 | "outputs": [], 225 | "source": [ 226 | "import math\n", 227 | "import numpy as np\n", 228 | "\n", 229 | "# Here is the formula for making an observation, given distance.\n", 230 | "def noisy_observation(D):\n", 231 | " # D = distance in meters\n", 232 | " d_oth = D+2 # crossing point distance, this is a crude approximation to simplify our exercise\n", 233 | " h = 1.5 # eye height\n", 234 | " observation_var = 0.1 # this is the sigma\n", 235 | " distance_var = d_oth * (1 - h / (D*math.tan(math.atan(h/D) + observation_var)))\n", 236 | " observed_distance = np.random.normal(D, distance_var)\n", 237 | " return observed_distance, distance_var\n", 238 | "\n", 239 | "\n", 240 | "# Let's see what kinds of observations we make from the true distance of 50 m\n", 241 | "for i in range(10):\n", 242 | " print(noisy_observation(50)[0])" 243 | ] 244 | }, 245 | { 246 | "cell_type": "code", 247 | "execution_count": null, 248 | "id": "9db93f30", 249 | "metadata": {}, 250 | "outputs": [], 251 | "source": [ 252 | "# So, that's not a very reliable observation! Here is the Kalman filter, let's see how it helps.\n", 253 | "def kalman_update(prior_mean, prior_var, observation, s):\n", 254 | " observation_gain = 1\n", 255 | " kalman_gain = prior_var * observation_gain / (prior_var * observation_gain**2 + s**2)\n", 256 | " posterior_mean = prior_mean + kalman_gain * (observation - prior_mean)\n", 257 | " posterior_var = (1 - kalman_gain * observation_gain) * prior_var\n", 258 | "\n", 259 | " return posterior_mean, posterior_var\n", 260 | " \n", 261 | "# Then, let's update both mean and variance estimates multiple times\n", 262 | "distance_prior = 100 # set an uninformed prior\n", 263 | "distance_var_prior = 1000 # basically a uniform prior\n", 264 | "for i in range(20):\n", 265 | " d, s = noisy_observation(50)\n", 266 | " distance_post, distance_var_post = kalman_update(distance_prior, distance_var_prior, d, s)\n", 267 | " # Store result\n", 268 | " print(distance_post)\n", 269 | " # Set the posterior as the new prior\n", 270 | " distance_prior = distance_post\n", 271 | " distance_var_prior = distance_var_post" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": null, 277 | "id": "47023b7a", 278 | "metadata": {}, 279 | "outputs": [], 280 | "source": [ 281 | "# Can we visualize it? Let's assume a constant velocity of 0.5 m per observational \"tick\".\n", 282 | "# Assuming one tick is 0.05s, this is 10m/s = 36km/h\n", 283 | "import matplotlib.pyplot as plt\n", 284 | "import seaborn as sns\n", 285 | "\n", 286 | "true_d = 50\n", 287 | "distance_prior = 100 # set an uninformed prior\n", 288 | "distance_var_prior = 1000 # basically a uniform prior\n", 289 | "velocity = 0.5\n", 290 | "data = []\n", 291 | "for i in range(100):\n", 292 | " d, s = noisy_observation(true_d)\n", 293 | " distance_post, distance_var_post = kalman_update(distance_prior, distance_var_prior, d, s)\n", 294 | " # Store result\n", 295 | " data.append([true_d, d, distance_post])\n", 296 | " # Set the posterior as the new prior\n", 297 | " distance_prior = distance_post\n", 298 | " distance_var_prior = distance_var_post\n", 299 | " # \"Move\" the approaching vehicle\n", 300 | " true_d -= velocity\n", 301 | " \n", 302 | "# Make the plot\n", 303 | "true_distance, observed_distance, estimated_distance = zip(*data)\n", 304 | "index = np.array(range(len(data))) / 20 # make into seconds\n", 305 | "sns.set_style('whitegrid')\n", 306 | "plt.plot(index, true_distance, label='True Distance', linestyle='-', marker='o')\n", 307 | "plt.plot(index, observed_distance, label='Observed Distance', linestyle='-', marker='o')\n", 308 | "plt.plot(index, estimated_distance, label='Estimated Distance', linestyle='-', marker='o')\n", 309 | "\n", 310 | "plt.xlabel('Time (s)', fontsize = 14)\n", 311 | "plt.ylabel('Distance', fontsize = 14)\n", 312 | "plt.legend(loc='best', fontsize = 14)\n", 313 | "\n", 314 | "plt.show()" 315 | ] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "id": "0fda6479", 320 | "metadata": {}, 321 | "source": [ 322 | "**Task 3.1** Change the prior (distance and variance) and investigate how that changes the convergence of the posterior over time. You can also try changing the velocity and starting distance." 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "id": "982fa065", 328 | "metadata": {}, 329 | "source": [ 330 | "# 4. Derive optimal policy\n", 331 | "\n", 332 | "For establishing the (bounded) optimal policy for the ideal and noisy observer agents, we will be using Proximal Policy Optimization, which is an on-policy deep RL algorithm. This notebook uses the OpenAI Stable Baselines implementation, but there are others as well. https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html\n", 333 | "\n", 334 | "Before we can derive the optimal policy, the environments (both internal and external) need to be modeled using Markov Decision Process (**MDP**, or in our case, due to partial observability, we are defining a **POMDP**). We won't go into the details of how the external driving environment is modeled, but it is a simple stepwise simulator, that for each time step (0.05 seconds by default), \"ticks\" the environment forward by moving the upcoming car's y-position according to its velocity, and in case the agent decides to go, moves it towards the side road along a predefined trajectory, in a constant velocity as well.\n", 335 | "\n", 336 | "For the internal environment, we need to model the belief update for distance of the oncoming car, given the equation and filtering that were defined above. All relevant information must be represented as the agent's belief such that we can pass that, along with the reward signal, to the RL agent for learning the optimal policy." 337 | ] 338 | }, 339 | { 340 | "cell_type": "code", 341 | "execution_count": null, 342 | "id": "a3b9e221", 343 | "metadata": {}, 344 | "outputs": [], 345 | "source": [ 346 | "# Here are the action and observation spaces of the agent\n", 347 | "\n", 348 | "from gymnasium.spaces import Discrete, Dict, Box\n", 349 | "\n", 350 | "action_space = Discrete(2) \n", 351 | "\n", 352 | "# Note that all observatons are normalized between 0 and 1.\n", 353 | "observation_space = Dict(\n", 354 | " spaces = {\n", 355 | " \"distance\": Box(0, 1, (1,)),\n", 356 | " \"passed\": Box(0, 1, (1,)),\n", 357 | " \"distance_var\": Box(0, 1, (1,)),\n", 358 | " \"speed\": Box(0, 1, (1,)),\n", 359 | " \"acceleration\": Box(0, 1, (1,)),\n", 360 | " \"ticks\": Box(0, 1, (1,))\n", 361 | " })\n", 362 | "\n", 363 | "# Sample the action space:\n", 364 | "print(\"A small sample of actions:\", action_space.sample(), action_space.sample(), action_space.sample())\n", 365 | "print(\"One sample of observation space:\", observation_space.sample())" 366 | ] 367 | }, 368 | { 369 | "cell_type": "markdown", 370 | "id": "fae9030e", 371 | "metadata": {}, 372 | "source": [ 373 | "So, the environment simulation works as intended. Let's now take a look at the reward function. Remember our definition:\n", 374 | "\n", 375 | "1. When the agent **turns left successfully**, there is a positive reward.\n", 376 | "2. For this positive reward, there is a penalty of **time spent** waiting.\n", 377 | "3. For a **collision**, there is a negative reward.\n", 378 | "\n", 379 | "We will establish the reward function along with one step of the model to see what actually happens when we step." 380 | ] 381 | }, 382 | { 383 | "cell_type": "code", 384 | "execution_count": null, 385 | "id": "8227436a", 386 | "metadata": {}, 387 | "outputs": [], 388 | "source": [ 389 | "def step(self, action):\n", 390 | " self.reward = 0\n", 391 | " # action: no go\n", 392 | " if action == 0:\n", 393 | " self.env.tick()\n", 394 | " self.ticks += 1\n", 395 | " # break if nothing ever happens\n", 396 | " if self.ticks > self.max_ticks:\n", 397 | " self.reward = -10\n", 398 | " self.done = True\n", 399 | " if self.env.get_distance() > self.max_distance:\n", 400 | " self.reward = -10\n", 401 | " self.done = True \n", 402 | " # action: go\n", 403 | " if action == 1:\n", 404 | " # Did we wait for the other car before going?\n", 405 | " if self.env.veh2_turn_pos[1] < self.env.veh1_straight_pos[1]:\n", 406 | " self.waited_before_go = True\n", 407 | " self.distance_at_go = self.env.get_distance()\n", 408 | " self.done = True\n", 409 | " self.collision, _ = self.env.simulate_go()\n", 410 | " if self.collision:\n", 411 | " self.reward = -10\n", 412 | " else:\n", 413 | " self.reward = 10 - self.penalty_per_tick * self.ticks\n", 414 | "\n", 415 | " self.belief = self.get_belief()\n", 416 | "\n", 417 | " return self.belief, self.reward, self.done, False, {}" 418 | ] 419 | }, 420 | { 421 | "cell_type": "code", 422 | "execution_count": null, 423 | "id": "8b439c41", 424 | "metadata": {}, 425 | "outputs": [], 426 | "source": [ 427 | "# Let's make and train the full observer agent.\n", 428 | "import go_nogo\n", 429 | "full_obs_agent = go_nogo.make_agent(sigma = 0, iters = 10)\n", 430 | "# In the output:\n", 431 | "# i = training iteration\n", 432 | "# t = number of ticks (1 tick = 0.05s)\n", 433 | "# r = average reward (10 is max)\n", 434 | "# d = average distance of the two vehicles at the time of go (in meters)\n", 435 | "# w = frequency of waits (agent waited the other car to pass before going)\n", 436 | "# c = frequency of collisions" 437 | ] 438 | }, 439 | { 440 | "cell_type": "code", 441 | "execution_count": null, 442 | "id": "6d409044", 443 | "metadata": {}, 444 | "outputs": [], 445 | "source": [ 446 | "# To keep training the agent without initializing it anew, use this.\n", 447 | "# Commented out so it won't run when running all cells\n", 448 | "# go_nogo.retrain_agent(full_obs_agent, iters = 10)" 449 | ] 450 | }, 451 | { 452 | "cell_type": "code", 453 | "execution_count": null, 454 | "id": "11e89821", 455 | "metadata": {}, 456 | "outputs": [], 457 | "source": [ 458 | "# Let's investigate if it has found a policy for the different distances.\n", 459 | "# Keep an eye on the \"critical\" y_start at around -13.\n", 460 | "HTML(go_nogo.animate_agent(full_obs_agent, y_start = -13.8, get_anim = True).to_jshtml())" 461 | ] 462 | }, 463 | { 464 | "cell_type": "code", 465 | "execution_count": null, 466 | "id": "e5572776", 467 | "metadata": {}, 468 | "outputs": [], 469 | "source": [ 470 | "# Train the noisy observer. Use more iters due to a more difficult learning task.\n", 471 | "noisy_obs_agent = go_nogo.make_agent(sigma = 0.1, iters = 15)" 472 | ] 473 | }, 474 | { 475 | "cell_type": "code", 476 | "execution_count": null, 477 | "id": "48c8d8ac", 478 | "metadata": {}, 479 | "outputs": [], 480 | "source": [ 481 | "# To keep training the agent without initializing it anew, use this.\n", 482 | "# Commented out so it won't run when running all cells\n", 483 | "#go_nogo.retrain_agent(noisy_obs_agent, iters = 10)" 484 | ] 485 | }, 486 | { 487 | "cell_type": "code", 488 | "execution_count": null, 489 | "id": "e081ef1d", 490 | "metadata": {}, 491 | "outputs": [], 492 | "source": [ 493 | "# Let's investigate if it has found a policy for the different distances.\n", 494 | "# Keep an eye on the \"critical\" y_start at around -13.\n", 495 | "HTML(go_nogo.animate_agent(noisy_obs_agent, y_start = -13.8, get_anim = True).to_jshtml())" 496 | ] 497 | }, 498 | { 499 | "cell_type": "markdown", 500 | "id": "c8ed7578", 501 | "metadata": {}, 502 | "source": [ 503 | "**Task 4.** Try to find out the critical distance where the two models, full and partial/noisy observer, differ in their go/no go policy." 504 | ] 505 | }, 506 | { 507 | "cell_type": "markdown", 508 | "id": "5b6401b0", 509 | "metadata": {}, 510 | "source": [ 511 | "# 5. Inspect model validity\n", 512 | "\n", 513 | "After having converged the model to an optimal policy, our aim is to utilize it for generating simulations of task behavior. The evaluation of the model's validity encompasses multiple stages, see the workflow.pdf draft for these. Here, we concentrate solely on its face validity, which addresses whether the model aligns with our initial predictions.\n", 514 | "\n", 515 | "Starting the model's validity assessment with face validity tests is a useful practice, as any discrepancies between the model's performance and our hypotheses at this stage may indicate issues with either the model's specification or our modeling assumptions. This is frequently an iterative procedure, during which we may observe the model's divergence from our expectations, resulting in identifying inadequate definitions of objectives, task environment, or cognitive constraints.\n", 516 | "\n", 517 | "Once the model successfully demonstrates face validity, it should be subjected to a rigorous validation process, wherein its predictions are compared against human data or some alternative benchmarks. The model should e.g., generate accurate summary statistics (across a broader human population), be capable of replicating individual performance by adjusting specific parameters, and operate reasonably under changes in the environment. The specific validation depends always on the details of the modeling work." 518 | ] 519 | }, 520 | { 521 | "cell_type": "code", 522 | "execution_count": null, 523 | "id": "329c93d6", 524 | "metadata": {}, 525 | "outputs": [], 526 | "source": [ 527 | "# Run an experiment for obtaining multiple samples from each agent.\n", 528 | "def wait_or_go_experiment(agent, y_range, n = 100, deterministic = False):\n", 529 | " data = []\n", 530 | " agent.env.veh1_straight_start_y_range = y_range\n", 531 | " for i in range(n):\n", 532 | " _, _, _, w, c = agent.run_episode(deterministic = deterministic)\n", 533 | " data.append([agent.observation_var, agent.env.y_start, w, c])\n", 534 | " \n", 535 | " agent.env.veh1_straight_start_y_range = [-25,-2] # return back to original\n", 536 | " return data\n", 537 | "\n", 538 | "import pandas as pd\n", 539 | "\n", 540 | "# Increase n to e.g., 2000 to get more robust final image, but note that it takes longer to run.\n", 541 | "data = wait_or_go_experiment(full_obs_agent, y_range = [-5,-25], n = 500, deterministic = True)\n", 542 | "data = data + wait_or_go_experiment(noisy_obs_agent, y_range = [-5,-25], n = 500, deterministic = True)\n", 543 | "columns = ['sigma', 'y_start', 'wait', 'collision']\n", 544 | "df = pd.DataFrame(data, columns=columns)" 545 | ] 546 | }, 547 | { 548 | "cell_type": "code", 549 | "execution_count": null, 550 | "id": "eb1b7231", 551 | "metadata": {}, 552 | "outputs": [], 553 | "source": [ 554 | "# Visualize individual go/no go decisions between different sigma models for various y_ranges.\n", 555 | "\n", 556 | "import numpy as np\n", 557 | "import matplotlib.pyplot as plt\n", 558 | "\n", 559 | "# Note: this is a bit slow and dirty, gets slow with a lot of data. Only use for diagnosis.\n", 560 | "def plot_data(df):\n", 561 | " fig, ax = plt.subplots(figsize=(10, 6))\n", 562 | "\n", 563 | " # Define custom colors and markers based on wait and collision values\n", 564 | " colors = {0: 'red', 1: 'blue'}\n", 565 | " markers = {0: 'o', 1: 'x'}\n", 566 | "\n", 567 | " # Create a dictionary to store the labels we've already added to the legend\n", 568 | " labels_added = {}\n", 569 | "\n", 570 | " for index, row in df.iterrows():\n", 571 | " sigma = row['sigma']\n", 572 | " y_start = row['y_start']\n", 573 | " wait = row['wait']\n", 574 | " collision = row['collision']\n", 575 | " label = f'Wait: {wait}, Collision: {collision}'\n", 576 | "\n", 577 | " # Add scatter plot point with custom color and marker\n", 578 | " ax.scatter(\n", 579 | " y_start,\n", 580 | " sigma,\n", 581 | " marker=markers[collision],\n", 582 | " color=colors[wait],\n", 583 | " label=label if label not in labels_added else \"\",\n", 584 | " alpha=0.7\n", 585 | " )\n", 586 | " \n", 587 | " # Remember that we've added this label to the legend\n", 588 | " labels_added[label] = True\n", 589 | "\n", 590 | " ax.set_xlabel('y_start')\n", 591 | " ax.set_ylabel('sigma')\n", 592 | " ax.legend(title=\"Wait, Collision\", bbox_to_anchor=(1.05, 1), loc='upper left')\n", 593 | " plt.title('Impact of y_start on wait and collision for different sigma agents')\n", 594 | " plt.show()\n", 595 | " \n", 596 | "plot_data(df)" 597 | ] 598 | }, 599 | { 600 | "cell_type": "markdown", 601 | "id": "86a7a58e", 602 | "metadata": {}, 603 | "source": [ 604 | "**Task 5.** Discuss the figure. What are we seeing here? What is the difference between the two models and how is it connected to the original hypotheses made?" 605 | ] 606 | }, 607 | { 608 | "cell_type": "code", 609 | "execution_count": null, 610 | "id": "44b80148", 611 | "metadata": {}, 612 | "outputs": [], 613 | "source": [ 614 | "# Visualize the probability of go/no go as the function of y_start, between different sigmas.\n", 615 | "\n", 616 | "# Change smoothness to larger to get less raggedy lines.\n", 617 | "# Also, increase the N in the experiment above to get more observations.\n", 618 | "smoothness = 1\n", 619 | "\n", 620 | "from scipy.ndimage import gaussian_filter1d\n", 621 | "\n", 622 | "import seaborn as sns\n", 623 | "\n", 624 | "def estimate_probability(df, sigma, y_start_values, window_size=1, collision = False):\n", 625 | " sub_df = df[df['sigma'] == sigma].sort_values('y_start')\n", 626 | " probabilities = []\n", 627 | "\n", 628 | " for y_start in y_start_values:\n", 629 | " window_df = sub_df[(sub_df['y_start'] >= y_start - window_size / 2) & (sub_df['y_start'] <= y_start + window_size / 2)]\n", 630 | " probability = window_df['collision' if collision else 'wait'].sum() / len(window_df)\n", 631 | " probabilities.append(probability)\n", 632 | "\n", 633 | " return probabilities\n", 634 | "\n", 635 | "def smooth_probabilities(probabilities, sigma=1):\n", 636 | " return gaussian_filter1d(probabilities, sigma=sigma)\n", 637 | "\n", 638 | "def plot_probability_lines(df, collision = False):\n", 639 | " sns.set_style('whitegrid')\n", 640 | " fig, ax = plt.subplots(figsize=(10, 6))\n", 641 | "\n", 642 | " y_start_range = np.linspace(df['y_start'].min(), df['y_start'].max(), num=500)\n", 643 | "\n", 644 | " for sigma in df['sigma'].unique():\n", 645 | " probabilities = estimate_probability(df, sigma, y_start_range, collision = collision)\n", 646 | " smoothed_probabilities = smooth_probabilities(probabilities, sigma = smoothness)\n", 647 | " ax.plot(y_start_range, smoothed_probabilities, label=f'{sigma}', linewidth=2)\n", 648 | "\n", 649 | " ax.set_xlabel('y_start', fontsize=14)\n", 650 | " ax.set_ylabel('Probability of Wait', fontsize=14)\n", 651 | " ax.legend(title=\"Sigma\", loc=\"upper left\", fontsize=12, title_fontsize=12)\n", 652 | " plt.xticks(fontsize=12)\n", 653 | " plt.yticks(fontsize=12)\n", 654 | " plt.title('Probability of Wait across y_start by sigma', fontsize=16)\n", 655 | " plt.grid(alpha=0.5)\n", 656 | " plt.show()\n", 657 | "\n", 658 | "plot_probability_lines(df)" 659 | ] 660 | }, 661 | { 662 | "cell_type": "markdown", 663 | "id": "f12364a6-9a39-401b-b236-64e55c97b6c9", 664 | "metadata": {}, 665 | "source": [ 666 | "# Extra: Changing the reward function\n", 667 | "\n", 668 | "With the model right above, we observe that the behavior follows our theoretical hypothesis, however it emphasizes is risk-taking in a way that does not feel human-like.\n", 669 | "\n", 670 | "The next step in modeling is to start adjusting the reward function for the desired performance. It sounds theoretically implausible that a collision is as \"bad\" (negatively rewarding) as a successful task is \"good\" (positively rewarding). We can change the model's predicted behavior by adjusting the relative strengths of the positive reward from a successful task and negative reward from a collision. The original values were 10 and -10, respectively." 671 | ] 672 | }, 673 | { 674 | "cell_type": "code", 675 | "execution_count": null, 676 | "id": "b825f31f", 677 | "metadata": {}, 678 | "outputs": [], 679 | "source": [ 680 | "# Set collision penalty to a considerable higher value. 'p' for 'penalty'\n", 681 | "# For ease of illustration using the old code, we set sigma to a slightly different value, which won't practically impact the simulation\n", 682 | "\n", 683 | "#noisy_obs_agent_p = go_nogo.make_agent(sigma = 0.1001, iters = 15, collision_reward = -500)" 684 | ] 685 | }, 686 | { 687 | "cell_type": "code", 688 | "execution_count": null, 689 | "id": "a72eb4ac-a31c-4cee-9da8-88dd4e5d439e", 690 | "metadata": {}, 691 | "outputs": [], 692 | "source": [ 693 | "# Collect data for all three models (full observation, noisy observation, and noisy observation with a large collision penalty\n", 694 | "# Bigger range for start values of the other car for clearer visualization.\n", 695 | "\n", 696 | "#data = wait_or_go_experiment(full_obs_agent, y_range = [-5,-40], n = 1000, deterministic = True)\n", 697 | "#data = data + wait_or_go_experiment(noisy_obs_agent, y_range = [-5,-40], n = 1000, deterministic = True)\n", 698 | "#data = data + wait_or_go_experiment(noisy_obs_agent_p, y_range = [-5,-40], n = 1000, deterministic = True)\n", 699 | "\n", 700 | "#columns = ['sigma', 'y_start', 'wait', 'collision']\n", 701 | "#df = pd.DataFrame(data, columns=columns)" 702 | ] 703 | }, 704 | { 705 | "cell_type": "code", 706 | "execution_count": null, 707 | "id": "4666aa97-293e-48b4-8f18-5e92a5213e5f", 708 | "metadata": {}, 709 | "outputs": [], 710 | "source": [] 711 | } 712 | ], 713 | "metadata": { 714 | "kernelspec": { 715 | "display_name": "Python 3 (ipykernel)", 716 | "language": "python", 717 | "name": "python3" 718 | }, 719 | "language_info": { 720 | "codemirror_mode": { 721 | "name": "ipython", 722 | "version": 3 723 | }, 724 | "file_extension": ".py", 725 | "mimetype": "text/x-python", 726 | "name": "python", 727 | "nbconvert_exporter": "python", 728 | "pygments_lexer": "ipython3", 729 | "version": "3.12.3" 730 | } 731 | }, 732 | "nbformat": 4, 733 | "nbformat_minor": 5 734 | } 735 | -------------------------------------------------------------------------------- /03-Modelbuilding/animate_trace.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import matplotlib 3 | from matplotlib import pyplot as plt 4 | from matplotlib import animation 5 | 6 | def animate_trace(trace, get_anim = False, x_lim = [-100,100], y_lim = [100,-100]): 7 | plt.close() 8 | interval = 0.05 9 | 10 | frames = len(trace) 11 | 12 | fig = plt.figure() 13 | ax = fig.add_subplot(1, 1, 1, aspect='equal') 14 | ax.set_xlim(x_lim[0], x_lim[1]) 15 | ax.set_ylim(y_lim[0], y_lim[1]) 16 | 17 | 18 | axl = plt.gca() 19 | axl.invert_yaxis() 20 | 21 | veh1 = plt.Circle((0, 0), 2, fc='orange') 22 | veh2 = plt.Circle((0, 0), 2, fc='red') 23 | 24 | time_text = ax.text(-30, 40, "Time:", fontsize=8) 25 | dist_text = ax.text(-30, 35, "Dist:", fontsize=8) 26 | coll_text = ax.text(-30, 30, "Coll:", fontsize=8) 27 | 28 | def init(): 29 | time_text.set_text('') 30 | dist_text.set_text('') 31 | coll_text.set_text('') 32 | return veh1, veh2, time_text, dist_text, coll_text 33 | 34 | def animate(i): 35 | if i == 0: 36 | ax.add_patch(veh1) 37 | ax.add_patch(veh2) 38 | t = trace[i][0] 39 | t = i * interval 40 | x = int(trace[i][1][0]) 41 | y = int(trace[i][1][1]) 42 | veh1.center = (x, y) 43 | x = int(trace[i][2][0]) 44 | y = int(trace[i][2][1]) 45 | veh2.center = (x, y) 46 | 47 | if trace[i][4]: 48 | veh2.set_color('r') 49 | 50 | time_text.set_text('Time {:1.2f}'.format(round(t,2))) 51 | d = round(trace[i][3],2) 52 | dist_text.set_text('Dist {:1.2f}'.format(d)) 53 | coll_text.set_text('Coll {:1.5s}'.format(str(trace[i][4]))) 54 | return veh1, veh2, time_text, dist_text, coll_text 55 | 56 | anim = animation.FuncAnimation( 57 | fig, 58 | animate, 59 | init_func=init, 60 | frames=frames, 61 | repeat=False, 62 | interval=1000 * interval, 63 | blit=True, 64 | ) 65 | 66 | if get_anim: 67 | return anim 68 | else: 69 | plt.show() 70 | -------------------------------------------------------------------------------- /03-Modelbuilding/corati_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/03-Modelbuilding/corati_model.png -------------------------------------------------------------------------------- /03-Modelbuilding/driver_agent_physics.py: -------------------------------------------------------------------------------- 1 | from gymnasium import Env 2 | from gymnasium.spaces import Discrete, Dict, Box 3 | from copy import copy 4 | 5 | from stable_baselines3 import PPO 6 | 7 | import numpy as np 8 | import math 9 | 10 | class driver_agent_physics(Env): 11 | def __init__(self, physics_env, goal_reward = 10, collision_reward = -10, observation_var = 0): 12 | self.env = physics_env 13 | 14 | self.goal_reward = goal_reward 15 | self.collision_reward = collision_reward 16 | 17 | self.max_ticks = 256 18 | self.max_distance = 80 19 | 20 | # go / no go 21 | self.action_space = Discrete(2) 22 | 23 | self.observation_space = Dict( 24 | spaces = { 25 | "distance": Box(0, 1, (1,)), 26 | "passed": Box(0, 1, (1,)), 27 | "distance_var": Box(0, 1, (1,)), 28 | "speed": Box(0, 1, (1,)), 29 | "acceleration": Box(0, 1, (1,)), 30 | "ticks": Box(0, 1, (1,)) 31 | }) 32 | 33 | self.penalty_per_tick = 0 34 | 35 | self.observation_var = observation_var 36 | 37 | self.agent = PPO("MultiInputPolicy", self, device = 'cuda', 38 | # learning_rate=0.00025, 39 | # ent_coef=0.01, 40 | # n_steps = 128, 41 | # batch_size = 64, 42 | verbose = 0) 43 | 44 | def reset(self, y_start = None, seed = None): 45 | self.env.reset(y_start = y_start) 46 | 47 | self.done = False 48 | self.ticks = 0 49 | 50 | self.collision = False 51 | self.distance_at_go = None 52 | self.waited_before_go = False 53 | 54 | # Init prior as basically flat. 55 | self.prior = {} 56 | self.prior['distance'] = self.max_distance/2 57 | self.prior['distance_var'] = 1000 58 | 59 | self.belief = self.get_belief() 60 | 61 | return self.belief, {} 62 | 63 | def step(self, action): 64 | self.reward = 0 65 | trunc = False 66 | # action: no go 67 | if action == 0: 68 | self.env.tick() 69 | self.ticks += 1 70 | # break if nothing ever happens 71 | if self.ticks > self.max_ticks: 72 | #print("Too many ticks") 73 | self.reward = self.collision_reward 74 | self.done = True 75 | trunc = True 76 | if self.env.get_distance() > self.max_distance: 77 | self.reward = self.collision_reward 78 | self.done = True 79 | # action: go 80 | if action == 1: 81 | # Did we wait for the other car before going? 82 | if self.env.veh2_turn_pos[1] < self.env.veh1_straight_pos[1]: 83 | self.waited_before_go = True 84 | self.distance_at_go = self.env.get_distance() 85 | self.done = True 86 | self.collision, _ = self.env.simulate_go() 87 | if self.collision: 88 | self.reward = self.collision_reward 89 | else: 90 | self.reward = self.goal_reward - self.penalty_per_tick * self.ticks 91 | 92 | self.belief = self.get_belief() 93 | 94 | return self.belief, self.reward, self.done, trunc, {} 95 | 96 | def get_belief(self): 97 | s = self.env.get_state() 98 | s['passed'] = [0] if self.env.veh2_turn_pos[1] > self.env.veh1_straight_pos[1] else [1] 99 | # Make observation noisy, as in https://github.com/gmarkkula/COMMOTIONSFramework 100 | D = s['distance'] 101 | d_oth = np.linalg.norm(self.env.veh1_straight_pos-[-1.825, -1.506512]) 102 | h = 1.5 103 | s['distance_var'] = d_oth * (1 - h / (D*math.tan(math.atan(h/D) + self.observation_var))) 104 | s['distance_var'] = max(0, s['distance_var']) 105 | s['distance'] = np.random.normal(s['distance'], s['distance_var']) 106 | if self.observation_var > 0: 107 | s['distance'], s['distance_var'] = \ 108 | self.kalman_update(self.prior['distance'], 109 | self.prior['distance_var'], 110 | np.random.normal(s['distance'], 111 | self.observation_var), 112 | self.observation_var) 113 | self.prior['distance'] = s['distance'] 114 | self.prior['distance_var'] = s['distance_var'] 115 | # Normalise into arrays 116 | # TODO: Set maxes as constants and obey them all over 117 | s['ticks'] = [self.ticks/self.max_ticks] 118 | if self.observation_var > 0: 119 | s['distance_var'] = [s['distance_var'] / (self.observation_var**2)] 120 | else: 121 | s['distance_var'] = [0] 122 | s['speed'] = [s['speed'] / 30] 123 | s['distance'] = [s['distance'] / (4*self.max_distance)] 124 | s['acceleration'] = [s['acceleration']] 125 | 126 | if s['speed'][0] > 1 or s['distance'][0] > 1 or s['acceleration'][0] > 1 or s['distance_var'][0] > 1: 127 | print("Box overflow") 128 | print(self.env.get_state()) 129 | print(s) 130 | 131 | return s 132 | 133 | def run_episode(self, render = False, deterministic = False, y_start = None): 134 | self.agent.policy.set_training_mode(False) 135 | # if render and self.carla_env.settings.no_rendering_mode: 136 | # self.carla_env.settings.no_rendering_mode = False 137 | # self.carla_env.world.apply_settings(self.carla_env.settings) 138 | self.reset(y_start = y_start) 139 | ticks = 0 140 | total_reward = 0 141 | 142 | while not self.done: 143 | ticks += 1 144 | a = self.agent.predict(self.belief, deterministic = deterministic)[0] 145 | self.step(a) 146 | total_reward += self.reward 147 | 148 | # if render: 149 | # self.carla_env.settings.no_rendering_mode = False 150 | # self.carla_env.world.apply_settings(self.carla_env.settings) 151 | 152 | self.agent.policy.set_training_mode(True) 153 | 154 | return ticks, total_reward, self.distance_at_go, 1 if self.waited_before_go else 0, 1 if self.collision else 0 155 | 156 | def train_agent(self, total_timesteps = 1000, iters = 10, print_debug = True): 157 | # i = iter 158 | # t = ticks 159 | # r = reward 160 | # d = distance at go 161 | # w = waited for the other car before go 162 | # c = collisions 163 | if print_debug: 164 | print("\ni\tt\tr\td\tw\tc") 165 | for i in range(iters): 166 | self.agent.learn(total_timesteps = total_timesteps) 167 | self.run_episodes(100, prefix = str(i), print_debug = print_debug) 168 | 169 | def run_episodes(self, n, prefix = "0", print_debug = True): 170 | ts = [] 171 | rs = [] 172 | ds = [] 173 | ws = [] 174 | cs = [] 175 | for i in range(n): 176 | t, r, d, w, c = self.run_episode() 177 | ts.append(t) 178 | rs.append(r) 179 | if d: ds.append(d) 180 | ws.append(w) 181 | cs.append(c) 182 | if len(ds) == 0: ds = [0] # to avoid warning in cases where zero gos were observed 183 | if print_debug: 184 | print(prefix, "\t", round(np.mean(ts),2), "\t", round(np.mean(rs),2), "\t", round(np.mean(ds),2), "\t", round(np.mean(ws),2), "\t", round(np.mean(cs),2), sep='') 185 | 186 | 187 | def kalman_update(self, prior_mean, prior_var, observation, s): 188 | """ 189 | Update the belief about x using the Kalman filter. 190 | 191 | Parameters: 192 | prior_mean (float): The mean of the prior belief about x. 193 | prior_var (float): The variance of the prior belief about x. 194 | observation (float): The noisy observation of x. 195 | s (float): The standard deviation of the observation noise. 196 | 197 | Returns: 198 | float, float: The mean and variance of the posterior belief about x. 199 | """ 200 | 201 | # The observation model has a gain of 1 (linear relationship) 202 | observation_gain = 1 203 | 204 | # Calculate the Kalman gain 205 | kalman_gain = prior_var * observation_gain / (prior_var * observation_gain**2 + s**2) 206 | 207 | # Update the mean and variance of the belief about x 208 | posterior_mean = prior_mean + kalman_gain * (observation - prior_mean) 209 | posterior_var = (1 - kalman_gain * observation_gain) * prior_var 210 | 211 | return posterior_mean, posterior_var 212 | -------------------------------------------------------------------------------- /03-Modelbuilding/go_nogo.py: -------------------------------------------------------------------------------- 1 | import importlib 2 | 3 | import matplotlib.pyplot as plt 4 | import numpy as np 5 | 6 | import driver_agent_physics 7 | import physics_env 8 | import animate_trace 9 | 10 | importlib.reload(driver_agent_physics) 11 | importlib.reload(physics_env) 12 | importlib.reload(animate_trace) 13 | 14 | def make_agent(sigma = 0, iters = 10, collision_reward = -10): 15 | e = physics_env.physics_env() 16 | agent = driver_agent_physics.driver_agent_physics(e, observation_var = sigma, collision_reward = collision_reward) 17 | agent.max_distance = 30 18 | agent.penalty_per_tick = 0.1 19 | e.veh1_straight_start_y_range = [-25,-2] 20 | agent.train_agent(total_timesteps = 10000, iters = iters) 21 | return agent 22 | 23 | def retrain_agent(agent, y_range = None, iters = 5): 24 | if y_range: 25 | agent.env.veh1_straight_start_y_range = [y_range[0],y_range[1]] 26 | agent.train_agent(total_timesteps = 10000, iters = iters) 27 | agent.env.veh1_straight_start_y_range = [-25,-2] 28 | 29 | def animate_agent(agent, y_start = None, get_anim = True, x_lim = [-50,50], y_lim = [50,-50]): 30 | agent.env.save_trace = True 31 | print(agent.run_episode(y_start = y_start)) 32 | agent.env.save_trace = False 33 | return animate_trace.animate_trace(agent.env.trace, get_anim = get_anim, x_lim = x_lim, y_lim = y_lim) 34 | 35 | def wait_or_go_experiment(agent, y_range, n = 100): 36 | data = [] 37 | a.env.veh1_straight_start_y_range = y_range 38 | for i in range(n): 39 | _, _, _, w, c = a.run_episode() 40 | data.append([a.observation_var, a.env.y_start, w, c]) 41 | 42 | agent.env.veh1_straight_start_y_range = [-25,-2] 43 | return data 44 | 45 | 46 | 47 | # Visualize the probability of go/no go as the function of y_start, between different sigmas. 48 | # Note that the lines may dip close to max y_start, this is an artefact of the smooting. 49 | 50 | # from scipy.stats import gaussian_kde 51 | 52 | # def estimate_probability(df, sigma, y_start_values): 53 | # sub_df = df[df['sigma'] == sigma] 54 | # kde = gaussian_kde(sub_df[['y_start', 'wait']].T) 55 | # probabilities = kde.evaluate(np.column_stack((y_start_values, np.ones_like(y_start_values))).T) 56 | # return probabilities 57 | 58 | # def plot_probability_lines(df): 59 | # fig, ax = plt.subplots() 60 | 61 | # y_start_range = np.linspace(df['y_start'].min(), df['y_start'].max(), num=500) 62 | 63 | # for sigma in df['sigma'].unique(): 64 | # probabilities = estimate_probability(df, sigma, y_start_range) 65 | # ax.plot(y_start_range, probabilities, label=f'Sigma: {sigma}') 66 | 67 | # ax.set_xlabel('y_start') 68 | # ax.set_ylabel('Probability of Wait') 69 | # ax.legend(title="Sigma", loc="upper left") 70 | # plt.title('Probability of wait across y_start by sigma') 71 | # plt.show() 72 | -------------------------------------------------------------------------------- /03-Modelbuilding/go_nogo_task.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/03-Modelbuilding/go_nogo_task.png -------------------------------------------------------------------------------- /03-Modelbuilding/physics_env.py: -------------------------------------------------------------------------------- 1 | import random 2 | import numpy as np 3 | 4 | 5 | class physics_env(): 6 | 7 | def __init__(self, veh1_straight_pos = [-1.825, -10], veh2_turn_pos = [3.65, 1.825], veh1_straight_vel = 8.94, veh2_turn_vel = 0.0 ,sim_freq = 0.05): 8 | self.veh1_straight_pos = np.array(veh1_straight_pos) 9 | self.veh1_straight_start_y_range = [-80,-2] 10 | self.veh2_turn_pos = np.array(veh2_turn_pos) 11 | self.veh1_straight_vel = veh1_straight_vel 12 | self.veh1_straight_vel_range = [5,20] 13 | self.veh2_turn_vel = veh2_turn_vel 14 | self.init_veh1_pos = np.array(veh1_straight_pos) 15 | self.init_veh2_pos = np.array(veh2_turn_pos) 16 | self.init_veh1_vel = veh1_straight_vel 17 | self.init_veh2_vel = veh2_turn_vel 18 | self.sim_freq = sim_freq 19 | self.time = 0 20 | self.save_trace = False 21 | 22 | def reset(self, y_start = None): 23 | self.veh1_straight_pos = self.init_veh1_pos 24 | self.veh1_straight_pos[1] = np.random.uniform(self.veh1_straight_start_y_range[0], 25 | self.veh1_straight_start_y_range[1]) 26 | if y_start: 27 | self.veh1_straight_pos[1] = y_start 28 | self.veh2_turn_pos = self.init_veh2_pos 29 | # self.veh1_straight_vel = np.random.uniform(self.veh1_straight_vel_range[0], 30 | # self.veh1_straight_vel_range[1]) 31 | self.y_start = self.veh1_straight_pos[1] # for logging 32 | self.veh2_turn_vel = self.init_veh2_vel 33 | self.trace = [] 34 | self.time = 0 35 | 36 | def get_random_b_spawn(self): 37 | return 38 | 39 | def get_state(self): 40 | vel = self.veh1_straight_vel 41 | # acc = self.actor_b.get_acceleration().x 42 | dist = self.get_distance() 43 | 44 | state = {} 45 | 46 | state['distance'] = dist 47 | state['speed'] = vel 48 | state['acceleration'] = 0 49 | 50 | return state 51 | 52 | def get_distance(self): 53 | return np.linalg.norm(self.veh1_straight_pos-self.veh2_turn_pos) 54 | 55 | def detect_collision(self): 56 | dist = np.fabs(self.veh1_straight_pos - self.veh2_turn_pos) 57 | return dist[1]<3.58 and dist[0]<1.645 58 | 59 | def simulate_go(self): 60 | done = False 61 | collision = False 62 | steps = 0 63 | self.veh2_turn_vel = 8.94 64 | # self.actor_a.enable_constant_velocity(carla.Vector3D(5,0.0,0.0)) 65 | while not done: 66 | steps += 1 67 | self.tick() 68 | # dist = np.linalg.norm(self.veh1_straight_pos-self.veh2_turn_pos) 69 | #print(dist) 70 | 71 | if self.detect_collision(): 72 | done = True 73 | collision = True 74 | # print(dist) 75 | # loc = self.actor_a.get_location() 76 | #print(loc.x, loc.y) 77 | # if self.veh2_turn_pos[0] <= 0 and self.veh2_turn_pos[1] <=-1.7: #old condition with loose constraint on y-axis position 78 | if self.veh2_turn_pos[0] <= -3.65: 79 | #print(self.veh2_turn_pos[0],self.veh2_turn_pos[1]) 80 | done = True 81 | # Stepped for too long: something wrong with the env? 82 | if steps > 1000: 83 | done = True 84 | 85 | return collision, steps 86 | 87 | def tick(self): 88 | self.time += self.sim_freq 89 | veh1_current_pos = self.veh1_straight_pos 90 | veh2_current_pos = self.veh2_turn_pos 91 | 92 | veh1_dist_travel = self.veh1_straight_vel*self.sim_freq 93 | veh2_dist_travel = self.veh2_turn_vel * self.sim_freq 94 | 95 | veh2_turn_radius = 13.301859 96 | veh2_turn_centre_x = -5.018750 97 | veh2_turn_centre_y = 11.406250 98 | 99 | self.veh1_straight_pos = self.veh1_straight_pos + np.array([0.0, veh1_dist_travel]) 100 | angle = np.arctan2(self.veh2_turn_pos[1]-veh2_turn_centre_y, 101 | self.veh2_turn_pos[0] - veh2_turn_centre_x) 102 | angle_increment = veh2_dist_travel/veh2_turn_radius 103 | angle -= angle_increment 104 | self.veh2_turn_pos = np.array([veh2_turn_radius*np.cos(angle)+veh2_turn_centre_x, 105 | veh2_turn_radius*np.sin(angle)+veh2_turn_centre_y]) 106 | 107 | if self.save_trace: 108 | self.trace.append([self.time, veh1_current_pos, veh2_current_pos, self.get_distance(), self.detect_collision()]) 109 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CogMod-Tutorial 2 | 3 | CogMod-Tutorial is a set of Python Jupyter Notebooks designed as an introduction to cognitive modeling in Human-Computer Interaction. 4 | 5 | Designed for delivery at CHI2022 and CHI2023, CHI2024 by Jussi Jokinen, Antti Oulasvirta and Andrew Howes: https://sites.google.com/view/modeling-chi24/ 6 | 7 | Before starting the tutorial it is worth knowing a little about the Python programming language and also about Jupyter Notebooks. If you are unfamiliar with notebooks and Python, you may want to start with the getting_started.ipynb notebook in the 01-Introduction/. Otherwise, you can start with the Fitts' Law notebook and proceed from there. 8 | --------------------------------------------------------------------------------