├── 01-Introduction
├── Fitts_Law.ipynb
├── fitts_task.png
└── getting_started.ipynb
├── 02-DeepRL
├── Gaze_Based_Interaction
│ ├── .ipynb_checkpoints
│ │ └── gaze_based_interaction-checkpoint.ipynb
│ ├── gaze_based_interaction.ipynb
│ ├── gazetools.py
│ ├── image
│ │ ├── cog_arch.png
│ │ ├── cognitive_POMDP.png
│ │ ├── gaze_task.png
│ │ ├── internal_env.png
│ │ ├── sub_movements.png
│ │ ├── time_v_target_size.png
│ │ └── visual_acuity.png
│ └── output
│ │ ├── behaviour_trace.csv
│ │ ├── monitor.csv
│ │ └── policy.zip
├── bayesian_state_estimation.ipynb
└── foveated_vision.ipynb
├── 03-Modelbuilding
├── Go_Nogo.ipynb
├── animate_trace.py
├── corati_model.png
├── driver_agent_physics.py
├── go_nogo.py
├── go_nogo_task.png
└── physics_env.py
└── README.md
/01-Introduction/Fitts_Law.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "attachments": {},
5 | "cell_type": "markdown",
6 | "id": "3e0c5f5d",
7 | "metadata": {},
8 | "source": [
9 | "[](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/01-Introduction/Fitts_Law.ipynb)\n"
10 | ]
11 | },
12 | {
13 | "attachments": {},
14 | "cell_type": "markdown",
15 | "id": "b1800c71-8555-4ffa-b993-249b80cb0234",
16 | "metadata": {},
17 | "source": [
18 | "# Fitts' Law\n",
19 | "\n",
20 | "Fitts' law provides a quantitative relationship between movement time in human pointing and the design of the pointing task. This model is instrumental in HCI for evaluating and designing user interfaces by optimizing the time and effort needed for common tasks. In this notebook, we will explore the mathematical underpinnings of Fitts' Law, demonstrating its application in analyzing and enhancing UI efficiency. The learning outcomes of this notebook are:\n",
21 | "\n",
22 | "* understanding of how mathematical formulas can describe psychological phenomena; and\n",
23 | "* being able to apply Fitts' law to make predictions of various UI designs.\n",
24 | "\n",
25 | "## Origins of Fitts' Law\n",
26 | "\n",
27 | "Fitts' Law was formulated in 1954 by psychologist Paul Fitts, originating from his empirical research on human motor skills. His work, deeply rooted in the principles of information theory and cybernetics, sought to understand the control aspects of human movement, particularly in the context of aiming or pointing tasks. Fitts' experiments led to the realization that a logarithmic relationship exists between the speed and accuracy of movements, encapsulated as an equation (the *Fitts' Law*). This discovery was pivotal in linking human cognitive processes with quantifiable metrics, laying a foundation for subsequent research in ergonomics, human factors, and HCI. Fitts' Law stands as a landmark in the history of cognitive psychology and its application to real-world problems.\n",
28 | "\n",
29 | "The original experiment conducted by Paul Fitts involved a simple setup to study the relationship between movement speed and target accuracy. Participants were asked to perform a series of rapid, repetitive pointing tasks using a hand-held stylus. The task required them to move the stylus back and forth between two flat, rectangular targets as quickly and accurately as possible. These targets varied in width (W) and distance (D) from each other. By manipulating these two parameters, Fitts observed how the speed of movement was influenced by the difficulty of the task, which he quantified as the ratio of the distance to the width of the target. This setup provided empirical data that led to the formulation of Fitts' Law, highlighting the logarithmic relationship between the time taken to complete the movement and the difficulty of the task.\n",
30 | "\n",
31 | "
\n",
32 | "Figure 1: Illustration of the original Fitts' experiment. Two target rectangles of width W are placed distance D apart, and the participant is tasked with moving a stylus between the targets."
33 | ]
34 | },
35 | {
36 | "cell_type": "markdown",
37 | "id": "65536e12-2ed3-4ea6-ad97-297adcbcdfb8",
38 | "metadata": {},
39 | "source": [
40 | "# Definition\n",
41 | "\n",
42 | "Fitts' Law is typically represented by the equation\n",
43 | "\n",
44 | "$\n",
45 | "MT = a + b \\cdot \\log_2\\left(\\frac{D}{W} + 1\\right),\n",
46 | "$\n",
47 | "\n",
48 | "where:\n",
49 | "- $MT$ is the movement time, the time it takes for a user to move the pointer to the target;\n",
50 | "- $a$ and $b$ are empirically determined constants through linear regression;\n",
51 | "- $D$ is the distance from the starting point to the center of the target; and\n",
52 | "- $W$ is the width of the target.\n",
53 | "\n",
54 | "We can define this as a function."
55 | ]
56 | },
57 | {
58 | "cell_type": "code",
59 | "execution_count": null,
60 | "id": "20223e8d-9749-489d-aa43-6e6950f0d3dd",
61 | "metadata": {},
62 | "outputs": [],
63 | "source": [
64 | "import numpy as np\n",
65 | "\n",
66 | "def fitts_law(D, W, a, b):\n",
67 | " \"\"\"\n",
68 | " Calculate the movement time as per Fitts' Law.\n",
69 | "\n",
70 | " Parameters:\n",
71 | " D (float): Distance from the starting point to the center of the target.\n",
72 | " W (float): Width of the target.\n",
73 | " a, b (float): Empirically determined constants.\n",
74 | "\n",
75 | " Returns:\n",
76 | " float: Estimated movement time.\n",
77 | " \"\"\"\n",
78 | " ID = np.log2(D / W + 1) # Index of Difficulty\n",
79 | " MT = a + b * ID\n",
80 | " return MT\n"
81 | ]
82 | },
83 | {
84 | "cell_type": "markdown",
85 | "id": "c1325bd7-66d0-4fbf-b138-197f7ad70785",
86 | "metadata": {},
87 | "source": [
88 | "Fitts' law is agnostic to the units of distance, because they cancel out in the equation. With UI design, a standard unit is a pixel. The predicted movement time is in seconds. Let's create a Fitts' law prediction for how a button width impacts movement time, when the distance between the origin of the pointing movement stays constant. You can explore how distance and the constants *a* and *b* impact predictions."
89 | ]
90 | },
91 | {
92 | "cell_type": "code",
93 | "execution_count": null,
94 | "id": "dc90c105-12eb-4f0b-a41b-20b87391d811",
95 | "metadata": {},
96 | "outputs": [],
97 | "source": [
98 | "import numpy as np\n",
99 | "import matplotlib.pyplot as plt\n",
100 | "\n",
101 | "# Fitts' Law function\n",
102 | "def fitts_law(D, W, a, b):\n",
103 | " ID = np.log2(D / W + 1) # Index of Difficulty\n",
104 | " MT = a + b * ID\n",
105 | " return MT\n",
106 | "\n",
107 | "# Constants (example values, should be determined empirically)\n",
108 | "a = 0.1\n",
109 | "b = 0.2\n",
110 | "\n",
111 | "# Fixed distance\n",
112 | "D = 1000 # example distance, you can adjust this\n",
113 | "\n",
114 | "# Generate a range of target widths\n",
115 | "target_widths = np.linspace(10, 600, 100)\n",
116 | "\n",
117 | "# Calculate movement times for each target width\n",
118 | "movement_times = [fitts_law(D, W, a, b) for W in target_widths]\n",
119 | "\n",
120 | "# Create the scatter plot\n",
121 | "plt.scatter(target_widths, movement_times)\n",
122 | "plt.title('Fitts\\' Law: Movement Time vs Target Width')\n",
123 | "plt.xlabel('Target Width (W)')\n",
124 | "plt.ylabel('Predicted Movement Time (s)')\n",
125 | "plt.grid(True)\n",
126 | "plt.show()"
127 | ]
128 | },
129 | {
130 | "cell_type": "markdown",
131 | "id": "dc77c452-7169-4c80-bcbe-c5465bcfe698",
132 | "metadata": {},
133 | "source": [
134 | "# Fitts' Law And UI Design\n",
135 | "\n",
136 | "Fitts' law can be directly used to predict movement times between UI elements, and summed to make predictions of trajectories. Let's create a simple code for creating mockup UIs and try out Fitts' law with it."
137 | ]
138 | },
139 | {
140 | "cell_type": "code",
141 | "execution_count": null,
142 | "id": "e1dc68a8-6b42-4fd3-b029-36157ed437d0",
143 | "metadata": {},
144 | "outputs": [],
145 | "source": [
146 | "%matplotlib inline\n",
147 | "\n",
148 | "import matplotlib.pyplot as plt\n",
149 | "import matplotlib.patches as patches\n",
150 | "from math import sqrt\n",
151 | "\n",
152 | "class element():\n",
153 | " def __init__(self, name, x = 0, y = 0, x_size = 1, y_size = 1, color = \"blue\"):\n",
154 | " self.name = name\n",
155 | " self.x = x\n",
156 | " self.y = y\n",
157 | " self.x_size = x_size\n",
158 | " self.y_size = y_size\n",
159 | " self.color = color\n",
160 | "\n",
161 | " def loc(self):\n",
162 | " return [int(self.x+self.x_size/2), int(self.y+self.y_size/2)]\n",
163 | " \n",
164 | " def get_size(self):\n",
165 | " return min(self.x_size, self.y_size)\n",
166 | "\n",
167 | " def distance_to(self, e):\n",
168 | " loc1 = self.loc()\n",
169 | " loc2 = e.loc()\n",
170 | " return sqrt( (loc1[0] - loc2[0])**2 + (loc1[1] - loc2[1])**2 )\n",
171 | "\n",
172 | "def draw_ui(elements, x_size = 0, y_size = 0, fig_width = 20, fig_height = 20):\n",
173 | " # Prepare plot\n",
174 | " plt.style.use('classic')\n",
175 | " fig, ax = plt.subplots()\n",
176 | " # Figure out size\n",
177 | " size_offset = 10 \n",
178 | " for e in elements:\n",
179 | " x_s = e.x + e.x_size + size_offset\n",
180 | " if x_size < x_s:\n",
181 | " x_size = x_s\n",
182 | " y_s = e.y + e.y_size + size_offset\n",
183 | " if y_size < y_s:\n",
184 | " y_size = y_s\n",
185 | " ax.set_xlim(0, x_size)\n",
186 | " ax.set_ylim(0, y_size)\n",
187 | " ax.invert_yaxis()\n",
188 | " ax.set_aspect('equal', adjustable='box')\n",
189 | " # Draw elements\n",
190 | " for e in elements:\n",
191 | " shape = patches.Rectangle((e.x, e.y), e.x_size, e.y_size, fc=e.color)\n",
192 | " ax.add_patch(shape)\n",
193 | " ax.text(e.x, e.y, str(e.name))\n",
194 | " plt.show()"
195 | ]
196 | },
197 | {
198 | "cell_type": "code",
199 | "execution_count": null,
200 | "id": "3fdc837b-f527-410e-8d47-4e3994e7d0b2",
201 | "metadata": {
202 | "scrolled": true
203 | },
204 | "outputs": [],
205 | "source": [
206 | "# Make a simple UI of four elements.\n",
207 | "elements = [\n",
208 | " element(\"logo\", x = 10, y = 10, x_size = 100, y_size = 100, color = \"blue\"),\n",
209 | " element(\"search\", x = 500, y = 10, x_size = 100, y_size = 20, color = \"grey\"),\n",
210 | " element(\"go\", x = 500, y = 200, x_size = 50, y_size = 50, color = \"red\")\n",
211 | "]\n",
212 | "draw_ui(elements)"
213 | ]
214 | },
215 | {
216 | "cell_type": "code",
217 | "execution_count": null,
218 | "id": "e149604b-c50b-454f-a9b9-2e1e883ae076",
219 | "metadata": {},
220 | "outputs": [],
221 | "source": [
222 | "# Create a wrapper for calculating Fitts' law between two elements: from element 1 to element 2.\n",
223 | "def fitts_mt(e1, e2, a = 0.1, b = 0.1):\n",
224 | " D = e1.distance_to(e2)\n",
225 | " W = e2.get_size() \n",
226 | " return fitts_law(D, W, a, b)\n",
227 | "\n",
228 | "def fitts_total_mt(elements, a = 0.1, b = 0.1):\n",
229 | " mt = 0\n",
230 | " for e in range(len(elements) - 1):\n",
231 | " mt += fitts_mt(elements[e], elements[e+1], a, b)\n",
232 | " return mt"
233 | ]
234 | },
235 | {
236 | "cell_type": "code",
237 | "execution_count": null,
238 | "id": "09265c14-383b-48f3-8c3d-08618f05ffbe",
239 | "metadata": {},
240 | "outputs": [],
241 | "source": [
242 | "# Calculate total movement time of pointing trajectory, starting from logo, going via search, and ending with go.\n",
243 | "fitts_total_mt(elements)"
244 | ]
245 | },
246 | {
247 | "cell_type": "markdown",
248 | "id": "11e6fa9f-a3ed-47f3-9d83-c1a8eb6ee0f9",
249 | "metadata": {},
250 | "source": [
251 | "With this general formulation of Fitts' law. We can investigate the impact of UI design (in this case, element size) on pointing time."
252 | ]
253 | },
254 | {
255 | "cell_type": "code",
256 | "execution_count": null,
257 | "id": "3bc41bfe-942c-4a58-af37-2cd3639cb522",
258 | "metadata": {},
259 | "outputs": [],
260 | "source": [
261 | "# make 'go' very small\n",
262 | "elements = [\n",
263 | " element(\"logo\", x = 10, y = 10, x_size = 100, y_size = 100, color = \"blue\"),\n",
264 | " element(\"search\", x = 500, y = 10, x_size = 1, y_size = 1, color = \"grey\"),\n",
265 | " element(\"go\", x = 10, y = 200, x_size = 1, y_size = 1, color = \"red\")\n",
266 | "]\n",
267 | "draw_ui(elements)\n",
268 | "print(fitts_total_mt(elements))"
269 | ]
270 | },
271 | {
272 | "cell_type": "code",
273 | "execution_count": null,
274 | "id": "82c21cae-074a-4ed1-b538-c88f717af19a",
275 | "metadata": {},
276 | "outputs": [],
277 | "source": [
278 | "# make 'go' very small\n",
279 | "elements = [\n",
280 | " element(\"logo\", x = 10, y = 10, x_size = 100, y_size = 100, color = \"blue\"),\n",
281 | " element(\"search\", x = 200, y = 10, x_size = 100, y_size = 100, color = \"grey\"),\n",
282 | " element(\"go\", x = 200, y = 120, x_size = 100, y_size = 100, color = \"red\")\n",
283 | "]\n",
284 | "draw_ui(elements)\n",
285 | "print(fitts_total_mt(elements))"
286 | ]
287 | },
288 | {
289 | "cell_type": "markdown",
290 | "id": "c5c6802f-38a9-4e19-9f6d-a78f74baa226",
291 | "metadata": {},
292 | "source": [
293 | "We can also use Fitts' law to calculate average time to a target, for instance assuming that pointing approaches the target uniformly from other UI elements."
294 | ]
295 | },
296 | {
297 | "cell_type": "code",
298 | "execution_count": null,
299 | "id": "20c7e964-5402-43e3-97c6-7b06c94d301b",
300 | "metadata": {},
301 | "outputs": [],
302 | "source": [
303 | "elements = [\n",
304 | " element(\"logo\", x = 10, y = 10, x_size = 100, y_size = 100, color = \"blue\"),\n",
305 | " element(\"header\", x = 150, y = 10, x_size = 300, y_size = 50, color = \"blue\"),\n",
306 | " element(\"search\", x = 500, y = 10, x_size = 100, y_size = 20, color = \"grey\"),\n",
307 | " element(\"go\", x = 500, y = 200, x_size = 50, y_size = 50, color = \"red\")\n",
308 | "]\n",
309 | "draw_ui(elements)\n",
310 | "fitts_total = 0\n",
311 | "for e in elements:\n",
312 | " if e.name != \"go\":\n",
313 | " # elements[-1] is the last element, which is \"go\"\n",
314 | " fitts_total += fitts_mt(e, elements[-1])\n",
315 | "# Print average\n",
316 | "print(fitts_total/(len(elements)-1))"
317 | ]
318 | },
319 | {
320 | "cell_type": "markdown",
321 | "id": "9f51f177-f956-4b35-bcdf-23d0103275b1",
322 | "metadata": {},
323 | "source": [
324 | "# Fitts' Law Parameters\n",
325 | "\n",
326 | "The parameters $a$ and $b$ in Fitts' law are fitted to a particular device and user. The parameter $a$ is the intercept, and it implies that regardless of how difficult a task is (long distance, small target), there is always some overhead associated with pointing, such as motor preparation. The parmaeter $b$ controls the slope: the larger this parameter, the more impactful distance and target width are on movement time.\n",
327 | "\n",
328 | "For instance, we can simulate a user with motor trouble by increasing both parameter values."
329 | ]
330 | },
331 | {
332 | "cell_type": "code",
333 | "execution_count": null,
334 | "id": "8e914eaf-f5da-4b4a-8eb2-b12239ccbc9d",
335 | "metadata": {},
336 | "outputs": [],
337 | "source": [
338 | "print(fitts_total_mt(elements, a = 0.1, b = 0.1))\n",
339 | "print(fitts_total_mt(elements, a = 0.2, b = 0.2))"
340 | ]
341 | },
342 | {
343 | "cell_type": "markdown",
344 | "id": "3f9d76ab-20c5-4c2e-bb00-3a4ef271374d",
345 | "metadata": {},
346 | "source": [
347 | "# Information Processing in Fitts' Law\n",
348 | "\n",
349 | "Fitts' law views motor manipulations as transmission of information between the human decision maker and the interactive system. A human user, equipped with a device such as a mouse, keyboard, joystick, eye movement sensor, or even a some sort of a neural link, transmits information to the system according to an index of performance (IP). The units of IP is bits/s, meaning that the higher the IP, the more information is being transmitted in a second. The task itself has an index of difficuly (ID), which determines how many bits need to be transmitted before the task is completed. Therefore, given movement time (MT), we define $IP = ID/MT$. As index of performance, given a user and a device, stays approximately constant, we can infer that as the task becomes more difficult (IP increases), movement time must equally increase.\n",
350 | "\n",
351 | "We can treat all \"motor\" actions by the user as transmission of information. For instance, perhaps a successful command will require 128 bits of information (this can encode, e.g., a short sentence). The IP computed for the human hand in the original Fitts' experiment is about 10 bits/s. We can solve how long it takes to transmit one sentence with the hand:\n",
352 | "\n",
353 | "$ IP = ID/MT $,\n",
354 | "$ MT = ID/IP $,\n",
355 | "$ MT = 128 bits / 10 bits/s = 12.8s $.\n",
356 | "\n",
357 | "In many time-critical tasks, this is too long. Fortunately, we can immediately cut the transmission time in half by assuming that two fingers use the keyboard to type. If we can make efficient use of ten-finger touch typing, we can achieve a theoretical transmission time of $1.28s$. This is a theoretical lower limit: generally a sentence cannot be optimally divided among 10 fingers.\n"
358 | ]
359 | },
360 | {
361 | "cell_type": "markdown",
362 | "id": "de171bb9-c313-4630-8d29-54f6ab3894b2",
363 | "metadata": {},
364 | "source": [
365 | "# Assignment\n",
366 | "\n",
367 | "Create a mock-up of a UI you are designing. Investigate how Fitts' law predicts movement time to a key element in the UI from various other elements (or all of them on average). Create a scatterplot that shows how the size of the target element impacts movement time, and investigate the shape of the plot: at what size does more size increase really not pay off anymore?"
368 | ]
369 | },
370 | {
371 | "cell_type": "markdown",
372 | "id": "fd8cf6d7-be8d-4c4c-b8bf-17f5906a7bae",
373 | "metadata": {},
374 | "source": [
375 | "# Sources\n",
376 | "\n",
377 | "- A good overview of Fitts' law in HCI: MacKenzie, I. S. (1992). Fitts' law as a research and design tool in human-computer interaction. Human-computer interaction, 7(1), 91-139.\n",
378 | "- The original paper: Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47, 381-391."
379 | ]
380 | },
381 | {
382 | "cell_type": "code",
383 | "execution_count": null,
384 | "id": "d7d2c89b-814d-4a66-b15a-a5e36fdb035d",
385 | "metadata": {},
386 | "outputs": [],
387 | "source": []
388 | }
389 | ],
390 | "metadata": {
391 | "kernelspec": {
392 | "display_name": "Python 3 (ipykernel)",
393 | "language": "python",
394 | "name": "python3"
395 | },
396 | "language_info": {
397 | "codemirror_mode": {
398 | "name": "ipython",
399 | "version": 3
400 | },
401 | "file_extension": ".py",
402 | "mimetype": "text/x-python",
403 | "name": "python",
404 | "nbconvert_exporter": "python",
405 | "pygments_lexer": "ipython3",
406 | "version": "3.12.3"
407 | }
408 | },
409 | "nbformat": 4,
410 | "nbformat_minor": 5
411 | }
412 |
--------------------------------------------------------------------------------
/01-Introduction/fitts_task.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/01-Introduction/fitts_task.png
--------------------------------------------------------------------------------
/01-Introduction/getting_started.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "id": "cardiovascular-breed",
6 | "metadata": {},
7 | "source": [
8 | "[](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/01-Introduction/getting_started.ipynb)\n"
9 | ]
10 | },
11 | {
12 | "cell_type": "markdown",
13 | "id": "operational-lloyd",
14 | "metadata": {},
15 | "source": [
16 | "# C28: Cognitive Modelling: From GOMS to Deep Reinforcement Learning\n",
17 | "\n",
18 | "\n",
19 | "\n",
20 | "\n",
21 | "Organizers:\n",
22 | "\n",
23 | "Jussi P. P. Jokinen, University of Jyväskylä
\n",
24 | "Antti Oulasvirta, Aalto University
\n",
25 | "Andrew Howes, University of Exeter\n",
26 | "\n",
27 | "Contact email: jussi.p.p.jokinen@jyu.fi\n",
28 | "\n",
29 | "This course introduces computational cognitive modeling for researchers and practitioners in the field of HCI. Cognitive models use computer programs to model how users perceive, think, and act in interactive tasks. They offer a powerful approach for understanding interaction and improving user interfaces. The course starts with a review of classic models such as Fitts's Law, GOMS and ACT-R. It builds on this review to provide an introduction to modern modelling approaches powered by machine learning methods, in particular deep learning, reinforcement learning (RL), and deep RL. The course is built around hands-on Python programming using Colab notebooks.\n",
30 | "\n",
31 | "A basic understanding of programming concepts is a prerequisite for the course and some familiarity with Python and Google Colab Notebooks (similar to Jupyter notebooks) is likely to be useful. Hopefully, you are reading this text having uploaded it to your private Google Colab -- if not then click on the \"Open in Colab\" link at the top of this page. This is a good start and it means that the document you are reading is not a static web page but, instead, an interactive environment. The key property of this environment is that it lets you write and execute code interactively. \n",
32 | "\n",
33 | "We will illustrate interactive code execution later in this notebook. We will also briefly review some of the historically important ideas in cognitive modelling for HCI. However, before we do that, we first state the learning objectives of the course and also propose a set of scientific objectives for a modern approach to cognitive modeling -- an approach that takes full advantage of recent advances in machine learning. The scientific objectives will be cruical to understanding the modeling approaches and techniques introduced later in the course. They will also be used as a basis for critiquing the historically important approaches from GOMS to ACT-R"
34 | ]
35 | },
36 | {
37 | "cell_type": "markdown",
38 | "id": "plastic-ghost",
39 | "metadata": {},
40 | "source": [
41 | "## Learning Objectives\n",
42 | "\n",
43 | "\n",
44 | "The learning objectives for the course are of two types, either \"understand...\" or \"be able to...\", where the latter is a practical skill. The objectives are grouped according to the three main modules of the course.\n",
45 | "\n",
46 | "* Understand the strengths and weaknesses of a cognitive model specified as a Partially Observable Markov Decision Process (POMDP) with foveated vision and Kalman state estimation.\n",
47 | "* Be able to use a Python reinforcement learning library -- running on a Colab server -- to find a solution to this POMDP.\n",
48 | "* Be able to test the above model under a range of conditions and generate results for comparison to human data.\n",
49 | "* Understand the HCI cognitive modelling problems for which RL is useful.\n",
50 | "\n",
51 | "\n",
52 | "* Understand the strengths and weaknesses of a cognitive model specified as an Artificial Neural Network (ANN).\n",
53 | "* Be able to make use of well-known pre-trained ANNs by incorporating Python libraries in your models.\n",
54 | "* Understand the HCI cognitive modelling problems for which ANNs are useful.\n",
55 | "\n",
56 | "\n",
57 | "* Understand modeling workflow, with an ability to provide a detailed case example.\n",
58 | "* Be able to use a reinforcement learning model of multitasking while driving.\n",
59 | "* Be able to use Bayesian likelihood free inference to fit model parameters.\n",
60 | "* Understand how parameterised simulation models can be used to explore design candidates."
61 | ]
62 | },
63 | {
64 | "cell_type": "markdown",
65 | "id": "medium-robinson",
66 | "metadata": {},
67 | "source": [
68 | "## Criteria for a modern approach to cognitive modeling\n",
69 | "\n",
70 | "While the tutorial seeks to cover a wide range of modeling techniques it is unashamedly committed to an emergent approach in cognitive science that is both routed in the history of information processing psychology and which seeks to take advantage of the recent convergence of cognitive science and HCI. Accordingly, the tutorial works towards introducing the student to modelling techniques which meet the following criteria:\n",
71 | "\n",
72 | "* Be firmly grounded in the information processing view of cognition (Miller, 1956).\n",
73 | "* Explain the adaptation of human behaviour to the environment (including the designed environment) (Brunswick, 1943). Brunswik proposed that humans be explained in terms of statistical descriptions of the structure of the environment. In HCI, models ought to be tested on tasks that are representative of the environment that computer users inhabit.\n",
74 | "* Build on Simon's (1955) insight that adaptation is bounded by two sources of constraint; not only those on on the environment, but also constraints on the mind, including perceptual/motor and memory constraints.\n",
75 | "* Clearly distinguish the explanatory role of invariant bounds on the mind, such as memory limits and visual acuity, from the strategies that are learned through experience, for example rehersal or knowing where to look for important information when a web page is opened. While people are bounded, much of what computer users do is strategic (Oulasvirta, et al., 2022).\n",
76 | "* Formally define theories of cognition as optimisation problems that incorporate bounds on both the environment and cognition (Lewis, Howes, Singh, 2014; Lieder, Griffiths, 2020). Human-like strategies can then be predicted by solving such optimisation problems.\n",
77 | "* Make use of the convergence between machine learning and cognitive science (Gershman, Tenenbaum and Horvitz, 2015), both to specify theories of the bounds on cognition and to derive the strategies that are a response to these bounds. Machine learning has the potential to add both rigour and convenience to the process of cognitive modeling in HCI.\n",
78 | "* Test theories of cognition using inverse modeling (Kangasrääsiö, et al., 2019).\n",
79 | "\n",
80 | "These are a challenging set of criteria that are not met in their entirety by any single cognitive model. However, they point in a direction for the future and can be used as a basis of understanding the relative stengths and weaknesses of each existing approach."
81 | ]
82 | },
83 | {
84 | "cell_type": "markdown",
85 | "id": "catholic-guess",
86 | "metadata": {},
87 | "source": [
88 | "## Background\n",
89 | "\n",
90 | "This section shows you how to interact with the notebooks.\n",
91 | "\n",
92 | "**Fitts's Law**\n",
93 | "\n",
94 | "Below find a short Python script -- in a code cell -- that computes a predicted movement time MT given the distance D and width W of a button on a screen."
95 | ]
96 | },
97 | {
98 | "cell_type": "code",
99 | "execution_count": null,
100 | "id": "roman-norfolk",
101 | "metadata": {},
102 | "outputs": [],
103 | "source": [
104 | "from math import *\n",
105 | "\n",
106 | "a = 1.03\n",
107 | "b = 0.096\n",
108 | "D = 8\n",
109 | "W = 1.25\n",
110 | "\n",
111 | "MT = a + b * log2(0.5+D/W)\n",
112 | "MT"
113 | ]
114 | },
115 | {
116 | "cell_type": "markdown",
117 | "id": "three-pastor",
118 | "metadata": {},
119 | "source": [
120 | "You can execute the code in the above cell by selecting it with a click and then pressing the play button to the left of the code, or by using the keyboard shortcut 'Command/Ctrl+Enter'. \n",
121 | "\n",
122 | "To edit the code, click the cell and start typing. For example, you can predict MT for a different distance by changing the value of D and re-running the cell. Try it now.\n",
123 | "\n",
124 | "Some readers may recognise the above formula as Fitts's Law, but whether you recognise it or not, you have have now used Python and Colab to execute your first model of a user. Congratulations, you are on the path to mastery!\n",
125 | "\n",
126 | "Fitts's Law illustrates one key property of models of users in HCI which is that they make predictions. If parameters 'a' and 'b' are known then Fitts's Law can be used to predict movement time given the distance and width of a target. For now, lets continue to learn about Colab's notebooks and Python. One important thing to know is that variables that you define in one cell can be used in other cells. For example, the value of MT is available in the code cell below:"
127 | ]
128 | },
129 | {
130 | "cell_type": "code",
131 | "execution_count": null,
132 | "id": "african-queens",
133 | "metadata": {},
134 | "outputs": [],
135 | "source": [
136 | "MT"
137 | ]
138 | },
139 | {
140 | "cell_type": "markdown",
141 | "id": "standing-harvest",
142 | "metadata": {},
143 | "source": [
144 | "Code cells can also be used to define Python functions."
145 | ]
146 | },
147 | {
148 | "cell_type": "code",
149 | "execution_count": null,
150 | "id": "local-stamp",
151 | "metadata": {},
152 | "outputs": [],
153 | "source": [
154 | "def fitts_mt(D,W):\n",
155 | " a = 1.03\n",
156 | " b = 0.096\n",
157 | " MT = a + b * log2(0.5+D/W)\n",
158 | " return MT"
159 | ]
160 | },
161 | {
162 | "cell_type": "markdown",
163 | "id": "lesbian-investigation",
164 | "metadata": {},
165 | "source": [
166 | "Run the code cell above to define the function. Once the function is defined, we can call it from other code cells."
167 | ]
168 | },
169 | {
170 | "cell_type": "code",
171 | "execution_count": null,
172 | "id": "flying-structure",
173 | "metadata": {},
174 | "outputs": [],
175 | "source": [
176 | "fitts_mt(2,1.25)"
177 | ]
178 | },
179 | {
180 | "cell_type": "markdown",
181 | "id": "instant-practice",
182 | "metadata": {},
183 | "source": [
184 | "Another important feature of Colab notebooks is that they can be used to visualise data. In the code cell below our fitts_mt() function is used to visualise the relationship between MT and distance. "
185 | ]
186 | },
187 | {
188 | "cell_type": "code",
189 | "execution_count": null,
190 | "id": "crude-lesson",
191 | "metadata": {
192 | "scrolled": true
193 | },
194 | "outputs": [],
195 | "source": [
196 | "import numpy as np\n",
197 | "from matplotlib import pyplot as plt\n",
198 | "\n",
199 | "xs = [x for x in range(1,17)]\n",
200 | "ys = [fitts_mt(x,1.25) for x in xs]\n",
201 | "\n",
202 | "plt.plot(xs, ys, '-')\n",
203 | "plt.xlabel(\"Distance D\")\n",
204 | "plt.ylabel(\"Movement time MT\")\n",
205 | "plt.title(\"Fitts's Law\")\n",
206 | "plt.show()"
207 | ]
208 | },
209 | {
210 | "cell_type": "markdown",
211 | "id": "developmental-stupid",
212 | "metadata": {},
213 | "source": [
214 | "You may have noticed the use of the Python import statement in the above code cells. As their name suggests these import code from libraries. There are many basic libraries in python that are used for maths and plotting, for example. More exciting are the libraries that support machine learning. We will see as the tutorial progresses that these libraries can be very useful for modeling cognition."
215 | ]
216 | },
217 | {
218 | "cell_type": "markdown",
219 | "id": "dutch-authorization",
220 | "metadata": {},
221 | "source": [
222 | "
\n",
223 | "
\n",
224 | "
\n",
225 | "\n",
226 | "Reflection
\n",
227 | "Fitts's Law has proved highly influential in HCI since it was used by Card, English and Burr (1978) to compare various input devices. There are many research papers that describe more recent developments, the application of Fitts's Law, and a number of limitations. One limitation is the fact that the law says nothing about speed/accuracy trade-offs, nor anything about errors; thereby failing to meet the demand for explaining adaptation in our above criteria for modelling. When a person moves a pointer to a target, they can choose whether to move quickly or slowly. If they choose to move quickly then the chance that they will make an error will increase. This is a topic that we will return to later in the tutorial when we look at reinforcement learning models of cognition. In these models, a reward function determines the desired speed/accuracy trade-off function and predictions of movement time emerge through learning an optimal policy for the reward function. The ability to model speed/accuracy trade-offs is critical to the scientific understanding of HCI because of the adaptive nature of human cognition.\n",
228 | "
\n",
229 | "
"
230 | ]
231 | },
232 | {
233 | "cell_type": "markdown",
234 | "id": "opposite-calculator",
235 | "metadata": {},
236 | "source": [
237 | "**GOMS**\n",
238 | "\n",
239 | "GOMS was an approach to cognitive modeling in Human-Computer Interaction in which human skill for computing systems was represented in terms of goals, operators, methods and selection rules. Many other methods for modeling task knowledge were based on similar concepts and GOMS gave rise to many variants through the 1980s and into the 2000s. Though their meanings have been refined, some GOMS concepts remain useful today, and it is worth briefly reviewing them. \n",
240 | "\n",
241 | "Goals are what the user has to accomplish and represent the end towards which the user is directed. Example goals might include making a set of changes to a text document or replying to an email. Goals typically have subgoals giving rise to the hierarchical structure of skill. For example, the goal of correcting the typos in a text might have as subgoals to \"delete text\" and \"insert word\". \n",
242 | "\n",
243 | "Operators model what a user can do in service of a goal. Operators may be theories of how a user makes changes to the external environment or they may be theories of mental operators that, for example, update memory. The very lowest level of operator could be indiviudal muscle axon innervations but cognitive models typically define operators at a much higher level than this. For example, a cognitive model of form filling might define operators at the word entry level, whereas a cognitive model of typing might define operators at a character level. \n",
244 | "\n",
245 | "Methods are sequences of subgoals and operators that achieve specific goals. For example, the goal of entering a name into a form might be accomplished with methods that we can represent with Python functions."
246 | ]
247 | },
248 | {
249 | "cell_type": "code",
250 | "execution_count": null,
251 | "id": "precious-cliff",
252 | "metadata": {},
253 | "outputs": [],
254 | "source": [
255 | "def enter_name():\n",
256 | " move_caret_to_location()\n",
257 | " print(\"ENTER NAME\")\n",
258 | "\n",
259 | "def move_caret_to_location():\n",
260 | " print(\"DETECT LOCATION\")\n",
261 | " print(\"MOVE MOUSE UNTIL POINTER AT LOCATION\")\n",
262 | " print(\"CLICK MOUSE\")\n",
263 | "\n",
264 | "enter_name()"
265 | ]
266 | },
267 | {
268 | "cell_type": "markdown",
269 | "id": "gentle-stanford",
270 | "metadata": {},
271 | "source": [
272 | "When the code cell above is executed, the GOMS methods for the goal ENTER NAME are expanded resulting in the required operator sequence being printed.\n",
273 | "\n",
274 | "Selection Rules choose between different methods for achieving a goal. For example, an alternative method for ENTER NAME might use TAB to move the caret from an earlier location rather than using the mouse. The selection rule might prefer one method or the other depending on the sequentiality of the entry process.\n",
275 | "\n",
276 | "The difference between a goal and an operator in GOMS is that operators are not broken down any further; they are at the bottom of a hierarchy of subgoals. Card, Moran and Newell demonstrated GOMS models at 4 levels of analysis. At one level the operators represented tasks taking about 30 seconds, at another the operators represented single keystrokes. As with Fitts's Law, the idea was that GOMS could be used to predict human performance time by adding up the durations of the sequence of operators required to achieve a goal. \n",
277 | "\n",
278 | "\n",
279 | "
\n",
280 | "
\n",
281 | "\n",
282 | "Reflection
\n",
283 | "GOMS embodied a number of concepts tha remain important to modeling today. These include the idea that human task knowledge is hierarchical and that goals are decomposed into subgoals and eventually operators. This ideas echoes in modern ideas concerning hierarchical reinforcement learning, for example.\n",
284 | "\n",
285 | "
But, there are a number of concepts missing in GOMS that are crucial to modern analyses of task knowledge. One of these concepts is that of utility (or reward). Another is the extent to which task knowledge is now known to be conditioned on state; that is on bottom-up information and not just on top-down hierarchical control. Cognition is embodied and interactive. GOMS therefore fails to meet the demand of our criteria; cognitive models must be sensitive to the statistical structure of the task environment; in other words the ecology.\n",
286 | " \n",
287 | "
In addition, GOMS models are difficult to build because they demand that the analyst hand-code a set of rules that represent a user's task knowledge -- which can require many interlated rules and be difficult to determine from human behaviour. GOMS, therefore, fails to meet our demand that cognitive modeling approaches should take full advantage of the recent convergence between ML and cognitive science. This convergence provides methods for learning task knowledge that, in for some tasks, results in human-level performance.\n",
288 | "
\n",
289 | "
\n"
290 | ]
291 | },
292 | {
293 | "cell_type": "markdown",
294 | "id": "unexpected-productivity",
295 | "metadata": {},
296 | "source": [
297 | "**Cognitive architectures**\n",
298 | "\n",
299 | "Cognitive architectures, particuarly ACT-R (Anderson, 2007) and EPIC (Kieras and Meyer, 1997), have played a crucial role in cognitive modeling for HCI. \n",
300 | "\n",
301 | "ACT-R consists of a number of integrated modules for modeling cognition. The modules include those for audition, vision, manual movement, declarative memory, and temporal memory. These modules are motivated by neuroscientific and behavioural evidence and empirically validated by many hundreds of human experiments. The modules are coordinated by a central production module which is responsible for matching, selecting and executing production rules. These rules make changes to the state of the other modules by, for example, issuing motor commands, or updating memories. ACT-R also provides theories of learning. In ACT-R's theory of procedural learning, skills are acquired by recruiting past problem solutions while solving new problems, thereby providing a theory of learning by doing and learning by example (Anderson and Schunn, 2000). ACT-R also models activation learning in declarative memory. Two factors influence this learning: how many times an item in memory was needed in the past, and how recently it was needed. The activation learning rule is derived from Bayesian statistics. ACT-R uses activation values of items in memory to order the chunks in the matching process.\n",
302 | "\n",
303 | "\n",
304 | "\n",
305 | "
\n",
306 | "
\n",
307 | "\n",
308 | "Reflection
\n",
309 | "One of the most significant theoretical contributions to HCI, information foraging theory, was grounded in ACT-R theory (Pirolli, 1997). EPIC has also played a significant role, for example, in models of visual search (Kieras and Hornoff, 2014). Arguably, ACT-R represents the state of the art in computational models of human memory.\n",
310 | "\n",
311 | "
ACT-R and EPIC meet many of the criteria for cognitive modeling that we set at the beginning of this notebook. They are both bounded information processing theories of cognition and they embrace the notion of adaptation. In ACT-R's case adaptation is modelled through various learning mechanisms built into the architecture. These can both be used to generate multiple strategies and to select between them, meeting another criteria. However, ACT-R and EPIC do not permit the definition of theories of cognition as optimisation problems in response to which modern machine learning algorithms can be used to automatically find human-like strategies. Instead, even with learning researchers are still required to program some production rules, considerably increasing the burden on the modeler, and also increasing the parametric flexibility of the model leading to difficulties associated with fitting ACT-R models to data.\n",
312 | " \n",
313 | "
\n",
314 | "
\n"
315 | ]
316 | },
317 | {
318 | "cell_type": "markdown",
319 | "id": "divided-congress",
320 | "metadata": {},
321 | "source": [
322 | "## Discussion\n",
323 | "\n",
324 | "You are now at the end of the \"getting started\" module of the CHI'2023 course on cognitive modeling. Well done, for getting this far. \n",
325 | "\n",
326 | "We have attempted to cover over 40 years of cognitive modeling in HCI in a few hundred words; as a consequence we are aware of many many omissions. However, our main purpose was to illustrate the contrast between the well-known approaches in HCI and the modern -- machine learning fuelled -- approach that has been emerging over the past few years. This new approach, sometimes known as computational rationality (or resource rationality), begins to meet all of the criteria for cognitive modeling that we articulated at the beginnig of this document. Accordingly, the main body of the tutorial will pick up on these criteria to motivate the use deep learning and deep reinforcement learning in cognitive models for HCI.\n",
327 | "\n",
328 | "If you have time to find out more then recommended, and highly recommended papers, are starred and double starred in the bibliogrpahy below. To find out more about Colab notebooks, see Overview of Colab. Colab notebooks are Jupyter notebooks that are hosted by Colab. To find out more about the Jupyter project, see jupyter.org.\n",
329 | "\n",
330 | "If you are a novice Python programmer then you should find out more about lists, dictionaries and about classes/objects.\n",
331 | "\n",
332 | "Further modules will be provided on the day of the tutorial. We look forward to seeing you there.\n",
333 | "\n",
334 | "Jussi Jokinen
\n",
335 | "Antti Oulasvirta
\n",
336 | "Andrew Howes\n"
337 | ]
338 | },
339 | {
340 | "cell_type": "markdown",
341 | "id": "brief-vehicle",
342 | "metadata": {},
343 | "source": [
344 | "**References**\n",
345 | "\n",
346 | "Anderson, J. R. (2007). How Can the Human Mind Exist in the Physical Universe? Oxford University Press.\n",
347 | "\n",
348 | "Card, S. M., & Newell, T. (1983). A.(1983)“The Psychology of Human-Computer Interaction.”.\n",
349 | "\n",
350 | "Card, S. K., English, W. K., Burr, B. J., 1978. Evaluation of mouse, rate controlled isometric joystick, step keys and text keys for text selection on a CRT. Ergonomics 21, 601-613.\n",
351 | "\n",
352 | "\\* Gershman, S. J., Horvitz, E. J., & Tenenbaum, J. B. (2015). Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science, 349(6245), 273-278.\n",
353 | "\n",
354 | "John, B. E., & Kieras, D. E. (1996). The GOMS family of user interface analysis techniques: Comparison and contrast. ACM Transactions on Computer-Human Interaction (TOCHI), 3(4), 320-351.\n",
355 | "\n",
356 | "\\* Kangasrääsiö, A., Jokinen, J. P., Oulasvirta, A., Howes, A., & Kaski, S. (2019). Parameter inference for computational cognitive models with Approximate Bayesian Computation. Cognitive science, 43(6), e12738.\n",
357 | "\n",
358 | "Kieras, D. E., & Meyer, D. E. (1997). An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Human–Computer Interaction, 12(4), 391-438.\n",
359 | "\n",
360 | "Kieras, D. E., & Hornof, A. J. (2014, April). Towards accurate and practical predictive models of active-vision-based visual search. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 3875-3884).\n",
361 | "\n",
362 | "** Lewis, R. L., Howes, A., & Singh, S. (2014). Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in cognitive science, 6(2), 279-311.\n",
363 | "\n",
364 | "** Lieder, F., & Griffiths, T. L. (2020). Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences, 43.\n",
365 | "\n",
366 | "Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2), 81.\n",
367 | "\n",
368 | "\\* Pirolli, P. (1997). Computational models of information scent-following in a very large browsable text collection. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (pp. 3-10).\n",
369 | "\n",
370 | "Simon, H. A. (1955). A behavioral model of rational choice. The quarterly journal of economics, 69(1), 99-118.\n"
371 | ]
372 | }
373 | ],
374 | "metadata": {
375 | "kernelspec": {
376 | "display_name": "Python 3 (ipykernel)",
377 | "language": "python",
378 | "name": "python3"
379 | },
380 | "language_info": {
381 | "codemirror_mode": {
382 | "name": "ipython",
383 | "version": 3
384 | },
385 | "file_extension": ".py",
386 | "mimetype": "text/x-python",
387 | "name": "python",
388 | "nbconvert_exporter": "python",
389 | "pygments_lexer": "ipython3",
390 | "version": "3.9.5"
391 | }
392 | },
393 | "nbformat": 4,
394 | "nbformat_minor": 5
395 | }
396 |
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/.ipynb_checkpoints/gaze_based_interaction-checkpoint.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "attachments": {},
5 | "cell_type": "markdown",
6 | "id": "3e0c5f5d",
7 | "metadata": {},
8 | "source": [
9 | "[](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/02-DeepRL/Gaze_Based_Interaction/gaze_based_interaction.ipynb)\n"
10 | ]
11 | },
12 | {
13 | "cell_type": "markdown",
14 | "id": "fitted-component",
15 | "metadata": {
16 | "id": "fitted-component"
17 | },
18 | "source": [
19 | "# A cognitive model of gaze-based interaction\n",
20 | "\n",
21 | "Some cognitive models describe behaviour and others predict it. Models that predict behaviour must be capable of generating output without that output being part of the input. In this notebook we demonstrate this property for a model of eye movements. The model is a reinforcement learner that is trained by performing hundreds of thousands of simulated eye movements in search of a target of varying size and distance. The model predicts how many eye movements people will tend to make to find a target of a given size and distance and, in addition, that the first eye movement to a target will tend to undershoot, rather than overshoot.\n",
22 | "\n",
23 | "### The task: Gaze-based interaction\n",
24 | "\n",
25 | "Gaze-based interaction is a mode of interaction in which users, including users with a range of movement disabilities, are able to indicate which display item they wish to select by fixating their eyes on it. Confirmation of selection is then made with one of a number of methods, including with a key press, or by holding the fixation for an extended duration of time. The model performs this task for targets with randomly selected location and size. \n",
26 | "\n",
27 | "<>\n",
28 | "\n",
29 | "### Model architecture\n",
30 | "\n",
31 | "The model has a simple architecture that you have previously seen in the introduction. The figure is reproduced here:\n",
32 | "\n",
33 | "
\n",
34 | "\n",
35 | "- The **control** module makes decisions about where to move the eyes with the oculomotor system. Decisions are conditioned on the current belief about the location of the target.\n",
36 | "- The **motor** module implements decisions but it is bounded by Gaussian noise, which models noise in the human motor system.\n",
37 | "- The **environment** models the physics of the world and the task (the location of the target). Given a response from the motor system, a saccade is made to the aim point of the eyes, and a fixation is initiated.\n",
38 | "- The **perception** module simulates the human capacity to localize a target with foveated vision. The accuracy of the location estimate generated by perception is negatively affected by the eccentricity of the target from the current fixation location.\n",
39 | "- The **Memory** module stores a representation of the current state. Over the course of an episode a sequence of location estimates will be made. Humans are known to integrate these estimates into a single integrated representation of the location. People are known to do this optimally using a process that can be modelled with Bayesian state estimation. The state estimation constitutes a belief about the location of the target.\n",
40 | "- The **Utility** module calculates a reward signal given the current belief about the enviornment. The reward signal is used to train the controller.\n",
41 | "\n",
42 | "### Prerequisites\n",
43 | "\n",
44 | "Before proceeding with this notebook you should firrst review the notebooks on foveated vision and on Bayesian integration."
45 | ]
46 | },
47 | {
48 | "cell_type": "markdown",
49 | "id": "a427be22",
50 | "metadata": {},
51 | "source": [
52 | "### Machine learning\n",
53 | "\n",
54 | "In order to learn how to perform the task, the model uses a set of implementations of reinforcement learning algorithms in PyTorch known as stable-baselines3."
55 | ]
56 | },
57 | {
58 | "cell_type": "code",
59 | "execution_count": null,
60 | "id": "vL8yMY6q_Rd-",
61 | "metadata": {
62 | "id": "vL8yMY6q_Rd-"
63 | },
64 | "outputs": [],
65 | "source": [
66 | "# Install stable_baselines3\n",
67 | "# This is a well known machine learning library that provides a suite of reinforcement learning methods.\n",
68 | "# Only needs to be run once\n",
69 | "\n",
70 | "!pip install --pre -U stable_baselines3"
71 | ]
72 | },
73 | {
74 | "cell_type": "code",
75 | "execution_count": null,
76 | "id": "oeJU5wJR7A8V",
77 | "metadata": {
78 | "colab": {
79 | "base_uri": "https://localhost:8080/"
80 | },
81 | "executionInfo": {
82 | "elapsed": 1652,
83 | "status": "ok",
84 | "timestamp": 1643210006123,
85 | "user": {
86 | "displayName": "Andrew Howes",
87 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64",
88 | "userId": "02694399383679444060"
89 | },
90 | "user_tz": 0
91 | },
92 | "id": "oeJU5wJR7A8V",
93 | "outputId": "f8e53d98-820c-4a31-b9f5-e4b5c46bd72b"
94 | },
95 | "outputs": [],
96 | "source": [
97 | "# Ignore this cell unless you need to save results to Google drive.\n",
98 | "# Mount Google drive and change directory into the project folder.\n",
99 | "# from google.colab import drive\n",
100 | "# drive.mount('/content/drive')\n",
101 | "# %cd '/content/drive/MyDrive/CHI22CMT/CHI22_CogMod_Tutorial/03-Reinforcement-Learning/034_Gaze_based_Interaction'"
102 | ]
103 | },
104 | {
105 | "cell_type": "code",
106 | "execution_count": null,
107 | "id": "P44tljjb8hBC",
108 | "metadata": {
109 | "colab": {
110 | "base_uri": "https://localhost:8080/"
111 | },
112 | "executionInfo": {
113 | "elapsed": 6305,
114 | "status": "ok",
115 | "timestamp": 1643209694066,
116 | "user": {
117 | "displayName": "Andrew Howes",
118 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64",
119 | "userId": "02694399383679444060"
120 | },
121 | "user_tz": 0
122 | },
123 | "id": "P44tljjb8hBC",
124 | "outputId": "d635fb70-1df0-4ab3-f15e-13b3d7b2fdc9"
125 | },
126 | "outputs": [],
127 | "source": [
128 | "# Load local modules\n",
129 | "# gazetools is a module that contains functions for modeling gaze-based interaction.\n",
130 | "\n",
131 | "!wget https://raw.githubusercontent.com/howesa/CHI22-CogMod-Tutorial/main/03-Reinforcement-Learning/034_Gaze_based_Interaction/gazetools.py\n",
132 | "\n",
133 | "import gazetools"
134 | ]
135 | },
136 | {
137 | "cell_type": "code",
138 | "execution_count": null,
139 | "id": "obvious-point",
140 | "metadata": {
141 | "executionInfo": {
142 | "elapsed": 358,
143 | "status": "ok",
144 | "timestamp": 1643209000116,
145 | "user": {
146 | "displayName": "Andrew Howes",
147 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64",
148 | "userId": "02694399383679444060"
149 | },
150 | "user_tz": 0
151 | },
152 | "id": "obvious-point"
153 | },
154 | "outputs": [],
155 | "source": [
156 | "# load required standard modules and configure matplotlib\n",
157 | "\n",
158 | "import numpy as np\n",
159 | "import math\n",
160 | "import matplotlib.pyplot as plt\n",
161 | "import sys\n",
162 | "\n",
163 | "import gymnasium as gym\n",
164 | "sys.modules[\"gym\"] = gym\n",
165 | "from gym import spaces\n",
166 | "\n",
167 | "import matplotlib as mpl\n",
168 | "%matplotlib inline\n",
169 | "mpl.style.use('ggplot')"
170 | ]
171 | },
172 | {
173 | "cell_type": "markdown",
174 | "id": "injured-leadership",
175 | "metadata": {
176 | "id": "injured-leadership"
177 | },
178 | "source": [
179 | "### Implementation of the Cognitive Architecture as a Python Class\n",
180 | "\n",
181 | "The first step to formalise the model architecture presented in the above figure. We do this by specifying a class of cognitive theories and will later define instances of this class.\n",
182 | "\n",
183 | "The class has only a single method, which defines a step through the processes defined in the figure. "
184 | ]
185 | },
186 | {
187 | "cell_type": "code",
188 | "execution_count": null,
189 | "id": "monetary-ivory",
190 | "metadata": {
191 | "executionInfo": {
192 | "elapsed": 270,
193 | "status": "ok",
194 | "timestamp": 1643209727593,
195 | "user": {
196 | "displayName": "Andrew Howes",
197 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64",
198 | "userId": "02694399383679444060"
199 | },
200 | "user_tz": 0
201 | },
202 | "id": "monetary-ivory"
203 | },
204 | "outputs": [],
205 | "source": [
206 | "class CognitivePOMDP():\n",
207 | "\n",
208 | " def __init__(self):\n",
209 | " self.internal_state = {}\n",
210 | " \n",
211 | " def step(self, ext, action):\n",
212 | " ''' Define the cognitive POMDP.'''\n",
213 | " self._update_state_with_action(action)\n",
214 | " response = self._get_response()\n",
215 | " external_state, done = ext.external_env(response)\n",
216 | " stimulus, stimulus_std = self._get_stimulus(ext.external_state)\n",
217 | " self._update_state_with_stimulus(stimulus, stimulus_std)\n",
218 | " obs = self._get_obs()\n",
219 | " reward = self._get_reward()\n",
220 | " return obs, reward, done"
221 | ]
222 | },
223 | {
224 | "cell_type": "markdown",
225 | "id": "brazilian-timeline",
226 | "metadata": {
227 | "id": "brazilian-timeline"
228 | },
229 | "source": [
230 | "### A theory of gaze-based interaction\n",
231 | "\n",
232 | "Each of the entities in CognitivePOMDP must be defined so as to state our theory of gaze-based interaction. The theory makes the following assumptions:\n",
233 | "\n",
234 | "* Target location stimuli are corrupted by Gaussian noise in human vision.\n",
235 | "* The standard deviation of noise increases linearly with eccentricity from the fovea.\n",
236 | "* Sequences of stimuli are noisily perceived and optimally integrated.\n",
237 | "* Intended eye movements (oculomotor actions) are corrupted by signal dependent Gaussian noise to generate responses.\n",
238 | "\n",
239 | "These assumptions are further described in Chen et al. (2021)."
240 | ]
241 | },
242 | {
243 | "cell_type": "code",
244 | "execution_count": null,
245 | "id": "accepted-reunion",
246 | "metadata": {
247 | "executionInfo": {
248 | "elapsed": 242,
249 | "status": "ok",
250 | "timestamp": 1643209731182,
251 | "user": {
252 | "displayName": "Andrew Howes",
253 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64",
254 | "userId": "02694399383679444060"
255 | },
256 | "user_tz": 0
257 | },
258 | "id": "accepted-reunion"
259 | },
260 | "outputs": [],
261 | "source": [
262 | "class GazeTheory(CognitivePOMDP):\n",
263 | "\n",
264 | " def __init__(self):\n",
265 | " ''' Initialise the theoretically motivated parameters.'''\n",
266 | " # weight eye movement noise with distance of saccade\n",
267 | " self.oculamotor_noise_weight = 0.01\n",
268 | " # weight noise with eccentricity\n",
269 | " self.stimulus_noise_weight = 0.09\n",
270 | " # step_cost for the reward function\n",
271 | " self.step_cost = -1\n",
272 | " # super.__init__()\n",
273 | "\n",
274 | " def reset_internal_env(self, external_state):\n",
275 | " ''' The internal state includes the fixation location, the latest estimate of \n",
276 | " the target location and the target uncertainty. Assumes that there is no \n",
277 | " uncertainty in the fixation location.\n",
278 | " Assumes that width is known. All numbers are on scale -1 to 1.\n",
279 | " The target_std represents the strength of the prior.'''\n",
280 | " self.internal_state = {'fixation': np.array([-1,-1]), \n",
281 | " 'target': np.array([0,0]), \n",
282 | " 'target_std': 0.1,\n",
283 | " 'width': external_state['width'],\n",
284 | " 'action': np.array([-1,-1])} \n",
285 | " return self._get_obs() \n",
286 | "\n",
287 | " def _update_state_with_action(self, action):\n",
288 | " self.internal_state['action'] = action\n",
289 | " \n",
290 | " def _get_response(self):\n",
291 | " ''' Take an action and add noise.'''\n",
292 | " # !!!! should take internal_state as parameter\n",
293 | " move_distance = gazetools.get_distance( self.internal_state['fixation'], \n",
294 | " self.internal_state['action'] )\n",
295 | " \n",
296 | " ocularmotor_noise = np.random.normal(0, self.oculamotor_noise_weight * move_distance, \n",
297 | " self.internal_state['action'].shape)\n",
298 | " # response is action plus noise\n",
299 | " response = self.internal_state['action'] + ocularmotor_noise\n",
300 | " \n",
301 | " # update the ocularmotor state (internal)\n",
302 | " self.internal_state['fixation'] = response\n",
303 | " \n",
304 | " # make an adjustment if response is out of range. \n",
305 | " response = np.clip(response,-1,1)\n",
306 | " return response\n",
307 | " \n",
308 | " def _get_stimulus(self, external_state):\n",
309 | " ''' define a psychologically plausible stimulus function in which acuity \n",
310 | " falls off with eccentricity.''' \n",
311 | " eccentricity = gazetools.get_distance( external_state['target'], external_state['fixation'] )\n",
312 | " stm_std = self.stimulus_noise_weight * eccentricity\n",
313 | " stimulus_noise = np.random.normal(0, stm_std, \n",
314 | " external_state['target'].shape)\n",
315 | " # stimulus is the external target location plus noise\n",
316 | " stm = external_state['target'] + stimulus_noise\n",
317 | " return stm, stm_std\n",
318 | "\n",
319 | " \n",
320 | " def _update_state_with_stimulus(self, stimulus, stimulus_std):\n",
321 | " posterior, posterior_std = self.bayes_update(stimulus, \n",
322 | " stimulus_std, \n",
323 | " self.internal_state['target'],\n",
324 | " self.internal_state['target_std'])\n",
325 | " self.internal_state['target'] = posterior\n",
326 | " self.internal_state['target_std'] = posterior_std\n",
327 | "\n",
328 | " def bayes_update(self, stimulus, stimulus_std, belief, belief_std):\n",
329 | " ''' A Bayes optimal function that integrates multiple stimuluss.\n",
330 | " The belief is the prior.'''\n",
331 | " z1, sigma1 = stimulus, stimulus_std\n",
332 | " z2, sigma2 = belief, belief_std\n",
333 | " w1 = sigma2**2 / (sigma1**2 + sigma2**2)\n",
334 | " w2 = sigma1**2 / (sigma1**2 + sigma2**2)\n",
335 | " posterior = w1*z1 + w2*z2\n",
336 | " posterior_std = np.sqrt( (sigma1**2 * sigma2**2)/(sigma1**2 + sigma2**2) )\n",
337 | " return posterior, posterior_std\n",
338 | " \n",
339 | " def _get_obs(self):\n",
340 | " # the Bayesian posterior has already been calculated so just return it.\n",
341 | " # also return the target_std so that the controller knows the uncertainty \n",
342 | " # of the observation.\n",
343 | " #return self.internal_state['target']\n",
344 | " return np.array([self.internal_state['target'][0],\n",
345 | " self.internal_state['target'][1],\n",
346 | " self.internal_state['target_std']])\n",
347 | " \n",
348 | " def _get_reward(self):\n",
349 | " distance = gazetools.get_distance(self.internal_state['fixation'], \n",
350 | " self.internal_state['target'])\n",
351 | " \n",
352 | " if distance < self.internal_state['width'] / 2:\n",
353 | " reward = 0\n",
354 | " else:\n",
355 | " reward = -distance # a much better model of the psychological reward function is possible.\n",
356 | " \n",
357 | " return reward"
358 | ]
359 | },
360 | {
361 | "cell_type": "markdown",
362 | "id": "boolean-missile",
363 | "metadata": {
364 | "id": "boolean-missile"
365 | },
366 | "source": [
367 | "### Task environment\n",
368 | "\n",
369 | "In order to test the theory we need to define the task environment. \n",
370 | "\n",
371 | "The task environment allows the theory to make predictions for a particular task. The theory makes predictions for many more tasks. For example, adaptation to mixed target widths and distances."
372 | ]
373 | },
374 | {
375 | "cell_type": "code",
376 | "execution_count": null,
377 | "id": "bacterial-archives",
378 | "metadata": {
379 | "executionInfo": {
380 | "elapsed": 472,
381 | "status": "ok",
382 | "timestamp": 1643209742334,
383 | "user": {
384 | "displayName": "Andrew Howes",
385 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64",
386 | "userId": "02694399383679444060"
387 | },
388 | "user_tz": 0
389 | },
390 | "id": "bacterial-archives"
391 | },
392 | "outputs": [],
393 | "source": [
394 | "class GazeTask():\n",
395 | " \n",
396 | " def __init__(self):\n",
397 | " self.target_width = 0.15\n",
398 | " self.target_loc_std = 0.3\n",
399 | "\n",
400 | " def reset_external_env(self):\n",
401 | " ''' The external_state includes the fixation and target location.\n",
402 | " Choose a new target location and reset to the first fixation location.'''\n",
403 | " \n",
404 | " def _get_new_target():\n",
405 | " x_target =np.clip(np.random.normal(0, self.target_loc_std),-1,1)\n",
406 | " y_target =np.clip(np.random.normal(0, self.target_loc_std),-1,1) \n",
407 | " return np.array( [x_target, y_target] )\n",
408 | " \n",
409 | " fx = np.array([-1,-1])\n",
410 | " tg = _get_new_target()\n",
411 | " self.external_state = {'fixation':fx, 'target':tg, 'width':self.target_width }\n",
412 | " \n",
413 | " def external_env(self, action):\n",
414 | " self.external_state['fixation'] = action\n",
415 | " \n",
416 | " # determine when the goal has been achieved.\n",
417 | " distance = gazetools.get_distance(self.external_state['fixation'], \n",
418 | " self.external_state['target'])\n",
419 | " if distance < self.external_state['width']/2 :\n",
420 | " done = True\n",
421 | " else:\n",
422 | " done = False\n",
423 | " \n",
424 | " return self.external_state, done\n",
425 | " "
426 | ]
427 | },
428 | {
429 | "cell_type": "markdown",
430 | "id": "combined-alliance",
431 | "metadata": {
432 | "id": "combined-alliance"
433 | },
434 | "source": [
435 | "### Gym environment\n",
436 | "\n",
437 | "In order to find an optimal policy we use the theory and external environment to define a machine learning problem, here, making use of the framework defined by one specific library called gym.\n",
438 | "\n",
439 | "For further information see: https://gym.openai.com/\n",
440 | "\n",
441 | "gym.Env is a class provided by this library. Note that Env here refers to all of the components of the, including both internal and external environment, with the exception of the controller."
442 | ]
443 | },
444 | {
445 | "cell_type": "code",
446 | "execution_count": null,
447 | "id": "applicable-sleeve",
448 | "metadata": {
449 | "executionInfo": {
450 | "elapsed": 231,
451 | "status": "ok",
452 | "timestamp": 1643209749681,
453 | "user": {
454 | "displayName": "Andrew Howes",
455 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64",
456 | "userId": "02694399383679444060"
457 | },
458 | "user_tz": 0
459 | },
460 | "id": "applicable-sleeve"
461 | },
462 | "outputs": [],
463 | "source": [
464 | "class GazeModel(gym.Env):\n",
465 | " \n",
466 | " def __init__(self):\n",
467 | " \n",
468 | " def default_box(x):\n",
469 | " return spaces.Box(low=-1, high=1, shape=(x, ), dtype=np.float64)\n",
470 | " \n",
471 | " self.GT = GazeTheory()\n",
472 | " self.TX = GazeTask() \n",
473 | " \n",
474 | " # Required by gym. These define the range of each variable.\n",
475 | " # Each action has an x,y coordinate therefore the box size is 2.\n",
476 | " # Each obs has a an x,y and an uncertainty therefore the box size is 3.\n",
477 | " self.action_space = default_box(2)\n",
478 | " self.observation_space = default_box(3)\n",
479 | " \n",
480 | " # max_fixations per episode. Used to curtail exploration early in training.\n",
481 | " self.max_steps = 500\n",
482 | " \n",
483 | " def reset(self):\n",
484 | " self.n_step = 0\n",
485 | " self.TX.reset_external_env()\n",
486 | " self.GT.reset_internal_env(self.TX.external_state)\n",
487 | " obs = self.GT.reset_internal_env( self.TX.external_state )\n",
488 | " return obs, {}\n",
489 | " \n",
490 | " def step(self, action):\n",
491 | " obs, reward, done = self.GT.step( self.TX, action )\n",
492 | " self.n_step+=1\n",
493 | "\n",
494 | " # give up if been looking for too long\n",
495 | " if self.n_step > self.max_steps:\n",
496 | " done = True\n",
497 | " \n",
498 | " info = self.get_info()\n",
499 | " truncated = False\n",
500 | " return obs, reward, done, truncated, info\n",
501 | " \n",
502 | " def get_info(self):\n",
503 | " return {'step': self.n_step,\n",
504 | " 'target_width': self.TX.target_width,\n",
505 | " 'target_x': self.TX.external_state['target'][0],\n",
506 | " 'target_y': self.TX.external_state['target'][1],\n",
507 | " 'fixate_x':self.TX.external_state['fixation'][0],\n",
508 | " 'fixate_y':self.TX.external_state['fixation'][1] }"
509 | ]
510 | },
511 | {
512 | "cell_type": "markdown",
513 | "id": "complex-covering",
514 | "metadata": {
515 | "id": "complex-covering"
516 | },
517 | "source": [
518 | "### Test the model\n",
519 | "\n",
520 | "Step through the untrained model to check for simple bugs. More comprehensive tests needed."
521 | ]
522 | },
523 | {
524 | "cell_type": "code",
525 | "execution_count": null,
526 | "id": "4ed00048",
527 | "metadata": {
528 | "colab": {
529 | "base_uri": "https://localhost:8080/"
530 | },
531 | "executionInfo": {
532 | "elapsed": 534,
533 | "status": "ok",
534 | "timestamp": 1643209756680,
535 | "user": {
536 | "displayName": "Andrew Howes",
537 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64",
538 | "userId": "02694399383679444060"
539 | },
540 | "user_tz": 0
541 | },
542 | "id": "4ed00048",
543 | "outputId": "95f13fd2-30ad-43d9-8cab-1d127f6359d6"
544 | },
545 | "outputs": [],
546 | "source": [
547 | "model = GazeModel()\n",
548 | "\n",
549 | "model.reset()\n",
550 | "\n",
551 | "i=0\n",
552 | "done = False\n",
553 | "while not done:\n",
554 | " # make a step with a randomly sampled action\n",
555 | " obs, reward, done, truncated, info = model.step(model.action_space.sample())\n",
556 | " i+=1\n",
557 | "\n",
558 | "print(i)"
559 | ]
560 | },
561 | {
562 | "cell_type": "markdown",
563 | "id": "cognitive-stevens",
564 | "metadata": {
565 | "id": "cognitive-stevens"
566 | },
567 | "source": [
568 | "### Train the model\n",
569 | "\n",
570 | "We can train the model to generate a controller.\n",
571 | "\n",
572 | "By plotting the learning curve we can see whether the performance improves with training and whether the model approaches an optimum performance. We are interested in approximately optimal performance, so if the training curve is not approaching asymptote then we need to train with more timesteps or revise the model.\n",
573 | "\n",
574 | "We can see that at first the model uses hundreds of fixations to find the target, this is because it has not yet learned to move the gaze in a way that is informed by the observation. As it learns to do this, it takes fewer steps to gaze at the target and its performance improves.\n",
575 | "\n",
576 | "If our problem definition is correct then the model will get more 'human-like' the more that it is trained. In other words, training makes it a better model of interaction.\n",
577 | "\n",
578 | "If we assume that people are computationally rational then the optimal solution to a cognitive problem predicts human behavior."
579 | ]
580 | },
581 | {
582 | "cell_type": "code",
583 | "execution_count": null,
584 | "id": "7d0910ac",
585 | "metadata": {
586 | "colab": {
587 | "base_uri": "https://localhost:8080/",
588 | "height": 299
589 | },
590 | "executionInfo": {
591 | "elapsed": 66705,
592 | "status": "ok",
593 | "timestamp": 1643209827031,
594 | "user": {
595 | "displayName": "Andrew Howes",
596 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64",
597 | "userId": "02694399383679444060"
598 | },
599 | "user_tz": 0
600 | },
601 | "id": "7d0910ac",
602 | "outputId": "d927873a-e745-4931-d6b8-fb6f3a668245"
603 | },
604 | "outputs": [],
605 | "source": [
606 | "timesteps = 200000\n",
607 | "#timesteps = 50000\n",
608 | "\n",
609 | "controller = gazetools.train(model, timesteps)\n",
610 | "gazetools.plot_learning_curve()"
611 | ]
612 | },
613 | {
614 | "cell_type": "markdown",
615 | "id": "japanese-georgia",
616 | "metadata": {
617 | "id": "japanese-georgia"
618 | },
619 | "source": [
620 | "### Run the model for N trials\n",
621 | "Run the trained model and save a trace of each episode to csv file. "
622 | ]
623 | },
624 | {
625 | "cell_type": "code",
626 | "execution_count": null,
627 | "id": "after-evans",
628 | "metadata": {
629 | "colab": {
630 | "base_uri": "https://localhost:8080/",
631 | "height": 35
632 | },
633 | "executionInfo": {
634 | "elapsed": 162308,
635 | "status": "ok",
636 | "timestamp": 1643210003598,
637 | "user": {
638 | "displayName": "Andrew Howes",
639 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GguyjUymXH2ndqd0p8hhQuI6UyIwWtm4lsMYWs0Ug=s64",
640 | "userId": "02694399383679444060"
641 | },
642 | "user_tz": 0
643 | },
644 | "id": "after-evans",
645 | "outputId": "838f8f2b-9414-454a-8700-acfd39062bb8"
646 | },
647 | "outputs": [],
648 | "source": [
649 | "data = gazetools.run_model( model, controller, 100, 'behaviour_trace.csv' )\n",
650 | "\n",
651 | "gazetools.animate_multiple_episodes(data, n=30)\n"
652 | ]
653 | },
654 | {
655 | "cell_type": "markdown",
656 | "id": "b6c4c96f",
657 | "metadata": {},
658 | "source": []
659 | },
660 | {
661 | "cell_type": "markdown",
662 | "id": "weekly-headquarters",
663 | "metadata": {
664 | "id": "weekly-headquarters"
665 | },
666 | "source": [
667 | "### References\n",
668 | "Chen, X., Acharya, A., Oulasvirta, A., & Howes, A. (2021, May). An adaptive model of gaze-based selection. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-11)."
669 | ]
670 | }
671 | ],
672 | "metadata": {
673 | "colab": {
674 | "collapsed_sections": [],
675 | "name": "gaze-based-interaction.ipynb",
676 | "provenance": []
677 | },
678 | "kernelspec": {
679 | "display_name": "Python 3 (ipykernel)",
680 | "language": "python",
681 | "name": "python3"
682 | },
683 | "language_info": {
684 | "codemirror_mode": {
685 | "name": "ipython",
686 | "version": 3
687 | },
688 | "file_extension": ".py",
689 | "mimetype": "text/x-python",
690 | "name": "python",
691 | "nbconvert_exporter": "python",
692 | "pygments_lexer": "ipython3",
693 | "version": "3.9.5"
694 | }
695 | },
696 | "nbformat": 4,
697 | "nbformat_minor": 5
698 | }
699 |
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/gaze_based_interaction.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "attachments": {},
5 | "cell_type": "markdown",
6 | "id": "3e0c5f5d",
7 | "metadata": {},
8 | "source": [
9 | "[](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/02-DeepRL/Gaze_Based_Interaction/gaze_based_interaction.ipynb)\n"
10 | ]
11 | },
12 | {
13 | "cell_type": "markdown",
14 | "id": "fitted-component",
15 | "metadata": {
16 | "id": "fitted-component"
17 | },
18 | "source": [
19 | "# A cognitive model of gaze-based interaction\n",
20 | "\n",
21 | "Some cognitive models describe behaviour and others predict it. Models that predict behaviour must be capable of generating output without that output being part of the input. In this notebook we demonstrate this property for a model of eye movements. The model is a reinforcement learner that is trained by performing hundreds of thousands of simulated eye movements in search of a target of varying size and distance. The model predicts how many eye movements people will tend to make to find a target of a given size and distance. It predicts inhibition of return. It **predicts** Fitts's Law like behaviour and it predicts that the first eye movement will usually undershoot, rather than overshoot the target.\n",
22 | "\n",
23 | "
\n",
24 | "\n",
25 | "(source: Meyer, D. E., Abrams, R. A., Kornblum, S., Wright, C. E., & Keith Smith, J. E. (1988). Optimality in human motor performance: ideal control of rapid aimed movements. Psychological review, 95(3), 340.)\n",
26 | "\n",
27 | "Note that what might seem the obvious strategy -- aim for where you believe the target is -- is not necessarily the optimal strategy.\n",
28 | "\n",
29 | "### The task: Gaze-based interaction\n",
30 | "\n",
31 | "Gaze-based interaction is a mode of interaction in which users, including users with a range of movement disabilities, are able to indicate which display item they wish to select by fixating their eyes on it. Confirmation of selection is then made with one of a number of methods, including with a key press, or by holding the fixation for an extended duration of time. The model performs this task for targets with randomly selected location and size.\n",
32 | "\n",
33 | "
\n",
34 | "\n",
35 | "In the figure, the red lines represent saccades (eye movements). Multiple eye movements are needed to reach the target (the black circle).\n",
36 | "\n",
37 | "### Model architecture\n",
38 | "\n",
39 | "The model has a simple architecture that you have previously seen in the introduction. The figure is reproduced here:\n",
40 | "\n",
41 | "
\n",
42 | "\n",
43 | "- The **control** module makes decisions about where to move the eyes with the oculomotor system. Decisions are conditioned on the current belief about the location of the target.\n",
44 | "- The **motor** module implements decisions but it is bounded by Gaussian noise, which models noise in the human motor system.\n",
45 | "- The **environment** models the physics of the world and the task (the location of the target). Given a response from the motor system, a saccade is made to the aim point of the eyes, and a fixation is initiated.\n",
46 | "- The **perception** module simulates the human capacity to localize a target with foveated vision. The accuracy of the location estimate generated by perception is negatively affected by the eccentricity of the target from the current fixation location.\n",
47 | "- The **Memory** module stores a representation of the current state. Over the course of an episode a sequence of location estimates will be made. Humans are known to integrate these estimates into a single integrated representation of the location. People are known to do this optimally using a process that can be modelled with Bayesian state estimation. The state estimation constitutes a belief about the location of the target.\n",
48 | "- The **Utility** module calculates a reward signal given the current belief about the enviornment. The reward signal is used to train the controller.\n",
49 | "\n",
50 | "### Prerequisites\n",
51 | "\n",
52 | "Before proceeding with this notebook you should firrst review the notebooks on foveated vision and on Bayesian integration. These explain how the perception and memory modules work.\n",
53 | "\n"
54 | ]
55 | },
56 | {
57 | "cell_type": "markdown",
58 | "id": "a427be22",
59 | "metadata": {
60 | "id": "a427be22"
61 | },
62 | "source": [
63 | "### Machine learning\n",
64 | "\n",
65 | "In order to learn how to perform the task, the model uses implementations of reinforcement learning algorithms in PyTorch known as stable-baselines3."
66 | ]
67 | },
68 | {
69 | "cell_type": "code",
70 | "execution_count": null,
71 | "id": "vL8yMY6q_Rd-",
72 | "metadata": {
73 | "colab": {
74 | "base_uri": "https://localhost:8080/"
75 | },
76 | "id": "vL8yMY6q_Rd-",
77 | "outputId": "0f80da38-b03b-4b02-f041-58f91d3d7c18"
78 | },
79 | "outputs": [],
80 | "source": [
81 | "# Install stable_baselines3 and the gymnasium environment\n",
82 | "# This is a well known machine learning library that provides a suite of reinforcement learning methods.\n",
83 | "# Only needs to be run once\n",
84 | "\n",
85 | "!pip install --pre -U stable_baselines3\n",
86 | "\n",
87 | "import gymnasium as gym\n",
88 | "from gymnasium import spaces"
89 | ]
90 | },
91 | {
92 | "cell_type": "code",
93 | "execution_count": null,
94 | "id": "P44tljjb8hBC",
95 | "metadata": {
96 | "id": "P44tljjb8hBC"
97 | },
98 | "outputs": [],
99 | "source": [
100 | "# Load local modules\n",
101 | "# gazetools is a module that contains support functions for modeling gaze-based interaction.\n",
102 | "# the code below makes use of them but we do not need to understand how they work in this tutorial.\n",
103 | "\n",
104 | "!wget https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/main/02-DeepRL/Gaze_Based_Interaction/gazetools.py\n",
105 | "\n",
106 | "import gazetools"
107 | ]
108 | },
109 | {
110 | "cell_type": "code",
111 | "execution_count": null,
112 | "id": "obvious-point",
113 | "metadata": {
114 | "id": "obvious-point"
115 | },
116 | "outputs": [],
117 | "source": [
118 | "# load required standard modules and configure matplotlib\n",
119 | "\n",
120 | "import numpy as np\n",
121 | "import math\n",
122 | "import matplotlib.pyplot as plt\n",
123 | "import sys\n",
124 | "\n",
125 | "import matplotlib as mpl\n",
126 | "%matplotlib inline\n",
127 | "mpl.style.use('ggplot')"
128 | ]
129 | },
130 | {
131 | "cell_type": "markdown",
132 | "id": "injured-leadership",
133 | "metadata": {
134 | "id": "injured-leadership"
135 | },
136 | "source": [
137 | "### Implementation of the Cognitive Architecture as a Python Class\n",
138 | "\n",
139 | "The first step to formalise the model architecture presented in the above figure. We do this by specifying a class of cognitive theories and will later define an instance of this class.\n",
140 | "\n",
141 | "The class has only a single method, which defines a cycle through the processes defined in the figure."
142 | ]
143 | },
144 | {
145 | "cell_type": "code",
146 | "execution_count": null,
147 | "id": "monetary-ivory",
148 | "metadata": {
149 | "id": "monetary-ivory"
150 | },
151 | "outputs": [],
152 | "source": [
153 | "class CognitivePOMDP():\n",
154 | "\n",
155 | " def __init__(self):\n",
156 | " self.internal_state = {}\n",
157 | "\n",
158 | " def step(self, ext, action):\n",
159 | " ''' Define the cognitive architecture.'''\n",
160 | " self._update_state_with_action(action)\n",
161 | " response = self._get_response()\n",
162 | " external_state, done = ext.external_env(response)\n",
163 | " stimulus, stimulus_std = self._get_stimulus(ext.external_state)\n",
164 | " self._update_state_with_stimulus(stimulus, stimulus_std)\n",
165 | " obs = self._get_obs()\n",
166 | " reward = self._get_reward()\n",
167 | " return obs, reward, done"
168 | ]
169 | },
170 | {
171 | "cell_type": "markdown",
172 | "id": "brazilian-timeline",
173 | "metadata": {
174 | "id": "brazilian-timeline"
175 | },
176 | "source": [
177 | "### A theory of gaze-based interaction\n",
178 | "\n",
179 | "Each of the entities in the cognitive architecture must be defined so as to make explicit our theory of gaze-based interaction. The theory makes the following assumptions:\n",
180 | "\n",
181 | "- The perception of the target location is corrupted by Gaussian noise in human vision.\n",
182 | "- The standard deviation of noise increases linearly with eccentricity from the fovea.\n",
183 | "- Sequences of stimuli are noisily perceived and optimally integrated.\n",
184 | "- Intended eye movements (oculomotor actions) are corrupted by signal dependent Gaussian noise to generate responses.\n",
185 | "\n",
186 | "These assumptions are further described in Chen et al. (2021)."
187 | ]
188 | },
189 | {
190 | "cell_type": "code",
191 | "execution_count": null,
192 | "id": "accepted-reunion",
193 | "metadata": {
194 | "id": "accepted-reunion"
195 | },
196 | "outputs": [],
197 | "source": [
198 | "class GazeTheory(CognitivePOMDP):\n",
199 | "\n",
200 | " def __init__(self):\n",
201 | " ''' Initialise the theoretically motivated parameters.'''\n",
202 | " # weight eye movement noise with distance of saccade\n",
203 | " self.oculamotor_noise_weight = 0.01\n",
204 | " # weight noise with eccentricity\n",
205 | " self.stimulus_noise_weight = 0.09\n",
206 | " # step_cost for the reward function\n",
207 | " self.step_cost = -1\n",
208 | " # super.__init__()\n",
209 | "\n",
210 | " def reset_internal_env(self, external_state):\n",
211 | " ''' The internal state includes the fixation location, the latest estimate of\n",
212 | " the target location and the target uncertainty. Assumes that there is no\n",
213 | " uncertainty in the fixation location.\n",
214 | " Assumes that width is known. All numbers are on scale -1 to 1.\n",
215 | " The target_std represents the strength of the prior.'''\n",
216 | " self.internal_state = {'fixation': np.array([-1,-1]),\n",
217 | " 'target': np.array([0,0]),\n",
218 | " 'target_std': 0.1,\n",
219 | " 'width': external_state['width'],\n",
220 | " 'action': np.array([-1,-1])}\n",
221 | " return self._get_obs()\n",
222 | "\n",
223 | " def _update_state_with_action(self, action):\n",
224 | " self.internal_state['action'] = action\n",
225 | "\n",
226 | " def _get_response(self):\n",
227 | " ''' Take an action and add noise.'''\n",
228 | " # !!!! should take internal_state as parameter\n",
229 | " move_distance = gazetools.get_distance( self.internal_state['fixation'],\n",
230 | " self.internal_state['action'] )\n",
231 | "\n",
232 | " ocularmotor_noise = np.random.normal(0, self.oculamotor_noise_weight * move_distance,\n",
233 | " self.internal_state['action'].shape)\n",
234 | " # response is action plus noise\n",
235 | " response = self.internal_state['action'] + ocularmotor_noise\n",
236 | "\n",
237 | " # update the ocularmotor state (internal)\n",
238 | " self.internal_state['fixation'] = response\n",
239 | "\n",
240 | " # make an adjustment if response is out of range.\n",
241 | " response = np.clip(response,-1,1)\n",
242 | " return response\n",
243 | "\n",
244 | " def _get_stimulus(self, external_state):\n",
245 | " ''' define a psychologically plausible stimulus function in which acuity\n",
246 | " falls off with eccentricity.'''\n",
247 | " eccentricity = gazetools.get_distance( external_state['target'], external_state['fixation'] )\n",
248 | " stm_std = self.stimulus_noise_weight * eccentricity\n",
249 | " stimulus_noise = np.random.normal(0, stm_std,\n",
250 | " external_state['target'].shape)\n",
251 | " # stimulus is the external target location plus noise\n",
252 | " stm = external_state['target'] + stimulus_noise\n",
253 | " return stm, stm_std\n",
254 | "\n",
255 | "\n",
256 | " def _update_state_with_stimulus(self, stimulus, stimulus_std):\n",
257 | " posterior, posterior_std = self.bayes_update(stimulus,\n",
258 | " stimulus_std,\n",
259 | " self.internal_state['target'],\n",
260 | " self.internal_state['target_std'])\n",
261 | " self.internal_state['target'] = posterior\n",
262 | " self.internal_state['target_std'] = posterior_std\n",
263 | "\n",
264 | " def bayes_update(self, stimulus, stimulus_std, belief, belief_std):\n",
265 | " ''' A Bayes optimal function that integrates multiple stimului.\n",
266 | " The belief is the prior.'''\n",
267 | " z1, sigma1 = stimulus, stimulus_std\n",
268 | " z2, sigma2 = belief, belief_std\n",
269 | " w1 = sigma2**2 / (sigma1**2 + sigma2**2)\n",
270 | " w2 = sigma1**2 / (sigma1**2 + sigma2**2)\n",
271 | " posterior = w1*z1 + w2*z2\n",
272 | " posterior_std = np.sqrt( (sigma1**2 * sigma2**2)/(sigma1**2 + sigma2**2) )\n",
273 | " return posterior, posterior_std\n",
274 | "\n",
275 | " def _get_obs(self):\n",
276 | " # the Bayesian posterior has already been calculated so just return it.\n",
277 | " # also return the target_std so that the controller knows the uncertainty\n",
278 | " # of the observation.\n",
279 | " #return self.internal_state['target']\n",
280 | " return np.array([self.internal_state['target'][0],\n",
281 | " self.internal_state['target'][1],\n",
282 | " self.internal_state['target_std']])\n",
283 | "\n",
284 | " def _get_reward(self):\n",
285 | " distance = gazetools.get_distance(self.internal_state['fixation'],\n",
286 | " self.internal_state['target'])\n",
287 | "\n",
288 | " if distance < self.internal_state['width'] / 2:\n",
289 | " reward = 0\n",
290 | " else:\n",
291 | " reward = -distance # a much better model of the psychological reward function is possible.\n",
292 | "\n",
293 | " return reward"
294 | ]
295 | },
296 | {
297 | "cell_type": "markdown",
298 | "id": "boolean-missile",
299 | "metadata": {
300 | "id": "boolean-missile"
301 | },
302 | "source": [
303 | "### Task environment\n",
304 | "\n",
305 | "In order to test the theory we need to define the task environment.\n",
306 | "\n",
307 | "The task environment allows the theory to make predictions for a particular task. The theory makes predictions for many more tasks. For example, adaptation to mixed target widths and distances."
308 | ]
309 | },
310 | {
311 | "cell_type": "code",
312 | "execution_count": null,
313 | "id": "bacterial-archives",
314 | "metadata": {
315 | "id": "bacterial-archives"
316 | },
317 | "outputs": [],
318 | "source": [
319 | "class GazeTask():\n",
320 | "\n",
321 | " def __init__(self):\n",
322 | " self.target_width = 0.15\n",
323 | " self.target_loc_std = 0.3\n",
324 | "\n",
325 | " def reset_external_env(self):\n",
326 | " ''' The external_state includes the fixation and target location.\n",
327 | " Choose a new target location and reset fixation to the first fixation location.'''\n",
328 | "\n",
329 | " def _get_new_target():\n",
330 | " x_target =np.clip(np.random.normal(0, self.target_loc_std),-1,1)\n",
331 | " y_target =np.clip(np.random.normal(0, self.target_loc_std),-1,1)\n",
332 | " return np.array( [x_target, y_target] )\n",
333 | "\n",
334 | " fx = np.array([-1,-1])\n",
335 | " tg = _get_new_target()\n",
336 | " self.external_state = {'fixation':fx, 'target':tg, 'width':self.target_width }\n",
337 | "\n",
338 | " def external_env(self, action):\n",
339 | " self.external_state['fixation'] = action\n",
340 | "\n",
341 | " # determine when the goal has been achieved.\n",
342 | " distance = gazetools.get_distance(self.external_state['fixation'],\n",
343 | " self.external_state['target'])\n",
344 | " if distance < self.external_state['width']/2 :\n",
345 | " done = True\n",
346 | " else:\n",
347 | " done = False\n",
348 | "\n",
349 | " return self.external_state, done\n"
350 | ]
351 | },
352 | {
353 | "cell_type": "markdown",
354 | "id": "combined-alliance",
355 | "metadata": {
356 | "id": "combined-alliance"
357 | },
358 | "source": [
359 | "### Gym (Gymnasium)\n",
360 | "\n",
361 | "In order to find an optimal policy we use the theory and external environment to define a machine learning problem, here, making use of the framework defined by one specific library called gym.\n",
362 | "\n",
363 | "For further information see: https://gymnasium.farama.org/\n",
364 | "\n",
365 | "gym.Env is a class provided by this library. Note that all of the modules of the cognitive architecture are part of gym.env except for the controller."
366 | ]
367 | },
368 | {
369 | "cell_type": "code",
370 | "execution_count": null,
371 | "id": "applicable-sleeve",
372 | "metadata": {
373 | "colab": {
374 | "base_uri": "https://localhost:8080/"
375 | },
376 | "id": "applicable-sleeve",
377 | "outputId": "f3e0499f-5ac2-4d2a-dcce-41131c66801b"
378 | },
379 | "outputs": [],
380 | "source": [
381 | "class GazeModel(gym.Env):\n",
382 | "\n",
383 | " def __init__(self):\n",
384 | "\n",
385 | " def default_box(x):\n",
386 | " return spaces.Box(low=-1, high=1, shape=(x, ), dtype=np.float64)\n",
387 | "\n",
388 | " self.GT = GazeTheory()\n",
389 | " self.TX = GazeTask()\n",
390 | "\n",
391 | " # Required by gym. These define the range of each variable.\n",
392 | " # Each action has an x,y coordinate therefore the box size is 2.\n",
393 | " # Each obs has a an x,y and an uncertainty therefore the box size is 3.\n",
394 | " self.action_space = default_box(2)\n",
395 | " self.observation_space = default_box(3)\n",
396 | "\n",
397 | " # max_fixations per episode. Used to curtail exploration early in training.\n",
398 | " self.max_steps = 500\n",
399 | "\n",
400 | " def reset(self, seed=None, options=None):\n",
401 | " ''' reset the model.'''\n",
402 | " self.n_step = 0\n",
403 | " self.TX.reset_external_env()\n",
404 | " self.GT.reset_internal_env(self.TX.external_state)\n",
405 | " obs = self.GT.reset_internal_env( self.TX.external_state )\n",
406 | " return obs, {}\n",
407 | "\n",
408 | " def step(self, action):\n",
409 | " ''' Step through one cycle of the model.'''\n",
410 | " obs, reward, done = self.GT.step( self.TX, action )\n",
411 | " self.n_step+=1\n",
412 | "\n",
413 | " # give up if been looking for too long\n",
414 | " if self.n_step > self.max_steps:\n",
415 | " done = True\n",
416 | "\n",
417 | " info = self.get_info()\n",
418 | " truncated = False\n",
419 | " return obs, reward, done, truncated, info\n",
420 | "\n",
421 | " def get_info(self):\n",
422 | " return {'step': self.n_step,\n",
423 | " 'target_width': self.TX.target_width,\n",
424 | " 'target_x': self.TX.external_state['target'][0],\n",
425 | " 'target_y': self.TX.external_state['target'][1],\n",
426 | " 'fixate_x':self.TX.external_state['fixation'][0],\n",
427 | " 'fixate_y':self.TX.external_state['fixation'][1] }"
428 | ]
429 | },
430 | {
431 | "cell_type": "markdown",
432 | "id": "complex-covering",
433 | "metadata": {
434 | "id": "complex-covering"
435 | },
436 | "source": [
437 | "### Test the model\n",
438 | "\n",
439 | "Step through the untrained model to check for simple bugs. More comprehensive tests needed."
440 | ]
441 | },
442 | {
443 | "cell_type": "code",
444 | "execution_count": null,
445 | "id": "4ed00048",
446 | "metadata": {
447 | "colab": {
448 | "base_uri": "https://localhost:8080/"
449 | },
450 | "id": "4ed00048",
451 | "outputId": "e4395a7a-aa41-4e5c-9ab1-3e6d77a6876c"
452 | },
453 | "outputs": [],
454 | "source": [
455 | "model = GazeModel()\n",
456 | "\n",
457 | "model.reset()\n",
458 | "\n",
459 | "i=0\n",
460 | "done = False\n",
461 | "while not done:\n",
462 | " # make a step with a randomly sampled action\n",
463 | " obs, reward, done, truncated, info = model.step(model.action_space.sample())\n",
464 | " i+=1\n",
465 | "\n",
466 | "print(i)"
467 | ]
468 | },
469 | {
470 | "cell_type": "markdown",
471 | "id": "cognitive-stevens",
472 | "metadata": {
473 | "id": "cognitive-stevens"
474 | },
475 | "source": [
476 | "### Train the model\n",
477 | "\n",
478 | "We can train the model to generate a policy for the controller.\n",
479 | "\n",
480 | "By plotting the learning curve we can see whether the performance improves with training and whether the model approaches an optimum performance. We are interested in approximately optimal performance, so if the training curve is not approaching asymptote then we need to train with more timesteps or revise the model.\n",
481 | "\n",
482 | "We can see that at first the model uses hundreds of fixations to find the target, this is because it has not yet learned to move the gaze in a way that is informed by the observation. As it learns to do this, it takes fewer steps to gaze at the target and its performance improves.\n",
483 | "\n",
484 | "If our problem definition is correct then the model will get more 'human-like' the more that it is trained. In other words, training makes it a better model of interaction.\n",
485 | "\n",
486 | "If we assume that people are computationally rational then the optimal solution to a cognitive problem predicts human behavior."
487 | ]
488 | },
489 | {
490 | "cell_type": "code",
491 | "execution_count": null,
492 | "id": "7d0910ac",
493 | "metadata": {
494 | "colab": {
495 | "base_uri": "https://localhost:8080/",
496 | "height": 470
497 | },
498 | "id": "7d0910ac",
499 | "outputId": "e01f85a6-70a8-4fb5-9bf2-a36469669684"
500 | },
501 | "outputs": [],
502 | "source": [
503 | "timesteps = 100000\n",
504 | "\n",
505 | "controller = gazetools.train(model, timesteps)"
506 | ]
507 | },
508 | {
509 | "cell_type": "code",
510 | "execution_count": null,
511 | "id": "dc728251-2e31-4f0f-ae3d-9f92028a48db",
512 | "metadata": {},
513 | "outputs": [],
514 | "source": [
515 | "gazetools.plot_learning_curve()"
516 | ]
517 | },
518 | {
519 | "cell_type": "markdown",
520 | "id": "ccba2f2c",
521 | "metadata": {
522 | "id": "ccba2f2c"
523 | },
524 | "source": [
525 | "### Increase timesteps\n",
526 | "\n",
527 | "100,000 timesteps is not enough to train this model. Try doubling the number of timesteps and train again."
528 | ]
529 | },
530 | {
531 | "cell_type": "markdown",
532 | "id": "japanese-georgia",
533 | "metadata": {
534 | "id": "japanese-georgia"
535 | },
536 | "source": [
537 | "### Run the model for N trials\n",
538 | "Run the trained model, save a trace of each episode to csv file, and animate the results."
539 | ]
540 | },
541 | {
542 | "cell_type": "code",
543 | "execution_count": null,
544 | "id": "after-evans",
545 | "metadata": {
546 | "colab": {
547 | "base_uri": "https://localhost:8080/",
548 | "height": 955
549 | },
550 | "id": "after-evans",
551 | "outputId": "3c6adc93-94cd-4ec9-915e-0274b306e8b3"
552 | },
553 | "outputs": [],
554 | "source": [
555 | "data = gazetools.run_model( model, controller, 100, 'behaviour_trace.csv' )\n",
556 | "\n",
557 | "gazetools.animate_multiple_episodes(data, n=30)"
558 | ]
559 | },
560 | {
561 | "cell_type": "markdown",
562 | "id": "b6c4c96f",
563 | "metadata": {
564 | "id": "b6c4c96f"
565 | },
566 | "source": [
567 | "### Exercises\n",
568 | "\n",
569 | "- Rerun the model with different parameter settings. Start by trying a different target size. What is the impact on the behaviour?\n",
570 | "- Discuss in groups how you would use the methods presented in the notebook to build a cognitive model of another task."
571 | ]
572 | },
573 | {
574 | "cell_type": "markdown",
575 | "id": "53e77c9c",
576 | "metadata": {
577 | "id": "53e77c9c"
578 | },
579 | "source": [
580 | "### Discussion\n",
581 | "\n",
582 | "- The cognitive model that we have described above accurately predicts human gaze-based interaction performance (Chen et al., 2021).\n",
583 | "- It is an example of a **computationally rational** cognitive model. This is because the behaviour is predicted from an approximately optimal policy given hypothesised bounds on cognition.\n",
584 | "- It should be possible to find an approximately optimal policy using any reinforcement learning algorithm. The only difference that the algorithm will make is to the efficiency with which solution is found.\n",
585 | "- The separation of cognitive theory and reinforcement learning algorithm is achieved through the statement of the architecture as what is known as a belief-state Markov Decision Process (a **belief-state MDP**), which is a type of Partially Observable Markov Decision Process (POMDP).\n",
586 | "- Using reinforcement learning (RL) to model cognition with the approximately optimal policy contrast does not model the human learning process. For work that does use RL to model human learning see Daw and Dayan (2008).\n",
587 | "- A number of CHI papers have made use of this architecture. See Oulasvirta et al., 2022 for a review.\n",
588 | "- An important issue concerns how model parameters are fitted to data. See Keuralinen et al., 2023."
589 | ]
590 | },
591 | {
592 | "cell_type": "markdown",
593 | "id": "weekly-headquarters",
594 | "metadata": {
595 | "id": "weekly-headquarters"
596 | },
597 | "source": [
598 | "### References\n",
599 | "Chen, X., Acharya, A., Oulasvirta, A., & Howes, A. (2021, May). An adaptive model of gaze-based selection. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-11).\n",
600 | "\n",
601 | "Chen, H., Chang, H. J., & Howes, A. (2021, May). Apparently Irrational Choice as Optimal Sequential Decision Making. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 1, pp. 792-800).\n",
602 | "\n",
603 | "Dayan, P., & Daw, N. D. (2008). Decision theory, reinforcement learning, and the brain. Cognitive, Affective, & Behavioral Neuroscience, 8(4), 429-453.\n",
604 | "\n",
605 | "Oulasvirta, A., Jokinen, J. P., & Howes, A. (2022, April). Computational rationality as a theory of interaction. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (pp. 1-14).\n",
606 | "\n",
607 | "Keurulainen, A., Westerlund, I. R., Keurulainen, O., & Howes, A. (2023, April). Amortised Experimental Design and Parameter Estimation for User Models of Pointing. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (pp. 1-17)."
608 | ]
609 | }
610 | ],
611 | "metadata": {
612 | "colab": {
613 | "provenance": []
614 | },
615 | "kernelspec": {
616 | "display_name": "Python 3 (ipykernel)",
617 | "language": "python",
618 | "name": "python3"
619 | },
620 | "language_info": {
621 | "codemirror_mode": {
622 | "name": "ipython",
623 | "version": 3
624 | },
625 | "file_extension": ".py",
626 | "mimetype": "text/x-python",
627 | "name": "python",
628 | "nbconvert_exporter": "python",
629 | "pygments_lexer": "ipython3",
630 | "version": "3.12.3"
631 | }
632 | },
633 | "nbformat": 4,
634 | "nbformat_minor": 5
635 | }
636 |
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/gazetools.py:
--------------------------------------------------------------------------------
1 |
2 | import os
3 | import csv
4 |
5 | import gymnasium as gym
6 | import numpy as np
7 | import matplotlib.pyplot as plt
8 | import pandas as pd
9 | import time
10 | from IPython import display
11 |
12 | from stable_baselines3 import PPO
13 | from stable_baselines3.common import results_plotter
14 | from stable_baselines3.common.monitor import Monitor
15 | from stable_baselines3.common.noise import NormalActionNoise
16 | from stable_baselines3.common.callbacks import BaseCallback
17 | from stable_baselines3.common.callbacks import CheckpointCallback
18 | from stable_baselines3.common.results_plotter import load_results, ts2xy, plot_results
19 |
20 |
21 | output_dir = 'output/'
22 | policy_file = 'policy'
23 |
24 |
25 | def train(env, timesteps):
26 | '''
27 | '''
28 | env = Monitor(env, output_dir)
29 | controller = PPO('MlpPolicy', env, verbose=0, clip_range=0.15)
30 | controller.learn(total_timesteps=int(timesteps))
31 | controller.save(f'{output_dir}{policy_file}')
32 | print('Done training.')
33 | return controller
34 |
35 | '''
36 | Plot learning curve
37 | '''
38 |
39 | def plot_learning_curve(title='Learning Curve'):
40 | """
41 | :param output_folder: (str) the save location of the results to plot
42 | :param title: (str) the title of the task to plot
43 | """
44 | x, y = ts2xy(load_results(output_dir), 'timesteps')
45 | y = moving_average(y, window=100)
46 | # Truncate x
47 | #x = x[len(x) - len(y):]
48 | fig = plt.figure(title)
49 | plt.plot(y)
50 | plt.xlabel('Number of Timesteps')
51 | plt.ylabel('Rewards')
52 |
53 | def moving_average(values, window):
54 | """
55 | Smooth values by doing a moving average
56 | :param values: (numpy array)
57 | :param window: (int)
58 | :return: (numpy array)
59 | """
60 | weights = np.repeat(1.0, window) / window
61 | return np.convolve(values, weights, 'valid')
62 |
63 | '''
64 | Get the Euclidean distance between points p and q
65 | '''
66 |
67 | def get_distance(p,q):
68 | return np.sqrt(np.sum((p-q)**2))
69 |
70 | '''
71 | '''
72 |
73 | def load_controller():
74 | controller = PPO.load(f'{output_dir}{policy_file}')
75 | return controller
76 |
77 | '''
78 | '''
79 |
80 | def run_model(env, controller, n_episodes, filename):
81 | '''
82 | run the model for n_episodes and save its behaviour in a csv file.
83 | Note that 'env' is a term used by Gym to describe everything but the controller.
84 | '''
85 | max_episodes = 900000
86 | if n_episodes > max_episodes:
87 | print(f'We ask that you limit training to a max of {max_episodes} on the School of Computer Science AWS account.')
88 | print(f'If you want to run more training episodes then please do so on a local computer.')
89 | return
90 |
91 | result = []
92 | # repeat for n episodes
93 | eps = 0
94 | while eps < n_episodes:
95 | done = False
96 | step = 0
97 | obs, _ = env.reset()
98 | # record the initial state
99 | info = env.get_info()
100 | info['episode'] = eps
101 | result.append(info)
102 |
103 | # repeat until the gaze is on the target.
104 | while not done:
105 | step+=1
106 | # get the next prediction action from the controller
107 | action, _ = controller.predict(obs,deterministic = True)
108 | obs, reward, done, _, info = env.step(action)
109 | info['episode'] = eps
110 | result.append(info)
111 | if done:
112 | eps+=1
113 | path = f'{output_dir}{filename}'
114 | df = pd.DataFrame(result)
115 | df.to_csv(path,index=False)
116 | return df
117 |
118 | '''
119 | '''
120 |
121 | def plot_gaze(gaze_x,gaze_y):
122 | plt.plot(gaze_x,gaze_y,'r+',markersize=20,linewidth=2)
123 |
124 | '''
125 | '''
126 |
127 | def update_display(gap_time):
128 | # update the display with a time step
129 | display.display(plt.gcf())
130 | display.clear_output(wait=True)
131 | time.sleep(gap_time)
132 |
133 | '''
134 | '''
135 |
136 | def set_canvas():
137 | time.sleep(2)
138 | #set the canvas
139 | plt.close()
140 | fig, ax = plt.subplots(figsize=(7,7)) # note we must use plt.subplots, not plt.subplot
141 | plt.xlim(-1,1)
142 | plt.ylim(-1,1)
143 | plt.gca().set_aspect('equal', adjustable='box')
144 | update_display(gap_time=1)
145 | return(ax)
146 |
147 | '''
148 | '''
149 |
150 | def animate_episode(ax, df_eps):
151 | gaze_x,gaze_y=df_eps.loc[0]['fixate_x'],df_eps.loc[0]['fixate_y']
152 | for t in range(1,len(df_eps)):
153 | # each time step t
154 |
155 | if t==2:
156 | # target
157 | target_x,target_y=df_eps.loc[t]['target_x'],df_eps.loc[t]['target_y']
158 | target_width=df_eps.loc[t]['target_width']
159 | circle1 = plt.Circle((target_x,target_y), target_width/2, color='k')
160 | ax.add_patch(circle1)
161 | update_display(gap_time=0.5)
162 |
163 | new_gaze_x,new_gaze_y=df_eps.loc[t]['fixate_x'],df_eps.loc[t]['fixate_y']
164 | plt.arrow(gaze_x,gaze_y, new_gaze_x-gaze_x,new_gaze_y-gaze_y,head_width=0.05,
165 | length_includes_head=True,linestyle='-',color='r')
166 | plot_gaze(new_gaze_x,new_gaze_y)
167 | update_display(gap_time=0.5)
168 |
169 | # new gaze becomes the current gaze
170 | gaze_x,gaze_y=new_gaze_x,new_gaze_y
171 |
172 | '''
173 | '''
174 |
175 | def animate_multiple_episodes(data, n):
176 | for eps in range(n):
177 | # each episode
178 | ax = set_canvas()
179 |
180 | # behaviour data for each episode
181 | df_eps=data.loc[data['episode']==eps]
182 | df_eps.reset_index(drop=True, inplace=True)
183 |
184 | # truncate the length of the episode if it is too long.
185 | if len(df_eps) > 5:
186 | df_eps = df_eps[0:5]
187 |
188 | animate_episode(ax, df_eps)
189 |
190 |
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/image/cog_arch.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/cog_arch.png
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/image/cognitive_POMDP.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/cognitive_POMDP.png
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/image/gaze_task.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/gaze_task.png
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/image/internal_env.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/internal_env.png
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/image/sub_movements.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/sub_movements.png
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/image/time_v_target_size.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/time_v_target_size.png
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/image/visual_acuity.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/image/visual_acuity.png
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/output/behaviour_trace.csv:
--------------------------------------------------------------------------------
1 | step,target_width,target_x,target_y,fixate_x,fixate_y,episode
2 | 0.0,0.15,0.06414962203309098,-0.09282954521216029,-1.0,-1.0,0.0
3 | 1.0,0.15,0.06414962203309098,-0.09282954521216029,0.022952957267056928,-0.006057112040038275,0.0
4 | 2.0,0.15,0.06414962203309098,-0.09282954521216029,0.04155284368127787,-0.09377532502789823,0.0
5 | 0.0,0.15,-0.6019876004334083,-0.01653672263142971,-1.0,-1.0,1.0
6 | 1.0,0.15,-0.6019876004334083,-0.01653672263142971,0.0517793655965969,-0.0027747333854521543,1.0
7 | 2.0,0.15,-0.6019876004334083,-0.01653672263142971,-0.46166395072500815,-0.04279898880266997,1.0
8 | 3.0,0.15,-0.6019876004334083,-0.01653672263142971,-0.6167801155204129,-0.01282450722378098,1.0
9 | 0.0,0.15,0.2735984614270531,0.07705367206811643,-1.0,-1.0,2.0
10 | 1.0,0.15,0.2735984614270531,0.07705367206811643,0.06467241975995563,0.01139736623678792,2.0
11 | 2.0,0.15,0.2735984614270531,0.07705367206811643,0.2567464278578047,0.09175435009612219,2.0
12 | 0.0,0.15,-0.4091416057153972,0.19017308200151792,-1.0,-1.0,3.0
13 | 1.0,0.15,-0.4091416057153972,0.19017308200151792,0.04194268164996753,0.01701461828107996,3.0
14 | 2.0,0.15,-0.4091416057153972,0.19017308200151792,-0.3241411211315401,0.2637026101099477,3.0
15 | 3.0,0.15,-0.4091416057153972,0.19017308200151792,-0.44797239496562263,0.251710953880758,3.0
16 | 0.0,0.15,-0.43270166291918166,0.17741346914408665,-1.0,-1.0,4.0
17 | 1.0,0.15,-0.43270166291918166,0.17741346914408665,0.06365889434067198,0.008223384887533575,4.0
18 | 2.0,0.15,-0.43270166291918166,0.17741346914408665,-0.2806227767187054,0.2571779935728619,4.0
19 | 3.0,0.15,-0.43270166291918166,0.17741346914408665,-0.4115424460223932,0.2277943909412183,4.0
20 | 0.0,0.15,-0.23129327102684388,0.4138951414435922,-1.0,-1.0,5.0
21 | 1.0,0.15,-0.23129327102684388,0.4138951414435922,0.0336275538898401,0.009590282598844277,5.0
22 | 2.0,0.15,-0.23129327102684388,0.4138951414435922,-0.15376627437176435,0.3822021336329997,5.0
23 | 3.0,0.15,-0.23129327102684388,0.4138951414435922,-0.2845421921926004,0.4287584537408162,5.0
24 | 0.0,0.15,0.13119399718688762,-0.2760649421438989,-1.0,-1.0,6.0
25 | 1.0,0.15,0.13119399718688762,-0.2760649421438989,0.05725105287474052,-0.018994036110992575,6.0
26 | 2.0,0.15,0.13119399718688762,-0.2760649421438989,0.1561455574500748,-0.28200717375522844,6.0
27 | 0.0,0.15,0.05492472079290004,-0.1091640358530668,-1.0,-1.0,7.0
28 | 1.0,0.15,0.05492472079290004,-0.1091640358530668,0.07426093370549192,0.0007454376495246442,7.0
29 | 2.0,0.15,0.05492472079290004,-0.1091640358530668,0.039504127171326134,-0.1235305008939485,7.0
30 | 0.0,0.15,-0.09426987324585008,0.44388258583777,-1.0,-1.0,8.0
31 | 1.0,0.15,-0.09426987324585008,0.44388258583777,0.02245436246505295,0.0006928245864493396,8.0
32 | 2.0,0.15,-0.09426987324585008,0.44388258583777,-0.1585791253892003,0.46898011871891543,8.0
33 | 0.0,0.15,0.1788326944590339,-0.10607480320702313,-1.0,-1.0,9.0
34 | 1.0,0.15,0.1788326944590339,-0.10607480320702313,0.03230664141133567,0.0164669214868451,9.0
35 | 2.0,0.15,0.1788326944590339,-0.10607480320702313,0.20853109964240588,-0.1549831528301417,9.0
36 | 0.0,0.15,0.297092196113112,0.21844844944428302,-1.0,-1.0,10.0
37 | 1.0,0.15,0.297092196113112,0.21844844944428302,0.040204626309107525,-0.008753829929950116,10.0
38 | 2.0,0.15,0.297092196113112,0.21844844944428302,0.22202757522189964,0.21357752121546333,10.0
39 | 3.0,0.15,0.297092196113112,0.21844844944428302,0.3014779727637586,0.23356455635149298,10.0
40 | 0.0,0.15,-0.0010135714283298935,0.03252154433349142,-1.0,-1.0,11.0
41 | 1.0,0.15,-0.0010135714283298935,0.03252154433349142,0.032673756789997545,0.006636726653141751,11.0
42 | 0.0,0.15,0.14311920421285432,-0.1904122733975137,-1.0,-1.0,12.0
43 | 1.0,0.15,0.14311920421285432,-0.1904122733975137,0.05320920868652877,0.01192599646948581,12.0
44 | 2.0,0.15,0.14311920421285432,-0.1904122733975137,0.17039800284622325,-0.2301220193302062,12.0
45 | 0.0,0.15,0.1304458056773149,-0.17277410465817883,-1.0,-1.0,13.0
46 | 1.0,0.15,0.1304458056773149,-0.17277410465817883,0.07206775310725205,0.0022185344400437314,13.0
47 | 2.0,0.15,0.1304458056773149,-0.17277410465817883,0.15132024072142863,-0.1810170380982514,13.0
48 | 0.0,0.15,-0.2319148262633478,0.11551146203429558,-1.0,-1.0,14.0
49 | 1.0,0.15,-0.2319148262633478,0.11551146203429558,0.06500919418862863,0.02093501024815099,14.0
50 | 2.0,0.15,-0.2319148262633478,0.11551146203429558,-0.25764939483544036,0.11023141618499335,14.0
51 | 0.0,0.15,0.08798296512857404,0.5050904680819654,-1.0,-1.0,15.0
52 | 1.0,0.15,0.08798296512857404,0.5050904680819654,0.03074062566277619,-0.014660190308663086,15.0
53 | 2.0,0.15,0.08798296512857404,0.5050904680819654,0.061462926050982136,0.47385098253255187,15.0
54 | 0.0,0.15,-0.2199571192121488,-0.38194913727322655,-1.0,-1.0,16.0
55 | 1.0,0.15,-0.2199571192121488,-0.38194913727322655,0.04103194201179049,-0.016897207485399093,16.0
56 | 2.0,0.15,-0.2199571192121488,-0.38194913727322655,-0.1824121517619583,-0.35504430366653866,16.0
57 | 0.0,0.15,-0.40502354625131043,0.24221279961199477,-1.0,-1.0,17.0
58 | 1.0,0.15,-0.40502354625131043,0.24221279961199477,0.01366919697760631,-0.011032686269866872,17.0
59 | 2.0,0.15,-0.40502354625131043,0.24221279961199477,-0.31228562332639914,0.1944457467497777,17.0
60 | 3.0,0.15,-0.40502354625131043,0.24221279961199477,-0.43113638296124585,0.2593286077752522,17.0
61 | 0.0,0.15,-0.22727139645640565,0.2946231485409358,-1.0,-1.0,18.0
62 | 1.0,0.15,-0.22727139645640565,0.2946231485409358,0.0520387559850861,0.025569081145251568,18.0
63 | 2.0,0.15,-0.22727139645640565,0.2946231485409358,-0.19906948297016727,0.3292951359416989,18.0
64 | 0.0,0.15,0.38715871532518803,0.20774700755126518,-1.0,-1.0,19.0
65 | 1.0,0.15,0.38715871532518803,0.20774700755126518,0.05716638781815297,0.02865869820594034,19.0
66 | 2.0,0.15,0.38715871532518803,0.20774700755126518,0.34506664507958873,0.2264340116256558,19.0
67 | 0.0,0.15,0.051585093869399294,-0.09859287099233052,-1.0,-1.0,20.0
68 | 1.0,0.15,0.051585093869399294,-0.09859287099233052,0.031677345778908836,-0.011722727661730548,20.0
69 | 2.0,0.15,0.051585093869399294,-0.09859287099233052,0.051921909518558566,-0.11337538849596154,20.0
70 | 0.0,0.15,-0.35042118098373126,-0.08159096141110943,-1.0,-1.0,21.0
71 | 1.0,0.15,-0.35042118098373126,-0.08159096141110943,0.04256702571677419,-0.03724214483816485,21.0
72 | 2.0,0.15,-0.35042118098373126,-0.08159096141110943,-0.36780503506284273,-0.12048313941483238,21.0
73 | 0.0,0.15,-0.05847971133248091,0.007466396681718116,-1.0,-1.0,22.0
74 | 1.0,0.15,-0.05847971133248091,0.007466396681718116,0.05882039862008667,0.009462747740083951,22.0
75 | 2.0,0.15,-0.05847971133248091,0.007466396681718116,-0.0735318983679421,-0.022065207603506876,22.0
76 | 0.0,0.15,-0.7121889593373226,-0.058211251887428206,-1.0,-1.0,23.0
77 | 1.0,0.15,-0.7121889593373226,-0.058211251887428206,0.07833873456267743,-0.0046009149695391705,23.0
78 | 2.0,0.15,-0.7121889593373226,-0.058211251887428206,-0.4888168532941895,-0.01542165319999887,23.0
79 | 3.0,0.15,-0.7121889593373226,-0.058211251887428206,-0.7052765656466627,-0.05884400570490193,23.0
80 | 0.0,0.15,-0.06377558808898039,0.07697389227769467,-1.0,-1.0,24.0
81 | 1.0,0.15,-0.06377558808898039,0.07697389227769467,0.032571280308437985,0.0034459504458742023,24.0
82 | 2.0,0.15,-0.06377558808898039,0.07697389227769467,-0.05672195932675608,0.09931880624900916,24.0
83 | 0.0,0.15,0.7335238138165546,-0.13170960527105272,-1.0,-1.0,25.0
84 | 1.0,0.15,0.7335238138165546,-0.13170960527105272,0.06745996712968717,-0.006104395446774213,25.0
85 | 2.0,0.15,0.7335238138165546,-0.13170960527105272,0.5608453285367376,-0.12081348609434935,25.0
86 | 3.0,0.15,0.7335238138165546,-0.13170960527105272,0.7560028011382774,-0.16552934483847775,25.0
87 | 0.0,0.15,0.21092527942879483,0.05766350974855612,-1.0,-1.0,26.0
88 | 1.0,0.15,0.21092527942879483,0.05766350974855612,0.04813346464752367,0.004511532595652595,26.0
89 | 2.0,0.15,0.21092527942879483,0.05766350974855612,0.15622527683658335,0.10737691450154997,26.0
90 | 0.0,0.15,0.013798287743157896,0.33217787272568056,-1.0,-1.0,27.0
91 | 1.0,0.15,0.013798287743157896,0.33217787272568056,0.03886747129985962,0.024548304971569446,27.0
92 | 2.0,0.15,0.013798287743157896,0.33217787272568056,-0.01732608431268916,0.3563519665124801,27.0
93 | 0.0,0.15,-0.360303984705875,-0.327273325302741,-1.0,-1.0,28.0
94 | 1.0,0.15,-0.360303984705875,-0.327273325302741,0.02726670469832683,0.012457424701257254,28.0
95 | 2.0,0.15,-0.360303984705875,-0.327273325302741,-0.26902971168434164,-0.29556970814055217,28.0
96 | 3.0,0.15,-0.360303984705875,-0.327273325302741,-0.3729469247514502,-0.3427752180041761,28.0
97 | 0.0,0.15,-0.03209989163813316,-0.45575960437966573,-1.0,-1.0,29.0
98 | 1.0,0.15,-0.03209989163813316,-0.45575960437966573,0.011335510217212637,0.030859051994577974,29.0
99 | 2.0,0.15,-0.03209989163813316,-0.45575960437966573,-0.139917793287394,-0.40786515358813574,29.0
100 | 3.0,0.15,-0.03209989163813316,-0.45575960437966573,-0.07065421227561385,-0.4917718377430215,29.0
101 | 0.0,0.15,-0.11114769843081844,-0.25760476017786643,-1.0,-1.0,30.0
102 | 1.0,0.15,-0.11114769843081844,-0.25760476017786643,0.046768882821610624,0.012103487681706472,30.0
103 | 2.0,0.15,-0.11114769843081844,-0.25760476017786643,-0.06716154994302884,-0.2680336906946792,30.0
104 | 0.0,0.15,-0.21572122240217956,0.21900657337141247,-1.0,-1.0,31.0
105 | 1.0,0.15,-0.21572122240217956,0.21900657337141247,0.04701869869948062,-0.005015993217710385,31.0
106 | 2.0,0.15,-0.21572122240217956,0.21900657337141247,-0.17726392496954035,0.28666828358076674,31.0
107 | 3.0,0.15,-0.21572122240217956,0.21900657337141247,-0.26830770770877527,0.2567624560905446,31.0
108 | 0.0,0.15,-0.2557688203408042,-0.3705300094035497,-1.0,-1.0,32.0
109 | 1.0,0.15,-0.2557688203408042,-0.3705300094035497,0.022672953975151525,0.011432524554609176,32.0
110 | 2.0,0.15,-0.2557688203408042,-0.3705300094035497,-0.21830717914722544,-0.39557011550092075,32.0
111 | 0.0,0.15,0.3925558133595611,0.009645602631218825,-1.0,-1.0,33.0
112 | 1.0,0.15,0.3925558133595611,0.009645602631218825,0.033052474580954706,0.002330908544370104,33.0
113 | 2.0,0.15,0.3925558133595611,0.009645602631218825,0.4051381077328279,0.00606755656341242,33.0
114 | 0.0,0.15,-0.5792107364870995,-0.1341705550315782,-1.0,-1.0,34.0
115 | 1.0,0.15,-0.5792107364870995,-0.1341705550315782,0.002772622556956948,-0.00596026714047021,34.0
116 | 2.0,0.15,-0.5792107364870995,-0.1341705550315782,-0.4994376094388785,-0.1406418191286596,34.0
117 | 3.0,0.15,-0.5792107364870995,-0.1341705550315782,-0.5785026175998892,-0.155571987275992,34.0
118 | 0.0,0.15,-0.09905383104177455,-0.16520292858999563,-1.0,-1.0,35.0
119 | 1.0,0.15,-0.09905383104177455,-0.16520292858999563,0.049125101213618225,-0.01537779278047801,35.0
120 | 2.0,0.15,-0.09905383104177455,-0.16520292858999563,-0.10655875053079945,-0.12863714554675929,35.0
121 | 0.0,0.15,0.014167280348627517,0.15857296217323158,-1.0,-1.0,36.0
122 | 1.0,0.15,0.014167280348627517,0.15857296217323158,0.03305959117787621,0.02879766143838659,36.0
123 | 2.0,0.15,0.014167280348627517,0.15857296217323158,0.02258934805537644,0.17435283824891662,36.0
124 | 0.0,0.15,-0.21258332296495472,-0.2738291738166111,-1.0,-1.0,37.0
125 | 1.0,0.15,-0.21258332296495472,-0.2738291738166111,0.05040203028709987,0.0009078370570102637,37.0
126 | 2.0,0.15,-0.21258332296495472,-0.2738291738166111,-0.21231669611041493,-0.2624300954214877,37.0
127 | 0.0,0.15,-0.11325574379967678,0.2606798163870956,-1.0,-1.0,38.0
128 | 1.0,0.15,-0.11325574379967678,0.2606798163870956,0.050771828488389306,-0.029084077030268635,38.0
129 | 2.0,0.15,-0.11325574379967678,0.2606798163870956,-0.15466377415483434,0.2451767228104351,38.0
130 | 0.0,0.15,0.2970797829940354,0.06893641225287236,-1.0,-1.0,39.0
131 | 1.0,0.15,0.2970797829940354,0.06893641225287236,0.05791949596829962,0.024893731115873983,39.0
132 | 2.0,0.15,0.2970797829940354,0.06893641225287236,0.3283271125241888,0.051679102169388466,39.0
133 | 0.0,0.15,-0.2963203554072132,0.12079854442864445,-1.0,-1.0,40.0
134 | 1.0,0.15,-0.2963203554072132,0.12079854442864445,0.06434721731752746,-0.0072914479440132195,40.0
135 | 2.0,0.15,-0.2963203554072132,0.12079854442864445,-0.2630085542712542,0.14108980161523205,40.0
136 | 0.0,0.15,-0.26988059977344403,0.019232442339303834,-1.0,-1.0,41.0
137 | 1.0,0.15,-0.26988059977344403,0.019232442339303834,0.04012862463736115,0.018798609769400062,41.0
138 | 2.0,0.15,-0.26988059977344403,0.019232442339303834,-0.3091295268470044,0.046094511674736195,41.0
139 | 0.0,0.15,0.3758137523899551,0.4781124222105405,-1.0,-1.0,42.0
140 | 1.0,0.15,0.3758137523899551,0.4781124222105405,0.02122804053703171,0.016671632512660284,42.0
141 | 2.0,0.15,0.3758137523899551,0.4781124222105405,0.3552325628681667,0.3914842955693962,42.0
142 | 3.0,0.15,0.3758137523899551,0.4781124222105405,0.36926595423414255,0.5183266431142247,42.0
143 | 0.0,0.15,0.15222996964533295,0.32291035215543207,-1.0,-1.0,43.0
144 | 1.0,0.15,0.15222996964533295,0.32291035215543207,0.037047387645389396,-0.017120727028373754,43.0
145 | 2.0,0.15,0.15222996964533295,0.32291035215543207,0.14624229001337818,0.3831583523828629,43.0
146 | 0.0,0.15,-0.08806591231709889,-0.025347802810632616,-1.0,-1.0,44.0
147 | 1.0,0.15,-0.08806591231709889,-0.025347802810632616,0.041929553923185,-0.005869781327381585,44.0
148 | 2.0,0.15,-0.08806591231709889,-0.025347802810632616,-0.10094349725108763,-0.014614378249899963,44.0
149 | 0.0,0.15,-0.12711103701755513,-0.17693277412880634,-1.0,-1.0,45.0
150 | 1.0,0.15,-0.12711103701755513,-0.17693277412880634,0.0593024206893812,0.004370286214580236,45.0
151 | 2.0,0.15,-0.12711103701755513,-0.17693277412880634,-0.12388865752662613,-0.18843359075306945,45.0
152 | 0.0,0.15,-0.08256924015915106,0.048593197245711324,-1.0,-1.0,46.0
153 | 1.0,0.15,-0.08256924015915106,0.048593197245711324,0.06810056120511035,-0.0024220661633051394,46.0
154 | 2.0,0.15,-0.08256924015915106,0.048593197245711324,-0.11214327627343487,0.09725742079152619,46.0
155 | 0.0,0.15,-0.13262781190697362,0.43200649111674316,-1.0,-1.0,47.0
156 | 1.0,0.15,-0.13262781190697362,0.43200649111674316,0.03844814118539968,-0.0027020399505069364,47.0
157 | 2.0,0.15,-0.13262781190697362,0.43200649111674316,-0.1190194025468621,0.3848054887445577,47.0
158 | 0.0,0.15,-0.11458794491331732,0.23435796026687644,-1.0,-1.0,48.0
159 | 1.0,0.15,-0.11458794491331732,0.23435796026687644,0.049030087262185926,0.011897053674317596,48.0
160 | 2.0,0.15,-0.11458794491331732,0.23435796026687644,-0.16008155223306866,0.260335089112164,48.0
161 | 0.0,0.15,0.6050736122918101,-0.4415878085778648,-1.0,-1.0,49.0
162 | 1.0,0.15,0.6050736122918101,-0.4415878085778648,0.0456558282825393,0.003535315417539779,49.0
163 | 2.0,0.15,0.6050736122918101,-0.4415878085778648,0.45241791523245317,-0.43710174506214855,49.0
164 | 3.0,0.15,0.6050736122918101,-0.4415878085778648,0.5947286574836945,-0.4250176157386889,49.0
165 | 0.0,0.15,0.028534804504455533,-0.5029034689888454,-1.0,-1.0,50.0
166 | 1.0,0.15,0.028534804504455533,-0.5029034689888454,0.04562523035287361,0.017295290789046375,50.0
167 | 2.0,0.15,0.028534804504455533,-0.5029034689888454,0.05679213503125965,-0.4251238137994234,50.0
168 | 3.0,0.15,0.028534804504455533,-0.5029034689888454,0.06277804943206478,-0.5421370560026485,50.0
169 | 0.0,0.15,0.13469486289502433,-0.018689808144753378,-1.0,-1.0,51.0
170 | 1.0,0.15,0.13469486289502433,-0.018689808144753378,0.051618668485832554,0.02309904429515406,51.0
171 | 2.0,0.15,0.13469486289502433,-0.018689808144753378,0.16401229018374777,-0.0032743946462125695,51.0
172 | 0.0,0.15,0.03472171946092767,0.08520911109949268,-1.0,-1.0,52.0
173 | 1.0,0.15,0.03472171946092767,0.08520911109949268,0.052852156548921635,-0.007758706006038601,52.0
174 | 2.0,0.15,0.03472171946092767,0.08520911109949268,0.04807617403923386,0.12117233702376463,52.0
175 | 0.0,0.15,0.07281945112720674,0.24121167965817558,-1.0,-1.0,53.0
176 | 1.0,0.15,0.07281945112720674,0.24121167965817558,0.05093923716587817,0.009334745376458958,53.0
177 | 2.0,0.15,0.07281945112720674,0.24121167965817558,0.07407276337486085,0.28688718218489656,53.0
178 | 0.0,0.15,0.6036099272755281,-0.6510761013245265,-1.0,-1.0,54.0
179 | 1.0,0.15,0.6036099272755281,-0.6510761013245265,0.040313000829106375,-0.005629327786859507,54.0
180 | 2.0,0.15,0.6036099272755281,-0.6510761013245265,0.41624308883742517,-0.43746266644469783,54.0
181 | 3.0,0.15,0.6036099272755281,-0.6510761013245265,0.5084991891979846,-0.6335392718309133,54.0
182 | 4.0,0.15,0.6036099272755281,-0.6510761013245265,0.5986746484057115,-0.6673983535708621,54.0
183 | 0.0,0.15,-0.38249193660182096,0.11278538372898168,-1.0,-1.0,55.0
184 | 1.0,0.15,-0.38249193660182096,0.11278538372898168,0.041975369073639676,-0.005791739047306393,55.0
185 | 2.0,0.15,-0.38249193660182096,0.11278538372898168,-0.372428313960294,0.10727477484414108,55.0
186 | 0.0,0.15,-0.048559227233096615,0.2876997488282835,-1.0,-1.0,56.0
187 | 1.0,0.15,-0.048559227233096615,0.2876997488282835,0.06717131599300977,-0.004576395134306483,56.0
188 | 2.0,0.15,-0.048559227233096615,0.2876997488282835,-0.06676296914803206,0.356423500298756,56.0
189 | 0.0,0.15,0.24699293463725366,-0.08012061281579443,-1.0,-1.0,57.0
190 | 1.0,0.15,0.24699293463725366,-0.08012061281579443,0.03374253273613548,0.01957981216463243,57.0
191 | 2.0,0.15,0.24699293463725366,-0.08012061281579443,0.25313774410125695,-0.09707887478864087,57.0
192 | 0.0,0.15,0.14328171731660966,-0.04604751295962332,-1.0,-1.0,58.0
193 | 1.0,0.15,0.14328171731660966,-0.04604751295962332,0.04398067553636723,0.00343140325174186,58.0
194 | 2.0,0.15,0.14328171731660966,-0.04604751295962332,0.11684359820270622,-0.05904696259540202,58.0
195 | 0.0,0.15,0.2420063453315825,0.09815412226409745,-1.0,-1.0,59.0
196 | 1.0,0.15,0.2420063453315825,0.09815412226409745,0.04364679433184551,0.01562795969513265,59.0
197 | 2.0,0.15,0.2420063453315825,0.09815412226409745,0.23876502648537987,0.10025650797433755,59.0
198 | 0.0,0.15,-0.3054900856834516,-0.4105104403641334,-1.0,-1.0,60.0
199 | 1.0,0.15,-0.3054900856834516,-0.4105104403641334,0.042615296846733854,-0.0036176807661398762,60.0
200 | 2.0,0.15,-0.3054900856834516,-0.4105104403641334,-0.2501658227882057,-0.38577027139134656,60.0
201 | 0.0,0.15,0.2528246121566485,0.4156339727682298,-1.0,-1.0,61.0
202 | 1.0,0.15,0.2528246121566485,0.4156339727682298,0.049234816588684636,-0.031877686718629686,61.0
203 | 2.0,0.15,0.2528246121566485,0.4156339727682298,0.19588534506519198,0.507019792245592,61.0
204 | 3.0,0.15,0.2528246121566485,0.4156339727682298,0.2288160164352217,0.48210417668230565,61.0
205 | 0.0,0.15,0.5339529395844452,0.12095781254025344,-1.0,-1.0,62.0
206 | 1.0,0.15,0.5339529395844452,0.12095781254025344,0.026651969980413203,0.008018272567443741,62.0
207 | 2.0,0.15,0.5339529395844452,0.12095781254025344,0.47177530234383414,0.11531164094678442,62.0
208 | 0.0,0.15,-0.30830156560046995,-0.09624866343194168,-1.0,-1.0,63.0
209 | 1.0,0.15,-0.30830156560046995,-0.09624866343194168,0.04563699167426992,0.002493875851612919,63.0
210 | 2.0,0.15,-0.30830156560046995,-0.09624866343194168,-0.2362059119647002,-0.06291433272060422,63.0
211 | 3.0,0.15,-0.30830156560046995,-0.09624866343194168,-0.3244653765558659,-0.0808664575364132,63.0
212 | 0.0,0.15,0.43446472653753404,0.14515415096782353,-1.0,-1.0,64.0
213 | 1.0,0.15,0.43446472653753404,0.14515415096782353,0.06381318600934156,-0.005277850829486452,64.0
214 | 2.0,0.15,0.43446472653753404,0.14515415096782353,0.44561120727588255,0.1874217070655866,64.0
215 | 0.0,0.15,0.021414007866896772,0.11803432419507363,-1.0,-1.0,65.0
216 | 1.0,0.15,0.021414007866896772,0.11803432419507363,0.01926858492225006,0.025178849131179643,65.0
217 | 2.0,0.15,0.021414007866896772,0.11803432419507363,-0.003380475971317909,0.1274972348273112,65.0
218 | 0.0,0.15,-0.4028512560702591,0.12961671617906234,-1.0,-1.0,66.0
219 | 1.0,0.15,-0.4028512560702591,0.12961671617906234,0.03989866960398037,0.011084014697792199,66.0
220 | 2.0,0.15,-0.4028512560702591,0.12961671617906234,-0.3798878087165995,0.18070576812395367,66.0
221 | 0.0,0.15,0.47926072233611483,-0.4620852377768718,-1.0,-1.0,67.0
222 | 1.0,0.15,0.47926072233611483,-0.4620852377768718,0.047054844119704,0.02043600240537462,67.0
223 | 2.0,0.15,0.47926072233611483,-0.4620852377768718,0.4716500912877814,-0.3519639376054769,67.0
224 | 3.0,0.15,0.47926072233611483,-0.4620852377768718,0.5005765673758799,-0.48503510289990187,67.0
225 | 0.0,0.15,-0.3539999907854146,-0.01494454868179744,-1.0,-1.0,68.0
226 | 1.0,0.15,-0.3539999907854146,-0.01494454868179744,0.028634466653953847,0.0025977557824362397,68.0
227 | 2.0,0.15,-0.3539999907854146,-0.01494454868179744,-0.3107122399766935,-0.05514276781928499,68.0
228 | 0.0,0.15,0.230716284717682,0.13640014049354968,-1.0,-1.0,69.0
229 | 1.0,0.15,0.230716284717682,0.13640014049354968,0.053067792362395824,-0.023428836725051597,69.0
230 | 2.0,0.15,0.230716284717682,0.13640014049354968,0.18790464103588556,0.106833651342968,69.0
231 | 0.0,0.15,0.05690541663401461,-0.35686683304857864,-1.0,-1.0,70.0
232 | 1.0,0.15,0.05690541663401461,-0.35686683304857864,0.05860590345151194,0.020754495498751593,70.0
233 | 2.0,0.15,0.05690541663401461,-0.35686683304857864,-0.012516840782491687,-0.3114224250867912,70.0
234 | 3.0,0.15,0.05690541663401461,-0.35686683304857864,0.0787443081123331,-0.3662274477420743,70.0
235 | 0.0,0.15,-0.1897894059320973,1.0,-1.0,-1.0,71.0
236 | 1.0,0.15,-0.1897894059320973,1.0,0.05493091159721517,-0.0012256752407652834,71.0
237 | 2.0,0.15,-0.1897894059320973,1.0,-0.11494042183996693,0.5839021517669961,71.0
238 | 3.0,0.15,-0.1897894059320973,1.0,-0.12845720778926661,0.9287329591181651,71.0
239 | 4.0,0.15,-0.1897894059320973,1.0,-0.17908269638778357,0.9958940571187267,71.0
240 | 0.0,0.15,0.2894556977298768,-0.19042996524257982,-1.0,-1.0,72.0
241 | 1.0,0.15,0.2894556977298768,-0.19042996524257982,0.06591334129256077,-0.021672374850798422,72.0
242 | 2.0,0.15,0.2894556977298768,-0.19042996524257982,0.26557900013650754,-0.18567349069256536,72.0
243 | 0.0,0.15,0.2289136167829819,0.27547648426779997,-1.0,-1.0,73.0
244 | 1.0,0.15,0.2289136167829819,0.27547648426779997,0.04146521939364157,0.007863348624264105,73.0
245 | 2.0,0.15,0.2289136167829819,0.27547648426779997,0.21140341118622086,0.31235243592848866,73.0
246 | 0.0,0.15,0.29464148446892996,-0.09670023612505375,-1.0,-1.0,74.0
247 | 1.0,0.15,0.29464148446892996,-0.09670023612505375,0.03473276501386732,0.02366963215703282,74.0
248 | 2.0,0.15,0.29464148446892996,-0.09670023612505375,0.29607887783420983,-0.038247222756002,74.0
249 | 0.0,0.15,0.3057012956840737,-0.1806763284059769,-1.0,-1.0,75.0
250 | 1.0,0.15,0.3057012956840737,-0.1806763284059769,0.05035416607246748,-0.00352481742655003,75.0
251 | 2.0,0.15,0.3057012956840737,-0.1806763284059769,0.35233367276661204,-0.17642769307522474,75.0
252 | 0.0,0.15,0.43012519249758574,0.22795864779806024,-1.0,-1.0,76.0
253 | 1.0,0.15,0.43012519249758574,0.22795864779806024,0.04891613582403732,0.004578535300660017,76.0
254 | 2.0,0.15,0.43012519249758574,0.22795864779806024,0.38309366193720434,0.18223495814373736,76.0
255 | 0.0,0.15,-0.13013651525088213,0.20875569325155852,-1.0,-1.0,77.0
256 | 1.0,0.15,-0.13013651525088213,0.20875569325155852,0.0278877339227807,0.010394419962968885,77.0
257 | 2.0,0.15,-0.13013651525088213,0.20875569325155852,-0.12935723776418123,0.19684372102835468,77.0
258 | 0.0,0.15,-0.45622768013490356,0.35583833588891983,-1.0,-1.0,78.0
259 | 1.0,0.15,-0.45622768013490356,0.35583833588891983,0.052425303454858484,0.012134049029832082,78.0
260 | 2.0,0.15,-0.45622768013490356,0.35583833588891983,-0.2880896228012208,0.2902996644808326,78.0
261 | 3.0,0.15,-0.45622768013490356,0.35583833588891983,-0.44996333641647984,0.35804634576498834,78.0
262 | 0.0,0.15,-0.34682336048798984,-0.26268658735084593,-1.0,-1.0,79.0
263 | 1.0,0.15,-0.34682336048798984,-0.26268658735084593,0.07159180011960406,-0.0033328598431632073,79.0
264 | 2.0,0.15,-0.34682336048798984,-0.26268658735084593,-0.2825572537592489,-0.21329305194325823,79.0
265 | 3.0,0.15,-0.34682336048798984,-0.26268658735084593,-0.3688974947478353,-0.2998556158160257,79.0
266 | 0.0,0.15,-0.07870547938499006,-0.3590920891782355,-1.0,-1.0,80.0
267 | 1.0,0.15,-0.07870547938499006,-0.3590920891782355,0.05104582879604966,-0.0044563645709609885,80.0
268 | 2.0,0.15,-0.07870547938499006,-0.3590920891782355,-0.07419677030806847,-0.3343262643939309,80.0
269 | 0.0,0.15,0.2595751172318591,-0.11938729575498783,-1.0,-1.0,81.0
270 | 1.0,0.15,0.2595751172318591,-0.11938729575498783,0.04395797955333259,0.025731629186670928,81.0
271 | 2.0,0.15,0.2595751172318591,-0.11938729575498783,0.24427088238909558,-0.18037443153479354,81.0
272 | 0.0,0.15,-0.4152503203327221,-0.3729230417884008,-1.0,-1.0,82.0
273 | 1.0,0.15,-0.4152503203327221,-0.3729230417884008,0.05659389401944776,0.0015146173653884172,82.0
274 | 2.0,0.15,-0.4152503203327221,-0.3729230417884008,-0.2907187899038383,-0.2575232744444416,82.0
275 | 3.0,0.15,-0.4152503203327221,-0.3729230417884008,-0.43542322006932327,-0.36194061745697864,82.0
276 | 0.0,0.15,0.06833655919544931,-0.3578982170855646,-1.0,-1.0,83.0
277 | 1.0,0.15,0.06833655919544931,-0.3578982170855646,0.062473579940861744,0.015880003671061238,83.0
278 | 2.0,0.15,0.06833655919544931,-0.3578982170855646,0.06477214586412991,-0.3608195398764706,83.0
279 | 0.0,0.15,-0.18782958245868078,-0.2957670484861216,-1.0,-1.0,84.0
280 | 1.0,0.15,-0.18782958245868078,-0.2957670484861216,0.04352090889416995,-0.01346503311007322,84.0
281 | 2.0,0.15,-0.18782958245868078,-0.2957670484861216,-0.14688455263980343,-0.33236018294895964,84.0
282 | 0.0,0.15,-0.4104674464924564,-0.24592297397902857,-1.0,-1.0,85.0
283 | 1.0,0.15,-0.4104674464924564,-0.24592297397902857,0.04062409915979944,0.023775673120202134,85.0
284 | 2.0,0.15,-0.4104674464924564,-0.24592297397902857,-0.3969725589743864,-0.23205898409435435,85.0
285 | 0.0,0.15,-0.0899391373110621,-0.24878143578794137,-1.0,-1.0,86.0
286 | 1.0,0.15,-0.0899391373110621,-0.24878143578794137,0.0384996928531391,0.007605025983544439,86.0
287 | 2.0,0.15,-0.0899391373110621,-0.24878143578794137,-0.10448857847222599,-0.320608550649896,86.0
288 | 0.0,0.15,0.22183678495821674,-0.044633705885785205,-1.0,-1.0,87.0
289 | 1.0,0.15,0.22183678495821674,-0.044633705885785205,0.06138592120157123,0.014220974837514366,87.0
290 | 2.0,0.15,0.22183678495821674,-0.044633705885785205,0.2275532498231114,-0.1003589818149725,87.0
291 | 0.0,0.15,0.17973235230787293,0.1054573166264953,-1.0,-1.0,88.0
292 | 1.0,0.15,0.17973235230787293,0.1054573166264953,0.04390686084806695,0.0068106848190078164,88.0
293 | 2.0,0.15,0.17973235230787293,0.1054573166264953,0.22745039782475637,0.10880135587843613,88.0
294 | 0.0,0.15,0.024232540309781267,0.08823578646862316,-1.0,-1.0,89.0
295 | 1.0,0.15,0.024232540309781267,0.08823578646862316,0.02177577870772335,-0.0023974348772158676,89.0
296 | 2.0,0.15,0.024232540309781267,0.08823578646862316,0.028704555081370504,0.11363150670307684,89.0
297 | 0.0,0.15,0.12510010345250125,0.5049281053802411,-1.0,-1.0,90.0
298 | 1.0,0.15,0.12510010345250125,0.5049281053802411,0.03159606715752105,-0.001350709820192308,90.0
299 | 2.0,0.15,0.12510010345250125,0.5049281053802411,0.16610943367172865,0.5622599427025267,90.0
300 | 0.0,0.15,0.37176927763744083,0.18149997923124098,-1.0,-1.0,91.0
301 | 1.0,0.15,0.37176927763744083,0.18149997923124098,0.030765377046042257,0.009001317167140309,91.0
302 | 2.0,0.15,0.37176927763744083,0.18149997923124098,0.3156905735020419,0.18738362679554607,91.0
303 | 0.0,0.15,0.35827533465789985,0.5231881817476288,-1.0,-1.0,92.0
304 | 1.0,0.15,0.35827533465789985,0.5231881817476288,0.03449691832105268,0.013570796050154075,92.0
305 | 2.0,0.15,0.35827533465789985,0.5231881817476288,0.27456614771874693,0.46944059683556405,92.0
306 | 3.0,0.15,0.35827533465789985,0.5231881817476288,0.3424637532830716,0.5725817368293623,92.0
307 | 0.0,0.15,-0.4749629527354867,0.21436051662154296,-1.0,-1.0,93.0
308 | 1.0,0.15,-0.4749629527354867,0.21436051662154296,0.04581524599597033,0.009300807472284733,93.0
309 | 2.0,0.15,-0.4749629527354867,0.21436051662154296,-0.3904651534498414,0.2106282446625461,93.0
310 | 3.0,0.15,-0.4749629527354867,0.21436051662154296,-0.49296192856726023,0.22788851750009345,93.0
311 | 0.0,0.15,0.2563971181361048,-0.45207121570246067,-1.0,-1.0,94.0
312 | 1.0,0.15,0.2563971181361048,-0.45207121570246067,0.062249544992495576,0.01421147584998032,94.0
313 | 2.0,0.15,0.2563971181361048,-0.45207121570246067,0.28873523938394974,-0.42431173724507887,94.0
314 | 0.0,0.15,-0.11699825099127519,-0.012616547848888224,-1.0,-1.0,95.0
315 | 1.0,0.15,-0.11699825099127519,-0.012616547848888224,0.036235154645072466,0.02240319662878835,95.0
316 | 2.0,0.15,-0.11699825099127519,-0.012616547848888224,-0.13635482594512197,0.023346386003769887,95.0
317 | 0.0,0.15,-0.563067614511279,-0.133422326825027,-1.0,-1.0,96.0
318 | 1.0,0.15,-0.563067614511279,-0.133422326825027,0.05633923033678578,0.01970467595343732,96.0
319 | 2.0,0.15,-0.563067614511279,-0.133422326825027,-0.415691901855508,-0.10306197502599378,96.0
320 | 3.0,0.15,-0.563067614511279,-0.133422326825027,-0.5576120003404873,-0.1320557464260784,96.0
321 | 0.0,0.15,0.09438061162636577,0.14078322003740132,-1.0,-1.0,97.0
322 | 1.0,0.15,0.09438061162636577,0.14078322003740132,0.04914396475444823,0.01293729776848159,97.0
323 | 2.0,0.15,0.09438061162636577,0.14078322003740132,0.11965079624613723,0.18327703015810212,97.0
324 | 0.0,0.15,0.3205021573192244,-0.31866902934058344,-1.0,-1.0,98.0
325 | 1.0,0.15,0.3205021573192244,-0.31866902934058344,0.030143631406051308,-0.014656394195799909,98.0
326 | 2.0,0.15,0.3205021573192244,-0.31866902934058344,0.2224616543168862,-0.2931398805230987,98.0
327 | 3.0,0.15,0.3205021573192244,-0.31866902934058344,0.31822088776472296,-0.3422053725624213,98.0
328 | 0.0,0.15,0.2818323846028262,0.19257608412474328,-1.0,-1.0,99.0
329 | 1.0,0.15,0.2818323846028262,0.19257608412474328,0.06266328374018314,-0.019065860245672105,99.0
330 | 2.0,0.15,0.2818323846028262,0.19257608412474328,0.2746391341886509,0.2463885597684051,99.0
331 |
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/output/monitor.csv:
--------------------------------------------------------------------------------
1 | #{"t_start": 1651051715.872973, "env_id": null}
2 | r,l,t
3 | -270.238114,197,0.126351
4 | -70.45687,41,0.149668
5 | -450.6688,331,0.377506
6 | -40.491495,27,0.401056
7 | -36.059652,31,0.424587
8 | -68.402355,33,0.449067
9 | -651.427712,501,0.731244
10 | -5.560864,4,0.733522
11 | -726.74203,501,0.99392
12 | -210.438871,246,1.139322
13 | -417.107543,259,2.040449
14 | -163.488867,90,2.095777
15 | -413.139406,286,2.253394
16 | -96.259316,102,2.311559
17 | -686.001824,426,2.548265
18 | -36.668556,28,2.563605
19 | -427.858066,318,2.738643
20 | -751.233692,501,3.011576
21 | -119.316984,103,3.067752
22 | -476.773689,328,4.013326
23 | -587.949785,409,4.243478
24 | -64.544827,34,4.261252
25 | -347.592196,233,4.389178
26 | -681.977659,501,4.659089
27 | -39.70289,27,4.674886
28 | -57.45577,33,4.693156
29 | -302.868888,246,4.829842
30 | -435.299536,310,5.777861
31 | -865.254316,501,6.063542
32 | -206.363768,158,6.158182
33 | -696.591703,501,6.446138
34 | -470.246127,261,6.590776
35 | -306.453634,201,6.704873
36 | -337.570682,362,6.912561
37 | -463.76346,468,7.953302
38 | -11.091099,9,7.958137
39 | -515.656413,325,8.143822
40 | -593.924865,343,8.328973
41 | -764.606413,403,8.537441
42 | -212.390134,155,8.617828
43 | -56.783953,48,8.642774
44 | -167.849232,127,8.709011
45 | -923.586156,501,9.723475
46 | -115.381516,88,9.770051
47 | -692.61731,501,10.034998
48 | -25.941193,18,10.044562
49 | -725.809703,482,10.31997
50 | -268.148157,350,10.500654
51 | -683.383136,501,11.511595
52 | -686.847391,501,11.769005
53 | -616.669334,501,12.044885
54 | -186.926686,101,12.098603
55 | -410.428062,242,12.240204
56 | -23.795001,13,12.247405
57 | -117.048887,58,12.278432
58 | -683.828573,501,13.307283
59 | -484.983503,386,13.507191
60 | -525.987955,432,13.73062
61 | -66.91409,50,13.756836
62 | -834.088138,501,14.02622
63 | -628.605142,501,14.300635
64 | -367.270673,344,15.233433
65 | -633.391248,501,15.493408
66 | -471.862407,315,15.658821
67 | -583.168004,501,15.917828
68 | -252.470917,156,16.00025
69 | -788.385194,501,17.104452
70 | -869.031968,501,17.387437
71 | -247.755699,236,17.512748
72 | -554.398024,501,17.775839
73 | -436.929084,501,18.050405
74 | -513.590731,313,19.005381
75 | -644.785843,496,19.274933
76 | -151.517457,161,19.374352
77 | -709.617392,501,19.640415
78 | -97.302595,72,19.677779
79 | -997.274582,501,19.953038
80 | -250.480285,217,20.845903
81 | -103.913595,65,20.879674
82 | -129.062583,105,20.934251
83 | -142.373265,72,20.971557
84 | -139.622501,78,21.012116
85 | -31.304996,34,21.029814
86 | -265.417721,204,21.134568
87 | -267.426301,166,21.249965
88 | -420.489312,313,21.423452
89 | -395.32882,254,21.557129
90 | -3.638601,3,21.558926
91 | -527.730012,385,21.761692
92 | -79.298416,44,21.784667
93 | -853.512807,501,22.79841
94 | -266.821066,247,22.925931
95 | -115.728642,68,22.960809
96 | -27.513952,23,22.973281
97 | -743.726516,501,23.233552
98 | -800.448276,501,23.491189
99 | -589.245595,501,23.750347
100 | -490.507758,308,24.65453
101 | -909.684089,470,24.903285
102 | -137.160766,122,24.966798
103 | -136.215509,124,25.031685
104 | -435.643075,258,25.170146
105 | -64.970162,47,25.19517
106 | -530.445917,344,25.373026
107 | -19.239661,19,25.382997
108 | -900.003598,501,26.395676
109 | -88.191976,53,26.423578
110 | -297.677491,217,26.536716
111 | -23.977437,18,26.546375
112 | -130.462261,97,26.596575
113 | -225.327652,220,26.709932
114 | -413.500658,369,26.900383
115 | -77.250564,58,26.930496
116 | -88.146758,56,26.95946
117 | -536.437296,402,27.167
118 | -5.064053,3,27.168782
119 | -207.727417,157,27.249876
120 | -1017.881826,501,28.237284
121 | -306.921896,258,28.370334
122 | -599.31209,501,28.629156
123 | -685.492016,501,28.888627
124 | -369.789108,225,29.004486
125 | -448.738238,501,29.994172
126 | -24.958855,15,30.002127
127 | -375.297178,284,30.148698
128 | -104.459979,72,30.186112
129 | -11.964718,13,30.192992
130 | -79.506423,46,30.216647
131 | -9.318598,7,30.2204
132 | -238.72573,193,30.31953
133 | -661.494081,501,30.57868
134 | -154.972209,96,30.62846
135 | -209.862287,157,30.709779
136 | -198.934958,244,30.835935
137 | -61.771382,31,30.852112
138 | -292.460423,197,30.954206
139 | -117.788986,91,31.734583
140 | -236.172908,124,31.799227
141 | -28.284777,24,31.811764
142 | -143.644375,114,31.870582
143 | -454.980379,297,32.02876
144 | -201.886001,162,32.11285
145 | -444.249692,219,32.234179
146 | -58.113063,43,32.256626
147 | -635.112842,428,32.477578
148 | -805.219098,501,32.736082
149 | -21.099919,17,32.74503
150 | -5.438016,5,32.747768
151 | -1128.80401,501,33.728304
152 | -294.507243,228,33.846657
153 | -31.774225,19,33.856771
154 | -683.94211,373,34.052945
155 | -85.592797,57,34.083025
156 | -217.012366,205,34.190326
157 | -726.560934,501,34.450016
158 | -417.775387,330,35.395153
159 | -6.336624,4,35.397486
160 | -30.662592,29,35.412612
161 | -245.292014,231,35.5324
162 | -713.621102,445,35.762142
163 | -231.620731,180,35.855667
164 | -222.690429,125,35.920385
165 | -102.781025,98,35.971138
166 | -253.187206,149,36.048243
167 | -94.243206,62,36.080373
168 | -115.859755,116,36.141463
169 | -451.575929,239,36.267143
170 | -20.113289,13,36.274027
171 | -298.460404,174,36.364168
172 | -575.523252,359,37.273887
173 | -147.530859,103,37.327122
174 | -563.0783,501,37.586466
175 | -402.632172,235,37.709309
176 | -514.287164,403,37.922127
177 | -387.283806,259,38.056219
178 | -231.749075,260,38.910821
179 | -456.299809,305,39.068392
180 | -410.415375,322,39.234573
181 | -352.316783,343,39.412163
182 | -75.116848,75,39.45099
183 | -303.791576,179,39.5431
184 | -643.682811,501,39.801513
185 | -443.261806,407,40.745171
186 | -317.704064,255,40.881954
187 | -432.956835,334,41.054438
188 | -189.132677,181,41.147997
189 | -1035.502595,501,41.407192
190 | -805.08485,501,41.665944
191 | -176.415396,132,41.734854
192 | -403.654102,291,42.63318
193 | -423.818276,434,42.856554
194 | -475.336539,349,43.036561
195 | -4.080923,4,43.038828
196 | -150.852508,83,43.081987
197 | -530.340171,501,43.340627
198 | -762.951706,378,43.535331
199 | -51.948013,50,44.289367
200 | -242.1572,166,44.375033
201 | -91.357708,65,44.40934
202 | -5.618529,5,44.412106
203 | -82.927803,47,44.436747
204 | -391.183684,245,44.563635
205 | -354.643619,241,44.688048
206 | -63.018276,38,44.707739
207 | -386.543786,344,44.884556
208 | -298.839204,200,44.988852
209 | -223.440882,166,45.074275
210 | -859.186875,501,45.333198
211 | -327.935273,168,46.140247
212 | -542.751992,501,46.399477
213 | -622.031937,399,46.605784
214 | -1089.394166,501,46.863352
215 | -143.512519,77,46.903026
216 | -285.352008,176,46.99401
217 | -608.80004,326,47.88989
218 | -41.278845,26,47.903778
219 | -573.983885,397,48.10836
220 | -22.793199,13,48.115264
221 | -496.469159,309,48.274795
222 | -737.621739,478,48.521487
223 | -235.202847,121,48.583923
224 | -694.884878,372,48.776056
225 | -377.053268,314,49.673651
226 | -560.061261,403,49.882402
227 | -511.953535,501,50.141807
228 | -65.172665,50,50.167615
229 | -404.84448,245,50.302659
230 | -90.536932,53,50.333036
231 | -488.327428,300,50.491577
232 | -319.27516,265,50.646296
233 | -354.305958,208,51.487087
234 | -297.069953,314,51.649384
235 | -3.936732,2,51.650627
236 | -71.701991,64,51.683701
237 | -9.730047,6,51.686969
238 | -882.081769,501,51.946964
239 | -148.432536,125,52.017169
240 | -173.665166,99,52.068771
241 | -345.257843,189,52.166533
242 | -24.031451,21,52.177608
243 | -480.069364,501,52.443967
244 | -189.52461,124,52.508083
245 | -262.018742,139,53.306486
246 | -182.435698,151,53.384191
247 | -671.736276,501,53.642752
248 | -333.472777,210,53.751138
249 | -683.660386,501,54.014385
250 | -314.596261,192,54.113867
251 | -273.901962,145,54.188488
252 | -168.131956,109,54.245066
253 | -1.828591,1,54.245774
254 | -263.462486,202,55.082513
255 | -210.324617,161,55.165542
256 | -121.74771,100,55.216918
257 | -563.595026,403,55.424777
258 | -145.824319,125,55.489594
259 | -1116.835224,501,55.748927
260 | -539.534915,501,56.007737
261 | -92.306932,83,56.05075
262 | -103.366579,57,56.080233
263 | -50.15664,34,56.097883
264 | -123.519471,73,56.864982
265 | -127.635617,119,56.926585
266 | -477.812803,251,57.056328
267 | -455.613034,334,57.229069
268 | -109.856141,76,57.268671
269 | -3.75146,3,57.270414
270 | -168.679902,110,57.328045
271 | -802.904051,501,57.587415
272 | -351.984541,249,57.716554
273 | -252.957009,292,57.866828
274 | -100.473388,78,57.907035
275 | -619.927454,340,58.811698
276 | -305.392081,171,58.899863
277 | -121.164008,96,58.949751
278 | -221.349855,129,59.016837
279 | -153.556921,119,59.078355
280 | -303.097736,287,59.22675
281 | -524.186145,501,59.4852
282 | -109.468051,58,59.515068
283 | -710.788964,444,60.487546
284 | -149.386425,148,60.564102
285 | -969.725995,501,60.82261
286 | -610.427908,501,61.080847
287 | -543.622891,438,61.30673
288 | -30.993787,28,61.321322
289 | -289.809657,213,61.431841
290 | -90.1746,75,61.470719
291 | -71.945301,56,61.500555
292 | -521.031288,387,62.444912
293 | -548.757956,304,62.60145
294 | -75.179505,52,62.628627
295 | -28.580685,21,62.6396
296 | -201.52732,103,62.692948
297 | -306.087152,205,62.79919
298 | -172.661651,96,62.84883
299 | -25.684064,19,62.858798
300 | -97.018955,44,62.881578
301 | -122.482675,96,62.931249
302 | -476.118654,258,63.064645
303 | -783.644057,433,63.288705
304 | -902.57989,501,64.265511
305 | -494.665819,238,64.388764
306 | -583.948686,468,64.630816
307 | -193.997325,308,64.793686
308 | -996.512255,501,65.052387
309 | -1231.665369,501,66.035455
310 | -363.617374,268,66.174089
311 | -445.838303,223,66.289533
312 | -729.60199,501,66.548133
313 | -473.51601,462,66.78628
314 | -30.895458,22,66.797897
315 | -111.852937,89,66.84375
316 | -558.064493,390,67.774793
317 | -21.871277,17,67.783736
318 | -264.608603,224,67.899299
319 | -655.66551,501,68.157769
320 | -207.217712,106,68.213306
321 | -165.940232,136,68.283434
322 | -143.161632,116,68.343706
323 | -159.489388,164,68.4337
324 | -473.571298,254,68.564858
325 | -142.008557,83,68.607805
326 | -167.248194,106,69.385032
327 | -539.720511,501,69.644215
328 | -40.995992,34,69.662717
329 | -332.566056,262,69.802239
330 | -799.222735,420,70.018402
331 | -106.354926,81,70.060902
332 | -14.220241,13,70.067844
333 | -79.698876,80,70.108954
334 | -84.838758,74,70.147464
335 | -315.054589,183,70.241472
336 | -163.881898,117,70.301802
337 | -483.657473,348,71.210978
338 | -605.93902,470,71.454153
339 | -367.485854,342,71.629674
340 | -179.130894,121,71.692191
341 | -828.842084,501,71.957032
342 | -988.919924,501,72.220826
343 | -542.026966,419,73.167031
344 | -457.748008,319,73.332162
345 | -284.786987,200,73.435587
346 | -519.097219,294,73.587529
347 | -713.859824,496,73.843223
348 | -56.269814,34,73.861081
349 | -123.790402,103,73.914311
350 | -87.996351,103,73.9679
351 | -385.043821,249,74.820072
352 | -657.911323,457,75.057352
353 | -200.138519,124,75.121424
354 | -643.586353,501,75.379185
355 | -68.547897,63,75.411848
356 | -217.525292,126,75.477393
357 | -564.838947,501,75.736238
358 | -97.21376,85,75.780316
359 | -886.64754,439,76.73552
360 | -351.442165,282,76.880911
361 | -133.061046,78,76.921094
362 | -41.755196,31,76.937195
363 | -95.475104,83,76.980063
364 | -139.162686,102,77.032542
365 | -15.608,10,77.037832
366 | -442.081879,407,77.247784
367 | -512.047465,341,77.424088
368 | -751.501455,501,78.432001
369 | -573.584814,367,78.621646
370 | -115.236664,62,78.653747
371 | -648.749659,501,78.912568
372 | -179.929357,89,78.958783
373 | -70.709235,44,78.982129
374 | -36.811894,25,78.99518
375 | -95.76711,64,79.028412
376 | -203.878792,134,79.09777
377 | -88.419949,72,79.134931
378 | -264.77556,121,79.197884
379 | -530.869704,330,79.369119
380 | -60.840846,35,79.387333
381 | -238.134076,160,80.203963
382 | -75.772265,53,80.23132
383 | -12.094617,9,80.236113
384 | -334.654188,246,80.362997
385 | -410.236705,288,80.510792
386 | -397.070975,314,80.673282
387 | -128.643713,90,80.720599
388 | -101.852827,85,80.764327
389 | -73.534807,56,80.793203
390 | -776.267783,353,80.978849
391 | -173.057712,106,81.0337
392 | -65.643663,47,81.058004
393 | -500.696606,286,81.206163
394 | -275.017173,341,82.113731
395 | -242.561784,188,82.211193
396 | -770.569878,501,82.478559
397 | -140.289288,135,82.548226
398 | -571.57828,501,82.806174
399 | -313.211247,227,82.922984
400 | -355.094128,220,83.957002
401 | -563.463271,410,84.2157
402 | -11.916761,10,84.22705
403 | -109.314511,99,84.278551
404 | -248.962878,186,84.374854
405 | -411.976699,319,84.54087
406 | -125.023701,90,84.587692
407 | -753.489905,501,84.847509
408 | -326.282063,229,84.965594
409 | -164.118769,84,85.009422
410 | -152.589359,107,85.803168
411 | -17.788048,11,85.809021
412 | -51.933488,40,85.829858
413 | -281.263543,180,85.923587
414 | -422.514302,279,86.068558
415 | -104.951833,84,86.112193
416 | -89.88128,87,86.157333
417 | -276.453867,205,86.264256
418 | -196.749055,145,86.339292
419 | -1128.657978,501,86.599219
420 | -49.378145,24,86.611773
421 | -520.488802,389,86.812664
422 | -321.047723,209,87.739039
423 | -362.532505,255,87.879349
424 | -286.463145,249,88.008787
425 | -578.860104,501,88.268366
426 | -589.305299,501,88.528816
427 | -106.932005,62,88.561038
428 | -365.061244,226,88.677537
429 |
--------------------------------------------------------------------------------
/02-DeepRL/Gaze_Based_Interaction/output/policy.zip:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/02-DeepRL/Gaze_Based_Interaction/output/policy.zip
--------------------------------------------------------------------------------
/02-DeepRL/bayesian_state_estimation.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "attachments": {},
5 | "cell_type": "markdown",
6 | "id": "77bab68c",
7 | "metadata": {},
8 | "source": [
9 | "[](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/02-DeepRL/bayesian_state_estimation.ipynb)\n"
10 | ]
11 | },
12 | {
13 | "cell_type": "markdown",
14 | "id": "15dc1fb8",
15 | "metadata": {},
16 | "source": [
17 | "# Bayesian estimation\n",
18 | "\n",
19 | "Andrew Howes\\\n",
20 | "School of Computer Science\\\n",
21 | "University of Birmingham\n",
22 | "\n",
23 | "What do people do with a sequence of observations (visual or otherwise)? Should they just use the most recent observation? Or perhaps the \"best\" observation? In fact, people do neither. Multiple sources of evidence suggests that people optimally integrate observations to generate a posterior estimate given all of the available information. One way to model this process is with Bayesian inference.\n",
24 | "\n",
25 | "This notebook gives one simple worked example of Bayesian inference for a human estimating the visual location of a target from a sequence of fixations at a fixed location. \n"
26 | ]
27 | },
28 | {
29 | "cell_type": "markdown",
30 | "id": "29ad7b7e",
31 | "metadata": {},
32 | "source": [
33 | "Given a prior location distribution $z_1$ with uncertainty $\\sigma^2_{z1}$, the user makes a visual observation $z_2$ with uncertainty $\\sigma^2_{z2}$. \n",
34 | "\n",
35 | "The user combines their prior and observation optimally using Bayesian estimation.\n",
36 | "\n",
37 | "The best estimate, given the prior and observation, is $\\mu$ with an associated error variance $ \\sigma^2$ as defined below. \n",
38 | "\n",
39 | "$$ \\mu =[\\sigma^2_{z_2}/(\\sigma^2_{z_1}+\\sigma^2_{z_2})] z_1 +[\\sigma^2_{z_1}/(\\sigma^2_{z_1}+\\sigma^2_{z_2})] z_2 $$\n",
40 | "\n",
41 | "\n",
42 | "$$1/ \\sigma^2=1/ \\sigma^2_{z_1}+1/ \\sigma^2_{z_2} $$"
43 | ]
44 | },
45 | {
46 | "cell_type": "markdown",
47 | "id": "cbf6a373",
48 | "metadata": {},
49 | "source": [
50 | "$\\sigma$ is less than either $\\sigma_{z_1}$ or $\\sigma_{z_2}$ , which is to say that the uncertainty in the user's estimate of location has been decreased by combining the two pieces of information (the prior and the observation). \n",
51 | "\n",
52 | "If $\\sigma_{z_1}$ were equal to $\\sigma_{z_2}$, which is to say that the prior and observation are of equal precision, then the equation says the optimal estimate of position is simply the average of the two measurements, as would be expected. On the other hand, if $\\sigma_{z_1}$ were larger than $\\sigma_{z_2}$, which is to say that the uncertainty in the prior $z_1$ is greater than that of the observation $z_2$ , then the equation dictates “weighting” $z2$ more heavily than $z1$. Finally, the variance of the estimate is less than $\\sigma_{z_1}$ , even if $\\sigma_{z_2}$ is very large: even poor quality data provide some information, and should thus increase the precision of the user's estimate."
53 | ]
54 | },
55 | {
56 | "cell_type": "markdown",
57 | "id": "40464312",
58 | "metadata": {},
59 | "source": [
60 | "The above equations can be reformulated.\n",
61 | "\n",
62 | "We have a Guassian prior $p(x)$, and a noisy observation $o$.\n",
63 | "\n",
64 | "The optimal location estimate $\\hat{X}$, that is the maximum of the posterior is:\n",
65 | "\n",
66 | "$$\\hat{X}=\\alpha o +(1- \\alpha) \\hat{\\mu}$$\n",
67 | "\n",
68 | "Where,\n",
69 | "\n",
70 | "$$\\alpha=\\dfrac{\\sigma^2_{p}}{\\sigma^2_{p}+\\sigma^2_{o}}$$\n"
71 | ]
72 | },
73 | {
74 | "cell_type": "code",
75 | "execution_count": null,
76 | "id": "nuclear-rachel",
77 | "metadata": {},
78 | "outputs": [],
79 | "source": [
80 | "# This cell and the following are only if you are running on Google Colab.\n",
81 | "\n",
82 | "from google.colab import drive\n",
83 | "drive.mount('/content/drive')"
84 | ]
85 | },
86 | {
87 | "cell_type": "code",
88 | "execution_count": null,
89 | "id": "cac37a84",
90 | "metadata": {},
91 | "outputs": [],
92 | "source": [
93 | "import matplotlib.pyplot as plt\n",
94 | "#%matplotlib inline\n",
95 | "import matplotlib as mpl\n",
96 | "\n",
97 | "import scipy.stats\n",
98 | "import numpy as np\n",
99 | "\n",
100 | "mpl.style.use('fivethirtyeight')\n",
101 | "\n",
102 | "\n",
103 | "def combine_two_guassian(m1,sigma1,m2,sigma2):\n",
104 | " '''\n",
105 | " Optimally combine two gaussians\n",
106 | " Return combine mean and std\n",
107 | " '''\n",
108 | " w1=sigma2**2/(sigma1**2+sigma2**2)\n",
109 | " w2=sigma1**2/(sigma1**2+sigma2**2)\n",
110 | "\n",
111 | " m=w1*m1+w2*m2\n",
112 | " sigma=np.sqrt( (sigma1**2 * sigma2**2)/(sigma1**2 + sigma2**2))\n",
113 | "\n",
114 | " return m,sigma\n",
115 | "\n",
116 | "def plot_gaussian(mean,sigma,fmt,label):\n",
117 | " '''\n",
118 | " plot the guassian pdf\n",
119 | " '''\n",
120 | " x_min = mean-3*sigma\n",
121 | " x_max = mean+3*sigma\n",
122 | " x = np.linspace(x_min, x_max, 100)\n",
123 | " y = scipy.stats.norm.pdf(x,mean,sigma)\n",
124 | " plt.xlim(-1,80)\n",
125 | " plt.ylim(0,0.06)\n",
126 | " plt.plot(x,y,fmt,label=label)\n"
127 | ]
128 | },
129 | {
130 | "cell_type": "code",
131 | "execution_count": null,
132 | "id": "f7f06dae",
133 | "metadata": {},
134 | "outputs": [],
135 | "source": [
136 | "fixation=0\n",
137 | "target=50\n",
138 | "m1,sigma1=40,20\n",
139 | "m2,sigma2=47,10\n",
140 | "\n",
141 | "plt.figure(figsize=(10,6))\n",
142 | "# obs 1\n",
143 | "\n",
144 | "plot_gaussian(m1,sigma1,'g:','prior')\n",
145 | "\n",
146 | "# plot the target line and fixation line\n",
147 | "plt.axvline(x = target, color = 'b', label = 'target')\n",
148 | "plt.axvline(x = fixation, color = 'k')\n",
149 | "\n",
150 | "plt.legend(loc='best')\n",
151 | "plt.xlabel('Eccentricity')\n"
152 | ]
153 | },
154 | {
155 | "cell_type": "code",
156 | "execution_count": null,
157 | "id": "c372d75b",
158 | "metadata": {},
159 | "outputs": [],
160 | "source": [
161 | "\n",
162 | "\n",
163 | "plt.figure(figsize=(10,6))\n",
164 | "# obs 1\n",
165 | "\n",
166 | "plot_gaussian(m1,sigma1,'g:','prior')\n",
167 | "\n",
168 | "# obs 2\n",
169 | "\n",
170 | "plot_gaussian(m2,sigma2,'r--','observation 1')\n",
171 | "\n",
172 | "\n",
173 | "# plot the target line and fixation line\n",
174 | "plt.axvline(x = target, color = 'b', label = 'target')\n",
175 | "plt.axvline(x = fixation, color = 'k')\n",
176 | "\n",
177 | "plt.legend(loc='best')\n",
178 | "plt.xlabel('Eccentricity')\n",
179 | "\n"
180 | ]
181 | },
182 | {
183 | "cell_type": "code",
184 | "execution_count": null,
185 | "id": "08f31602",
186 | "metadata": {},
187 | "outputs": [],
188 | "source": [
189 | "fixation=0\n",
190 | "target=50\n",
191 | "\n",
192 | "plt.figure(figsize=(10,6))\n",
193 | "# obs 1\n",
194 | "\n",
195 | "plot_gaussian(m1,sigma1,'g:','prior')\n",
196 | "\n",
197 | "# obs 2\n",
198 | "\n",
199 | "plot_gaussian(m2,sigma2,'r--','observation 1')\n",
200 | "\n",
201 | "# combine obs1 and obs2\n",
202 | "m3,sigma3=combine_two_guassian(m1,sigma1,m2,sigma2)\n",
203 | "plot_gaussian(m3,sigma3,'y-','posterior')\n",
204 | "\n",
205 | "# plot the target line and fixation line\n",
206 | "plt.axvline(x = target, color = 'b', label = 'target')\n",
207 | "plt.axvline(x = fixation, color = 'k')\n",
208 | "\n",
209 | "plt.legend(loc='best')\n",
210 | "plt.xlabel('Eccentricity')\n",
211 | "\n",
212 | "print(m3)\n",
213 | "print(sigma3)\n"
214 | ]
215 | },
216 | {
217 | "cell_type": "markdown",
218 | "id": "6e11290c",
219 | "metadata": {},
220 | "source": [
221 | "## Exercises\n",
222 | "\n",
223 | "1. Try different values of the mean and variance of the distributions. Satisfy yourself that the posterior estimate is always more accurate than the prior and the observation.\n",
224 | "\n",
225 | "2. Given a prior with standard deviation of 20 and mean 40, imagine three observations each with different standard deviation (5,10,15) but the same location (70). Illustrate the effect of each observation on the posterior.\n",
226 | "\n",
227 | "Advanced\n",
228 | "\n",
229 | "3. Write a function that takes as input a sequences of noisy observations, possibly from foveated vision(!), of arbitrary length and generates a posterior estimate of the target location. This will be your first simple model of human vision.\n"
230 | ]
231 | },
232 | {
233 | "cell_type": "code",
234 | "execution_count": null,
235 | "id": "b88774f7",
236 | "metadata": {},
237 | "outputs": [],
238 | "source": []
239 | }
240 | ],
241 | "metadata": {
242 | "kernelspec": {
243 | "display_name": "Python 3",
244 | "language": "python",
245 | "name": "python3"
246 | },
247 | "language_info": {
248 | "codemirror_mode": {
249 | "name": "ipython",
250 | "version": 3
251 | },
252 | "file_extension": ".py",
253 | "mimetype": "text/x-python",
254 | "name": "python",
255 | "nbconvert_exporter": "python",
256 | "pygments_lexer": "ipython3",
257 | "version": "3.8.8"
258 | }
259 | },
260 | "nbformat": 4,
261 | "nbformat_minor": 5
262 | }
263 |
--------------------------------------------------------------------------------
/02-DeepRL/foveated_vision.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "attachments": {},
5 | "cell_type": "markdown",
6 | "id": "2847b331",
7 | "metadata": {},
8 | "source": [
9 | "[](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/02-DeepRL/foveated_vision.ipynb)\n"
10 | ]
11 | },
12 | {
13 | "attachments": {},
14 | "cell_type": "markdown",
15 | "id": "recovered-bottle",
16 | "metadata": {},
17 | "source": [
18 | "# Foveated vision\n",
19 | "\n",
20 | "Andrew Howes\n",
21 | "\n",
22 | "This notebook illustrates the effect of retinal eccentricity on spatial resolution and, therefore, on visual acuity in the human eye.\n",
23 | "\n",
24 | "
\n",
25 | "\n",
26 | "(source: Geisler, W. S. (2011). Contributions of ideal observer theory to vision research. Vision research, 51(7), 771-781.)\n",
27 | "\n",
28 | "The retina is the layer of photoreceptors at the back of the eye that captures photons and transmits information to the brain. The fovea is a small depression in the middle of the retina with a particularly high density of photoreceptors. It is where visual acuity is highest. People use eye movements to bring the fovea to bear on locations about which they require more information.\n",
29 | "\n",
30 | "As retinal eccentricity from the fovea increases, visual acuity falls exponentially. By just 2.5 degrees of retinal eccentricity, acuity has fallen by 50%.\n",
31 | "\n",
32 | "As a consequence, our ability to estimate the location of an item on a display decreases exponentially with eccentricity from the fovea. The less accurate a target location estimate then the more diffcult it is to move the eyes to the target.\n",
33 | "\n",
34 | "In order to model this 'bound' on cognition, it is assumed that noise in location estimates increases with eccentricity and that the noise is Gaussian distributed.\n",
35 | "\n",
36 | "In order to start modeling let us first define some parameters."
37 | ]
38 | },
39 | {
40 | "cell_type": "code",
41 | "execution_count": null,
42 | "id": "homeless-young",
43 | "metadata": {},
44 | "outputs": [],
45 | "source": [
46 | "# fixed_noise is the noise at the fovea.\n",
47 | "fixed_noise = 2\n",
48 | "\n",
49 | "# the noise parameter determines how much noise increases with eccentricity\n",
50 | "noise_parameter = 0.25"
51 | ]
52 | },
53 | {
54 | "attachments": {},
55 | "cell_type": "markdown",
56 | "id": "0afe8d0b",
57 | "metadata": {},
58 | "source": [
59 | "### Import libraries\n",
60 | "\n",
61 | "Next we can import standard libraries for maths, statistics and plotting. "
62 | ]
63 | },
64 | {
65 | "cell_type": "code",
66 | "execution_count": null,
67 | "id": "growing-alarm",
68 | "metadata": {},
69 | "outputs": [],
70 | "source": [
71 | "import matplotlib.pyplot as plt\n",
72 | "import matplotlib as mpl\n",
73 | "import scipy.stats\n",
74 | "import numpy as np\n",
75 | "\n",
76 | "# set a style for the plots.\n",
77 | "\n",
78 | "mpl.style.use('fivethirtyeight')"
79 | ]
80 | },
81 | {
82 | "attachments": {},
83 | "cell_type": "markdown",
84 | "id": "f04c5865",
85 | "metadata": {},
86 | "source": [
87 | "### Plot\n",
88 | "\n",
89 | "Now we define a function that plots a Gaussian distribution. \n",
90 | "\n",
91 | "Below, we will use this function to represent the distribution of probable target locations given a noisy observation."
92 | ]
93 | },
94 | {
95 | "cell_type": "code",
96 | "execution_count": null,
97 | "id": "78d162ae",
98 | "metadata": {},
99 | "outputs": [],
100 | "source": [
101 | "def plot_gaussian(mean,sigma,fmt,label):\n",
102 | " # plot a Gaussian distributed at 'mean' with standard deviation sigma.\n",
103 | " # fmt provides a string of line parameters (colour etc.) and 'label' is a label for the plotted line.\n",
104 | " x_min = mean-3*sigma\n",
105 | " x_max = mean+3*sigma\n",
106 | " x = np.linspace(x_min, x_max, 100)\n",
107 | " y = scipy.stats.norm.pdf(x,mean,abs(sigma))\n",
108 | " plt.xlim(-80,80)\n",
109 | " plt.ylim(0,0.2)\n",
110 | " plt.plot(x,y,fmt,label=label)\n"
111 | ]
112 | },
113 | {
114 | "cell_type": "code",
115 | "execution_count": null,
116 | "id": "d1119022",
117 | "metadata": {},
118 | "outputs": [],
119 | "source": [
120 | "\n",
121 | "eccentricity = np.arange(-60,70,20)\n",
122 | "\n",
123 | "plt.figure(figsize=(10,6))\n",
124 | "\n",
125 | "for i in eccentricity:\n",
126 | " plot_gaussian(i, fixed_noise+abs(noise_parameter*i), 'g:',f'{i}')\n",
127 | "\n",
128 | "x = plt.xlabel('Eccentricity')"
129 | ]
130 | },
131 | {
132 | "attachments": {},
133 | "cell_type": "markdown",
134 | "id": "portable-accommodation",
135 | "metadata": {},
136 | "source": [
137 | "In the figure above it is assumed that the fovea (the gaze location) is at 0 eccentricity. Each distribution then represents the perceived target location probability given that the actual target location is at the centre of the distribution. \n",
138 | "\n",
139 | "So, for example, if the target is at eccentricity 60 then the probability of perceiving it at 60 is about 0.025, whereas if the target is at 20 then the probability of perceiving it at 20 is over 0.05.\n",
140 | "\n",
141 | "Parameter values do not represent the actual human acuity function and are for illustration only."
142 | ]
143 | },
144 | {
145 | "attachments": {},
146 | "cell_type": "markdown",
147 | "id": "metallic-gather",
148 | "metadata": {},
149 | "source": [
150 | "### Exercise\n",
151 | "\n",
152 | "- Build a Python model of human vision which returns the (stochastic) perceived location of a target given the true location.\n",
153 | "\n",
154 | "- Hint: \n",
155 | " 1. Use the numpy function random.uniform() to generate a target location between some lower and upper bound of eccentricity. \n",
156 | " 2. Use the eccentricity and the function random.normal() to generate an estimate of the target location.\n",
157 | "\n",
158 | "### Advanced exercises\n",
159 | "\n",
160 | "- Assume that the eyes are moved to the perceived location and a new observation is made of the target (which has not moved). Show, through simulated trials, how the error in the perceived location reduces as each successive observation is made.\n",
161 | "\n",
162 | "- Explain why the accuracy increases."
163 | ]
164 | }
165 | ],
166 | "metadata": {
167 | "kernelspec": {
168 | "display_name": "Python 3",
169 | "language": "python",
170 | "name": "python3"
171 | },
172 | "language_info": {
173 | "codemirror_mode": {
174 | "name": "ipython",
175 | "version": 3
176 | },
177 | "file_extension": ".py",
178 | "mimetype": "text/x-python",
179 | "name": "python",
180 | "nbconvert_exporter": "python",
181 | "pygments_lexer": "ipython3",
182 | "version": "3.8.8"
183 | }
184 | },
185 | "nbformat": 4,
186 | "nbformat_minor": 5
187 | }
188 |
--------------------------------------------------------------------------------
/03-Modelbuilding/Go_Nogo.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "attachments": {},
5 | "cell_type": "markdown",
6 | "id": "3e0c5f5d",
7 | "metadata": {},
8 | "source": [
9 | "[](https://colab.research.google.com/github/jussippjokinen/CogMod-Tutorial/blob/main/03-Modelbuilding/Go_Nogo.ipynb)\n"
10 | ]
11 | },
12 | {
13 | "cell_type": "code",
14 | "execution_count": 1,
15 | "id": "20e1ba45",
16 | "metadata": {},
17 | "outputs": [],
18 | "source": [
19 | "# Install the files to local.\n",
20 | "# Please note that the go / no go model we are using here is still being developed,\n",
21 | "# and should not yet be adapted for other work! The development is in close collaboration\n",
22 | "# with the Commotions project at Leeds, led by Gustav Markkula, and collaborated with Aravinda Srinivasan.\n",
23 | "! wget https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/main/03-Modelbuilding/animate_trace.py\n",
24 | "! wget https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/main/03-Modelbuilding/driver_agent_physics.py\n",
25 | "! wget https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/main/03-Modelbuilding/go_nogo.py\n",
26 | "! wget https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/main/03-Modelbuilding/physics_env.py"
27 | ]
28 | },
29 | {
30 | "cell_type": "code",
31 | "execution_count": null,
32 | "id": "02706422",
33 | "metadata": {},
34 | "outputs": [],
35 | "source": [
36 | "# Install the required library.\n",
37 | "! pip install stable_baselines3"
38 | ]
39 | },
40 | {
41 | "cell_type": "markdown",
42 | "id": "50888742",
43 | "metadata": {},
44 | "source": [
45 | "# Module 3: Building a Model\n",
46 | "\n",
47 | "In this module, we take a step-by-step walktrough of how to create a computational rational (CR) model using deep reinforcement learning (RL). This notebook does not cover the full workflow of CR modeling, which is long and detailed. It can be found here, make sure to follow it when creating your own models. For the purpose of this notebook, the simplified workflow looks like this:\n",
48 | "\n",
49 | "1. Define the goals.\n",
50 | "\n",
51 | "2. Define the environment.\n",
52 | "\n",
53 | "3. Define the cognitive limitations.\n",
54 | "\n",
55 | "4. Derive the optimal behavior.\n",
56 | "\n",
57 | "5. Inspect model validity.\n",
58 | "\n",
59 | "The model will be defined according to the standard CR flow of information.\n",
60 | "\n",
61 | "
\n"
62 | ]
63 | },
64 | {
65 | "cell_type": "markdown",
66 | "id": "ac368997",
67 | "metadata": {},
68 | "source": [
69 | "# 1. Define the agent's goals\n",
70 | "\n",
71 | "The task explored in this notebook is a fairly simple one: in a junction, when turning against the oncoming traffic, the driver needs to decide if they can go, or if they need to wait for an oncoming car before they can cross. In left-handed traffic, this means that the agent driver is turning right. Here is the illustration:\n",
72 | "\n",
73 | "
\n",
74 | "\n",
75 | "The agent (the yellow car) has two main goals:\n",
76 | "\n",
77 | "1. **Proceed** to the destination by turning left.\n",
78 | "\n",
79 | "2. Drive **safely**, avoiding collisions.\n",
80 | "\n",
81 | "First, we need to analyze these goals. First, the agent wants to drive efficiently and not get stuck on the road for too long. They probably want to get to their destination, and also they would be blocking the traffic behind them if they wait unreasonably long. So we can analyse these goals into a reward function.\n",
82 | "\n",
83 | "1. When the agent **turns successfully**, there is a positive reward.\n",
84 | "2. For this positive reward, there is a penalty of **time spent** waiting.\n",
85 | "3. For a **collision**, there is a negative reward."
86 | ]
87 | },
88 | {
89 | "cell_type": "markdown",
90 | "id": "03fe18e6",
91 | "metadata": {},
92 | "source": [
93 | "**Task 1**. Discuss alternative goals that the driver may have. Can you specify them in terms of the scalar reward function?"
94 | ]
95 | },
96 | {
97 | "cell_type": "markdown",
98 | "id": "3a685485",
99 | "metadata": {},
100 | "source": [
101 | "# 2. Define the task environment\n",
102 | "\n",
103 | "For simplicity, the environment has only two cars, in a 2-dimensional environment: the agent's car and the oncoming car. The cars are particles that have their individual `(x,y)` coordinates. The agent's car is stationary until the agent decides to turn left. The oncoming car has a variable position, which moves along the y-axis of the environment according to its velocity, which is fixed constant. A collision is detected if the vehicles are too close to each other regardless of their velocities."
104 | ]
105 | },
106 | {
107 | "cell_type": "code",
108 | "execution_count": null,
109 | "id": "c7e8921b",
110 | "metadata": {},
111 | "outputs": [],
112 | "source": [
113 | "# Let's define an environment and an agent and see how it works.\n",
114 | "import physics_env\n",
115 | "import driver_agent_physics\n",
116 | "\n",
117 | "e = physics_env.physics_env()\n",
118 | "agent = driver_agent_physics.driver_agent_physics(e, observation_var = 0) # a full observer: no noise\n",
119 | "agent.reset()"
120 | ]
121 | },
122 | {
123 | "cell_type": "code",
124 | "execution_count": null,
125 | "id": "24a692d5",
126 | "metadata": {},
127 | "outputs": [],
128 | "source": [
129 | "# Let's wait for 20 ticks, then go.\n",
130 | "from IPython.display import HTML, display\n",
131 | "import animate_trace\n",
132 | "\n",
133 | "agent.reset(y_start = -30)\n",
134 | "agent.env.save_trace = True\n",
135 | "for i in range(20):\n",
136 | " agent.step(0)\n",
137 | "agent.step(1)\n",
138 | "agent.env.save_trace = False\n",
139 | "\n",
140 | "HTML(animate_trace.animate_trace(agent.env.trace, get_anim = True, x_lim = [-50,50], y_lim = [10,-80]).to_jshtml())"
141 | ]
142 | },
143 | {
144 | "cell_type": "markdown",
145 | "id": "ff573d60",
146 | "metadata": {},
147 | "source": [
148 | "**Task 2.** Change the starting distance of the incoming car. Try to collide the cars! (Note that the y-coordinate needs to be negative.)"
149 | ]
150 | },
151 | {
152 | "cell_type": "markdown",
153 | "id": "3ed88fdb",
154 | "metadata": {},
155 | "source": [
156 | "# 3. Define the relevant cognitive bounds of the agent\n",
157 | "\n",
158 | "The agent must make a decision to go (turn left) or wait. It bases this decision on a noisy observation of the oncoming car's distance to the agent's car. If the distance is long enough, the agent can go and save time. If the car is too close, the agent must wait for it to pass to avoid a collision.\n",
159 | "\n",
160 | "For modeling noisy observation, we will be using the formula from Markkula, et al. (2022). Explaining human interactions on the road requires large-scale integration of psychological theory. https://psyarxiv.com/hdxbs/\n",
161 | "\n",
162 | "$ \\hat{D} = D_{oth} \\cdot \\left(1 - \\frac{h}{D \\cdot \\tan \\left(\\arctan \\left(\\frac{h}{D}\\right) + \\sigma \\right)}\\right) $,\n",
163 | "\n",
164 | "where $ \\hat{D} $ is a noisy observation of distance, $D_{oth}$ is the oncoming car's longitudinal distance to the crossing point, and $h$ is the observer's eye height over ground (1.5m). The important parameter here is $\\sigma$, which describes how noisy the observation is.\n",
165 | "\n",
166 | "The noisy observation is not used directly, but via Bayesian filtering. This considers prior belief about the distance and integrates the new observation with it. Additionally, we represent the uncertainty associated with the belief.\n",
167 | "\n",
168 | "Before deriving the optimal behavior, we want to establish what hypotheses our model in fact makes, so that we can then assess the plausibility of model predictions against them. While modeling is virtually always interactive and the hypotheses might develop during iteration, but it is important to have a strong initial assumption about how our model will behave. Here are initial proposals, which would then be turned into exact, testable hypotheses:\n",
169 | "\n",
170 | "P1. For a non-noisy (full) observer, there is a precise distance: if the other car is farther than this distance, the agent decides to go. If the other car is closer, the agent chooses to wait. If the car has passed, the agent always goes.\n",
171 | "\n",
172 | "P2. For a noisy (partial) observer, the decision to go is more conservative and uncertain: the distance that determines the go / no go decision is larger, and due to noisy estimates, the choice is probabilistic given distance."
173 | ]
174 | },
175 | {
176 | "cell_type": "code",
177 | "execution_count": null,
178 | "id": "b960cbd8",
179 | "metadata": {},
180 | "outputs": [],
181 | "source": [
182 | "# Here is a theoretical illustration of our proposals.\n",
183 | "import numpy as np\n",
184 | "import matplotlib.pyplot as plt\n",
185 | "import seaborn as sns\n",
186 | "sns.set_style('whitegrid')\n",
187 | "x1 = np.linspace(-1, 0, 100)\n",
188 | "x2 = np.linspace(-1, 0, 100)\n",
189 | "y1 = np.zeros(100)\n",
190 | "y1[50:] = 1\n",
191 | "y2 = 1 / (1 + np.exp(-10*(x2+0.5)))\n",
192 | "plt.plot(x1, y1, label='Full Observer', linewidth = 2)\n",
193 | "plt.plot(x2, y2, label='Noisy Observer', linewidth = 2)\n",
194 | "plt.ylim(0, 1.01)\n",
195 | "plt.xlabel('Distance', fontsize=14)\n",
196 | "plt.ylabel('Probability of wait', fontsize=14)\n",
197 | "plt.legend(fontsize=14)\n",
198 | "plt.show()"
199 | ]
200 | },
201 | {
202 | "cell_type": "markdown",
203 | "id": "1eebb341",
204 | "metadata": {},
205 | "source": [
206 | "**Task 3.** Discuss what the hypothesis actually means. What do the lines tell us?"
207 | ]
208 | },
209 | {
210 | "cell_type": "markdown",
211 | "id": "8610046c",
212 | "metadata": {},
213 | "source": [
214 | "## How does the noisy observation happen?\n",
215 | "\n",
216 | "The observation of the approaching vehicle distance is noisy. In practice, we simulate noisy observation by taking the true observation and then adding noise from a normal distribution, according to some parameter sigma. However, instead of using the returned noisy value as such, we assume that the human visual system exploits the fact that each noisy value is based on the true value, and the noise comes from a known distribution. We can then use a Kalman filter to approximate the true state. With more observations, the approximation becomes better. In our simulation, this is confounded by the formula above, which states that observations get better as the distance shortens. Let's investigate this behavior."
217 | ]
218 | },
219 | {
220 | "cell_type": "code",
221 | "execution_count": null,
222 | "id": "994429e4",
223 | "metadata": {},
224 | "outputs": [],
225 | "source": [
226 | "import math\n",
227 | "import numpy as np\n",
228 | "\n",
229 | "# Here is the formula for making an observation, given distance.\n",
230 | "def noisy_observation(D):\n",
231 | " # D = distance in meters\n",
232 | " d_oth = D+2 # crossing point distance, this is a crude approximation to simplify our exercise\n",
233 | " h = 1.5 # eye height\n",
234 | " observation_var = 0.1 # this is the sigma\n",
235 | " distance_var = d_oth * (1 - h / (D*math.tan(math.atan(h/D) + observation_var)))\n",
236 | " observed_distance = np.random.normal(D, distance_var)\n",
237 | " return observed_distance, distance_var\n",
238 | "\n",
239 | "\n",
240 | "# Let's see what kinds of observations we make from the true distance of 50 m\n",
241 | "for i in range(10):\n",
242 | " print(noisy_observation(50)[0])"
243 | ]
244 | },
245 | {
246 | "cell_type": "code",
247 | "execution_count": null,
248 | "id": "9db93f30",
249 | "metadata": {},
250 | "outputs": [],
251 | "source": [
252 | "# So, that's not a very reliable observation! Here is the Kalman filter, let's see how it helps.\n",
253 | "def kalman_update(prior_mean, prior_var, observation, s):\n",
254 | " observation_gain = 1\n",
255 | " kalman_gain = prior_var * observation_gain / (prior_var * observation_gain**2 + s**2)\n",
256 | " posterior_mean = prior_mean + kalman_gain * (observation - prior_mean)\n",
257 | " posterior_var = (1 - kalman_gain * observation_gain) * prior_var\n",
258 | "\n",
259 | " return posterior_mean, posterior_var\n",
260 | " \n",
261 | "# Then, let's update both mean and variance estimates multiple times\n",
262 | "distance_prior = 100 # set an uninformed prior\n",
263 | "distance_var_prior = 1000 # basically a uniform prior\n",
264 | "for i in range(20):\n",
265 | " d, s = noisy_observation(50)\n",
266 | " distance_post, distance_var_post = kalman_update(distance_prior, distance_var_prior, d, s)\n",
267 | " # Store result\n",
268 | " print(distance_post)\n",
269 | " # Set the posterior as the new prior\n",
270 | " distance_prior = distance_post\n",
271 | " distance_var_prior = distance_var_post"
272 | ]
273 | },
274 | {
275 | "cell_type": "code",
276 | "execution_count": null,
277 | "id": "47023b7a",
278 | "metadata": {},
279 | "outputs": [],
280 | "source": [
281 | "# Can we visualize it? Let's assume a constant velocity of 0.5 m per observational \"tick\".\n",
282 | "# Assuming one tick is 0.05s, this is 10m/s = 36km/h\n",
283 | "import matplotlib.pyplot as plt\n",
284 | "import seaborn as sns\n",
285 | "\n",
286 | "true_d = 50\n",
287 | "distance_prior = 100 # set an uninformed prior\n",
288 | "distance_var_prior = 1000 # basically a uniform prior\n",
289 | "velocity = 0.5\n",
290 | "data = []\n",
291 | "for i in range(100):\n",
292 | " d, s = noisy_observation(true_d)\n",
293 | " distance_post, distance_var_post = kalman_update(distance_prior, distance_var_prior, d, s)\n",
294 | " # Store result\n",
295 | " data.append([true_d, d, distance_post])\n",
296 | " # Set the posterior as the new prior\n",
297 | " distance_prior = distance_post\n",
298 | " distance_var_prior = distance_var_post\n",
299 | " # \"Move\" the approaching vehicle\n",
300 | " true_d -= velocity\n",
301 | " \n",
302 | "# Make the plot\n",
303 | "true_distance, observed_distance, estimated_distance = zip(*data)\n",
304 | "index = np.array(range(len(data))) / 20 # make into seconds\n",
305 | "sns.set_style('whitegrid')\n",
306 | "plt.plot(index, true_distance, label='True Distance', linestyle='-', marker='o')\n",
307 | "plt.plot(index, observed_distance, label='Observed Distance', linestyle='-', marker='o')\n",
308 | "plt.plot(index, estimated_distance, label='Estimated Distance', linestyle='-', marker='o')\n",
309 | "\n",
310 | "plt.xlabel('Time (s)', fontsize = 14)\n",
311 | "plt.ylabel('Distance', fontsize = 14)\n",
312 | "plt.legend(loc='best', fontsize = 14)\n",
313 | "\n",
314 | "plt.show()"
315 | ]
316 | },
317 | {
318 | "cell_type": "markdown",
319 | "id": "0fda6479",
320 | "metadata": {},
321 | "source": [
322 | "**Task 3.1** Change the prior (distance and variance) and investigate how that changes the convergence of the posterior over time. You can also try changing the velocity and starting distance."
323 | ]
324 | },
325 | {
326 | "cell_type": "markdown",
327 | "id": "982fa065",
328 | "metadata": {},
329 | "source": [
330 | "# 4. Derive optimal policy\n",
331 | "\n",
332 | "For establishing the (bounded) optimal policy for the ideal and noisy observer agents, we will be using Proximal Policy Optimization, which is an on-policy deep RL algorithm. This notebook uses the OpenAI Stable Baselines implementation, but there are others as well. https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html\n",
333 | "\n",
334 | "Before we can derive the optimal policy, the environments (both internal and external) need to be modeled using Markov Decision Process (**MDP**, or in our case, due to partial observability, we are defining a **POMDP**). We won't go into the details of how the external driving environment is modeled, but it is a simple stepwise simulator, that for each time step (0.05 seconds by default), \"ticks\" the environment forward by moving the upcoming car's y-position according to its velocity, and in case the agent decides to go, moves it towards the side road along a predefined trajectory, in a constant velocity as well.\n",
335 | "\n",
336 | "For the internal environment, we need to model the belief update for distance of the oncoming car, given the equation and filtering that were defined above. All relevant information must be represented as the agent's belief such that we can pass that, along with the reward signal, to the RL agent for learning the optimal policy."
337 | ]
338 | },
339 | {
340 | "cell_type": "code",
341 | "execution_count": null,
342 | "id": "a3b9e221",
343 | "metadata": {},
344 | "outputs": [],
345 | "source": [
346 | "# Here are the action and observation spaces of the agent\n",
347 | "\n",
348 | "from gymnasium.spaces import Discrete, Dict, Box\n",
349 | "\n",
350 | "action_space = Discrete(2) \n",
351 | "\n",
352 | "# Note that all observatons are normalized between 0 and 1.\n",
353 | "observation_space = Dict(\n",
354 | " spaces = {\n",
355 | " \"distance\": Box(0, 1, (1,)),\n",
356 | " \"passed\": Box(0, 1, (1,)),\n",
357 | " \"distance_var\": Box(0, 1, (1,)),\n",
358 | " \"speed\": Box(0, 1, (1,)),\n",
359 | " \"acceleration\": Box(0, 1, (1,)),\n",
360 | " \"ticks\": Box(0, 1, (1,))\n",
361 | " })\n",
362 | "\n",
363 | "# Sample the action space:\n",
364 | "print(\"A small sample of actions:\", action_space.sample(), action_space.sample(), action_space.sample())\n",
365 | "print(\"One sample of observation space:\", observation_space.sample())"
366 | ]
367 | },
368 | {
369 | "cell_type": "markdown",
370 | "id": "fae9030e",
371 | "metadata": {},
372 | "source": [
373 | "So, the environment simulation works as intended. Let's now take a look at the reward function. Remember our definition:\n",
374 | "\n",
375 | "1. When the agent **turns left successfully**, there is a positive reward.\n",
376 | "2. For this positive reward, there is a penalty of **time spent** waiting.\n",
377 | "3. For a **collision**, there is a negative reward.\n",
378 | "\n",
379 | "We will establish the reward function along with one step of the model to see what actually happens when we step."
380 | ]
381 | },
382 | {
383 | "cell_type": "code",
384 | "execution_count": null,
385 | "id": "8227436a",
386 | "metadata": {},
387 | "outputs": [],
388 | "source": [
389 | "def step(self, action):\n",
390 | " self.reward = 0\n",
391 | " # action: no go\n",
392 | " if action == 0:\n",
393 | " self.env.tick()\n",
394 | " self.ticks += 1\n",
395 | " # break if nothing ever happens\n",
396 | " if self.ticks > self.max_ticks:\n",
397 | " self.reward = -10\n",
398 | " self.done = True\n",
399 | " if self.env.get_distance() > self.max_distance:\n",
400 | " self.reward = -10\n",
401 | " self.done = True \n",
402 | " # action: go\n",
403 | " if action == 1:\n",
404 | " # Did we wait for the other car before going?\n",
405 | " if self.env.veh2_turn_pos[1] < self.env.veh1_straight_pos[1]:\n",
406 | " self.waited_before_go = True\n",
407 | " self.distance_at_go = self.env.get_distance()\n",
408 | " self.done = True\n",
409 | " self.collision, _ = self.env.simulate_go()\n",
410 | " if self.collision:\n",
411 | " self.reward = -10\n",
412 | " else:\n",
413 | " self.reward = 10 - self.penalty_per_tick * self.ticks\n",
414 | "\n",
415 | " self.belief = self.get_belief()\n",
416 | "\n",
417 | " return self.belief, self.reward, self.done, False, {}"
418 | ]
419 | },
420 | {
421 | "cell_type": "code",
422 | "execution_count": null,
423 | "id": "8b439c41",
424 | "metadata": {},
425 | "outputs": [],
426 | "source": [
427 | "# Let's make and train the full observer agent.\n",
428 | "import go_nogo\n",
429 | "full_obs_agent = go_nogo.make_agent(sigma = 0, iters = 10)\n",
430 | "# In the output:\n",
431 | "# i = training iteration\n",
432 | "# t = number of ticks (1 tick = 0.05s)\n",
433 | "# r = average reward (10 is max)\n",
434 | "# d = average distance of the two vehicles at the time of go (in meters)\n",
435 | "# w = frequency of waits (agent waited the other car to pass before going)\n",
436 | "# c = frequency of collisions"
437 | ]
438 | },
439 | {
440 | "cell_type": "code",
441 | "execution_count": null,
442 | "id": "6d409044",
443 | "metadata": {},
444 | "outputs": [],
445 | "source": [
446 | "# To keep training the agent without initializing it anew, use this.\n",
447 | "# Commented out so it won't run when running all cells\n",
448 | "# go_nogo.retrain_agent(full_obs_agent, iters = 10)"
449 | ]
450 | },
451 | {
452 | "cell_type": "code",
453 | "execution_count": null,
454 | "id": "11e89821",
455 | "metadata": {},
456 | "outputs": [],
457 | "source": [
458 | "# Let's investigate if it has found a policy for the different distances.\n",
459 | "# Keep an eye on the \"critical\" y_start at around -13.\n",
460 | "HTML(go_nogo.animate_agent(full_obs_agent, y_start = -13.8, get_anim = True).to_jshtml())"
461 | ]
462 | },
463 | {
464 | "cell_type": "code",
465 | "execution_count": null,
466 | "id": "e5572776",
467 | "metadata": {},
468 | "outputs": [],
469 | "source": [
470 | "# Train the noisy observer. Use more iters due to a more difficult learning task.\n",
471 | "noisy_obs_agent = go_nogo.make_agent(sigma = 0.1, iters = 15)"
472 | ]
473 | },
474 | {
475 | "cell_type": "code",
476 | "execution_count": null,
477 | "id": "48c8d8ac",
478 | "metadata": {},
479 | "outputs": [],
480 | "source": [
481 | "# To keep training the agent without initializing it anew, use this.\n",
482 | "# Commented out so it won't run when running all cells\n",
483 | "#go_nogo.retrain_agent(noisy_obs_agent, iters = 10)"
484 | ]
485 | },
486 | {
487 | "cell_type": "code",
488 | "execution_count": null,
489 | "id": "e081ef1d",
490 | "metadata": {},
491 | "outputs": [],
492 | "source": [
493 | "# Let's investigate if it has found a policy for the different distances.\n",
494 | "# Keep an eye on the \"critical\" y_start at around -13.\n",
495 | "HTML(go_nogo.animate_agent(noisy_obs_agent, y_start = -13.8, get_anim = True).to_jshtml())"
496 | ]
497 | },
498 | {
499 | "cell_type": "markdown",
500 | "id": "c8ed7578",
501 | "metadata": {},
502 | "source": [
503 | "**Task 4.** Try to find out the critical distance where the two models, full and partial/noisy observer, differ in their go/no go policy."
504 | ]
505 | },
506 | {
507 | "cell_type": "markdown",
508 | "id": "5b6401b0",
509 | "metadata": {},
510 | "source": [
511 | "# 5. Inspect model validity\n",
512 | "\n",
513 | "After having converged the model to an optimal policy, our aim is to utilize it for generating simulations of task behavior. The evaluation of the model's validity encompasses multiple stages, see the workflow.pdf draft for these. Here, we concentrate solely on its face validity, which addresses whether the model aligns with our initial predictions.\n",
514 | "\n",
515 | "Starting the model's validity assessment with face validity tests is a useful practice, as any discrepancies between the model's performance and our hypotheses at this stage may indicate issues with either the model's specification or our modeling assumptions. This is frequently an iterative procedure, during which we may observe the model's divergence from our expectations, resulting in identifying inadequate definitions of objectives, task environment, or cognitive constraints.\n",
516 | "\n",
517 | "Once the model successfully demonstrates face validity, it should be subjected to a rigorous validation process, wherein its predictions are compared against human data or some alternative benchmarks. The model should e.g., generate accurate summary statistics (across a broader human population), be capable of replicating individual performance by adjusting specific parameters, and operate reasonably under changes in the environment. The specific validation depends always on the details of the modeling work."
518 | ]
519 | },
520 | {
521 | "cell_type": "code",
522 | "execution_count": null,
523 | "id": "329c93d6",
524 | "metadata": {},
525 | "outputs": [],
526 | "source": [
527 | "# Run an experiment for obtaining multiple samples from each agent.\n",
528 | "def wait_or_go_experiment(agent, y_range, n = 100, deterministic = False):\n",
529 | " data = []\n",
530 | " agent.env.veh1_straight_start_y_range = y_range\n",
531 | " for i in range(n):\n",
532 | " _, _, _, w, c = agent.run_episode(deterministic = deterministic)\n",
533 | " data.append([agent.observation_var, agent.env.y_start, w, c])\n",
534 | " \n",
535 | " agent.env.veh1_straight_start_y_range = [-25,-2] # return back to original\n",
536 | " return data\n",
537 | "\n",
538 | "import pandas as pd\n",
539 | "\n",
540 | "# Increase n to e.g., 2000 to get more robust final image, but note that it takes longer to run.\n",
541 | "data = wait_or_go_experiment(full_obs_agent, y_range = [-5,-25], n = 500, deterministic = True)\n",
542 | "data = data + wait_or_go_experiment(noisy_obs_agent, y_range = [-5,-25], n = 500, deterministic = True)\n",
543 | "columns = ['sigma', 'y_start', 'wait', 'collision']\n",
544 | "df = pd.DataFrame(data, columns=columns)"
545 | ]
546 | },
547 | {
548 | "cell_type": "code",
549 | "execution_count": null,
550 | "id": "eb1b7231",
551 | "metadata": {},
552 | "outputs": [],
553 | "source": [
554 | "# Visualize individual go/no go decisions between different sigma models for various y_ranges.\n",
555 | "\n",
556 | "import numpy as np\n",
557 | "import matplotlib.pyplot as plt\n",
558 | "\n",
559 | "# Note: this is a bit slow and dirty, gets slow with a lot of data. Only use for diagnosis.\n",
560 | "def plot_data(df):\n",
561 | " fig, ax = plt.subplots(figsize=(10, 6))\n",
562 | "\n",
563 | " # Define custom colors and markers based on wait and collision values\n",
564 | " colors = {0: 'red', 1: 'blue'}\n",
565 | " markers = {0: 'o', 1: 'x'}\n",
566 | "\n",
567 | " # Create a dictionary to store the labels we've already added to the legend\n",
568 | " labels_added = {}\n",
569 | "\n",
570 | " for index, row in df.iterrows():\n",
571 | " sigma = row['sigma']\n",
572 | " y_start = row['y_start']\n",
573 | " wait = row['wait']\n",
574 | " collision = row['collision']\n",
575 | " label = f'Wait: {wait}, Collision: {collision}'\n",
576 | "\n",
577 | " # Add scatter plot point with custom color and marker\n",
578 | " ax.scatter(\n",
579 | " y_start,\n",
580 | " sigma,\n",
581 | " marker=markers[collision],\n",
582 | " color=colors[wait],\n",
583 | " label=label if label not in labels_added else \"\",\n",
584 | " alpha=0.7\n",
585 | " )\n",
586 | " \n",
587 | " # Remember that we've added this label to the legend\n",
588 | " labels_added[label] = True\n",
589 | "\n",
590 | " ax.set_xlabel('y_start')\n",
591 | " ax.set_ylabel('sigma')\n",
592 | " ax.legend(title=\"Wait, Collision\", bbox_to_anchor=(1.05, 1), loc='upper left')\n",
593 | " plt.title('Impact of y_start on wait and collision for different sigma agents')\n",
594 | " plt.show()\n",
595 | " \n",
596 | "plot_data(df)"
597 | ]
598 | },
599 | {
600 | "cell_type": "markdown",
601 | "id": "86a7a58e",
602 | "metadata": {},
603 | "source": [
604 | "**Task 5.** Discuss the figure. What are we seeing here? What is the difference between the two models and how is it connected to the original hypotheses made?"
605 | ]
606 | },
607 | {
608 | "cell_type": "code",
609 | "execution_count": null,
610 | "id": "44b80148",
611 | "metadata": {},
612 | "outputs": [],
613 | "source": [
614 | "# Visualize the probability of go/no go as the function of y_start, between different sigmas.\n",
615 | "\n",
616 | "# Change smoothness to larger to get less raggedy lines.\n",
617 | "# Also, increase the N in the experiment above to get more observations.\n",
618 | "smoothness = 1\n",
619 | "\n",
620 | "from scipy.ndimage import gaussian_filter1d\n",
621 | "\n",
622 | "import seaborn as sns\n",
623 | "\n",
624 | "def estimate_probability(df, sigma, y_start_values, window_size=1, collision = False):\n",
625 | " sub_df = df[df['sigma'] == sigma].sort_values('y_start')\n",
626 | " probabilities = []\n",
627 | "\n",
628 | " for y_start in y_start_values:\n",
629 | " window_df = sub_df[(sub_df['y_start'] >= y_start - window_size / 2) & (sub_df['y_start'] <= y_start + window_size / 2)]\n",
630 | " probability = window_df['collision' if collision else 'wait'].sum() / len(window_df)\n",
631 | " probabilities.append(probability)\n",
632 | "\n",
633 | " return probabilities\n",
634 | "\n",
635 | "def smooth_probabilities(probabilities, sigma=1):\n",
636 | " return gaussian_filter1d(probabilities, sigma=sigma)\n",
637 | "\n",
638 | "def plot_probability_lines(df, collision = False):\n",
639 | " sns.set_style('whitegrid')\n",
640 | " fig, ax = plt.subplots(figsize=(10, 6))\n",
641 | "\n",
642 | " y_start_range = np.linspace(df['y_start'].min(), df['y_start'].max(), num=500)\n",
643 | "\n",
644 | " for sigma in df['sigma'].unique():\n",
645 | " probabilities = estimate_probability(df, sigma, y_start_range, collision = collision)\n",
646 | " smoothed_probabilities = smooth_probabilities(probabilities, sigma = smoothness)\n",
647 | " ax.plot(y_start_range, smoothed_probabilities, label=f'{sigma}', linewidth=2)\n",
648 | "\n",
649 | " ax.set_xlabel('y_start', fontsize=14)\n",
650 | " ax.set_ylabel('Probability of Wait', fontsize=14)\n",
651 | " ax.legend(title=\"Sigma\", loc=\"upper left\", fontsize=12, title_fontsize=12)\n",
652 | " plt.xticks(fontsize=12)\n",
653 | " plt.yticks(fontsize=12)\n",
654 | " plt.title('Probability of Wait across y_start by sigma', fontsize=16)\n",
655 | " plt.grid(alpha=0.5)\n",
656 | " plt.show()\n",
657 | "\n",
658 | "plot_probability_lines(df)"
659 | ]
660 | },
661 | {
662 | "cell_type": "markdown",
663 | "id": "f12364a6-9a39-401b-b236-64e55c97b6c9",
664 | "metadata": {},
665 | "source": [
666 | "# Extra: Changing the reward function\n",
667 | "\n",
668 | "With the model right above, we observe that the behavior follows our theoretical hypothesis, however it emphasizes is risk-taking in a way that does not feel human-like.\n",
669 | "\n",
670 | "The next step in modeling is to start adjusting the reward function for the desired performance. It sounds theoretically implausible that a collision is as \"bad\" (negatively rewarding) as a successful task is \"good\" (positively rewarding). We can change the model's predicted behavior by adjusting the relative strengths of the positive reward from a successful task and negative reward from a collision. The original values were 10 and -10, respectively."
671 | ]
672 | },
673 | {
674 | "cell_type": "code",
675 | "execution_count": null,
676 | "id": "b825f31f",
677 | "metadata": {},
678 | "outputs": [],
679 | "source": [
680 | "# Set collision penalty to a considerable higher value. 'p' for 'penalty'\n",
681 | "# For ease of illustration using the old code, we set sigma to a slightly different value, which won't practically impact the simulation\n",
682 | "\n",
683 | "#noisy_obs_agent_p = go_nogo.make_agent(sigma = 0.1001, iters = 15, collision_reward = -500)"
684 | ]
685 | },
686 | {
687 | "cell_type": "code",
688 | "execution_count": null,
689 | "id": "a72eb4ac-a31c-4cee-9da8-88dd4e5d439e",
690 | "metadata": {},
691 | "outputs": [],
692 | "source": [
693 | "# Collect data for all three models (full observation, noisy observation, and noisy observation with a large collision penalty\n",
694 | "# Bigger range for start values of the other car for clearer visualization.\n",
695 | "\n",
696 | "#data = wait_or_go_experiment(full_obs_agent, y_range = [-5,-40], n = 1000, deterministic = True)\n",
697 | "#data = data + wait_or_go_experiment(noisy_obs_agent, y_range = [-5,-40], n = 1000, deterministic = True)\n",
698 | "#data = data + wait_or_go_experiment(noisy_obs_agent_p, y_range = [-5,-40], n = 1000, deterministic = True)\n",
699 | "\n",
700 | "#columns = ['sigma', 'y_start', 'wait', 'collision']\n",
701 | "#df = pd.DataFrame(data, columns=columns)"
702 | ]
703 | },
704 | {
705 | "cell_type": "code",
706 | "execution_count": null,
707 | "id": "4666aa97-293e-48b4-8f18-5e92a5213e5f",
708 | "metadata": {},
709 | "outputs": [],
710 | "source": []
711 | }
712 | ],
713 | "metadata": {
714 | "kernelspec": {
715 | "display_name": "Python 3 (ipykernel)",
716 | "language": "python",
717 | "name": "python3"
718 | },
719 | "language_info": {
720 | "codemirror_mode": {
721 | "name": "ipython",
722 | "version": 3
723 | },
724 | "file_extension": ".py",
725 | "mimetype": "text/x-python",
726 | "name": "python",
727 | "nbconvert_exporter": "python",
728 | "pygments_lexer": "ipython3",
729 | "version": "3.12.3"
730 | }
731 | },
732 | "nbformat": 4,
733 | "nbformat_minor": 5
734 | }
735 |
--------------------------------------------------------------------------------
/03-Modelbuilding/animate_trace.py:
--------------------------------------------------------------------------------
1 | import numpy as np
2 | import matplotlib
3 | from matplotlib import pyplot as plt
4 | from matplotlib import animation
5 |
6 | def animate_trace(trace, get_anim = False, x_lim = [-100,100], y_lim = [100,-100]):
7 | plt.close()
8 | interval = 0.05
9 |
10 | frames = len(trace)
11 |
12 | fig = plt.figure()
13 | ax = fig.add_subplot(1, 1, 1, aspect='equal')
14 | ax.set_xlim(x_lim[0], x_lim[1])
15 | ax.set_ylim(y_lim[0], y_lim[1])
16 |
17 |
18 | axl = plt.gca()
19 | axl.invert_yaxis()
20 |
21 | veh1 = plt.Circle((0, 0), 2, fc='orange')
22 | veh2 = plt.Circle((0, 0), 2, fc='red')
23 |
24 | time_text = ax.text(-30, 40, "Time:", fontsize=8)
25 | dist_text = ax.text(-30, 35, "Dist:", fontsize=8)
26 | coll_text = ax.text(-30, 30, "Coll:", fontsize=8)
27 |
28 | def init():
29 | time_text.set_text('')
30 | dist_text.set_text('')
31 | coll_text.set_text('')
32 | return veh1, veh2, time_text, dist_text, coll_text
33 |
34 | def animate(i):
35 | if i == 0:
36 | ax.add_patch(veh1)
37 | ax.add_patch(veh2)
38 | t = trace[i][0]
39 | t = i * interval
40 | x = int(trace[i][1][0])
41 | y = int(trace[i][1][1])
42 | veh1.center = (x, y)
43 | x = int(trace[i][2][0])
44 | y = int(trace[i][2][1])
45 | veh2.center = (x, y)
46 |
47 | if trace[i][4]:
48 | veh2.set_color('r')
49 |
50 | time_text.set_text('Time {:1.2f}'.format(round(t,2)))
51 | d = round(trace[i][3],2)
52 | dist_text.set_text('Dist {:1.2f}'.format(d))
53 | coll_text.set_text('Coll {:1.5s}'.format(str(trace[i][4])))
54 | return veh1, veh2, time_text, dist_text, coll_text
55 |
56 | anim = animation.FuncAnimation(
57 | fig,
58 | animate,
59 | init_func=init,
60 | frames=frames,
61 | repeat=False,
62 | interval=1000 * interval,
63 | blit=True,
64 | )
65 |
66 | if get_anim:
67 | return anim
68 | else:
69 | plt.show()
70 |
--------------------------------------------------------------------------------
/03-Modelbuilding/corati_model.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/03-Modelbuilding/corati_model.png
--------------------------------------------------------------------------------
/03-Modelbuilding/driver_agent_physics.py:
--------------------------------------------------------------------------------
1 | from gymnasium import Env
2 | from gymnasium.spaces import Discrete, Dict, Box
3 | from copy import copy
4 |
5 | from stable_baselines3 import PPO
6 |
7 | import numpy as np
8 | import math
9 |
10 | class driver_agent_physics(Env):
11 | def __init__(self, physics_env, goal_reward = 10, collision_reward = -10, observation_var = 0):
12 | self.env = physics_env
13 |
14 | self.goal_reward = goal_reward
15 | self.collision_reward = collision_reward
16 |
17 | self.max_ticks = 256
18 | self.max_distance = 80
19 |
20 | # go / no go
21 | self.action_space = Discrete(2)
22 |
23 | self.observation_space = Dict(
24 | spaces = {
25 | "distance": Box(0, 1, (1,)),
26 | "passed": Box(0, 1, (1,)),
27 | "distance_var": Box(0, 1, (1,)),
28 | "speed": Box(0, 1, (1,)),
29 | "acceleration": Box(0, 1, (1,)),
30 | "ticks": Box(0, 1, (1,))
31 | })
32 |
33 | self.penalty_per_tick = 0
34 |
35 | self.observation_var = observation_var
36 |
37 | self.agent = PPO("MultiInputPolicy", self, device = 'cuda',
38 | # learning_rate=0.00025,
39 | # ent_coef=0.01,
40 | # n_steps = 128,
41 | # batch_size = 64,
42 | verbose = 0)
43 |
44 | def reset(self, y_start = None, seed = None):
45 | self.env.reset(y_start = y_start)
46 |
47 | self.done = False
48 | self.ticks = 0
49 |
50 | self.collision = False
51 | self.distance_at_go = None
52 | self.waited_before_go = False
53 |
54 | # Init prior as basically flat.
55 | self.prior = {}
56 | self.prior['distance'] = self.max_distance/2
57 | self.prior['distance_var'] = 1000
58 |
59 | self.belief = self.get_belief()
60 |
61 | return self.belief, {}
62 |
63 | def step(self, action):
64 | self.reward = 0
65 | trunc = False
66 | # action: no go
67 | if action == 0:
68 | self.env.tick()
69 | self.ticks += 1
70 | # break if nothing ever happens
71 | if self.ticks > self.max_ticks:
72 | #print("Too many ticks")
73 | self.reward = self.collision_reward
74 | self.done = True
75 | trunc = True
76 | if self.env.get_distance() > self.max_distance:
77 | self.reward = self.collision_reward
78 | self.done = True
79 | # action: go
80 | if action == 1:
81 | # Did we wait for the other car before going?
82 | if self.env.veh2_turn_pos[1] < self.env.veh1_straight_pos[1]:
83 | self.waited_before_go = True
84 | self.distance_at_go = self.env.get_distance()
85 | self.done = True
86 | self.collision, _ = self.env.simulate_go()
87 | if self.collision:
88 | self.reward = self.collision_reward
89 | else:
90 | self.reward = self.goal_reward - self.penalty_per_tick * self.ticks
91 |
92 | self.belief = self.get_belief()
93 |
94 | return self.belief, self.reward, self.done, trunc, {}
95 |
96 | def get_belief(self):
97 | s = self.env.get_state()
98 | s['passed'] = [0] if self.env.veh2_turn_pos[1] > self.env.veh1_straight_pos[1] else [1]
99 | # Make observation noisy, as in https://github.com/gmarkkula/COMMOTIONSFramework
100 | D = s['distance']
101 | d_oth = np.linalg.norm(self.env.veh1_straight_pos-[-1.825, -1.506512])
102 | h = 1.5
103 | s['distance_var'] = d_oth * (1 - h / (D*math.tan(math.atan(h/D) + self.observation_var)))
104 | s['distance_var'] = max(0, s['distance_var'])
105 | s['distance'] = np.random.normal(s['distance'], s['distance_var'])
106 | if self.observation_var > 0:
107 | s['distance'], s['distance_var'] = \
108 | self.kalman_update(self.prior['distance'],
109 | self.prior['distance_var'],
110 | np.random.normal(s['distance'],
111 | self.observation_var),
112 | self.observation_var)
113 | self.prior['distance'] = s['distance']
114 | self.prior['distance_var'] = s['distance_var']
115 | # Normalise into arrays
116 | # TODO: Set maxes as constants and obey them all over
117 | s['ticks'] = [self.ticks/self.max_ticks]
118 | if self.observation_var > 0:
119 | s['distance_var'] = [s['distance_var'] / (self.observation_var**2)]
120 | else:
121 | s['distance_var'] = [0]
122 | s['speed'] = [s['speed'] / 30]
123 | s['distance'] = [s['distance'] / (4*self.max_distance)]
124 | s['acceleration'] = [s['acceleration']]
125 |
126 | if s['speed'][0] > 1 or s['distance'][0] > 1 or s['acceleration'][0] > 1 or s['distance_var'][0] > 1:
127 | print("Box overflow")
128 | print(self.env.get_state())
129 | print(s)
130 |
131 | return s
132 |
133 | def run_episode(self, render = False, deterministic = False, y_start = None):
134 | self.agent.policy.set_training_mode(False)
135 | # if render and self.carla_env.settings.no_rendering_mode:
136 | # self.carla_env.settings.no_rendering_mode = False
137 | # self.carla_env.world.apply_settings(self.carla_env.settings)
138 | self.reset(y_start = y_start)
139 | ticks = 0
140 | total_reward = 0
141 |
142 | while not self.done:
143 | ticks += 1
144 | a = self.agent.predict(self.belief, deterministic = deterministic)[0]
145 | self.step(a)
146 | total_reward += self.reward
147 |
148 | # if render:
149 | # self.carla_env.settings.no_rendering_mode = False
150 | # self.carla_env.world.apply_settings(self.carla_env.settings)
151 |
152 | self.agent.policy.set_training_mode(True)
153 |
154 | return ticks, total_reward, self.distance_at_go, 1 if self.waited_before_go else 0, 1 if self.collision else 0
155 |
156 | def train_agent(self, total_timesteps = 1000, iters = 10, print_debug = True):
157 | # i = iter
158 | # t = ticks
159 | # r = reward
160 | # d = distance at go
161 | # w = waited for the other car before go
162 | # c = collisions
163 | if print_debug:
164 | print("\ni\tt\tr\td\tw\tc")
165 | for i in range(iters):
166 | self.agent.learn(total_timesteps = total_timesteps)
167 | self.run_episodes(100, prefix = str(i), print_debug = print_debug)
168 |
169 | def run_episodes(self, n, prefix = "0", print_debug = True):
170 | ts = []
171 | rs = []
172 | ds = []
173 | ws = []
174 | cs = []
175 | for i in range(n):
176 | t, r, d, w, c = self.run_episode()
177 | ts.append(t)
178 | rs.append(r)
179 | if d: ds.append(d)
180 | ws.append(w)
181 | cs.append(c)
182 | if len(ds) == 0: ds = [0] # to avoid warning in cases where zero gos were observed
183 | if print_debug:
184 | print(prefix, "\t", round(np.mean(ts),2), "\t", round(np.mean(rs),2), "\t", round(np.mean(ds),2), "\t", round(np.mean(ws),2), "\t", round(np.mean(cs),2), sep='')
185 |
186 |
187 | def kalman_update(self, prior_mean, prior_var, observation, s):
188 | """
189 | Update the belief about x using the Kalman filter.
190 |
191 | Parameters:
192 | prior_mean (float): The mean of the prior belief about x.
193 | prior_var (float): The variance of the prior belief about x.
194 | observation (float): The noisy observation of x.
195 | s (float): The standard deviation of the observation noise.
196 |
197 | Returns:
198 | float, float: The mean and variance of the posterior belief about x.
199 | """
200 |
201 | # The observation model has a gain of 1 (linear relationship)
202 | observation_gain = 1
203 |
204 | # Calculate the Kalman gain
205 | kalman_gain = prior_var * observation_gain / (prior_var * observation_gain**2 + s**2)
206 |
207 | # Update the mean and variance of the belief about x
208 | posterior_mean = prior_mean + kalman_gain * (observation - prior_mean)
209 | posterior_var = (1 - kalman_gain * observation_gain) * prior_var
210 |
211 | return posterior_mean, posterior_var
212 |
--------------------------------------------------------------------------------
/03-Modelbuilding/go_nogo.py:
--------------------------------------------------------------------------------
1 | import importlib
2 |
3 | import matplotlib.pyplot as plt
4 | import numpy as np
5 |
6 | import driver_agent_physics
7 | import physics_env
8 | import animate_trace
9 |
10 | importlib.reload(driver_agent_physics)
11 | importlib.reload(physics_env)
12 | importlib.reload(animate_trace)
13 |
14 | def make_agent(sigma = 0, iters = 10, collision_reward = -10):
15 | e = physics_env.physics_env()
16 | agent = driver_agent_physics.driver_agent_physics(e, observation_var = sigma, collision_reward = collision_reward)
17 | agent.max_distance = 30
18 | agent.penalty_per_tick = 0.1
19 | e.veh1_straight_start_y_range = [-25,-2]
20 | agent.train_agent(total_timesteps = 10000, iters = iters)
21 | return agent
22 |
23 | def retrain_agent(agent, y_range = None, iters = 5):
24 | if y_range:
25 | agent.env.veh1_straight_start_y_range = [y_range[0],y_range[1]]
26 | agent.train_agent(total_timesteps = 10000, iters = iters)
27 | agent.env.veh1_straight_start_y_range = [-25,-2]
28 |
29 | def animate_agent(agent, y_start = None, get_anim = True, x_lim = [-50,50], y_lim = [50,-50]):
30 | agent.env.save_trace = True
31 | print(agent.run_episode(y_start = y_start))
32 | agent.env.save_trace = False
33 | return animate_trace.animate_trace(agent.env.trace, get_anim = get_anim, x_lim = x_lim, y_lim = y_lim)
34 |
35 | def wait_or_go_experiment(agent, y_range, n = 100):
36 | data = []
37 | a.env.veh1_straight_start_y_range = y_range
38 | for i in range(n):
39 | _, _, _, w, c = a.run_episode()
40 | data.append([a.observation_var, a.env.y_start, w, c])
41 |
42 | agent.env.veh1_straight_start_y_range = [-25,-2]
43 | return data
44 |
45 |
46 |
47 | # Visualize the probability of go/no go as the function of y_start, between different sigmas.
48 | # Note that the lines may dip close to max y_start, this is an artefact of the smooting.
49 |
50 | # from scipy.stats import gaussian_kde
51 |
52 | # def estimate_probability(df, sigma, y_start_values):
53 | # sub_df = df[df['sigma'] == sigma]
54 | # kde = gaussian_kde(sub_df[['y_start', 'wait']].T)
55 | # probabilities = kde.evaluate(np.column_stack((y_start_values, np.ones_like(y_start_values))).T)
56 | # return probabilities
57 |
58 | # def plot_probability_lines(df):
59 | # fig, ax = plt.subplots()
60 |
61 | # y_start_range = np.linspace(df['y_start'].min(), df['y_start'].max(), num=500)
62 |
63 | # for sigma in df['sigma'].unique():
64 | # probabilities = estimate_probability(df, sigma, y_start_range)
65 | # ax.plot(y_start_range, probabilities, label=f'Sigma: {sigma}')
66 |
67 | # ax.set_xlabel('y_start')
68 | # ax.set_ylabel('Probability of Wait')
69 | # ax.legend(title="Sigma", loc="upper left")
70 | # plt.title('Probability of wait across y_start by sigma')
71 | # plt.show()
72 |
--------------------------------------------------------------------------------
/03-Modelbuilding/go_nogo_task.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jussippjokinen/CogMod-Tutorial/06306081c89506e18ea98f56f3373668a22019d5/03-Modelbuilding/go_nogo_task.png
--------------------------------------------------------------------------------
/03-Modelbuilding/physics_env.py:
--------------------------------------------------------------------------------
1 | import random
2 | import numpy as np
3 |
4 |
5 | class physics_env():
6 |
7 | def __init__(self, veh1_straight_pos = [-1.825, -10], veh2_turn_pos = [3.65, 1.825], veh1_straight_vel = 8.94, veh2_turn_vel = 0.0 ,sim_freq = 0.05):
8 | self.veh1_straight_pos = np.array(veh1_straight_pos)
9 | self.veh1_straight_start_y_range = [-80,-2]
10 | self.veh2_turn_pos = np.array(veh2_turn_pos)
11 | self.veh1_straight_vel = veh1_straight_vel
12 | self.veh1_straight_vel_range = [5,20]
13 | self.veh2_turn_vel = veh2_turn_vel
14 | self.init_veh1_pos = np.array(veh1_straight_pos)
15 | self.init_veh2_pos = np.array(veh2_turn_pos)
16 | self.init_veh1_vel = veh1_straight_vel
17 | self.init_veh2_vel = veh2_turn_vel
18 | self.sim_freq = sim_freq
19 | self.time = 0
20 | self.save_trace = False
21 |
22 | def reset(self, y_start = None):
23 | self.veh1_straight_pos = self.init_veh1_pos
24 | self.veh1_straight_pos[1] = np.random.uniform(self.veh1_straight_start_y_range[0],
25 | self.veh1_straight_start_y_range[1])
26 | if y_start:
27 | self.veh1_straight_pos[1] = y_start
28 | self.veh2_turn_pos = self.init_veh2_pos
29 | # self.veh1_straight_vel = np.random.uniform(self.veh1_straight_vel_range[0],
30 | # self.veh1_straight_vel_range[1])
31 | self.y_start = self.veh1_straight_pos[1] # for logging
32 | self.veh2_turn_vel = self.init_veh2_vel
33 | self.trace = []
34 | self.time = 0
35 |
36 | def get_random_b_spawn(self):
37 | return
38 |
39 | def get_state(self):
40 | vel = self.veh1_straight_vel
41 | # acc = self.actor_b.get_acceleration().x
42 | dist = self.get_distance()
43 |
44 | state = {}
45 |
46 | state['distance'] = dist
47 | state['speed'] = vel
48 | state['acceleration'] = 0
49 |
50 | return state
51 |
52 | def get_distance(self):
53 | return np.linalg.norm(self.veh1_straight_pos-self.veh2_turn_pos)
54 |
55 | def detect_collision(self):
56 | dist = np.fabs(self.veh1_straight_pos - self.veh2_turn_pos)
57 | return dist[1]<3.58 and dist[0]<1.645
58 |
59 | def simulate_go(self):
60 | done = False
61 | collision = False
62 | steps = 0
63 | self.veh2_turn_vel = 8.94
64 | # self.actor_a.enable_constant_velocity(carla.Vector3D(5,0.0,0.0))
65 | while not done:
66 | steps += 1
67 | self.tick()
68 | # dist = np.linalg.norm(self.veh1_straight_pos-self.veh2_turn_pos)
69 | #print(dist)
70 |
71 | if self.detect_collision():
72 | done = True
73 | collision = True
74 | # print(dist)
75 | # loc = self.actor_a.get_location()
76 | #print(loc.x, loc.y)
77 | # if self.veh2_turn_pos[0] <= 0 and self.veh2_turn_pos[1] <=-1.7: #old condition with loose constraint on y-axis position
78 | if self.veh2_turn_pos[0] <= -3.65:
79 | #print(self.veh2_turn_pos[0],self.veh2_turn_pos[1])
80 | done = True
81 | # Stepped for too long: something wrong with the env?
82 | if steps > 1000:
83 | done = True
84 |
85 | return collision, steps
86 |
87 | def tick(self):
88 | self.time += self.sim_freq
89 | veh1_current_pos = self.veh1_straight_pos
90 | veh2_current_pos = self.veh2_turn_pos
91 |
92 | veh1_dist_travel = self.veh1_straight_vel*self.sim_freq
93 | veh2_dist_travel = self.veh2_turn_vel * self.sim_freq
94 |
95 | veh2_turn_radius = 13.301859
96 | veh2_turn_centre_x = -5.018750
97 | veh2_turn_centre_y = 11.406250
98 |
99 | self.veh1_straight_pos = self.veh1_straight_pos + np.array([0.0, veh1_dist_travel])
100 | angle = np.arctan2(self.veh2_turn_pos[1]-veh2_turn_centre_y,
101 | self.veh2_turn_pos[0] - veh2_turn_centre_x)
102 | angle_increment = veh2_dist_travel/veh2_turn_radius
103 | angle -= angle_increment
104 | self.veh2_turn_pos = np.array([veh2_turn_radius*np.cos(angle)+veh2_turn_centre_x,
105 | veh2_turn_radius*np.sin(angle)+veh2_turn_centre_y])
106 |
107 | if self.save_trace:
108 | self.trace.append([self.time, veh1_current_pos, veh2_current_pos, self.get_distance(), self.detect_collision()])
109 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # CogMod-Tutorial
2 |
3 | CogMod-Tutorial is a set of Python Jupyter Notebooks designed as an introduction to cognitive modeling in Human-Computer Interaction.
4 |
5 | Designed for delivery at CHI2022 and CHI2023, CHI2024 by Jussi Jokinen, Antti Oulasvirta and Andrew Howes: https://sites.google.com/view/modeling-chi24/
6 |
7 | Before starting the tutorial it is worth knowing a little about the Python programming language and also about Jupyter Notebooks. If you are unfamiliar with notebooks and Python, you may want to start with the getting_started.ipynb notebook in the 01-Introduction/. Otherwise, you can start with the Fitts' Law notebook and proceed from there.
8 |
--------------------------------------------------------------------------------