├── .gitattributes ├── .gitignore ├── README.md ├── dqn ├── CaptureSurround.t7 ├── LICENSE ├── MyGame.lua ├── MyRandomFunction.lua ├── NeuralQLearner.lua ├── Rectifier.lua ├── Scale.lua ├── TransitionTable.lua ├── convnet.lua ├── convnet_evasion.lua ├── initenv.lua ├── maxqrewardstep.t7 ├── mylast.t7 ├── mynote.sh ├── mypara.t7 ├── net_downsample_2x_full_y.lua ├── nnutils.lua ├── plotthree.m ├── qmax.t7 ├── qmaxtest.t7 ├── qnowtest.t7 ├── reward_history.t7 ├── step_history.t7 ├── subtitle.m ├── test_agent_cpu.lua ├── test_agent_gpu.lua ├── train_agent_cpu.lua └── train_agent_gpu.lua ├── gifs ├── 3.gif └── Evasion_Test.gif ├── install_depen2.sh ├── install_dependencies.sh ├── run_cpu ├── run_gpu ├── test_cpu └── test_gpu /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | 4 | # Custom for Visual Studio 5 | *.cs diff=csharp 6 | 7 | # Standard to msysgit 8 | *.doc diff=astextplain 9 | *.DOC diff=astextplain 10 | *.docx diff=astextplain 11 | *.DOCX diff=astextplain 12 | *.dot diff=astextplain 13 | *.DOT diff=astextplain 14 | *.pdf diff=astextplain 15 | *.PDF diff=astextplain 16 | *.rtf diff=astextplain 17 | *.RTF diff=astextplain 18 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Windows image file caches 2 | Thumbs.db 3 | ehthumbs.db 4 | 5 | # Folder config file 6 | Desktop.ini 7 | 8 | # Recycle Bin used on file shares 9 | $RECYCLE.BIN/ 10 | 11 | # Windows Installer files 12 | *.cab 13 | *.msi 14 | *.msm 15 | *.msp 16 | 17 | # Windows shortcuts 18 | *.lnk 19 | 20 | # ========================= 21 | # Operating System Files 22 | # ========================= 23 | 24 | # OSX 25 | # ========================= 26 | 27 | .DS_Store 28 | .AppleDouble 29 | .LSOverride 30 | 31 | # Thumbnails 32 | ._* 33 | 34 | # Files that might appear in the root of a volume 35 | .DocumentRevisions-V100 36 | .fseventsd 37 | .Spotlight-V100 38 | .TemporaryItems 39 | .Trashes 40 | .VolumeIcon.icns 41 | 42 | # Directories potentially created on remote AFP share 43 | .AppleDB 44 | .AppleDesktop 45 | Network Trash Folder 46 | Temporary Items 47 | .apdisk 48 | 49 | *_FULL_Y.t7 50 | dqn/*_FULL_Y.t7 51 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | This repo holds the codes for the paper "Learning Evasion Strategy in Pursuit-Evasion by Deep Q-Network, ICPR 2018". 2 | 3 | To replicate the experiment results, a number of dependencies need to be 4 | installed, namely: 5 | * LuaJIT and Torch 7.0 6 | * nngraph 7 | * Xitari (fork of the Arcade Learning Environment (Bellemare et al., 2013)) 8 | * AleWrap (a lua interface to Xitari) 9 | An install script for these dependencies is provided. 10 | 11 | Two run scripts are provided: run_cpu and run_gpu. As the names imply, 12 | the former trains the DQN network using regular CPUs, while the latter uses 13 | GPUs (CUDA), which typically results in a significant speed-up. 14 | 15 | Installation instructions 16 | ------------------------- 17 | 18 | The installation requires Linux with apt-get. 19 | 20 | Note: In order to run the GPU version of DQN, you should additionally have the 21 | NVIDIA® CUDA® (version 5.5 or later) toolkit installed prior to the Torch 22 | installation below. 23 | This can be downloaded from https://developer.nvidia.com/cuda-toolkit 24 | and installation instructions can be found in 25 | http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux 26 | 27 | 28 | To train the DQN, the following components must be installed: 29 | * LuaJIT and Torch 7.0 30 | * nngraph 31 | * Xitari 32 | * AleWrap 33 | 34 | To install all of the above in a subdirectory called 'torch', it should be enough to run 35 | 36 | ./install_dependencies.sh 37 | 38 | from the base directory of the package. 39 | 40 | 41 | Note: The above install script will install the following packages via apt-get: 42 | build-essential, gcc, g++, cmake, curl, libreadline-dev, git-core, libjpeg-dev, 43 | libpng-dev, ncurses-dev, imagemagick, unzip 44 | 45 | Training the DQN 46 | --------------------------- 47 | 48 | ./run_gpu 49 | 50 | 51 | 52 | 53 | ## Citation 54 | Please cite the following paper if you feel this repository useful. 55 | ``` 56 | @inproceedings{PURSUITEVASION2018ICPR, 57 | author = {Jiagang Zhu and 58 | Wei Zou and 59 | Zheng Zhu}, 60 | title = {Learning Evasion Strategy in Pursuit-Evasion by Deep Q-Network}, 61 | booktitle = {ICPR}, 62 | year = {2018}, 63 | } 64 | 65 | ``` 66 | 67 | 68 | ## Contact 69 | For any question, please contact 70 | ``` 71 | Jiagang Zhu: zhujiagang2015@ia.ac.cn 72 | ``` 73 | -------------------------------------------------------------------------------- /dqn/CaptureSurround.t7: -------------------------------------------------------------------------------- 1 | 4 2 | 1 3 | 3 4 | V 1 5 | 17 6 | torch.FloatTensor 7 | 1 8 | 2 9 | 1 10 | 1 11 | 4 12 | 2 13 | 3 14 | V 1 15 | 18 16 | torch.FloatStorage 17 | 2 18 | 1 68 19 | -------------------------------------------------------------------------------- /dqn/LICENSE: -------------------------------------------------------------------------------- 1 | 2 | LIMITED LICENSE: 3 | 4 | Copyright (c) 2014 Google Inc. 5 | Limited License: Under no circumstance is commercial use, reproduction, or 6 | distribution permitted. Use, reproduction, and distribution are permitted 7 | solely for academic use in evaluating and reviewing claims made in 8 | "Human-level control through deep reinforcement learning", Nature 518, 529–533 9 | (26 February 2015) doi:10.1038/nature14236, provided that the following 10 | conditions are met: 11 | 12 | * Any reproduction or distribution of source code must retain the above 13 | copyright notice and the full text of this license including the following 14 | disclaimer.
 15 | 16 | * Any reproduction or distribution in binary form must reproduce the above 17 | copyright notice and the full text of this license including the following 18 | disclaimer
 in the documentation and/or other materials provided with the 19 | distribution. 20 | 21 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 22 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 23 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 24 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 25 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 26 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 27 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 28 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 29 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 30 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 31 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 32 | -------------------------------------------------------------------------------- /dqn/MyGame.lua: -------------------------------------------------------------------------------- 1 | require 'gnuplot' 2 | require 'torch' 3 | require 'image' 4 | require 'nn' 5 | require 'MyRandomFunction' 6 | 7 | 8 | metric=84 9 | 10 | XBound=metric/2 11 | escape_speed=1 12 | puesue_speed=escape_speed*2/3 13 | 14 | initlen=4 15 | gridiv=2 16 | gridunit=1 17 | 18 | grayescape=100 19 | graypursue=200 20 | 21 | escapeinit=4*initlen 22 | 23 | terminalscope=1 24 | separatescope=4 25 | 26 | bound=false 27 | 28 | function addlimit(r,a) 29 | r=r+a; 30 | if r>XBound then 31 | r=XBound; 32 | elseif r<-XBound+1 then 33 | r=-XBound+1; 34 | end 35 | return r; 36 | end 37 | 38 | 39 | function escape(x,y,actionindex) 40 | local a,b 41 | if actionindex==0 then 42 | a=0 43 | b=0 44 | escapeangle=4*math.pi 45 | else 46 | a=escape_speed*math.cos(actionindex*math.pi/gridiv) 47 | b=escape_speed*math.sin(actionindex*math.pi/gridiv) 48 | escapeangle=actionindex*math.pi/gridiv 49 | end 50 | 51 | x=addlimit(x,a) 52 | y=addlimit(y,b) 53 | 54 | return x,y,escapeangle 55 | end 56 | 57 | 58 | function pursue(x1,y1,x,y,centerx,centery) 59 | 60 | local base =math.sqrt(math.pow(x-centerx,2)+math.pow(y-centery,2)) 61 | local a,b 62 | 63 | if base == 0 then 64 | a=0; 65 | b=0; 66 | else 67 | a=puesue_speed*(x-centerx)/base 68 | b=puesue_speed*(y-centery)/base 69 | end 70 | 71 | x1=addlimit(x1,a) 72 | y1=addlimit(y1,b) 73 | 74 | return x1,y1 75 | end 76 | 77 | function leftrightlimit1(maxx,maxxid,minx,minxid,a,xy,xyx,l,r) 78 | local i 79 | if maxx+a>XBound then 80 | for i=l,r do 81 | if xyx[i]~=maxxid then 82 | xy[xyx[i]][1]=xy[xyx[i]][1]+XBound-maxx 83 | end 84 | end 85 | maxx=XBound; 86 | 87 | elseif minx+a<-XBound+1 then 88 | for i=l,r do 89 | if xyx[i]~=minxid then 90 | xy[xyx[i]][1]=xy[xyx[i]][1]-XBound+1-minx 91 | end 92 | end 93 | minx=-XBound+1; 94 | else 95 | for i=l,r do 96 | xy[xyx[i]][1]=xy[xyx[i]][1]+a 97 | end 98 | end 99 | return xy 100 | end 101 | 102 | 103 | function leftrightlimit2(maxx,maxxid,minx,minxid,a,xy,xyx,l,r) 104 | local i 105 | if maxx+a>XBound then 106 | for i=l,r do 107 | if xyx[i]~=maxxid then 108 | xy[xyx[i]][2]=xy[xyx[i]][2]+XBound-maxx 109 | end 110 | end 111 | maxx=XBound; 112 | 113 | elseif minx+a<-XBound+1 then 114 | for i=l,r do 115 | if xyx[i]~=minxid then 116 | xy[xyx[i]][2]=xy[xyx[i]][2]-XBound+1-minx 117 | end 118 | end 119 | minx=-XBound+1; 120 | 121 | else 122 | for i=l,r do 123 | xy[xyx[i]][2]=xy[xyx[i]][2]+a 124 | end 125 | end 126 | return xy 127 | end 128 | 129 | function leftrightbound(maxx,minx,a) 130 | if maxx+a>XBound then 131 | bound=true 132 | elseif minx+a<-XBound+1 then 133 | bound=true 134 | end 135 | end 136 | 137 | function pursuebound(xy,x,y,centerx,centery,xyx,l,r) 138 | local i 139 | 140 | local base = math.sqrt(math.pow(x-centerx,2)+math.pow(y-centery,2)) 141 | local a,b 142 | if base == 0 then 143 | a=0; 144 | b=0; 145 | else 146 | a=puesue_speed*(x-centerx)/base 147 | b=puesue_speed*(y-centery)/base 148 | end 149 | 150 | maxidx=xyx[l] 151 | maxidy=xyx[l] 152 | minidx=xyx[l] 153 | minidy=xyx[l] 154 | maxx,maxy,minx,miny=xy[xyx[l]][1],xy[xyx[l]][2],xy[xyx[l]][1],xy[xyx[l]][2] 155 | 156 | for i=l+1,r do 157 | if maxxxy[xyx[i]][1] then 161 | minidx=xyx[i] 162 | minx=xy[xyx[i]][1] 163 | end 164 | if maxyxy[xyx[i]][2] then 168 | minidy=xyx[i] 169 | miny=xy[xyx[i]][2] 170 | end 171 | end 172 | bound=false 173 | leftrightbound(maxx,minx,a) 174 | leftrightbound(maxy,miny,b) 175 | return bound 176 | end 177 | 178 | 179 | 180 | function pursuetogether(xy,x,y,centerx,centery,xyx,l,r) 181 | local i 182 | local base =math.sqrt(math.pow(x-centerx,2)+math.pow(y-centery,2)) 183 | local a,b 184 | if base == 0 then 185 | a=0; 186 | b=0; 187 | else 188 | a=puesue_speed*(x-centerx)/base 189 | b=puesue_speed*(y-centery)/base 190 | end 191 | 192 | maxidx=xyx[l] 193 | maxidy=xyx[l] 194 | minidx=xyx[l] 195 | minidy=xyx[l] 196 | maxx,maxy,minx,miny=xy[xyx[l]][1],xy[xyx[l]][2],xy[xyx[l]][1],xy[xyx[l]][2] 197 | 198 | for i=l+1,r do 199 | if maxxxy[xyx[i]][1] then 203 | minidx=xyx[i] 204 | minx=xy[xyx[i]][1] 205 | end 206 | if maxyxy[xyx[i]][2] then 210 | minidy=xyx[i] 211 | miny=xy[xyx[i]][2] 212 | end 213 | end 214 | 215 | xy=leftrightlimit1(maxx,maxxid,minx,minxid,a,xy,xyx,l,r) 216 | xy=leftrightlimit2(maxy,maxyid,miny,minyid,b,xy,xyx,l,r) 217 | 218 | return xy 219 | end 220 | 221 | function body1(x,y,xxx,id) 222 | yb=XBound+1 223 | xxx[x+yb][y+yb] = id 224 | xxx[x+yb+1][y+yb] = id 225 | xxx[x+yb-1][y+yb] = id 226 | xxx[x+yb][y+yb+1] = id 227 | xxx[x+yb][y+yb-1] = id 228 | 229 | return xxx 230 | end 231 | 232 | function body2(x,y,xxx,id) 233 | yb=XBound+1 234 | xxx[x+yb][y+yb] = id 235 | xxx[x+yb+1][y+yb] = id 236 | xxx[x+yb-1][y+yb] = id 237 | xxx[x+yb][y+yb+1] = id 238 | xxx[x+yb][y+yb-1] = id 239 | 240 | xxx[x+yb+1][y+yb-1] = id 241 | xxx[x+yb+1][y+yb+1] = id 242 | xxx[x+yb-1][y+yb+1] = id 243 | xxx[x+yb-1][y+yb-1] = id 244 | return xxx 245 | end 246 | 247 | function imgsave() 248 | xxx = torch.zeros(metric+4, metric+4) 249 | xxx = body1(x,y,xxx,grayescape) 250 | 251 | for i=1,iiii do 252 | xxx = body2(xy[i][1],xy[i][2],xxx,graypursue) 253 | end 254 | 255 | return xxx 256 | end 257 | 258 | function body(x,y,xx,id) 259 | xx[x+XBound][y+XBound] = id 260 | -- xx[x+XBound+1+1][y+XBound+1] = id 261 | -- xx[x+XBound-1+1][y+XBound+1] = id 262 | -- xx[x+XBound+1][y+XBound+1+1] = id 263 | -- xx[x+XBound+1][y+XBound-1+1] = id 264 | 265 | -- xx[x+XBound+1][y+XBound-1] = id 266 | -- xx[x+XBound+1][y+XBound+1] = id 267 | -- xx[x+XBound-1][y+XBound+1] = id 268 | -- xx[x+XBound+1][y+XBound+1] = id 269 | return xx 270 | end 271 | 272 | 273 | 274 | 275 | function imgdisplay() 276 | local i 277 | 278 | xx = torch.zeros(metric, metric) 279 | xx=body(x,y,xx,grayescape) 280 | 281 | for i=1,iiii do 282 | xx=body(xy[i][1],xy[i][2],xx,graypursue) 283 | end 284 | 285 | return xx 286 | end 287 | 288 | function parainit() 289 | local i,j 290 | Sigma=1; 291 | GradNow=0; 292 | GradLast=0; 293 | 294 | GradWalltemp=torch.zeros(XBound,1); 295 | GradWall=torch.zeros(2*XBound,2*XBound); 296 | 297 | for i=-XBound+1,0 do 298 | GradWalltemp[i+XBound][1]=math.exp(-(i+XBound)*(i+XBound)/(2*Sigma*Sigma))/(Sigma*math.sqrt(2*math.pi)); 299 | end 300 | 301 | for i=-XBound+1,XBound do 302 | for j=-XBound+1,XBound do 303 | if j<=0 and j>=-XBound+1 then 304 | GradWall[i+XBound][j+XBound]=GradWall[i+XBound][j+XBound]+GradWalltemp[j+XBound][1] 305 | end 306 | if j>0 and j<=XBound then 307 | GradWall[i+XBound][j+XBound]=GradWall[i+XBound][j+XBound]+GradWalltemp[-j+XBound+1][1] 308 | end 309 | end 310 | end 311 | 312 | for j=-XBound+1,XBound do 313 | for i=-XBound+1,XBound do 314 | if i<=0 and i>=-XBound+1 then 315 | GradWall[i+XBound][j+XBound]=GradWall[i+XBound][j+XBound]+GradWalltemp[i+XBound][1] 316 | end 317 | if i>0 and i<=XBound then 318 | GradWall[i+XBound][j+XBound]=GradWall[i+XBound][j+XBound]+GradWalltemp[-i+XBound+1][1] 319 | end 320 | end 321 | end 322 | end 323 | 324 | 325 | 326 | function NewGameInit(index,testing) 327 | local i,j 328 | if index==1 then 329 | -- local iii=torch.random(0,3); 330 | local iii 331 | if not testing then 332 | iii=myrandom(0,2); 333 | else 334 | iii=3 335 | end 336 | x=escapeinit*math.cos(iii*math.pi/gridiv) 337 | y=escapeinit*math.sin(iii*math.pi/gridiv) 338 | iiii=4 339 | xy = torch.zeros(iiii, 2) 340 | xy[1][1] = initlen 341 | xy[1][2] = initlen 342 | xy[2][1] = initlen 343 | xy[2][2] = -initlen 344 | xy[3][1] = -initlen 345 | xy[3][2] = initlen 346 | xy[4][1] = -initlen 347 | xy[4][2] = -initlen 348 | elseif index==2 then 349 | -- local iii=torch.random(0,3); 350 | -- local iii=myrandom(0,3); 351 | local iii 352 | if not testing then 353 | iii=myrandom(1,3); 354 | else 355 | iii=0 356 | end 357 | x=escapeinit*math.cos(iii*math.pi/gridiv) 358 | y=escapeinit*math.sin(iii*math.pi/gridiv) 359 | 360 | iiii=myrandom(4,8); 361 | 362 | xy = torch.zeros(iiii, 2) 363 | 364 | xy[1][1]=torch.random(-initlen,initlen); 365 | xy[1][2]=torch.random(-initlen,initlen); 366 | for i=2,iiii do 367 | xy[i][1]=torch.random(-initlen,initlen); 368 | xy[i][2]=torch.random(-initlen,initlen); 369 | for j=1,i-1 do 370 | if xy[i][1]==xy[j][1] and xy[i][2]==xy[j][2] then 371 | -- xy[i][1]=torch.random(-initlen,initlen); 372 | xy[i][1]=torch.random(-initlen,initlen); 373 | xy[i][2]=torch.random(-initlen,initlen); 374 | j=1; 375 | end 376 | end 377 | end 378 | elseif index==3 or index==4 or index==5 then 379 | if index==4 then 380 | iiii=4 381 | else 382 | iiii=myrandom(4,8) 383 | end 384 | xy = torch.zeros(iiii, 2) 385 | unready=true 386 | while unready do 387 | xy[1][1]=torch.random(-XBound+1,XBound); 388 | xy[1][2]=torch.random(-XBound+1,XBound); 389 | for i=2,iiii do 390 | xy[i][1]=torch.random(-XBound+1,XBound); 391 | xy[i][2]=torch.random(-XBound+1,XBound); 392 | for j=1,i-1 do 393 | if xy[i][1]==xy[j][1] and xy[i][2]==xy[j][2] then 394 | xy[i][1]=torch.random(-XBound+1,XBound); 395 | xy[i][2]=torch.random(-XBound+1,XBound); 396 | j=1; 397 | end 398 | if xy[i][1]*xy[i][1] + xy[i][2]*xy[i][2]0 and y>0 and not testing then 423 | unready=true 424 | end 425 | if (x<0 or y<0) and testing then 426 | unready=true 427 | end 428 | end 429 | elseif index==6 then 430 | iiii=myrandom(4,8) 431 | if not testing then 432 | unready=true 433 | while unready do 434 | x=torch.random(-XBound/2+1,XBound/2); 435 | y=torch.random(-XBound/2+1,XBound/2); 436 | unready=false 437 | if x<0 and y<0 then 438 | unready=true 439 | end 440 | end 441 | else 442 | while true do 443 | x=torch.random(-XBound/2+1,XBound/2); 444 | y=torch.random(-XBound/2+1,XBound/2); 445 | if x<0 and y<0 then 446 | break 447 | end 448 | end 449 | end 450 | 451 | unready=true 452 | while unready do 453 | xy = torch.zeros(iiii, 2) 454 | xy[1][1]=torch.random(-XBound+1,XBound) 455 | xy[1][2]=torch.random(-XBound+1,XBound) 456 | for i=2,iiii do 457 | xy[i][1]=torch.random(-XBound+1,XBound); 458 | xy[i][2]=torch.random(-XBound+1,XBound); 459 | for j=1,i-1 do 460 | if xy[i][1]==xy[j][1] and xy[i][2]==xy[j][2] then 461 | xy[i][1]=torch.random(-XBound+1,XBound); 462 | xy[i][2]=torch.random(-XBound+1,XBound); 463 | j=1; 464 | end 465 | end 466 | end 467 | unready=false 468 | for j=1,iiii do 469 | if x==xy[j][1] and y==xy[j][2] then 470 | unready=true 471 | end 472 | end 473 | end 474 | end 475 | 476 | Terminal=false; 477 | Reward=0; 478 | ImageCnt=0; 479 | 480 | xx=imgdisplay() 481 | 482 | -- xxx=imgsave() 483 | -- return xx,Reward,Terminal--,xxx 484 | end 485 | 486 | 487 | function GameSetting(actionindex,index) 488 | local i 489 | x,y,eangle=escape(x,y,actionindex) 490 | xyx = torch.zeros(iiii) 491 | xyxid = torch.zeros(iiii) 492 | for i=1,iiii do 493 | xyx[i]=(xy[i][1]-x)*(xy[i][1]-x)+(xy[i][2]-y)*(xy[i][2]-y) 494 | xyxid[i]=i 495 | end 496 | i=1 497 | for i = 1, iiii do 498 | for j = i+1,iiii do 499 | if (xyx[i] > xyx[j]) then 500 | local temp = xyx[i]; 501 | xyx[i] = xyx[j]; 502 | xyx[j] = temp; 503 | 504 | temp = xyxid[i]; 505 | xyxid[i] = xyxid[j]; 506 | xyxid[j] = temp; 507 | end 508 | end 509 | end 510 | l=torch.min(xy,1) 511 | r=torch.max(xy,1) 512 | left=l[1][1] 513 | down=l[1][2] 514 | right=r[1][1] 515 | up=r[1][2] 516 | if index==1 or index==2 then 517 | bound1=false 518 | if index==2 then 519 | bound1=pursuebound(xy,x,y,xy[xyxid[1]][1],xy[xyxid[1]][2],xyxid,1,iiii) 520 | end 521 | if (xleft-1 and ydown-1) or bound1 then 522 | i=1 523 | for i=1,iiii do 524 | xy[i][1],xy[i][2]=pursue(xy[i][1],xy[i][2],x,y,xy[i][1],xy[i][2]) 525 | end 526 | else 527 | xy=pursuetogether(xy,x,y,xy[xyxid[1]][1],xy[xyxid[1]][2],xyxid,1,iiii) 528 | end 529 | elseif index==3 then 530 | i=1 531 | for i=1,iiii do 532 | xy[i][1],xy[i][2]=pursue(xy[i][1],xy[i][2],x,y,xy[i][1],xy[i][2]) 533 | end 534 | elseif index==4 or index==5 or index==6 then 535 | 536 | bound1=pursuebound(xy,x,y,xy[xyxid[1]][1],xy[xyxid[1]][2],xyxid,1,torch.floor(iiii/2)) 537 | tt=torch.floor(iiii/2)+1 538 | bound2=pursuebound(xy,x,y,xy[xyxid[tt]][1],xy[xyxid[tt]][2],xyxid,tt,iiii) 539 | 540 | if bound1 or bound2 then 541 | i=1 542 | for i=1,iiii do 543 | xy[i][1],xy[i][2]=pursue(xy[i][1],xy[i][2],x,y,xy[i][1],xy[i][2]) 544 | end 545 | else 546 | if (xleft and ydown) then 547 | i=1 548 | for i=1,iiii do 549 | xy[i][1],xy[i][2]=pursue(xy[i][1],xy[i][2],x,y,xy[i][1],xy[i][2]) 550 | end 551 | first=1 552 | else 553 | local tempx=0 554 | local tempy=0 555 | i=1 556 | for i=1,torch.floor(iiii/2) do 557 | tempx=tempx+xy[xyxid[i]][1] 558 | tempy=tempy+xy[xyxid[i]][2] 559 | end 560 | 561 | local centerx=tempx/torch.floor(iiii/2); 562 | local centery=tempy/torch.floor(iiii/2); 563 | 564 | local tempx=0 565 | local tempy=0 566 | i=torch.floor(iiii/2)+1 567 | for i=torch.floor(iiii/2)+1,iiii do 568 | tempx=tempx+xy[xyxid[i]][1] 569 | tempy=tempy+xy[xyxid[i]][2] 570 | end 571 | 572 | local centerx1=tempx/(iiii-torch.floor(iiii/2)); 573 | local centery1=tempy/(iiii-torch.floor(iiii/2)); 574 | 575 | xy=pursuetogether(xy,x,y,xy[xyxid[1]][1],xy[xyxid[1]][2],xyxid,1,torch.floor(iiii/2)) 576 | 577 | if math.pow(centerx1-centerx,2)+math.pow(centery1-centery,2)left and ydown) then 617 | Surround=true 618 | local differforcex=0 619 | local differforcey=0 620 | for i=1,iiii do 621 | local r2=math.pow(x-xy[i][1],2)+math.pow(y-xy[i][2],2) 622 | if r2 == 0 then 623 | Terminal=true 624 | break 625 | else 626 | differforcex=differforcex+(x-xy[i][1])/(r2*r2) 627 | differforcey=differforcey+(y-xy[i][2])/(r2*r2) 628 | end 629 | end 630 | 631 | differforcex=differforcex+1*((x-XBound)/(math.pow(x-XBound,4)+0.0001)+(x+XBound)/(math.pow(x+XBound,4)+0.0001)) 632 | differforcey=differforcey+1*((y-XBound)/(math.pow(y-XBound,4)+0.0001)+(y+XBound)/(math.pow(y+XBound,4)+0.0001)) 633 | 634 | local dangle=torch.atan(differforcey/differforcex) 635 | if differforcey<0 and differforcex>0 then 636 | dangle=dangle+2*math.pi 637 | elseif differforcey<0 and differforcex<0 then 638 | dangle=dangle+math.pi 639 | elseif differforcey>0 and differforcex<0 then 640 | dangle=dangle+math.pi 641 | end 642 | if eangle>=dangle-math.pi/4 and eangle<=dangle+math.pi/4 then 643 | Reward=1; 644 | else 645 | Reward=-1; 646 | end 647 | elseif (x==XBound or x==-XBound+1 or y==XBound or y==-XBound+1) then 648 | Reward=-1; 649 | elseif (mint4*initlen*initlen and GradNow= 0 then 113 | self.network:cuda() 114 | else 115 | self.network:float() 116 | end 117 | -- Load preprocessing network. 118 | if not (type(self.preproc == 'string')) then 119 | error('The preprocessing is not a string') 120 | end 121 | msg, err = pcall(require, self.preproc) 122 | if not msg then 123 | error("Error loading preprocessing net") 124 | end 125 | self.preproc = err 126 | self.preproc = self:preproc() 127 | self.preproc:float() 128 | 129 | if self.gpu and self.gpu >= 0 then 130 | self.network:cuda() 131 | self.tensor_type = torch.CudaTensor 132 | else 133 | self.network:float() 134 | self.tensor_type = torch.FloatTensor 135 | end 136 | 137 | -- Create transition table. 138 | ---- assuming the transition table always gets floating point input 139 | ---- (Foat or Cuda tensors) and always returns one of the two, as required 140 | ---- internally it always uses ByteTensors for states, scaling and 141 | ---- converting accordingly 142 | local transition_args = { 143 | stateDim = self.state_dim, numActions = self.n_actions, 144 | histLen = self.hist_len, gpu = self.gpu, 145 | maxSize = self.replay_memory, histType = self.histType, 146 | histSpacing = self.histSpacing, nonTermProb = self.nonTermProb, 147 | bufferSize = self.bufferSize 148 | } 149 | 150 | self.transitions = dqn.TransitionTable(transition_args) 151 | 152 | self.lastState = nil 153 | self.lastAction = nil 154 | self.v_avg = 0 -- V running average. 155 | self.tderr_avg = 0 -- TD error running average. 156 | self.q_max = 1 157 | self.r_max = 1 158 | self.w, self.dw = self.network:getParameters() 159 | self.dw:zero() 160 | 161 | self.deltas = self.dw:clone():fill(0) 162 | 163 | self.tmp= self.dw:clone():fill(0) 164 | self.g = self.dw:clone():fill(0) 165 | self.g2 = self.dw:clone():fill(0) 166 | 167 | self.SnapshotCnt=0 168 | 169 | if self.target_q then 170 | self.target_network = self.network:clone() 171 | end 172 | end 173 | 174 | 175 | function nql:reset(state) 176 | if not state then 177 | return 178 | end 179 | self.best_network = state.best_network 180 | self.network = state.model 181 | self.w, self.dw = self.network:getParameters() 182 | self.dw:zero() 183 | self.numSteps = 0 184 | print("RESET STATE SUCCESFULLY") 185 | end 186 | 187 | 188 | function nql:preprocess(rawstate) 189 | if self.preproc then 190 | return self.preproc:forward(rawstate:float()) 191 | :clone():reshape(self.state_dim) 192 | end 193 | 194 | return rawstate 195 | end 196 | 197 | 198 | function nql:getQUpdate(args) 199 | local s, a, r, s2, term, delta 200 | local q, q2, q2_max 201 | 202 | s = args.s 203 | a = args.a 204 | r = args.r 205 | s2 = args.s2 206 | term = args.term 207 | 208 | -- The order of calls to forward is a bit odd in order 209 | -- to avoid unnecessary calls (we only need 2). 210 | 211 | -- delta = r + (1-terminal) * gamma * max_a Q(s2, a) - Q(s, a) 212 | term = term:clone():float():mul(-1):add(1) 213 | 214 | local target_q_net 215 | if self.target_q then 216 | target_q_net = self.target_network 217 | else 218 | target_q_net = self.network 219 | end 220 | 221 | -- Compute max_a Q(s_2, a). 222 | q2_max = target_q_net:forward(s2):float():max(2) 223 | 224 | -- print(q2_max) 225 | 226 | -- Compute q2 = (1-terminal) * gamma * max_a Q(s2, a) 227 | q2 = q2_max:clone():mul(self.discount):cmul(term) 228 | 229 | delta = r:clone():float() 230 | 231 | if self.rescale_r then 232 | delta:div(self.r_max) 233 | end 234 | delta:add(q2) 235 | 236 | -- q = Q(s,a) 237 | local q_all = self.network:forward(s):float() 238 | 239 | q = torch.FloatTensor(q_all:size(1)) 240 | for i=1,q_all:size(1) do 241 | q[i] = q_all[i][a[i]] 242 | -- print(q[i]) 243 | end 244 | -- print(q) 245 | delta:add(-1, q) 246 | 247 | if self.clip_delta then 248 | delta[delta:ge(self.clip_delta)] = self.clip_delta 249 | delta[delta:le(-self.clip_delta)] = -self.clip_delta 250 | end 251 | 252 | local targets = torch.zeros(self.minibatch_size, self.n_actions):float() 253 | 254 | 255 | for i=1,math.min(self.minibatch_size,a:size(1)) do 256 | targets[i][a[i]] = delta[i] 257 | end 258 | 259 | if self.gpu >= 0 then targets = targets:cuda() end 260 | 261 | return targets, delta, q2_max 262 | end 263 | 264 | 265 | function nql:qLearnMinibatch() 266 | -- Perform a minibatch Q-learning update: 267 | -- w += alpha * (r + gamma max Q(s2,a2) - Q(s,a)) * dQ(s,a)/dw 268 | assert(self.transitions:size() > self.minibatch_size) 269 | 270 | local s, a, r, s2, term = self.transitions:sample(self.minibatch_size) 271 | 272 | local targets, delta, q2_max = self:getQUpdate{s=s, a=a, r=r, s2=s2, 273 | term=term, update_qmax=true} 274 | 275 | -- zero gradients of parameters 276 | self.dw:zero() 277 | 278 | -- get new gradient 279 | self.network:backward(s, targets) 280 | 281 | -- add weight cost to gradient 282 | self.dw:add(-self.wc, self.w) 283 | 284 | -- compute linearly annealed learning rate 285 | local t = math.max(0, self.numSteps - self.learn_start) 286 | self.lr = (self.lr_start - self.lr_end) * (self.lr_endt - t)/self.lr_endt + 287 | self.lr_end 288 | self.lr = math.max(self.lr, self.lr_end) 289 | 290 | -- use gradients 291 | self.g:mul(0.95):add(0.05, self.dw) 292 | self.tmp:cmul(self.dw, self.dw) 293 | self.g2:mul(0.95):add(0.05, self.tmp) 294 | self.tmp:cmul(self.g, self.g) 295 | self.tmp:mul(-1) 296 | self.tmp:add(self.g2) 297 | self.tmp:add(0.01) 298 | self.tmp:sqrt() 299 | 300 | -- self.g2:cmul(self.dw, self.dw) 301 | -- self.tmp:mul(0.95):add(0.05, self.g2) 302 | -- self.tmp:add(0.01) 303 | -- self.tmp:sqrt() 304 | 305 | -- print(self.deltas:mul(0):addcdiv(self.lr, self.dw, self.tmp)) 306 | -- print(self.tmp:size()) 307 | -- print(tt) 308 | -- print(self.tmp) 309 | 310 | -- accumulate update 311 | self.deltas:mul(0):addcdiv(self.lr, self.dw, self.tmp) 312 | self.w:add(self.deltas) 313 | end 314 | 315 | 316 | function nql:sample_validation_data() 317 | local s, a, r, s2, term = self.transitions:sample(self.valid_size) 318 | self.valid_s = s:clone() 319 | self.valid_a = a:clone() 320 | self.valid_r = r:clone() 321 | self.valid_s2 = s2:clone() 322 | self.valid_term = term:clone() 323 | end 324 | 325 | 326 | function nql:compute_validation_statistics() 327 | local targets, delta, q2_max = self:getQUpdate{s=self.valid_s, 328 | a=self.valid_a, r=self.valid_r, s2=self.valid_s2, term=self.valid_term} 329 | 330 | self.v_avg = self.q_max * q2_max:mean() 331 | self.tderr_avg = delta:clone():abs():mean() 332 | end 333 | 334 | 335 | function nql:perceive(reward, rawstate, terminal, testing, testing_ep) 336 | -- Preprocess state (will be set to nil if terminal) 337 | local state = self:preprocess(rawstate):float() 338 | local curState 339 | 340 | -- print(state:size()) 341 | if self.max_reward then 342 | reward = math.min(reward, self.max_reward) 343 | end 344 | if self.min_reward then 345 | reward = math.max(reward, self.min_reward) 346 | end 347 | if self.rescale_r then 348 | self.r_max = math.max(self.r_max, reward) 349 | end 350 | 351 | self.transitions:add_recent_state(state, terminal) 352 | 353 | local currentFullState = self.transitions:get_recent() 354 | -- print(currentFullState:size()) 355 | --Store transition s, a, r, s' 356 | if self.lastState and not testing then 357 | self.transitions:add(self.lastState, self.lastAction, reward, 358 | self.lastTerminal, priority) 359 | end 360 | 361 | -- if not Snapshot and self.numSteps == self.learn_start+1 and not testing then 362 | -- self:sample_validation_data() 363 | -- end 364 | 365 | -- if Snapshot and self.SnapshotCnt == self.learn_start+1 and not testing then 366 | -- self:sample_validation_data() 367 | -- end 368 | 369 | curState= self.transitions:get_recent() 370 | curState = curState:resize(1, unpack(self.input_dims)) 371 | 372 | 373 | local lasthiddenlayer ={} 374 | if testing then 375 | lasthiddenlayer=self.network:get(10).output[1]:float() 376 | end 377 | 378 | qnow = self:getQgreedy(curState) 379 | 380 | -- Select action 381 | local actionIndex = 1 382 | if not terminal then 383 | actionIndex = self:eGreedy(curState, testing_ep, self.numSteps) 384 | end 385 | 386 | self.transitions:add_recent_action(actionIndex) 387 | 388 | --Do some Q-learning updates 389 | if not Snapshot and self.numSteps > self.learn_start and not testing and 390 | self.numSteps % self.update_freq == 0 then 391 | for i = 1, self.n_replay do 392 | self:qLearnMinibatch() 393 | end 394 | end 395 | 396 | if Snapshot and self.SnapshotCnt > self.learn_start and not testing and 397 | self.numSteps % self.update_freq == 0 then 398 | for i = 1, self.n_replay do 399 | self:qLearnMinibatch() 400 | end 401 | end 402 | 403 | if not testing and not Snapshot then 404 | self.numSteps = self.numSteps + 1 405 | end 406 | 407 | if not testing and Snapshot then 408 | self.SnapshotCnt=self.SnapshotCnt + 1 409 | if self.SnapshotCnt > self.learn_start then 410 | self.numSteps = self.numSteps + 1 411 | end 412 | end 413 | 414 | -- print(self.numSteps) 415 | 416 | self.lastState = state:clone() 417 | self.lastAction = actionIndex 418 | self.lastTerminal = terminal 419 | 420 | if not Snapshot and self.target_q and self.numSteps % self.target_q == 1 then 421 | self.target_network = self.network:clone() 422 | end 423 | 424 | if Snapshot and self.target_q and self.SnapshotCnt % self.target_q == 1 then 425 | self.target_network = self.network:clone() 426 | end 427 | 428 | if not terminal then 429 | return actionIndex,lasthiddenlayer,qnow 430 | else 431 | return 0,lasthiddenlayer,qnow 432 | end 433 | end 434 | 435 | 436 | function nql:eGreedy(state, testing_ep, curSteps) 437 | 438 | 439 | -- --- epsilon annealing 440 | -- self.ep_start = args.ep or 1 441 | -- self.ep = self.ep_start -- Exploration probability. 442 | -- self.ep_end = args.ep_end or self.ep 443 | -- self.ep_endt = args.ep_endt or 1000000 444 | 445 | 446 | self.ep = testing_ep or (self.ep_end + 447 | math.max(0, (self.ep_start - self.ep_end) * (self.ep_endt - 448 | math.max(0, curSteps - self.learn_start))/self.ep_endt)) 449 | -- Epsilon greedy 450 | if torch.uniform() < self.ep then 451 | self.bestq = 0 452 | return myrandom(1, self.n_actions) 453 | else 454 | return self:greedy(state) 455 | end 456 | end 457 | 458 | function nql:getQgreedy(state) 459 | -- Turn single state into minibatch. Needed for convolutional nets. 460 | if state:dim() == 2 then 461 | assert(false, 'Input must be at least 3D') 462 | state = state:resize(1, state:size(1), state:size(2)) 463 | end 464 | 465 | if self.gpu >= 0 then 466 | state = state:cuda() 467 | end 468 | 469 | 470 | local q = self.network:forward(state):float():squeeze() 471 | -- print(q) 472 | return q 473 | end 474 | 475 | function nql:greedy(state) 476 | -- Turn single state into minibatch. Needed for convolutional nets. 477 | if state:dim() == 2 then 478 | assert(false, 'Input must be at least 3D') 479 | state = state:resize(1, state:size(1), state:size(2)) 480 | end 481 | 482 | if self.gpu >= 0 then 483 | state = state:cuda() 484 | end 485 | 486 | local q = self.network:forward(state):float():squeeze() 487 | -- print(q) 488 | 489 | local maxq = q[1] 490 | local besta = {1} 491 | 492 | -- Evaluate all other actions (with random tie-breaking) 493 | for a = 2, self.n_actions do 494 | if q[a] > maxq then 495 | besta = { a } 496 | maxq = q[a] 497 | elseif q[a] == maxq then 498 | besta[#besta+1] = a 499 | end 500 | end 501 | self.bestq = maxq 502 | 503 | local r = myrandom(1, #besta) 504 | 505 | self.lastAction = besta[r] 506 | 507 | return besta[r] 508 | end 509 | 510 | 511 | function nql:createNetwork() 512 | local n_hid = 128 513 | local mlp = nn.Sequential() 514 | mlp:add(nn.Reshape(self.hist_len*self.ncols*self.state_dim)) 515 | mlp:add(nn.Linear(self.hist_len*self.ncols*self.state_dim, n_hid)) 516 | mlp:add(nn.Rectifier()) 517 | mlp:add(nn.Linear(n_hid, n_hid)) 518 | mlp:add(nn.Rectifier()) 519 | mlp:add(nn.Linear(n_hid, self.n_actions)) 520 | 521 | return mlp 522 | end 523 | 524 | 525 | function nql:_loadNet() 526 | local net = self.network 527 | if self.gpu then 528 | net:cuda() 529 | else 530 | net:float() 531 | end 532 | return net 533 | end 534 | 535 | 536 | function nql:init(arg) 537 | self.actions = arg.actions 538 | self.n_actions = #self.actions 539 | self.network = self:_loadNet() 540 | -- Generate targets. 541 | self.transitions:empty() 542 | end 543 | 544 | 545 | function nql:report() 546 | print(get_weight_norms(self.network)) 547 | print(get_grad_norms(self.network)) 548 | end 549 | -------------------------------------------------------------------------------- /dqn/Rectifier.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (c) 2014 Google Inc. 3 | 4 | See LICENSE file for full terms of limited license. 5 | ]] 6 | 7 | --[[ Rectified Linear Unit. 8 | 9 | The output is max(0, input). 10 | --]] 11 | 12 | local Rectifier, parent = torch.class('nn.Rectifier', 'nn.Module') 13 | 14 | -- This module accepts minibatches 15 | function Rectifier:updateOutput(input) 16 | return self.output:resizeAs(input):copy(input):abs():add(input):div(2) 17 | end 18 | 19 | function Rectifier:updateGradInput(input, gradOutput) 20 | self.gradInput:resizeAs(self.output) 21 | return self.gradInput:sign(self.output):cmul(gradOutput) 22 | end -------------------------------------------------------------------------------- /dqn/Scale.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (c) 2014 Google Inc. 3 | 4 | See LICENSE file for full terms of limited license. 5 | ]] 6 | 7 | require "nn" 8 | require "image" 9 | 10 | local scale = torch.class('nn.Scale', 'nn.Module') 11 | 12 | 13 | function scale:__init(height, width) 14 | self.height = height 15 | self.width = width 16 | end 17 | 18 | function scale:forward(x) 19 | local x = x 20 | if x:dim() > 3 then 21 | x = x[1] 22 | end 23 | 24 | -- x = image.rgb2y(x) 25 | x = image.scale(x, self.width, self.height, 'bilinear') 26 | 27 | return x 28 | end 29 | 30 | function scale:updateOutput(input) 31 | return self:forward(input) 32 | end 33 | 34 | function scale:float() 35 | end 36 | -------------------------------------------------------------------------------- /dqn/TransitionTable.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (c) 2014 Google Inc. 3 | 4 | See LICENSE file for full terms of limited license. 5 | ]] 6 | 7 | require 'image' 8 | 9 | local trans = torch.class('dqn.TransitionTable') 10 | 11 | function trans:__init(args) 12 | self.stateDim = args.stateDim 13 | self.numActions = args.numActions 14 | self.histLen = args.histLen 15 | self.maxSize = args.maxSize or 1024^2 16 | self.bufferSize = args.bufferSize or 1024 17 | self.histType = args.histType or "linear" 18 | self.histSpacing = args.histSpacing or 1 19 | self.zeroFrames = args.zeroFrames or 1 20 | self.nonTermProb = args.nonTermProb or 1 21 | self.nonEventProb = args.nonEventProb or 1 22 | self.gpu = args.gpu 23 | self.numEntries = 0 24 | self.insertIndex = 0 25 | 26 | self.histIndices = {} 27 | local histLen = self.histLen 28 | if self.histType == "linear" then 29 | -- History is the last histLen frames. 30 | self.recentMemSize = self.histSpacing*histLen 31 | for i=1,histLen do 32 | self.histIndices[i] = i*self.histSpacing 33 | end 34 | elseif self.histType == "exp2" then 35 | -- The ith history frame is from 2^(i-1) frames ago. 36 | self.recentMemSize = 2^(histLen-1) 37 | self.histIndices[1] = 1 38 | for i=1,histLen-1 do 39 | self.histIndices[i+1] = self.histIndices[i] + 2^(7-i) 40 | end 41 | elseif self.histType == "exp1.25" then 42 | -- The ith history frame is from 1.25^(i-1) frames ago. 43 | self.histIndices[histLen] = 1 44 | for i=histLen-1,1,-1 do 45 | self.histIndices[i] = math.ceil(1.25*self.histIndices[i+1])+1 46 | end 47 | self.recentMemSize = self.histIndices[1] 48 | for i=1,histLen do 49 | self.histIndices[i] = self.recentMemSize - self.histIndices[i] + 1 50 | end 51 | end 52 | 53 | self.s = torch.ByteTensor(self.maxSize, self.stateDim):fill(0) 54 | self.a = torch.LongTensor(self.maxSize):fill(0) 55 | self.r = torch.zeros(self.maxSize) 56 | self.t = torch.ByteTensor(self.maxSize):fill(0) 57 | self.action_encodings = torch.eye(self.numActions) 58 | 59 | -- Tables for storing the last histLen states. They are used for 60 | -- constructing the most recent agent state more easily. 61 | self.recent_s = {} 62 | self.recent_a = {} 63 | self.recent_t = {} 64 | 65 | local s_size = self.stateDim*histLen 66 | self.buf_a = torch.LongTensor(self.bufferSize):fill(0) 67 | self.buf_r = torch.zeros(self.bufferSize) 68 | self.buf_term = torch.ByteTensor(self.bufferSize):fill(0) 69 | self.buf_s = torch.ByteTensor(self.bufferSize, s_size):fill(0) 70 | self.buf_s2 = torch.ByteTensor(self.bufferSize, s_size):fill(0) 71 | 72 | if self.gpu and self.gpu >= 0 then 73 | self.gpu_s = self.buf_s:float():cuda() 74 | self.gpu_s2 = self.buf_s2:float():cuda() 75 | end 76 | end 77 | 78 | 79 | function trans:reset() 80 | self.numEntries = 0 81 | self.insertIndex = 0 82 | end 83 | 84 | function trans:size() 85 | return self.numEntries 86 | end 87 | 88 | function trans:empty() 89 | return self.numEntries == 0 90 | end 91 | 92 | function trans:fill_buffer() 93 | assert(self.numEntries >= self.bufferSize) 94 | -- clear CPU buffers 95 | self.buf_ind = 1 96 | local ind 97 | for buf_ind=1,self.bufferSize do 98 | local s, a, r, s2, term = self:sample_one(1) 99 | self.buf_s[buf_ind]:copy(s) 100 | self.buf_a[buf_ind] = a 101 | self.buf_r[buf_ind] = r 102 | self.buf_s2[buf_ind]:copy(s2) 103 | self.buf_term[buf_ind] = term 104 | end 105 | self.buf_s = self.buf_s:float():div(255) 106 | self.buf_s2 = self.buf_s2:float():div(255) 107 | if self.gpu and self.gpu >= 0 then 108 | self.gpu_s:copy(self.buf_s) 109 | self.gpu_s2:copy(self.buf_s2) 110 | end 111 | end 112 | 113 | 114 | function trans:sample_one() 115 | assert(self.numEntries > 1) 116 | local index 117 | local valid = false 118 | -- self.nonTermProb=0.25 119 | -- self.nonEventProb=0.25 120 | 121 | while not valid do 122 | -- start at 2 because of previous action 123 | index = torch.random(2, self.numEntries-self.recentMemSize) 124 | if self.t[index+self.recentMemSize-1] == 0 then 125 | valid = true 126 | end 127 | -- if self.nonTermProb < 1 and self.t[index+self.recentMemSize] == 0 and 128 | -- torch.uniform() > self.nonTermProb then 129 | -- -- Discard non-terminal states with probability (1-nonTermProb). 130 | -- -- Note that this is the terminal flag for s_{t+1}. 131 | -- valid = false 132 | -- end 133 | if self.nonEventProb < 1 and self.t[index+self.recentMemSize] == 0 and 134 | self.r[index+self.recentMemSize-1] <= 0 and 135 | torch.uniform() > self.nonTermProb then 136 | -- Discard non-terminal and non-reward states with 137 | -- probability (1-nonTermProb). 138 | valid = false 139 | end 140 | end 141 | 142 | return self:get(index) 143 | end 144 | 145 | 146 | function trans:sample(batch_size) 147 | local batch_size = batch_size or 1 148 | assert(batch_size < self.bufferSize) 149 | 150 | if not self.buf_ind or self.buf_ind + batch_size - 1 > self.bufferSize then 151 | self:fill_buffer() 152 | end 153 | 154 | local index = self.buf_ind 155 | 156 | self.buf_ind = self.buf_ind+batch_size 157 | local range = {{index, index+batch_size-1}} 158 | 159 | local buf_s, buf_s2, buf_a, buf_r, buf_term = self.buf_s, self.buf_s2, 160 | self.buf_a, self.buf_r, self.buf_term 161 | if self.gpu and self.gpu >=0 then 162 | buf_s = self.gpu_s 163 | buf_s2 = self.gpu_s2 164 | end 165 | 166 | return buf_s[range], buf_a[range], buf_r[range], buf_s2[range], buf_term[range] 167 | end 168 | 169 | 170 | function trans:concatFrames(index, use_recent) 171 | if use_recent then 172 | s, t = self.recent_s, self.recent_t 173 | else 174 | s, t = self.s, self.t 175 | end 176 | 177 | local fullstate = s[1].new() 178 | fullstate:resize(self.histLen, unpack(s[1]:size():totable())) 179 | 180 | -- Zero out frames from all but the most recent episode. 181 | local zero_out = false 182 | local episode_start = self.histLen 183 | 184 | for i=self.histLen-1,1,-1 do 185 | if not zero_out then 186 | for j=index+self.histIndices[i]-1,index+self.histIndices[i+1]-2 do 187 | if t[j] == 1 then 188 | zero_out = true 189 | break 190 | end 191 | end 192 | end 193 | 194 | if zero_out then 195 | fullstate[i]:zero() 196 | else 197 | episode_start = i 198 | end 199 | end 200 | 201 | if self.zeroFrames == 0 then 202 | episode_start = 1 203 | end 204 | 205 | -- Copy frames from the current episode. 206 | for i=episode_start,self.histLen do 207 | fullstate[i]:copy(s[index+self.histIndices[i]-1]) 208 | end 209 | 210 | return fullstate 211 | end 212 | 213 | 214 | function trans:concatActions(index, use_recent) 215 | local act_hist = torch.FloatTensor(self.histLen, self.numActions) 216 | if use_recent then 217 | a, t = self.recent_a, self.recent_t 218 | else 219 | a, t = self.a, self.t 220 | end 221 | 222 | -- Zero out frames from all but the most recent episode. 223 | local zero_out = false 224 | local episode_start = self.histLen 225 | 226 | for i=self.histLen-1,1,-1 do 227 | if not zero_out then 228 | for j=index+self.histIndices[i]-1,index+self.histIndices[i+1]-2 do 229 | if t[j] == 1 then 230 | zero_out = true 231 | break 232 | end 233 | end 234 | end 235 | 236 | if zero_out then 237 | act_hist[i]:zero() 238 | else 239 | episode_start = i 240 | end 241 | end 242 | 243 | if self.zeroFrames == 0 then 244 | episode_start = 1 245 | end 246 | 247 | -- Copy frames from the current episode. 248 | for i=episode_start,self.histLen do 249 | act_hist[i]:copy(self.action_encodings[a[index+self.histIndices[i]-1]]) 250 | end 251 | 252 | return act_hist 253 | end 254 | 255 | 256 | function trans:get_recent() 257 | -- Assumes that the most recent state has been added, but the action has not 258 | return self:concatFrames(1, true):float():div(255) 259 | end 260 | 261 | 262 | function trans:get(index) 263 | local s = self:concatFrames(index) 264 | local s2 = self:concatFrames(index+1) 265 | local ar_index = index+self.recentMemSize-1 266 | 267 | return s, self.a[ar_index], self.r[ar_index], s2, self.t[ar_index+1] 268 | end 269 | 270 | 271 | function trans:add(s, a, r, term) 272 | assert(s, 'State cannot be nil') 273 | assert(a, 'Action cannot be nil') 274 | assert(r, 'Reward cannot be nil') 275 | 276 | -- Incremenet until at full capacity 277 | if self.numEntries < self.maxSize then 278 | self.numEntries = self.numEntries + 1 279 | end 280 | 281 | -- Always insert at next index, then wrap around 282 | self.insertIndex = self.insertIndex + 1 283 | -- Overwrite oldest experience once at capacity 284 | if self.insertIndex > self.maxSize then 285 | self.insertIndex = 1 286 | end 287 | 288 | -- Overwrite (s,a,r,t) at insertIndex 289 | self.s[self.insertIndex] = s:clone():float():mul(255) 290 | self.a[self.insertIndex] = a 291 | self.r[self.insertIndex] = r 292 | if term then 293 | self.t[self.insertIndex] = 1 294 | else 295 | self.t[self.insertIndex] = 0 296 | end 297 | end 298 | 299 | 300 | function trans:add_recent_state(s, term) 301 | local s = s:clone():float():mul(255):byte() 302 | if #self.recent_s == 0 then 303 | for i=1,self.recentMemSize do 304 | table.insert(self.recent_s, s:clone():zero()) 305 | table.insert(self.recent_t, 1) 306 | end 307 | end 308 | 309 | table.insert(self.recent_s, s) 310 | if term then 311 | table.insert(self.recent_t, 1) 312 | else 313 | table.insert(self.recent_t, 0) 314 | end 315 | 316 | -- Keep recentMemSize states. 317 | if #self.recent_s > self.recentMemSize then 318 | table.remove(self.recent_s, 1) 319 | table.remove(self.recent_t, 1) 320 | end 321 | end 322 | 323 | 324 | function trans:add_recent_action(a) 325 | if #self.recent_a == 0 then 326 | for i=1,self.recentMemSize do 327 | table.insert(self.recent_a, 1) 328 | end 329 | end 330 | 331 | table.insert(self.recent_a, a) 332 | 333 | -- Keep recentMemSize steps. 334 | if #self.recent_a > self.recentMemSize then 335 | table.remove(self.recent_a, 1) 336 | end 337 | end 338 | 339 | 340 | --[[ 341 | Override the write function to serialize this class into a file. 342 | We do not want to store anything into the file, just the necessary info 343 | to create an empty transition table. 344 | 345 | @param file (FILE object ) @see torch.DiskFile 346 | --]] 347 | function trans:write(file) 348 | file:writeObject({self.stateDim, 349 | self.numActions, 350 | self.histLen, 351 | self.maxSize, 352 | self.bufferSize, 353 | self.numEntries, 354 | self.insertIndex, 355 | self.recentMemSize, 356 | self.histIndices}) 357 | end 358 | 359 | 360 | --[[ 361 | Override the read function to desearialize this class from file. 362 | Recreates an empty table. 363 | 364 | @param file (FILE object ) @see torch.DiskFile 365 | --]] 366 | function trans:read(file) 367 | local stateDim, numActions, histLen, maxSize, bufferSize, numEntries, insertIndex, recentMemSize, histIndices = unpack(file:readObject()) 368 | self.stateDim = stateDim 369 | self.numActions = numActions 370 | self.histLen = histLen 371 | self.maxSize = maxSize 372 | self.bufferSize = bufferSize 373 | self.recentMemSize = recentMemSize 374 | self.histIndices = histIndices 375 | self.numEntries = 0 376 | self.insertIndex = 0 377 | 378 | self.s = torch.ByteTensor(self.maxSize, self.stateDim):fill(0) 379 | self.a = torch.LongTensor(self.maxSize):fill(0) 380 | self.r = torch.zeros(self.maxSize) 381 | self.t = torch.ByteTensor(self.maxSize):fill(0) 382 | self.action_encodings = torch.eye(self.numActions) 383 | 384 | -- Tables for storing the last histLen states. They are used for 385 | -- constructing the most recent agent state more easily. 386 | self.recent_s = {} 387 | self.recent_a = {} 388 | self.recent_t = {} 389 | 390 | self.buf_a = torch.LongTensor(self.bufferSize):fill(0) 391 | self.buf_r = torch.zeros(self.bufferSize) 392 | self.buf_term = torch.ByteTensor(self.bufferSize):fill(0) 393 | self.buf_s = torch.ByteTensor(self.bufferSize, self.stateDim * self.histLen):fill(0) 394 | self.buf_s2 = torch.ByteTensor(self.bufferSize, self.stateDim * self.histLen):fill(0) 395 | 396 | if self.gpu and self.gpu >= 0 then 397 | self.gpu_s = self.buf_s:float():cuda() 398 | self.gpu_s2 = self.buf_s2:float():cuda() 399 | end 400 | end 401 | -------------------------------------------------------------------------------- /dqn/convnet.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (c) 2014 Google Inc. 3 | 4 | See LICENSE file for full terms of limited license. 5 | ]] 6 | 7 | require "initenv" 8 | 9 | function create_network(args) 10 | 11 | local net = nn.Sequential() 12 | net:add(nn.Reshape(unpack(args.input_dims))) 13 | 14 | --- first convolutional layer 15 | local convLayer = nn.SpatialConvolution 16 | 17 | net:add(convLayer(args.hist_len*args.ncols, args.n_units[1], 18 | args.filter_size[1], args.filter_size[1], 19 | args.filter_stride[1], args.filter_stride[1],1)) 20 | net:add(args.nl()) 21 | 22 | -- Add convolutional layers 23 | for i=1,(#args.n_units-1) do 24 | -- second convolutional layer 25 | net:add(convLayer(args.n_units[i], args.n_units[i+1], 26 | args.filter_size[i+1], args.filter_size[i+1], 27 | args.filter_stride[i+1], args.filter_stride[i+1])) 28 | net:add(args.nl()) 29 | end 30 | 31 | local nel 32 | if args.gpu >= 0 then 33 | nel = net:cuda():forward(torch.zeros(1,unpack(args.input_dims)) 34 | :cuda()):nElement() 35 | else 36 | nel = net:forward(torch.zeros(1,unpack(args.input_dims))):nElement() 37 | end 38 | 39 | -- reshape all feature planes into a vector per example 40 | net:add(nn.Reshape(nel)) 41 | 42 | -- fully connected layer 43 | net:add(nn.Linear(nel, args.n_hid[1])) 44 | net:add(args.nl()) 45 | local last_layer_size = args.n_hid[1] 46 | 47 | for i=1,(#args.n_hid-1) do 48 | -- add Linear layer 49 | last_layer_size = args.n_hid[i+1] 50 | net:add(nn.Linear(args.n_hid[i], last_layer_size)) 51 | net:add(args.nl()) 52 | end 53 | 54 | -- add the last fully connected layer (to actions) 55 | net:add(nn.Linear(last_layer_size, args.n_actions)) 56 | 57 | if args.gpu >=0 then 58 | net:cuda() 59 | end 60 | if args.verbose >= 2 then 61 | print(net) 62 | print('Convolutional layers flattened output size:', nel) 63 | end 64 | return net 65 | end 66 | -------------------------------------------------------------------------------- /dqn/convnet_evasion.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (c) 2014 Google Inc. 3 | 4 | See LICENSE file for full terms of limited license. 5 | ]] 6 | 7 | require 'convnet' 8 | 9 | return function(args) 10 | args.n_units = {32, 64, 64} 11 | args.filter_size = {8, 4, 3} 12 | args.filter_stride = {4, 2, 1} 13 | args.n_hid = {512} 14 | args.nl = nn.Rectifier 15 | 16 | return create_network(args) 17 | end 18 | 19 | -------------------------------------------------------------------------------- /dqn/initenv.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (c) 2014 Google Inc. 3 | 4 | See LICENSE file for full terms of limited license. 5 | ]] 6 | dqn = {} 7 | 8 | -- mygame={} 9 | 10 | require 'torch' 11 | require 'nn' 12 | require 'nngraph' 13 | require 'nnutils' 14 | require 'image' 15 | require 'Scale' 16 | require 'NeuralQLearner' 17 | require 'TransitionTable' 18 | require 'Rectifier' 19 | require 'MyGame' 20 | require 'MyRandomFunction' 21 | function torchSetup(_opt) 22 | _opt = _opt or {} 23 | local opt = table.copy(_opt) 24 | assert(opt) 25 | 26 | -- preprocess options: 27 | --- convert options strings to tables 28 | if opt.pool_frms then 29 | opt.pool_frms = str_to_table(opt.pool_frms) 30 | end 31 | -- if opt.env_params then 32 | -- opt.env_params = str_to_table(opt.env_params) 33 | -- end 34 | if opt.agent_params then 35 | opt.agent_params = str_to_table(opt.agent_params) 36 | opt.agent_params.gpu = opt.gpu 37 | opt.agent_params.best = opt.best 38 | opt.agent_params.verbose = opt.verbose 39 | if opt.network ~= '' then 40 | opt.agent_params.network = opt.network 41 | end 42 | end 43 | 44 | --- general setup 45 | opt.tensorType = opt.tensorType or 'torch.FloatTensor' 46 | torch.setdefaulttensortype(opt.tensorType) 47 | if not opt.threads then 48 | opt.threads = 4 49 | end 50 | torch.setnumthreads(opt.threads) 51 | if not opt.verbose then 52 | opt.verbose = 10 53 | end 54 | if opt.verbose >= 1 then 55 | print('Torch Threads:', torch.getnumthreads()) 56 | end 57 | 58 | --- set gpu device 59 | if opt.gpu and opt.gpu >= 0 then 60 | require 'cutorch' 61 | require 'cunn' 62 | if opt.gpu == 0 then 63 | -- print('opt.gpu == 0:') 64 | local gpu_id = tonumber(os.getenv('GPU_ID')) 65 | if gpu_id then opt.gpu = gpu_id+1 end 66 | end 67 | if opt.gpu > 0 then cutorch.setDevice(opt.gpu) end 68 | -- print('opt.gpu > 0:') 69 | opt.gpu = cutorch.getDevice() 70 | print('Using GPU device id:', opt.gpu-1) 71 | else 72 | opt.gpu = -1 73 | if opt.verbose >= 1 then 74 | print('Using CPU code only. GPU device id:', opt.gpu) 75 | end 76 | end 77 | 78 | --- set up random number generators 79 | -- removing lua RNG; seeding torch RNG with opt.seed and setting cutorch 80 | -- RNG seed to the first uniform random int32 from the previous RNG; 81 | -- this is preferred because using the same seed for both generators 82 | -- may introduce correlations; we assume that both torch RNGs ensure 83 | -- adequate dispersion for different seeds. 84 | math.random = nil 85 | opt.seed = opt.seed or 1 86 | torch.manualSeed(opt.seed) 87 | if opt.verbose >= 1 then 88 | print('Torch Seed:', torch.initialSeed()) 89 | end 90 | local firstRandInt = torch.random() 91 | if opt.gpu >= 0 then 92 | cutorch.manualSeed(firstRandInt) 93 | if opt.verbose >= 1 then 94 | print('CUTorch Seed:', cutorch.initialSeed()) 95 | end 96 | end 97 | 98 | return opt 99 | end 100 | 101 | 102 | function setup(_opt) 103 | assert(_opt) 104 | --preprocess options: 105 | --- convert options strings to tables 106 | _opt.pool_frms = str_to_table(_opt.pool_frms) 107 | -- _opt.env_params = str_to_table(_opt.env_params) 108 | _opt.agent_params = str_to_table(_opt.agent_params) 109 | if _opt.agent_params.transition_params then 110 | _opt.agent_params.transition_params = 111 | str_to_table(_opt.agent_params.transition_params) 112 | end 113 | --- first things first 114 | local opt = torchSetup(_opt) 115 | -- load training framework and environment 116 | -- local framework = require(opt.framework) 117 | -- assert(framework) 118 | 119 | -- local gameEnv = framework.GameEnvironment(opt) 120 | -- local gameActions = gameEnv:getActions() 121 | -- local gameActions=torch.Tensor({0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16}) 122 | -- agent options 123 | -- _opt.agent_params.actions = gameActions 124 | _opt.agent_params.actions = {0,1,2,3,4} 125 | _opt.agent_params.gpu = _opt.gpu 126 | _opt.agent_params.best = _opt.best 127 | if _opt.network ~= '' then 128 | _opt.agent_params.network = _opt.network 129 | end 130 | _opt.agent_params.verbose = _opt.verbose 131 | if not _opt.agent_params.state_dim then 132 | -- _opt.agent_params.state_dim = gameEnv:nObsFeature() 133 | end 134 | 135 | local agent = dqn[_opt.agent](_opt.agent_params) 136 | 137 | -- if opt.verbose >= 1 then 138 | -- print('Set up Torch using these options:') 139 | -- for k, v in pairs(opt) do 140 | -- print(k, v) 141 | -- end 142 | -- end 143 | 144 | return agent, opt 145 | end 146 | 147 | 148 | 149 | --- other functions 150 | 151 | function str_to_table(str) 152 | if type(str) == 'table' then 153 | return str 154 | end 155 | if not str or type(str) ~= 'string' then 156 | if type(str) == 'table' then 157 | return str 158 | end 159 | return {} 160 | end 161 | local ttr 162 | if str ~= '' then 163 | local ttx=tt 164 | loadstring('tt = {' .. str .. '}')() 165 | ttr = tt 166 | tt = ttx 167 | else 168 | ttr = {} 169 | end 170 | return ttr 171 | end 172 | 173 | function table.copy(t) 174 | if t == nil then return nil end 175 | local nt = {} 176 | for k, v in pairs(t) do 177 | if type(v) == 'table' then 178 | nt[k] = table.copy(v) 179 | else 180 | nt[k] = v 181 | end 182 | end 183 | setmetatable(nt, table.copy(getmetatable(t))) 184 | return nt 185 | end 186 | -------------------------------------------------------------------------------- /dqn/maxqrewardstep.t7: -------------------------------------------------------------------------------- 1 | 3 2 | 1 3 | 99 4 | 1 5 | 1 6 | 4 7 | 2 8 | 3 9 | V 1 10 | 17 11 | torch.FloatTensor 12 | 1 13 | 3 14 | 1 15 | 1 16 | 4 17 | 3 18 | 3 19 | V 1 20 | 18 21 | torch.FloatStorage 22 | 3 23 | 0 9 71 24 | 1 25 | 2 26 | 4 27 | 4 28 | 3 29 | V 1 30 | 17 31 | torch.FloatTensor 32 | 1 33 | 3 34 | 1 35 | 1 36 | 4 37 | 5 38 | 3 39 | V 1 40 | 18 41 | torch.FloatStorage 42 | 3 43 | 0 10 78 44 | 1 45 | 3 46 | 4 47 | 6 48 | 3 49 | V 1 50 | 17 51 | torch.FloatTensor 52 | 1 53 | 3 54 | 1 55 | 1 56 | 4 57 | 7 58 | 3 59 | V 1 60 | 18 61 | torch.FloatStorage 62 | 3 63 | 0 25 64 64 | 1 65 | 4 66 | 4 67 | 8 68 | 3 69 | V 1 70 | 17 71 | torch.FloatTensor 72 | 1 73 | 3 74 | 1 75 | 1 76 | 4 77 | 9 78 | 3 79 | V 1 80 | 18 81 | torch.FloatStorage 82 | 3 83 | 0 30 68 84 | 1 85 | 5 86 | 4 87 | 10 88 | 3 89 | V 1 90 | 17 91 | torch.FloatTensor 92 | 1 93 | 3 94 | 1 95 | 1 96 | 4 97 | 11 98 | 3 99 | V 1 100 | 18 101 | torch.FloatStorage 102 | 3 103 | 0.740333319 58 135 104 | 1 105 | 6 106 | 4 107 | 12 108 | 3 109 | V 1 110 | 17 111 | torch.FloatTensor 112 | 1 113 | 3 114 | 1 115 | 1 116 | 4 117 | 13 118 | 3 119 | V 1 120 | 18 121 | torch.FloatStorage 122 | 3 123 | 1.67058182 43 83 124 | 1 125 | 7 126 | 4 127 | 14 128 | 3 129 | V 1 130 | 17 131 | torch.FloatTensor 132 | 1 133 | 3 134 | 1 135 | 1 136 | 4 137 | 15 138 | 3 139 | V 1 140 | 18 141 | torch.FloatStorage 142 | 3 143 | 3.5103128 150 298 144 | 1 145 | 8 146 | 4 147 | 16 148 | 3 149 | V 1 150 | 17 151 | torch.FloatTensor 152 | 1 153 | 3 154 | 1 155 | 1 156 | 4 157 | 17 158 | 3 159 | V 1 160 | 18 161 | torch.FloatStorage 162 | 3 163 | 4.97735596 382 928 164 | 1 165 | 9 166 | 4 167 | 18 168 | 3 169 | V 1 170 | 17 171 | torch.FloatTensor 172 | 1 173 | 3 174 | 1 175 | 1 176 | 4 177 | 19 178 | 3 179 | V 1 180 | 18 181 | torch.FloatStorage 182 | 3 183 | 9.07875347 92 364 184 | 1 185 | 10 186 | 4 187 | 20 188 | 3 189 | V 1 190 | 17 191 | torch.FloatTensor 192 | 1 193 | 3 194 | 1 195 | 1 196 | 4 197 | 21 198 | 3 199 | V 1 200 | 18 201 | torch.FloatStorage 202 | 3 203 | 11.1555119 56 243 204 | 1 205 | 11 206 | 4 207 | 22 208 | 3 209 | V 1 210 | 17 211 | torch.FloatTensor 212 | 1 213 | 3 214 | 1 215 | 1 216 | 4 217 | 23 218 | 3 219 | V 1 220 | 18 221 | torch.FloatStorage 222 | 3 223 | 16.4745884 101 232 224 | 1 225 | 12 226 | 4 227 | 24 228 | 3 229 | V 1 230 | 17 231 | torch.FloatTensor 232 | 1 233 | 3 234 | 1 235 | 1 236 | 4 237 | 25 238 | 3 239 | V 1 240 | 18 241 | torch.FloatStorage 242 | 3 243 | 22.2508125 239 685 244 | 1 245 | 13 246 | 4 247 | 26 248 | 3 249 | V 1 250 | 17 251 | torch.FloatTensor 252 | 1 253 | 3 254 | 1 255 | 1 256 | 4 257 | 27 258 | 3 259 | V 1 260 | 18 261 | torch.FloatStorage 262 | 3 263 | 30.4644299 120 646 264 | 1 265 | 14 266 | 4 267 | 28 268 | 3 269 | V 1 270 | 17 271 | torch.FloatTensor 272 | 1 273 | 3 274 | 1 275 | 1 276 | 4 277 | 29 278 | 3 279 | V 1 280 | 18 281 | torch.FloatStorage 282 | 3 283 | 35.6908722 98 370 284 | 1 285 | 15 286 | 4 287 | 30 288 | 3 289 | V 1 290 | 17 291 | torch.FloatTensor 292 | 1 293 | 3 294 | 1 295 | 1 296 | 4 297 | 31 298 | 3 299 | V 1 300 | 18 301 | torch.FloatStorage 302 | 3 303 | 42.8036079 99 586 304 | 1 305 | 16 306 | 4 307 | 32 308 | 3 309 | V 1 310 | 17 311 | torch.FloatTensor 312 | 1 313 | 3 314 | 1 315 | 1 316 | 4 317 | 33 318 | 3 319 | V 1 320 | 18 321 | torch.FloatStorage 322 | 3 323 | 42.2582169 77 414 324 | 1 325 | 17 326 | 4 327 | 34 328 | 3 329 | V 1 330 | 17 331 | torch.FloatTensor 332 | 1 333 | 3 334 | 1 335 | 1 336 | 4 337 | 35 338 | 3 339 | V 1 340 | 18 341 | torch.FloatStorage 342 | 3 343 | 49.8918991 127 344 344 | 1 345 | 18 346 | 4 347 | 36 348 | 3 349 | V 1 350 | 17 351 | torch.FloatTensor 352 | 1 353 | 3 354 | 1 355 | 1 356 | 4 357 | 37 358 | 3 359 | V 1 360 | 18 361 | torch.FloatStorage 362 | 3 363 | 63.1538048 154 812 364 | 1 365 | 19 366 | 4 367 | 38 368 | 3 369 | V 1 370 | 17 371 | torch.FloatTensor 372 | 1 373 | 3 374 | 1 375 | 1 376 | 4 377 | 39 378 | 3 379 | V 1 380 | 18 381 | torch.FloatStorage 382 | 3 383 | 48.2694435 212 769 384 | 1 385 | 20 386 | 4 387 | 40 388 | 3 389 | V 1 390 | 17 391 | torch.FloatTensor 392 | 1 393 | 3 394 | 1 395 | 1 396 | 4 397 | 41 398 | 3 399 | V 1 400 | 18 401 | torch.FloatStorage 402 | 3 403 | 54.5002365 210 667 404 | 1 405 | 21 406 | 4 407 | 42 408 | 3 409 | V 1 410 | 17 411 | torch.FloatTensor 412 | 1 413 | 3 414 | 1 415 | 1 416 | 4 417 | 43 418 | 3 419 | V 1 420 | 18 421 | torch.FloatStorage 422 | 3 423 | 50.1117592 347 796 424 | 1 425 | 22 426 | 4 427 | 44 428 | 3 429 | V 1 430 | 17 431 | torch.FloatTensor 432 | 1 433 | 3 434 | 1 435 | 1 436 | 4 437 | 45 438 | 3 439 | V 1 440 | 18 441 | torch.FloatStorage 442 | 3 443 | 53.0823822 217 859 444 | 1 445 | 23 446 | 4 447 | 46 448 | 3 449 | V 1 450 | 17 451 | torch.FloatTensor 452 | 1 453 | 3 454 | 1 455 | 1 456 | 4 457 | 47 458 | 3 459 | V 1 460 | 18 461 | torch.FloatStorage 462 | 3 463 | 53.57341 561 1395 464 | 1 465 | 24 466 | 4 467 | 48 468 | 3 469 | V 1 470 | 17 471 | torch.FloatTensor 472 | 1 473 | 3 474 | 1 475 | 1 476 | 4 477 | 49 478 | 3 479 | V 1 480 | 18 481 | torch.FloatStorage 482 | 3 483 | 56.7803459 687 1572 484 | 1 485 | 25 486 | 4 487 | 50 488 | 3 489 | V 1 490 | 17 491 | torch.FloatTensor 492 | 1 493 | 3 494 | 1 495 | 1 496 | 4 497 | 51 498 | 3 499 | V 1 500 | 18 501 | torch.FloatStorage 502 | 3 503 | 57.1201363 773 1549 504 | 1 505 | 26 506 | 4 507 | 52 508 | 3 509 | V 1 510 | 17 511 | torch.FloatTensor 512 | 1 513 | 3 514 | 1 515 | 1 516 | 4 517 | 53 518 | 3 519 | V 1 520 | 18 521 | torch.FloatStorage 522 | 3 523 | 51.1403465 2206 4639 524 | 1 525 | 27 526 | 4 527 | 54 528 | 3 529 | V 1 530 | 17 531 | torch.FloatTensor 532 | 1 533 | 3 534 | 1 535 | 1 536 | 4 537 | 55 538 | 3 539 | V 1 540 | 18 541 | torch.FloatStorage 542 | 3 543 | 60.4080429 2041 3776 544 | 1 545 | 28 546 | 4 547 | 56 548 | 3 549 | V 1 550 | 17 551 | torch.FloatTensor 552 | 1 553 | 3 554 | 1 555 | 1 556 | 4 557 | 57 558 | 3 559 | V 1 560 | 18 561 | torch.FloatStorage 562 | 3 563 | 54.1427727 2002 4223 564 | 1 565 | 29 566 | 4 567 | 58 568 | 3 569 | V 1 570 | 17 571 | torch.FloatTensor 572 | 1 573 | 3 574 | 1 575 | 1 576 | 4 577 | 59 578 | 3 579 | V 1 580 | 18 581 | torch.FloatStorage 582 | 3 583 | 50.7730713 2557 4374 584 | 1 585 | 30 586 | 4 587 | 60 588 | 3 589 | V 1 590 | 17 591 | torch.FloatTensor 592 | 1 593 | 3 594 | 1 595 | 1 596 | 4 597 | 61 598 | 3 599 | V 1 600 | 18 601 | torch.FloatStorage 602 | 3 603 | 54.7723007 2031 3928 604 | 1 605 | 31 606 | 4 607 | 62 608 | 3 609 | V 1 610 | 17 611 | torch.FloatTensor 612 | 1 613 | 3 614 | 1 615 | 1 616 | 4 617 | 63 618 | 3 619 | V 1 620 | 18 621 | torch.FloatStorage 622 | 3 623 | 49.0708351 3727 6782 624 | 1 625 | 32 626 | 4 627 | 64 628 | 3 629 | V 1 630 | 17 631 | torch.FloatTensor 632 | 1 633 | 3 634 | 1 635 | 1 636 | 4 637 | 65 638 | 3 639 | V 1 640 | 18 641 | torch.FloatStorage 642 | 3 643 | 54.3541298 3863 6736 644 | 1 645 | 33 646 | 4 647 | 66 648 | 3 649 | V 1 650 | 17 651 | torch.FloatTensor 652 | 1 653 | 3 654 | 1 655 | 1 656 | 4 657 | 67 658 | 3 659 | V 1 660 | 18 661 | torch.FloatStorage 662 | 3 663 | 52.9082794 3868 6722 664 | 1 665 | 34 666 | 4 667 | 68 668 | 3 669 | V 1 670 | 17 671 | torch.FloatTensor 672 | 1 673 | 3 674 | 1 675 | 1 676 | 4 677 | 69 678 | 3 679 | V 1 680 | 18 681 | torch.FloatStorage 682 | 3 683 | 57.386425 3935 6902 684 | 1 685 | 35 686 | 4 687 | 70 688 | 3 689 | V 1 690 | 17 691 | torch.FloatTensor 692 | 1 693 | 3 694 | 1 695 | 1 696 | 4 697 | 71 698 | 3 699 | V 1 700 | 18 701 | torch.FloatStorage 702 | 3 703 | 58.6187859 6821 11873 704 | 1 705 | 36 706 | 4 707 | 72 708 | 3 709 | V 1 710 | 17 711 | torch.FloatTensor 712 | 1 713 | 3 714 | 1 715 | 1 716 | 4 717 | 73 718 | 3 719 | V 1 720 | 18 721 | torch.FloatStorage 722 | 3 723 | 59.384182 3624 6144 724 | 1 725 | 37 726 | 4 727 | 74 728 | 3 729 | V 1 730 | 17 731 | torch.FloatTensor 732 | 1 733 | 3 734 | 1 735 | 1 736 | 4 737 | 75 738 | 3 739 | V 1 740 | 18 741 | torch.FloatStorage 742 | 3 743 | 59.5744209 5624 9192 744 | 1 745 | 38 746 | 4 747 | 76 748 | 3 749 | V 1 750 | 17 751 | torch.FloatTensor 752 | 1 753 | 3 754 | 1 755 | 1 756 | 4 757 | 77 758 | 3 759 | V 1 760 | 18 761 | torch.FloatStorage 762 | 3 763 | 62.9725151 6092 10061 764 | 1 765 | 39 766 | 4 767 | 78 768 | 3 769 | V 1 770 | 17 771 | torch.FloatTensor 772 | 1 773 | 3 774 | 1 775 | 1 776 | 4 777 | 79 778 | 3 779 | V 1 780 | 18 781 | torch.FloatStorage 782 | 3 783 | 65.6488037 4765 7524 784 | 1 785 | 40 786 | 4 787 | 80 788 | 3 789 | V 1 790 | 17 791 | torch.FloatTensor 792 | 1 793 | 3 794 | 1 795 | 1 796 | 4 797 | 81 798 | 3 799 | V 1 800 | 18 801 | torch.FloatStorage 802 | 3 803 | 66.6990585 11254 17801 804 | 1 805 | 41 806 | 4 807 | 82 808 | 3 809 | V 1 810 | 17 811 | torch.FloatTensor 812 | 1 813 | 3 814 | 1 815 | 1 816 | 4 817 | 83 818 | 3 819 | V 1 820 | 18 821 | torch.FloatStorage 822 | 3 823 | 69.6797104 5079 8529 824 | 1 825 | 42 826 | 4 827 | 84 828 | 3 829 | V 1 830 | 17 831 | torch.FloatTensor 832 | 1 833 | 3 834 | 1 835 | 1 836 | 4 837 | 85 838 | 3 839 | V 1 840 | 18 841 | torch.FloatStorage 842 | 3 843 | 74.2961731 6302 10619 844 | 1 845 | 43 846 | 4 847 | 86 848 | 3 849 | V 1 850 | 17 851 | torch.FloatTensor 852 | 1 853 | 3 854 | 1 855 | 1 856 | 4 857 | 87 858 | 3 859 | V 1 860 | 18 861 | torch.FloatStorage 862 | 3 863 | 76.4080963 9542 15355 864 | 1 865 | 44 866 | 4 867 | 88 868 | 3 869 | V 1 870 | 17 871 | torch.FloatTensor 872 | 1 873 | 3 874 | 1 875 | 1 876 | 4 877 | 89 878 | 3 879 | V 1 880 | 18 881 | torch.FloatStorage 882 | 3 883 | 72.7449112 4562 7281 884 | 1 885 | 45 886 | 4 887 | 90 888 | 3 889 | V 1 890 | 17 891 | torch.FloatTensor 892 | 1 893 | 3 894 | 1 895 | 1 896 | 4 897 | 91 898 | 3 899 | V 1 900 | 18 901 | torch.FloatStorage 902 | 3 903 | 77.9146423 3862 6212 904 | 1 905 | 46 906 | 4 907 | 92 908 | 3 909 | V 1 910 | 17 911 | torch.FloatTensor 912 | 1 913 | 3 914 | 1 915 | 1 916 | 4 917 | 93 918 | 3 919 | V 1 920 | 18 921 | torch.FloatStorage 922 | 3 923 | 74.2404251 7030 10773 924 | 1 925 | 47 926 | 4 927 | 94 928 | 3 929 | V 1 930 | 17 931 | torch.FloatTensor 932 | 1 933 | 3 934 | 1 935 | 1 936 | 4 937 | 95 938 | 3 939 | V 1 940 | 18 941 | torch.FloatStorage 942 | 3 943 | 75.4080658 3687 6176 944 | 1 945 | 48 946 | 4 947 | 96 948 | 3 949 | V 1 950 | 17 951 | torch.FloatTensor 952 | 1 953 | 3 954 | 1 955 | 1 956 | 4 957 | 97 958 | 3 959 | V 1 960 | 18 961 | torch.FloatStorage 962 | 3 963 | 77.0724106 7552 12157 964 | 1 965 | 49 966 | 4 967 | 98 968 | 3 969 | V 1 970 | 17 971 | torch.FloatTensor 972 | 1 973 | 3 974 | 1 975 | 1 976 | 4 977 | 99 978 | 3 979 | V 1 980 | 18 981 | torch.FloatStorage 982 | 3 983 | 76.1054077 6318 9896 984 | 1 985 | 50 986 | 4 987 | 100 988 | 3 989 | V 1 990 | 17 991 | torch.FloatTensor 992 | 1 993 | 3 994 | 1 995 | 1 996 | 4 997 | 101 998 | 3 999 | V 1 1000 | 18 1001 | torch.FloatStorage 1002 | 3 1003 | 76.9298019 6750 10780 1004 | 1 1005 | 51 1006 | 4 1007 | 102 1008 | 3 1009 | V 1 1010 | 17 1011 | torch.FloatTensor 1012 | 1 1013 | 3 1014 | 1 1015 | 1 1016 | 4 1017 | 103 1018 | 3 1019 | V 1 1020 | 18 1021 | torch.FloatStorage 1022 | 3 1023 | 78.5791702 4164 6589 1024 | 1 1025 | 52 1026 | 4 1027 | 104 1028 | 3 1029 | V 1 1030 | 17 1031 | torch.FloatTensor 1032 | 1 1033 | 3 1034 | 1 1035 | 1 1036 | 4 1037 | 105 1038 | 3 1039 | V 1 1040 | 18 1041 | torch.FloatStorage 1042 | 3 1043 | 79.0923309 5144 7751 1044 | 1 1045 | 53 1046 | 4 1047 | 106 1048 | 3 1049 | V 1 1050 | 17 1051 | torch.FloatTensor 1052 | 1 1053 | 3 1054 | 1 1055 | 1 1056 | 4 1057 | 107 1058 | 3 1059 | V 1 1060 | 18 1061 | torch.FloatStorage 1062 | 3 1063 | 81.6695404 8813 13901 1064 | 1 1065 | 54 1066 | 4 1067 | 108 1068 | 3 1069 | V 1 1070 | 17 1071 | torch.FloatTensor 1072 | 1 1073 | 3 1074 | 1 1075 | 1 1076 | 4 1077 | 109 1078 | 3 1079 | V 1 1080 | 18 1081 | torch.FloatStorage 1082 | 3 1083 | 81.9793549 8062 12746 1084 | 1 1085 | 55 1086 | 4 1087 | 110 1088 | 3 1089 | V 1 1090 | 17 1091 | torch.FloatTensor 1092 | 1 1093 | 3 1094 | 1 1095 | 1 1096 | 4 1097 | 111 1098 | 3 1099 | V 1 1100 | 18 1101 | torch.FloatStorage 1102 | 3 1103 | 82.4294891 5447 8654 1104 | 1 1105 | 56 1106 | 4 1107 | 112 1108 | 3 1109 | V 1 1110 | 17 1111 | torch.FloatTensor 1112 | 1 1113 | 3 1114 | 1 1115 | 1 1116 | 4 1117 | 113 1118 | 3 1119 | V 1 1120 | 18 1121 | torch.FloatStorage 1122 | 3 1123 | 82.9039383 8628 13428 1124 | 1 1125 | 57 1126 | 4 1127 | 114 1128 | 3 1129 | V 1 1130 | 17 1131 | torch.FloatTensor 1132 | 1 1133 | 3 1134 | 1 1135 | 1 1136 | 4 1137 | 115 1138 | 3 1139 | V 1 1140 | 18 1141 | torch.FloatStorage 1142 | 3 1143 | 83.3780594 4661 7197 1144 | 1 1145 | 58 1146 | 4 1147 | 116 1148 | 3 1149 | V 1 1150 | 17 1151 | torch.FloatTensor 1152 | 1 1153 | 3 1154 | 1 1155 | 1 1156 | 4 1157 | 117 1158 | 3 1159 | V 1 1160 | 18 1161 | torch.FloatStorage 1162 | 3 1163 | 83.8745575 4756 7501 1164 | 1 1165 | 59 1166 | 4 1167 | 118 1168 | 3 1169 | V 1 1170 | 17 1171 | torch.FloatTensor 1172 | 1 1173 | 3 1174 | 1 1175 | 1 1176 | 4 1177 | 119 1178 | 3 1179 | V 1 1180 | 18 1181 | torch.FloatStorage 1182 | 3 1183 | 82.5935211 5159 7816 1184 | 1 1185 | 60 1186 | 4 1187 | 120 1188 | 3 1189 | V 1 1190 | 17 1191 | torch.FloatTensor 1192 | 1 1193 | 3 1194 | 1 1195 | 1 1196 | 4 1197 | 121 1198 | 3 1199 | V 1 1200 | 18 1201 | torch.FloatStorage 1202 | 3 1203 | 82.6272812 8388 13021 1204 | 1 1205 | 61 1206 | 4 1207 | 122 1208 | 3 1209 | V 1 1210 | 17 1211 | torch.FloatTensor 1212 | 1 1213 | 3 1214 | 1 1215 | 1 1216 | 4 1217 | 123 1218 | 3 1219 | V 1 1220 | 18 1221 | torch.FloatStorage 1222 | 3 1223 | 82.3874969 5808 9449 1224 | 1 1225 | 62 1226 | 4 1227 | 124 1228 | 3 1229 | V 1 1230 | 17 1231 | torch.FloatTensor 1232 | 1 1233 | 3 1234 | 1 1235 | 1 1236 | 4 1237 | 125 1238 | 3 1239 | V 1 1240 | 18 1241 | torch.FloatStorage 1242 | 3 1243 | 82.8035889 6504 10096 1244 | 1 1245 | 63 1246 | 4 1247 | 126 1248 | 3 1249 | V 1 1250 | 17 1251 | torch.FloatTensor 1252 | 1 1253 | 3 1254 | 1 1255 | 1 1256 | 4 1257 | 127 1258 | 3 1259 | V 1 1260 | 18 1261 | torch.FloatStorage 1262 | 3 1263 | 84.6659088 8956 14075 1264 | 1 1265 | 64 1266 | 4 1267 | 128 1268 | 3 1269 | V 1 1270 | 17 1271 | torch.FloatTensor 1272 | 1 1273 | 3 1274 | 1 1275 | 1 1276 | 4 1277 | 129 1278 | 3 1279 | V 1 1280 | 18 1281 | torch.FloatStorage 1282 | 3 1283 | 84.2061691 6904 10176 1284 | 1 1285 | 65 1286 | 4 1287 | 130 1288 | 3 1289 | V 1 1290 | 17 1291 | torch.FloatTensor 1292 | 1 1293 | 3 1294 | 1 1295 | 1 1296 | 4 1297 | 131 1298 | 3 1299 | V 1 1300 | 18 1301 | torch.FloatStorage 1302 | 3 1303 | 85.2420883 6278 9714 1304 | 1 1305 | 66 1306 | 4 1307 | 132 1308 | 3 1309 | V 1 1310 | 17 1311 | torch.FloatTensor 1312 | 1 1313 | 3 1314 | 1 1315 | 1 1316 | 4 1317 | 133 1318 | 3 1319 | V 1 1320 | 18 1321 | torch.FloatStorage 1322 | 3 1323 | 85.2520599 4314 6684 1324 | 1 1325 | 67 1326 | 4 1327 | 134 1328 | 3 1329 | V 1 1330 | 17 1331 | torch.FloatTensor 1332 | 1 1333 | 3 1334 | 1 1335 | 1 1336 | 4 1337 | 135 1338 | 3 1339 | V 1 1340 | 18 1341 | torch.FloatStorage 1342 | 3 1343 | 85.6229401 5913 9050 1344 | 1 1345 | 68 1346 | 4 1347 | 136 1348 | 3 1349 | V 1 1350 | 17 1351 | torch.FloatTensor 1352 | 1 1353 | 3 1354 | 1 1355 | 1 1356 | 4 1357 | 137 1358 | 3 1359 | V 1 1360 | 18 1361 | torch.FloatStorage 1362 | 3 1363 | 85.287178 5459 8267 1364 | 1 1365 | 69 1366 | 4 1367 | 138 1368 | 3 1369 | V 1 1370 | 17 1371 | torch.FloatTensor 1372 | 1 1373 | 3 1374 | 1 1375 | 1 1376 | 4 1377 | 139 1378 | 3 1379 | V 1 1380 | 18 1381 | torch.FloatStorage 1382 | 3 1383 | 85.1850815 6481 9718 1384 | 1 1385 | 70 1386 | 4 1387 | 140 1388 | 3 1389 | V 1 1390 | 17 1391 | torch.FloatTensor 1392 | 1 1393 | 3 1394 | 1 1395 | 1 1396 | 4 1397 | 141 1398 | 3 1399 | V 1 1400 | 18 1401 | torch.FloatStorage 1402 | 3 1403 | 87.5705261 3898 6204 1404 | 1 1405 | 71 1406 | 4 1407 | 142 1408 | 3 1409 | V 1 1410 | 17 1411 | torch.FloatTensor 1412 | 1 1413 | 3 1414 | 1 1415 | 1 1416 | 4 1417 | 143 1418 | 3 1419 | V 1 1420 | 18 1421 | torch.FloatStorage 1422 | 3 1423 | 86.317482 5838 9262 1424 | 1 1425 | 72 1426 | 4 1427 | 144 1428 | 3 1429 | V 1 1430 | 17 1431 | torch.FloatTensor 1432 | 1 1433 | 3 1434 | 1 1435 | 1 1436 | 4 1437 | 145 1438 | 3 1439 | V 1 1440 | 18 1441 | torch.FloatStorage 1442 | 3 1443 | 87.01017 4208 6778 1444 | 1 1445 | 73 1446 | 4 1447 | 146 1448 | 3 1449 | V 1 1450 | 17 1451 | torch.FloatTensor 1452 | 1 1453 | 3 1454 | 1 1455 | 1 1456 | 4 1457 | 147 1458 | 3 1459 | V 1 1460 | 18 1461 | torch.FloatStorage 1462 | 3 1463 | 87.115303 6400 10003 1464 | 1 1465 | 74 1466 | 4 1467 | 148 1468 | 3 1469 | V 1 1470 | 17 1471 | torch.FloatTensor 1472 | 1 1473 | 3 1474 | 1 1475 | 1 1476 | 4 1477 | 149 1478 | 3 1479 | V 1 1480 | 18 1481 | torch.FloatStorage 1482 | 3 1483 | 86.6254044 8624 13036 1484 | 1 1485 | 75 1486 | 4 1487 | 150 1488 | 3 1489 | V 1 1490 | 17 1491 | torch.FloatTensor 1492 | 1 1493 | 3 1494 | 1 1495 | 1 1496 | 4 1497 | 151 1498 | 3 1499 | V 1 1500 | 18 1501 | torch.FloatStorage 1502 | 3 1503 | 85.488266 10790 16112 1504 | 1 1505 | 76 1506 | 4 1507 | 152 1508 | 3 1509 | V 1 1510 | 17 1511 | torch.FloatTensor 1512 | 1 1513 | 3 1514 | 1 1515 | 1 1516 | 4 1517 | 153 1518 | 3 1519 | V 1 1520 | 18 1521 | torch.FloatStorage 1522 | 3 1523 | 87.7770615 4946 8627 1524 | 1 1525 | 77 1526 | 4 1527 | 154 1528 | 3 1529 | V 1 1530 | 17 1531 | torch.FloatTensor 1532 | 1 1533 | 3 1534 | 1 1535 | 1 1536 | 4 1537 | 155 1538 | 3 1539 | V 1 1540 | 18 1541 | torch.FloatStorage 1542 | 3 1543 | 87.3532104 4106 6422 1544 | 1 1545 | 78 1546 | 4 1547 | 156 1548 | 3 1549 | V 1 1550 | 17 1551 | torch.FloatTensor 1552 | 1 1553 | 3 1554 | 1 1555 | 1 1556 | 4 1557 | 157 1558 | 3 1559 | V 1 1560 | 18 1561 | torch.FloatStorage 1562 | 3 1563 | 87.0052185 6746 10285 1564 | 1 1565 | 79 1566 | 4 1567 | 158 1568 | 3 1569 | V 1 1570 | 17 1571 | torch.FloatTensor 1572 | 1 1573 | 3 1574 | 1 1575 | 1 1576 | 4 1577 | 159 1578 | 3 1579 | V 1 1580 | 18 1581 | torch.FloatStorage 1582 | 3 1583 | 87.2277298 5690 8891 1584 | 1 1585 | 80 1586 | 4 1587 | 160 1588 | 3 1589 | V 1 1590 | 17 1591 | torch.FloatTensor 1592 | 1 1593 | 3 1594 | 1 1595 | 1 1596 | 4 1597 | 161 1598 | 3 1599 | V 1 1600 | 18 1601 | torch.FloatStorage 1602 | 3 1603 | 86.9693985 5286 8242 1604 | 1 1605 | 81 1606 | 4 1607 | 162 1608 | 3 1609 | V 1 1610 | 17 1611 | torch.FloatTensor 1612 | 1 1613 | 3 1614 | 1 1615 | 1 1616 | 4 1617 | 163 1618 | 3 1619 | V 1 1620 | 18 1621 | torch.FloatStorage 1622 | 3 1623 | 86.6926041 5513 8543 1624 | 1 1625 | 82 1626 | 4 1627 | 164 1628 | 3 1629 | V 1 1630 | 17 1631 | torch.FloatTensor 1632 | 1 1633 | 3 1634 | 1 1635 | 1 1636 | 4 1637 | 165 1638 | 3 1639 | V 1 1640 | 18 1641 | torch.FloatStorage 1642 | 3 1643 | 88.5110855 6996 10471 1644 | 1 1645 | 83 1646 | 4 1647 | 166 1648 | 3 1649 | V 1 1650 | 17 1651 | torch.FloatTensor 1652 | 1 1653 | 3 1654 | 1 1655 | 1 1656 | 4 1657 | 167 1658 | 3 1659 | V 1 1660 | 18 1661 | torch.FloatStorage 1662 | 3 1663 | 89.9347839 8207 12309 1664 | 1 1665 | 84 1666 | 4 1667 | 168 1668 | 3 1669 | V 1 1670 | 17 1671 | torch.FloatTensor 1672 | 1 1673 | 3 1674 | 1 1675 | 1 1676 | 4 1677 | 169 1678 | 3 1679 | V 1 1680 | 18 1681 | torch.FloatStorage 1682 | 3 1683 | 89.7893753 10998 16435 1684 | 1 1685 | 85 1686 | 4 1687 | 170 1688 | 3 1689 | V 1 1690 | 17 1691 | torch.FloatTensor 1692 | 1 1693 | 3 1694 | 1 1695 | 1 1696 | 4 1697 | 171 1698 | 3 1699 | V 1 1700 | 18 1701 | torch.FloatStorage 1702 | 3 1703 | 90.5599976 6800 10707 1704 | 1 1705 | 86 1706 | 4 1707 | 172 1708 | 3 1709 | V 1 1710 | 17 1711 | torch.FloatTensor 1712 | 1 1713 | 3 1714 | 1 1715 | 1 1716 | 4 1717 | 173 1718 | 3 1719 | V 1 1720 | 18 1721 | torch.FloatStorage 1722 | 3 1723 | 89.9406281 7532 11382 1724 | 1 1725 | 87 1726 | 4 1727 | 174 1728 | 3 1729 | V 1 1730 | 17 1731 | torch.FloatTensor 1732 | 1 1733 | 3 1734 | 1 1735 | 1 1736 | 4 1737 | 175 1738 | 3 1739 | V 1 1740 | 18 1741 | torch.FloatStorage 1742 | 3 1743 | 87.7954407 4418 6998 1744 | 1 1745 | 88 1746 | 4 1747 | 176 1748 | 3 1749 | V 1 1750 | 17 1751 | torch.FloatTensor 1752 | 1 1753 | 3 1754 | 1 1755 | 1 1756 | 4 1757 | 177 1758 | 3 1759 | V 1 1760 | 18 1761 | torch.FloatStorage 1762 | 3 1763 | 89.2815781 7843 11650 1764 | 1 1765 | 89 1766 | 4 1767 | 178 1768 | 3 1769 | V 1 1770 | 17 1771 | torch.FloatTensor 1772 | 1 1773 | 3 1774 | 1 1775 | 1 1776 | 4 1777 | 179 1778 | 3 1779 | V 1 1780 | 18 1781 | torch.FloatStorage 1782 | 3 1783 | 89.3388443 6840 10774 1784 | 1 1785 | 90 1786 | 4 1787 | 180 1788 | 3 1789 | V 1 1790 | 17 1791 | torch.FloatTensor 1792 | 1 1793 | 3 1794 | 1 1795 | 1 1796 | 4 1797 | 181 1798 | 3 1799 | V 1 1800 | 18 1801 | torch.FloatStorage 1802 | 3 1803 | 88.8973236 8567 12933 1804 | 1 1805 | 91 1806 | 4 1807 | 182 1808 | 3 1809 | V 1 1810 | 17 1811 | torch.FloatTensor 1812 | 1 1813 | 3 1814 | 1 1815 | 1 1816 | 4 1817 | 183 1818 | 3 1819 | V 1 1820 | 18 1821 | torch.FloatStorage 1822 | 3 1823 | 88.9192276 5706 8671 1824 | 1 1825 | 92 1826 | 4 1827 | 184 1828 | 3 1829 | V 1 1830 | 17 1831 | torch.FloatTensor 1832 | 1 1833 | 3 1834 | 1 1835 | 1 1836 | 4 1837 | 185 1838 | 3 1839 | V 1 1840 | 18 1841 | torch.FloatStorage 1842 | 3 1843 | 88.8432922 5065 7987 1844 | 1 1845 | 93 1846 | 4 1847 | 186 1848 | 3 1849 | V 1 1850 | 17 1851 | torch.FloatTensor 1852 | 1 1853 | 3 1854 | 1 1855 | 1 1856 | 4 1857 | 187 1858 | 3 1859 | V 1 1860 | 18 1861 | torch.FloatStorage 1862 | 3 1863 | 88.2233963 8316 12211 1864 | 1 1865 | 94 1866 | 4 1867 | 188 1868 | 3 1869 | V 1 1870 | 17 1871 | torch.FloatTensor 1872 | 1 1873 | 3 1874 | 1 1875 | 1 1876 | 4 1877 | 189 1878 | 3 1879 | V 1 1880 | 18 1881 | torch.FloatStorage 1882 | 3 1883 | 88.7269135 11130 17414 1884 | 1 1885 | 95 1886 | 4 1887 | 190 1888 | 3 1889 | V 1 1890 | 17 1891 | torch.FloatTensor 1892 | 1 1893 | 3 1894 | 1 1895 | 1 1896 | 4 1897 | 191 1898 | 3 1899 | V 1 1900 | 18 1901 | torch.FloatStorage 1902 | 3 1903 | 88.4081726 7678 12858 1904 | 1 1905 | 96 1906 | 4 1907 | 192 1908 | 3 1909 | V 1 1910 | 17 1911 | torch.FloatTensor 1912 | 1 1913 | 3 1914 | 1 1915 | 1 1916 | 4 1917 | 193 1918 | 3 1919 | V 1 1920 | 18 1921 | torch.FloatStorage 1922 | 3 1923 | 88.0464096 6222 9552 1924 | 1 1925 | 97 1926 | 4 1927 | 194 1928 | 3 1929 | V 1 1930 | 17 1931 | torch.FloatTensor 1932 | 1 1933 | 3 1934 | 1 1935 | 1 1936 | 4 1937 | 195 1938 | 3 1939 | V 1 1940 | 18 1941 | torch.FloatStorage 1942 | 3 1943 | 97.8261108 8228 12751 1944 | 1 1945 | 98 1946 | 4 1947 | 196 1948 | 3 1949 | V 1 1950 | 17 1951 | torch.FloatTensor 1952 | 1 1953 | 3 1954 | 1 1955 | 1 1956 | 4 1957 | 197 1958 | 3 1959 | V 1 1960 | 18 1961 | torch.FloatStorage 1962 | 3 1963 | 87.6725845 8593 13048 1964 | 1 1965 | 99 1966 | 4 1967 | 198 1968 | 3 1969 | V 1 1970 | 17 1971 | torch.FloatTensor 1972 | 1 1973 | 3 1974 | 1 1975 | 1 1976 | 4 1977 | 199 1978 | 3 1979 | V 1 1980 | 18 1981 | torch.FloatStorage 1982 | 3 1983 | 87.074173 13020 19933 1984 | -------------------------------------------------------------------------------- /dqn/mylast.t7: -------------------------------------------------------------------------------- 1 | 3 2 | 1 3 | 738 4 | 1 5 | 1 6 | 4 7 | 2 8 | 3 9 | V 1 10 | 17 11 | torch.FloatTensor 12 | 1 13 | 512 14 | 1 15 | 1 16 | 4 17 | 3 18 | 3 19 | V 1 20 | 18 21 | torch.FloatStorage 22 | 512 23 | 0 0 0 0 0 0 0 0 0 0 0 0 6.58622837 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3.344594 0 0 0 0 0 0 0 0 0 0 0 0 0 31.7757397 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 190.779022 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35.1762314 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9.61548805 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 36.4697495 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 24.4820023 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 24 | 1 25 | 2 26 | 4 27 | 4 28 | 3 29 | V 1 30 | 17 31 | torch.FloatTensor 32 | 1 33 | 512 34 | 1 35 | 1 36 | 4 37 | 5 38 | 3 39 | V 1 40 | 18 41 | torch.FloatStorage 42 | 512 43 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 32.4019089 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 37.9879723 0 0 0 0 0 0 11.4234066 0 0 0 123.595909 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 47.10112 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 71.7981339 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6.06885815 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 44 | 1 45 | 3 46 | 4 47 | 6 48 | 3 49 | V 1 50 | 17 51 | torch.FloatTensor 52 | 1 53 | 512 54 | 1 55 | 1 56 | 4 57 | 7 58 | 3 59 | V 1 60 | 18 61 | torch.FloatStorage 62 | 512 63 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9.76747799 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5.57563782 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 39.5400467 0 0 0 0 0 0 5.02919292 0 0 0 147.964554 0 0 0 0 0 0 0 0 0 0 0 0 14.0224094 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8.78799343 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 54.5700951 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13.6398649 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 64 | 1 65 | 4 66 | 4 67 | 8 68 | 3 69 | V 1 70 | 17 71 | torch.FloatTensor 72 | 1 73 | 512 74 | 1 75 | 1 76 | 4 77 | 9 78 | 3 79 | V 1 80 | 18 81 | torch.FloatStorage 82 | 512 83 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 24.1775379 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6.33089638 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19.3304119 0 0 0 0 0 0 0 0 0 0 154.748093 0 0 0 0 0 0 0 0 0 0 0 0 43.4271622 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11.491353 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 61.9754639 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 84 | 1 85 | 5 86 | 4 87 | 10 88 | 3 89 | V 1 90 | 17 91 | torch.FloatTensor 92 | 1 93 | 512 94 | 1 95 | 1 96 | 4 97 | 11 98 | 3 99 | V 1 100 | 18 101 | torch.FloatStorage 102 | 512 103 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30.0381927 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11.4009027 0 0 0 0 0 0 0 0 0 0 163.856079 0 0 0 0 0 0 0 0 0 0 0 0 51.4670258 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12.8769541 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2.12077022 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.560262799 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.07576883 0 73.5429535 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 104 | 1 105 | 6 106 | 4 107 | 12 108 | 3 109 | V 1 110 | 17 111 | torch.FloatTensor 112 | 1 113 | 512 114 | 1 115 | 1 116 | 4 117 | 13 118 | 3 119 | V 1 120 | 18 121 | torch.FloatStorage 122 | 512 123 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30.6641045 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 22.5346432 0 0 0 0 0 0 0 0 0 0 171.061829 0 0 0 0 0 0 0 0 0 0 0 0 40.4784966 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11.1698961 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2.51682806 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5.17310381 0 60.5071602 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 124 | 1 125 | 7 126 | 4 127 | 14 128 | 3 129 | V 1 130 | 17 131 | torch.FloatTensor 132 | 1 133 | 512 134 | 1 135 | 1 136 | 4 137 | 15 138 | 3 139 | V 1 140 | 18 141 | torch.FloatStorage 142 | 512 143 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 39.1534271 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 33.3445015 0 0 0 0 0 0 0 0 0 0 170.686066 0 0 0 0 0 0 0 0 0 0 0 0 24.5942192 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 24.2273102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8.67008305 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 17.5755196 0 56.7241402 0 3.70744944 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3.1662693 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 144 | 1 145 | 8 146 | 4 147 | 16 148 | 3 149 | V 1 150 | 17 151 | torch.FloatTensor 152 | 1 153 | 512 154 | 1 155 | 1 156 | 4 157 | 17 158 | 3 159 | V 1 160 | 18 161 | torch.FloatStorage 162 | 512 163 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 43.7858658 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 34.5896683 0 0 0 0 0 0 0 0 0 0 179.550507 0 0 0 0 0 0 0 0 0 0 0 0 25.7089577 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 27.147171 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14.7311239 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 22.5616665 0 52.8098869 0 7.43758869 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14.0994234 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 164 | 1 165 | 9 166 | 4 167 | 18 168 | 3 169 | V 1 170 | 17 171 | torch.FloatTensor 172 | 1 173 | 512 174 | 1 175 | 1 176 | 4 177 | 19 178 | 3 179 | V 1 180 | 18 181 | torch.FloatStorage 182 | 512 183 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3.77072763 0 0 0 0 0 0 0 47.7512054 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 39.8951912 0 0 0 0 0 0 0 0 0 0 182.945389 0 0 0 0 0 0 0 0 0 0 0 0 9.66460419 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 31.9172554 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 21.6739616 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20.4382133 0 51.8563881 0 8.29326344 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20.3934097 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 184 | 1 185 | 10 186 | 4 187 | 20 188 | 3 189 | V 1 190 | 17 191 | torch.FloatTensor 192 | 1 193 | 512 194 | 1 195 | 1 196 | 4 197 | 21 198 | 3 199 | V 1 200 | 18 201 | torch.FloatStorage 202 | 512 203 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 48.3215904 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 24.4473724 0 0 0 0 0 0 0 0 0 0 186.425156 0 0 0 0 0 0 0 0 0 0 0 0 4.85952711 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19.7040005 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19.2309704 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 28.8020439 0 44.4254723 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 24.2894039 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 204 | 1 205 | 11 206 | 4 207 | 22 208 | 3 209 | V 1 210 | 17 211 | torch.FloatTensor 212 | 1 213 | 512 214 | 1 215 | 1 216 | 4 217 | 23 218 | 3 219 | V 1 220 | 18 221 | torch.FloatStorage 222 | 512 223 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 48.3792305 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 27.7522945 0 0 0 0 0 0 0 0 0 0 182.708557 0 0 0 0 0 0 0 0 0 0 0 0 0.179861426 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6.27080345 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9.85739708 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6.85564137 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16.0797024 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 28.1864147 0 47.9325066 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 30.3913231 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 224 | 1 225 | 12 226 | 4 227 | 24 228 | 3 229 | V 1 230 | 17 231 | torch.FloatTensor 232 | 1 233 | 512 234 | 1 235 | 1 236 | 4 237 | 25 238 | 3 239 | V 1 240 | 18 241 | torch.FloatStorage 242 | 512 243 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 41.8446083 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 26.8422794 0 0 0 0 0 0 0 0 0 0 190.713165 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6.63244152 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16.2418976 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.73563457 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16.8558006 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16.6317844 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 28.089159 0 39.3096352 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 21.4746494 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 244 | 1 245 | 13 246 | 4 247 | 26 248 | 3 249 | V 1 250 | 17 251 | torch.FloatTensor 252 | 1 253 | 512 254 | 1 255 | 1 256 | 4 257 | 27 258 | 3 259 | V 1 260 | 18 261 | torch.FloatStorage 262 | 512 263 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2.09928846 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 31.5450935 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15.0281086 0 0 0 0 0 0 0 0 0 0 192.79039 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8.85061359 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8.58797169 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.96751618 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 21.0787258 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 23.0694752 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 33.4809456 0 27.988081 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 26.8990536 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 264 | 1 265 | 14 266 | 4 267 | 28 268 | 3 269 | V 1 270 | 17 271 | torch.FloatTensor 272 | 1 273 | 512 274 | 1 275 | 1 276 | 4 277 | 29 278 | 3 279 | V 1 280 | 18 281 | torch.FloatStorage 282 | 512 283 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35.8579521 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13.6722507 0 0 0 0 0 0 0 0 0 0 195.030212 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 17.2185707 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15.6566248 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16.8687534 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 18.9966393 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 32.4675293 0 26.1146374 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 17.6728439 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 284 | 1 285 | 15 286 | 4 287 | 30 288 | 3 289 | V 1 290 | 17 291 | torch.FloatTensor 292 | 1 293 | 512 294 | 1 295 | 1 296 | 4 297 | 31 298 | 3 299 | V 1 300 | 18 301 | torch.FloatStorage 302 | 512 303 | 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.38269663 0 0 0 0 0 0 0 27.4811707 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10.0424509 0 0 0 0 0 0 0 0 0 0 192.128799 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13.8090286 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10.2724047 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 25.1244564 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19.7997684 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 23.8001614 0 23.9346619 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20.3793659 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 304 | -------------------------------------------------------------------------------- /dqn/mynote.sh: -------------------------------------------------------------------------------- 1 | /home/user/torch/install/bin/qlua train_agent_cpu.lua 2 | -------------------------------------------------------------------------------- /dqn/mypara.t7: -------------------------------------------------------------------------------- 1 | 1 2 | 10000000 3 | -------------------------------------------------------------------------------- /dqn/net_downsample_2x_full_y.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (c) 2014 Google Inc. 3 | 4 | See LICENSE file for full terms of limited license. 5 | ]] 6 | 7 | require "image" 8 | require "Scale" 9 | 10 | local function create_network(args) 11 | -- Y (luminance) 12 | return nn.Scale(84, 84, true) 13 | end 14 | 15 | return create_network 16 | -------------------------------------------------------------------------------- /dqn/nnutils.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (c) 2014 Google Inc. 3 | 4 | See LICENSE file for full terms of limited license. 5 | ]] 6 | 7 | require "torch" 8 | 9 | function recursive_map(module, field, func) 10 | local str = "" 11 | if module[field] or module.modules then 12 | str = str .. torch.typename(module) .. ": " 13 | end 14 | if module[field] then 15 | str = str .. func(module[field]) 16 | end 17 | if module.modules then 18 | str = str .. "[" 19 | for i, submodule in ipairs(module.modules) do 20 | local submodule_str = recursive_map(submodule, field, func) 21 | str = str .. submodule_str 22 | if i < #module.modules and string.len(submodule_str) > 0 then 23 | str = str .. " " 24 | end 25 | end 26 | str = str .. "]" 27 | end 28 | 29 | return str 30 | end 31 | 32 | function abs_mean(w) 33 | return torch.mean(torch.abs(w:clone():float())) 34 | end 35 | 36 | function abs_max(w) 37 | return torch.abs(w:clone():float()):max() 38 | end 39 | 40 | -- Build a string of average absolute weight values for the modules in the 41 | -- given network. 42 | function get_weight_norms(module) 43 | return "Weight norms:\n" .. recursive_map(module, "weight", abs_mean) .. 44 | "\nWeight max:\n" .. recursive_map(module, "weight", abs_max) 45 | end 46 | 47 | -- Build a string of average absolute weight gradient values for the modules 48 | -- in the given network. 49 | function get_grad_norms(module) 50 | return "Weight grad norms:\n" .. 51 | recursive_map(module, "gradWeight", abs_mean) .. 52 | "\nWeight grad max:\n" .. recursive_map(module, "gradWeight", abs_max) 53 | end 54 | -------------------------------------------------------------------------------- /dqn/plotthree.m: -------------------------------------------------------------------------------- 1 | clc; 2 | clear all; 3 | close all; 4 | qmax=importdata('qmax.t7'); 5 | length(qmax) 6 | 7 | txtsize=17; 8 | linewid=2.2; 9 | 10 | subplot(1,3,1) 11 | % plot(ex,'color','c') 12 | % hold on 13 | plot(qmax(7:4:length(qmax)),'color','k','linewidth',linewid) 14 | set(gca,'linewidth',2.5,'FontSize',txtsize); 15 | set(gca,'yticklabel',get(gca,'ytick')); 16 | box off; 17 | ylabel('Average action value (Q)','FontSize',txtsize) 18 | xlabel('Training epochs','FontSize',txtsize) 19 | 20 | 21 | subplot(1,3,2) 22 | rewardhistory=importdata('reward_history.t7'); 23 | % plot(ex,'color','c') 24 | % hold on 25 | plot(rewardhistory(7:4:length(rewardhistory)),'color','k','linewidth',linewid) 26 | axis([0 100 -1000 10000]) 27 | set(gca,'linewidth',2.5,'FontSize',txtsize); 28 | set(gca,'yticklabel',get(gca,'ytick')); 29 | box off; 30 | ylabel('Average total reward per episode','FontSize',txtsize) 31 | xlabel('Training epochs','FontSize',txtsize) 32 | 33 | subplot(1,3,3) 34 | stephistory=importdata('step_history.t7'); 35 | % plot(ex,'color','c') 36 | % hold on 37 | 38 | plot(stephistory(7:4:length(stephistory)),'color','k','linewidth',linewid) 39 | axis([0 100 0 10000]) 40 | set(gca,'linewidth',2.5,'FontSize',txtsize); 41 | set(gca,'yticklabel',get(gca,'ytick')); 42 | 43 | box off; 44 | ylabel('Average step per episode','FontSize',txtsize) 45 | xlabel('Training epochs','FontSize',txtsize) 46 | 47 | %ht=subtitle(3,'G4',txtsize) -------------------------------------------------------------------------------- /dqn/qmax.t7: -------------------------------------------------------------------------------- 1 | 3 2 | 1 3 | 99 4 | 1 5 | 1 6 | 1 7 | -0.94052603675127033 8 | 1 9 | 2 10 | 1 11 | -2.0589655377340317 12 | 1 13 | 3 14 | 1 15 | -3.4158912809848787 16 | 1 17 | 4 18 | 1 19 | -4.1567733057403569 20 | 1 21 | 5 22 | 1 23 | -3.9434267574238033 24 | 1 25 | 6 26 | 1 27 | -4.114846419904679 28 | 1 29 | 7 30 | 1 31 | -2.4990199849436432 32 | 1 33 | 8 34 | 1 35 | -1.531911589487791 36 | 1 37 | 9 38 | 1 39 | -0.64734987956923895 40 | 1 41 | 10 42 | 1 43 | 0.54491011246001353 44 | 1 45 | 11 46 | 1 47 | 2.1911295783019042 48 | 1 49 | 12 50 | 1 51 | 3.0740450541308335 52 | 1 53 | 13 54 | 1 55 | 4.4106505028454963 56 | 1 57 | 14 58 | 1 59 | 5.8987613309765239 60 | 1 61 | 15 62 | 1 63 | 7.6281759147756736 64 | 1 65 | 16 66 | 1 67 | 7.5947204641835393 68 | 1 69 | 17 70 | 1 71 | 8.4127548737582565 72 | 1 73 | 18 74 | 1 75 | 10.197525554956645 76 | 1 77 | 19 78 | 1 79 | 9.1705753738239402 80 | 1 81 | 20 82 | 1 83 | 11.171364045099319 84 | 1 85 | 21 86 | 1 87 | 10.675169847895949 88 | 1 89 | 22 90 | 1 91 | 12.425039221524001 92 | 1 93 | 23 94 | 1 95 | 12.337962995467038 96 | 1 97 | 24 98 | 1 99 | 13.860071660462916 100 | 1 101 | 25 102 | 1 103 | 14.5141255852893 104 | 1 105 | 26 106 | 1 107 | 16.901763572939039 108 | 1 109 | 27 110 | 1 111 | 18.926226292175947 112 | 1 113 | 28 114 | 1 115 | 21.987632332718967 116 | 1 117 | 29 118 | 1 119 | 23.42032977346301 120 | 1 121 | 30 122 | 1 123 | 23.547721241955163 124 | 1 125 | 31 126 | 1 127 | 27.800400224268735 128 | 1 129 | 32 130 | 1 131 | 31.072763123872281 132 | 1 133 | 33 134 | 1 135 | 33.853365470071438 136 | 1 137 | 34 138 | 1 139 | 34.10473333477259 140 | 1 141 | 35 142 | 1 143 | 36.838147461956737 144 | 1 145 | 36 146 | 1 147 | 39.703069608470202 148 | 1 149 | 37 150 | 1 151 | 42.080675353623626 152 | 1 153 | 38 154 | 1 155 | 43.409004212523698 156 | 1 157 | 39 158 | 1 159 | 46.310916860136984 160 | 1 161 | 40 162 | 1 163 | 48.443040041227341 164 | 1 165 | 41 166 | 1 167 | 47.786211693658828 168 | 1 169 | 42 170 | 1 171 | 50.065713537521361 172 | 1 173 | 43 174 | 1 175 | 52.712301878614426 176 | 1 177 | 44 178 | 1 179 | 52.935438262395856 180 | 1 181 | 45 182 | 1 183 | 54.076176730604175 184 | 1 185 | 46 186 | 1 187 | 55.292571673243046 188 | 1 189 | 47 190 | 1 191 | 55.520742277717588 192 | 1 193 | 48 194 | 1 195 | 59.474774130783082 196 | 1 197 | 49 198 | 1 199 | 59.549391605949403 200 | 1 201 | 50 202 | 1 203 | 59.772800973854068 204 | 1 205 | 51 206 | 1 207 | 58.511589849444626 208 | 1 209 | 52 210 | 1 211 | 60.915906529779434 212 | 1 213 | 53 214 | 1 215 | 61.223390424737929 216 | 1 217 | 54 218 | 1 219 | 62.193703423328401 220 | 1 221 | 55 222 | 1 223 | 62.865579730796817 224 | 1 225 | 56 226 | 1 227 | 63.286728921699527 228 | 1 229 | 57 230 | 1 231 | 63.105299624414442 232 | 1 233 | 58 234 | 1 235 | 64.54187728328705 236 | 1 237 | 59 238 | 1 239 | 65.585406292667386 240 | 1 241 | 60 242 | 1 243 | 65.208390701065071 244 | 1 245 | 61 246 | 1 247 | 66.86819714420318 248 | 1 249 | 62 250 | 1 251 | 66.4524175623703 252 | 1 253 | 63 254 | 1 255 | 66.974456021690372 256 | 1 257 | 64 258 | 1 259 | 66.621116739120481 260 | 1 261 | 65 262 | 1 263 | 67.51424145103455 264 | 1 265 | 66 266 | 1 267 | 66.961817172737128 268 | 1 269 | 67 270 | 1 271 | 68.84207217830658 272 | 1 273 | 68 274 | 1 275 | 68.419022272720341 276 | 1 277 | 69 278 | 1 279 | 68.627728112449645 280 | 1 281 | 70 282 | 1 283 | 69.047618763351437 284 | 1 285 | 71 286 | 1 287 | 69.348824579629891 288 | 1 289 | 72 290 | 1 291 | 68.430430072212218 292 | 1 293 | 73 294 | 1 295 | 68.882705075340269 296 | 1 297 | 74 298 | 1 299 | 69.553904914417274 300 | 1 301 | 75 302 | 1 303 | 71.363296307601928 304 | 1 305 | 76 306 | 1 307 | 70.32370199645996 308 | 1 309 | 77 310 | 1 311 | 70.605553074111938 312 | 1 313 | 78 314 | 1 315 | 72.345923363037116 316 | 1 317 | 79 318 | 1 319 | 71.242913959960944 320 | 1 321 | 80 322 | 1 323 | 71.846558366584773 324 | 1 325 | 81 326 | 1 327 | 72.415102837638855 328 | 1 329 | 82 330 | 1 331 | 73.257329670600896 332 | 1 333 | 83 334 | 1 335 | 73.205130924224846 336 | 1 337 | 84 338 | 1 339 | 73.102385663833616 340 | 1 341 | 85 342 | 1 343 | 74.024071583747869 344 | 1 345 | 86 346 | 1 347 | 72.849275372085572 348 | 1 349 | 87 350 | 1 351 | 73.214492692222592 352 | 1 353 | 88 354 | 1 355 | 74.137988716011051 356 | 1 357 | 89 358 | 1 359 | 73.46419211212158 360 | 1 361 | 90 362 | 1 363 | 73.570429764213557 364 | 1 365 | 91 366 | 1 367 | 74.286845762672428 368 | 1 369 | 92 370 | 1 371 | 71.505440451774604 372 | 1 373 | 93 374 | 1 375 | 72.979671493072516 376 | 1 377 | 94 378 | 1 379 | 73.192729105567935 380 | 1 381 | 95 382 | 1 383 | 72.207506627140049 384 | 1 385 | 96 386 | 1 387 | 72.484382523994441 388 | 1 389 | 97 390 | 1 391 | 71.849904086685186 392 | 1 393 | 98 394 | 1 395 | 72.26073483739853 396 | 1 397 | 99 398 | 1 399 | 72.176550920009618 400 | -------------------------------------------------------------------------------- /dqn/qmaxtest.t7: -------------------------------------------------------------------------------- 1 | 3 2 | 1 3 | 737 4 | 1 5 | 1 6 | 1 7 | 38.056934356689453 8 | 1 9 | 2 10 | 1 11 | 40.346858978271484 12 | 1 13 | 3 14 | 1 15 | 40.807018280029297 16 | 1 17 | 4 18 | 1 19 | 43.096240997314453 20 | 1 21 | 5 22 | 1 23 | 44.780548095703125 24 | 1 25 | 6 26 | 1 27 | 47.319183349609375 28 | 1 29 | 7 30 | 1 31 | 50.034114837646484 32 | 1 33 | 8 34 | 1 35 | 52.718963623046875 36 | 1 37 | 9 38 | 1 39 | 52.037887573242188 40 | 1 41 | 10 42 | 1 43 | 52.2239990234375 44 | 1 45 | 11 46 | 1 47 | 53.200469970703125 48 | 1 49 | 12 50 | 1 51 | 52.470546722412109 52 | 1 53 | 13 54 | 1 55 | 52.607158660888672 56 | 1 57 | 14 58 | 1 59 | 51.471355438232422 60 | 1 61 | 15 62 | 1 63 | 51.68121337890625 64 | 1 65 | 16 66 | 1 67 | 50.728984832763672 68 | 1 69 | 17 70 | 1 71 | 51.027652740478516 72 | 1 73 | 18 74 | 1 75 | 51.08782958984375 76 | 1 77 | 19 78 | 1 79 | 51.990161895751953 80 | 1 81 | 20 82 | 1 83 | 0 84 | 1 85 | 21 86 | 1 87 | 52.060878753662109 88 | 1 89 | 22 90 | 1 91 | 52.142368316650391 92 | 1 93 | 23 94 | 1 95 | 51.019493103027344 96 | 1 97 | 24 98 | 1 99 | 49.832042694091797 100 | 1 101 | 25 102 | 1 103 | 52.322837829589844 104 | 1 105 | 26 106 | 1 107 | 53.875595092773438 108 | 1 109 | 27 110 | 1 111 | 51.985164642333984 112 | 1 113 | 28 114 | 1 115 | 54.878089904785156 116 | 1 117 | 29 118 | 1 119 | 53.213977813720703 120 | 1 121 | 30 122 | 1 123 | 53.048057556152344 124 | 1 125 | 31 126 | 1 127 | 52.347286224365234 128 | 1 129 | 32 130 | 1 131 | 53.260871887207031 132 | 1 133 | 33 134 | 1 135 | 53.510211944580078 136 | 1 137 | 34 138 | 1 139 | 54.876670837402344 140 | 1 141 | 35 142 | 1 143 | 50.878734588623047 144 | 1 145 | 36 146 | 1 147 | 52.962947845458984 148 | 1 149 | 37 150 | 1 151 | 53.1766357421875 152 | 1 153 | 38 154 | 1 155 | 54.138813018798828 156 | 1 157 | 39 158 | 1 159 | 53.55242919921875 160 | 1 161 | 40 162 | 1 163 | 51.606178283691406 164 | 1 165 | 41 166 | 1 167 | 52.663486480712891 168 | 1 169 | 42 170 | 1 171 | 51.328365325927734 172 | 1 173 | 43 174 | 1 175 | 51.952907562255859 176 | 1 177 | 44 178 | 1 179 | 53.271541595458984 180 | 1 181 | 45 182 | 1 183 | 54.620510101318359 184 | 1 185 | 46 186 | 1 187 | 0 188 | 1 189 | 47 190 | 1 191 | 52.408843994140625 192 | 1 193 | 48 194 | 1 195 | 54.483821868896484 196 | 1 197 | 49 198 | 1 199 | 53.624858856201172 200 | 1 201 | 50 202 | 1 203 | 51.395343780517578 204 | 1 205 | 51 206 | 1 207 | 54.583663940429688 208 | 1 209 | 52 210 | 1 211 | 56.450962066650391 212 | 1 213 | 53 214 | 1 215 | 54.141151428222656 216 | 1 217 | 54 218 | 1 219 | 59.03997802734375 220 | 1 221 | 55 222 | 1 223 | 56.731029510498047 224 | 1 225 | 56 226 | 1 227 | 59.409915924072266 228 | 1 229 | 57 230 | 1 231 | 58.673591613769531 232 | 1 233 | 58 234 | 1 235 | 53.102706909179688 236 | 1 237 | 59 238 | 1 239 | 54.309459686279297 240 | 1 241 | 60 242 | 1 243 | 53.875045776367188 244 | 1 245 | 61 246 | 1 247 | 54.868156433105469 248 | 1 249 | 62 250 | 1 251 | 56.032585144042969 252 | 1 253 | 63 254 | 1 255 | 56.405452728271484 256 | 1 257 | 64 258 | 1 259 | 59.038017272949219 260 | 1 261 | 65 262 | 1 263 | 54.165988922119141 264 | 1 265 | 66 266 | 1 267 | 58.144172668457031 268 | 1 269 | 67 270 | 1 271 | 58.504894256591797 272 | 1 273 | 68 274 | 1 275 | 0 276 | 1 277 | 69 278 | 1 279 | 0 280 | 1 281 | 70 282 | 1 283 | 54.391822814941406 284 | 1 285 | 71 286 | 1 287 | 56.431629180908203 288 | 1 289 | 72 290 | 1 291 | 64.52105712890625 292 | 1 293 | 73 294 | 1 295 | 62.786304473876953 296 | 1 297 | 74 298 | 1 299 | 61.214916229248047 300 | 1 301 | 75 302 | 1 303 | 67.216110229492188 304 | 1 305 | 76 306 | 1 307 | 68.947250366210938 308 | 1 309 | 77 310 | 1 311 | 66.888946533203125 312 | 1 313 | 78 314 | 1 315 | 67.682899475097656 316 | 1 317 | 79 318 | 1 319 | 67.680389404296875 320 | 1 321 | 80 322 | 1 323 | 65.709762573242188 324 | 1 325 | 81 326 | 1 327 | 66.512725830078125 328 | 1 329 | 82 330 | 1 331 | 67.21197509765625 332 | 1 333 | 83 334 | 1 335 | 69.4805908203125 336 | 1 337 | 84 338 | 1 339 | 68.398223876953125 340 | 1 341 | 85 342 | 1 343 | 66.73211669921875 344 | 1 345 | 86 346 | 1 347 | 68.82855224609375 348 | 1 349 | 87 350 | 1 351 | 68.786056518554688 352 | 1 353 | 88 354 | 1 355 | 66.921859741210938 356 | 1 357 | 89 358 | 1 359 | 67.441093444824219 360 | 1 361 | 90 362 | 1 363 | 67.461334228515625 364 | 1 365 | 91 366 | 1 367 | 67.543411254882812 368 | 1 369 | 92 370 | 1 371 | 0 372 | 1 373 | 93 374 | 1 375 | 63.748893737792969 376 | 1 377 | 94 378 | 1 379 | 63.255157470703125 380 | 1 381 | 95 382 | 1 383 | 63.053901672363281 384 | 1 385 | 96 386 | 1 387 | 64.932723999023438 388 | 1 389 | 97 390 | 1 391 | 62.215663909912109 392 | 1 393 | 98 394 | 1 395 | 63.359489440917969 396 | 1 397 | 99 398 | 1 399 | 65.133415222167969 400 | 1 401 | 100 402 | 1 403 | 65.854537963867188 404 | 1 405 | 101 406 | 1 407 | 64.66314697265625 408 | 1 409 | 102 410 | 1 411 | 59.982913970947266 412 | 1 413 | 103 414 | 1 415 | 59.673755645751953 416 | 1 417 | 104 418 | 1 419 | 60.670639038085938 420 | 1 421 | 105 422 | 1 423 | 63.422554016113281 424 | 1 425 | 106 426 | 1 427 | 60.812511444091797 428 | 1 429 | 107 430 | 1 431 | 57.351352691650391 432 | 1 433 | 108 434 | 1 435 | 53.785427093505859 436 | 1 437 | 109 438 | 1 439 | 49.490150451660156 440 | 1 441 | 110 442 | 1 443 | 49.492443084716797 444 | 1 445 | 111 446 | 1 447 | 52.520076751708984 448 | 1 449 | 112 450 | 1 451 | 50.363273620605469 452 | 1 453 | 113 454 | 1 455 | 50.234184265136719 456 | 1 457 | 114 458 | 1 459 | 50.8941650390625 460 | 1 461 | 115 462 | 1 463 | 49.741958618164062 464 | 1 465 | 116 466 | 1 467 | 50.492462158203125 468 | 1 469 | 117 470 | 1 471 | 53.265869140625 472 | 1 473 | 118 474 | 1 475 | 53.467304229736328 476 | 1 477 | 119 478 | 1 479 | 46.896629333496094 480 | 1 481 | 120 482 | 1 483 | 0 484 | 1 485 | 121 486 | 1 487 | 43.186725616455078 488 | 1 489 | 122 490 | 1 491 | 44.426822662353516 492 | 1 493 | 123 494 | 1 495 | 0 496 | 1 497 | 124 498 | 1 499 | 44.578559875488281 500 | 1 501 | 125 502 | 1 503 | 43.030876159667969 504 | 1 505 | 126 506 | 1 507 | 44.40911865234375 508 | 1 509 | 127 510 | 1 511 | 38.167301177978516 512 | 1 513 | 128 514 | 1 515 | 35.750946044921875 516 | 1 517 | 129 518 | 1 519 | 0 520 | 1 521 | 130 522 | 1 523 | 38.536155700683594 524 | 1 525 | 131 526 | 1 527 | 29.725673675537109 528 | 1 529 | 132 530 | 1 531 | 35.257919311523438 532 | 1 533 | 133 534 | 1 535 | 37.817710876464844 536 | 1 537 | 134 538 | 1 539 | 39.817806243896484 540 | 1 541 | 135 542 | 1 543 | 41.225353240966797 544 | 1 545 | 136 546 | 1 547 | 41.658573150634766 548 | 1 549 | 137 550 | 1 551 | 41.776111602783203 552 | 1 553 | 138 554 | 1 555 | 42.112602233886719 556 | 1 557 | 139 558 | 1 559 | 43.435779571533203 560 | 1 561 | 140 562 | 1 563 | 42.865283966064453 564 | 1 565 | 141 566 | 1 567 | 42.263668060302734 568 | 1 569 | 142 570 | 1 571 | 43.231643676757812 572 | 1 573 | 143 574 | 1 575 | 42.664886474609375 576 | 1 577 | 144 578 | 1 579 | 42.897319793701172 580 | 1 581 | 145 582 | 1 583 | 42.183818817138672 584 | 1 585 | 146 586 | 1 587 | 42.805538177490234 588 | 1 589 | 147 590 | 1 591 | 41.155704498291016 592 | 1 593 | 148 594 | 1 595 | 39.10052490234375 596 | 1 597 | 149 598 | 1 599 | 40.414905548095703 600 | 1 601 | 150 602 | 1 603 | 41.130714416503906 604 | 1 605 | 151 606 | 1 607 | 40.134853363037109 608 | 1 609 | 152 610 | 1 611 | 41.507804870605469 612 | 1 613 | 153 614 | 1 615 | 41.620712280273438 616 | 1 617 | 154 618 | 1 619 | 42.011440277099609 620 | 1 621 | 155 622 | 1 623 | 41.812904357910156 624 | 1 625 | 156 626 | 1 627 | 41.849288940429688 628 | 1 629 | 157 630 | 1 631 | 42.585708618164062 632 | 1 633 | 158 634 | 1 635 | 43.593338012695312 636 | 1 637 | 159 638 | 1 639 | 0 640 | 1 641 | 160 642 | 1 643 | 40.049442291259766 644 | 1 645 | 161 646 | 1 647 | 40.519454956054688 648 | 1 649 | 162 650 | 1 651 | 42.724800109863281 652 | 1 653 | 163 654 | 1 655 | 41.2593994140625 656 | 1 657 | 164 658 | 1 659 | 43.116519927978516 660 | 1 661 | 165 662 | 1 663 | 44.292640686035156 664 | 1 665 | 166 666 | 1 667 | 42.944808959960938 668 | 1 669 | 167 670 | 1 671 | 43.92144775390625 672 | 1 673 | 168 674 | 1 675 | 38.3319091796875 676 | 1 677 | 169 678 | 1 679 | 42.923069000244141 680 | 1 681 | 170 682 | 1 683 | 41.915149688720703 684 | 1 685 | 171 686 | 1 687 | 39.773105621337891 688 | 1 689 | 172 690 | 1 691 | 39.037326812744141 692 | 1 693 | 173 694 | 1 695 | 42.4510498046875 696 | 1 697 | 174 698 | 1 699 | 43.746467590332031 700 | 1 701 | 175 702 | 1 703 | 44.6651611328125 704 | 1 705 | 176 706 | 1 707 | 44.635494232177734 708 | 1 709 | 177 710 | 1 711 | 43.994987487792969 712 | 1 713 | 178 714 | 1 715 | 41.191608428955078 716 | 1 717 | 179 718 | 1 719 | 41.010810852050781 720 | 1 721 | 180 722 | 1 723 | 0 724 | 1 725 | 181 726 | 1 727 | 36.916889190673828 728 | 1 729 | 182 730 | 1 731 | 38.593112945556641 732 | 1 733 | 183 734 | 1 735 | 36.553256988525391 736 | 1 737 | 184 738 | 1 739 | 36.048843383789062 740 | 1 741 | 185 742 | 1 743 | 37.2176513671875 744 | 1 745 | 186 746 | 1 747 | 36.440769195556641 748 | 1 749 | 187 750 | 1 751 | 0 752 | 1 753 | 188 754 | 1 755 | 35.606182098388672 756 | 1 757 | 189 758 | 1 759 | 35.976318359375 760 | 1 761 | 190 762 | 1 763 | 33.787914276123047 764 | 1 765 | 191 766 | 1 767 | 34.227764129638672 768 | 1 769 | 192 770 | 1 771 | 37.012981414794922 772 | 1 773 | 193 774 | 1 775 | 36.105152130126953 776 | 1 777 | 194 778 | 1 779 | 37.571189880371094 780 | 1 781 | 195 782 | 1 783 | 39.042564392089844 784 | 1 785 | 196 786 | 1 787 | 0 788 | 1 789 | 197 790 | 1 791 | 36.362323760986328 792 | 1 793 | 198 794 | 1 795 | 35.714473724365234 796 | 1 797 | 199 798 | 1 799 | 36.276664733886719 800 | 1 801 | 200 802 | 1 803 | 37.082481384277344 804 | 1 805 | 201 806 | 1 807 | 36.220779418945312 808 | 1 809 | 202 810 | 1 811 | 38.90966796875 812 | 1 813 | 203 814 | 1 815 | 39.672065734863281 816 | 1 817 | 204 818 | 1 819 | 39.991737365722656 820 | 1 821 | 205 822 | 1 823 | 38.766265869140625 824 | 1 825 | 206 826 | 1 827 | 36.689777374267578 828 | 1 829 | 207 830 | 1 831 | 39.492080688476562 832 | 1 833 | 208 834 | 1 835 | 42.530265808105469 836 | 1 837 | 209 838 | 1 839 | 44.439666748046875 840 | 1 841 | 210 842 | 1 843 | 41.213424682617188 844 | 1 845 | 211 846 | 1 847 | 41.219276428222656 848 | 1 849 | 212 850 | 1 851 | 42.597793579101562 852 | 1 853 | 213 854 | 1 855 | 39.806655883789062 856 | 1 857 | 214 858 | 1 859 | 41.204608917236328 860 | 1 861 | 215 862 | 1 863 | 44.167194366455078 864 | 1 865 | 216 866 | 1 867 | 44.024875640869141 868 | 1 869 | 217 870 | 1 871 | 47.188247680664062 872 | 1 873 | 218 874 | 1 875 | 45.024448394775391 876 | 1 877 | 219 878 | 1 879 | 0 880 | 1 881 | 220 882 | 1 883 | 39.596240997314453 884 | 1 885 | 221 886 | 1 887 | 43.041313171386719 888 | 1 889 | 222 890 | 1 891 | 31.177335739135742 892 | 1 893 | 223 894 | 1 895 | 38.138389587402344 896 | 1 897 | 224 898 | 1 899 | 0 900 | 1 901 | 225 902 | 1 903 | 25.902925491333008 904 | 1 905 | 226 906 | 1 907 | 0 908 | 1 909 | 227 910 | 1 911 | 33.034351348876953 912 | 1 913 | 228 914 | 1 915 | 0 916 | 1 917 | 229 918 | 1 919 | 26.848373413085938 920 | 1 921 | 230 922 | 1 923 | 57.685756683349609 924 | 1 925 | 231 926 | 1 927 | 61.741416931152344 928 | 1 929 | 232 930 | 1 931 | 61.58538818359375 932 | 1 933 | 233 934 | 1 935 | 59.789390563964844 936 | 1 937 | 234 938 | 1 939 | 58.254905700683594 940 | 1 941 | 235 942 | 1 943 | 57.419010162353516 944 | 1 945 | 236 946 | 1 947 | 57.912334442138672 948 | 1 949 | 237 950 | 1 951 | 56.41156005859375 952 | 1 953 | 238 954 | 1 955 | 54.752494812011719 956 | 1 957 | 239 958 | 1 959 | 54.469245910644531 960 | 1 961 | 240 962 | 1 963 | 0 964 | 1 965 | 241 966 | 1 967 | 55.306217193603516 968 | 1 969 | 242 970 | 1 971 | 52.592288970947266 972 | 1 973 | 243 974 | 1 975 | 53.850139617919922 976 | 1 977 | 244 978 | 1 979 | 53.182655334472656 980 | 1 981 | 245 982 | 1 983 | 53.31988525390625 984 | 1 985 | 246 986 | 1 987 | 53.274410247802734 988 | 1 989 | 247 990 | 1 991 | 53.545566558837891 992 | 1 993 | 248 994 | 1 995 | 52.869918823242188 996 | 1 997 | 249 998 | 1 999 | 50.68560791015625 1000 | 1 1001 | 250 1002 | 1 1003 | 52.605575561523438 1004 | 1 1005 | 251 1006 | 1 1007 | 53.908241271972656 1008 | 1 1009 | 252 1010 | 1 1011 | 55.713119506835938 1012 | 1 1013 | 253 1014 | 1 1015 | 58.166183471679688 1016 | 1 1017 | 254 1018 | 1 1019 | 55.630157470703125 1020 | 1 1021 | 255 1022 | 1 1023 | 54.959026336669922 1024 | 1 1025 | 256 1026 | 1 1027 | 56.543300628662109 1028 | 1 1029 | 257 1030 | 1 1031 | 56.329357147216797 1032 | 1 1033 | 258 1034 | 1 1035 | 54.323993682861328 1036 | 1 1037 | 259 1038 | 1 1039 | 52.104171752929688 1040 | 1 1041 | 260 1042 | 1 1043 | 53.363311767578125 1044 | 1 1045 | 261 1046 | 1 1047 | 51.045242309570312 1048 | 1 1049 | 262 1050 | 1 1051 | 0 1052 | 1 1053 | 263 1054 | 1 1055 | 51.555316925048828 1056 | 1 1057 | 264 1058 | 1 1059 | 49.777088165283203 1060 | 1 1061 | 265 1062 | 1 1063 | 50.013458251953125 1064 | 1 1065 | 266 1066 | 1 1067 | 49.978160858154297 1068 | 1 1069 | 267 1070 | 1 1071 | 49.235801696777344 1072 | 1 1073 | 268 1074 | 1 1075 | 48.980941772460938 1076 | 1 1077 | 269 1078 | 1 1079 | 0 1080 | 1 1081 | 270 1082 | 1 1083 | 50.297389984130859 1084 | 1 1085 | 271 1086 | 1 1087 | 52.063022613525391 1088 | 1 1089 | 272 1090 | 1 1091 | 0 1092 | 1 1093 | 273 1094 | 1 1095 | 49.278251647949219 1096 | 1 1097 | 274 1098 | 1 1099 | 49.414676666259766 1100 | 1 1101 | 275 1102 | 1 1103 | 49.142383575439453 1104 | 1 1105 | 276 1106 | 1 1107 | 48.620330810546875 1108 | 1 1109 | 277 1110 | 1 1111 | 49.869239807128906 1112 | 1 1113 | 278 1114 | 1 1115 | 51.323089599609375 1116 | 1 1117 | 279 1118 | 1 1119 | 50.263313293457031 1120 | 1 1121 | 280 1122 | 1 1123 | 50.760765075683594 1124 | 1 1125 | 281 1126 | 1 1127 | 50.45611572265625 1128 | 1 1129 | 282 1130 | 1 1131 | 50.661293029785156 1132 | 1 1133 | 283 1134 | 1 1135 | 47.874553680419922 1136 | 1 1137 | 284 1138 | 1 1139 | 47.542655944824219 1140 | 1 1141 | 285 1142 | 1 1143 | 46.510936737060547 1144 | 1 1145 | 286 1146 | 1 1147 | 46.093475341796875 1148 | 1 1149 | 287 1150 | 1 1151 | 44.066703796386719 1152 | 1 1153 | 288 1154 | 1 1155 | 44.107276916503906 1156 | 1 1157 | 289 1158 | 1 1159 | 43.871932983398438 1160 | 1 1161 | 290 1162 | 1 1163 | 45.114593505859375 1164 | 1 1165 | 291 1166 | 1 1167 | 43.011306762695312 1168 | 1 1169 | 292 1170 | 1 1171 | 43.475311279296875 1172 | 1 1173 | 293 1174 | 1 1175 | 44.547714233398438 1176 | 1 1177 | 294 1178 | 1 1179 | 45.716682434082031 1180 | 1 1181 | 295 1182 | 1 1183 | 45.854957580566406 1184 | 1 1185 | 296 1186 | 1 1187 | 47.137542724609375 1188 | 1 1189 | 297 1190 | 1 1191 | 45.907520294189453 1192 | 1 1193 | 298 1194 | 1 1195 | 44.429847717285156 1196 | 1 1197 | 299 1198 | 1 1199 | 41.528274536132812 1200 | 1 1201 | 300 1202 | 1 1203 | 41.195125579833984 1204 | 1 1205 | 301 1206 | 1 1207 | 41.171836853027344 1208 | 1 1209 | 302 1210 | 1 1211 | 40.660011291503906 1212 | 1 1213 | 303 1214 | 1 1215 | 42.921478271484375 1216 | 1 1217 | 304 1218 | 1 1219 | 0 1220 | 1 1221 | 305 1222 | 1 1223 | 42.168521881103516 1224 | 1 1225 | 306 1226 | 1 1227 | 40.575443267822266 1228 | 1 1229 | 307 1230 | 1 1231 | 0 1232 | 1 1233 | 308 1234 | 1 1235 | 36.799968719482422 1236 | 1 1237 | 309 1238 | 1 1239 | 39.282085418701172 1240 | 1 1241 | 310 1242 | 1 1243 | 38.757099151611328 1244 | 1 1245 | 311 1246 | 1 1247 | 41.079765319824219 1248 | 1 1249 | 312 1250 | 1 1251 | 44.588676452636719 1252 | 1 1253 | 313 1254 | 1 1255 | 43.90435791015625 1256 | 1 1257 | 314 1258 | 1 1259 | 43.966800689697266 1260 | 1 1261 | 315 1262 | 1 1263 | 44.217315673828125 1264 | 1 1265 | 316 1266 | 1 1267 | 46.651069641113281 1268 | 1 1269 | 317 1270 | 1 1271 | 45.339298248291016 1272 | 1 1273 | 318 1274 | 1 1275 | 45.857257843017578 1276 | 1 1277 | 319 1278 | 1 1279 | 48.064262390136719 1280 | 1 1281 | 320 1282 | 1 1283 | 47.191753387451172 1284 | 1 1285 | 321 1286 | 1 1287 | 52.151378631591797 1288 | 1 1289 | 322 1290 | 1 1291 | 49.319709777832031 1292 | 1 1293 | 323 1294 | 1 1295 | 42.722930908203125 1296 | 1 1297 | 324 1298 | 1 1299 | 52.456027984619141 1300 | 1 1301 | 325 1302 | 1 1303 | 49.650390625 1304 | 1 1305 | 326 1306 | 1 1307 | 59.803943634033203 1308 | 1 1309 | 327 1310 | 1 1311 | 61.563869476318359 1312 | 1 1313 | 328 1314 | 1 1315 | 61.515827178955078 1316 | 1 1317 | 329 1318 | 1 1319 | 60.878810882568359 1320 | 1 1321 | 330 1322 | 1 1323 | 62.861858367919922 1324 | 1 1325 | 331 1326 | 1 1327 | 61.305515289306641 1328 | 1 1329 | 332 1330 | 1 1331 | 64.863754272460938 1332 | 1 1333 | 333 1334 | 1 1335 | 58.5421142578125 1336 | 1 1337 | 334 1338 | 1 1339 | 61.964885711669922 1340 | 1 1341 | 335 1342 | 1 1343 | 62.829944610595703 1344 | 1 1345 | 336 1346 | 1 1347 | 61.546672821044922 1348 | 1 1349 | 337 1350 | 1 1351 | 61.820579528808594 1352 | 1 1353 | 338 1354 | 1 1355 | 65.403205871582031 1356 | 1 1357 | 339 1358 | 1 1359 | 68.82568359375 1360 | 1 1361 | 340 1362 | 1 1363 | 70.853645324707031 1364 | 1 1365 | 341 1366 | 1 1367 | 67.763046264648438 1368 | 1 1369 | 342 1370 | 1 1371 | 69.194015502929688 1372 | 1 1373 | 343 1374 | 1 1375 | 66.981964111328125 1376 | 1 1377 | 344 1378 | 1 1379 | 64.669189453125 1380 | 1 1381 | 345 1382 | 1 1383 | 61.596946716308594 1384 | 1 1385 | 346 1386 | 1 1387 | 61.827602386474609 1388 | 1 1389 | 347 1390 | 1 1391 | 60.304763793945312 1392 | 1 1393 | 348 1394 | 1 1395 | 60.478401184082031 1396 | 1 1397 | 349 1398 | 1 1399 | 60.516109466552734 1400 | 1 1401 | 350 1402 | 1 1403 | 61.474929809570312 1404 | 1 1405 | 351 1406 | 1 1407 | 64.402587890625 1408 | 1 1409 | 352 1410 | 1 1411 | 63.159137725830078 1412 | 1 1413 | 353 1414 | 1 1415 | 62.208076477050781 1416 | 1 1417 | 354 1418 | 1 1419 | 63.545524597167969 1420 | 1 1421 | 355 1422 | 1 1423 | 62.672519683837891 1424 | 1 1425 | 356 1426 | 1 1427 | 59.757118225097656 1428 | 1 1429 | 357 1430 | 1 1431 | 56.658042907714844 1432 | 1 1433 | 358 1434 | 1 1435 | 57.235862731933594 1436 | 1 1437 | 359 1438 | 1 1439 | 56.501148223876953 1440 | 1 1441 | 360 1442 | 1 1443 | 55.973930358886719 1444 | 1 1445 | 361 1446 | 1 1447 | 57.576503753662109 1448 | 1 1449 | 362 1450 | 1 1451 | 59.713157653808594 1452 | 1 1453 | 363 1454 | 1 1455 | 59.672142028808594 1456 | 1 1457 | 364 1458 | 1 1459 | 58.883846282958984 1460 | 1 1461 | 365 1462 | 1 1463 | 57.935794830322266 1464 | 1 1465 | 366 1466 | 1 1467 | 0 1468 | 1 1469 | 367 1470 | 1 1471 | 56.721355438232422 1472 | 1 1473 | 368 1474 | 1 1475 | 56.797920227050781 1476 | 1 1477 | 369 1478 | 1 1479 | 55.610248565673828 1480 | 1 1481 | 370 1482 | 1 1483 | 54.680278778076172 1484 | 1 1485 | 371 1486 | 1 1487 | 54.158809661865234 1488 | 1 1489 | 372 1490 | 1 1491 | 55.242687225341797 1492 | 1 1493 | 373 1494 | 1 1495 | 54.432723999023438 1496 | 1 1497 | 374 1498 | 1 1499 | 52.556758880615234 1500 | 1 1501 | 375 1502 | 1 1503 | 49.688835144042969 1504 | 1 1505 | 376 1506 | 1 1507 | 51.154045104980469 1508 | 1 1509 | 377 1510 | 1 1511 | 51.707084655761719 1512 | 1 1513 | 378 1514 | 1 1515 | 51.135391235351562 1516 | 1 1517 | 379 1518 | 1 1519 | 0 1520 | 1 1521 | 380 1522 | 1 1523 | 51.447715759277344 1524 | 1 1525 | 381 1526 | 1 1527 | 49.534767150878906 1528 | 1 1529 | 382 1530 | 1 1531 | 46.768169403076172 1532 | 1 1533 | 383 1534 | 1 1535 | 47.564785003662109 1536 | 1 1537 | 384 1538 | 1 1539 | 46.879001617431641 1540 | 1 1541 | 385 1542 | 1 1543 | 46.392654418945312 1544 | 1 1545 | 386 1546 | 1 1547 | 44.303207397460938 1548 | 1 1549 | 387 1550 | 1 1551 | 0 1552 | 1 1553 | 388 1554 | 1 1555 | 41.196395874023438 1556 | 1 1557 | 389 1558 | 1 1559 | 39.788276672363281 1560 | 1 1561 | 390 1562 | 1 1563 | 40.055290222167969 1564 | 1 1565 | 391 1566 | 1 1567 | 38.906982421875 1568 | 1 1569 | 392 1570 | 1 1571 | 38.346435546875 1572 | 1 1573 | 393 1574 | 1 1575 | 36.833522796630859 1576 | 1 1577 | 394 1578 | 1 1579 | 35.62744140625 1580 | 1 1581 | 395 1582 | 1 1583 | 35.610946655273438 1584 | 1 1585 | 396 1586 | 1 1587 | 34.86431884765625 1588 | 1 1589 | 397 1590 | 1 1591 | 0 1592 | 1 1593 | 398 1594 | 1 1595 | 35.034263610839844 1596 | 1 1597 | 399 1598 | 1 1599 | 34.993061065673828 1600 | 1 1601 | 400 1602 | 1 1603 | 32.532257080078125 1604 | 1 1605 | 401 1606 | 1 1607 | 32.562553405761719 1608 | 1 1609 | 402 1610 | 1 1611 | 28.18474006652832 1612 | 1 1613 | 403 1614 | 1 1615 | 28.437402725219727 1616 | 1 1617 | 404 1618 | 1 1619 | 28.144992828369141 1620 | 1 1621 | 405 1622 | 1 1623 | 0 1624 | 1 1625 | 406 1626 | 1 1627 | 0 1628 | 1 1629 | 407 1630 | 1 1631 | 27.448335647583008 1632 | 1 1633 | 408 1634 | 1 1635 | 23.793703079223633 1636 | 1 1637 | 409 1638 | 1 1639 | 23.338033676147461 1640 | 1 1641 | 410 1642 | 1 1643 | 22.114604949951172 1644 | 1 1645 | 411 1646 | 1 1647 | 26.099836349487305 1648 | 1 1649 | 412 1650 | 1 1651 | 0 1652 | 1 1653 | 413 1654 | 1 1655 | 28.489349365234375 1656 | 1 1657 | 414 1658 | 1 1659 | 35.78790283203125 1660 | 1 1661 | 415 1662 | 1 1663 | 35.532924652099609 1664 | 1 1665 | 416 1666 | 1 1667 | 35.257556915283203 1668 | 1 1669 | 417 1670 | 1 1671 | 27.820722579956055 1672 | 1 1673 | 418 1674 | 1 1675 | 32.545787811279297 1676 | 1 1677 | 419 1678 | 1 1679 | 43.310234069824219 1680 | 1 1681 | 420 1682 | 1 1683 | 43.266075134277344 1684 | 1 1685 | 421 1686 | 1 1687 | 47.482456207275391 1688 | 1 1689 | 422 1690 | 1 1691 | 48.885757446289062 1692 | 1 1693 | 423 1694 | 1 1695 | 50.169349670410156 1696 | 1 1697 | 424 1698 | 1 1699 | 50.301036834716797 1700 | 1 1701 | 425 1702 | 1 1703 | 51.448879241943359 1704 | 1 1705 | 426 1706 | 1 1707 | 53.972366333007812 1708 | 1 1709 | 427 1710 | 1 1711 | 59.210594177246094 1712 | 1 1713 | 428 1714 | 1 1715 | 60.85772705078125 1716 | 1 1717 | 429 1718 | 1 1719 | 60.591846466064453 1720 | 1 1721 | 430 1722 | 1 1723 | 60.017608642578125 1724 | 1 1725 | 431 1726 | 1 1727 | 0 1728 | 1 1729 | 432 1730 | 1 1731 | 57.352512359619141 1732 | 1 1733 | 433 1734 | 1 1735 | 57.568294525146484 1736 | 1 1737 | 434 1738 | 1 1739 | 59.909133911132812 1740 | 1 1741 | 435 1742 | 1 1743 | 56.616863250732422 1744 | 1 1745 | 436 1746 | 1 1747 | 54.939018249511719 1748 | 1 1749 | 437 1750 | 1 1751 | 56.394683837890625 1752 | 1 1753 | 438 1754 | 1 1755 | 57.710826873779297 1756 | 1 1757 | 439 1758 | 1 1759 | 57.947139739990234 1760 | 1 1761 | 440 1762 | 1 1763 | 55.136207580566406 1764 | 1 1765 | 441 1766 | 1 1767 | 56.280109405517578 1768 | 1 1769 | 442 1770 | 1 1771 | 57.726768493652344 1772 | 1 1773 | 443 1774 | 1 1775 | 57.782093048095703 1776 | 1 1777 | 444 1778 | 1 1779 | 59.483699798583984 1780 | 1 1781 | 445 1782 | 1 1783 | 58.899120330810547 1784 | 1 1785 | 446 1786 | 1 1787 | 0 1788 | 1 1789 | 447 1790 | 1 1791 | 54.316867828369141 1792 | 1 1793 | 448 1794 | 1 1795 | 54.026603698730469 1796 | 1 1797 | 449 1798 | 1 1799 | 53.572284698486328 1800 | 1 1801 | 450 1802 | 1 1803 | 55.074253082275391 1804 | 1 1805 | 451 1806 | 1 1807 | 54.514003753662109 1808 | 1 1809 | 452 1810 | 1 1811 | 53.255668640136719 1812 | 1 1813 | 453 1814 | 1 1815 | 53.296974182128906 1816 | 1 1817 | 454 1818 | 1 1819 | 52.985847473144531 1820 | 1 1821 | 455 1822 | 1 1823 | 54.540348052978516 1824 | 1 1825 | 456 1826 | 1 1827 | 53.695064544677734 1828 | 1 1829 | 457 1830 | 1 1831 | 51.827522277832031 1832 | 1 1833 | 458 1834 | 1 1835 | 51.713054656982422 1836 | 1 1837 | 459 1838 | 1 1839 | 53.039756774902344 1840 | 1 1841 | 460 1842 | 1 1843 | 49.20989990234375 1844 | 1 1845 | 461 1846 | 1 1847 | 48.694316864013672 1848 | 1 1849 | 462 1850 | 1 1851 | 48.334049224853516 1852 | 1 1853 | 463 1854 | 1 1855 | 48.373725891113281 1856 | 1 1857 | 464 1858 | 1 1859 | 48.819728851318359 1860 | 1 1861 | 465 1862 | 1 1863 | 0 1864 | 1 1865 | 466 1866 | 1 1867 | 46.440998077392578 1868 | 1 1869 | 467 1870 | 1 1871 | 0 1872 | 1 1873 | 468 1874 | 1 1875 | 47.289047241210938 1876 | 1 1877 | 469 1878 | 1 1879 | 0 1880 | 1 1881 | 470 1882 | 1 1883 | 46.967029571533203 1884 | 1 1885 | 471 1886 | 1 1887 | 47.538700103759766 1888 | 1 1889 | 472 1890 | 1 1891 | 48.018245697021484 1892 | 1 1893 | 473 1894 | 1 1895 | 45.917354583740234 1896 | 1 1897 | 474 1898 | 1 1899 | 0 1900 | 1 1901 | 475 1902 | 1 1903 | 45.486804962158203 1904 | 1 1905 | 476 1906 | 1 1907 | 45.697910308837891 1908 | 1 1909 | 477 1910 | 1 1911 | 46.535400390625 1912 | 1 1913 | 478 1914 | 1 1915 | 44.740287780761719 1916 | 1 1917 | 479 1918 | 1 1919 | 43.816535949707031 1920 | 1 1921 | 480 1922 | 1 1923 | 44.679088592529297 1924 | 1 1925 | 481 1926 | 1 1927 | 44.516727447509766 1928 | 1 1929 | 482 1930 | 1 1931 | 45.575565338134766 1932 | 1 1933 | 483 1934 | 1 1935 | 46.02734375 1936 | 1 1937 | 484 1938 | 1 1939 | 42.993186950683594 1940 | 1 1941 | 485 1942 | 1 1943 | 44.851818084716797 1944 | 1 1945 | 486 1946 | 1 1947 | 42.958911895751953 1948 | 1 1949 | 487 1950 | 1 1951 | 42.995914459228516 1952 | 1 1953 | 488 1954 | 1 1955 | 44.714370727539062 1956 | 1 1957 | 489 1958 | 1 1959 | 40.863685607910156 1960 | 1 1961 | 490 1962 | 1 1963 | 42.404205322265625 1964 | 1 1965 | 491 1966 | 1 1967 | 45.395305633544922 1968 | 1 1969 | 492 1970 | 1 1971 | 43.168075561523438 1972 | 1 1973 | 493 1974 | 1 1975 | 43.777767181396484 1976 | 1 1977 | 494 1978 | 1 1979 | 46.158138275146484 1980 | 1 1981 | 495 1982 | 1 1983 | 47.241580963134766 1984 | 1 1985 | 496 1986 | 1 1987 | 48.583316802978516 1988 | 1 1989 | 497 1990 | 1 1991 | 0 1992 | 1 1993 | 498 1994 | 1 1995 | 48.822032928466797 1996 | 1 1997 | 499 1998 | 1 1999 | 52.766365051269531 2000 | 1 2001 | 500 2002 | 1 2003 | 53.364658355712891 2004 | 1 2005 | 501 2006 | 1 2007 | 52.833366394042969 2008 | 1 2009 | 502 2010 | 1 2011 | 56.949058532714844 2012 | 1 2013 | 503 2014 | 1 2015 | 0 2016 | 1 2017 | 504 2018 | 1 2019 | 60.394939422607422 2020 | 1 2021 | 505 2022 | 1 2023 | 60.827007293701172 2024 | 1 2025 | 506 2026 | 1 2027 | 61.340419769287109 2028 | 1 2029 | 507 2030 | 1 2031 | 61.219295501708984 2032 | 1 2033 | 508 2034 | 1 2035 | 60.155731201171875 2036 | 1 2037 | 509 2038 | 1 2039 | 62.775856018066406 2040 | 1 2041 | 510 2042 | 1 2043 | 61.903827667236328 2044 | 1 2045 | 511 2046 | 1 2047 | 63.936431884765625 2048 | 1 2049 | 512 2050 | 1 2051 | 66.327011108398438 2052 | 1 2053 | 513 2054 | 1 2055 | 0 2056 | 1 2057 | 514 2058 | 1 2059 | 61.542064666748047 2060 | 1 2061 | 515 2062 | 1 2063 | 62.921424865722656 2064 | 1 2065 | 516 2066 | 1 2067 | 65.84686279296875 2068 | 1 2069 | 517 2070 | 1 2071 | 66.542823791503906 2072 | 1 2073 | 518 2074 | 1 2075 | 65.811271667480469 2076 | 1 2077 | 519 2078 | 1 2079 | 64.69927978515625 2080 | 1 2081 | 520 2082 | 1 2083 | 67.429214477539062 2084 | 1 2085 | 521 2086 | 1 2087 | 66.03802490234375 2088 | 1 2089 | 522 2090 | 1 2091 | 64.56890869140625 2092 | 1 2093 | 523 2094 | 1 2095 | 64.6201171875 2096 | 1 2097 | 524 2098 | 1 2099 | 66.365158081054688 2100 | 1 2101 | 525 2102 | 1 2103 | 63.959087371826172 2104 | 1 2105 | 526 2106 | 1 2107 | 62.844341278076172 2108 | 1 2109 | 527 2110 | 1 2111 | 61.879875183105469 2112 | 1 2113 | 528 2114 | 1 2115 | 63.521865844726562 2116 | 1 2117 | 529 2118 | 1 2119 | 0 2120 | 1 2121 | 530 2122 | 1 2123 | 64.67987060546875 2124 | 1 2125 | 531 2126 | 1 2127 | 66.428924560546875 2128 | 1 2129 | 532 2130 | 1 2131 | 68.879913330078125 2132 | 1 2133 | 533 2134 | 1 2135 | 0 2136 | 1 2137 | 534 2138 | 1 2139 | 65.130950927734375 2140 | 1 2141 | 535 2142 | 1 2143 | 65.611991882324219 2144 | 1 2145 | 536 2146 | 1 2147 | 64.107040405273438 2148 | 1 2149 | 537 2150 | 1 2151 | 63.789985656738281 2152 | 1 2153 | 538 2154 | 1 2155 | 62.760818481445312 2156 | 1 2157 | 539 2158 | 1 2159 | 62.649490356445312 2160 | 1 2161 | 540 2162 | 1 2163 | 62.522792816162109 2164 | 1 2165 | 541 2166 | 1 2167 | 0 2168 | 1 2169 | 542 2170 | 1 2171 | 60.808414459228516 2172 | 1 2173 | 543 2174 | 1 2175 | 62.451889038085938 2176 | 1 2177 | 544 2178 | 1 2179 | 62.409381866455078 2180 | 1 2181 | 545 2182 | 1 2183 | 60.490501403808594 2184 | 1 2185 | 546 2186 | 1 2187 | 59.079132080078125 2188 | 1 2189 | 547 2190 | 1 2191 | 57.137424468994141 2192 | 1 2193 | 548 2194 | 1 2195 | 56.590927124023438 2196 | 1 2197 | 549 2198 | 1 2199 | 57.075119018554688 2200 | 1 2201 | 550 2202 | 1 2203 | 57.873844146728516 2204 | 1 2205 | 551 2206 | 1 2207 | 57.030788421630859 2208 | 1 2209 | 552 2210 | 1 2211 | 57.094814300537109 2212 | 1 2213 | 553 2214 | 1 2215 | 0 2216 | 1 2217 | 554 2218 | 1 2219 | 55.493064880371094 2220 | 1 2221 | 555 2222 | 1 2223 | 54.822807312011719 2224 | 1 2225 | 556 2226 | 1 2227 | 52.702838897705078 2228 | 1 2229 | 557 2230 | 1 2231 | 0 2232 | 1 2233 | 558 2234 | 1 2235 | 52.137603759765625 2236 | 1 2237 | 559 2238 | 1 2239 | 52.419696807861328 2240 | 1 2241 | 560 2242 | 1 2243 | 53.277393341064453 2244 | 1 2245 | 561 2246 | 1 2247 | 51.489490509033203 2248 | 1 2249 | 562 2250 | 1 2251 | 50.325786590576172 2252 | 1 2253 | 563 2254 | 1 2255 | 49.804153442382812 2256 | 1 2257 | 564 2258 | 1 2259 | 46.915901184082031 2260 | 1 2261 | 565 2262 | 1 2263 | 47.064006805419922 2264 | 1 2265 | 566 2266 | 1 2267 | 47.38787841796875 2268 | 1 2269 | 567 2270 | 1 2271 | 46.216667175292969 2272 | 1 2273 | 568 2274 | 1 2275 | 45.419071197509766 2276 | 1 2277 | 569 2278 | 1 2279 | 44.456233978271484 2280 | 1 2281 | 570 2282 | 1 2283 | 42.834327697753906 2284 | 1 2285 | 571 2286 | 1 2287 | 0 2288 | 1 2289 | 572 2290 | 1 2291 | 37.125686645507812 2292 | 1 2293 | 573 2294 | 1 2295 | 37.675048828125 2296 | 1 2297 | 574 2298 | 1 2299 | 35.830543518066406 2300 | 1 2301 | 575 2302 | 1 2303 | 37.642166137695312 2304 | 1 2305 | 576 2306 | 1 2307 | 38.331069946289062 2308 | 1 2309 | 577 2310 | 1 2311 | 38.826240539550781 2312 | 1 2313 | 578 2314 | 1 2315 | 36.328910827636719 2316 | 1 2317 | 579 2318 | 1 2319 | 36.904449462890625 2320 | 1 2321 | 580 2322 | 1 2323 | 35.366607666015625 2324 | 1 2325 | 581 2326 | 1 2327 | 35.311546325683594 2328 | 1 2329 | 582 2330 | 1 2331 | 33.671661376953125 2332 | 1 2333 | 583 2334 | 1 2335 | 37.592750549316406 2336 | 1 2337 | 584 2338 | 1 2339 | 30.027578353881836 2340 | 1 2341 | 585 2342 | 1 2343 | 25.730741500854492 2344 | 1 2345 | 586 2346 | 1 2347 | 32.360519409179688 2348 | 1 2349 | 587 2350 | 1 2351 | 31.796012878417969 2352 | 1 2353 | 588 2354 | 1 2355 | 30.761096954345703 2356 | 1 2357 | 589 2358 | 1 2359 | 35.057285308837891 2360 | 1 2361 | 590 2362 | 1 2363 | 39.622276306152344 2364 | 1 2365 | 591 2366 | 1 2367 | 44.624946594238281 2368 | 1 2369 | 592 2370 | 1 2371 | 46.416229248046875 2372 | 1 2373 | 593 2374 | 1 2375 | 47.120033264160156 2376 | 1 2377 | 594 2378 | 1 2379 | 52.176666259765625 2380 | 1 2381 | 595 2382 | 1 2383 | 53.356117248535156 2384 | 1 2385 | 596 2386 | 1 2387 | 51.416587829589844 2388 | 1 2389 | 597 2390 | 1 2391 | 54.782356262207031 2392 | 1 2393 | 598 2394 | 1 2395 | 48.74810791015625 2396 | 1 2397 | 599 2398 | 1 2399 | 52.692489624023438 2400 | 1 2401 | 600 2402 | 1 2403 | 55.960136413574219 2404 | 1 2405 | 601 2406 | 1 2407 | 55.469940185546875 2408 | 1 2409 | 602 2410 | 1 2411 | 50.822052001953125 2412 | 1 2413 | 603 2414 | 1 2415 | 54.740074157714844 2416 | 1 2417 | 604 2418 | 1 2419 | 52.069847106933594 2420 | 1 2421 | 605 2422 | 1 2423 | 56.515449523925781 2424 | 1 2425 | 606 2426 | 1 2427 | 59.775459289550781 2428 | 1 2429 | 607 2430 | 1 2431 | 0 2432 | 1 2433 | 608 2434 | 1 2435 | 60.480308532714844 2436 | 1 2437 | 609 2438 | 1 2439 | 60.754676818847656 2440 | 1 2441 | 610 2442 | 1 2443 | 47.889358520507812 2444 | 1 2445 | 611 2446 | 1 2447 | 41.120338439941406 2448 | 1 2449 | 612 2450 | 1 2451 | 24.422828674316406 2452 | 1 2453 | 613 2454 | 1 2455 | 23.835556030273438 2456 | 1 2457 | 614 2458 | 1 2459 | 32.273597717285156 2460 | 1 2461 | 615 2462 | 1 2463 | 37.689735412597656 2464 | 1 2465 | 616 2466 | 1 2467 | 57.899456024169922 2468 | 1 2469 | 617 2470 | 1 2471 | 55.526908874511719 2472 | 1 2473 | 618 2474 | 1 2475 | 58.97418212890625 2476 | 1 2477 | 619 2478 | 1 2479 | 33.303321838378906 2480 | 1 2481 | 620 2482 | 1 2483 | 34.066219329833984 2484 | 1 2485 | 621 2486 | 1 2487 | 37.609207153320312 2488 | 1 2489 | 622 2490 | 1 2491 | 37.821922302246094 2492 | 1 2493 | 623 2494 | 1 2495 | 37.547943115234375 2496 | 1 2497 | 624 2498 | 1 2499 | 37.69171142578125 2500 | 1 2501 | 625 2502 | 1 2503 | 0 2504 | 1 2505 | 626 2506 | 1 2507 | 40.943660736083984 2508 | 1 2509 | 627 2510 | 1 2511 | 40.996112823486328 2512 | 1 2513 | 628 2514 | 1 2515 | 42.571376800537109 2516 | 1 2517 | 629 2518 | 1 2519 | 46.339088439941406 2520 | 1 2521 | 630 2522 | 1 2523 | 0 2524 | 1 2525 | 631 2526 | 1 2527 | 47.902252197265625 2528 | 1 2529 | 632 2530 | 1 2531 | 47.8214111328125 2532 | 1 2533 | 633 2534 | 1 2535 | 51.529125213623047 2536 | 1 2537 | 634 2538 | 1 2539 | 52.077926635742188 2540 | 1 2541 | 635 2542 | 1 2543 | 53.990184783935547 2544 | 1 2545 | 636 2546 | 1 2547 | 54.640365600585938 2548 | 1 2549 | 637 2550 | 1 2551 | 54.258499145507812 2552 | 1 2553 | 638 2554 | 1 2555 | 54.445835113525391 2556 | 1 2557 | 639 2558 | 1 2559 | 0 2560 | 1 2561 | 640 2562 | 1 2563 | 54.056655883789062 2564 | 1 2565 | 641 2566 | 1 2567 | 53.232143402099609 2568 | 1 2569 | 642 2570 | 1 2571 | 53.053577423095703 2572 | 1 2573 | 643 2574 | 1 2575 | 53.962169647216797 2576 | 1 2577 | 644 2578 | 1 2579 | 53.359325408935547 2580 | 1 2581 | 645 2582 | 1 2583 | 52.841468811035156 2584 | 1 2585 | 646 2586 | 1 2587 | 53.443248748779297 2588 | 1 2589 | 647 2590 | 1 2591 | 54.79180908203125 2592 | 1 2593 | 648 2594 | 1 2595 | 51.975971221923828 2596 | 1 2597 | 649 2598 | 1 2599 | 0 2600 | 1 2601 | 650 2602 | 1 2603 | 51.875892639160156 2604 | 1 2605 | 651 2606 | 1 2607 | 51.765594482421875 2608 | 1 2609 | 652 2610 | 1 2611 | 51.560855865478516 2612 | 1 2613 | 653 2614 | 1 2615 | 51.188869476318359 2616 | 1 2617 | 654 2618 | 1 2619 | 49.849441528320312 2620 | 1 2621 | 655 2622 | 1 2623 | 50.233608245849609 2624 | 1 2625 | 656 2626 | 1 2627 | 48.408954620361328 2628 | 1 2629 | 657 2630 | 1 2631 | 47.120121002197266 2632 | 1 2633 | 658 2634 | 1 2635 | 0 2636 | 1 2637 | 659 2638 | 1 2639 | 42.920875549316406 2640 | 1 2641 | 660 2642 | 1 2643 | 41.67633056640625 2644 | 1 2645 | 661 2646 | 1 2647 | 41.309501647949219 2648 | 1 2649 | 662 2650 | 1 2651 | 40.018898010253906 2652 | 1 2653 | 663 2654 | 1 2655 | 40.650295257568359 2656 | 1 2657 | 664 2658 | 1 2659 | 40.261207580566406 2660 | 1 2661 | 665 2662 | 1 2663 | 40.716976165771484 2664 | 1 2665 | 666 2666 | 1 2667 | 39.852592468261719 2668 | 1 2669 | 667 2670 | 1 2671 | 40.606613159179688 2672 | 1 2673 | 668 2674 | 1 2675 | 37.659530639648438 2676 | 1 2677 | 669 2678 | 1 2679 | 40.889511108398438 2680 | 1 2681 | 670 2682 | 1 2683 | 39.807807922363281 2684 | 1 2685 | 671 2686 | 1 2687 | 40.276443481445312 2688 | 1 2689 | 672 2690 | 1 2691 | 39.77203369140625 2692 | 1 2693 | 673 2694 | 1 2695 | 39.689445495605469 2696 | 1 2697 | 674 2698 | 1 2699 | 39.6455078125 2700 | 1 2701 | 675 2702 | 1 2703 | 40.834945678710938 2704 | 1 2705 | 676 2706 | 1 2707 | 39.816444396972656 2708 | 1 2709 | 677 2710 | 1 2711 | 0 2712 | 1 2713 | 678 2714 | 1 2715 | 39.476600646972656 2716 | 1 2717 | 679 2718 | 1 2719 | 39.740974426269531 2720 | 1 2721 | 680 2722 | 1 2723 | 38.505104064941406 2724 | 1 2725 | 681 2726 | 1 2727 | 38.39129638671875 2728 | 1 2729 | 682 2730 | 1 2731 | 39.475845336914062 2732 | 1 2733 | 683 2734 | 1 2735 | 39.428009033203125 2736 | 1 2737 | 684 2738 | 1 2739 | 39.177009582519531 2740 | 1 2741 | 685 2742 | 1 2743 | 38.311286926269531 2744 | 1 2745 | 686 2746 | 1 2747 | 37.878330230712891 2748 | 1 2749 | 687 2750 | 1 2751 | 38.454536437988281 2752 | 1 2753 | 688 2754 | 1 2755 | 36.954132080078125 2756 | 1 2757 | 689 2758 | 1 2759 | 37.202316284179688 2760 | 1 2761 | 690 2762 | 1 2763 | 37.617465972900391 2764 | 1 2765 | 691 2766 | 1 2767 | 36.968086242675781 2768 | 1 2769 | 692 2770 | 1 2771 | 37.186393737792969 2772 | 1 2773 | 693 2774 | 1 2775 | 34.547344207763672 2776 | 1 2777 | 694 2778 | 1 2779 | 34.75335693359375 2780 | 1 2781 | 695 2782 | 1 2783 | 35.72283935546875 2784 | 1 2785 | 696 2786 | 1 2787 | 36.05078125 2788 | 1 2789 | 697 2790 | 1 2791 | 36.763832092285156 2792 | 1 2793 | 698 2794 | 1 2795 | 36.116325378417969 2796 | 1 2797 | 699 2798 | 1 2799 | 33.254058837890625 2800 | 1 2801 | 700 2802 | 1 2803 | 32.187549591064453 2804 | 1 2805 | 701 2806 | 1 2807 | 31.454631805419922 2808 | 1 2809 | 702 2810 | 1 2811 | 0 2812 | 1 2813 | 703 2814 | 1 2815 | 32.189945220947266 2816 | 1 2817 | 704 2818 | 1 2819 | 24.59382438659668 2820 | 1 2821 | 705 2822 | 1 2823 | 25.796806335449219 2824 | 1 2825 | 706 2826 | 1 2827 | 29.984529495239258 2828 | 1 2829 | 707 2830 | 1 2831 | 32.774551391601562 2832 | 1 2833 | 708 2834 | 1 2835 | 0 2836 | 1 2837 | 709 2838 | 1 2839 | 30.591981887817383 2840 | 1 2841 | 710 2842 | 1 2843 | 32.949398040771484 2844 | 1 2845 | 711 2846 | 1 2847 | 36.027584075927734 2848 | 1 2849 | 712 2850 | 1 2851 | 37.547817230224609 2852 | 1 2853 | 713 2854 | 1 2855 | 34.11529541015625 2856 | 1 2857 | 714 2858 | 1 2859 | 32.469001770019531 2860 | 1 2861 | 715 2862 | 1 2863 | 31.980913162231445 2864 | 1 2865 | 716 2866 | 1 2867 | 34.152938842773438 2868 | 1 2869 | 717 2870 | 1 2871 | 29.737852096557617 2872 | 1 2873 | 718 2874 | 1 2875 | 28.946714401245117 2876 | 1 2877 | 719 2878 | 1 2879 | 22.647958755493164 2880 | 1 2881 | 720 2882 | 1 2883 | 21.538455963134766 2884 | 1 2885 | 721 2886 | 1 2887 | 18.428827285766602 2888 | 1 2889 | 722 2890 | 1 2891 | 15.448187828063965 2892 | 1 2893 | 723 2894 | 1 2895 | 17.797248840332031 2896 | 1 2897 | 724 2898 | 1 2899 | 18.166584014892578 2900 | 1 2901 | 725 2902 | 1 2903 | 21.137378692626953 2904 | 1 2905 | 726 2906 | 1 2907 | 22.462091445922852 2908 | 1 2909 | 727 2910 | 1 2911 | 22.468355178833008 2912 | 1 2913 | 728 2914 | 1 2915 | 21.915153503417969 2916 | 1 2917 | 729 2918 | 1 2919 | 21.36534309387207 2920 | 1 2921 | 730 2922 | 1 2923 | 18.135143280029297 2924 | 1 2925 | 731 2926 | 1 2927 | 17.339061737060547 2928 | 1 2929 | 732 2930 | 1 2931 | 14.967717170715332 2932 | 1 2933 | 733 2934 | 1 2935 | 14.017934799194336 2936 | 1 2937 | 734 2938 | 1 2939 | 11.324968338012695 2940 | 1 2941 | 735 2942 | 1 2943 | 15.5147705078125 2944 | 1 2945 | 736 2946 | 1 2947 | 18.815456390380859 2948 | 1 2949 | 737 2950 | 1 2951 | 17.580638885498047 2952 | -------------------------------------------------------------------------------- /dqn/reward_history.t7: -------------------------------------------------------------------------------- 1 | 3 2 | 1 3 | 99 4 | 1 5 | 1 6 | 1 7 | -16.370121951219513 8 | 1 9 | 2 10 | 1 11 | -16.762320648783533 12 | 1 13 | 3 14 | 1 15 | -7.3712653778558872 16 | 1 17 | 4 18 | 1 19 | -4.6380952380952385 20 | 1 21 | 5 22 | 1 23 | -2.9360613810741687 24 | 1 25 | 6 26 | 1 27 | -5.839316239316239 28 | 1 29 | 7 30 | 1 31 | -6.8236301369863011 32 | 1 33 | 8 34 | 1 35 | -6.9407558733401427 36 | 1 37 | 9 38 | 1 39 | -1.7319109461966604 40 | 1 41 | 10 42 | 1 43 | -0.10476190476190476 44 | 1 45 | 11 46 | 1 47 | 4.8308823529411766 48 | 1 49 | 12 50 | 1 51 | 7.6679586563307494 52 | 1 53 | 13 54 | 1 55 | 6.2855191256830603 56 | 1 57 | 14 58 | 1 59 | 3.3367496339677891 60 | 1 61 | 15 62 | 1 63 | 8.2196339434276204 64 | 1 65 | 16 66 | 1 67 | 7.7170111287758347 68 | 1 69 | 17 70 | 1 71 | 8.8457711442786078 72 | 1 73 | 18 74 | 1 75 | 11.246846846846847 76 | 1 77 | 19 78 | 1 79 | 15.010330578512397 80 | 1 81 | 20 82 | 1 83 | 20.952688172043011 84 | 1 85 | 21 86 | 1 87 | 22.300000000000001 88 | 1 89 | 22 90 | 1 91 | 22.745454545454546 92 | 1 93 | 23 94 | 1 95 | 27.560096153846153 96 | 1 97 | 24 98 | 1 99 | 39.485955056179776 100 | 1 101 | 25 102 | 1 103 | 48.178683385579937 104 | 1 105 | 26 106 | 1 107 | 196.35849056603774 108 | 1 109 | 27 110 | 1 111 | 204.33980582524271 112 | 1 113 | 28 114 | 1 115 | 183.46491228070175 116 | 1 117 | 29 118 | 1 119 | 306.06410256410254 120 | 1 121 | 30 122 | 1 123 | 314.45454545454544 124 | 1 125 | 31 126 | 1 127 | 485.76470588235293 128 | 1 129 | 32 130 | 1 131 | 497.60784313725492 132 | 1 133 | 33 134 | 1 135 | 905.03999999999996 136 | 1 137 | 34 138 | 1 139 | 534 140 | 1 141 | 35 142 | 1 143 | 867.39393939393938 144 | 1 145 | 36 146 | 1 147 | 751.43243243243239 148 | 1 149 | 37 150 | 1 151 | 1430.8571428571429 152 | 1 153 | 38 154 | 1 155 | 819.52777777777783 156 | 1 157 | 39 158 | 1 159 | 1163.6153846153845 160 | 1 161 | 40 162 | 1 163 | 1268.125 164 | 1 165 | 41 166 | 1 167 | 929.07142857142856 168 | 1 169 | 42 170 | 1 171 | 1279 172 | 1 173 | 43 174 | 1 175 | 1615.5625 176 | 1 177 | 44 178 | 1 179 | 867.5625 180 | 1 181 | 45 182 | 1 183 | 903.969696969697 184 | 1 185 | 46 186 | 1 187 | 1297.6875 188 | 1 189 | 47 190 | 1 191 | 501.65384615384613 192 | 1 193 | 48 194 | 1 195 | 1175.75 196 | 1 197 | 49 198 | 1 199 | 1181.6800000000001 200 | 1 201 | 50 202 | 1 203 | 1183.1666666666667 204 | 1 205 | 51 206 | 1 207 | 649.51111111111106 208 | 1 209 | 52 210 | 1 211 | 606.51999999999998 212 | 1 213 | 53 214 | 1 215 | 811.41935483870964 216 | 1 217 | 54 218 | 1 219 | 990.96296296296293 220 | 1 221 | 55 222 | 1 223 | 811.05405405405406 224 | 1 225 | 56 226 | 1 227 | 1038.8571428571429 228 | 1 229 | 57 230 | 1 231 | 577.80851063829789 232 | 1 233 | 58 234 | 1 235 | 705.29999999999995 236 | 1 237 | 59 238 | 1 239 | 1206.8800000000001 240 | 1 241 | 60 242 | 1 243 | 888.38235294117646 244 | 1 245 | 61 246 | 1 247 | 1153.0416666666667 248 | 1 249 | 62 250 | 1 251 | 1141.5833333333333 252 | 1 253 | 63 254 | 1 255 | 970.25 256 | 1 257 | 64 258 | 1 259 | 701.41860465116281 260 | 1 261 | 65 262 | 1 263 | 745.42105263157896 264 | 1 265 | 66 266 | 1 267 | 805.97222222222217 268 | 1 269 | 67 270 | 1 271 | 1061.391304347826 272 | 1 273 | 68 274 | 1 275 | 907.83870967741939 276 | 1 277 | 69 278 | 1 279 | 1079.4482758620691 280 | 1 281 | 70 282 | 1 283 | 797.65517241379314 284 | 1 285 | 71 286 | 1 287 | 891.94117647058829 288 | 1 289 | 72 290 | 1 291 | 848.02857142857147 292 | 1 293 | 73 294 | 1 295 | 641.02127659574467 296 | 1 297 | 74 298 | 1 299 | 703.69767441860461 300 | 1 301 | 75 302 | 1 303 | 1713.2222222222222 304 | 1 305 | 76 306 | 1 307 | 729.76315789473688 308 | 1 309 | 77 310 | 1 311 | 1053.1379310344828 312 | 1 313 | 78 314 | 1 315 | 1341.8095238095239 316 | 1 317 | 79 318 | 1 319 | 748.71428571428567 320 | 1 321 | 80 322 | 1 323 | 826.88235294117646 324 | 1 325 | 81 326 | 1 327 | 1753.875 328 | 1 329 | 82 330 | 1 331 | 1278.6818181818182 332 | 1 333 | 83 334 | 1 335 | 967.22222222222217 336 | 1 337 | 84 338 | 1 339 | 960 340 | 1 341 | 85 342 | 1 343 | 1346 344 | 1 345 | 86 346 | 1 347 | 910.75757575757575 348 | 1 349 | 87 350 | 1 351 | 886.86666666666667 352 | 1 353 | 88 354 | 1 355 | 1004.0714285714286 356 | 1 357 | 89 358 | 1 359 | 777.78378378378375 360 | 1 361 | 90 362 | 1 363 | 1243.9583333333333 364 | 1 365 | 91 366 | 1 367 | 1331.6818181818182 368 | 1 369 | 92 370 | 1 371 | 608.51111111111106 372 | 1 373 | 93 374 | 1 375 | 939.72727272727275 376 | 1 377 | 94 378 | 1 379 | 1607.8947368421052 380 | 1 381 | 95 382 | 1 383 | 1746 384 | 1 385 | 96 386 | 1 387 | 1501.05 388 | 1 389 | 97 390 | 1 391 | 838.11111111111109 392 | 1 393 | 98 394 | 1 395 | 1620 396 | 1 397 | 99 398 | 1 399 | 1409.952380952381 400 | -------------------------------------------------------------------------------- /dqn/step_history.t7: -------------------------------------------------------------------------------- 1 | 3 2 | 1 3 | 99 4 | 1 5 | 1 6 | 1 7 | 30.487804878048781 8 | 1 9 | 2 10 | 1 11 | 31.191515907673114 12 | 1 13 | 3 14 | 1 15 | 21.968365553602812 16 | 1 17 | 4 18 | 1 19 | 26.455026455026456 20 | 1 21 | 5 22 | 1 23 | 42.625745950554133 24 | 1 25 | 6 26 | 1 27 | 28.490028490028489 28 | 1 29 | 7 30 | 1 31 | 28.538812785388128 32 | 1 33 | 8 34 | 1 35 | 51.072522982635341 36 | 1 37 | 9 38 | 1 39 | 46.382189239332099 40 | 1 41 | 10 42 | 1 43 | 52.910052910052912 44 | 1 45 | 11 46 | 1 47 | 61.274509803921568 48 | 1 49 | 12 50 | 1 51 | 64.599483204134373 52 | 1 53 | 13 54 | 1 55 | 68.306010928961754 56 | 1 57 | 14 58 | 1 59 | 73.206442166910691 60 | 1 61 | 15 62 | 1 63 | 83.194675540765388 64 | 1 65 | 16 66 | 1 67 | 79.491255961844203 68 | 1 69 | 17 70 | 1 71 | 82.91873963515755 72 | 1 73 | 18 74 | 1 75 | 90.090090090090087 76 | 1 77 | 19 78 | 1 79 | 103.30578512396694 80 | 1 81 | 20 82 | 1 83 | 107.52688172043011 84 | 1 85 | 21 86 | 1 87 | 116.27906976744185 88 | 1 89 | 22 90 | 1 91 | 113.63636363636364 92 | 1 93 | 23 94 | 1 95 | 120.19230769230769 96 | 1 97 | 24 98 | 1 99 | 140.44943820224719 100 | 1 101 | 25 102 | 1 103 | 156.73981191222572 104 | 1 105 | 26 106 | 1 107 | 471.69811320754718 108 | 1 109 | 27 110 | 1 111 | 485.43689320388347 112 | 1 113 | 28 114 | 1 115 | 438.59649122807019 116 | 1 117 | 29 118 | 1 119 | 641.02564102564099 120 | 1 121 | 30 122 | 1 123 | 649.35064935064941 124 | 1 125 | 31 126 | 1 127 | 980.39215686274508 128 | 1 129 | 32 130 | 1 131 | 980.39215686274508 132 | 1 133 | 33 134 | 1 135 | 2000 136 | 1 137 | 34 138 | 1 139 | 1086.9565217391305 140 | 1 141 | 35 142 | 1 143 | 1515.1515151515152 144 | 1 145 | 36 146 | 1 147 | 1351.3513513513512 148 | 1 149 | 37 150 | 1 151 | 2380.9523809523807 152 | 1 153 | 38 154 | 1 155 | 1388.8888888888889 156 | 1 157 | 39 158 | 1 159 | 1923.0769230769231 160 | 1 161 | 40 162 | 1 163 | 2083.3333333333335 164 | 1 165 | 41 166 | 1 167 | 1785.7142857142858 168 | 1 169 | 42 170 | 1 171 | 2272.7272727272725 172 | 1 173 | 43 174 | 1 175 | 3125 176 | 1 177 | 44 178 | 1 179 | 1562.5 180 | 1 181 | 45 182 | 1 183 | 1515.1515151515152 184 | 1 185 | 46 186 | 1 187 | 3125 188 | 1 189 | 47 190 | 1 191 | 961.53846153846155 192 | 1 193 | 48 194 | 1 195 | 2500 196 | 1 197 | 49 198 | 1 199 | 2000 200 | 1 201 | 50 202 | 1 203 | 2083.3333333333335 204 | 1 205 | 51 206 | 1 207 | 1111.1111111111111 208 | 1 209 | 52 210 | 1 211 | 1000 212 | 1 213 | 53 214 | 1 215 | 1612.9032258064517 216 | 1 217 | 54 218 | 1 219 | 1851.851851851852 220 | 1 221 | 55 222 | 1 223 | 1351.3513513513512 224 | 1 225 | 56 226 | 1 227 | 1785.7142857142858 228 | 1 229 | 57 230 | 1 231 | 1063.8297872340424 232 | 1 233 | 58 234 | 1 235 | 1250 236 | 1 237 | 59 238 | 1 239 | 2000 240 | 1 241 | 60 242 | 1 243 | 1470.5882352941176 244 | 1 245 | 61 246 | 1 247 | 2083.3333333333335 248 | 1 249 | 62 250 | 1 251 | 2083.3333333333335 252 | 1 253 | 63 254 | 1 255 | 1562.5 256 | 1 257 | 64 258 | 1 259 | 1162.7906976744187 260 | 1 261 | 65 262 | 1 263 | 1315.7894736842106 264 | 1 265 | 66 266 | 1 267 | 1388.8888888888889 268 | 1 269 | 67 270 | 1 271 | 2173.913043478261 272 | 1 273 | 68 274 | 1 275 | 1612.9032258064517 276 | 1 277 | 69 278 | 1 279 | 1724.1379310344828 280 | 1 281 | 70 282 | 1 283 | 1724.1379310344828 284 | 1 285 | 71 286 | 1 287 | 1470.5882352941176 288 | 1 289 | 72 290 | 1 291 | 1428.5714285714287 292 | 1 293 | 73 294 | 1 295 | 1063.8297872340424 296 | 1 297 | 74 298 | 1 299 | 1162.7906976744187 300 | 1 301 | 75 302 | 1 303 | 2777.7777777777778 304 | 1 305 | 76 306 | 1 307 | 1315.7894736842106 308 | 1 309 | 77 310 | 1 311 | 1724.1379310344828 312 | 1 313 | 78 314 | 1 315 | 2380.9523809523807 316 | 1 317 | 79 318 | 1 319 | 1428.5714285714287 320 | 1 321 | 80 322 | 1 323 | 1470.5882352941176 324 | 1 325 | 81 326 | 1 327 | 3125 328 | 1 329 | 82 330 | 1 331 | 2272.7272727272725 332 | 1 333 | 83 334 | 1 335 | 1851.851851851852 336 | 1 337 | 84 338 | 1 339 | 1851.851851851852 340 | 1 341 | 85 342 | 1 343 | 3846.1538461538462 344 | 1 345 | 86 346 | 1 347 | 1515.1515151515152 348 | 1 349 | 87 350 | 1 351 | 1666.6666666666667 352 | 1 353 | 88 354 | 1 355 | 1785.7142857142858 356 | 1 357 | 89 358 | 1 359 | 1351.3513513513512 360 | 1 361 | 90 362 | 1 363 | 2083.3333333333335 364 | 1 365 | 91 366 | 1 367 | 2272.7272727272725 368 | 1 369 | 92 370 | 1 371 | 1111.1111111111111 372 | 1 373 | 93 374 | 1 375 | 2272.7272727272725 376 | 1 377 | 94 378 | 1 379 | 2631.5789473684213 380 | 1 381 | 95 382 | 1 383 | 3333.3333333333335 384 | 1 385 | 96 386 | 1 387 | 2500 388 | 1 389 | 97 390 | 1 391 | 1388.8888888888889 392 | 1 393 | 98 394 | 1 395 | 2631.5789473684213 396 | 1 397 | 99 398 | 1 399 | 2380.9523809523807 400 | -------------------------------------------------------------------------------- /dqn/subtitle.m: -------------------------------------------------------------------------------- 1 | function ht = subtitle(kn,text,txtsize) 2 | h1 = get(gcf,'children'); 3 | axis1 = get(h1(end),'Position'); 4 | axis2 = get(h1(end-kn+1),'Position'); 5 | axest = [axis1(1),axis1(2)+axis1(4),axis2(1)+axis1(3)-axis1(1),0.01]; 6 | ht = axes('Position',axest); 7 | axis(ht,'off') 8 | title(ht,text,'FontSize',txtsize) -------------------------------------------------------------------------------- /dqn/test_agent_cpu.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (c) 2014 Google Inc. 3 | See LICENSE file for full terms of limited license. 4 | ]] 5 | 6 | gd = require "gd" 7 | 8 | if not dqn then 9 | require "initenv" 10 | end 11 | 12 | NETWORK="DQN3_0_1_Evasion_FULL_Y.t7" 13 | agent="NeuralQLearner" 14 | n_replay=1 15 | netfile="\"convnet_evasion\"" 16 | update_freq=4 17 | actrep=4 18 | discount=0.99 19 | seed=1 20 | learn_start=50000 21 | pool_frms_type="\"max\"" 22 | pool_frms_size=2 23 | initial_priority="false" 24 | replay_memory=1000000 25 | eps_end=0.1 26 | eps_endt=replay_memory 27 | lr=0.00025 28 | agent_type="DQN3_0_1" 29 | preproc_net="\"net_downsample_2x_full_y\"" 30 | agent_name="DQN3_0_1_Evasion_FULL_Y" 31 | state_dim=7056 32 | ncols=1 33 | 34 | agent_params="lr=" .. lr .. ",ep=1,ep_end=" .. eps_end .. ",ep_endt=" .. eps_endt .. ",discount=" .. discount .. ",hist_len=4,learn_start=" .. learn_start .. ",replay_memory=" .. replay_memory .. ",update_freq=" .. update_freq .. ",n_replay=" .. n_replay .. ",network=" .. netfile .. ",preproc=" .. preproc_net .. ",state_dim=" .. state_dim .. ",minibatch_size=32,rescale_r=1,ncols=" .. ncols .. ",bufferSize=512,valid_size=500,target_q=10000,clip_delta=1,min_reward=-1,max_reward=1" 35 | gif_file="../gifs/Evasion_Test.gif" 36 | gpu=-1 37 | random_starts=30 38 | pool_frms="type=\"max\",size=2" 39 | num_threads=4 40 | -- args="-name $agent_name -env_params $env_params -agent $agent -agent_params $agent_params -actrep $actrep -gpu $gpu -random_starts $random_starts -pool_frms $pool_frms -seed $seed -threads $num_threads -network $NETWORK -gif_file $gif_file" 41 | -- print(args) 42 | 43 | arg[1] = "-name" 44 | arg[2] = agent_name 45 | arg[3] = "-agent" 46 | arg[4] = agent 47 | arg[5] = "-agent_params" 48 | arg[6] = agent_params 49 | arg[7] = "-actrep" 50 | arg[8] = actrep 51 | arg[9] = "-gpu" 52 | arg[10] = gpu 53 | arg[11] = "-random_starts" 54 | arg[12] = random_starts 55 | arg[13] = "-pool_frms" 56 | arg[14] = pool_frms 57 | arg[15] = "-seed" 58 | arg[16] = seed 59 | arg[17] = "-threads" 60 | arg[18] = num_threads 61 | 62 | arg[19] = "-network" 63 | arg[20] = NETWORK 64 | 65 | arg[21] = "-gif_file" 66 | arg[22] = gif_file 67 | 68 | local cmd = torch.CmdLine() 69 | cmd:text() 70 | cmd:text('Train Agent in Environment:') 71 | cmd:text() 72 | cmd:text('Options:') 73 | 74 | cmd:option('-pool_frms', '', 75 | 'string of frame pooling parameters (e.g.: size=2,type="max")') 76 | cmd:option('-actrep', 1, 'how many times to repeat action') 77 | cmd:option('-random_starts', 0, 'play action 0 between 1 and random_starts ' .. 78 | 'number of times at the start of each training episode') 79 | 80 | cmd:option('-name', '', 'filename used for saving network and training history') 81 | cmd:option('-network', '', 'reload pretrained network') 82 | cmd:option('-agent', '', 'name of agent file to use') 83 | cmd:option('-agent_params', '', 'string of agent parameters') 84 | cmd:option('-seed', 1, 'fixed input seed for repeatable experiments') 85 | 86 | cmd:option('-verbose', 2, 87 | 'the higher the level, the more information is printed to screen') 88 | cmd:option('-threads', 1, 'number of BLAS threads') 89 | cmd:option('-gpu', -1, 'gpu flag') 90 | cmd:option('-gif_file', '', 'GIF path to write session screens') 91 | cmd:option('-csv_file', '', 'CSV path to write session data') 92 | 93 | cmd:text() 94 | 95 | local opt = cmd:parse(arg) 96 | 97 | --- General setup. 98 | local agent, opt = setup(opt) 99 | 100 | -- override print to always flush the output 101 | local old_print = print 102 | local print = function(...) 103 | old_print(...) 104 | io.flush() 105 | end 106 | 107 | -- file names from command line 108 | local gif_filename = opt.gif_file 109 | -- start a new game 110 | game_actions = {0,1,2,3,4} 111 | parainit() 112 | screen, reward, terminal = NewGame() 113 | 114 | -- compress screen to JPEG with 100% quality 115 | local jpg = image.compressJPG(screen:squeeze(), 100) 116 | -- create gd image from JPEG string 117 | local im = gd.createFromJpegStr(jpg:storage():string()) 118 | -- convert truecolor to palette 119 | im:trueColorToPalette(false, 256) 120 | 121 | -- write GIF header, use global palette and infinite looping 122 | im:gifAnimBegin(gif_filename, true, 0) 123 | -- write first frame 124 | im:gifAnimAdd(gif_filename, false, 0, 0, 7, gd.DISPOSAL_NONE) 125 | 126 | -- remember the image and show it first 127 | local previm = im 128 | local win = image.display({image=screen}) 129 | 130 | print("Started playing...") 131 | -- play one episode (game) 132 | while not terminal do 133 | -- if action was chosen randomly, Q-value is 0 134 | agent.bestq = 0 135 | -- choose the best action 136 | local action_index = agent:perceive(reward, screen, terminal, true, 0.05) 137 | -- play game in test mode (episodes don't end when losing a life) 138 | -- screen, reward, terminal = game_env:step(game_actions[action_index], false) 139 | screen, reward, terminal = EvasionGame(game_actions[action_index]) 140 | print(action_index) 141 | -- display screen 142 | image.display({image=screen, win=win}) 143 | -- create gd image from tensor 144 | jpg = image.compressJPG(screen:squeeze(), 100) 145 | im = gd.createFromJpegStr(jpg:storage():string()) 146 | 147 | -- use palette from previous (first) image 148 | im:trueColorToPalette(false, 256) 149 | im:paletteCopy(previm) 150 | 151 | -- write new GIF frame, no local palette, starting from left-top, 7ms delay 152 | im:gifAnimAdd(gif_filename, false, 0, 0, 7, gd.DISPOSAL_NONE) 153 | -- remember previous screen for optimal compression 154 | previm = im 155 | end 156 | 157 | -- end GIF animation and close CSV file 158 | gd.gifAnimEnd(gif_filename) 159 | 160 | print("Finished playing, close window to exit!") -------------------------------------------------------------------------------- /dqn/test_agent_gpu.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | Copyright (c) 2014 Google Inc. 3 | See LICENSE file for full terms of limited license. 4 | ]] 5 | 6 | gd = require "gd" 7 | image = require "image" 8 | if not dqn then 9 | require "initenv" 10 | end 11 | -- Gameindex=4 12 | 13 | NETWORK="DQN3_0_1_Evasion_FULL_Y.t7" 14 | agent="NeuralQLearner" 15 | n_replay=1 16 | netfile="\"convnet_evasion\"" 17 | update_freq=4 18 | actrep=4 19 | discount=0.99 20 | seed=1 21 | learn_start=50000 22 | pool_frms_type="\"max\"" 23 | pool_frms_size=2 24 | initial_priority="false" 25 | replay_memory=1000000 26 | eps_end=0.1 27 | eps_endt=replay_memory 28 | lr=0.00025 29 | agent_type="DQN3_0_1" 30 | preproc_net="\"net_downsample_2x_full_y\"" 31 | agent_name="DQN3_0_1_Evasion_FULL_Y" 32 | state_dim=7056 33 | ncols=1 34 | 35 | agent_params="lr=" .. lr .. ",ep=1,ep_end=" .. eps_end .. ",ep_endt=" .. eps_endt .. ",discount=" .. discount .. ",hist_len=4,learn_start=" .. learn_start .. ",replay_memory=" .. replay_memory .. ",update_freq=" .. update_freq .. ",n_replay=" .. n_replay .. ",network=" .. netfile .. ",preproc=" .. preproc_net .. ",state_dim=" .. state_dim .. ",minibatch_size=32,rescale_r=1,ncols=" .. ncols .. ",bufferSize=512,valid_size=500,target_q=10000,clip_delta=1,min_reward=-1,max_reward=1" 36 | gif_file="../gifs/Evasion_Test.gif" 37 | gpu=0 38 | random_starts=30 39 | pool_frms="type=\"max\",size=2" 40 | num_threads=4 41 | -- args="-name $agent_name -env_params $env_params -agent $agent -agent_params $agent_params -actrep $actrep -gpu $gpu -random_starts $random_starts -pool_frms $pool_frms -seed $seed -threads $num_threads -network $NETWORK -gif_file $gif_file" 42 | -- print(args) 43 | 44 | arg[1] = "-name" 45 | arg[2] = agent_name 46 | arg[3] = "-agent" 47 | arg[4] = agent 48 | arg[5] = "-agent_params" 49 | arg[6] = agent_params 50 | arg[7] = "-actrep" 51 | arg[8] = actrep 52 | arg[9] = "-gpu" 53 | arg[10] = gpu 54 | arg[11] = "-random_starts" 55 | arg[12] = random_starts 56 | arg[13] = "-pool_frms" 57 | arg[14] = pool_frms 58 | arg[15] = "-seed" 59 | arg[16] = seed 60 | arg[17] = "-threads" 61 | arg[18] = num_threads 62 | 63 | arg[19] = "-network" 64 | arg[20] = NETWORK 65 | 66 | arg[21] = "-gif_file" 67 | arg[22] = gif_file 68 | 69 | local cmd = torch.CmdLine() 70 | cmd:text() 71 | cmd:text('Train Agent in Environment:') 72 | cmd:text() 73 | cmd:text('Options:') 74 | 75 | cmd:option('-pool_frms', '', 76 | 'string of frame pooling parameters (e.g.: size=2,type="max")') 77 | cmd:option('-actrep', 1, 'how many times to repeat action') 78 | cmd:option('-random_starts', 0, 'play action 0 between 1 and random_starts ' .. 79 | 'number of times at the start of each training episode') 80 | 81 | cmd:option('-name', '', 'filename used for saving network and training history') 82 | cmd:option('-network', '', 'reload pretrained network') 83 | cmd:option('-agent', '', 'name of agent file to use') 84 | cmd:option('-agent_params', '', 'string of agent parameters') 85 | cmd:option('-seed', 1, 'fixed input seed for repeatable experiments') 86 | 87 | cmd:option('-verbose', 2, 88 | 'the higher the level, the more information is printed to screen') 89 | cmd:option('-threads', 1, 'number of BLAS threads') 90 | cmd:option('-gpu', -1, 'gpu flag') 91 | cmd:option('-gif_file', '', 'GIF path to write session screens') 92 | cmd:option('-csv_file', '', 'CSV path to write session data') 93 | 94 | cmd:text() 95 | 96 | local opt = cmd:parse(arg) 97 | 98 | --- General setup. 99 | local agent, opt = setup(opt) 100 | 101 | -- override print to always flush the output 102 | local old_print = print 103 | local print = function(...) 104 | old_print(...) 105 | io.flush() 106 | end 107 | 108 | -- file names from command line 109 | local gif_filename = opt.gif_file 110 | -- start a new game 111 | game_actions = {0,1,2,3,4} 112 | 113 | local maxqq 114 | local lasthid={} 115 | qmaxtest={} 116 | lasthidden={} 117 | qnowtest={} 118 | filename1 = "mylast" 119 | filename2 = "qmaxtest" 120 | filename3 = "qnowtest" 121 | 122 | for Gameindex=3,3 do 123 | Gameindex=4 124 | parainit() 125 | 126 | screen, reward, terminal,testscreen = NewGame(Gameindex,true) 127 | image.save('G' .. Gameindex .. '.jpg',testscreen) 128 | -- compress screen to JPEG with 100% quality 129 | local jpg = image.compressJPG(screen:squeeze(), 100) 130 | -- create gd image from JPEG string 131 | local im = gd.createFromJpegStr(jpg:storage():string()) 132 | -- convert truecolor to palette 133 | im:trueColorToPalette(false, 256) 134 | 135 | -- write GIF header, use global palette and infinite looping 136 | im:gifAnimBegin(gif_filename, true, 0) 137 | -- write first frame 138 | im:gifAnimAdd(gif_filename, false, 0, 0, 7, gd.DISPOSAL_NONE) 139 | 140 | -- remember the image and show it first 141 | local previm = im 142 | local win = image.display({image=screen}) 143 | 144 | print("Started playing...",'Game:',Gameindex) 145 | -- play one episode (game) 146 | 147 | local i=0; 148 | CaptureCnt=0 149 | SurroundCnt=0 150 | Maxiter=100000 151 | while i< Maxiter do 152 | -- if action was chosen randomly, Q-value is 0 153 | i=i+1 154 | -- choose the best action 155 | local action_index ,lasthid,qtemp = agent:perceive(reward, screen, terminal, true, 0.05) 156 | --local action_index =torch.random(1,5) 157 | -- filename1 = "lasthiddenlayer" 158 | -- lasthidden[i]=lasthid 159 | -- torch.save(filename1 .. i .. ".t7",lasthidden,'ascii') 160 | 161 | -- lasthidden[i]=lasthid 162 | -- torch.save(filename1 .. ".t7",lasthidden,'ascii') 163 | 164 | -- qmaxtest[i]=agent.bestq 165 | -- torch.save(filename2 .. ".t7",qmaxtest,'ascii') 166 | 167 | -- qnowtest[i]=qtemp 168 | -- torch.save(filename3 .. ".t7",qnowtest,'ascii') 169 | 170 | -- image.save(tostring(i) .. '.jpg',testscreen) 171 | 172 | -- play game in test mode (episodes don't end when losing a life) 173 | -- screen, reward, terminal = game_env:step(game_actions[action_index], false) 174 | screen, reward, terminal,Surround = EvasionGame(game_actions[action_index],Gameindex) 175 | if terminal then 176 | screen, reward, terminal = NextRandGame(Gameindex,30,true) 177 | CaptureCnt = CaptureCnt+1 178 | end 179 | if Surround then 180 | SurroundCnt = SurroundCnt+1 181 | end 182 | -- display screen 183 | image.display({image=screen, win=win}) 184 | -- create gd image from tensor 185 | jpg = image.compressJPG(screen:squeeze(), 100) 186 | im = gd.createFromJpegStr(jpg:storage():string()) 187 | 188 | -- use palette from previous (first) image 189 | im:trueColorToPalette(false, 256) 190 | im:paletteCopy(previm) 191 | 192 | -- write new GIF frame, no local palette, starting from left-top, 7ms delay 193 | im:gifAnimAdd(gif_filename, false, 0, 0, 7, gd.DISPOSAL_NONE) 194 | -- remember previous screen for optimal compression 195 | previm = im 196 | end 197 | -- CaptureSurround=torch.Tensor{CaptureCnt,SurroundCnt} 198 | -- local filename4 = "CaptureSurround" 199 | -- torch.save(filename4 .. ".t7",CaptureSurround,'ascii') 200 | 201 | print('Game:',Gameindex,'SurroundCnt :',SurroundCnt,'CaptureCnt:',CaptureCnt) 202 | 203 | -- end GIF animation and close CSV file 204 | gd.gifAnimEnd(gif_filename) 205 | 206 | end 207 | 208 | print("Finished playing, close window to exit!") -------------------------------------------------------------------------------- /dqn/train_agent_cpu.lua: -------------------------------------------------------------------------------- 1 | require "initenv" 2 | -- gd = require "gd" 3 | 4 | NETWORK="DQN3_0_1_Evasion_FULL_Y.t7" 5 | agent="NeuralQLearner" 6 | n_replay=1 7 | netfile="\"convnet_evasion\"" 8 | update_freq=4 9 | actrep=4 10 | discount=0.99 11 | seed=1 12 | learn_start=50000 13 | pool_frms_type="\"max\"" 14 | pool_frms_size=2 15 | initial_priority="false" 16 | replay_memory=1000000 17 | eps_end=0.1 18 | eps_endt=replay_memory 19 | lr=0.00025 20 | agent_type="DQN3_0_1" 21 | preproc_net="\"net_downsample_2x_full_y\"" 22 | agent_name="DQN3_0_1_Evasion_FULL_Y" 23 | state_dim=7056 24 | ncols=1 25 | 26 | agent_params="lr=" .. lr .. ",ep=1,ep_end=" .. eps_end .. ",ep_endt=" .. eps_endt .. ",discount=" .. discount .. ",hist_len=4,learn_start=" .. learn_start .. ",replay_memory=" .. replay_memory .. ",update_freq=" .. update_freq .. ",n_replay=" .. n_replay .. ",network=" .. netfile .. ",preproc=" .. preproc_net .. ",state_dim=" .. state_dim .. ",minibatch_size=32,rescale_r=1,ncols=" .. ncols .. ",bufferSize=512,valid_size=500,target_q=10000,clip_delta=1,min_reward=-1,max_reward=1" 27 | steps=50000000 28 | eval_freq=250000 29 | eval_steps=125000 30 | prog_freq=5000 31 | save_freq=125000 32 | gpu=-1 33 | random_starts=30 34 | pool_frms="type=\"max\",size=2" 35 | num_threads=4 36 | 37 | -- args="-name " .. agent_name .. " -agent " .. agent .. " -agent_params " .. agent_params .. " -steps " .. steps .. " -eval_freq " .. eval_freq .. " -eval_steps " .. eval_steps .. " -prog_freq " .. prog_freq .. " -save_freq " .. save_freq .. " -actrep " .. actrep .. " -gpu " .. gpu .. " -random_starts " .. random_starts .. " -pool_frms " .. pool_frms .. " -seed " .. seed .. " -threads " .. num_threads 38 | -- print(args) 39 | 40 | arg[1] = "-name" 41 | arg[2] = agent_name 42 | arg[3] = "-agent" 43 | arg[4] = agent 44 | arg[5] = "-agent_params" 45 | arg[6] = agent_params 46 | arg[7] = "-steps" 47 | arg[8] = steps 48 | arg[9] = "-eval_freq" 49 | arg[10] = eval_freq 50 | arg[11] = "-eval_steps" 51 | arg[12] = eval_steps 52 | arg[13] = "-prog_freq" 53 | arg[14] = prog_freq 54 | 55 | arg[15] = "-save_freq" 56 | arg[16] = save_freq 57 | arg[17] = "-actrep" 58 | arg[18] = actrep 59 | arg[19] = "-gpu" 60 | arg[20] = gpu 61 | arg[21] = "-random_starts" 62 | arg[22] = random_starts 63 | arg[23] = "-pool_frms" 64 | arg[24] = pool_frms 65 | arg[25] = "-seed" 66 | arg[26] = seed 67 | arg[27] = "-threads" 68 | arg[28] = num_threads 69 | arg[29] = "-network" 70 | arg[30] = NETWORK 71 | 72 | local cmd = torch.CmdLine() 73 | cmd:text() 74 | cmd:text('Train Agent in Environment:') 75 | cmd:text() 76 | cmd:text('Options:') 77 | 78 | cmd:option('-pool_frms', '', 79 | 'string of frame pooling parameters (e.g.: size=2,type="max")') 80 | cmd:option('-actrep', 1, 'how many times to repeat action') 81 | cmd:option('-random_starts', 0, 'play action 0 between 1 and random_starts ' .. 82 | 'number of times at the start of each training episode') 83 | 84 | cmd:option('-name', '', 'filename used for saving network and training history') 85 | cmd:option('-network', '', 'reload pretrained network') 86 | cmd:option('-agent', '', 'name of agent file to use') 87 | cmd:option('-agent_params', '', 'string of agent parameters') 88 | cmd:option('-seed', 1, 'fixed input seed for repeatable experiments') 89 | cmd:option('-saveNetworkParams', false, 90 | 'saves the agent network in a separate file') 91 | cmd:option('-prog_freq', 5*10^3, 'frequency of progress output') 92 | cmd:option('-save_freq', 5*10^4, 'the model is saved every save_freq steps') 93 | cmd:option('-eval_freq', 10^4, 'frequency of greedy evaluation') 94 | cmd:option('-save_versions', 0, '') 95 | 96 | cmd:option('-steps', 10^5, 'number of training steps to perform') 97 | cmd:option('-eval_steps', 10^5, 'number of evaluation steps') 98 | 99 | cmd:option('-verbose', 2, 100 | 'the higher the level, the more information is printed to screen') 101 | cmd:option('-threads', 1, 'number of BLAS threads') 102 | cmd:option('-gpu', -1, 'gpu flag') 103 | 104 | cmd:text() 105 | 106 | 107 | local opt = cmd:parse(arg) 108 | 109 | --- General setup. 110 | local agent, opt = setup(opt) 111 | 112 | -- override print to always flush the output 113 | local old_print = print 114 | local print = function(...) 115 | old_print(...) 116 | io.flush() 117 | end 118 | 119 | local learn_start = agent.learn_start 120 | local start_time = sys.clock() 121 | local reward_counts = {} 122 | local episode_counts = {} 123 | local time_history = {} 124 | local v_history = {} 125 | local qmax_history = {} 126 | local td_history = {} 127 | local reward_history = {} 128 | local step = 0 129 | time_history[1] = 0 130 | 131 | local total_reward 132 | local nrewards 133 | local nepisodes 134 | local episode_reward 135 | 136 | game_actions = {0,1,2,3,4} 137 | parainit() 138 | local screen, reward, terminal = NewGame() 139 | 140 | print("Iteration ..", step) 141 | local win = nil 142 | 143 | while step < opt.steps do 144 | -- local t1 = sys.clock() 145 | step = step + 1 146 | local action_index = agent:perceive(reward, screen, terminal) 147 | -- print(step) 148 | -- game over? get next game! 149 | if not terminal then 150 | screen, reward, terminal = EvasionGame(game_actions[action_index]) 151 | else 152 | if opt.random_starts > 0 then 153 | screen, reward, terminal = NewGame() 154 | screen, reward, terminal = NextRandGame() 155 | else 156 | screen, reward, terminal = NewGame() 157 | end 158 | end 159 | -- print(screen:size()) 160 | -- display screen 161 | win = image.display({image=screen, win=win}) 162 | 163 | if step % opt.prog_freq == 0 then 164 | assert(step==agent.numSteps, 'trainer step: ' .. step .. 165 | ' & agent.numSteps: ' .. agent.numSteps) 166 | print("Steps: ", step) 167 | agent:report() 168 | collectgarbage() 169 | end 170 | 171 | if step%1000 == 0 then collectgarbage() end 172 | 173 | if step % opt.eval_freq == 0 and step > learn_start then 174 | screen, reward, terminal = NewGame() 175 | total_reward = 0 176 | nrewards = 0 177 | nepisodes = 0 178 | episode_reward = 0 179 | local eval_time = sys.clock() 180 | for estep=1,opt.eval_steps do 181 | local action_index = agent:perceive(reward, screen, terminal, true, 0.05) 182 | -- Play game in test mode (episodes don't end when losing a life) 183 | -- screen, reward, terminal = game_env:step(game_actions[action_index]) 184 | -- display screen 185 | win = image.display({image=screen, win=win}) 186 | if not terminal then 187 | -- screen, reward, terminal = game_env:step(game_actions[action_index], true) 188 | screen, reward, terminal = EvasionGame(game_actions[action_index]) 189 | end 190 | 191 | if estep%1000 == 0 then collectgarbage() end 192 | 193 | -- record every reward 194 | episode_reward = episode_reward + reward 195 | if reward ~= 0 then 196 | nrewards = nrewards + 1 197 | end 198 | 199 | if terminal then 200 | total_reward = total_reward + episode_reward 201 | episode_reward = 0 202 | nepisodes = nepisodes + 1 203 | -- screen, reward, terminal = game_env:nextRandomGame() 204 | screen, reward, terminal = NewGame() 205 | screen, reward, terminal = NextRandGame() 206 | end 207 | end 208 | 209 | eval_time = sys.clock() - eval_time 210 | start_time = start_time + eval_time 211 | agent:compute_validation_statistics() 212 | local ind = #reward_history+1 213 | total_reward = total_reward/math.max(1, nepisodes) 214 | 215 | if #reward_history == 0 or total_reward > torch.Tensor(reward_history):max() then 216 | agent.best_network = agent.network:clone() 217 | end 218 | 219 | if agent.v_avg then 220 | v_history[ind] = agent.v_avg 221 | td_history[ind] = agent.tderr_avg 222 | qmax_history[ind] = agent.q_max 223 | end 224 | print("V", v_history[ind], "TD error", td_history[ind], "Qmax", qmax_history[ind]) 225 | 226 | reward_history[ind] = total_reward 227 | reward_counts[ind] = nrewards 228 | episode_counts[ind] = nepisodes 229 | 230 | time_history[ind+1] = sys.clock() - start_time 231 | 232 | local time_dif = time_history[ind+1] - time_history[ind] 233 | 234 | local training_rate = opt.actrep*opt.eval_freq/time_dif 235 | 236 | print(string.format( 237 | '\nSteps: %d (frames: %d), reward: %.2f, epsilon: %.2f, lr: %G, ' .. 238 | 'training time: %ds, training rate: %dfps, testing time: %ds, ' .. 239 | 'testing rate: %dfps, num. ep.: %d, num. rewards: %d', 240 | step, step*opt.actrep, total_reward, agent.ep, agent.lr, time_dif, 241 | training_rate, eval_time, opt.actrep*opt.eval_steps/eval_time, 242 | nepisodes, nrewards)) 243 | end 244 | 245 | if step % opt.save_freq == 0 or step == opt.steps then 246 | local s, a, r, s2, term = agent.valid_s, agent.valid_a, agent.valid_r, 247 | agent.valid_s2, agent.valid_term 248 | agent.valid_s, agent.valid_a, agent.valid_r, agent.valid_s2, 249 | agent.valid_term = nil, nil, nil, nil, nil, nil, nil 250 | local w, dw, g, g2, delta, delta2, deltas, tmp = agent.w, agent.dw, 251 | agent.g, agent.g2, agent.delta, agent.delta2, agent.deltas, agent.tmp 252 | agent.w, agent.dw, agent.g, agent.g2, agent.delta, agent.delta2, 253 | agent.deltas, agent.tmp = nil, nil, nil, nil, nil, nil, nil, nil 254 | 255 | local filename = opt.name 256 | if opt.save_versions > 0 then 257 | filename = filename .. "_" .. math.floor(step / opt.save_versions) 258 | end 259 | filename = filename 260 | torch.save(filename .. ".t7", {agent = agent, 261 | model = agent.network, 262 | best_model = agent.best_network, 263 | reward_history = reward_history, 264 | reward_counts = reward_counts, 265 | episode_counts = episode_counts, 266 | time_history = time_history, 267 | v_history = v_history, 268 | td_history = td_history, 269 | qmax_history = qmax_history, 270 | arguments=opt}) 271 | if opt.saveNetworkParams then 272 | local nets = {network=w:clone():float()} 273 | torch.save(filename..'.params.t7', nets, 'ascii') 274 | end 275 | agent.valid_s, agent.valid_a, agent.valid_r, agent.valid_s2, 276 | agent.valid_term = s, a, r, s2, term 277 | agent.w, agent.dw, agent.g, agent.g2, agent.delta, agent.delta2, 278 | agent.deltas, agent.tmp = w, dw, g, g2, delta, delta2, deltas, tmp 279 | print('Saved:', filename .. '.t7') 280 | io.flush() 281 | collectgarbage() 282 | end 283 | end -------------------------------------------------------------------------------- /dqn/train_agent_gpu.lua: -------------------------------------------------------------------------------- 1 | require "initenv" 2 | 3 | Gameindex=4 4 | 5 | NETWORK="DQN3_0_1_Evasion_FULL_Y.t7" 6 | agent="NeuralQLearner" 7 | n_replay=1 8 | netfile="\"convnet_evasion\"" 9 | update_freq=4 10 | actrep=4 11 | discount=0.99 12 | seed=1 13 | learn_start=50000 14 | pool_frms_type="\"max\"" 15 | pool_frms_size=2 16 | initial_priority="false" 17 | replay_memory=1000000 18 | eps_end=0.1 19 | eps_endt=replay_memory 20 | lr=0.00025 21 | agent_type="DQN3_0_1" 22 | preproc_net="\"net_downsample_2x_full_y\"" 23 | agent_name="DQN3_0_1_Evasion_FULL_Y" 24 | state_dim=7056 25 | ncols=1 26 | 27 | agent_params="lr=" .. lr .. ",ep=1,ep_end=" .. eps_end .. ",ep_endt=" .. eps_endt .. ",discount=" .. discount .. ",hist_len=4,learn_start=" .. learn_start .. ",replay_memory=" .. replay_memory .. ",update_freq=" .. update_freq .. ",n_replay=" .. n_replay .. ",network=" .. netfile .. ",preproc=" .. preproc_net .. ",state_dim=" .. state_dim .. ",minibatch_size=32,rescale_r=1,ncols=" .. ncols .. ",bufferSize=512,valid_size=500,target_q=10000,clip_delta=1,min_reward=-1,max_reward=1" 28 | steps=6000000 29 | 30 | prog_freq=50000 31 | save_freq=50000 32 | 33 | eval_freq=50000 34 | eval_steps=50000 35 | 36 | gpu=0 37 | random_starts=30 38 | pool_frms="type=\"max\",size=2" 39 | num_threads=4 40 | 41 | -- args="-name " .. agent_name .. " -agent " .. agent .. " -agent_params " .. agent_params .. " -steps " .. steps .. " -eval_freq " .. eval_freq .. " -eval_steps " .. eval_steps .. " -prog_freq " .. prog_freq .. " -save_freq " .. save_freq .. " -actrep " .. actrep .. " -gpu " .. gpu .. " -random_starts " .. random_starts .. " -pool_frms " .. pool_frms .. " -seed " .. seed .. " -threads " .. num_threads 42 | -- print(args) 43 | 44 | arg[1] = "-name" 45 | arg[2] = agent_name 46 | arg[3] = "-agent" 47 | arg[4] = agent 48 | arg[5] = "-agent_params" 49 | arg[6] = agent_params 50 | arg[7] = "-steps" 51 | arg[8] = steps 52 | arg[9] = "-eval_freq" 53 | arg[10] = eval_freq 54 | arg[11] = "-eval_steps" 55 | arg[12] = eval_steps 56 | arg[13] = "-prog_freq" 57 | arg[14] = prog_freq 58 | 59 | arg[15] = "-save_freq" 60 | arg[16] = save_freq 61 | arg[17] = "-actrep" 62 | arg[18] = actrep 63 | arg[19] = "-gpu" 64 | arg[20] = gpu 65 | arg[21] = "-random_starts" 66 | arg[22] = random_starts 67 | arg[23] = "-pool_frms" 68 | arg[24] = pool_frms 69 | arg[25] = "-seed" 70 | arg[26] = seed 71 | arg[27] = "-threads" 72 | arg[28] = num_threads 73 | arg[29] = "-network" 74 | arg[30] = NETWORK 75 | 76 | local cmd = torch.CmdLine() 77 | cmd:text() 78 | cmd:text('Train Agent in Environment:') 79 | cmd:text() 80 | cmd:text('Options:') 81 | 82 | cmd:option('-pool_frms', '', 83 | 'string of frame pooling parameters (e.g.: size=2,type="max")') 84 | cmd:option('-actrep', 1, 'how many times to repeat action') 85 | cmd:option('-random_starts', 0, 'play action 0 between 1 and random_starts ' .. 86 | 'number of times at the start of each training episode') 87 | 88 | cmd:option('-name', '', 'filename used for saving network and training history') 89 | cmd:option('-network', '', 'reload pretrained network') 90 | cmd:option('-agent', '', 'name of agent file to use') 91 | cmd:option('-agent_params', '', 'string of agent parameters') 92 | cmd:option('-seed', 1, 'fixed input seed for repeatable experiments') 93 | cmd:option('-saveNetworkParams', false, 94 | 'saves the agent network in a separate file') 95 | cmd:option('-prog_freq', 5*10^3, 'frequency of progress output') 96 | cmd:option('-save_freq', 5*10^4, 'the model is saved every save_freq steps') 97 | cmd:option('-eval_freq', 10^4, 'frequency of greedy evaluation') 98 | cmd:option('-save_versions', 0, '') 99 | 100 | cmd:option('-steps', 10^5, 'number of training steps to perform') 101 | cmd:option('-eval_steps', 10^5, 'number of evaluation steps') 102 | 103 | cmd:option('-verbose', 2, 104 | 'the higher the level, the more information is printed to screen') 105 | cmd:option('-threads', 1, 'number of BLAS threads') 106 | cmd:option('-gpu', -1, 'gpu flag') 107 | 108 | cmd:text() 109 | 110 | local opt = cmd:parse(arg) 111 | 112 | --- General setup. 113 | local agent, opt = setup(opt) 114 | 115 | -- override print to always flush the output 116 | local old_print = print 117 | local print = function(...) 118 | old_print(...) 119 | io.flush() 120 | end 121 | 122 | local learn_start = agent.learn_start 123 | local start_time = sys.clock() 124 | local reward_counts = {} 125 | local episode_counts = {} 126 | local time_history = {} 127 | local v_history = {} 128 | local qmax_history = {} 129 | local td_history = {} 130 | local reward_history = {} 131 | local step_history = {} 132 | local maxqrewardstep={} 133 | local step = 0 134 | local qmax_ave_history={} 135 | time_history[1] = 0 136 | 137 | local total_reward 138 | local nrewards 139 | local nepisodes 140 | local episode_reward 141 | local total_qmax 142 | 143 | game_actions = {0,1,2,3,4} 144 | 145 | parainit() 146 | 147 | print('Game:',Gameindex) 148 | local screen, reward, terminal = NewGame(Gameindex) 149 | local action_index=0 150 | print("Iteration ..", step) 151 | local win = nil 152 | 153 | local SnapshotCnt=0 154 | 155 | if Snapshot and arg[29] and arg[30] then 156 | Snapshot=false 157 | local msg, err = pcall(require, NETWORK) 158 | if not msg then 159 | -- try to load saved agent 160 | -- print(self.network) 161 | local err_msg, exp = pcall(torch.load, NETWORK) 162 | if not err_msg then 163 | error("Could not find network file ") 164 | end 165 | if exp.model and exp.reward_history and exp.step_history and exp.maxqrewardstep and exp.step and exp.qmax_ave_history 166 | and exp.time_history then 167 | Snapshot=true 168 | reward_history = exp.reward_history 169 | step_history = exp.step_history 170 | maxqrewardstep = exp.maxqrewardstep 171 | step = exp.step 172 | qmax_ave_history = exp.qmax_ave_history 173 | time_history = exp.time_history 174 | else 175 | reward_history = {} 176 | step_history = {} 177 | maxqrewardstep={} 178 | step = 0 179 | qmax_ave_history={} 180 | end 181 | end 182 | end 183 | 184 | print('Snapshot',Snapshot,'step',step) 185 | 186 | while step < opt.steps do 187 | if Snapshot then 188 | SnapshotCnt = SnapshotCnt+1 189 | -- print(SnapshotCnt) 190 | action_index = agent:perceive(reward, screen, terminal) 191 | -- game over? get next game! 192 | if not terminal then 193 | -- screen, reward, terminal = game_env:step(game_actions[action_index], true) 194 | screen, reward, terminal = EvasionGame(game_actions[action_index],Gameindex) 195 | else 196 | -- print(opt.random_starts) 197 | if opt.random_starts > 0 then 198 | -- screen, reward, terminal = NewGame(Gameindex) 199 | screen, reward, terminal = NextRandGame(Gameindex,opt.random_starts) 200 | else 201 | screen, reward, terminal = NewGame(Gameindex) 202 | end 203 | end 204 | -- display screen 205 | win = image.display({image=screen, win=win}) 206 | if SnapshotCnt > learn_start then 207 | Snapshot = false 208 | step = step + 1 209 | end 210 | else 211 | step = step + 1 212 | action_index = agent:perceive(reward, screen, terminal) 213 | -- print(step) 214 | -- game over? get next game! 215 | if not terminal then 216 | -- screen, reward, terminal = game_env:step(game_actions[action_index], true) 217 | screen, reward, terminal = EvasionGame(game_actions[action_index],Gameindex) 218 | else 219 | if opt.random_starts > 0 then 220 | -- screen, reward, terminal = NewGame(Gameindex) 221 | screen, reward, terminal = NextRandGame(Gameindex,opt.random_starts) 222 | else 223 | screen, reward, terminal = NewGame(Gameindex) 224 | end 225 | end 226 | -- print(screen:size()) 227 | -- display screen 228 | win = image.display({image=screen, win=win}) 229 | 230 | if step % opt.prog_freq == 0 then 231 | assert(step==agent.numSteps, 'trainer step: ' .. step .. 232 | ' & agent.numSteps: ' .. agent.numSteps) 233 | print("Steps: ", step) 234 | print('Game:',Gameindex) 235 | agent:report() 236 | collectgarbage() 237 | end 238 | 239 | if step%1000 == 0 then collectgarbage() end 240 | 241 | if step % opt.eval_freq == 0 and step > learn_start then 242 | 243 | screen, reward, terminal = NewGame(Gameindex,true) 244 | total_reward = 0 245 | total_qmax=0 246 | nrewards = 0 247 | nepisodes = 0 248 | episode_reward = 0 249 | local maxreward=0 250 | local maxstep=0 251 | local maxq=0 252 | local eval_time = sys.clock() 253 | local cntstep=0 254 | for estep=1,opt.eval_steps do 255 | action_index = agent:perceive(reward, screen, terminal, true, 0.05) 256 | -- Play game in test mode (episodes don't end when losing a life) 257 | -- screen, reward, terminal = game_env:step(game_actions[action_index]) 258 | -- display screen 259 | win = image.display({image=screen, win=win}) 260 | if not terminal then 261 | -- screen, reward, terminal = game_env:step(game_actions[action_index], true) 262 | screen, reward, terminal = EvasionGame(game_actions[action_index],Gameindex) 263 | end 264 | 265 | if estep%1000 == 0 then collectgarbage() end 266 | 267 | -- record every reward 268 | episode_reward = episode_reward + reward 269 | if reward ~= 0 then 270 | nrewards = nrewards + 1 271 | end 272 | 273 | cntstep=cntstep+1 274 | 275 | if terminal then 276 | total_reward = total_reward + episode_reward 277 | 278 | nepisodes = nepisodes+1 279 | if cntstep>maxstep then 280 | maxstep=cntstep 281 | end 282 | 283 | if episode_reward>maxreward then 284 | maxreward=episode_reward 285 | end 286 | 287 | cntstep=0 288 | episode_reward = 0 289 | 290 | -- screen, reward, terminal = NewGame(Gameindex) 291 | screen, reward, terminal = NextRandGame(Gameindex,opt.random_starts,true) 292 | end 293 | if maxq torch.Tensor(reward_history):max() then 320 | agent.best_network = agent.network:clone() 321 | end 322 | 323 | -- if agent.v_avg then 324 | -- v_history[ind] = agent.v_avg 325 | -- td_history[ind] = agent.tderr_avg 326 | -- qmax_history[ind] = agent.q_max 327 | -- end 328 | 329 | -- print("V", v_history[ind], "TD error", td_history[ind], "Qmax", qmax_history[ind]) 330 | 331 | reward_history[ind] = total_reward 332 | local filename1 = "reward_history" 333 | torch.save(filename1 .. ".t7",reward_history,'ascii') 334 | 335 | step_history[ind] = opt.eval_steps/nepisodes 336 | local filename2 = "step_history" 337 | torch.save(filename2 .. ".t7",step_history,'ascii') 338 | 339 | qmax_ave_history[ind] = total_qmax 340 | local filename3 = "qmax" 341 | torch.save(filename3 .. ".t7",qmax_ave_history,'ascii') 342 | 343 | maxqrewardstep[ind]=torch.Tensor{maxq,maxreward,maxstep} 344 | local filename4 = "maxqrewardstep" 345 | torch.save(filename4 .. ".t7",maxqrewardstep,'ascii') 346 | 347 | reward_counts[ind] = nrewards 348 | episode_counts[ind] = nepisodes 349 | 350 | time_history[ind+1] = sys.clock() - start_time 351 | 352 | local time_dif = time_history[ind+1] - time_history[ind] 353 | 354 | local training_rate = opt.actrep*opt.eval_freq/time_dif 355 | 356 | print(string.format( 357 | '\nSteps: %d (frames: %d), reward: %.2f, epsilon: %.2f, lr: %G, ' .. 358 | 'training time: %ds, training rate: %dfps, testing time: %ds, ' .. 359 | 'testing rate: %dfps, num. ep.: %d, num. rewards: %d', 360 | step, step*opt.actrep, total_reward, agent.ep, agent.lr, time_dif, 361 | training_rate, eval_time, opt.actrep*opt.eval_steps/eval_time, 362 | nepisodes, nrewards)) 363 | end 364 | 365 | if step % opt.save_freq == 0 or step == opt.steps then 366 | print('Game:',Gameindex) 367 | -- if step % 100 == 0 or step == opt.steps then 368 | local s, a, r, s2, term = agent.valid_s, agent.valid_a, agent.valid_r, 369 | agent.valid_s2, agent.valid_term 370 | agent.valid_s, agent.valid_a, agent.valid_r, agent.valid_s2, 371 | agent.valid_term = nil, nil, nil, nil, nil, nil, nil 372 | local w, dw, g, g2, delta, delta2, deltas, tmp = agent.w, agent.dw, 373 | agent.g, agent.g2, agent.delta, agent.delta2, agent.deltas, agent.tmp 374 | agent.w, agent.dw, agent.g, agent.g2, agent.delta, agent.delta2, 375 | agent.deltas, agent.tmp = nil, nil, nil, nil, nil, nil, nil, nil 376 | 377 | local filename = opt.name 378 | if opt.save_versions > 0 then 379 | filename = filename .. "_" .. math.floor(step / opt.save_versions) 380 | end 381 | filename = filename 382 | torch.save(filename .. ".t7", {agent = agent, 383 | model = agent.network, 384 | best_model = agent.best_network, 385 | reward_history = reward_history, 386 | step_history = step_history, 387 | qmax_ave_history=qmax_ave_history, 388 | maxqrewardstep=maxqrewardstep, 389 | reward_counts = reward_counts, 390 | episode_counts = episode_counts, 391 | time_history = time_history, 392 | v_history = v_history, 393 | td_history = td_history, 394 | qmax_history = qmax_history, 395 | arguments=opt, 396 | step=step 397 | }) 398 | 399 | if opt.saveNetworkParams then 400 | local nets = {network=w:clone():float()} 401 | torch.save(filename..'.params.t7', nets, 'ascii') 402 | end 403 | agent.valid_s, agent.valid_a, agent.valid_r, agent.valid_s2, 404 | agent.valid_term = s, a, r, s2, term 405 | agent.w, agent.dw, agent.g, agent.g2, agent.delta, agent.delta2, 406 | agent.deltas, agent.tmp = w, dw, g, g2, delta, delta2, deltas, tmp 407 | print('Saved:', filename .. '.t7') 408 | io.flush() 409 | collectgarbage() 410 | end 411 | end 412 | end -------------------------------------------------------------------------------- /gifs/3.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zhujiagang/pursuit-evasion-code/c0bcfb6f8a45f495152f559a0612f02f1e5ef357/gifs/3.gif -------------------------------------------------------------------------------- /gifs/Evasion_Test.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zhujiagang/pursuit-evasion-code/c0bcfb6f8a45f495152f559a0612f02f1e5ef357/gifs/Evasion_Test.gif -------------------------------------------------------------------------------- /install_depen2.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | ###################################################################### 4 | # Torch install 5 | ###################################################################### 6 | 7 | 8 | TOPDIR=$PWD 9 | rm -rf /home/user/torch 10 | # Prefix: 11 | PREFIX=/home/user/torch/install 12 | echo "Installing Torch into: $PREFIX" 13 | 14 | if [[ `uname` != 'Linux' ]]; then 15 | echo 'Platform unsupported, only available for Linux' 16 | exit 17 | fi 18 | if [[ `which apt-get` == '' ]]; then 19 | echo 'apt-get not found, platform not supported' 20 | exit 21 | fi 22 | 23 | # Install dependencies for Torch: 24 | sudo apt-get update 25 | sudo apt-get install -qqy build-essential 26 | sudo apt-get install -qqy gcc g++ 27 | sudo apt-get install -qqy cmake 28 | sudo apt-get install -qqy curl 29 | sudo apt-get install -qqy libreadline-dev 30 | sudo apt-get install -qqy git-core 31 | sudo apt-get install -qqy libjpeg-dev 32 | sudo apt-get install -qqy libpng-dev 33 | sudo apt-get install -qqy ncurses-dev 34 | sudo apt-get install -qqy imagemagick 35 | sudo apt-get install -qqy unzip 36 | sudo apt-get install -qqy libqt4-dev 37 | sudo apt-get install -qqy liblua5.1-0-dev 38 | sudo apt-get install -qqy libgd-dev 39 | sudo apt-get update 40 | 41 | 42 | echo "==> Torch7's dependencies have been installed" 43 | 44 | 45 | 46 | 47 | 48 | # Build and install Torch7 49 | cd /tmp 50 | rm -rf luajit-rocks 51 | git clone https://github.com/torch/luajit-rocks.git 52 | cd luajit-rocks 53 | mkdir -p build 54 | cd build 55 | git checkout master; git pull 56 | rm -f CMakeCache.txt 57 | cmake .. -DCMAKE_INSTALL_PREFIX=$PREFIX -DCMAKE_BUILD_TYPE=Release 58 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 59 | make 60 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 61 | make install 62 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 63 | 64 | export PATH=/usr/local/cuda-8.0/bin:$PATH 65 | export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:$LD_LIBRARY_PATH 66 | path_to_nvcc=$(which nvcc) 67 | if [ -x "$path_to_nvcc" ] 68 | then 69 | cutorch=ok 70 | cunn=ok 71 | fi 72 | 73 | # Install base packages: 74 | $PREFIX/bin/luarocks install cwrap 75 | $PREFIX/bin/luarocks install paths 76 | $PREFIX/bin/luarocks install torch 77 | $PREFIX/bin/luarocks install nn 78 | [ -n "$cutorch" ] && \ 79 | ($PREFIX/bin/luarocks install cutorch) 80 | [ -n "$cunn" ] && \ 81 | ($PREFIX/bin/luarocks install cunn) 82 | 83 | $PREFIX/bin/luarocks install luafilesystem 84 | $PREFIX/bin/luarocks install penlight 85 | $PREFIX/bin/luarocks install sys 86 | $PREFIX/bin/luarocks install xlua 87 | $PREFIX/bin/luarocks install image 88 | $PREFIX/bin/luarocks install env 89 | $PREFIX/bin/luarocks install qtlua 90 | $PREFIX/bin/luarocks install qttorch 91 | 92 | sudo apt-get install gnuplot 93 | sudo apt-get install gnuplot-qt 94 | echo "" 95 | echo "=> Torch7 has been installed successfully" 96 | echo "" 97 | 98 | 99 | echo "Installing nngraph ... " 100 | $PREFIX/bin/luarocks install nngraph 101 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 102 | echo "nngraph installation completed" 103 | 104 | echo "Installing Xitari ... " 105 | cd /tmp 106 | rm -rf xitari 107 | git clone https://github.com/deepmind/xitari.git 108 | cd xitari 109 | $PREFIX/bin/luarocks make 110 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 111 | echo "Xitari installation completed" 112 | 113 | echo "Installing Alewrap ... " 114 | cd /tmp 115 | rm -rf alewrap 116 | git clone https://github.com/deepmind/alewrap.git 117 | cd alewrap 118 | $PREFIX/bin/luarocks make 119 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 120 | echo "Alewrap installation completed" 121 | 122 | echo "Installing Lua-GD ... " 123 | mkdir $PREFIX/src 124 | cd $PREFIX/src 125 | rm -rf lua-gd 126 | git clone https://github.com/ittner/lua-gd.git 127 | cd lua-gd 128 | sed -i "s/LUABIN=lua5.1/LUABIN=..\/..\/bin\/luajit/" Makefile 129 | $PREFIX/bin/luarocks make 130 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 131 | echo "Lua-GD installation completed" 132 | 133 | 134 | 135 | 136 | echo 137 | echo "You can run experiments by executing: " 138 | echo 139 | echo " ./run_cpu game_name" 140 | echo 141 | echo " or " 142 | echo 143 | echo " ./run_gpu game_name" 144 | echo 145 | echo "For this you need to provide the rom files of the respective games (game_name.bin) in the roms/ directory" 146 | echo 147 | 148 | -------------------------------------------------------------------------------- /install_dependencies.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | ###################################################################### 4 | # Torch install 5 | ###################################################################### 6 | 7 | 8 | TOPDIR=$PWD 9 | 10 | # Prefix: 11 | PREFIX=$PWD/torch 12 | echo "Installing Torch into: $PREFIX" 13 | 14 | if [[ `uname` != 'Linux' ]]; then 15 | echo 'Platform unsupported, only available for Linux' 16 | exit 17 | fi 18 | if [[ `which apt-get` == '' ]]; then 19 | echo 'apt-get not found, platform not supported' 20 | exit 21 | fi 22 | 23 | # Install dependencies for Torch: 24 | sudo apt-get update 25 | sudo apt-get install -qqy build-essential 26 | sudo apt-get install -qqy gcc g++ 27 | sudo apt-get install -qqy cmake 28 | sudo apt-get install -qqy curl 29 | sudo apt-get install -qqy libreadline-dev 30 | sudo apt-get install -qqy git-core 31 | sudo apt-get install -qqy libjpeg-dev 32 | sudo apt-get install -qqy libpng-dev 33 | sudo apt-get install -qqy ncurses-dev 34 | sudo apt-get install -qqy imagemagick 35 | sudo apt-get install -qqy unzip 36 | sudo apt-get install -qqy libqt4-dev 37 | sudo apt-get install -qqy liblua5.1-0-dev 38 | sudo apt-get install -qqy libgd-dev 39 | sudo apt-get update 40 | 41 | 42 | echo "==> Torch7's dependencies have been installed" 43 | 44 | 45 | 46 | 47 | 48 | # Build and install Torch7 49 | cd /tmp 50 | rm -rf luajit-rocks 51 | git clone https://github.com/torch/luajit-rocks.git 52 | cd luajit-rocks 53 | mkdir -p build 54 | cd build 55 | git checkout master; git pull 56 | rm -f CMakeCache.txt 57 | cmake .. -DCMAKE_INSTALL_PREFIX=$PREFIX -DCMAKE_BUILD_TYPE=Release 58 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 59 | make 60 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 61 | make install 62 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 63 | 64 | 65 | path_to_nvcc=$(which nvcc) 66 | if [ -x "$path_to_nvcc" ] 67 | then 68 | cutorch=ok 69 | cunn=ok 70 | fi 71 | 72 | # Install base packages: 73 | $PREFIX/bin/luarocks install cwrap 74 | $PREFIX/bin/luarocks install paths 75 | $PREFIX/bin/luarocks install torch 76 | $PREFIX/bin/luarocks install nn 77 | 78 | [ -n "$cutorch" ] && \ 79 | ($PREFIX/bin/luarocks install cutorch) 80 | [ -n "$cunn" ] && \ 81 | ($PREFIX/bin/luarocks install cunn) 82 | 83 | $PREFIX/bin/luarocks install luafilesystem 84 | $PREFIX/bin/luarocks install penlight 85 | $PREFIX/bin/luarocks install sys 86 | $PREFIX/bin/luarocks install xlua 87 | $PREFIX/bin/luarocks install image 88 | $PREFIX/bin/luarocks install env 89 | $PREFIX/bin/luarocks install qtlua 90 | $PREFIX/bin/luarocks install qttorch 91 | 92 | echo "" 93 | echo "=> Torch7 has been installed successfully" 94 | echo "" 95 | 96 | 97 | echo "Installing nngraph ... " 98 | $PREFIX/bin/luarocks install nngraph 99 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 100 | echo "nngraph installation completed" 101 | 102 | echo "Installing Xitari ... " 103 | cd /tmp 104 | rm -rf xitari 105 | git clone https://github.com/deepmind/xitari.git 106 | cd xitari 107 | $PREFIX/bin/luarocks make 108 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 109 | echo "Xitari installation completed" 110 | 111 | echo "Installing Alewrap ... " 112 | cd /tmp 113 | rm -rf alewrap 114 | git clone https://github.com/deepmind/alewrap.git 115 | cd alewrap 116 | $PREFIX/bin/luarocks make 117 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 118 | echo "Alewrap installation completed" 119 | 120 | echo "Installing Lua-GD ... " 121 | mkdir $PREFIX/src 122 | cd $PREFIX/src 123 | rm -rf lua-gd 124 | git clone https://github.com/ittner/lua-gd.git 125 | cd lua-gd 126 | sed -i "s/LUABIN=lua5.1/LUABIN=..\/..\/bin\/luajit/" Makefile 127 | $PREFIX/bin/luarocks make 128 | RET=$?; if [ $RET -ne 0 ]; then echo "Error. Exiting."; exit $RET; fi 129 | echo "Lua-GD installation completed" 130 | 131 | echo 132 | echo "You can run experiments by executing: " 133 | echo 134 | echo " ./run_cpu game_name" 135 | echo 136 | echo " or " 137 | echo 138 | echo " ./run_gpu game_name" 139 | echo 140 | echo "For this you need to provide the rom files of the respective games (game_name.bin) in the roms/ directory" 141 | echo 142 | 143 | -------------------------------------------------------------------------------- /run_cpu: -------------------------------------------------------------------------------- 1 | 2 | cd dqn 3 | 4 | /home/user/torch/install/bin/qlua train_agent_cpu.lua 5 | -------------------------------------------------------------------------------- /run_gpu: -------------------------------------------------------------------------------- 1 | cd dqn 2 | export PATH=/usr/local/cuda-8.0/bin:$PATH 3 | export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:$LD_LIBRARY_PATH 4 | find ./ -name "*.jpg" | xargs rm -rf 5 | /home/user/torch/install/bin/qlua train_agent_gpu.lua -------------------------------------------------------------------------------- /test_cpu: -------------------------------------------------------------------------------- 1 | 2 | cd dqn 3 | 4 | /home/user/torch/install/bin/qlua test_agent_cpu.lua 5 | -------------------------------------------------------------------------------- /test_gpu: -------------------------------------------------------------------------------- 1 | cd dqn 2 | export PATH=/usr/local/cuda-8.0/bin:$PATH 3 | export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:$LD_LIBRARY_PATH 4 | find ./ -name "*.jpg" | xargs rm -rf 5 | /home/user/torch/install/bin/qlua test_agent_gpu.lua --------------------------------------------------------------------------------