├── .gitignore ├── Deep_Q_Learning.pdf ├── EXPERIMENTS_DQL ├── agents │ ├── 40000_MLP-DQN │ ├── 500000_MLP-DQN │ └── 5000_MLP-DQN ├── dqn.py ├── dqn_helpers.py ├── enjoy_dense.py ├── figures │ ├── ddqn_01.png │ ├── ddqn_02.png │ ├── ddqn_03.png │ ├── ddqn_04.png │ ├── ddqn_gamma.png │ ├── ddqn_target_update.png │ ├── dqn-timeline.jpeg │ ├── dqn_batch_size.png │ ├── dqn_batchsize.png │ ├── dqn_buffer.png │ ├── dqn_lrate.png │ ├── dqn_target_update.png │ ├── dueling_dqn_all.png │ ├── dueling_dqn_arch.png │ ├── per_dqn_er.png │ ├── per_dqn_is.png │ └── per_dqn_temp.png ├── general_helpers.py ├── gridworld │ ├── __init__.py │ ├── __init__.pyc │ └── dense.py ├── logs │ └── old │ │ ├── ddqn_gamma_09.csv │ │ ├── ddqn_gamma_095.csv │ │ ├── ddqn_gamma_099.csv │ │ ├── ddqn_soft_update_001.csv │ │ ├── ddqn_soft_update_005.csv │ │ ├── ddqn_soft_update_01.csv │ │ ├── double_dueling_dqn.csv │ │ ├── dqn_batchsize_128.csv │ │ ├── dqn_batchsize_32.csv │ │ ├── dqn_batchsize_64.csv │ │ ├── dqn_capacity_100.csv │ │ ├── dqn_capacity_1000.csv │ │ ├── dqn_capacity_20000.csv │ │ ├── dqn_capacity_40000.csv │ │ ├── dqn_capacity_5000.csv │ │ ├── dqn_hard_update_1.csv │ │ ├── dqn_hard_update_100.csv │ │ ├── dqn_lrate_1e3.csv │ │ ├── dqn_lrate_1e4.csv │ │ ├── dqn_lrate_1e5.csv │ │ ├── dqn_soft_update_001.csv │ │ ├── dqn_soft_update_005.csv │ │ ├── dueling_dqn.csv │ │ ├── dueling_dqn_128.csv │ │ ├── dueling_dqn_256.csv │ │ ├── dueling_dqn_64.csv │ │ ├── per_double_dueling_dqn.csv │ │ ├── per_dqn_alpha_02.csv │ │ ├── per_dqn_alpha_05.csv │ │ ├── per_dqn_alpha_08.csv │ │ ├── per_dqn_beta_02.csv │ │ ├── per_dqn_beta_05.csv │ │ ├── per_dqn_beta_08.csv │ │ ├── per_dqn_capacity_100.csv │ │ ├── per_dqn_capacity_1000.csv │ │ ├── per_dqn_capacity_20000.csv │ │ ├── per_dqn_capacity_40000.csv │ │ ├── per_dqn_capacity_5000.csv │ │ └── per_dueling_dqn.csv ├── movies │ ├── 40000_MLP-DQN.mp4 │ ├── 500000_MLP-DQN.mp4 │ ├── 5000_MLP-DQN.mp4 │ ├── DQN-Gridworld.gif │ └── DQN-Gridworld.mp4 ├── requirements.txt ├── run_experiments_dqn.sh ├── train_dqn.py └── viz_results.ipynb ├── EXPERIMENTS_PG ├── configs │ ├── base_a2c.json │ ├── base_ddpg.json │ └── base_vpg.json ├── experiments │ └── 2020-04-09_base_a2c │ │ ├── 2020-04-09_base_a2c.ckpth │ │ ├── 2020-04-09_base_a2c.hdf5 │ │ └── 2020-04-09_base_a2c.json ├── train_policy_gradients.py └── viz_results.ipynb └── Readme.md /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints/ 2 | .DS_Store 3 | .sync-config.cson 4 | __pycache__ 5 | Deep_Q_Learning.key 6 | Policy_Gradients.key 7 | -------------------------------------------------------------------------------- /Deep_Q_Learning.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/Deep_Q_Learning.pdf -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/agents/40000_MLP-DQN: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/agents/40000_MLP-DQN -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/agents/500000_MLP-DQN: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/agents/500000_MLP-DQN -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/agents/5000_MLP-DQN: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/agents/5000_MLP-DQN -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/dqn.py: -------------------------------------------------------------------------------- 1 | import math 2 | import random 3 | import numpy as np 4 | 5 | import torch 6 | import torch.nn as nn 7 | import torch.autograd as autograd 8 | import torch.optim as optim 9 | 10 | from general_helpers import init_weights 11 | import warnings 12 | warnings.filterwarnings("ignore") 13 | 14 | # Set device config variables 15 | USE_CUDA = torch.cuda.is_available() 16 | Variable = lambda *args, **kwargs: autograd.Variable(*args, **kwargs).cuda() if USE_CUDA else autograd.Variable(*args, **kwargs) 17 | 18 | 19 | def init_dqn(model, L_RATE, USE_CUDA, 20 | INPUT_DIM, HIDDEN_SIZE, NUM_ACTIONS, 21 | load_checkpoint_path=None): 22 | """ 23 | Out: Model (or dictionay) as well as optimizer 24 | """ 25 | agents = {"current": model(INPUT_DIM, HIDDEN_SIZE, NUM_ACTIONS), 26 | "target": model(INPUT_DIM, HIDDEN_SIZE, NUM_ACTIONS)} 27 | 28 | # Copy over initial parameters to the target network 29 | for target_param, param in zip(agents["target"].parameters(), 30 | agents["current"].parameters()): 31 | target_param.data.copy_(param.data) 32 | 33 | if USE_CUDA: 34 | agents["current"] = agents["current"].cuda() 35 | agents["target"] = agents["target"].cuda() 36 | 37 | if load_checkpoint_path is not None: 38 | checkpoint = torch.load(load_checkpoint_path, 39 | map_location='cpu') 40 | agents["current"].load_state_dict(checkpoint) 41 | agents["target"].load_state_dict(checkpoint) 42 | 43 | # Initialize optimizer object - single agent 44 | optimizers = optim.Adam(params=agents["current"].parameters(), lr=L_RATE) 45 | return agents, optimizers 46 | 47 | 48 | class MLP_DQN(nn.Module): 49 | def __init__(self, INPUT_DIM, HIDDEN_SIZE, NUM_ACTIONS): 50 | super(MLP_DQN, self).__init__() 51 | self.action_space_size = NUM_ACTIONS 52 | self.layers = nn.Sequential( 53 | nn.Linear(INPUT_DIM, HIDDEN_SIZE), 54 | nn.ReLU(), 55 | nn.Linear(HIDDEN_SIZE, HIDDEN_SIZE), 56 | nn.ReLU(), 57 | nn.Linear(HIDDEN_SIZE, self.action_space_size) 58 | ) 59 | self.layers.apply(init_weights) 60 | 61 | def forward(self, x): 62 | return self.layers(x) 63 | 64 | def act(self, state, epsilon): 65 | if random.random() > epsilon: 66 | state = Variable(torch.FloatTensor(state).unsqueeze(0), volatile=True) 67 | q_value = self.forward(state) 68 | action = q_value.max(1)[1].data[0] 69 | else: 70 | action = random.randrange(self.action_space_size) 71 | return action 72 | 73 | 74 | class MLP_DuelingDQN(nn.Module): 75 | def __init__(self, INPUT_DIM, HIDDEN_SIZE, NUM_ACTIONS): 76 | super(MLP_DuelingDQN, self).__init__() 77 | # Implements a Dueling DQN agent based on MLP 78 | self.action_space_size = NUM_ACTIONS 79 | 80 | self.feature = nn.Sequential( 81 | nn.Linear(INPUT_DIM, HIDDEN_SIZE), 82 | nn.ReLU() 83 | ) 84 | self.feature.apply(init_weights) 85 | 86 | self.advantage = nn.Sequential( 87 | nn.Linear(HIDDEN_SIZE, HIDDEN_SIZE), 88 | nn.ReLU(), 89 | nn.Linear(HIDDEN_SIZE, self.action_space_size) 90 | ) 91 | self.advantage.apply(init_weights) 92 | 93 | self.value = nn.Sequential( 94 | nn.Linear(HIDDEN_SIZE, HIDDEN_SIZE), 95 | nn.ReLU(), 96 | nn.Linear(HIDDEN_SIZE, 1) 97 | ) 98 | self.value.apply(init_weights) 99 | 100 | def forward(self, x): 101 | x = self.feature(x) 102 | advantage = self.advantage(x) 103 | value = self.value(x) 104 | return value + advantage - advantage.mean() 105 | 106 | def act(self, state, epsilon): 107 | if random.random() > epsilon: 108 | state = Variable(torch.FloatTensor(state).unsqueeze(0)) 109 | q_value = self.forward(state) 110 | action = q_value.max(1)[1].data[0] 111 | else: 112 | action = random.randrange(self.action_space_size) 113 | return action 114 | -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/dqn_helpers.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import math 3 | import random 4 | import pandas as pd 5 | import numpy as np 6 | from collections import deque 7 | import gym 8 | 9 | import torch 10 | import torch.autograd as autograd 11 | 12 | 13 | def command_line_dqn(): 14 | parser = argparse.ArgumentParser() 15 | # General logging/saving and device arguments 16 | parser.add_argument('-roll_upd', '--ROLLOUT_EVERY', action="store", 17 | default=500, type=int, 18 | help='Rollout test performance after # batch updates.') 19 | parser.add_argument('-n_roll', '--NUM_ROLLOUTS', action="store", 20 | default=5, type=int, 21 | help='# rollouts for tracking learning progrees') 22 | parser.add_argument('-n_runs', '--RUN_TIMES', action="store", 23 | default=1, type=int, 24 | help='# Times to run agent learning') 25 | parser.add_argument('-n_upd', '--NUM_UPDATES', action="store", 26 | default=80000, type=int, 27 | help='# SGD updates/iterations to train for') 28 | parser.add_argument('-max_steps', '--MAX_STEPS', action="store", 29 | default=200, type=int, 30 | help='Max # of steps before episode terminated') 31 | parser.add_argument('-v', '--VERBOSE', action="store_true", default=False, 32 | help='Get training progress printed out') 33 | parser.add_argument('-print', '--PRINT_EVERY', action="store", 34 | default=5000, type=int, 35 | help='#Episodes after which to print if verbose.') 36 | parser.add_argument('-s', '--SAVE', action="store_true", 37 | default=False, help='Save final agents and log') 38 | parser.add_argument('-s_agent', '--SAVE_AGENT', action="store_true", 39 | default=False, help='Save final agents and log') 40 | parser.add_argument('-device', '--device_id', action="store", 41 | default=0, type=int, help='Device id on which to train') 42 | parser.add_argument('-fname', '--SAVE_FNAME', action="store", 43 | default="temp", type=str, help='Filename to which to save logs') 44 | # Network architecture arguments 45 | parser.add_argument('-input', '--INPUT_DIM', action="store", 46 | default=1200, type=int, help='Input Dimension') 47 | parser.add_argument('-hidden', '--HIDDEN_SIZE', action="store", 48 | default=128, type=int, help='Hidden Dimension') 49 | parser.add_argument('-num_actions', '--NUM_ACTIONS', action="store", 50 | default=4, type=int, help='Number of Actions') 51 | 52 | parser.add_argument('-gamma', '--GAMMA', action="store", 53 | default=0.99, type=float, 54 | help='Discount factor') 55 | parser.add_argument('-l_r', '--L_RATE', action="store", default=0.001, 56 | type=float, help='Save network and learning stats after # epochs') 57 | parser.add_argument('-train_batch', '--TRAIN_BATCH_SIZE', action="store", 58 | default=32, type=int, help='# images in training batch') 59 | 60 | parser.add_argument('-soft_tau', '--SOFT_TAU', action="store", 61 | default=0., type=float, 62 | help='Polyak Averaging tau for target network update') 63 | parser.add_argument('-update_upd', '--UPDATE_EVERY', action="store", 64 | default=500, type=int, 65 | help='Update target network after # batch updates') 66 | 67 | parser.add_argument('-e_start', '--EPS_START', action="store", default=1, 68 | type=float, help='Start Exploration Rate') 69 | parser.add_argument('-e_stop', '--EPS_STOP', action="store", default=0.01, 70 | type=float, help='Start Exploration Rate') 71 | 72 | parser.add_argument('-p', '--PER', action="store_true", default=False, 73 | help='Perform prioritized experience replay sampling update.') 74 | parser.add_argument('-b_start', '--BETA_START', action="store", default=0.4, 75 | type=float, help='Initial beta to start learning with.') 76 | parser.add_argument('-b_steps', '--BETA_STEPS', action="store", default=5000, 77 | type=int, help='Number of steps until which beta is annealed to 1.') 78 | parser.add_argument('-alpha', '--ALPHA', action="store", default=0.6, 79 | type=float, help='Temperature is priority Boltzmann distribution.') 80 | 81 | parser.add_argument('-agent', '--AGENT', action="store", 82 | default="MLP-DQN", type=str, help='Agent model') 83 | parser.add_argument('-d', '--DOUBLE', action="store_true", default=False, 84 | help='Perform double Q-Learning update.') 85 | parser.add_argument('-capacity', '--CAPACITY', action="store", 86 | default=20000, type=int, help='Storage capacity of ER buffer') 87 | 88 | return parser.parse_args() 89 | 90 | 91 | def compute_td_loss(agents, optimizer, replay_buffer, beta, 92 | TRAIN_BATCH_SIZE, GAMMA, Variable, TRAIN_DOUBLE): 93 | 94 | if beta is None: 95 | obs, acts, reward, next_obs, done = replay_buffer.sample(TRAIN_BATCH_SIZE) 96 | else: 97 | obs, acts, reward, next_obs, done, indices, weights = replay_buffer.sample(TRAIN_BATCH_SIZE, beta) 98 | 99 | pyT = lambda array: Variable(torch.FloatTensor(array)) 100 | obs = np.float32([ob.flatten() for ob in obs]) 101 | next_obs = np.float32([next_ob.flatten() for next_ob in next_obs]) 102 | obs, next_obs, reward = pyT(obs), pyT(next_obs), pyT(reward) 103 | action = Variable(torch.LongTensor(acts)) 104 | done = Variable(torch.FloatTensor(done)) 105 | 106 | if beta is not None: 107 | weights = pyT(weights) 108 | 109 | q_values = agents["current"](obs) 110 | q_value = q_values.gather(1, action.unsqueeze(1)).squeeze(1) 111 | 112 | if TRAIN_DOUBLE: 113 | next_q_values = agents["current"](next_obs) 114 | next_q_state_values = agents["target"](next_obs) 115 | next_q_value = next_q_state_values.gather(1, torch.max(next_q_values, 1)[1].unsqueeze(1)).squeeze(1) 116 | else: 117 | next_q_values = agents["target"](next_obs) 118 | next_q_value = next_q_values.max(1)[0] 119 | 120 | expected_q_value = reward + GAMMA* next_q_value * (1 - done) 121 | loss = (q_value - expected_q_value.detach()).pow(2) 122 | 123 | if beta is not None: 124 | loss *= weights 125 | prios = loss + 1e-5 126 | replay_buffer.update_priorities(indices, prios.data.cpu().numpy()) 127 | loss = loss.mean() 128 | 129 | # Perform optimization step for agent 130 | optimizer.zero_grad() 131 | loss.backward() 132 | torch.nn.utils.clip_grad_norm(agents["current"].parameters(), 0.5) 133 | optimizer.step() 134 | return loss 135 | -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/enjoy_dense.py: -------------------------------------------------------------------------------- 1 | import gym 2 | import gridworld 3 | from dqn import init_dqn, MLP_DQN 4 | from pycolab import rendering 5 | import torch 6 | 7 | import numpy as np 8 | import matplotlib 9 | matplotlib.use("Agg") 10 | import matplotlib.pyplot as plt 11 | import matplotlib.animation as manimation 12 | import glob 13 | import os 14 | import subprocess 15 | import argparse 16 | 17 | 18 | def rgb_rescale(v): 19 | return v/255 20 | 21 | 22 | COLOUR_FG = {' ': tuple([rgb_rescale(v) for v in (123, 132, 150)]), # Background 23 | '$': tuple([rgb_rescale(v) for v in (214, 182, 79)]), # Coins 24 | '@': tuple([rgb_rescale(v) for v in (66, 6, 13)]), # Poison 25 | '#': tuple([rgb_rescale(v) for v in (119, 107, 122)]), # Walls of the maze 26 | 'P': tuple([rgb_rescale(v) for v in (153, 85, 74)]), # Player 27 | 'a': tuple([rgb_rescale(v) for v in (107, 132, 102)]), # Patroller A 28 | 'b': tuple([rgb_rescale(v) for v in (107, 132, 102)])} # Patroller B 29 | 30 | 31 | def converter(obs): 32 | converter = rendering.ObservationToArray(COLOUR_FG, permute=(0,1,2)) 33 | converted = np.swapaxes(converter(obs), 1, 2).T 34 | return converted 35 | 36 | 37 | def main(LOAD_CKPT, save_fname, title): 38 | FFMpegWriter = manimation.writers['ffmpeg'] 39 | metadata = dict(title='Movie Test', artist='Matplotlib', 40 | comment='Movie support!') 41 | writer = FFMpegWriter(fps=15, metadata=metadata) 42 | 43 | env = gym.make("dense-v0") 44 | 45 | USE_CUDA = torch.cuda.is_available() 46 | agents, optimizer = init_dqn(MLP_DQN, 0, USE_CUDA, 47 | 1200, 128, 4, LOAD_CKPT) 48 | 49 | obs, screen_obs = env.reset_with_render() 50 | done = False 51 | episode_rew = 0 52 | converted = converter(screen_obs) 53 | my_plot = plt.imshow(converted) 54 | steps = 0 55 | while steps < 50 and not done: 56 | #obs, rew, done, _ , screen_obs = env.step_with_render(act(obs)[0]) 57 | #obs, rew, done, _ , screen_obs = env.step_with_render(env.action_space.sample()) 58 | action = agents["current"].act(obs.flatten(), epsilon=0.05) 59 | obs, rew, done, _ , screen_obs = env.step_with_render(action) 60 | converted = converter(screen_obs) 61 | plt.ion() 62 | my_plot.autoscale() 63 | my_plot.set_data(converted) 64 | plt.title(title + "- Step: {}".format(steps + 1)) 65 | plt.pause(.05) 66 | plt.draw() 67 | plt.axis("off") 68 | steps += 1 69 | # if steps == 1: 70 | # plt.savefig("example_frame.png", dpi=300) 71 | plt.savefig("movies/file%02d.png" % steps) 72 | #print("action: ", act(obs)[0]) 73 | episode_rew += rew 74 | print("Episode reward", episode_rew) 75 | 76 | os.chdir(os.getcwd() + "/movies") 77 | subprocess.call([ 78 | 'ffmpeg', '-framerate', '8', '-i', 'file%02d.png', '-r', '30', '-pix_fmt', 'yuv420p', 79 | save_fname 80 | ]) 81 | for file_name in glob.glob("*.png"): 82 | os.remove(file_name) 83 | 84 | 85 | if __name__ == '__main__': 86 | parser = argparse.ArgumentParser() 87 | # General logging/saving and device arguments 88 | parser.add_argument('-agent_fname', '--AGENT', action="store", 89 | default="500000_MLP-DQN", type=str, 90 | help='Filename of DQN agent.') 91 | parser.add_argument('-title', '--TITLE', action="store", 92 | default="500000", type=str, 93 | help='Iteration Title on top of frame.') 94 | args = parser.parse_args() 95 | 96 | 97 | LOAD_CKPT = "agents/" + args.AGENT 98 | save_fname = args.AGENT + ".mp4" 99 | title = "DQN Agent after {} Iterations".format(args.TITLE) 100 | main(LOAD_CKPT, save_fname, title) 101 | -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/ddqn_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/ddqn_01.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/ddqn_02.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/ddqn_02.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/ddqn_03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/ddqn_03.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/ddqn_04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/ddqn_04.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/ddqn_gamma.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/ddqn_gamma.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/ddqn_target_update.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/ddqn_target_update.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/dqn-timeline.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/dqn-timeline.jpeg -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/dqn_batch_size.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/dqn_batch_size.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/dqn_batchsize.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/dqn_batchsize.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/dqn_buffer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/dqn_buffer.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/dqn_lrate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/dqn_lrate.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/dqn_target_update.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/dqn_target_update.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/dueling_dqn_all.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/dueling_dqn_all.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/dueling_dqn_arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/dueling_dqn_arch.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/per_dqn_er.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/per_dqn_er.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/per_dqn_is.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/per_dqn_is.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/figures/per_dqn_temp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/figures/per_dqn_temp.png -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/general_helpers.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.autograd as autograd 4 | 5 | import gym 6 | import gridworld 7 | 8 | import copy 9 | import time 10 | import math 11 | import random 12 | import numpy as np 13 | import pandas as pd 14 | from collections import deque 15 | 16 | import torch.multiprocessing as mp 17 | 18 | # Set device config variables 19 | USE_CUDA = torch.cuda.is_available() 20 | Variable = lambda *args, **kwargs: autograd.Variable(*args, **kwargs).cuda() if USE_CUDA else autograd.Variable(*args, **kwargs) 21 | 22 | 23 | def init_weights(m): 24 | # Xavier initialization weights in network 25 | if type(m) == nn.Linear: 26 | torch.nn.init.xavier_uniform(m.weight) 27 | m.bias.data.fill_(0.01) 28 | 29 | 30 | class ReplayBuffer(object): 31 | def __init__(self, capacity, record_macros=False): 32 | self.buffer = deque(maxlen=capacity) 33 | 34 | def push(self, ep_id, step, state, action, 35 | reward, next_state, done): 36 | self.buffer.append((ep_id, step, state, action, reward, next_state, done)) 37 | 38 | def sample(self, batch_size): 39 | ep_id, step, state, action, reward, next_state, done = zip(*random.sample(self.buffer, batch_size)) 40 | return np.stack(state), action, reward, np.stack(next_state), done 41 | 42 | def num_episodes(self): 43 | return len(np.unique(np.array(self.buffer)[:,0])) 44 | 45 | def __len__(self): 46 | return len(self.buffer) 47 | 48 | 49 | class NaivePrioritizedBuffer(object): 50 | def __init__(self, capacity, prob_alpha=0.6): 51 | self.prob_alpha = prob_alpha 52 | self.capacity = capacity 53 | self.buffer = [] 54 | self.pos = 0 55 | self.priorities = np.zeros((capacity,), dtype=np.float32) 56 | 57 | def push(self, ep_id, step, state, action, reward, next_state, done): 58 | max_prio = self.priorities.max() if self.buffer else 1.0 59 | 60 | if len(self.buffer) < self.capacity: 61 | self.buffer.append((ep_id, step, state, action, 62 | reward, next_state, done)) 63 | else: 64 | self.buffer[self.pos] = (ep_id, step, state, action, 65 | reward, next_state, done) 66 | 67 | self.priorities[self.pos] = max_prio 68 | self.pos = (self.pos + 1) % self.capacity 69 | 70 | def sample(self, batch_size, beta=0.4): 71 | if len(self.buffer) == self.capacity: prios = self.priorities 72 | else: prios = self.priorities[:self.pos] 73 | 74 | probs = prios ** self.prob_alpha 75 | probs /= probs.sum() 76 | 77 | indices = np.random.choice(len(self.buffer), batch_size, p=probs) 78 | samples = [self.buffer[idx] for idx in indices] 79 | 80 | N = len(self.buffer) 81 | weights = (N * probs[indices]) ** (-beta) 82 | weights /= weights.max() 83 | weights = np.array(weights, dtype=np.float32) 84 | batch = list(zip(*samples)) 85 | states, next_states = np.stack(batch[2]), np.stack(batch[5]) 86 | actions, rewards, dones = batch[3], batch[4], batch[6] 87 | return states, actions, rewards, next_states, dones, indices, weights 88 | 89 | def update_priorities(self, batch_indices, batch_priorities): 90 | for idx, prio in zip(batch_indices, batch_priorities): 91 | self.priorities[idx] = prio 92 | 93 | def __len__(self): 94 | return len(self.buffer) 95 | 96 | 97 | def epsilon_by_update(opt_counter, epsilon_start, epsilon_final, epsilon_decay): 98 | # Exponentially decaying exploration strategy 99 | eps = min(epsilon_final, epsilon_start + opt_counter * (epsilon_final - epsilon_start) / epsilon_decay) 100 | return eps 101 | 102 | 103 | def beta_by_update(opt_counter, beta_start=0.4, beta_steps=2000): 104 | # Linearly increasing temperature parameter 105 | beta = min(1.0, beta_start + opt_counter * (1.0 - beta_start) / beta_steps) 106 | return beta 107 | 108 | 109 | def update_target(current_model, target_model): 110 | # Transfer parameters from current model to target model 111 | target_model.load_state_dict(current_model.state_dict()) 112 | 113 | 114 | def polyak_update_target(current_model, target_model, soft_tau): 115 | for target_param, current_param in zip(target_model.parameters(), 116 | current_model.parameters()): 117 | target_param.data.copy_( 118 | target_param.data * (1. - soft_tau) + current_param.data * soft_tau 119 | ) 120 | 121 | 122 | def get_logging_stats(opt_counter, agent, GAMMA, 123 | NUM_ROLLOUTS, MAX_STEPS, AGENT): 124 | steps = [] 125 | rew = [] 126 | 127 | for i in range(NUM_ROLLOUTS): 128 | step_temp, reward_temp, buffer = rollout_episode(agent, GAMMA, 129 | MAX_STEPS, AGENT) 130 | steps.append(step_temp) 131 | rew.append(reward_temp) 132 | 133 | steps = np.array(steps) 134 | rew = np.array(rew) 135 | 136 | reward_stats = pd.DataFrame(columns=["opt_counter", "rew_mean", "rew_sd", 137 | "rew_median", 138 | "rew_10th_p", "rew_90th_p"]) 139 | 140 | steps_stats = pd.DataFrame(columns=["opt_counter", "steps_mean", "steps_sd", 141 | "steps_median", 142 | "steps_10th_p", "steps_90th_p"]) 143 | 144 | reward_stats.loc[0] = [opt_counter, rew.mean(), rew.std(), np.median(rew), 145 | np.percentile(rew, 10), np.percentile(rew, 90)] 146 | 147 | steps_stats.loc[0] = [opt_counter, steps.mean(), steps.std(), np.median(steps), 148 | np.percentile(steps, 10), np.percentile(steps, 90)] 149 | 150 | return reward_stats, steps_stats 151 | 152 | 153 | def rollout_episode(agent, GAMMA, MAX_STEPS, AGENT): 154 | env = gym.make("dense-v0") 155 | # Rollout the policy for a single episode - greedy! 156 | replay_buffer = ReplayBuffer(capacity=5000) 157 | 158 | obs = env.reset() 159 | episode_rew = 0 160 | steps = 0 161 | GAMMA = 0.95 162 | 163 | while steps < MAX_STEPS: 164 | action = agent["current"].act(obs.flatten(), epsilon=0.05) 165 | next_obs, reward, done, _ = env.step(action) 166 | steps += 1 167 | 168 | replay_buffer.push(0, steps, obs, action, 169 | reward, next_obs, done) 170 | 171 | obs = next_obs 172 | 173 | episode_rew += GAMMA**(steps - 1) * reward 174 | if done: 175 | break 176 | return steps, episode_rew, replay_buffer.buffer 177 | 178 | 179 | def run_multiple_times(args, run_fct): 180 | cpu_count = mp.cpu_count() 181 | gpu_count = torch.cuda.device_count() 182 | 183 | # Clone arguments into list & Distribute workload across GPUs 184 | args_across_workers = [copy.deepcopy(args) for r in range(args.RUN_TIMES)] 185 | if gpu_count > 0: 186 | gpu_counter = 0 187 | for r in range(args.RUN_TIMES): 188 | args_across_workers[r].device_id = gpu_counter 189 | gpu_counter += 1 190 | if gpu_counter > gpu_count-1: 191 | gpu_counter = 0 192 | 193 | # Execute different runs/random seeds in parallel 194 | pool = mp.Pool(cpu_count-1) 195 | df_across_runs = pool.map(run_fct, args_across_workers) 196 | pool.close() 197 | 198 | # Post process results 199 | df_concat = pd.concat(df_across_runs) 200 | by_row_index = df_concat.groupby(df_concat.index) 201 | df_means, df_stds = by_row_index.mean(), by_row_index.std() 202 | if args.SAVE: 203 | df_means.to_csv("logs/" + args.SAVE_FNAME + ".csv") 204 | return df_means, df_stds 205 | -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/gridworld/__init__.py: -------------------------------------------------------------------------------- 1 | from gym.envs.registration import register 2 | 3 | 4 | register( 5 | id='dense-v0', 6 | entry_point='gridworld.dense:MazeEnv', 7 | kwargs=dict(), 8 | nondeterministic = False, 9 | ) 10 | -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/gridworld/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/gridworld/__init__.pyc -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/gridworld/dense.py: -------------------------------------------------------------------------------- 1 | import gym 2 | import sys 3 | import numpy as np 4 | 5 | from pycolab import ascii_art 6 | from pycolab import things as plab_things 7 | from pycolab.prefab_parts import sprites as prefab_sprites 8 | 9 | 10 | MAZES_ART = [ 11 | ['####################', 12 | '#P $ @ $ @ $ @ $ @ #', 13 | '# @ $ $ $ $ $ $ $ $#', 14 | '###### a #######', 15 | '#$ $ @ $ $ $ $ @ $ #', 16 | '# $ $ $ $ @ $ $ $ $#', 17 | '########## b ###', 18 | '#$ $ @ $ $ $ $ $ @ #', 19 | '# $ $ $ @ $ $ @ $ $#', 20 | '####################']] 21 | 22 | 23 | # The "teaser observations" (see docstring) have their top-left corners at these 24 | # row, column maze locations. (The teaser window is 12 rows by 20 columns.) 25 | TEASER_CORNER = [(0, 0)] 26 | 27 | # For dramatic effect, none of the levels start the game with the first 28 | # observation centred on the player; instead, the view in the window is shifted 29 | # such that the player is this many rows, columns away from the centre. 30 | STARTER_OFFSET = [(0, 0)] 31 | 32 | 33 | # These colours are only for humans to see in the CursesUi. 34 | COLOUR_FG = {' ': (0, 0, 0), # Default black background 35 | '$': (999, 862, 110), # Shimmering golden coins 36 | '@': (66, 6, 13), # Poison 37 | '#': (764, 0, 999), # Walls of the maze 38 | 'P': (0, 999, 999), # Player 39 | 'a': (999, 0, 780), # Patroller A 40 | 'b': (145, 987, 341)} # Patroller B 41 | 42 | COLOUR_BG = {'$': (0, 0, 0), 43 | '@': (0, 0, 0)} 44 | 45 | 46 | def make_game(level): 47 | """Builds and returns a Better Scrolly Maze game for the selected level.""" 48 | return ascii_art.ascii_art_to_game( 49 | MAZES_ART[level], what_lies_beneath=' ', 50 | sprites={ 51 | 'P': PlayerSprite, 52 | 'a': PatrollerSprite, 53 | 'b': PatrollerSprite}, 54 | drapes={ 55 | '$': CashDrape, 56 | '@': PoisonDrape}, 57 | update_schedule=['a', 'b', 'P', '$','@'], 58 | z_order='ab$@P') 59 | 60 | 61 | class PlayerSprite(prefab_sprites.MazeWalker): 62 | """A `Sprite` for our player, the maze explorer.""" 63 | 64 | def __init__(self, corner, position, character): 65 | """Constructor: just tells `MazeWalker` we can't walk through walls.""" 66 | super(PlayerSprite, self).__init__( 67 | corner, position, character, impassable='#') 68 | self.num_steps = 0 69 | 70 | def update(self, actions, board, layers, backdrop, things, the_plot): 71 | del backdrop, things, layers # Unused 72 | self.num_steps += 1 73 | 74 | if actions == 0: # go upward 75 | self._north(board, the_plot) 76 | elif actions == 1: # go downward 77 | self._south(board, the_plot) 78 | elif actions == 2: # go leftward 79 | self._west(board, the_plot) 80 | elif actions == 3: # go rightward 81 | self._east(board, the_plot) 82 | elif actions == 4: # stay put 83 | self._stay(board, the_plot) 84 | if self.num_steps == 200: # terminate when reached max episode steps 85 | the_plot.terminate_episode() 86 | self.num_steps = 0 87 | 88 | 89 | class PatrollerSprite(prefab_sprites.MazeWalker): 90 | """Wanders back and forth horizontally, killing the player on contact.""" 91 | 92 | def __init__(self, corner, position, character): 93 | """Constructor: list impassables, initialise direction.""" 94 | super(PatrollerSprite, self).__init__( 95 | corner, position, character, impassable='#') 96 | # Choose our initial direction based on our character value. 97 | self._moving_east = bool(ord(character) % 2) 98 | 99 | def update(self, actions, board, layers, backdrop, things, the_plot): 100 | del actions, backdrop # Unused. 101 | 102 | # We only move once every two game iterations. 103 | if the_plot.frame % 2: 104 | self._stay(board, the_plot) # Also not strictly necessary. 105 | return 106 | 107 | # If there is a wall next to us, we ought to switch direction. 108 | row, col = self.position 109 | if layers['#'][row, col-1]: self._moving_east = True 110 | if layers['#'][row, col+1]: self._moving_east = False 111 | 112 | # Make our move. If we're now in the same cell as the player, it's instant 113 | # game over! 114 | (self._east if self._moving_east else self._west)(board, the_plot) 115 | if self.position == things['P'].position: the_plot.terminate_episode() 116 | 117 | 118 | class CashDrape(plab_things.Drape): 119 | """A `Drape` handling all of the coins. 120 | This Drape detects when a player traverses a coin, removing the coin and 121 | crediting the player for the collection. Terminates if all coins are gone. 122 | """ 123 | 124 | def update(self, actions, board, layers, backdrop, things, the_plot): 125 | # If the player has reached a coin, credit reward 100 and remove the coin 126 | # from the scrolling pattern. If the player has obtained all coins, quit! 127 | player_pattern_position = things['P'].position 128 | 129 | if self.curtain[player_pattern_position]: 130 | the_plot.log('Coin collected at {}!'.format(player_pattern_position)) 131 | the_plot.add_reward(100) 132 | self.curtain[player_pattern_position] = False 133 | if not self.curtain.any(): the_plot.terminate_episode() 134 | 135 | 136 | class PoisonDrape(plab_things.Drape): 137 | """A `Drape` handling all of the coins. 138 | This Drape detects when a player traverses a coin, removing the coin and 139 | crediting the player for the collection. Terminates if all coins are gone. 140 | """ 141 | 142 | def update(self, actions, board, layers, backdrop, things, the_plot): 143 | # If the player has reached a poison, deleted 100 reward and remove the coin 144 | # from the scrolling pattern. 145 | player_pattern_position = things['P'].position 146 | 147 | if self.curtain[player_pattern_position]: 148 | the_plot.log('Poison collected at {}!'.format(player_pattern_position)) 149 | the_plot.add_reward(-100) 150 | self.curtain[player_pattern_position] = False 151 | if not self.curtain.any(): the_plot.terminate_episode() 152 | 153 | 154 | class MazeEnv(gym.Env): 155 | """ 156 | Wrapper to adapt to OpenAI's gym interface. 157 | """ 158 | action_space = gym.spaces.Discrete(4) 159 | observation_space = gym.spaces.Box(low=0, high=1, shape=[10, 20, 6], dtype=np.uint8) 160 | 161 | def _to_obs(self, observation): 162 | hallway = observation.layers[' '] 163 | ob = np.stack([observation.layers[c] for c in 'Pab$@'] + [hallway], axis=2).astype(np.uint8) 164 | return ob 165 | 166 | def reset(self): 167 | self._game = make_game(0) 168 | observation, _, _ = self._game.its_showtime() 169 | return self._to_obs(observation) 170 | 171 | def reset_with_render(self): 172 | self._game = make_game(0) 173 | observation, _ , _ = self._game.its_showtime() 174 | return self._to_obs(observation), observation 175 | 176 | def step(self, action): 177 | observation, reward, _ = self._game.play(action) 178 | if reward is None: reward = 0 179 | done = self._game.game_over 180 | info = {} 181 | return self._to_obs(observation), reward, done, info 182 | 183 | def step_with_render(self, action): 184 | observation, reward, _ = self._game.play(action) 185 | if reward is None: reward = 0 186 | done = self._game.game_over 187 | info = {} 188 | return self._to_obs(observation), reward, done, info, observation 189 | -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/logs/old/dqn_capacity_100.csv: -------------------------------------------------------------------------------- 1 | ,index,opt_counter,rew_mean,rew_sd,rew_median,rew_10th_p,rew_90th_p,steps_mean,steps_sd,steps_median,steps_10th_p,steps_90th_p 2 | 0,0,19.0,57.385967853755204,30.21749431011665,47.14858821219911,31.022782296883165,91.2001052026117,194.775,8.450000000000001,199.0,186.325,199.0 3 | 1,0,39.0,74.82383213438727,57.91332847591154,46.802486008369044,32.95199529731548,139.54476070568302,199.0,0.0,199.0,199.0,199.0 4 | 2,0,59.0,50.059596702523045,28.62658878187775,42.32640147231556,25.788654812999184,80.5443962320225,194.375,9.25,199.0,185.125,199.0 5 | 3,0,79.0,57.46107568779058,32.59892927565897,40.76744151474528,37.52141682053562,92.44587511491656,194.575,8.85,199.0,185.725,199.0 6 | 4,0,99.0,62.95071670257349,25.578256176321815,53.11725495561023,42.17745586829,91.29082881164337,194.775,8.450000000000001,199.0,186.325,199.0 7 | 5,0,119.0,37.782029365839925,31.8411743646264,19.026551005688567,15.975885036292112,73.22208684119676,199.0,0.0,199.0,199.0,199.0 8 | 6,0,139.0,34.51995098870401,21.841097917842287,25.736095826135895,21.319288636257326,56.469542788664434,194.45,6.654227979262508,199.0,186.475,199.0 9 | 7,0,159.0,42.20422823759989,36.73282422028845,27.60616881267637,19.66217355615818,79.62386068727182,199.0,0.0,199.0,199.0,199.0 10 | 8,0,179.0,24.028831013795266,4.86246846121028,21.527341173155534,20.61632258442105,29.230813056972462,199.0,0.0,199.0,199.0,199.0 11 | 9,0,199.0,27.708377697089237,16.444395607448207,20.398800346241746,17.573657348550142,45.018841365736925,185.625,10.926744711944176,177.375,176.375,199.0 12 | 10,0,219.0,33.003849209964486,22.047488929673484,22.468906998004925,18.93469094936844,56.112835861486175,197.625,2.75,199.0,194.875,199.0 13 | 11,0,239.0,59.68944963519298,37.926562555985505,37.53039147834352,35.39791367964567,101.03233780512014,198.225,1.55,199.0,196.675,199.0 14 | 12,0,259.0,58.1539639872434,25.15072017845592,60.08777132800769,34.48162613842946,82.95086733161736,199.0,0.0,199.0,199.0,199.0 15 | 13,0,279.0,42.237205483086726,22.288949220358383,43.61522838811194,21.199862424091283,64.07256929737093,194.475,9.05,199.0,185.425,199.0 16 | 14,0,299.0,97.01103222178324,43.4311626635985,92.06667528926445,58.45633816945138,136.42664469490018,199.0,0.0,199.0,199.0,199.0 17 | 15,0,319.0,53.21533494798623,11.596434617925617,55.59460681031198,41.75456583615248,62.28994145917845,196.0,4.50624566573994,199.0,190.675,199.0 18 | 16,0,339.0,115.54515669776237,36.392098648739484,99.09243635377615,90.08737494089787,153.78174375578337,199.0,0.0,199.0,199.0,199.0 19 | 17,0,359.0,95.0322770668618,25.31775654102195,95.4834434658699,70.56705669470239,118.57454353317026,190.95,16.1,199.0,174.85,199.0 20 | 18,0,379.0,82.58983830238861,24.914045406903938,77.12820493369568,59.33367637859861,107.42578986026831,199.0,0.0,199.0,199.0,199.0 21 | 19,0,399.0,73.91559922405452,21.42520841684603,66.5442335536025,57.178948525126586,95.8302211041063,197.54999999999998,2.9000000000000004,199.0,194.65,199.0 22 | 20,0,419.0,83.37087744472726,39.63617026827194,72.67233945429183,55.46163902639245,121.92931647390932,176.525,26.661543246046243,187.625,147.175,198.35 23 | 21,0,439.0,104.22072773289872,31.564384880989603,98.22536457729163,78.80437885742073,137.13127544576923,182.29999999999998,17.688992111815566,176.875,166.6,199.0 24 | 22,0,459.0,119.13803956631213,23.655939789072796,117.24938360098112,99.07812559207582,142.11972659150408,192.075,12.319638025079826,199.0,179.0,199.0 25 | 23,0,479.0,119.72042146996102,26.086756318274595,118.31210662887244,96.88828650380917,141.79971566498574,176.875,17.4,176.625,163.95,190.05 26 | 24,0,499.0,106.9260596954047,19.53165165310363,107.78960641263977,89.44198154861895,123.08918175474376,162.1,11.256498123306377,153.75,152.5,175.875 27 | 25,0,519.0,90.50460589461494,25.84812344343142,88.31018262767151,65.50607356550758,114.13973465229012,171.75,7.949999999999999,175.625,163.85,175.775 28 | 26,0,539.0,86.40383156682857,17.233594418448057,90.75931249657037,68.47818920158399,100.28414033258778,175.95,25.474725769564305,175.625,150.825,199.0 29 | 27,0,559.0,107.52478065299722,19.216214046200797,104.39896580125493,91.73730943512194,125.4390129137843,168.47500000000002,8.329238505292766,167.25,160.275,176.625 30 | 28,0,579.0,80.79255302077331,23.981631075658857,77.4008464052743,58.67630224950892,106.50677291037246,199.0,0.0,199.0,199.0,199.0 31 | 29,0,599.0,76.04031776617113,31.111924942206226,66.67123298796048,54.86924353315257,106.48325540115513,196.525,4.95,199.0,191.575,199.0 32 | 30,0,619.0,67.81424775227332,20.368184402489096,67.76748628123295,48.46541312237747,88.51093964766727,191.925,12.623895235301857,199.0,178.45,199.0 33 | 31,0,639.0,78.09198555572635,20.301380050586932,84.5404908718534,57.16261787024961,93.88603030506792,194.675,8.65,199.0,186.025,199.0 34 | 32,0,659.0,97.50770289053357,37.42924112214025,93.17200676021656,64.92453085552046,133.6048958239981,189.57500000000002,18.85,199.0,170.725,199.0 35 | 33,0,679.0,96.28139792220063,19.158376414570196,90.03978369317173,79.92841806338899,116.8734876141733,183.8,19.224120235892045,194.375,161.675,199.0 36 | 34,0,699.0,96.14300212357458,22.643266997311855,86.87434728440648,78.89743108165837,120.55173998172134,189.425,7.3893064656686755,191.375,181.55,195.95 37 | 35,0,719.0,104.06305920142081,28.52657847729949,100.08365415022664,80.89267021271004,130.32584816158285,193.65,10.7,199.0,182.95,199.0 38 | 36,0,739.0,97.23414001782417,28.952288230115762,90.01984448736442,70.30215663718543,130.79610070232067,199.0,0.0,199.0,199.0,199.0 39 | 37,0,759.0,74.03083461688298,16.82481503564099,70.04212062405969,61.416927451376594,90.5776824810187,194.825,8.350000000000001,199.0,186.475,199.0 40 | 38,0,779.0,85.66297860272321,27.563778041894334,75.44304356152081,62.36046124804785,114.88305208523309,194.0,7.419989892176404,199.0,185.175,199.0 41 | 39,0,799.0,100.22982574702907,11.486739116181669,97.7789137068435,91.22059986168823,111.64448051887419,195.975,6.05,199.0,189.925,199.0 42 | 40,0,819.0,102.88571075332503,19.407778517608378,103.95328300297204,83.68328517492873,120.78091365725119,191.4,15.2,199.0,176.2,199.0 43 | 41,0,839.0,83.38467887643179,19.53413866300685,86.97364416690712,62.25555968913402,99.92496259926219,186.35,16.51301263435993,190.875,168.8,199.0 44 | 42,0,859.0,93.28725350062297,20.49754683168199,92.96450198072606,75.22093231161612,112.76822550114942,199.0,0.0,199.0,199.0,199.0 45 | 43,0,879.0,89.86567015741635,31.914768652253834,82.22121829165356,62.642730885469916,121.95536563340988,198.075,1.8499999999999999,199.0,196.225,199.0 46 | 44,0,899.0,82.93822545605413,14.063826389788698,76.49106952249669,74.746308563109,97.35714339758789,189.3,13.091564251018536,194.375,174.8,199.0 47 | 45,0,919.0,98.39969023617434,17.90209752669749,92.55363714761027,85.26503848412617,117.38906120326985,196.275,5.45,199.0,190.825,199.0 48 | 46,0,939.0,83.84112415913758,20.65387881024879,74.23128573559134,71.60990908165675,105.00333376491531,198.075,1.8499999999999999,199.0,196.225,199.0 49 | 47,0,959.0,98.6784617199267,34.849907439781056,93.58131191432862,68.6625362521444,134.42981121057068,192.375,13.25,199.0,179.125,199.0 50 | 48,0,979.0,80.46767149748347,12.03967289876199,80.793999887505,68.59875288913216,92.24582451255709,192.9,12.2,199.0,180.7,199.0 51 | 49,0,999.0,112.91560910274563,33.23775380525235,112.86121324420652,80.06109488032105,144.5956165354441,195.075,7.85,199.0,187.225,199.0 52 | 50,0,1019.0,119.14333276868572,28.41954623727023,118.10749607749233,94.94763888249199,144.81995375735457,191.725,14.55,199.0,177.175,199.0 53 | 51,0,1039.0,121.84065131889962,22.27460439195885,116.17419089342089,102.69324413184751,144.67025073190135,199.0,0.0,199.0,199.0,199.0 54 | 52,0,1059.0,113.88633059543886,17.436169729393963,115.3256987145586,98.95133771528693,129.59030750049487,199.0,0.0,199.0,199.0,199.0 55 | 53,0,1079.0,117.00649519196692,9.244445766521046,112.43787897919012,108.99661499283756,127.47405398437346,199.0,0.0,199.0,199.0,199.0 56 | 54,0,1099.0,117.12327010792389,17.226343303504443,117.33494518468736,101.27835287261341,133.34261463438293,199.0,0.0,199.0,199.0,199.0 57 | 55,0,1119.0,119.05689292394467,24.222732555801343,116.57831003425882,99.49457476741647,140.15194074247574,192.95,12.100000000000001,199.0,180.85000000000002,199.0 58 | 56,0,1139.0,119.06885885194694,15.089026963870923,116.5901709677205,107.613453450289,132.64318992306215,195.375,7.25,199.0,188.125,199.0 59 | 57,0,1159.0,120.84056180388565,21.23126751084355,113.48118674260702,103.78547441416245,142.25684271384796,199.0,0.0,199.0,199.0,199.0 60 | 58,0,1179.0,100.54488051140058,17.45724949923464,100.84717206517047,85.30142384186121,115.46806519968672,199.0,0.0,199.0,199.0,199.0 61 | 59,0,1199.0,119.43575286251274,26.75646664195434,126.98055287571222,93.24788845546786,139.8461907517477,195.075,7.85,199.0,187.225,199.0 62 | 60,0,1219.0,99.52814875394039,23.930053437290383,95.36245371691727,80.67777750269184,122.3354323141121,182.5,26.061466598954723,199.0,153.975,199.0 63 | 61,0,1239.0,101.2210853150633,14.782933223209252,101.65259256110653,88.45311796410725,114.09298880204167,197.225,3.5500000000000003,199.0,193.675,199.0 64 | 62,0,1259.0,106.63122635033162,11.332323515631307,107.00574115141796,96.00912340391743,118.1702799733723,194.675,8.65,199.0,186.025,199.0 65 | 63,0,1279.0,80.8182766491899,10.602537013008448,80.83846481233286,71.64615455567957,90.05861742071967,198.275,1.45,199.0,196.825,199.0 66 | 64,0,1299.0,91.32702544598828,25.59983539971659,81.42894508741769,69.66594821724297,117.35867861867845,199.0,0.0,199.0,199.0,199.0 67 | 65,0,1319.0,87.73958984733427,12.696660563033769,81.03897628561212,77.39659567534417,102.05366752521567,199.0,0.0,199.0,199.0,199.0 68 | 66,0,1339.0,82.21945507647506,15.191640106402732,78.07171177584875,69.64366248433988,97.9298101876622,195.075,7.8500000000000005,199.0,187.225,199.0 69 | 67,0,1359.0,95.43809909159376,35.70476861304532,81.10368877128175,70.68376906068701,132.8099388390175,195.075,7.8500000000000005,199.0,187.225,199.0 70 | 68,0,1379.0,84.72334753587654,30.97595399529311,85.7871124911231,57.344897383173034,115.01408092631362,190.15,10.838992111815562,199.0,176.875,199.0 71 | 69,0,1399.0,110.84395796852459,34.982843484646445,104.66563210281787,83.54228005707093,145.3366678779159,184.04999999999998,14.011466598954723,176.625,172.05,199.0 72 | 70,0,1419.0,89.11121925851525,21.843071252618106,86.12587954809236,71.25453518676643,110.27727285170056,184.85,12.411466598954721,176.625,174.45,199.0 73 | 71,0,1439.0,121.11520407143848,28.54386401415852,133.44014716719693,91.45339193263482,139.08264467947282,199.0,0.0,199.0,199.0,199.0 74 | 72,0,1459.0,112.03948070407687,29.22928734926469,123.42390352782297,81.94136533111502,132.08611048670062,198.975,0.05,199.0,198.925,199.0 75 | 73,0,1479.0,137.37472421130968,11.805288245818145,131.233102677612,130.01031000427386,150.1443359679134,188.375,14.388992111815563,199.0,171.55,199.0 76 | 74,0,1499.0,113.70636615105472,21.424241774630737,122.80086928294762,89.1387312968356,131.87894706075892,194.575,8.85,199.0,185.725,199.0 77 | 75,0,1519.0,122.93779967079652,16.239604037460563,115.69881617067345,109.37937046522889,141.05432253248375,199.0,0.0,199.0,199.0,199.0 78 | 76,0,1539.0,101.67734627863037,27.835702262354026,103.29533696513099,72.52821760957742,129.0742938688584,199.0,0.0,199.0,199.0,199.0 79 | 77,0,1559.0,111.7221567845782,14.996768079807866,116.8905069274269,96.92717896682541,121.64164214253613,199.0,0.0,199.0,199.0,199.0 80 | 78,0,1579.0,81.62390713555588,14.439332863442978,74.95787704127845,71.37627405977884,97.72990710653367,199.0,0.0,199.0,199.0,199.0 81 | 79,0,1599.0,73.47936771624568,18.953526730602526,72.04079326826266,55.72215646801234,92.66839005534237,199.0,0.0,199.0,199.0,199.0 82 | 80,0,1619.0,73.95428719877552,15.46414662926189,75.53445145177366,59.807932127494524,86.83794890521187,199.0,0.0,199.0,199.0,199.0 83 | 81,0,1639.0,72.11975223641305,11.983108304397401,73.45719758394617,60.93659826807564,82.17212222550639,196.575,4.8500000000000005,199.0,191.725,199.0 84 | 82,0,1659.0,80.63155020474032,34.95370407815834,74.14682966679882,49.60200626009369,113.6272075409438,199.0,0.0,199.0,199.0,199.0 85 | 83,0,1679.0,71.59263246970028,9.720663801095641,72.1785959516912,62.426736395797604,81.14411183842228,191.75,14.5,199.0,177.25,199.0 86 | 84,0,1699.0,89.46834655872779,35.065271175499426,72.76764483060812,62.83137569049374,128.02660731732792,199.0,0.0,199.0,199.0,199.0 87 | 85,0,1719.0,78.13075488458726,13.529650213189486,73.57595996195263,65.96552658215668,92.28469270351236,199.0,0.0,199.0,199.0,199.0 88 | 86,0,1739.0,82.63964748983378,14.722053263575997,79.9383424437788,69.10542771636577,98.64245493588851,199.0,0.0,199.0,199.0,199.0 89 | 87,0,1769.0,71.33146178782164,5.673725786010834,72.70338616899426,65.75800494043295,75.99571603998378,199.0,0.0,199.0,199.0,199.0 90 | 88,0,1789.0,71.40216691232483,15.895631463206335,71.04286403574653,57.2138442202573,85.33231330415141,198.925,0.15,199.0,198.775,199.0 91 | 89,0,1809.0,74.39770523536605,13.616137561419599,79.10810585708921,60.22723623415272,84.61537822215782,199.0,0.0,199.0,199.0,199.0 92 | 90,0,1829.0,68.23583395581028,15.027759632543193,69.02468738699986,54.79515695710814,80.65729482402273,199.0,0.0,199.0,199.0,199.0 93 | 91,0,1849.0,81.68444918591842,26.771318640631172,75.17455180846814,62.40450233965113,107.55834814489492,199.0,0.0,199.0,199.0,199.0 94 | 92,0,1869.0,76.83520188557598,14.524371478891915,77.42394026019208,61.91911774413671,90.37833239955796,199.0,0.0,199.0,199.0,199.0 95 | 93,0,1889.0,81.501319974939,18.159581060104465,79.66382898172486,68.14277159781197,97.76426153826687,199.0,0.0,199.0,199.0,199.0 96 | 94,0,1909.0,79.55030686046585,6.888731514859199,78.17775341850471,74.39368340525414,86.10268046768569,197.375,3.25,199.0,194.125,199.0 97 | 95,0,1929.0,78.10610545302859,13.980538873981509,82.64661529856673,62.39857072009621,90.55666690495053,198.975,0.05,199.0,198.925,199.0 98 | 96,0,1949.0,107.01549615622736,17.97218972145279,112.43819133954966,86.31910146069006,124.43481945911108,188.725,9.115028798637994,188.375,178.575,199.0 99 | 97,0,1969.0,91.08420463913659,11.039603838399978,94.75644235438762,79.96490358635117,98.62372041099285,193.225,9.491564251018536,199.0,182.6,199.0 100 | 98,0,1989.0,86.10929405929016,8.351632012920929,87.30576768923103,77.97831477310473,93.42719320975415,194.775,8.450000000000001,199.0,186.325,199.0 101 | 99,0,2009.0,81.39548172612903,21.911072163159066,82.60167076722357,61.54171027232808,99.71729801567525,199.0,0.0,199.0,199.0,199.0 102 | 100,0,2029.0,94.4177771998166,23.66423941609181,88.72528893284421,75.97233854625257,117.14614848144436,192.35,13.3,199.0,179.05,199.0 103 | 101,0,2049.0,73.9467164474218,9.81536816940652,79.84142760269859,62.410446074776004,81.4369979121735,199.0,0.0,199.0,199.0,199.0 104 | 102,0,2069.0,87.22001798164148,24.681701906299562,79.87842866947744,69.56610078503167,111.37852371660215,199.0,0.0,199.0,199.0,199.0 105 | 103,0,2089.0,89.66863725329618,15.018854262614228,86.78911399707911,77.0975347208576,105.34095164261304,194.575,8.85,199.0,185.725,199.0 106 | 104,0,2109.0,76.84527026701394,5.697000951892724,79.25090152692376,70.59974607698771,81.27994684180373,199.0,0.0,199.0,199.0,199.0 107 | 105,0,2129.0,92.5708724831743,7.0226904957125384,89.95931592314614,88.16119123884587,99.56505847724794,195.325,7.35,199.0,187.975,199.0 108 | 106,0,2149.0,90.51125337589326,16.32923928616917,81.68138339879845,78.44991103556491,108.7734624211894,199.0,0.0,199.0,199.0,199.0 109 | 107,0,2169.0,93.90089908087175,22.471135199620903,88.7462514690743,72.67893054645026,117.232124741405,196.625,4.75,199.0,191.875,199.0 110 | 108,0,2189.0,89.06925459028437,12.321536547310458,87.43819698388323,77.80793406723785,101.1703481741311,199.0,0.0,199.0,199.0,199.0 111 | 109,0,2211.5,89.10358184914675,16.99363423384582,91.82682127747943,71.16321776418125,105.8312435096632,195.95,4.263654535724019,199.0,190.725,199.0 112 | 110,0,2231.5,80.08708786453099,16.621954159299,71.06336654241994,68.81179952147602,98.97515090904788,199.0,0.0,199.0,199.0,199.0 113 | 111,0,2251.5,74.41974089974708,13.123075415469252,74.06790984938156,62.57535931316886,85.67493709635909,199.0,0.0,199.0,199.0,199.0 114 | 112,0,2271.5,72.07449863946391,9.550623377904026,74.90175464248784,61.859199510605116,79.78055233896158,198.575,0.85,199.0,197.725,199.0 115 | 113,0,2291.5,91.26117463529953,12.605995272916648,83.84794836270967,81.32383116247117,105.40534308298277,199.0,0.0,199.0,199.0,199.0 116 | 114,0,2311.5,76.93040942541938,10.390129888697064,77.06417070323863,67.78050181804272,86.11884824920116,199.0,0.0,199.0,199.0,199.0 117 | 115,0,2331.5,98.0900639263609,16.852353647685973,97.80926776338998,83.20792185473556,113.46680335781281,199.0,0.0,199.0,199.0,199.0 118 | 116,0,2351.5,81.55233440775821,7.398193435351409,79.52150983403146,76.55092077930409,88.60024183868298,195.9,6.2,199.0,189.7,199.0 119 | 117,0,2371.5,95.42478521182623,18.62798504182748,93.16769515665166,76.12514141203654,115.50783315533272,185.8,17.048374201742725,199.0,164.9,199.0 120 | 118,0,2391.5,85.05819462690317,15.289838834074013,82.49148284023681,72.2981055224527,100.46556069301654,191.7,9.490188678221841,199.0,180.15,199.0 121 | 119,0,2411.5,105.90888242515865,21.824232929771874,101.78377685492441,88.69909202778136,127.36889846561681,197.775,2.4499999999999997,199.0,195.325,199.0 122 | 120,0,2431.5,102.8437085835307,14.26449349365033,100.6700516843834,91.6729857509132,116.5135016901075,198.025,1.95,199.0,196.075,199.0 123 | 121,0,2451.5,93.78778982820607,11.759168728062145,98.78992671410124,80.52095588878215,103.16328773750843,179.025,12.44044334092938,176.5,169.6,191.05 124 | 122,0,2471.5,102.01954526584524,14.46048993695396,98.12523289281282,88.39833353648619,117.9222829482289,184.0,15.058673532516199,178.125,170.75,199.0 125 | 123,0,2491.5,132.873664415941,20.706797324957677,141.42620490158043,109.4344488489959,149.86552123017958,187.95,6.916285130039102,188.125,180.525,195.55 126 | 124,0,2511.5,142.69356992761558,14.688756197420537,151.25932277393736,125.48714589319607,153.3086642648223,188.05,7.925276020429825,190.125,178.925,196.65 127 | 125,0,2531.5,109.21061030427114,15.159931734348653,115.36457851640607,93.18448853550727,120.79897869506397,199.0,0.0,199.0,199.0,199.0 128 | 126,0,2551.5,142.6583104573355,20.109053657482317,147.4900278018816,122.43403086847806,156.9306240255225,196.825,4.35,199.0,192.475,199.0 129 | 127,0,2571.5,124.54379153934016,19.932041675803493,132.89856253555408,103.84907651757705,136.78326775775534,194.775,8.450000000000001,199.0,186.325,199.0 130 | 128,0,2591.5,158.77002965142097,14.772479192650435,162.09277250744788,142.79644284779238,172.17702954589146,190.55,16.900000000000002,199.0,173.65,199.0 131 | 129,0,2611.5,131.44502837159354,10.427613689440978,133.61751375817803,121.4829436815784,139.4720224326825,198.175,1.6500000000000001,199.0,196.525,199.0 132 | 130,0,2631.5,134.59486561569744,23.923816329886765,129.191295788859,118.76689014992898,156.30610321171167,194.775,8.450000000000001,199.0,186.325,199.0 133 | 131,0,2651.5,129.1161854896532,13.895461881299664,129.88681534171405,117.1253257095462,141.14666816280933,199.0,0.0,199.0,199.0,199.0 134 | 132,0,2671.5,113.2758840902635,20.278628503680416,112.61477316919871,95.06976175700314,130.80420111842201,199.0,0.0,199.0,199.0,199.0 135 | 133,0,2691.5,95.31332991921191,28.454944551864024,94.90474946898435,69.44066264528546,121.60404569292993,191.45,15.100000000000001,199.0,176.35000000000002,199.0 136 | 134,0,2711.5,116.0273140411946,26.668136573235937,123.61820269510855,86.36920328983219,141.3108326529034,199.0,0.0,199.0,199.0,199.0 137 | 135,0,2731.5,127.23293411886127,23.674234727870395,128.54461622124768,105.26569996994262,148.52934376210465,191.75,14.5,199.0,177.25,199.0 138 | 136,0,2754.0,88.15207838606942,16.313978038904576,91.90429499533694,70.71071552777522,103.10949715098762,199.0,0.0,199.0,199.0,199.0 139 | 137,0,2774.0,117.04436805526475,16.357398492702327,125.42032939598302,98.96044201963312,128.05611428568517,194.775,8.450000000000001,199.0,186.325,199.0 140 | 138,0,2794.0,122.172228960779,9.363210190767557,126.91117280585294,112.55187408503355,127.37192407570815,190.55,16.900000000000002,199.0,173.65,199.0 141 | 139,0,2814.0,118.56557972220881,12.081786144290135,126.91966045396241,104.02560215137888,127.04572949647422,199.0,0.0,199.0,199.0,199.0 142 | 140,0,2834.0,119.02434406347763,27.829193306505477,126.48125967612216,91.75017552122854,138.97555986700945,199.0,0.0,199.0,199.0,199.0 143 | 141,0,2854.0,129.03930004041024,30.70256311615379,134.43292674281201,101.0957701293953,152.15152977092862,199.0,0.0,199.0,199.0,199.0 144 | 142,0,2874.0,121.17200557853349,9.841985645641367,125.40336986404677,110.67481028907817,127.76649738766125,196.575,4.8500000000000005,199.0,191.725,199.0 145 | 143,0,2894.0,120.08152409738578,16.579620444958,125.28054916342668,103.38942452621264,131.7267705878981,199.0,0.0,199.0,199.0,199.0 146 | 144,0,2914.0,114.09733743800732,21.478357881167756,122.81647809435842,90.29763893944653,130.03427544475855,199.0,0.0,199.0,199.0,199.0 147 | 145,0,2934.0,125.43953189621142,15.345219240645823,125.94088012330504,112.85175582723522,137.95196787057463,199.0,0.0,199.0,199.0,199.0 148 | 146,0,2954.0,122.77893089762063,12.643177615103841,123.35220154619326,110.20539203390696,135.20481881186737,199.0,0.0,199.0,199.0,199.0 149 | 147,0,2974.0,117.28383240165911,22.743866083451636,127.29750930756919,90.49207639049531,135.92332509781306,194.775,8.450000000000001,199.0,186.325,199.0 150 | 148,0,2994.0,118.8729374717162,15.776638050649648,120.29066039180842,102.46315098230144,132.89356740718728,199.0,0.0,199.0,199.0,199.0 151 | 149,0,3014.0,131.08261007313493,7.781981471144301,134.31792964504365,123.28105713822211,135.71740350864624,199.0,0.0,199.0,199.0,199.0 152 | 150,0,3034.0,130.0391267054085,15.306572929343723,135.7000725481726,115.32350318869356,139.32695866236884,199.0,0.0,199.0,199.0,199.0 153 | 151,0,3054.0,138.67244671219555,7.425128876578437,137.42054328582643,131.46273697170713,146.40596680835623,195.375,7.25,199.0,188.125,199.0 154 | 152,0,3074.0,150.77641134937116,11.53303656730533,151.80616564781656,140.5510669454272,160.27624726884568,178.425,3.2707352111026946,179.375,175.04999999999998,181.175 155 | 153,0,3094.0,153.9902532769332,14.348931372662722,155.3537017492683,138.8170617193627,168.3947973394774,195.225,7.55,199.0,187.675,199.0 156 | 154,0,3114.0,162.8044363390918,15.154099971340903,166.36784846278542,148.6016608064658,174.33221802805204,195.25,4.969280632043234,199.0,189.025,199.0 157 | 155,0,3134.0,148.86635883278072,15.347017678260096,145.41709023379437,136.9931677965745,163.82557179186725,191.15,9.66093939531762,199.0,179.075,199.0 158 | 156,0,3154.0,153.23473142043974,9.019552533830389,151.5991404472858,144.48100740245417,162.61140631766463,199.0,0.0,199.0,199.0,199.0 159 | 157,0,3174.0,149.96863106496113,26.388373182320713,155.0079318140743,125.45547084272019,171.08617635681878,183.15,18.00477629251465,180.625,166.14999999999998,199.0 160 | 158,0,3194.0,151.54295060356637,17.56470828439822,158.8633356651158,131.90548338946857,167.2290391854288,194.775,8.450000000000001,199.0,186.325,199.0 161 | 159,0,3214.0,145.38816823346056,16.70363695372999,145.0628341820825,129.6393501079813,161.68983163617884,197.325,3.35,199.0,193.975,199.0 162 | 160,0,3234.0,157.66676975549302,22.964541305342596,162.77305493161433,134.04744134618383,176.73057713587963,199.0,0.0,199.0,199.0,199.0 163 | 161,0,3254.0,168.20514353529776,12.857483229759463,163.76794551320995,158.56098095980218,181.44413533420183,193.25,11.5,199.0,181.75,199.0 164 | 162,0,3274.0,144.31064521175324,14.7562951237357,149.83798052197633,127.98975909492779,156.85140982393602,194.775,8.450000000000001,199.0,186.325,199.0 165 | 163,0,3294.0,143.7356191761627,26.535821188924135,135.85814160800328,125.0095830556474,169.72275050793175,199.0,0.0,199.0,199.0,199.0 166 | 164,0,3314.0,128.4174362239366,24.728082959877803,135.07318327544715,102.91182448844486,151.2679191274801,191.65,14.700000000000001,199.0,176.95,199.0 167 | 165,0,3334.0,145.0935354160012,20.90583910315022,143.998092540345,124.99272750876328,166.0077913784195,194.775,8.450000000000001,199.0,186.325,199.0 168 | 166,0,3354.0,114.86647580620158,36.08861605895721,102.9393025459467,87.52229793429564,152.76928141196856,188.125,14.873426660164842,199.0,170.60000000000002,199.0 169 | 167,0,3374.0,92.62841412737127,17.027400420442596,91.50548874328385,79.65035893382372,107.33449556140346,181.35,8.838269061303802,176.625,176.625,190.55 170 | 168,0,3394.0,85.18370896533243,12.972874629503073,81.27087924460457,74.27457043318297,98.4643604552167,182.1,8.45,177.875,177.875,190.55 171 | 169,0,3414.0,98.90397025357183,17.53825338923923,100.64403685232043,80.25210420467049,116.73275865872176,191.075,15.849999999999998,199.0,175.22500000000002,199.0 172 | 170,0,3434.0,108.12285176894895,27.81549223161127,103.76707455283702,85.83757382939325,134.41419548953053,170.75,12.304530711280801,177.875,157.425,177.875 173 | 171,0,3454.0,121.25662577285252,26.228085899503625,124.78265470887129,95.4190988596011,144.8855422265396,177.825,29.248188326517855,199.0,143.925,199.0 174 | 172,0,3474.0,137.57470726234806,27.146072808444362,130.0158415776911,117.15691248987983,164.06700601092788,186.325,10.349094163258927,177.875,177.875,199.0 175 | 173,0,3494.0,128.07740397917604,14.064824227406486,134.13311801581347,113.69764952527395,136.33392387315456,181.275,20.16308430336519,177.875,162.72500000000002,199.0 176 | 174,0,3514.0,95.00219453615213,17.454588582675974,97.13424108443122,78.42493340976785,108.65690334737357,176.675,17.799999999999997,176.625,163.35,190.05 177 | 175,0,3534.0,88.1263425906742,18.135629973798395,86.27496258408678,71.75921627611864,104.918319793709,192.02499999999998,13.95,199.0,178.075,199.0 178 | 176,0,3554.0,95.44511175628168,23.488005236149483,87.65806864783855,78.39975391897335,120.51349255499295,194.45,9.1,199.0,185.35,199.0 179 | 177,0,3574.0,93.36372725433637,5.06536871038666,94.73801316166576,88.45224680640726,97.20708234539516,196.275,5.449999999999999,199.0,190.825,199.0 180 | 178,0,3594.0,113.43627597979052,17.736229390933374,123.07278115577745,92.74352173571873,128.05051565483382,194.775,8.450000000000001,199.0,186.325,199.0 181 | 179,0,3614.0,108.97567427453387,13.208947679005231,107.90686361964868,98.32908290448714,120.88888272525995,193.05,11.899999999999999,199.0,181.15,199.0 182 | 180,0,3634.0,115.76025447330434,21.074660737239356,109.18279176920073,97.02078891368438,138.79148713325966,199.0,0.0,199.0,199.0,199.0 183 | 181,0,3654.0,106.11996570567159,21.868229360496652,106.11633130488106,85.58999786055647,126.36477336027812,196.175,5.65,199.0,190.525,199.0 184 | 182,0,3674.0,120.87596833733585,22.305812236743186,122.59457673443029,100.22413098585008,140.0292194721468,182.3,26.59195302065386,199.0,153.125,199.0 185 | 183,0,3694.0,133.1233732413612,27.260497345462213,126.62130110947939,111.88993687577462,159.4109053097336,192.10000000000002,13.8,199.0,178.3,199.0 186 | 184,0,3714.0,109.60034064812164,21.061070636270827,110.15984637135806,92.40325889084524,126.36692289918196,192.75,12.5,199.0,180.25,199.0 187 | 185,0,3734.0,122.36965888001609,18.100003287055493,120.28886622063031,108.42391623345627,138.84919220312017,199.0,0.0,199.0,199.0,199.0 188 | 186,0,3754.0,98.95183562440367,7.298018979226821,95.57722381278836,92.85616887964841,106.87976742336947,197.575,2.8499999999999996,199.0,194.725,199.0 189 | 187,0,3774.0,103.25692983149308,14.965664100603222,102.5487971723838,89.17403661583289,117.20659974878939,197.0,4.0,199.0,193.0,199.0 190 | 188,0,3794.0,140.33093587443304,19.031873690911105,142.37881622343497,123.05062994012488,155.62707584184358,199.0,0.0,199.0,199.0,199.0 191 | 189,0,3814.0,168.74244759603306,6.879976350938028,170.38749502629454,160.96157300135587,175.23452843994974,199.0,0.0,199.0,199.0,199.0 192 | 190,0,3836.5,138.92150492659636,30.731826242942255,136.36419913262782,112.8280751511405,166.92735105164292,194.9,8.2,199.0,186.70000000000002,199.0 193 | 191,0,3856.5,135.0571459926496,23.812862475873974,126.25832561527876,116.02905408599545,160.51658441986007,193.625,10.750000000000002,199.0,182.875,199.0 194 | 192,0,3876.5,110.52035989922771,27.764157791437086,117.70318123452618,83.7647206963769,131.05459755032285,199.0,0.0,199.0,199.0,199.0 195 | 193,0,3896.5,117.58249721789633,12.397954881065525,118.58275405748698,106.8025433486531,127.39070459322154,190.325,17.35,199.0,172.975,199.0 196 | 194,0,3919.0,112.55558813603363,36.2388090230371,100.0506204136115,87.22769961562803,149.58989490827923,193.85,6.919808523362478,199.0,185.225,199.0 197 | 195,0,3939.0,105.03499082721736,25.10473252542161,97.13939847462689,82.89935608216966,132.267317540438,194.3,9.399999999999999,199.0,184.9,199.0 198 | 196,0,3959.0,109.88749751240233,30.22427705196172,98.24985732312537,87.8664192360318,141.58026497780247,199.0,0.0,199.0,199.0,199.0 199 | 197,0,3979.0,114.55005707961698,13.293827072232833,109.8026474138247,104.75431682310432,128.26227295303488,199.0,0.0,199.0,199.0,199.0 200 | 198,0,3999.0,114.23377918077452,8.741847140439111,111.96886698788691,105.49378276111723,123.81067373333465,192.3,8.209217380481528,199.0,182.175,199.0 201 | 199,0,4019.0,116.48803080494092,32.1648758172551,109.38403519463577,89.0077523950742,150.42680891680234,195.35,7.237661915287284,199.0,188.075,199.0 202 | 200,0,4039.0,117.96270288468386,31.557391240511187,111.33130869103793,92.6171379341422,149.7226014356743,194.475,9.05,199.0,185.425,199.0 203 | 201,0,4059.0,110.50405233001581,16.092039123893716,108.37724093408646,97.66725335676496,125.0708667934819,195.375,7.25,199.0,188.125,199.0 204 | 202,0,4079.0,116.10490210968507,17.764321828221945,108.8075000149255,102.16220033913858,135.55588659281872,188.125,16.812640133355018,199.0,169.10000000000002,199.0 205 | 203,0,4099.0,96.16904065563052,22.194298709045036,104.09578548112715,72.67205223940903,111.13771486782836,194.525,8.95,199.0,185.57500000000002,199.0 206 | 204,0,4119.0,171.71382688169928,23.934653359304452,170.57949266547578,147.8047757071489,196.5936909392409,191.85,8.898525158699053,199.0,180.625,199.0 207 | 205,0,4139.0,179.79980340926764,41.40441931630262,185.79399584195522,142.5704170001552,215.66611714864348,190.10000000000002,17.800000000000004,199.0,172.3,199.0 208 | 206,0,4159.0,179.61706393785727,25.545731038881886,184.1589584493336,154.23047460786125,201.55832228915898,199.0,0.0,199.0,199.0,199.0 209 | 207,0,4179.0,155.6214537727059,16.76750589040237,157.3248095632972,137.76633946812328,171.492481647196,186.02499999999998,19.39909416325893,199.0,164.3,199.0 210 | 208,0,4199.0,183.70325254184854,22.132716685905244,185.94798413391516,161.1896941038604,202.98141850718855,198.675,0.65,199.0,198.025,199.0 211 | 209,0,4219.0,156.94036823502586,27.7244928883094,154.2904612716907,132.32292824675594,182.45440149432284,199.0,0.0,199.0,199.0,199.0 212 | 210,0,4239.0,176.03975271507056,22.34940677324777,182.07472409750847,151.7844296004739,196.15642155250893,199.0,0.0,199.0,199.0,199.0 213 | 211,0,4259.0,169.99689477951296,26.988018288921666,175.4067110275494,141.38382472195008,193.24089488007755,186.325,18.79909416325893,199.0,165.2,199.0 214 | 212,0,4279.0,140.07664163075074,25.925780036550023,142.65771298756502,114.78297817597326,162.05881869730783,190.55,16.900000000000002,199.0,173.65,199.0 215 | 213,0,4301.5,144.49913592328994,26.11315938582453,145.07336311763578,121.3359726890722,166.76492204309972,190.925,16.15,199.0,174.775,199.0 216 | 214,0,4321.5,142.92556356896276,20.48025147343525,142.10718745967858,125.18821097516576,160.4596249929207,198.175,1.6500000000000001,199.0,196.525,199.0 217 | 215,0,4341.5,148.66152267399264,15.415265775862016,145.50027618768084,137.90645307782015,162.83714081712088,197.675,2.65,199.0,195.025,199.0 218 | 216,0,4361.5,97.64441386978363,31.798986700383267,89.68229611328732,72.12018689980047,130.7180118343697,190.95,9.879461017687149,199.0,178.675,199.0 219 | 217,0,4381.5,103.04831435654336,21.2863990576344,99.2447092411249,85.35005929029792,125.82890525328452,188.175,16.977499637786146,199.0,168.95,199.0 220 | 218,0,4401.5,107.47048217286451,20.994916597239047,100.4603327134628,91.69609238362267,128.58049034294197,199.0,0.0,199.0,199.0,199.0 221 | 219,0,4421.5,93.65906151538947,11.037725792090631,98.35475004346493,81.52493536380388,102.60816008864396,199.0,0.0,199.0,199.0,199.0 222 | 220,0,4441.5,148.31888702128234,19.87411470013829,147.54382461894258,132.18974796075972,166.31776479612785,199.0,0.0,199.0,199.0,199.0 223 | 221,0,4461.5,137.71632327233698,27.286770419862783,134.4277726863694,115.35864754509298,162.98451866302202,186.775,24.450000000000003,199.0,162.325,199.0 224 | 222,0,4481.5,145.36530027522392,36.98193913447842,145.94600066734188,112.78688779428332,179.1440528090158,193.725,10.55,199.0,183.175,199.0 225 | 223,0,4501.5,138.88757690241368,15.168330919158015,136.4218137938354,127.28237728967413,153.43263085992032,199.0,0.0,199.0,199.0,199.0 226 | 224,0,4521.5,127.65603985171562,15.427075049049424,130.6113285009564,111.57576127091605,140.77091573350484,178.875,24.361466598954724,176.625,156.525,199.0 227 | 225,0,4541.5,215.66092975054573,13.275775486063282,220.19215403821815,201.36704415668305,226.2818251745887,199.0,0.0,199.0,199.0,199.0 228 | 226,0,4561.5,206.56901770654207,15.711957472083107,214.10313005468234,189.89422905346723,216.57036549464308,194.775,8.450000000000001,199.0,186.325,199.0 229 | 227,0,4581.5,189.58527738437542,17.776806579728074,192.56232603042687,172.08265244479674,204.78684899735111,177.025,21.40163911905983,176.875,155.0,199.0 230 | 228,0,4601.5,137.77264512605998,26.810614247931582,144.30270504029184,108.12638630832996,159.06445252763416,171.125,13.5,177.875,157.625,177.875 231 | 229,0,4621.5,187.2891557366109,28.845400676818077,198.91017559616964,155.6823203913735,208.84102035683503,184.50000000000003,22.59025269559102,199.0,159.425,199.0 232 | 230,0,4641.5,195.8530586619802,18.544978786231287,203.4269648116096,174.73067583779596,210.23575680022515,194.775,8.450000000000001,199.0,186.325,199.0 233 | 231,0,4661.5,200.03052946139832,27.04530397310604,205.79669107688886,173.63617121795,221.94817016230635,172.9,8.450000000000001,177.125,164.45,177.125 234 | 232,0,4681.5,195.97251477643243,17.588220878552054,204.06811065418304,176.11644362692564,209.15222274569047,199.0,0.0,199.0,199.0,199.0 235 | 233,0,4704.0,182.7521596094204,3.9560970415401053,183.70747586460158,178.8540982593191,185.63877232036094,196.825,4.35,199.0,192.475,199.0 236 | 234,0,4724.0,158.89685850808647,8.340287344246395,160.01385219629523,151.51213181996118,165.24253865539,199.0,0.0,199.0,199.0,199.0 237 | 235,0,4744.0,154.17106773408207,38.08559537705044,155.56985424510097,122.41848262889826,186.7981517508956,199.0,0.0,199.0,199.0,199.0 238 | 236,0,4764.0,127.10356832225624,41.252370761584395,121.03557275556498,95.54837349337807,165.70621185156457,182.52499999999998,17.839576935258414,183.75,163.725,199.0 239 | 237,0,4784.0,140.2170498465182,28.543639153506803,137.59137471761267,115.89252677707503,168.18934314023286,185.55,23.064920924067074,199.0,160.925,199.0 240 | 238,0,4804.0,121.48517906696478,47.03386740409899,108.1464731856773,81.49830181038055,171.96082132263825,195.45,7.1,199.0,188.35000000000002,199.0 241 | 239,0,4824.0,154.9729031389613,40.0206898600866,160.11603991853514,117.19847431105376,188.71700958614773,188.525,15.223535305431596,199.0,171.0,199.0 242 | 240,0,4844.0,147.44812993286564,25.948080262435006,159.09537019966152,118.93642716500564,166.40044451839577,199.0,0.0,199.0,199.0,199.0 243 | 241,0,4864.0,150.29087111172464,34.46342081666972,144.40937477227715,122.28292378324586,183.60854141496498,185.15,21.96181677642135,199.0,160.825,199.0 244 | 242,0,4884.0,106.29928995078083,15.966645988758028,110.62790781123961,89.2059321110246,120.65539272095775,194.775,8.450000000000001,199.0,186.325,199.0 245 | 243,0,4904.0,104.34930083593447,25.611035089202076,108.14672734354671,80.49949540151073,127.51543406068734,199.0,0.0,199.0,199.0,199.0 246 | 244,0,4924.0,114.95317142960434,27.63205911960344,114.51752097917088,90.33833474824395,139.68149152913398,197.175,3.65,199.0,193.525,199.0 247 | 245,0,4944.0,92.64242698259007,15.882739895899242,89.8497781970397,80.56983876949688,107.0764172596325,197.475,3.05,199.0,194.425,199.0 248 | 246,0,4964.0,87.62094853895451,27.72495406826128,82.22649931825977,62.325344226908186,116.21830824518203,194.35,9.299999999999999,199.0,185.05,199.0 249 | 247,0,4984.0,95.09380169095162,13.52899072384126,95.16789063749758,83.80580989673592,106.38166224786994,199.0,0.0,199.0,199.0,199.0 250 | 248,0,5004.0,98.14803708128197,19.91036443937362,96.96279855152925,81.67326321374402,116.18385158273959,189.85,6.907604505181228,191.375,181.675,197.15 251 | 249,0,5024.0,95.5087661502864,7.657746915339345,94.57544864881622,89.09557461008325,102.97432427205351,196.275,5.449999999999999,199.0,190.825,199.0 252 | 250,0,5044.0,108.29471258688443,15.571948230133735,105.00555808682535,94.08586364302637,124.10386317264158,189.7,11.390401661047777,199.0,175.725,199.0 253 | 251,0,5064.0,102.88018603986043,21.071016782823033,100.83460494995008,85.61701674217596,122.69634560541681,199.0,0.0,199.0,199.0,199.0 254 | 252,0,5084.0,98.06475392929507,22.161094048917057,100.47950467082822,78.26082767404299,116.0466327311345,196.575,4.8500000000000005,199.0,191.725,199.0 255 | 253,0,5104.0,97.05151256169412,9.019429580156022,96.07628983124283,90.36941083095627,105.24074529428313,196.05,4.698803039072824,199.0,190.675,199.0 256 | 254,0,5124.0,93.93565606702514,17.217878541115013,96.13270410218004,77.77602023228204,107.5868436003944,194.55,5.4524077250330425,199.0,187.825,199.0 257 | 255,0,5146.5,92.50008312715049,16.25584545829024,95.83751416097952,77.86305153944033,104.4665212355285,190.6,10.324545994860985,199.0,177.725,199.0 258 | 256,0,5166.5,94.7777338864791,24.2072248407009,96.02647061244289,73.46688773263077,114.64468976216261,196.8,4.3999999999999995,199.0,192.4,199.0 259 | 257,0,5186.5,100.7759414199068,18.137604221585192,102.27317191515452,83.98239379239439,117.34236958830306,191.925,14.149999999999999,199.0,177.775,199.0 260 | 258,0,5206.5,106.11986898581081,16.755955269301214,99.53656437585693,92.86336865607166,125.0942598888229,194.475,9.05,199.0,185.425,199.0 261 | 259,0,5226.5,93.2829452752404,20.23945135274425,91.96947688641968,75.83016327864229,113.60500412580676,195.1,2.748181216732259,194.375,192.175,198.35 262 | 260,0,5246.5,106.20422765462774,24.178158617998513,96.79765899245204,90.13400993233238,131.68927224573773,189.775,18.450000000000003,199.0,171.325,199.0 263 | 261,0,5266.5,105.8340637813348,35.45444316774742,99.82151667947366,78.49747426490552,139.16002618288476,199.0,0.0,199.0,199.0,199.0 264 | 262,0,5286.5,94.55069877378108,22.33488298816132,93.70209461728052,73.80274092404996,115.37423357296092,189.425,15.214134743056212,199.0,172.2,199.0 265 | 263,0,5306.5,106.14771506377136,19.469509577410786,97.18339730032743,93.2279346965587,126.83819085347189,196.175,5.65,199.0,190.525,199.0 266 | 264,0,5326.5,119.8858276714023,17.49536719432591,127.43265585532899,100.98412855991225,131.55410727474114,194.775,8.450000000000001,199.0,186.325,199.0 267 | 265,0,5346.5,138.60295886034453,13.123201924952898,134.40881682479525,128.72336346120113,152.03402603079394,199.0,0.0,199.0,199.0,199.0 268 | 266,0,5366.5,121.74967856723396,18.113893276052732,125.34349854013442,103.58105686254707,137.1820835304258,186.925,18.24968111726091,199.0,166.10000000000002,199.0 269 | 267,0,5386.5,135.83187819478343,22.089634194420633,136.3871393118689,115.0351754221603,155.5667274948839,195.125,7.75,199.0,187.375,199.0 270 | 268,0,5406.5,117.77084431049163,11.616200043382563,115.63299343515646,105.9289078068656,130.5572178601841,199.0,0.0,199.0,199.0,199.0 271 | 269,0,5426.5,106.02959979641015,22.7348245808943,106.28619685202331,85.44592611954363,125.85062068032265,194.725,8.549999999999999,199.0,186.175,199.0 272 | 270,0,5446.5,121.8864937215157,13.926056253356702,114.14345697200217,110.83561042034965,138.03113014536476,199.0,0.0,199.0,199.0,199.0 273 | 271,0,5466.5,119.25324106340243,22.64330803433657,109.22862620018259,105.31415498343146,142.68936954732948,194.75,8.5,199.0,186.25,199.0 274 | 272,0,5486.5,116.98117126428939,14.40647705589966,126.8727374640058,99.23257177867151,128.48854939012986,191.075,13.045414330181368,199.0,176.89999999999998,199.0 275 | 273,0,5506.5,135.08473771334855,16.488448950421976,137.05845564157514,118.77362678463976,148.4221432879399,197.425,3.1500000000000004,199.0,194.275,199.0 276 | 274,0,5526.5,142.20347535526264,10.931439388303264,136.24597872152393,134.16316122954606,154.80955878546004,191.75,14.5,199.0,177.25,199.0 277 | 275,0,5546.5,127.1185301287769,12.833951137940359,124.86155877872595,117.21146588299604,139.74267627685475,188.975,13.499094163258928,199.0,173.15,199.0 278 | 276,0,5566.5,78.14778986668782,22.40985413201666,72.7671138590288,60.19241938731568,101.4773415968734,199.0,0.0,199.0,199.0,199.0 279 | 277,0,5586.5,92.77127499332941,26.60606238544369,84.87660359434125,68.38681643276992,122.67243054328804,199.0,0.0,199.0,199.0,199.0 280 | 278,0,5606.5,91.75829213863037,19.754859732850583,86.02597288716724,78.98141937334233,110.78257071544039,195.175,7.650000000000001,199.0,187.525,199.0 281 | 279,0,5626.5,122.5778842834279,15.426955638592476,117.93973290241992,110.1474755559777,139.23869620322054,199.0,0.0,199.0,199.0,199.0 282 | 280,0,5646.5,157.09759473462458,7.305603185623539,156.17132413976424,151.4978100663193,163.61552837768144,199.0,0.0,199.0,199.0,199.0 283 | 281,0,5666.5,162.68380666058752,18.635144224170894,155.65621213970252,148.23587093592602,181.42506934007437,194.575,4.2519113349175095,194.375,189.725,199.0 284 | 282,0,5686.5,157.91834482144884,21.066627379666823,154.82871235755366,137.28929243856908,179.80101312626417,189.7,9.782511353640263,192.375,178.1,199.0 285 | 283,0,5706.5,152.83257579859009,15.962282767167018,158.1523805844584,135.49686116399832,165.29170062249239,191.575,7.030558299310234,192.125,183.275,199.0 286 | 284,0,5726.5,175.14069569744046,31.056964328153313,171.51042384493098,147.40513317868027,204.34937319973827,187.4,7.638880808076534,189.875,178.775,195.45 287 | 285,0,5749.0,130.86011393352197,23.270926251957153,120.29734900686844,111.79803096474309,156.8930183070752,196.275,5.45,199.0,190.825,199.0 288 | 286,0,5769.0,133.5973735569794,22.60857989831547,126.5887924962305,116.22979934796786,156.32623411182868,191.9,14.2,199.0,177.7,199.0 289 | 287,0,5789.0,140.7671184990648,16.607180382359616,141.96714340960537,125.6483685934067,154.91495719131538,195.075,7.85,199.0,187.225,199.0 290 | 288,0,5809.0,165.5859141885979,38.028918720509445,174.23920618764203,127.09307161188138,193.5198134203145,199.0,0.0,199.0,199.0,199.0 291 | 289,0,5829.0,176.67181195712422,26.616981824983906,176.46892787363453,153.76560672362237,200.85710845940116,190.55,10.349094163258927,199.0,177.875,199.0 292 | 290,0,5849.0,200.4402504065254,36.95597230225507,216.10170407082157,161.8982869433588,226.3973232685031,199.0,0.0,199.0,199.0,199.0 293 | 291,0,5869.0,202.21296888496136,22.87158767565694,212.95021379016748,177.57171662114263,217.23005729586478,195.025,7.95,199.0,187.075,199.0 294 | 292,0,5889.0,192.0448805424004,37.419360891592405,202.4531287267045,153.06485138148037,220.34244089910618,199.0,0.0,199.0,199.0,199.0 295 | 293,0,5909.0,211.68913903794024,32.800965987666686,215.25015499827575,177.19127840920245,242.36840021272428,194.775,8.450000000000001,199.0,186.325,199.0 296 | 294,0,5929.0,207.61646769954973,37.527313518619565,224.7121460052113,168.48610308639175,231.65409397445117,186.6,24.8,199.0,161.8,199.0 297 | 295,0,5949.0,157.49793828240396,31.347239660037815,171.27210589861096,123.17578532664362,182.00429971636984,196.825,4.35,199.0,192.475,199.0 298 | 296,0,5969.0,150.79248820858734,25.559525434813082,152.2544634059144,122.50436823307915,175.43117198439322,192.65,12.700000000000001,199.0,179.95,199.0 299 | 297,0,5989.0,169.68969863252988,36.307398939806404,157.6085794251983,140.82790756476732,208.01171078729797,192.05,8.836218082415124,199.0,180.875,199.0 300 | 298,0,6009.0,150.48709775244706,29.14136383111253,166.27174387932382,116.33892768748532,173.83597197849156,196.825,4.35,199.0,192.475,199.0 301 | 299,0,6029.0,171.5288542351264,25.296696593032987,168.4525139228607,148.7825437932571,196.9249462294561,184.5,22.692935004291783,199.0,159.425,199.0 302 | 300,0,6049.0,199.82013373353084,37.37554959912712,199.6781951639499,165.76262177648528,237.66123656839443,195.125,7.75,199.0,187.375,199.0 303 | 301,0,6069.0,211.0845969003459,21.170753545111168,224.84138995986075,186.38698169869343,225.4145912811315,190.275,17.450000000000003,199.0,172.825,199.0 304 | 302,0,6089.0,222.6354609138112,26.118607048206652,222.26851540556416,201.13662305121642,245.76551360003603,192.7,12.600000000000001,199.0,180.1,199.0 305 | 303,0,6109.0,225.38904896115565,24.44208172934262,220.33135403395897,206.4643124937798,249.13388921329914,190.85,16.3,199.0,174.55,199.0 306 | 304,0,6129.0,205.38753806670945,25.507964677354195,215.12063197799483,175.80442162323823,227.66313287137683,184.95,17.119126253550586,188.625,166.95,199.0 307 | 305,0,6149.0,198.7248005884084,18.40525577073729,207.22775838435643,178.00799495767023,211.92498767214704,194.725,8.549999999999999,199.0,186.175,199.0 308 | 306,0,6169.0,190.2565112967992,27.966734492142436,190.78936876505318,162.19048063581408,217.16294720891034,184.07500000000002,24.54379101038097,194.625,158.325,199.0 309 | 307,0,6189.0,188.57216053556488,28.59180517225594,194.43332276775638,159.6711269651073,210.52560126384708,197.025,3.95,199.0,193.075,199.0 310 | 308,0,6209.0,208.74156068544266,18.19455808711857,217.78838580613308,189.24135324087982,220.46714355388787,197.475,3.05,199.0,194.425,199.0 311 | 309,0,6229.0,200.15100543926889,32.78806124684568,217.70952652188947,163.02638279697265,222.73669209227955,177.375,30.08974476272129,199.0,142.425,199.0 312 | 310,0,6249.0,231.7084127656497,34.471854226387975,244.48115825097074,195.16275797858742,255.28785092332487,187.575,22.85,199.0,164.72500000000002,199.0 313 | 311,0,6269.0,235.0708643417922,34.60281652347651,242.75847057252892,202.01076870003752,260.36580098824163,194.475,7.795421716908276,199.0,186.15,199.0 314 | 312,0,6289.0,199.46166620869423,25.922614683684188,210.5904165139632,170.4165708443092,222.39363341469843,194.4,9.200000000000001,199.0,185.2,199.0 315 | 313,0,6309.0,195.48138069099804,57.57905495237174,180.3488453173169,146.7229405956121,256.09906971616067,189.325,12.686435023070466,199.0,174.0,199.0 316 | 314,0,6329.0,195.75731185444204,36.93920901151747,203.95126826989727,155.3167684207484,228.05237065764237,184.35,22.74909416325893,199.0,159.275,199.0 317 | 315,0,6349.0,226.60118194117837,25.617712602134052,235.2413383685497,201.95494115203118,243.10899357544338,194.775,8.450000000000001,199.0,186.325,199.0 318 | 316,0,6369.0,223.9355117796678,23.240926321625746,230.84822821644235,198.295021324512,244.8781196872324,179.12500000000003,29.30448074308181,193.875,146.64999999999998,199.0 319 | 317,0,6389.0,195.79030427890444,48.80359965792316,206.32480620091346,144.697758764442,239.50736511708715,185.225,13.5168382034234,186.375,170.4,199.0 320 | 318,0,6409.0,207.81691918505157,30.398387040075832,219.97503928495192,175.12042429014403,231.17118739128605,182.5,26.123426660164846,199.0,153.72500000000002,199.0 321 | 319,0,6429.0,155.69829519458983,27.640186091236956,165.31228135211,123.58605177676121,179.99304080270497,191.25,15.5,199.0,175.75,199.0 322 | 320,0,6449.0,198.49294811911977,43.36276343104086,212.176358826879,152.28373247620044,231.10255672731319,188.475,21.050000000000004,199.0,167.42499999999998,199.0 323 | 321,0,6469.0,205.47464673950884,31.769631711685157,221.28445059310343,171.59767850105786,224.5909704357181,186.0,19.449094163258927,199.0,164.225,199.0 324 | 322,0,6489.0,195.66795626726488,37.9157314709521,196.6570072310726,156.8645983303793,231.57310476826785,183.70000000000002,30.6,199.0,153.1,199.0 325 | 323,0,6509.0,198.26038496959774,44.65060910968763,212.53269244209466,153.43807650416122,231.15841854249226,191.375,14.461951060010163,199.0,176.45,199.0 326 | 324,0,6529.0,200.4949641402848,28.690389184856485,202.35009257786606,172.6476818357354,227.69367394493355,193.25,8.235214022719749,199.0,183.275,199.0 327 | 325,0,6549.0,208.2612335425275,16.84353421950854,215.95568806986725,191.2666079193104,217.63431289076766,199.0,0.0,199.0,199.0,199.0 328 | 326,0,6569.0,199.22371950036904,30.373085065923206,208.1336825573873,168.31846329356438,223.12926041333856,190.55,16.900000000000002,199.0,173.65,199.0 329 | 327,0,6589.0,198.45260228507547,29.50807497554516,213.36106998325738,166.87859497074604,216.0985961673552,194.375,9.25,199.0,185.125,199.0 330 | 328,0,6609.0,203.00155684938005,30.423165955458124,208.84130205271708,173.78996254997105,227.8720739260508,188.475,17.94740332748399,199.0,168.95,199.0 331 | 329,0,6629.0,202.42428055691803,25.541052273469493,211.13197984980616,174.282585808212,223.8852495484797,199.0,0.0,199.0,199.0,199.0 332 | 330,0,6649.0,215.11270697467188,23.535302066474646,211.39485625874306,196.49259160710076,237.3064375489601,186.55,20.26447288315584,199.0,164.325,199.0 333 | 331,0,6669.0,197.2181629474696,28.920067403030078,215.07559468189373,165.1375757574891,216.18538593541973,196.875,4.25,199.0,192.625,199.0 334 | 332,0,6689.0,205.9100098812987,27.613617264684617,211.12576833448614,176.6908414401817,230.45651845563958,182.975,26.305680092863376,199.0,154.25,199.0 335 | 333,0,6709.0,152.29241568714045,13.941342217545705,155.2135796947406,136.88051061388438,165.2694892237708,199.0,0.0,199.0,199.0,199.0 336 | 334,0,6731.5,148.57248593252118,29.830048376173906,154.02553546535063,117.12716667759064,177.94080069340384,197.425,3.1500000000000004,199.0,194.275,199.0 337 | 335,0,6751.5,167.00457272403298,17.68571936408849,173.35540499997273,148.93420004651608,181.61927460871243,195.025,7.888446607650083,199.0,187.1,199.0 338 | 336,0,6771.5,156.32324093892657,15.245095844318692,161.85583984042154,139.24927732785187,168.80132426120485,189.85,18.3,199.0,171.55,199.0 339 | 337,0,6791.5,150.42683000239853,27.093902084311225,158.41996457220898,121.75060490897731,172.92778008250662,188.375,21.25,199.0,167.125,199.0 340 | 338,0,6811.5,159.01526889435456,26.256943554129112,160.1168095695607,135.32010891575231,180.31936060201514,198.825,0.35,199.0,198.475,199.0 341 | 339,0,6831.5,164.89703895601917,12.419864756345708,164.54906836441228,152.63707370746107,177.26168672143834,192.05,13.899999999999999,199.0,178.15,199.0 342 | 340,0,6851.5,157.9946778419294,15.16643506407257,164.29000471986345,141.50096719301285,169.89435309658612,195.04999999999998,7.8999999999999995,199.0,187.15,199.0 343 | 341,0,6871.5,209.4412622317734,11.920724478729602,212.39667642654342,197.47114947446278,219.32479881340373,193.5,11.000000000000002,199.0,182.5,199.0 344 | 342,0,6891.5,212.0968885908094,15.80158060825356,216.48208783175446,196.72600873402737,223.87364953381217,194.775,8.450000000000001,199.0,186.325,199.0 345 | 343,0,6911.5,202.7290969874202,22.93458491704575,215.4265830970201,176.83034352910295,219.76100482442288,199.0,0.0,199.0,199.0,199.0 346 | 344,0,6931.5,204.6202800639668,14.016994139633525,212.06721183873216,188.11206325401352,216.77404588791245,194.525,8.950000000000001,199.0,185.575,199.0 347 | 345,0,6951.5,201.49974888717168,24.816992466494543,215.29950380209658,174.03875509423028,217.32701690672098,184.425,5.230678732248808,182.625,180.825,189.825 348 | 346,0,6971.5,205.55797035964838,16.900167381923143,216.27957776552796,186.2849095804312,216.56541177583006,190.15,12.54801781352888,193.625,177.14999999999998,199.0 349 | 347,0,6991.5,199.52087941876357,24.744638207645163,208.6419502198553,171.69501859675677,220.91210943167448,180.75,23.166176592941703,188.625,155.525,199.0 350 | 348,0,7011.5,208.60866697155114,24.188200573010906,212.7510264108846,185.674855384768,227.94417461331025,195.075,7.8500000000000005,199.0,187.225,199.0 351 | 349,0,7031.5,206.1286897491121,27.568736350554445,215.59233429089934,177.85546463160415,228.723086948684,189.675,18.65,199.0,171.025,199.0 352 | 350,0,7051.5,195.9637176553106,39.1580730069958,215.983685174703,155.052645120637,221.90362223360654,199.0,0.0,199.0,199.0,199.0 353 | 351,0,7071.5,206.87773298260427,19.221953870330044,211.8439150063781,187.37391380208828,222.30020810791083,191.4,15.2,199.0,176.20000000000002,199.0 354 | 352,0,7091.5,208.3220811319658,22.651895826482015,213.19432171830874,186.85054784434556,226.69998440855758,199.0,0.0,199.0,199.0,199.0 355 | 353,0,7111.5,210.45181824572225,20.12721653212902,221.2401410611755,187.0262325334502,225.25658060253377,194.45,9.100000000000001,199.0,185.35000000000002,199.0 356 | 354,0,7131.5,215.75209374337828,14.890816247182688,225.2905336639428,198.5669667244207,225.45587194794393,194.775,8.450000000000001,199.0,186.325,199.0 357 | 355,0,7151.5,180.7794193763695,29.090704416699516,178.33101207699397,151.47437435445534,210.86690324092382,188.975,20.050000000000004,199.0,168.925,199.0 358 | 356,0,7171.5,165.1003927767129,33.06354070381069,168.13527500110564,135.07491485302114,191.0262670390416,199.0,0.0,199.0,199.0,199.0 359 | 357,0,7191.5,168.91778759063175,27.906134774379435,171.35588859301882,142.78625452587104,192.18275728393144,199.0,0.0,199.0,199.0,199.0 360 | 358,0,7211.5,169.86429303058014,14.322321293168258,172.12576711841535,155.8598144310337,181.92188458879474,191.4,15.2,199.0,176.2,199.0 361 | 359,0,7231.5,212.30600832530234,18.612753240842768,224.95440960630071,190.50522944357584,226.35963146621395,199.0,0.0,199.0,199.0,199.0 362 | 360,0,7254.0,191.13168884806788,49.79548876086788,213.72572180630277,134.4182652145811,230.81590938951544,199.0,0.0,199.0,199.0,199.0 363 | 361,0,7274.0,211.46212096154068,20.961509890815137,224.99022462830757,186.91671717066197,225.4370818770085,199.0,0.0,199.0,199.0,199.0 364 | 362,0,7294.0,204.51251641450386,36.00755712321476,221.06565199500497,166.35904906417232,229.4488607865947,194.775,8.450000000000001,199.0,186.325,199.0 365 | 363,0,7314.0,215.107147896968,13.982756112137196,220.0642661118377,199.7462118676211,225.95214620616252,194.775,8.450000000000001,199.0,186.325,199.0 366 | 364,0,7334.0,193.14546012636288,27.05893473811333,197.5400388945178,166.61691223256014,215.0606567722889,186.25,18.949094163258927,199.0,164.975,199.0 367 | 365,0,7354.0,220.03023077688894,8.892884457107064,224.22335712260255,210.58113658731057,225.43836174551265,199.0,0.0,199.0,199.0,199.0 368 | 366,0,7374.0,200.4052803138036,31.009394942992156,215.5199899967566,163.17409887505164,229.12766209385282,190.1,17.800000000000004,199.0,172.3,199.0 369 | 367,0,7394.0,212.18437801121465,18.993055734983486,225.4434677496976,189.86612447122155,225.445111290721,198.025,1.95,199.0,196.075,199.0 370 | 368,0,7414.0,208.84584863665276,28.87190024278129,215.61537309915047,178.655079144585,233.72071448568454,194.775,8.450000000000001,199.0,186.325,199.0 371 | 369,0,7434.0,216.74969745012223,22.38848908355411,218.58304424361222,195.20137883495767,237.335171174706,190.475,17.05,199.0,173.425,199.0 372 | 370,0,7454.0,217.9534657157085,24.520847167318152,222.8376252969929,193.80535885146514,236.9445951469848,190.55,16.900000000000002,199.0,173.64999999999998,199.0 373 | 371,0,7474.0,212.02764247379955,23.64620468052724,225.22679835629364,184.95924232823893,229.09897203978107,187.27499999999998,16.899094163258926,199.0,168.05,199.0 374 | 372,0,7494.0,204.1333204866844,27.727804217244834,212.46655028621885,175.08937314938504,228.93782878642506,190.55,16.900000000000002,199.0,173.65,199.0 375 | 373,0,7514.0,173.39068355640023,25.58574153509302,179.12755576646353,149.83918398704128,192.1708432202998,194.775,8.450000000000001,199.0,186.325,199.0 376 | 374,0,7534.0,203.81698800481144,21.81317162379069,212.607427505079,180.2626277682034,218.95885506860697,196.225,5.55,199.0,190.675,199.0 377 | 375,0,7554.0,189.49645838890356,23.89471520493001,185.8672197903786,168.07566753773492,212.37294490533176,185.875,10.7479649236495,177.875,176.525,199.0 378 | 376,0,7574.0,191.56474892442787,34.770506981511524,192.07125596518054,155.61058876843873,224.7934145783292,190.55,10.349094163258927,199.0,177.875,199.0 379 | 377,0,7594.0,195.80748446592085,32.53936070182078,203.99576204457713,161.03934142279064,225.94209776125547,191.4,15.200000000000001,199.0,176.2,199.0 380 | 378,0,7614.0,198.63782104085792,23.01581573155887,210.29067061313933,172.15140534669115,215.3650676437935,199.0,0.0,199.0,199.0,199.0 381 | 379,0,7634.0,174.242698649066,32.942315522595614,175.21860751406916,146.05238631888705,202.72626204005442,199.0,0.0,199.0,199.0,199.0 382 | 380,0,7654.0,163.2432333040086,22.636551781516822,168.4293804189884,138.79557957877694,183.89130241663466,199.0,0.0,199.0,199.0,199.0 383 | 381,0,7674.0,154.62096727418108,28.422782937627808,156.68553172978253,128.8056907244119,180.41514872511965,199.0,0.0,199.0,199.0,199.0 384 | 382,0,7694.0,158.29699330052063,17.0509102224122,157.38216337392734,143.53141791973388,173.81011931880764,198.625,0.75,199.0,197.875,199.0 385 | 383,0,7714.0,142.14261989094197,21.902845282157386,137.6541848078899,123.14058124161706,164.89977186494363,190.775,16.450000000000003,199.0,174.325,199.0 386 | 384,0,7734.0,148.53580434215542,18.717922192604505,148.30782861601963,129.7579993413989,166.39831498789493,199.0,0.0,199.0,199.0,199.0 387 | 385,0,7754.0,162.49517526336172,18.848841415947554,164.84033312359026,142.24381168929506,181.22149735917793,194.325,9.350000000000001,199.0,184.975,199.0 388 | 386,0,7774.0,164.6148257613949,36.05658465100554,177.009152592925,125.61708398755489,195.10233568789278,194.775,8.450000000000001,199.0,186.325,199.0 389 | 387,0,7794.0,165.40981411363265,12.857005094168152,171.6540407610402,150.44753231677214,176.5358007529382,192.05,13.900000000000002,199.0,178.14999999999998,199.0 390 | 388,0,7816.5,165.04450740886364,23.936321850710215,170.08745957641284,141.30871244867672,184.38959687650336,179.45,29.58089130885766,191.125,148.35000000000002,199.0 391 | 389,0,7836.5,157.36020835217568,23.13824946241963,168.8082571668369,131.44506446374368,174.50407783687754,194.775,8.450000000000001,199.0,186.325,199.0 392 | 390,0,7856.5,161.18339891217855,25.83782620284111,176.21883919223498,129.85513340189445,182.66115168453305,177.425,29.72252082342377,199.0,142.725,199.0 393 | 391,0,7876.5,159.12416731414274,29.28482737801874,161.43320261797496,127.82837814070075,188.56525017442812,194.775,8.450000000000001,199.0,186.325,199.0 394 | 392,0,7899.0,154.74061654609284,22.053219526938463,155.5643587404339,136.60602927286226,172.54594714205098,191.975,10.551464676150296,199.0,179.75,199.0 395 | 393,0,7919.0,173.213971193753,9.433097044232994,176.2835808966734,162.74107121142762,180.5715505332543,192.025,13.950000000000001,199.0,178.075,199.0 396 | 394,0,7939.0,158.1059824365,19.903429973254887,154.13783278674708,139.6254189262001,178.80236513724432,189.5,18.937732624901802,199.0,170.525,199.0 397 | 395,0,7959.0,196.27983788171113,30.577617510628315,202.2855675241465,167.64554828953862,219.357702157678,194.775,8.450000000000001,199.0,186.325,199.0 398 | 396,0,7979.0,145.82097149233985,34.74721414406535,148.05905417804448,112.01316098369448,179.48914978851298,181.975,15.816987172514963,186.875,166.1,194.25 399 | 397,0,7999.0,168.05081841084586,16.06448090482763,177.04376749694347,149.39281545299377,180.49942055565455,189.125,19.75,199.0,169.375,199.0 400 | 398,0,8019.0,190.3895678252673,28.225240451528624,192.7372206306375,163.9688889921985,213.6531733634164,193.95,6.877262972810417,199.0,185.55,199.0 401 | 399,0,8039.0,169.0155578141323,18.657576906412316,176.033349510081,148.95161747860155,183.8403890431101,191.0,16.000000000000004,199.0,175.0,199.0 402 | 400,0,8059.0,172.83179722326136,25.64973772132451,191.6968337829517,141.33843710534634,194.00713870608934,194.17142857142858,9.657142857142858,199.0,184.5142857142857,199.0 403 | 401,0,8079.0,166.863441232852,38.45570980928295,175.80609924503335,128.8735354443917,194.93638610633437,199.0,0.0,199.0,199.0,199.0 404 | 402,0,8099.0,181.51833074995903,41.37459104664495,203.7918829228359,130.78382796199276,218.6517806814722,181.2,35.599999999999994,199.0,145.6,199.0 405 | 403,0,8119.0,154.60780573724355,11.516865568074351,154.02924871534185,142.90774265228427,164.94706814351662,199.0,0.0,199.0,199.0,199.0 406 | 404,0,8139.0,89.95681650352465,0.07848167232814579,89.9996770753982,89.87382869523975,89.99999995220864,199.0,0.0,199.0,199.0,199.0 407 | 405,0,8159.0,66.26539734858,22.14751868200305,61.75704635189999,42.018176156399996,90.0,199.0,0.0,199.0,199.0,199.0 408 | 406,0,8179.0,86.54869875133018,6.377508155741371,90.0,79.86160152727665,90.0,199.0,0.0,199.0,199.0,199.0 409 | -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/logs/old/per_dqn_capacity_100.csv: -------------------------------------------------------------------------------- 1 | ,index,opt_counter,rew_mean,rew_sd,rew_median,rew_10th_p,rew_90th_p,steps_mean,steps_sd,steps_median,steps_10th_p,steps_90th_p 2 | 0,0,19.0,9.921072796292387,3.506075035454329,10.3772623327951,6.507089927782208,12.639868124113939,199.0,0.0,199.0,199.0,199.0 3 | 1,0,39.0,10.257512258786743,3.7527878582331473,10.049901545661978,6.934670233209087,13.916112543818391,199.0,0.0,199.0,199.0,199.0 4 | 2,0,59.0,18.920614301446697,7.2033001465737945,15.440787852042012,14.608949323349716,26.417263888515915,199.0,0.0,199.0,199.0,199.0 5 | 3,0,79.0,22.180582237418946,23.834831302241728,19.149544302586715,4.49580265106769,44.0780804936477,199.0,0.0,199.0,199.0,199.0 6 | 4,0,99.0,9.553248441736947,26.984447649874394,5.640381717474626,-14.722655441674027,38.50790832271547,191.15,15.700000000000001,199.0,175.45,199.0 7 | 5,0,119.0,39.420262404832265,18.810652843817323,42.98902318843216,20.99324720735011,55.48484145463064,199.0,0.0,199.0,199.0,199.0 8 | 6,0,139.0,4.673480866243622,21.811629117768575,3.3028523856780363,-16.631193782180567,24.538148487621925,199.0,0.0,199.0,199.0,199.0 9 | 7,0,159.0,57.019534779149176,26.737831350683333,49.81695766941061,39.18606517369547,82.33133616324263,190.15,10.838992111815562,199.0,176.875,199.0 10 | 8,0,179.0,9.73891401399166,9.283319319282498,4.853553884198554,4.371178447326471,19.555031808050494,199.0,0.0,199.0,199.0,199.0 11 | 9,0,199.0,18.386185811161575,10.647903129321111,16.561918437593544,7.785938840224615,30.44492979715617,185.875,19.23394108609388,199.0,164.15,199.0 12 | 10,0,219.0,51.37258089924842,41.90721136516937,36.340665663125,16.718180327923562,97.11635601029886,198.075,1.8499999999999999,199.0,196.225,199.0 13 | 11,0,239.0,123.48442604974387,26.76078805712572,131.0595625869009,96.34428301023134,145.3619335175452,199.0,0.0,199.0,199.0,199.0 14 | 12,0,259.0,102.04624211794471,23.8008384578891,109.51501835713154,76.6131499448246,120.99382821611879,185.575,10.961466598954722,176.625,176.625,199.0 15 | 13,0,279.0,88.00211587810092,16.819294853454746,93.34781709195903,70.60342076137326,100.21861697630723,185.575,10.961466598954722,176.625,176.625,199.0 16 | 14,0,299.0,111.39095130510769,48.12895067963085,103.78074023330814,67.63850062217273,158.75333544731788,184.9,12.311466598954722,176.625,174.6,199.0 17 | 15,0,319.0,58.36840942035987,23.92155979800837,57.821613146077,36.664990178725944,79.24386485529405,182.7,16.594625389754057,176.625,168.0,199.0 18 | 16,0,339.0,121.96529648794487,46.66137451258148,94.94982465475948,84.45333574346996,177.1297808026287,173.89999999999998,23.35,176.625,155.025,190.05 19 | 17,0,359.0,114.15662920403717,43.207682830796315,116.06304988551142,72.09047043045393,156.98088536153347,176.425,18.09141868488981,176.625,162.625,190.05 20 | 18,0,379.0,89.13565166409272,35.12844744335389,82.57454219208756,57.87619372887269,126.78456138586581,177.57500000000002,27.50937085787843,178.375,151.47500000000002,199.0 21 | 19,0,399.0,90.10901200914718,73.39658923036025,47.04387318257783,36.57529489274308,174.32977941853596,199.0,0.0,199.0,199.0,199.0 22 | 20,0,419.0,93.82967103219464,40.254659231478044,74.96023463895177,66.34330430709508,137.3668602144949,175.625,0.0,175.625,175.625,175.625 23 | 21,0,439.0,65.85820316636755,38.52264138153312,56.80632140570623,31.46264180000383,103.99438694421217,193.45,6.799172743797587,199.0,185.075,199.0 24 | 22,0,459.0,96.79903609070665,31.045812057180267,86.17005156314644,73.93693099944066,128.4284883850215,191.25,15.5,199.0,175.75,199.0 25 | 23,0,479.0,74.06316380762293,32.7846659329431,63.92527057981924,47.90851922361759,107.380652356215,192.325,12.574823940751468,199.0,179.3,199.0 26 | 24,0,499.0,78.00024013055405,33.32582121358256,70.06987400618287,53.10671284757878,108.69019085099146,178.175,13.670246703948191,179.375,165.05,191.375 27 | 25,0,519.0,77.3324777801766,11.75008348207878,74.07746874206845,68.31054605352058,89.16771078199463,186.3,19.034317270829547,199.0,164.675,199.0 28 | 26,0,539.0,83.2624480264875,4.93807026143847,83.07314561906722,78.8167691101723,87.99015095109384,195.475,4.284711192134191,198.375,190.075,199.0 29 | 27,0,559.0,105.73688272904266,30.59250178713095,104.83201307522607,80.92647826108707,132.29795627325998,198.025,1.95,199.0,196.075,199.0 30 | 28,0,579.0,83.50418545554139,17.41232532795211,84.21469594510808,66.52128476643091,100.0793537427006,194.725,8.549999999999999,199.0,186.175,199.0 31 | 29,0,599.0,112.72182902127692,25.542257462366646,107.2300299546558,90.26073910187185,138.70622188167312,177.975,36.25246181512456,199.0,139.1,199.0 32 | 30,0,619.0,126.24289924923666,26.16606162547189,139.07810893075867,99.01927095474653,141.69842717783217,190.15,10.838992111815562,199.0,176.875,199.0 33 | 31,0,639.0,134.71456884576824,29.252785300248476,135.55425119295683,107.38337884126133,160.52013734279515,175.77499999999998,31.153613125370857,177.625,146.47500000000002,199.0 34 | 32,0,659.0,101.88050560904806,13.2640157908898,101.8351007648687,89.8640998865224,113.99393885669453,186.025,10.594043137537245,177.375,177.375,199.0 35 | 33,0,679.0,132.6559561018165,10.930373395001677,136.6485019162523,120.86820744156573,140.81389772926084,199.0,0.0,199.0,199.0,199.0 36 | 34,0,699.0,117.47931192429492,21.64411784195475,117.73352083372814,98.2193216697811,136.57555913060264,185.35,27.299999999999997,199.0,158.05,199.0 37 | 35,0,719.0,96.91160933793115,9.130504555354008,99.92799345132632,86.55903894030278,104.66997309794851,172.47500000000002,21.44481151691125,182.625,147.675,191.975 38 | 36,0,739.0,134.75280954690743,23.22326839630265,138.8883913005399,112.67327442597303,152.8286808261203,169.925,18.127622241081838,159.75,157.05,190.05 39 | 37,0,759.0,141.33719354634525,11.72100959851268,145.22428610269708,128.50363743902435,150.5713584238253,190.325,16.799601900733563,199.0,173.2,199.0 40 | 38,0,779.0,107.47991736805152,35.95213148161648,106.14031712198862,78.9285616535712,138.38207136913883,188.52499999999998,20.95,199.0,167.575,199.0 41 | 39,0,799.0,133.07409040567865,38.45628777508173,142.29799066805413,92.98557549374522,167.0859376249188,190.55,10.349094163258927,199.0,177.875,199.0 42 | 40,0,819.0,139.7462886374559,14.863008125810207,143.18031140384784,124.6322947751455,153.17008033971945,190.05,10.961466598954722,199.0,176.625,199.0 43 | 41,0,839.0,121.30725512590354,16.477046636212386,128.11696038156526,104.34738108981766,131.2666299342072,181.1,8.95,176.625,176.625,190.05 44 | 42,0,859.0,114.80853116386152,13.440513980362768,119.02943761585563,100.85615155177689,126.38829048368103,181.25,19.553806159688563,176.875,163.55,199.0 45 | 43,0,879.0,128.94978084024314,9.335722171203562,134.66351370128083,117.89762147231241,137.25606965072768,194.725,8.549999999999999,199.0,186.175,199.0 46 | 44,0,899.0,129.26052857776733,16.58808566384738,135.97610620483988,110.33123719873267,142.63630850788374,194.725,8.549999999999999,199.0,186.175,199.0 47 | 45,0,919.0,133.60665183869014,13.835103265018542,140.9758871130297,117.81165394994153,143.61078529892694,199.0,0.0,199.0,199.0,199.0 48 | 46,0,939.0,128.1728262757461,25.53956850144639,141.7375931345396,100.23108695453465,144.11960999353744,194.575,8.85,199.0,185.725,199.0 49 | 47,0,959.0,138.42060504684457,9.854521477972764,142.8308513457289,127.77094142297446,145.1735271608823,196.825,4.35,199.0,192.475,199.0 50 | 48,0,979.0,136.35683731512464,12.438142379936904,136.39291464953263,124.2982762949591,149.0468633570404,196.475,5.050000000000001,199.0,191.425,199.0 51 | 49,0,999.0,133.56584284106984,29.30382736113911,138.9943225752975,107.33719261215067,154.7076878796404,190.70000000000002,16.6,199.0,174.1,199.0 52 | 50,0,1019.0,105.52318113599452,12.310166705561802,111.53684339497342,91.45875333071365,115.02935797505273,199.0,0.0,199.0,199.0,199.0 53 | 51,0,1041.5,102.51729539170273,10.688872998812041,102.63006312484094,91.91973514765081,113.19724403769216,184.85,17.415448998942406,199.0,163.1,199.0 54 | 52,0,1061.5,122.72312579176078,10.29739088757185,121.88558295735157,114.77589361424567,132.3784273711773,194.625,8.75,199.0,185.875,199.0 55 | 53,0,1081.5,104.05559260739625,12.492305455255856,107.34285393427348,91.31512094629012,114.67723120349791,199.0,0.0,199.0,199.0,199.0 56 | 54,0,1101.5,115.52999966004278,13.708384999130674,114.84638311494624,102.34334071383722,128.30196503576394,195.075,7.85,199.0,187.225,199.0 57 | 55,0,1121.5,105.52592955250589,12.372337806033057,108.76997459985917,92.67536949613152,115.72009306119631,199.0,0.0,199.0,199.0,199.0 58 | 56,0,1141.5,140.55217983820702,17.856177450228124,147.22088033464328,121.30720369391953,154.26934363524896,199.0,0.0,199.0,199.0,199.0 59 | 57,0,1161.5,133.77264770014244,17.28600642039682,134.83093273069554,118.9845341339168,148.14169348481883,195.075,7.8500000000000005,199.0,187.225,199.0 60 | 58,0,1181.5,140.5086930230791,17.98146810901206,144.77630812204117,122.05903492649826,154.45866216370374,199.0,0.0,199.0,199.0,199.0 61 | 59,0,1201.5,134.00176118408433,14.857860648591416,135.47250593916692,119.95716561830622,145.49755494466172,198.375,1.25,199.0,197.125,199.0 62 | 60,0,1221.5,140.77408359340274,20.121104701905708,146.21733280579315,120.12316161167956,156.0708309866088,199.0,0.0,199.0,199.0,199.0 63 | 61,0,1241.5,149.57426083189392,22.368570387637575,146.37561029934452,128.60917727250114,172.32579389305036,199.0,0.0,199.0,199.0,199.0 64 | 62,0,1261.5,129.83323690395468,22.767981054909566,136.40509464271761,105.62485027809123,147.16620051469204,199.0,0.0,199.0,199.0,199.0 65 | 63,0,1281.5,144.67242654336567,9.487895962844167,147.72234274891002,134.77184334999288,151.71666818476083,191.625,14.687640553537912,199.0,176.9,199.0 66 | 64,0,1301.5,135.82946221228946,30.785035046463474,134.81594169796978,110.29181188953316,163.35753881296006,194.475,9.05,199.0,185.425,199.0 67 | 65,0,1321.5,111.30456401756294,13.038151743521805,112.94517054394782,96.73352217661105,124.74216191294084,199.0,0.0,199.0,199.0,199.0 68 | 66,0,1341.5,113.98788631969363,16.915756615270844,119.5041156874282,95.37672154592612,128.36441606844122,196.575,4.85,199.0,191.725,199.0 69 | 67,0,1361.5,107.18149535538338,13.675235309892363,112.48212157373186,92.15215831091282,117.37103922616893,199.0,0.0,199.0,199.0,199.0 70 | 68,0,1381.5,112.55367304278693,12.28958683983092,112.51832333825186,100.26048873017591,123.29067354517693,199.0,0.0,199.0,199.0,199.0 71 | 69,0,1401.5,127.97984161684262,22.917497184372657,131.5367892816736,108.00941127836542,144.46069647218118,188.9,5.933169473392784,189.375,182.775,195.35 72 | 70,0,1421.5,131.9010707258028,13.5193595061482,132.7843256038445,117.37904367320604,145.9459232542007,195.175,7.650000000000001,199.0,187.525,199.0 73 | 71,0,1441.5,137.61814178667382,21.229755478200005,141.70126211076493,116.98667983137572,155.2115744294759,199.0,0.0,199.0,199.0,199.0 74 | 72,0,1461.5,140.88926206223263,36.99196697641975,143.3229398056552,104.54023890610722,173.736428365339,185.925,26.150000000000006,199.0,159.775,199.0 75 | 73,0,1481.5,146.80689079391718,32.07196449954032,158.96770816428662,114.12365774231793,169.21045369761313,185.25,11.315963908234703,176.375,175.9,199.0 76 | 74,0,1501.5,142.1195767880454,20.993840712518594,148.758146558459,120.3390062587894,158.96297379274725,183.325,31.35,199.0,151.97500000000002,199.0 77 | 75,0,1521.5,162.42986565461507,22.31265290291755,169.77716693538864,139.35032385085262,178.43642508288426,177.14999999999998,26.03950022090704,176.875,152.025,199.0 78 | 76,0,1541.5,207.26012317368952,14.168740798888214,209.6050137723245,192.1486944809164,220.15020085526396,178.77499999999998,19.975807712746864,178.625,157.8,199.0 79 | 77,0,1564.0,170.79575403927925,14.124176878507324,167.31032895922246,159.336813535572,186.09843143384683,192.625,7.8271299333365905,199.0,183.05,199.0 80 | 78,0,1584.0,181.66352768130957,17.256838854368006,188.0760413667579,163.60905776636224,194.8914547142153,196.825,4.35,199.0,192.475,199.0 81 | 79,0,1604.0,185.83708971494286,34.513934929383524,191.67319279045765,149.92463829941,218.54243293058417,169.725,30.543766367937863,176.875,136.425,199.0 82 | 80,0,1624.0,234.8867716369552,18.206742737301386,240.59609656329405,215.1378915818134,250.52901838283125,189.57500000000002,13.475183282487247,199.0,174.04999999999998,199.0 83 | 81,0,1644.0,235.03252604206682,18.801199277213403,239.58693235407324,216.65509574951898,248.57762910164658,178.875,33.69909416325893,199.0,142.85,199.0 84 | 82,0,1664.0,235.13248530244203,30.68306634809259,236.3596126874797,204.2071359949698,263.4837201466558,171.75,30.028162486939006,176.625,140.29999999999998,199.0 85 | 83,0,1684.0,222.50044664692103,28.589426345698048,220.65967365408866,193.67641593714842,249.9683816192996,185.525,20.088992111815564,199.0,163.0,199.0 86 | 84,0,1704.0,208.4685935915325,17.751476352091483,208.63821193731692,190.5565148387094,224.8722913342931,190.15,10.857428332713045,199.0,176.675,199.0 87 | 85,0,1724.0,186.54658782161493,30.420846865712733,193.25851870533384,156.50239546999768,210.74235941712683,176.075,30.789819048987848,185.375,143.9,199.0 88 | 86,0,1744.0,155.61711408307457,12.196989936381192,157.62349712496228,142.9413045244471,166.40528404658357,186.42499999999998,18.444043137537246,199.0,165.6,199.0 89 | 87,0,1771.5,234.3857153647122,13.729460768618146,238.54452201869194,221.2358309445317,243.76754956160067,189.57500000000002,15.628134539066114,199.0,172.25,199.0 90 | 88,0,1791.5,145.44033160647223,21.167192181031695,147.79416856797437,124.15463385394986,165.77488836911758,176.075,17.8,175.625,162.95,189.65 91 | 89,0,1811.5,201.1892757954199,31.22861130873197,204.49014356172495,167.5858584540639,231.18606638919482,181.49999999999997,28.12342666016484,199.0,150.725,199.0 92 | 90,0,1834.0,228.02791095755254,26.706656745508038,231.78310956525172,200.37029439451328,253.94302637299324,187.175,9.701997217068246,180.875,178.375,199.0 93 | 91,0,1854.0,250.37241088007548,35.82622965938402,245.80579268221402,217.02561595063983,284.4671447297465,171.675,9.633355148267313,176.25,161.275,177.875 94 | 92,0,1876.5,227.58627239475868,49.309665793405955,239.89238065108952,173.50014715966216,269.79364448491407,164.60000000000002,55.09642044452862,199.0,104.45,199.0 95 | 93,0,1896.5,231.77062442213733,39.60555991662361,231.67727778242934,191.7258590750392,270.5816647365969,183.3,25.16046398502626,199.0,155.875,199.0 96 | 94,0,1916.5,198.99420869943282,34.49433161711724,202.6680141168141,166.84627769119294,229.09526667719706,190.55,16.900000000000002,199.0,173.64999999999998,199.0 97 | 95,0,1936.5,226.20370968454534,22.449430334958603,228.72725596807976,203.0345921114485,247.8374871975567,185.925,26.15,199.0,159.77499999999998,199.0 98 | 96,0,1956.5,199.24959104987852,26.484015127390112,199.1476041306654,172.69792366237073,226.27800665203654,172.65,6.45,175.875,166.2,175.875 99 | 97,0,1976.5,229.96629890993694,30.238281747238933,237.405000176385,199.50710126281368,252.50977818727264,172.475,36.6288900603722,175.875,137.925,199.0 100 | 98,0,1996.5,217.28137332852558,24.766982064371348,223.76190944924883,191.6877121842834,235.52561225111327,194.775,8.450000000000001,199.0,186.325,199.0 101 | 99,0,2016.5,232.72498835245125,36.3611143610284,240.70783948088126,195.87559355430594,262.71105679415604,170.0,23.325100988098033,177.375,147.475,190.35 102 | 100,0,2036.5,222.95521910903963,27.14145060116696,221.5644133566123,197.93303440850923,250.27733121119934,194.7,8.6,199.0,186.1,199.0 103 | 101,0,2056.5,231.64960045861918,27.48624202010398,244.03189391783906,201.75793842822873,252.19926602451528,173.07500000000002,19.997366596101028,182.125,153.175,183.925 104 | 102,0,2076.5,225.59485780334984,40.12664402109519,244.92333670621093,180.97420790943062,254.25626893056807,185.925,19.59909416325893,199.0,164.0,199.0 105 | 103,0,2096.5,238.45505329670152,34.49139726156832,249.78325596920678,199.7819461817848,266.1170335143639,186.075,24.50355747039448,199.0,160.8,199.0 106 | 104,0,2116.5,243.05433511960928,25.79062751046709,250.76568997713454,217.16741299932676,260.4235819047515,179.125,32.5788900603722,199.0,144.0,199.0 107 | 105,0,2136.5,224.81405700861086,31.054965790170954,226.98563156011397,192.01247345126518,254.78080235434223,179.125,17.0778349497163,180.625,164.35,192.15 108 | 106,0,2156.5,228.93068964342456,27.29136279982943,243.54840999566954,198.80806236654374,248.40049943866018,181.57500000000002,24.68177890595466,199.0,152.975,199.0 109 | 107,0,2176.5,199.09871878483315,30.027476869466874,199.169492020153,169.15512194047747,228.8357326701855,192.55,12.900000000000002,199.0,179.65,199.0 110 | 108,0,2199.0,210.20214099697642,30.337061152782407,225.00463173558379,178.22705210778463,228.345823193042,176.82500000000002,28.106415573233043,176.125,150.775,199.0 111 | 109,0,2219.0,230.05016812369234,38.11743651972579,252.8255644486285,187.51646467189738,255.55632370635465,181.75,27.62342666016484,199.0,151.47500000000002,199.0 112 | 110,0,2239.0,223.9950511740766,32.473583662559605,236.56687557377842,186.3270522440426,254.76784116038107,176.475,17.6,176.125,163.45,189.85 113 | 111,0,2259.0,227.3110761345386,31.538828688064456,230.51468335466512,197.33925091454634,254.55250850471566,163.45,18.79909416325893,176.125,142.325,176.125 114 | 112,0,2279.0,241.79923965350412,35.01290061517376,245.03836919330098,204.23393306060203,275.16210442691806,186.32500000000002,25.35,199.0,160.975,199.0 115 | 113,0,2299.0,248.584913419457,33.935746259431326,248.46899813059616,212.20185344596158,282.7406341071537,183.10000000000002,15.55641557323304,176.125,169.6,199.0 116 | 114,0,2319.0,276.32376975253453,46.85973629115282,286.19407576651054,229.50103754568417,317.3693746080709,190.55,16.900000000000002,199.0,173.64999999999998,199.0 117 | 115,0,2339.0,260.3187160228842,43.057799609182524,255.88827478110528,222.33016173979786,302.14250835604,176.275,38.35641557323304,199.0,135.4,199.0 118 | 116,0,2359.0,258.08254360300197,55.100848900086866,261.9774251732832,209.47610769638715,303.04596626017974,160.675,31.292231464055103,135.125,135.125,199.0 119 | 117,0,2379.0,252.2927683621545,51.370754052350435,265.4598780929403,202.18756218313302,294.60893318293904,176.225,29.128890060372203,175.875,149.175,199.0 120 | 118,0,2399.0,255.3852377430003,63.49004466200134,285.06409786383904,184.22536471787345,305.3504460350582,174.025,13.832455532033677,181.125,160.02499999999998,181.125 121 | 119,0,2419.0,299.7075684976601,23.639733954183885,310.48274904616653,273.4392037888995,318.1001016957883,189.79999999999998,18.400000000000002,199.0,171.4,199.0 122 | 120,0,2439.0,291.3219662534474,41.259046756923695,313.97781722052264,242.47166117515764,327.0613231505896,188.175,21.65,199.0,166.525,199.0 123 | 121,0,2459.0,296.05613661435507,32.15766573304049,312.0326087016212,258.25702251659084,324.6866793772605,179.95,21.518832059088563,179.375,158.775,199.0 124 | 122,0,2479.0,293.04099493044276,55.35021549531367,321.7713965078117,233.1560342726272,329.5324610970308,182.325,24.972687933820183,199.0,153.825,199.0 125 | 123,0,2499.0,275.45978543428,39.42148931735409,294.0365817857307,232.31400711083722,302.7132065264987,176.725,21.641321878328622,175.875,154.65,199.0 126 | 124,0,2519.0,264.58997117519465,34.6669011143255,272.9784276348432,227.94722457516735,293.18156795865735,165.1,24.5288900603722,153.5,147.125,190.05 127 | 125,0,2539.0,246.51963688068702,47.93134592423279,263.2129629539886,194.66885434468077,283.24915132436195,167.475,20.167159121676,153.5,153.5,190.55 128 | 126,0,2559.0,246.05131507589022,48.313778725710314,245.1978252796241,196.3651042952506,292.80788695962246,178.70000000000002,37.966530447647436,199.0,139.27500000000003,199.0 129 | 127,0,2579.0,197.1492145801203,32.98662051290667,218.6919821992982,158.04133852655406,223.24231795636967,190.15,17.28773905854667,199.0,172.625,199.0 130 | 128,0,2599.0,184.91653488592405,32.95321315748929,178.94891977642803,157.0145505865491,217.39791018731415,192.475,13.05,199.0,179.425,199.0 131 | 129,0,2619.0,218.00543478592397,49.92978637875188,210.318253873595,177.17883575942898,268.5910828583097,181.375,29.37256462695887,199.0,149.5,199.0 132 | 130,0,2639.0,202.91213649329097,38.122284971998496,203.49605884930824,163.79926366617676,239.57379581859993,186.475,25.050000000000004,199.0,161.425,199.0 133 | 131,0,2659.0,204.1138412562433,25.429047048949574,209.2942157111753,175.57459099641414,227.6009423488642,199.0,0.0,199.0,199.0,199.0 134 | 132,0,2679.0,239.84356541895687,44.575788008048804,247.79189382487695,191.53086302815186,281.28694493807666,180.62500000000003,36.75000000000001,199.0,143.875,199.0 135 | 133,0,2699.0,178.98031099867646,28.522624441688713,175.95534091725585,155.49053769100516,204.4850467096192,180.5,9.25,175.875,175.875,189.75 136 | 134,0,2719.0,218.26901796927444,69.3992735856225,223.90104675706465,153.16834808722993,278.21988474574846,181.89999999999998,20.944317143451464,199.0,156.2,199.0 137 | 135,0,2739.0,238.62713798999658,64.44834823021961,242.9112636366189,173.32402197495114,301.8344663269058,199.0,0.0,199.0,199.0,199.0 138 | 136,0,2759.0,244.9710556985996,38.584336093175516,255.73718288283786,202.98115961395257,280.5115366044337,194.775,8.450000000000001,199.0,186.325,199.0 139 | 137,0,2779.0,272.14364473088074,32.878870547382036,284.4081830638456,234.41419706399802,300.20046220300594,190.55,10.349094163258927,199.0,177.875,199.0 140 | 138,0,2799.0,279.26125717261687,41.99390472744794,292.62877194303377,235.0030720313348,310.18460213713337,194.775,8.450000000000001,199.0,186.325,199.0 141 | 139,0,2819.0,244.87531418796502,29.920948247431404,255.88632545742286,211.834407577276,269.7612012115716,183.72500000000002,30.550000000000004,199.0,153.17499999999998,199.0 142 | 140,0,2839.0,246.87458301111587,31.181116294179727,263.8718293518224,210.35147884336456,270.74708791551444,185.875,19.373426660164842,199.0,163.85,199.0 143 | 141,0,2859.0,238.81939829764437,40.0793078617357,253.74336863447968,193.83895841952204,271.6574128896358,183.0,32.00000000000001,199.0,151.0,199.0 144 | 142,0,2879.0,258.6643098213392,30.758171792993267,267.58847779792234,226.17591186695233,285.68117934830184,199.0,0.0,199.0,199.0,199.0 145 | 143,0,2899.0,271.79562169024564,30.50934525219309,280.91382356231594,240.04841167372516,296.51680787014664,195.325,7.35,199.0,187.975,199.0 146 | 144,0,2919.0,224.1068311658526,34.044782233766334,241.1325333372918,187.31068385144448,245.49873617004374,190.625,15.961362704153874,199.0,174.2,199.0 147 | 145,0,2939.0,229.1163267366523,33.252320284896406,248.06822727236434,190.05102153388748,255.9932991142028,194.775,8.450000000000001,199.0,186.325,199.0 148 | 146,0,2959.0,219.36673500973993,30.572541067908354,229.56767737039536,185.59237876018292,243.5658197697391,199.0,0.0,199.0,199.0,199.0 149 | 147,0,2979.0,211.03069508070422,37.52157705494794,224.35631481096772,169.35034511582782,239.43191929053262,192.45,13.1,199.0,179.35,199.0 150 | 148,0,2999.0,198.92988462587397,17.578851251445084,206.22617556446818,179.95383027590884,210.4940377088678,194.775,8.450000000000001,199.0,186.325,199.0 151 | 149,0,3019.0,215.72667911948002,20.0234070430243,220.4233700171001,193.83813330643648,234.19112126091096,194.15,9.700000000000001,199.0,184.45,199.0 152 | 150,0,3039.0,240.52740689588015,28.992506538992608,240.28813424805162,216.94502603752568,265.5862071805127,199.0,0.0,199.0,199.0,199.0 153 | 151,0,3059.0,238.714073788786,44.283443679205,264.7977726427189,187.70558700781316,271.56872387931526,190.55,16.900000000000002,199.0,173.64999999999998,199.0 154 | 152,0,3079.0,256.4112198107189,18.172427160643696,261.15875468326135,237.75619043007694,270.61698462828514,196.45,3.1270992948737653,199.0,192.575,199.0 155 | 153,0,3099.0,259.22702560340934,33.836205853705124,261.4019862678241,227.9753616823697,288.4419500609085,185.125,11.3288900603722,175.875,175.875,199.0 156 | 154,0,3119.0,235.43366051545382,23.17445455370272,238.76396665272506,211.2090517123623,256.3226710193038,187.29999999999998,17.15919621470229,199.0,167.925,199.0 157 | 155,0,3139.0,223.55280759576868,33.59058578421682,231.8622216436278,188.21985360779718,251.9999457877706,180.975,21.292337808373503,178.875,161.225,199.0 158 | 156,0,3159.0,189.67714544935114,24.789173382708608,191.66638499375625,162.4434938471366,213.54438934877595,185.7,17.59089411558312,199.0,163.95000000000002,199.0 159 | 157,0,3179.0,192.35167050795823,34.88057653964412,199.22100289930322,155.08826064515108,224.02379946782568,186.075,19.112609905646927,199.0,164.45,199.0 160 | 158,0,3199.0,210.92677837950626,39.512888286486394,215.0235850256446,172.24411445095262,245.83383429399942,189.1,17.984765841548707,199.0,170.125,199.0 161 | 159,0,3219.0,215.00943885423715,51.18563342030721,236.28788941137276,157.8092868442925,255.01620957868946,194.325,9.350000000000001,199.0,184.975,199.0 162 | 160,0,3239.0,190.98146051379834,38.817455721361085,197.13859804322027,147.25715211309918,229.19120407024866,185.5,27.000000000000004,199.0,158.5,199.0 163 | 161,0,3259.0,213.10887075937848,54.517897579001485,216.75950645761696,160.83878299460866,261.79001759618376,181.82500000000002,34.35,199.0,147.47500000000002,199.0 164 | 162,0,3279.0,199.26209941707836,32.16921071346021,209.03110188742545,163.08790231632872,229.3067663050686,178.17499999999998,17.63945112778886,179.625,162.85,192.15 165 | 163,0,3299.0,203.22217694469145,33.257441063076676,210.00658762988385,166.16399807934292,234.4303211315311,199.0,0.0,199.0,199.0,199.0 166 | 164,0,3319.0,209.28616439207394,32.567423309180754,213.51283325056332,179.20123515812404,235.45029367229267,175.875,33.46249137930624,182.375,142.975,199.0 167 | 165,0,3339.0,177.4098978294173,40.81409066822334,179.51653408130392,138.5649032146752,214.20944906079885,176.0,26.714031132771773,178.875,148.875,199.0 168 | 166,0,3359.0,189.611977263838,33.04481374990495,195.92878788443866,156.86897963975966,218.07265996089842,194.775,8.450000000000001,199.0,186.325,199.0 169 | 167,0,3379.0,200.9560334337635,46.455105435344805,214.1188069458343,156.16869594285293,238.47780037789596,194.475,9.05,199.0,185.425,199.0 170 | 168,0,3399.0,188.50488378851972,44.780280774756605,208.1840412209658,138.70424691162228,220.87348808095965,181.42499999999998,20.090462582103463,187.25,158.275,199.0 171 | 169,0,3419.0,191.3570939690661,43.549496949966695,198.55865802276287,147.29583362849385,228.24614484737822,195.775,4.773279273229244,199.0,190.3,199.0 172 | 170,0,3439.0,189.9083979182471,31.799022708931552,198.23172184359325,156.86816804223892,220.8350987095267,191.2,10.06853226357483,198.625,178.65,199.0 173 | 171,0,3464.0,225.2838183293172,25.37233093599456,222.17739677687507,200.29053340679604,252.05898312482336,173.125,28.763848107335363,175.0,143.825,199.0 174 | 172,0,3484.0,216.1323028026708,18.420039287228004,217.16041530857154,196.81902662392406,233.0843198763642,193.425,11.149999999999999,199.0,182.275,199.0 175 | 173,0,3504.0,225.19860617045293,47.010592411505975,233.9969241057903,176.53073767562663,265.397404455551,189.475,7.860263990477674,184.625,182.125,199.0 176 | 174,0,3524.0,199.17268478894596,31.467509552908467,198.99302622565855,168.8141527881291,228.77072373198376,191.15,15.700000000000001,199.0,175.45,199.0 177 | 175,0,3544.0,217.65816928914094,33.101329278210656,207.13043846024397,188.55118005848968,253.6684980125701,188.3,11.96527209416505,191.375,174.8,199.0 178 | 176,0,3564.0,207.68115916958095,34.959666394494775,224.6347061679286,168.69170322632448,233.0769819861173,183.6,24.045638650168726,199.0,156.92499999999998,199.0 179 | 177,0,3584.0,206.82178371430413,48.31642149789287,200.49558909078613,163.39849722138678,253.7160039792911,190.72500000000002,16.55,199.0,174.175,199.0 180 | 178,0,3604.0,222.19436491264673,38.55525618608424,237.02895274858875,181.05883572840833,254.23844933045206,189.375,15.435163700339709,199.0,172.35,199.0 181 | 179,0,3624.0,256.73312436876773,31.0836487449294,258.89345703670386,225.66849440239662,286.8091872333303,188.125,16.375183282487246,199.0,169.7,199.0 182 | 180,0,3644.0,251.33692673863624,42.05011283278648,247.4110156380994,213.6925004917905,293.01733486101125,168.7,28.238278288764015,176.375,142.5,189.95 183 | 181,0,3664.0,250.3511322091117,53.21044009164536,247.65681451966387,200.1268620089008,298.49390026378455,185.70000000000002,17.664323089625196,193.875,166.2,199.0 184 | 182,0,3684.0,260.8456942386955,27.68027186651684,256.4429383218654,234.59475896088094,289.52507962492945,186.32500000000002,18.79909416325893,199.0,165.2,199.0 185 | 183,0,3704.0,245.74287034192173,31.10143648573169,258.0946928632254,209.87754619398507,271.4096182463211,180.475,30.91876399551141,199.0,147.15,199.0 186 | 184,0,3724.0,263.60409798155877,42.126173142432556,268.8342817238733,218.36227121237744,302.2504100790535,199.0,0.0,199.0,199.0,199.0 187 | 185,0,3744.0,224.97059291912075,30.90424739478578,222.6769415209033,195.23638498005022,257.45837313763695,199.0,0.0,199.0,199.0,199.0 188 | 186,0,3764.0,204.67347777102924,30.37525291077611,212.97158155443842,172.2113736699456,228.0886499577995,172.0,39.98631839308524,199.0,126.85,199.0 189 | 187,0,3784.0,205.51915237610828,35.9521539952261,210.41935035455492,170.01620815166032,239.54006723558194,182.875,23.405968055537848,199.0,155.175,199.0 190 | 188,0,3804.0,195.70113793312757,30.987665154374792,202.15542095099252,164.62284856703383,222.42870459979045,173.55,19.501364547511358,178.875,155.5,190.95 191 | 189,0,3824.0,246.30769501914182,31.755555079848143,251.25420948054602,215.03576325045856,275.7919763858686,187.325,9.741855572733565,181.375,178.075,199.0 192 | 190,0,3844.0,254.70178218183628,33.263786360469744,261.97443747820716,218.72649451158543,282.95703552934964,176.475,0.3,176.625,176.175,176.625 193 | 191,0,3864.0,235.54223068307897,18.38302023582571,237.0320713703099,216.62880436001964,251.90379339918755,192.4,13.200000000000001,199.0,179.2,199.0 194 | 192,0,3884.0,215.6232244664024,45.83002872341968,229.98681281493458,166.14216016527865,250.32960699945568,186.75,18.068148799068588,199.0,166.32500000000002,199.0 195 | 193,0,3904.0,186.9148200913483,34.41608052220883,201.20056905763315,150.20496710620063,210.42047785638042,163.35000000000002,20.148333127521198,158.0,148.375,182.975 196 | 194,0,3924.0,208.04127606673333,47.27769015610886,226.6390729666634,156.34087261953013,245.4547303747269,176.22500000000002,25.404136750847755,183.875,152.14999999999998,192.95 197 | 195,0,3946.5,188.3241611557379,16.298472922842098,192.9969260481732,171.4306332188785,202.640360692768,190.47500000000002,17.049999999999997,199.0,173.425,199.0 198 | 196,0,3966.5,162.93701698518953,26.853434769935593,156.7358916614238,142.80254448893191,188.78302785297674,179.5,10.536953767623375,175.875,173.25,189.75 199 | 197,0,3986.5,171.874783158577,29.404723792205715,163.85039497513833,150.33360245084052,202.04160134458633,187.925,14.9788900603722,199.0,170.4,199.0 200 | 198,0,4006.5,188.5450905817873,21.13658624798913,197.82897169545566,165.53050565649446,202.69184743613465,167.425,16.900000000000002,175.875,150.52499999999998,175.875 201 | 199,0,4026.5,207.30196699656932,24.08223683925921,212.88876438780636,181.06111005445888,230.36876685753128,192.05,13.9,199.0,178.14999999999998,199.0 202 | 200,0,4046.5,228.87091744147966,36.91684593474923,215.36172007349757,203.6280393805793,266.04554296404115,178.975,21.428375560121378,182.75,154.825,199.0 203 | 201,0,4066.5,203.77445027706895,31.567206296966127,212.8881946469338,172.06177717702644,227.9584774837309,192.35,13.3,199.0,179.04999999999998,199.0 204 | 202,0,4086.5,238.141027101493,43.8787352429743,253.68194227647538,188.97337328717515,273.685881129589,188.425,21.150000000000002,199.0,167.275,199.0 205 | 203,0,4106.5,244.36788411872337,35.520149606772065,258.1577537179849,203.97997685451776,272.70966282774725,190.55,16.900000000000002,199.0,173.64999999999998,199.0 206 | 204,0,4126.5,256.5285819503761,35.975349843396685,272.0163012419455,217.4160232940337,283.61120109134833,186.27499999999998,19.220600285696918,199.0,164.7,199.0 207 | 205,0,4146.5,238.02430690839168,34.24500993948811,248.94301144035174,198.34261951088172,268.39729029647907,174.55,19.99337840752002,159.75,157.35,199.0 208 | 206,0,4166.5,238.29925599166813,38.91894908336975,245.37965595170462,196.8276935407591,271.9180960190471,184.04999999999998,25.66906506128373,199.0,156.275,199.0 209 | 207,0,4186.5,251.8682532400855,31.11108698014413,265.9568864476459,216.7771419284631,275.07655788101516,192.05,13.900000000000002,199.0,178.15,199.0 210 | 208,0,4206.5,250.81775335919605,16.362657600295712,258.9437414628149,232.09462965155046,263.9525113992504,180.425,23.39177275328482,179.625,158.775,199.0 211 | 209,0,4229.0,218.5663880978372,13.35408718542219,219.9793080913965,204.39883371445987,230.97441481346195,186.35,6.941901756723442,182.375,180.725,194.75 212 | 210,0,4249.0,249.95940953338743,29.22092964597624,260.8549718585465,218.3739036267193,271.0106232509121,177.25,29.564247240423974,179.375,149.45,199.0 213 | 211,0,4269.0,254.62067524051594,26.936370433727912,264.47351537648643,224.68226582115028,276.1192580730314,182.375,27.164247240423975,199.0,153.05,199.0 214 | 212,0,4289.0,249.89850588966118,26.877105705502522,264.0584136511324,218.34481559799525,271.2243638029269,174.55,24.921066013941044,179.375,153.125,191.15 215 | 213,0,4309.0,246.04701550806487,36.83818565152433,262.2273515144271,204.15778348227428,275.5912195767751,186.875,18.164247240423975,199.0,166.55,199.0 216 | 214,0,4329.0,211.44708273730055,27.406902238316423,228.05611714531614,179.7931515481747,230.45378046859986,186.925,18.064247240423974,199.0,166.7,199.0 217 | 215,0,4349.0,206.74784814117496,28.35509584904512,224.93311749734488,172.33808897822945,228.57344473820666,199.0,0.0,199.0,199.0,199.0 218 | 216,0,4369.0,155.91080504749533,31.028809715574972,160.31286827004183,123.22617179899868,186.12242704232122,170.125,27.24902187576601,179.375,144.075,191.15 219 | 217,0,4389.0,176.68965958090143,16.41078993407045,179.22085684880292,158.93734604356428,193.0542638292146,189.775,12.054145188980609,199.0,175.45,199.0 220 | 218,0,4409.0,182.7167570386452,26.947194543127818,178.77865545479335,160.19877286968,209.72606645224317,184.7,7.792464308548356,179.125,179.125,194.25 221 | 219,0,4429.0,178.6102214937303,40.657266171223206,173.6299081850259,146.61406847286742,216.05806628016325,199.0,0.0,199.0,199.0,199.0 222 | 220,0,4449.0,161.8742703451869,37.71407578268485,170.918487158964,121.37746400360771,194.18878237371788,190.55,10.349094163258929,199.0,177.875,199.0 223 | 221,0,4469.0,207.74680917874687,45.09257052636366,210.85744499855065,164.11780296032634,246.05144927406548,194.525,8.950000000000001,199.0,185.575,199.0 224 | 222,0,4489.0,222.34460473033,10.17307057113375,225.5034571889708,212.20686121285985,228.82840408504867,194.775,8.450000000000001,199.0,186.325,199.0 225 | 223,0,4509.0,172.3896552331321,28.63404770011774,182.78226835652438,140.72220073848962,194.6682059132804,183.125,16.38343143989449,188.375,164.45,199.0 226 | 224,0,4529.0,180.59363023319176,56.375982444267784,173.9135502008022,131.4025295370381,235.52864450105838,196.575,4.85,199.0,191.725,199.0 227 | 225,0,4549.0,149.60900641296666,24.2055674966374,156.8086984242813,122.03618958905041,170.01102729744417,193.25,11.5,199.0,181.75,199.0 228 | 226,0,4569.0,177.92727400251985,33.25141363790668,190.71219830783113,141.73942532747438,204.99210652312885,199.0,0.0,199.0,199.0,199.0 229 | 227,0,4589.0,169.83667208333577,33.594287151856705,181.29232752278358,132.26953219586204,199.038329156318,194.775,8.450000000000001,199.0,186.325,199.0 230 | 228,0,4609.0,187.4232444219472,35.60345662974079,194.904793202634,150.97438356979222,218.7135862242072,187.525,17.529854073717267,199.0,167.6,199.0 231 | 229,0,4629.0,187.54117889495407,16.532255212184655,183.86645399613172,175.82620562461716,202.73599331610484,191.375,15.25,199.0,176.125,199.0 232 | 230,0,4651.5,171.50508815866627,39.6143660283124,188.21705400605836,129.7455158478858,201.66246435699617,190.55,16.900000000000002,199.0,173.64999999999998,199.0 233 | 231,0,4671.5,189.75793489860283,21.25078769613118,187.68818253585607,172.8825183115003,210.5127543986441,191.8,14.399999999999999,199.0,177.4,199.0 234 | 232,0,4691.5,183.35103462372456,24.191640179038952,196.15274077905121,155.8003839060372,201.51339440419582,191.0,16.0,199.0,175.0,199.0 235 | 233,0,4711.5,222.85560395096257,33.26363880264393,240.01673920880242,185.38459736968127,246.8815509639738,194.65,8.700000000000001,199.0,185.95,199.0 236 | 234,0,4731.5,218.9120800787517,32.03050421435322,225.1021994208753,184.32959685823448,246.60542762967094,191.20000000000002,15.600000000000001,199.0,175.6,199.0 237 | 235,0,4751.5,212.55178951061663,58.934867321210724,216.7655931250661,155.71822058669807,269.4079363935958,174.425,35.98588242277679,199.0,133.75,199.0 238 | 236,0,4771.5,180.5718501915246,27.073452732827363,182.74790549105165,153.07747863301623,206.0756151422872,189.025,19.950000000000003,199.0,169.075,199.0 239 | 237,0,4791.5,153.68987959551498,21.464294322435762,156.5375077321258,132.71302870139144,174.60731792377237,184.925,18.617343836635776,199.0,162.475,199.0 240 | 238,0,4811.5,188.2985688157084,41.44395103372487,190.66588542179383,148.51309392912455,227.33335252785065,194.14999999999998,9.7,199.0,184.45,199.0 241 | 239,0,4831.5,178.9246421206291,34.08471210976964,181.60655386256255,145.4684698676192,208.02415563572976,188.725,15.920190760616492,199.0,170.89999999999998,199.0 242 | 240,0,4851.5,213.7853865027862,45.93828701772583,205.46295358928498,176.49978006196045,258.91624672609464,191.75,14.5,199.0,177.25,199.0 243 | 241,0,4871.5,207.93081578420598,20.791546795251488,209.88155280679283,186.3479645667942,228.19893415641053,188.575,13.97342666016484,199.0,171.95000000000002,199.0 244 | 242,0,4891.5,178.8367354560568,38.60145696594709,202.26070002192736,132.99205516125375,210.9154711205221,191.75,8.92993561007021,199.0,180.575,199.0 245 | 243,0,4911.5,204.93888014049963,29.79479267563564,207.84191608426062,175.79946097919242,232.27271447804125,192.04999999999998,13.899999999999999,199.0,178.15,199.0 246 | 244,0,4931.5,199.81057581043325,37.73576123188319,196.91389733782236,168.5166725431456,232.3822961874323,194.775,8.450000000000001,199.0,186.325,199.0 247 | 245,0,4951.5,230.3933175836413,31.86075981690515,240.20048448523707,197.88741494202412,255.90660115964508,187.57500000000002,7.531315281252561,187.875,179.70000000000002,195.35 248 | 246,0,4971.5,238.4367684134047,38.32112768552098,239.96486753467428,201.47485533050184,271.5070196504332,188.875,20.25,199.0,168.625,199.0 249 | 247,0,4991.5,230.78303852921357,45.275870063935066,244.43126466517526,186.3380059208826,264.7827032081254,192.05,13.9,199.0,178.15,199.0 250 | 248,0,5011.5,233.90397511332742,23.354145355475207,242.2533054841949,208.8500728984049,254.3729486879818,178.12499999999997,25.973918104010398,177.875,153.60000000000002,199.0 251 | 249,0,5031.5,210.3013273078264,29.762242710071703,218.4045814969068,177.31810303539686,237.14084637344325,177.875,0.0,177.875,177.875,177.875 252 | 250,0,5051.5,174.0939869693394,42.54791719480106,170.54562129026476,134.8405353648255,216.81240474990562,189.22500000000002,19.55,199.0,169.67499999999998,199.0 253 | 251,0,5071.5,234.75453978320888,27.422476308089443,245.02012912580696,205.42733198278142,256.0964433967027,196.425,5.15,199.0,191.275,199.0 254 | 252,0,5091.5,224.141481308943,33.01435563172915,230.0153237148931,190.63832951895537,250.25230973623442,186.60000000000002,18.249094163258928,199.0,166.02499999999998,199.0 255 | 253,0,5111.5,216.46456161189974,47.0662625671784,235.7333923726883,164.72572081303622,250.76147234204132,179.25,25.846916437937146,194.125,149.325,199.0 256 | 254,0,5131.5,232.620397238932,35.73640050932521,244.60202243964397,193.58939469352464,262.01136264547455,177.5,24.05602579360365,185.375,154.175,193.55 257 | 255,0,5151.5,226.28531416994326,45.91426385647789,234.8738056817519,181.43377078997094,265.07444849835514,190.55,16.900000000000002,199.0,173.64999999999998,199.0 258 | 256,0,5171.5,237.04868838749172,26.695175087779088,237.97873719209827,212.6623738200244,261.7406928297692,194.775,8.450000000000001,199.0,186.325,199.0 259 | 257,0,5191.5,186.55958213774005,46.05499566694619,201.5397113461025,139.84376630579226,220.13903674484993,199.0,0.0,199.0,199.0,199.0 260 | 258,0,5211.5,196.73839330359326,34.185721597746934,204.24821191438022,163.2844234585312,227.787575852182,189.625,16.635887245741905,199.0,171.79999999999998,199.0 261 | 259,0,5231.5,209.13234526003242,44.256889779104185,208.14362500731048,173.18759109324668,246.47064957249785,195.02499999999998,7.95,199.0,187.075,199.0 262 | 260,0,5254.0,198.34603130042476,30.83056387139562,203.1958762381908,167.7472378862941,226.69010795434338,187.225,17.805680092863373,199.0,167.0,199.0 263 | 261,0,5274.0,190.95541428916334,32.58589558218799,201.1611236982851,156.1626861476277,217.66208322328623,199.0,0.0,199.0,199.0,199.0 264 | 262,0,5294.0,203.43941697920746,32.88335143224246,214.01574540623423,169.80258434667158,226.35560518072026,194.775,8.450000000000001,199.0,186.325,199.0 265 | 263,0,5314.0,228.29715109514473,26.76942012480022,238.29037546135842,198.23691177372012,249.82775762102762,195.5,7.0,199.0,188.5,199.0 266 | 264,0,5334.0,235.0086759972927,22.35462343956469,242.81497919877282,211.99499038355063,252.98795266409945,188.725,15.920190760616492,199.0,170.89999999999998,199.0 267 | 265,0,5354.0,227.8473259360839,45.61742997474228,232.80915289020734,185.44329550284732,264.73727915039825,199.0,0.0,199.0,199.0,199.0 268 | 266,0,5374.0,236.52360618423506,31.34964033371369,242.57209210183498,203.0341113576735,265.3820086044045,197.475,3.05,199.0,194.425,199.0 269 | 267,0,5394.0,234.80526354435062,16.20644200888431,241.70919946403302,216.42672907512735,248.25734160480414,190.15,15.947022402208157,199.0,173.37499999999997,199.0 270 | 268,0,5414.0,226.4711147209726,30.727082633219496,237.4557996781926,192.61147684333687,251.25990754510985,191.45000000000002,15.100000000000001,199.0,176.35,199.0 271 | 269,0,5434.0,229.56888581808968,29.84074571115017,240.09954926376102,197.58360135244266,256.93839157584387,187.85000000000002,21.530469738590245,199.0,165.87499999999997,199.0 272 | 270,0,5454.0,229.41413006853958,23.74232900139149,246.23169857275474,200.82072946665028,246.9876702858616,182.4,20.350517924621467,199.0,157.25,199.0 273 | 271,0,5474.0,213.66996659797593,31.100549462983754,226.13578499710056,178.950490030465,236.749622759772,190.55,16.900000000000002,199.0,173.64999999999998,199.0 274 | 272,0,5494.0,226.88842209655468,43.67301124227234,246.4105897298641,182.6466235988696,253.26764414229194,194.45,9.100000000000001,199.0,185.35,199.0 275 | 273,0,5514.0,230.0763030853808,28.95646366684469,240.69369014931067,200.28169516616336,249.96987164193177,199.0,0.0,199.0,199.0,199.0 276 | 274,0,5534.0,225.30635224956796,36.4122302089364,241.4939173804283,184.79100434847484,255.2776957705407,193.55,6.939290309534542,199.0,184.775,199.0 277 | 275,0,5554.0,187.22066532237082,44.52321760120209,192.8074986699949,143.99526081808938,228.3526391732555,190.55,16.900000000000002,199.0,173.64999999999998,199.0 278 | 276,0,5574.0,233.5174913055642,27.072820673771734,241.1327812835307,207.76863946575224,251.70353453044743,187.65,20.031578100293594,199.0,166.22500000000002,199.0 279 | 277,0,5594.0,214.2942941214165,46.52601033726482,242.34188136202033,158.75206998144415,250.05157114509615,182.45,20.37772462458293,199.0,157.0,199.0 280 | 278,0,5614.0,209.8404082692793,24.063389422691095,197.57326265864606,189.86449222471816,236.58386123331354,185.9,20.62993561007021,199.0,163.02499999999998,199.0 281 | 279,0,5634.0,216.70580220828796,45.08658385401516,243.8226784822123,165.3405721778911,251.25559925023074,194.8,5.144535936311456,199.0,188.475,199.0 282 | 280,0,5654.0,235.07040889472032,47.22116782929067,246.22693934165807,191.8191326214866,267.91760951611093,193.29999999999998,11.399999999999999,199.0,181.9,199.0 283 | 281,0,5674.0,176.75454070313873,32.84126597442862,187.71931353426433,141.36100981096644,206.41428979762503,199.0,0.0,199.0,199.0,199.0 284 | 282,0,5694.0,200.02515411063996,26.850688445666012,190.87794744540184,181.20093387111694,227.80665106270027,198.675,0.65,199.0,198.025,199.0 285 | 283,0,5714.0,184.92660856415867,32.008962716141674,178.62607063333147,157.97068058932356,216.70508195958666,180.775,24.18128665192716,199.0,151.39999999999998,199.0 286 | 284,0,5734.0,247.09587703926223,25.800569588729275,248.61753250373684,221.58321023406043,270.67848575618126,194.775,8.450000000000001,199.0,186.325,199.0 287 | 285,0,5756.5,231.6291297546426,50.72410230357465,256.08384554297436,173.68961184126016,269.8688250561765,165.3,24.148746293363565,165.0,142.20000000000002,186.075 288 | 286,0,5776.5,259.53523134112163,23.716479112225496,262.82628988888445,233.64663964168636,282.4502283230053,183.65,16.46879575100735,179.125,168.9,199.0 289 | 287,0,5796.5,250.28724415873376,30.72891929729174,250.5507868686933,217.90824472313997,280.8250295793897,179.375,25.196326624036548,179.375,155.875,199.0 290 | 288,0,5816.5,221.31443735331663,45.60459589927522,239.94769412863337,169.6858705902204,261.05039731458686,199.0,0.0,199.0,199.0,199.0 291 | 289,0,5836.5,249.46190787011355,35.33249364721472,254.10701575331848,213.37830400332237,281.3446904851789,186.025,25.950000000000003,199.0,160.075,199.0 292 | 290,0,5859.0,170.9500257129461,56.67584711369322,180.41848980432928,116.57032240996662,215.3230624503416,173.225,34.911923694136,199.0,131.95,199.0 293 | 291,0,5879.0,155.48144790906076,28.47080209421052,164.36967428780702,124.19988177662891,179.6294424662127,170.3,31.418658153638095,177.875,142.475,190.55 294 | 292,0,5899.0,235.3638172146566,18.71296430329687,236.67457184866657,215.97484247943027,253.74337244290825,164.1,36.598960831940374,177.125,129.15,190.25 295 | 293,0,5919.0,202.51700079401496,43.636417887580635,207.11512423729636,158.02189077198875,243.97592202030572,171.75,26.601368476443227,177.875,148.0,190.55 296 | 294,0,5939.0,233.09994899363983,26.35395534485754,241.8764607778975,205.24510822752273,251.61106499488258,195.975,6.05,199.0,189.925,199.0 297 | 295,0,5959.0,221.33543146520145,45.52311342091875,233.23350913009466,169.62423268087156,263.61673945324907,183.25,24.94909416325893,199.0,155.975,199.0 298 | 296,0,5979.0,230.732513440131,22.69093737174782,235.88869219960316,205.3803669333415,253.0078599343464,194.775,8.450000000000001,199.0,186.325,199.0 299 | 297,0,5999.0,262.5999673709411,14.145230279565116,264.9659628854379,248.76171775376028,275.88315840953516,173.25,25.44085236118309,177.875,151.375,190.55 300 | 298,0,6019.0,233.94939665544058,37.58214794862016,237.13380978962263,196.10997621812183,269.2145493187917,199.0,0.0,199.0,199.0,199.0 301 | 299,0,6039.0,253.9208961703153,41.37665711739255,260.7490497589249,208.35688849824473,294.39935932246203,182.25,33.5,199.0,148.75,199.0 302 | 300,0,6059.0,229.16504539660812,32.94654667032481,223.38579948522872,197.26365145543525,261.336911336994,187.6,19.142747680997402,199.0,166.925,199.0 303 | 301,0,6079.0,208.88845137407097,44.69320445069995,219.81576289577333,166.39290779788024,241.75264532447434,189.52499999999998,18.950000000000003,199.0,170.575,199.0 304 | 302,0,6099.0,181.27995777152995,25.73882634609849,179.07858910901976,157.7586964766955,205.6559815423173,194.20000000000002,9.600000000000001,199.0,184.6,199.0 305 | 303,0,6119.0,217.1283246900096,38.931077524088074,225.84678293783793,179.123157563611,248.675222578059,183.9,30.200000000000003,199.0,153.70000000000002,199.0 306 | 304,0,6139.0,264.40511596215947,26.04464740252765,269.02471667681704,237.66182524659973,288.8352984528688,190.55,16.900000000000002,199.0,173.64999999999998,199.0 307 | 305,0,6159.0,258.08191867019804,42.179240185554,270.1311202628817,210.4476764243942,296.0817613486556,184.05,23.349094163258926,199.0,158.375,199.0 308 | 306,0,6181.5,270.1776507442216,26.491910352912715,272.8862284184078,242.35021289473997,294.89041731330497,193.1,11.8,199.0,181.3,199.0 309 | 307,0,6201.5,277.108507056748,26.855357678261093,283.2720484549231,251.03278067979292,297.17972270929454,199.0,0.0,199.0,199.0,199.0 310 | 308,0,6221.5,225.9178829041885,50.332649168983934,239.00814196538846,178.45971147823968,262.1422783199757,178.17499999999998,41.650000000000006,199.0,136.525,199.0 311 | 309,0,6241.5,275.8992659386689,46.9893624156573,289.6651201236207,227.7147871243022,310.65261475446835,181.35,19.411466598954725,176.625,163.95,199.0 312 | 310,0,6261.5,242.89851090436832,31.12073414785436,252.68328040155282,210.08753075831862,266.8215573787001,190.55,10.349094163258927,199.0,177.875,199.0 313 | 311,0,6281.5,236.98029106991538,22.272081404102646,250.86560291788902,210.51643198943134,253.90454879653782,195.975,6.05,199.0,189.925,199.0 314 | 312,0,6301.5,266.3494342800475,45.93884493437095,271.04383969045807,219.39967821805115,308.11978731504763,187.82500000000002,22.35,199.0,165.475,199.0 315 | 313,0,6321.5,291.1617803233943,48.5380236064731,319.84974904296945,235.8303933718772,324.03280013476035,190.55,16.900000000000002,199.0,173.65,199.0 316 | 314,0,6341.5,286.4409182774866,41.53817557147788,294.15025877483157,243.5222816880338,322.67774269848337,177.20000000000002,17.529971546977436,181.125,159.15,194.35 317 | 315,0,6361.5,299.7860620612918,33.18728947285603,308.827158994543,263.2401112155904,326.9325391522116,180.625,33.11474403956757,199.0,145.64999999999998,199.0 318 | 316,0,6381.5,314.231589536751,38.23988799832034,321.71309505240214,276.63772243934204,346.385922469784,181.4,21.55662511347272,199.0,154.95,199.0 319 | 317,0,6401.5,308.4288917275217,33.134698408568845,323.93191218382844,271.9272305406427,331.6080338856082,199.0,0.0,199.0,199.0,199.0 320 | 318,0,6421.5,248.492517225538,42.784027577889375,250.95115237788409,206.1201979526096,289.17674700012947,161.42499999999998,33.20092601946772,175.875,127.15,184.775 321 | 319,0,6441.5,229.7177564816597,32.23154871608442,237.8063613883361,194.62385454265575,255.77587619900206,190.15,17.700000000000003,199.0,172.45,199.0 322 | 320,0,6461.5,191.7639059860178,62.8635238370679,184.79175863023843,136.4109659317788,248.8373533754865,173.8,28.496676434701786,177.875,144.675,199.0 323 | 321,0,6481.5,207.70019181987678,33.23621387483855,212.39097821743417,177.0634317980474,236.00144238493516,194.55,8.899999999999999,199.0,185.65,199.0 324 | 322,0,6501.5,258.09390764260024,53.59057588537188,279.5673652030372,204.25393386099745,290.41344320268263,188.425,21.150000000000002,199.0,167.275,199.0 325 | 323,0,6521.5,261.2610110920106,26.048524202319015,267.7598051355434,234.7024553882064,283.6310926334313,196.25,4.513175157247943,199.0,191.175,199.0 326 | 324,0,6541.5,265.4030019800007,25.965019032597166,271.36041527122495,237.7715776548567,287.1356421140254,194.775,8.450000000000001,199.0,186.325,199.0 327 | 325,0,6561.5,234.72457044653862,43.528656640737154,238.8463338873509,192.00014512632373,273.2964927764721,179.025,33.088992111815564,199.0,143.5,199.0 328 | 326,0,6581.5,256.0960432109326,32.25588803413767,252.22939617875306,230.38959401295415,286.3189045614737,195.675,6.65,199.0,189.025,199.0 329 | 327,0,6601.5,218.05437969708817,33.90792499632451,222.1991477754306,187.08596747212226,246.11224264979978,195.1,7.8,199.0,187.3,199.0 330 | 328,0,6621.5,224.34654058978546,34.770817170198555,219.46214973369064,190.95266022436843,260.7077156338591,180.775,3.829490827773322,179.375,178.325,184.775 331 | 329,0,6641.5,222.83791661673243,15.974701861289992,229.59413425609023,206.20921043944998,234.07605449810052,199.0,0.0,199.0,199.0,199.0 332 | 330,0,6661.5,216.29399758390772,31.262078861739802,229.46048770859156,181.4436030472213,241.44308883048876,190.55,16.900000000000002,199.0,173.64999999999998,199.0 333 | 331,0,6681.5,229.91579480568888,32.043701917531315,244.02465679989103,193.1958727795327,255.64864877991994,186.225,25.550000000000004,199.0,160.675,199.0 334 | 332,0,6701.5,234.99566794853655,37.53823871470954,244.04229102704727,200.62342867240645,262.4007494900879,195.375,7.25,199.0,188.125,199.0 335 | 333,0,6721.5,211.2695986771484,46.02926946633912,219.95919511313792,165.43380778579197,251.27661652605306,185.75,25.138097474538647,199.0,159.82500000000002,199.0 336 | 334,0,6741.5,214.73693610386061,26.93026960378936,227.79435202156475,184.93065846701245,236.12746324619647,194.775,8.450000000000001,199.0,186.325,199.0 337 | 335,0,6764.0,216.2366163166731,53.924295012324656,225.22016763310762,164.47457075298843,259.6567292382537,183.325,28.33581679282758,199.0,153.35,199.0 338 | 336,0,6784.0,228.12724773322574,41.12675277359824,224.876308414972,192.81583793382163,268.1849125308155,188.12500000000003,21.75,199.0,166.375,199.0 339 | 337,0,6804.0,231.41281588713193,41.56293367922428,232.52655957481034,192.83168295437937,266.69839597410737,163.675,42.900000000000006,175.875,125.4,189.75 340 | 338,0,6824.0,227.08334613451126,43.89467642375982,230.26715108177422,187.8618026090328,265.3055080285037,187.125,16.977010301104404,199.0,167.6,199.0 341 | 339,0,6844.0,209.30293713879453,34.516732334471556,215.36964717977384,175.87130073130464,236.77207360059876,186.475,25.050000000000004,199.0,161.425,199.0 342 | 340,0,6864.0,178.8215802073249,23.851027378913688,179.20822158862546,158.65510437583043,199.43415700772456,173.82500000000002,21.94453667295492,179.375,153.475,191.15 343 | 341,0,6884.0,288.0125163076656,20.9083055374223,292.95980726171837,267.35428056975337,303.81767968116145,179.32500000000002,27.282782360184534,199.0,147.125,199.0 344 | 342,0,6904.0,194.54809974168148,33.7943066734809,199.02815215105056,157.71334962983818,225.7815837444341,188.225,9.799681117260908,188.875,176.975,199.0 345 | 343,0,6924.0,198.23939566791626,34.788313683889385,199.35540621979527,167.80342236538164,226.81767227069022,199.0,0.0,199.0,199.0,199.0 346 | 344,0,6944.0,185.90044104139832,39.81764814384009,198.10793528533947,144.5416935589624,217.14174324098715,181.65,28.192485265806255,199.0,150.875,199.0 347 | 345,0,6964.0,171.50094849846812,25.17504253246079,166.34727742281444,153.4754545254878,195.71194935728414,195.1,7.8,199.0,187.3,199.0 348 | 346,0,6984.0,171.74635156111873,30.0937709358336,169.67142090696282,146.6374858546759,198.69188150737867,190.0,18.000000000000004,199.0,172.00000000000003,199.0 349 | 347,0,7004.0,188.9171263510579,27.94231450838567,195.52446150052538,157.39345318092847,214.4772899487772,191.35,15.3,199.0,176.04999999999998,199.0 350 | 348,0,7024.0,234.3887784020403,32.57464426297015,225.58032988930157,208.32152472581646,268.0192954546828,182.675,21.020575718506663,183.875,162.32500000000002,199.0 351 | 349,0,7044.0,204.94069840211495,31.409921825166506,214.052328000625,171.9163705549231,231.81783398929008,142.4,29.959642418181044,154.625,110.60000000000001,162.55 352 | 350,0,7064.0,175.07172677477217,35.47458936452333,183.30386976729312,139.84147152661552,202.6190045694702,192.75,12.5,199.0,180.25,199.0 353 | 351,0,7084.0,194.17405558484487,34.7660006029081,188.75893578105538,167.9253200372328,225.75144006513992,199.0,0.0,199.0,199.0,199.0 354 | 352,0,7104.0,141.95698420762795,55.7091440515208,150.2357120431809,87.13845750705515,190.85101868167305,169.525,29.80219653059501,175.0,140.95,192.15 355 | 353,0,7124.0,200.43012727697726,36.13856615174777,215.73408564009972,157.5994617325374,231.62558724466203,192.95,7.491912305946994,199.0,183.525,199.0 356 | 354,0,7144.0,214.1774719235243,27.235806979797516,218.82302061100012,186.1574882898963,237.77771694017596,193.0,12.0,199.0,181.0,199.0 357 | 355,0,7164.0,185.97145183264143,29.23864709617421,186.43858744565705,159.58451561819203,211.24820637793886,197.475,3.05,199.0,194.425,199.0 358 | 356,0,7184.0,188.15758393140234,25.14179859707741,185.4422198438458,162.61324297637728,215.41119108505018,171.95,41.50267757678295,192.875,126.97500000000001,199.0 359 | 357,0,7204.0,168.65275864959023,43.935098567158725,175.4469956516241,125.90873578635083,208.3519101446897,189.125,15.37161469600844,199.0,172.14999999999998,199.0 360 | 358,0,7224.0,178.20294081845714,30.83445032012318,177.47402248683093,153.2127110602313,204.94514465076747,194.925,8.149999999999999,199.0,186.775,199.0 361 | 359,0,7244.0,210.8486174229467,49.191266181707206,213.09046093864046,161.71007935025045,259.0587608173688,194.775,8.450000000000001,199.0,186.325,199.0 362 | 360,0,7264.0,182.14465611817195,43.808343545894665,188.01649740502384,139.0797517495925,218.30008065111903,174.25,42.25136454751136,199.0,129.425,199.0 363 | 361,0,7284.0,225.4778237102092,31.167534849318688,232.90683851324206,193.78098789463766,250.53429399366422,182.37499999999997,15.87226947939466,181.875,165.125,199.0 364 | 362,0,7304.0,224.1915055245756,53.05795489862995,256.1222498252783,165.21002371875068,257.87567692935596,194.775,8.450000000000001,199.0,186.325,199.0 365 | 363,0,7324.0,190.48827650793737,25.784293414676878,196.25393050166076,164.4901321579712,212.70625158538033,184.5,16.910718054633428,183.875,168.2,199.0 366 | 364,0,7344.0,193.1629035152577,40.4531891466457,201.4716891343379,151.45827111829686,228.54432019806256,194.775,8.450000000000001,199.0,186.325,199.0 367 | 365,0,7364.0,164.63606387407197,27.238685018085768,170.41382624600877,137.8265483594642,185.30279013247235,187.5,19.39438291935636,199.0,166.575,199.0 368 | 366,0,7384.0,154.95758728637466,47.39549548454863,159.7609536252109,110.26088252340901,194.80547458003196,184.025,24.696393396949645,197.125,157.375,199.0 369 | 367,0,7404.0,143.13618615682236,41.42728148454168,144.5013954346123,100.49201975793216,182.99840488774666,188.125,17.077499637786143,199.0,168.79999999999998,199.0 370 | 368,0,7424.0,194.096330197039,30.1297379844406,202.29157637716324,159.41257970001112,222.2096265041714,171.82500000000002,30.070806610183503,179.375,145.14999999999998,191.15 371 | 369,0,7446.5,210.33780606931998,34.402366690053576,223.85575459444073,171.73997151227798,239.59004440549603,194.775,8.450000000000001,199.0,186.325,199.0 372 | 370,0,7466.5,200.10363067857273,31.508917097985226,204.93942780756117,167.30672478957743,227.42981645227457,185.55,20.835027244382864,199.0,162.375,199.0 373 | 371,0,7486.5,232.15983999656677,51.326621240565956,257.18741186462944,176.97705636791966,265.1147072459244,177.625,28.292935004291785,179.375,150.72500000000002,199.0 374 | 372,0,7506.5,227.4232120497224,52.05667212012878,250.5375887252341,171.95251470668904,263.5392867468395,172.225,11.8,178.125,160.425,178.125 375 | 373,0,7526.5,243.389356958585,32.72077676895537,255.96340258220422,207.1481045414324,271.1598382827155,172.575,37.92664436125074,199.0,129.0,199.0 376 | 374,0,7546.5,245.65679474620762,62.28364457068872,268.59838457654917,180.8722845299595,286.99710179357595,172.89999999999998,25.956353651475304,176.375,150.57500000000002,191.15 377 | 375,0,7566.5,241.06043636589118,26.64175689410493,252.22891163481046,212.71105682303659,259.59230434764686,192.85,12.3,199.0,180.54999999999998,199.0 378 | 376,0,7586.5,241.2265355255562,33.87273416817867,249.10462016428204,204.46810773579224,269.9354354911768,176.4,24.492315719990405,185.375,152.075,193.55 379 | 377,0,7606.5,200.53421963372608,40.646096167712514,212.21296007733937,157.89603899622253,233.41599313539513,183.95000000000002,23.54909416325893,199.0,158.075,199.0 380 | 378,0,7626.5,298.9839475444882,26.82719162567481,298.1483115369844,276.0729920606518,322.9971472419428,189.075,19.85,199.0,169.225,199.0 381 | 379,0,7646.5,252.8144694592495,35.28903443158337,268.7585576624337,211.7253943831995,283.42394815748577,172.75,9.65,176.375,163.7,178.175 382 | 380,0,7666.5,293.8843274813954,33.91756778779875,305.78183914245375,256.34860731655647,321.7996521325884,186.32500000000002,25.35,199.0,160.97500000000002,199.0 383 | 381,0,7686.5,284.8774905266173,27.122626667062377,295.5040459679917,252.70375409762227,309.00871506460186,176.975,27.983941086093886,176.375,151.02499999999998,199.0 384 | 382,0,7706.5,322.1865617849465,26.915351310554048,335.59717065023335,294.02039699758205,337.6267062917361,193.79999999999998,10.4,199.0,183.4,199.0 385 | 383,0,7726.5,315.47725935934074,35.18959215587353,329.3065050270595,275.56110683241104,344.2424207805937,192.5,13.0,199.0,179.5,199.0 386 | 384,0,7746.5,314.50473906151257,45.14623628634473,330.80386120672125,265.95778122131,345.9813270623205,188.775,20.45,199.0,168.325,199.0 387 | 385,0,7766.5,270.4654098557579,37.93746760788832,289.22776858570535,228.36573129334244,297.1138017210781,187.47500000000002,23.05,199.0,164.425,199.0 388 | 386,0,7786.5,251.52310519449702,37.86302231233162,258.31112399135725,210.0668672528185,287.46387824246995,165.525,24.1,176.375,142.025,178.175 389 | 387,0,7806.5,252.05565630483616,36.42321244620109,266.8556052404728,212.80345218619033,279.20230168347575,180.025,14.399999999999999,179.375,169.55,191.15 390 | 388,0,7826.5,259.49390625808326,22.361119417029737,271.7495604073607,234.66444324314332,274.01325373925675,192.65,12.700000000000001,199.0,179.95,199.0 391 | 389,0,7846.5,266.0432009053888,35.335658365666276,273.87277247740224,225.49792362559853,300.6238879568868,183.6,25.229935610070214,199.0,156.125,199.0 392 | 390,0,7869.0,222.65302615584852,52.563926946151255,231.05017168483207,169.11188126842308,264.9427128001166,186.05,19.99968111726091,199.0,163.475,199.0 393 | 391,0,7889.0,224.33880722479563,43.03440881190071,236.58364813011275,178.83773031887662,262.30079115885894,185.225,23.081541950298412,199.0,159.89999999999998,199.0 394 | 392,0,7909.0,249.30279310134512,35.23417775271811,255.56211833630556,211.50935174293744,280.2558862252829,177.275,17.276805764261496,176.375,163.7,191.15 395 | 393,0,7929.0,230.8136606367921,48.59295746893807,244.9439811384147,180.70957731307163,270.63746172070904,194.775,8.450000000000001,199.0,186.325,199.0 396 | 394,0,7949.0,245.94114457084868,45.72834998483253,246.31178546750186,200.82244761729334,289.7709725784969,181.5,35.00000000000001,199.0,146.5,199.0 397 | 395,0,7969.0,217.34097606313688,41.28891786576846,234.80378519306777,168.64027372522156,253.48932335362295,179.57500000000002,32.1854757842254,199.0,144.95,199.0 398 | 396,0,7989.0,219.48800397090457,44.522875062535455,236.0376971970564,170.89744798181027,255.22398309265503,169.65,34.332405480781425,177.875,138.72500000000002,191.75 399 | 397,0,8009.0,272.1787479647576,28.72350893907229,283.59729628233174,242.05648280590515,294.10815543536796,189.225,19.55,199.0,169.675,199.0 400 | 398,0,8027.571428571428,275.7664573223853,45.32019670924672,292.07188766299356,222.41974242476334,314.3826297719706,169.4,22.157000660077138,174.85714285714286,148.82857142857142,189.34285714285716 401 | 399,0,8049.0,370.2139443395956,47.51664521386739,398.1181799063419,315.37424753938285,402.58675164919475,182.8,19.871713564763358,199.0,158.15,199.0 402 | 400,0,8069.0,385.3070425497325,16.119674656312935,387.7732255849852,368.0592744799848,400.89702426669726,194.1,8.271336046854824,199.0,184.95,199.0 403 | 401,0,8089.0,368.1919463989282,48.38337096020878,393.10745866136045,311.69850908650204,404.227987013076,174.64999999999998,36.37601088518899,199.0,133.8,199.0 404 | 402,0,8109.0,372.54829419168465,31.054554465229277,384.3122104812452,338.01719157373805,397.80305243706164,190.55,16.900000000000002,199.0,173.65,199.0 405 | 403,0,8129.0,373.5298809547862,30.02843931347497,385.3657197780957,338.8841107190908,403.06477372723555,199.0,0.0,199.0,199.0,199.0 406 | -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/movies/40000_MLP-DQN.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/movies/40000_MLP-DQN.mp4 -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/movies/500000_MLP-DQN.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/movies/500000_MLP-DQN.mp4 -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/movies/5000_MLP-DQN.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/movies/5000_MLP-DQN.mp4 -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/movies/DQN-Gridworld.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/movies/DQN-Gridworld.gif -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/movies/DQN-Gridworld.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_DQL/movies/DQN-Gridworld.mp4 -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/requirements.txt: -------------------------------------------------------------------------------- 1 | bayesian-optimization==1.0.1 2 | cycler==0.10.0 3 | future==0.17.1 4 | gym==0.12.5 5 | kiwisolver==1.1.0 6 | matplotlib==3.1.0 7 | numpy==1.16.4 8 | pandas==0.24.2 9 | pycolab==1.2 10 | pyglet==1.3.2 11 | pyparsing==2.4.0 12 | python-dateutil==2.8.0 13 | pytz==2019.1 14 | scipy==1.3.0 15 | six==1.12.0 16 | torch==1.1.0 17 | -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/run_experiments_dqn.sh: -------------------------------------------------------------------------------- 1 | if [[ "$1" == "dqn" ]] 2 | then 3 | echo "Run DQN Experiments" 4 | ############################################################################### 5 | # DQN EXPERIMENTS 6 | ############################################################################### 7 | # DQN - ER Capacity Experiments 8 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -capacity 5000 -fname dqn_capacity_5000 -n_upd 1500000 9 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -capacity 20000 -fname dqn_capacity_20000 -n_upd 1500000 10 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -capacity 40000 -fname dqn_capacity_40000 -n_upd 1500000 11 | # DQN - Batch Size Experiments 12 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -train_batch 32 -fname dqn_batchsize_32 -n_upd 1500000 13 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -train_batch 64 -fname dqn_batchsize_64 -n_upd 1500000 14 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -train_batch 128 -fname dqn_batchsize_128 -n_upd 1500000 15 | # DQN - Learning Rate Experiments 16 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -l_r 0.01 -fname dqn_lrate_1e3 -n_upd 1500000 17 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -l_r 0.001 -fname dqn_lrate_1e4 -n_upd 1500000 18 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -l_r 0.0001 -fname dqn_lrate_1e5 -n_upd 1500000 19 | # DQN - Hard vs Soft Updates 20 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -update_upd 1 -fname dqn_hard_update_1 -n_upd 1500000 21 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -update_upd 500 -fname dqn_hard_update_100 -n_upd 1500000 22 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -soft_tau 0.01 -fname dqn_soft_update_001 -n_upd 1500000 23 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -soft_tau 0.05 -fname dqn_soft_update_005 -n_upd 1500000 24 | elif [[ "$1" == "double-dqn" ]] 25 | then 26 | echo "Run Double DQN Experiments" 27 | ############################################################################### 28 | # DOUBLE DQN EXPERIMENTS 29 | ############################################################################### 30 | # DDQN - Updates of Target Networks 31 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --DOUBLE -soft_tau 0.01 -fname ddqn_soft_update_001 -n_upd 1500000 32 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --DOUBLE -soft_tau 0.05 -fname ddqn_soft_update_005 -n_upd 1500000 33 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --DOUBLE -soft_tau 0.1 -fname ddqn_soft_update_01 -n_upd 1500000 34 | # DDQN - Discounting Future Bias 35 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --DOUBLE -gamma 0.9 -fname ddqn_gamma_09 -n_upd 1500000 36 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --DOUBLE -gamma 0.95 -fname ddqn_gamma_095 -n_upd 1500000 37 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --DOUBLE -gamma 0.99 -fname ddqn_gamma_099 -n_upd 1500000 38 | elif [[ "$1" == "per-dqn" ]] 39 | then 40 | echo "Run Prioritized Experience Replay Experiments" 41 | ############################################################################### 42 | # PRIORITIZED ER DQN EXPERIMENTS 43 | ############################################################################### 44 | # PER - Priority Distribution Temperature 45 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --PER --ALPHA 0.2 -fname per_dqn_alpha_02 -n_upd 1500000 46 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --PER --ALPHA 0.5 -fname per_dqn_alpha_05 -n_upd 1500000 47 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --PER --ALPHA 0.8 -fname per_dqn_alpha_08 -n_upd 1500000 48 | # PER - Importance Sampling Start Beta 49 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --PER --BETA_START 0.2 -fname per_dqn_beta_02 -n_upd 1500000 50 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --PER --BETA_START 0.5 -fname per_dqn_beta_05 -n_upd 1500000 51 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --PER --BETA_START 0.8 -fname per_dqn_beta_08 -n_upd 1500000 52 | # PER - Buffer Capacity 53 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --PER -capacity 5000 -fname per_dqn_capacity_5000 -n_upd 1500000 54 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --PER -capacity 20000 -fname per_dqn_capacity_20000 -n_upd 1500000 55 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 --PER -capacity 40000 -fname per_dqn_capacity_40000 -n_upd 1500000 56 | elif [[ "$1" == "dueling-dqn" ]] 57 | then 58 | echo "Run Dueling DQN Experiments" 59 | ############################################################################### 60 | # DUELING DQN EXPERIMENTS 61 | ############################################################################### 62 | # Dueling DQN - Network Capacity 63 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -agent MLP-DUELING --HIDDEN_SIZE 64 -fname dueling_dqn_64 64 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -agent MLP-DUELING --HIDDEN_SIZE 128 -fname dueling_dqn_128 65 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -agent MLP-DUELING --HIDDEN_SIZE 256 -fname dueling_dqn_256 66 | # Dueling DQN - Combining the improvements 67 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -agent MLP-DUELING -fname dueling_dqn 68 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -agent MLP-DUELING --DOUBLE -fname double_dueling_dqn 69 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -agent MLP-DUELING --PER -fname per_dueling_dqn 70 | CUDA_VISIBLE_DEVICES=$2 python train_dqn.py -s -n_runs 5 -agent MLP-DUELING --DOUBLE --PER -fname per_double_dueling_dqn 71 | else 72 | echo "Provide valid argument to bash script" 73 | fi 74 | -------------------------------------------------------------------------------- /EXPERIMENTS_DQL/train_dqn.py: -------------------------------------------------------------------------------- 1 | import gym 2 | import time 3 | import numpy as np 4 | import pandas as pd 5 | import gridworld 6 | 7 | import torch 8 | import torch.autograd as autograd 9 | import torch.multiprocessing as mp 10 | 11 | from dqn import MLP_DQN, MLP_DuelingDQN, init_dqn 12 | from dqn_helpers import command_line_dqn, compute_td_loss 13 | from general_helpers import epsilon_by_update, beta_by_update 14 | from general_helpers import update_target, polyak_update_target 15 | from general_helpers import ReplayBuffer, NaivePrioritizedBuffer 16 | from general_helpers import get_logging_stats, run_multiple_times 17 | 18 | torch.manual_seed(0) 19 | np.random.seed(0) 20 | 21 | def run_dqn_learning(args): 22 | log_template = "Step {:>2} | T {:.1f} | Median R {:.1f} | Mean R {:.1f} | Median S {:.1f} | Mean S {:.1f}" 23 | 24 | # Set the GPU device on which to run the agent 25 | USE_CUDA = torch.cuda.is_available() 26 | if USE_CUDA: 27 | torch.cuda.set_device(args.device_id) 28 | else: 29 | print("USING CPU") 30 | Variable = lambda *args, **kwargs: autograd.Variable(*args, **kwargs).cuda() if USE_CUDA else autograd.Variable(*args, **kwargs) 31 | start = time.time() 32 | 33 | if args.DOUBLE: TRAIN_DOUBLE = True 34 | else: TRAIN_DOUBLE = False 35 | 36 | # Setup agent, replay replay_buffer, logging stats df 37 | if args.AGENT == "MLP-DQN": 38 | agents, optimizer = init_dqn(MLP_DQN, args.L_RATE, USE_CUDA, 39 | args.INPUT_DIM, args.HIDDEN_SIZE, 40 | args.NUM_ACTIONS) 41 | elif args.AGENT == "MLP-DUELING": 42 | agents, optimizer = init_dqn(MLP_DuelingDQN, args.L_RATE, USE_CUDA, 43 | args.INPUT_DIM, args.HIDDEN_SIZE, 44 | args.NUM_ACTIONS) 45 | 46 | if not args.PER: 47 | replay_buffer = ReplayBuffer(capacity=args.CAPACITY) 48 | else: 49 | replay_buffer = NaivePrioritizedBuffer(capacity=args.CAPACITY, 50 | prob_alpha=args.ALPHA) 51 | 52 | reward_stats = pd.DataFrame(columns=["opt_counter", "rew_mean", "rew_sd", 53 | "rew_median", "rew_10th_p", "rew_90th_p"]) 54 | 55 | step_stats = pd.DataFrame(columns=["opt_counter", "steps_mean", "steps_sd", 56 | "steps_median", "steps_10th_p", "steps_90th_p"]) 57 | 58 | # Initialize optimization update counter and environment 59 | opt_counter = 0 60 | env = gym.make("dense-v0") 61 | ep_id = 0 62 | # RUN TRAINING LOOP OVER EPISODES 63 | while opt_counter < args.NUM_UPDATES: 64 | obs = env.reset() 65 | steps = 0 66 | 67 | while steps < args.MAX_STEPS: 68 | epsilon = epsilon_by_update(opt_counter + 1, args.EPS_START, 69 | args.EPS_STOP, args.NUM_UPDATES) 70 | if args.PER: 71 | beta = beta_by_update(opt_counter + 1, args.BETA_START, 72 | args.BETA_STEPS) 73 | else: 74 | beta = None 75 | 76 | action = agents["current"].act(obs.flatten(), epsilon) 77 | next_obs, rew, done, _ = env.step(action) 78 | steps += 1 79 | 80 | # Push transition to ER Buffer 81 | replay_buffer.push(ep_id, steps, obs, action, 82 | rew, next_obs, done) 83 | 84 | if len(replay_buffer) > args.TRAIN_BATCH_SIZE: 85 | opt_counter += 1 86 | loss = compute_td_loss(agents, optimizer, replay_buffer, beta, 87 | args.TRAIN_BATCH_SIZE, args.GAMMA, Variable, 88 | TRAIN_DOUBLE) 89 | 90 | if args.SOFT_TAU != 0: 91 | polyak_update_target(agents["current"], agents["target"], 92 | args.SOFT_TAU) 93 | 94 | 95 | # Go to next episode if current one terminated or update obs 96 | if done: break 97 | else: obs = next_obs 98 | 99 | # On-Policy Rollout for Performance evaluation 100 | if (opt_counter+1) % args.ROLLOUT_EVERY == 0: 101 | r_stats, s_stats = get_logging_stats(opt_counter, agents, 102 | args.GAMMA, args.NUM_ROLLOUTS, 103 | args.MAX_STEPS, args.AGENT) 104 | reward_stats = pd.concat([reward_stats, r_stats], axis=0) 105 | step_stats = pd.concat([step_stats, s_stats], axis=0) 106 | 107 | if args.SOFT_TAU == 0 and (opt_counter+1) % args.UPDATE_EVERY == 0: 108 | update_target(agents["current"], agents["target"]) 109 | 110 | if args.VERBOSE and (opt_counter+1) % args.PRINT_EVERY == 0: 111 | stop = time.time() 112 | print(log_template.format(opt_counter+1, stop-start, 113 | r_stats.loc[0, "rew_median"], 114 | r_stats.loc[0, "rew_mean"], 115 | s_stats.loc[0, "steps_median"], 116 | s_stats.loc[0, "steps_mean"])) 117 | start = time.time() 118 | 119 | if args.SAVE_AGENT and opt_counter+1 in [10000, 100000, 500000, 1000000]: 120 | torch.save(agents["current"].state_dict(), "agents/" + str(opt_counter+1) + "_" + args.AGENT) 121 | print("Saved agent after {} SGD updates".format(opt_counter+1)) 122 | ep_id += 1 123 | 124 | # Save the logging dataframe 125 | df_to_save = pd.concat([reward_stats, step_stats], axis=1) 126 | df_to_save = df_to_save.loc[:,~df_to_save.columns.duplicated()] 127 | df_to_save = df_to_save.reset_index() 128 | 129 | return df_to_save 130 | 131 | 132 | if __name__ == "__main__": 133 | mp.set_start_method('forkserver') 134 | args = command_line_dqn() 135 | 136 | if args.RUN_TIMES == 1: 137 | print("START RUNNING {} AGENT LEARNING FOR 1 TIME".format(args.AGENT)) 138 | run_dqn_learning(args) 139 | else: 140 | start_t = time.time() 141 | run_multiple_times(args, run_dqn_learning) 142 | print("Done - {} experiment after {:.2f} secs".format(args.SAVE_FNAME, 143 | time.time() - start_t)) 144 | -------------------------------------------------------------------------------- /EXPERIMENTS_PG/configs/base_a2c.json: -------------------------------------------------------------------------------- 1 | { 2 | "train_config": {"agent_name": "A2C", # Class of agents to train 3 | "seed_id": 0, # Training seed for init 4 | "num_train_updates": 20000, # Batch updates to train with 5 | "env_name": "Pendulum-v0", # Name of training environment 6 | "device_name": "cpu", # Device on which to run sim 7 | # Optimization-spec. hyperparameters 8 | "loss_type": "a2c", 9 | "opt_type": {"policy": "Adam", 10 | "value": "Adam"}, 11 | "l_rate": {"policy": 1e-5, 12 | "value": 1e-5}, 13 | "batch_size": 32, # Divides by threads for sync 14 | "num_threads": 16, # Threads 15 | "steps_in_trajectory": 20, 16 | "optimization_steps_in_mini_epoch": 1, # Updates per batch 17 | # A2C-spec. hyperparams (policy type) 18 | "return_type": "advantage-critic", # vanilla/gae 19 | "shared_torso": 0, 20 | "entropy_beta": 0.02, 21 | "clip_action_to_range": 1, 22 | "policy_type": "gaussian", 23 | # MDP-spec. hyperparams (Steps, Discount) 24 | "max_steps_in_episode": 200, 25 | "train_discount_factor": 1, 26 | "test_discount_factor": 1, 27 | # Logging-spec. hyperparams (When/how much) 28 | "evaluate_every_optimization_steps": 100 29 | }, 30 | "log_config": {"time_to_track": ["ep_counter", "optim_counter", 31 | "step_counter", "t_since_last_opt"], 32 | "what_to_track": ["rew_mean", "rew_sd", "rew_median", 33 | "rew_10th_p", "rew_90th_p", 34 | "steps_mean", "steps_sd", "steps_median", 35 | "steps_10th_p", "steps_90th_p", 36 | "success_rate", "loss"], 37 | "experiment_dir": "experiments/", 38 | "seed_id": 0, 39 | "time_to_print": ["optim_counter", "t_since_last_opt"], 40 | "what_to_print": ["rew_mean", "rew_median", 41 | "steps_mean", "steps_median", "success_rate"]}, 42 | "net_config": {# Specify the actor architecture 43 | "policy": {"input_dim": [1, 3], 44 | "layers_info": [["flatten"], 45 | ["linear", 32, 1], 46 | ["linear", 32, 1], 47 | ["linear", 1, 1]], 48 | "output_act": "identity", 49 | "hidden_act": "relu", 50 | "learn_constant": 1 # whether to learn log std 51 | }, 52 | # Specify the critic architecture 53 | "value": {"input_dim": [1, 3], 54 | "layers_info": [["flatten"], 55 | ["linear", 32, 1], 56 | ["linear", 32, 1], 57 | ["linear", 1, 1]], 58 | "output_act": "identity", 59 | "hidden_act": "relu", 60 | "dropout": 0.0, 61 | "batch_norm": 0}} 62 | } 63 | -------------------------------------------------------------------------------- /EXPERIMENTS_PG/configs/base_ddpg.json: -------------------------------------------------------------------------------- 1 | { 2 | "train_config": {"agent_name": "A2C", # Class of agents to train 3 | "seed_id": 0, # Training seed for init 4 | "num_train_updates": 20000, # Batch updates to train with 5 | "env_name": "Pendulum-v0", # Name of training environment 6 | "device_name": "cpu", # Device on which to run sim 7 | # Optimization-spec. hyperparameters 8 | "loss_type": "a2c", 9 | "opt_type": {"policy": "Adam", 10 | "value": "Adam"}, 11 | "l_rate": {"policy": 1e-5, 12 | "value": 1e-5}, 13 | "batch_size": 32, # Divides by threads for sync 14 | "num_threads": 16, # Threads 15 | "steps_in_trajectory": 20, 16 | "optimization_steps_in_mini_epoch": 1, # Updates per batch 17 | # A2C-spec. hyperparams (policy type) 18 | "return_type": "advantage-critic", # vanilla/gae 19 | "shared_torso": 0, 20 | "entropy_beta": 0.02, 21 | "clip_action_to_range": 1, 22 | "policy_type": "gaussian", 23 | # MDP-spec. hyperparams (Steps, Discount) 24 | "max_steps_in_episode": 200, 25 | "train_discount_factor": 1, 26 | "test_discount_factor": 1, 27 | # Logging-spec. hyperparams (When/how much) 28 | "evaluate_every_optimization_steps": 100 29 | }, 30 | "log_config": {"time_to_track": ["ep_counter", "optim_counter", 31 | "step_counter", "t_since_last_opt"], 32 | "what_to_track": ["rew_mean", "rew_sd", "rew_median", 33 | "rew_10th_p", "rew_90th_p", 34 | "steps_mean", "steps_sd", "steps_median", 35 | "steps_10th_p", "steps_90th_p", 36 | "success_rate", "loss"], 37 | "experiment_dir": "experiments/", 38 | "seed_id": 0, 39 | "time_to_print": ["optim_counter", "t_since_last_opt"], 40 | "what_to_print": ["rew_mean", "rew_median", 41 | "steps_mean", "steps_median", "success_rate"]}, 42 | "net_config": {# Specify the actor architecture 43 | "policy": {"input_dim": [1, 3], 44 | "layers_info": [["flatten"], 45 | ["linear", 32, 1], 46 | ["linear", 32, 1], 47 | ["linear", 1, 1]], 48 | "output_act": "identity", 49 | "hidden_act": "relu", 50 | "learn_constant": 1 # whether to learn log std 51 | }, 52 | # Specify the critic architecture 53 | "value": {"input_dim": [1, 3], 54 | "layers_info": [["flatten"], 55 | ["linear", 32, 1], 56 | ["linear", 32, 1], 57 | ["linear", 1, 1]], 58 | "output_act": "identity", 59 | "hidden_act": "relu", 60 | "dropout": 0.0, 61 | "batch_norm": 0}} 62 | } 63 | -------------------------------------------------------------------------------- /EXPERIMENTS_PG/configs/base_vpg.json: -------------------------------------------------------------------------------- 1 | { 2 | "train_config": {"agent_name": "VPG", # Class of agents to train 3 | "seed_id": 0, # Training seed for init 4 | "num_train_updates": 10000, # Batch updates to train with 5 | "env_name": "Pendulum-v0", # Name of training environment 6 | "device_name": "cpu", # Device on which to run sim 7 | # Optimization-spec. hyperparameters 8 | "loss_type": "vpg", 9 | "opt_type": "Adam", 10 | "l_rate": 1e-4, 11 | "batch_size": 16, # Batchsize - divides by threads 12 | "num_threads": 16, # Threads 13 | "optimization_steps_in_mini_epoch": 1, # Updates per batch 14 | # VPG-spec. hyperparams (policy type) 15 | "entropy_beta": 0.02, 16 | "clip_action_to_range": 1, 17 | "policy_type": "gaussian", 18 | # MDP-spec. hyperparams (Steps, Discount) 19 | "max_steps_in_episode": 200, 20 | "train_discount_factor": 1, 21 | "test_discount_factor": 1, 22 | # Logging-spec. hyperparams (When/how much) 23 | "evaluate_every_optimization_steps": 100 24 | }, 25 | "log_config": {"time_to_track": ["ep_counter", "optim_counter", 26 | "step_counter", "t_since_last_opt"], 27 | "what_to_track": ["rew_mean", "rew_sd", "rew_median", 28 | "rew_10th_p", "rew_90th_p", 29 | "steps_mean", "steps_sd", "steps_median", 30 | "steps_10th_p", "steps_90th_p", 31 | "success_rate", "loss"], 32 | "experiment_dir": "experiments/", 33 | "seed_id": 0, 34 | "time_to_print": ["optim_counter", "t_since_last_opt"], 35 | "what_to_print": ["rew_mean", "rew_median", 36 | "steps_mean", "steps_median", "success_rate"]}, 37 | "net_config": {"input_dim": [1, 3], 38 | "layers_info": [["flatten"], 39 | ["linear", 32, 1], 40 | ["linear", 32, 1], 41 | ["linear", 1, 1]], 42 | "output_act": "identity", 43 | "hidden_act": "relu", 44 | "learn_constant": 1} 45 | } 46 | -------------------------------------------------------------------------------- /EXPERIMENTS_PG/experiments/2020-04-09_base_a2c/2020-04-09_base_a2c.ckpth: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_PG/experiments/2020-04-09_base_a2c/2020-04-09_base_a2c.ckpth -------------------------------------------------------------------------------- /EXPERIMENTS_PG/experiments/2020-04-09_base_a2c/2020-04-09_base_a2c.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/RobertTLange/deep-rl-tutorial/d334551ac9cf36760d232288c31406a46d7dfb80/EXPERIMENTS_PG/experiments/2020-04-09_base_a2c/2020-04-09_base_a2c.hdf5 -------------------------------------------------------------------------------- /EXPERIMENTS_PG/experiments/2020-04-09_base_a2c/2020-04-09_base_a2c.json: -------------------------------------------------------------------------------- 1 | { 2 | "train_config": {"agent_name": "A2C", # Class of agents to train 3 | "seed_id": 0, # Training seed for init 4 | "num_train_updates": 20000, # Batch updates to train with 5 | "env_name": "Pendulum-v0", # Name of training environment 6 | "device_name": "cpu", # Device on which to run sim 7 | # Optimization-spec. hyperparameters 8 | "loss_type": "a2c", 9 | "opt_type": {"policy": "Adam", 10 | "value": "Adam"}, 11 | "l_rate": {"policy": 1e-5, 12 | "value": 1e-5}, 13 | "batch_size": 32, # Divides by threads for sync 14 | "num_threads": 16, # Threads 15 | "steps_in_trajectory": 20, 16 | "optimization_steps_in_mini_epoch": 1, # Updates per batch 17 | # A2C-spec. hyperparams (policy type) 18 | "return_type": "advantage-critic", # vanilla/gae 19 | "shared_torso": 0, 20 | "entropy_beta": 0.02, 21 | "clip_action_to_range": 1, 22 | "policy_type": "gaussian", 23 | # MDP-spec. hyperparams (Steps, Discount) 24 | "max_steps_in_episode": 200, 25 | "train_discount_factor": 1, 26 | "test_discount_factor": 1, 27 | # Logging-spec. hyperparams (When/how much) 28 | "evaluate_every_optimization_steps": 100 29 | }, 30 | "log_config": {"time_to_track": ["ep_counter", "optim_counter", 31 | "step_counter", "t_since_last_opt"], 32 | "what_to_track": ["rew_mean", "rew_sd", "rew_median", 33 | "rew_10th_p", "rew_90th_p", 34 | "steps_mean", "steps_sd", "steps_median", 35 | "steps_10th_p", "steps_90th_p", 36 | "success_rate", "loss"], 37 | "experiment_dir": "experiments/", 38 | "seed_id": 0, 39 | "time_to_print": ["optim_counter", "t_since_last_opt"], 40 | "what_to_print": ["rew_mean", "rew_median", 41 | "steps_mean", "steps_median", "success_rate"]}, 42 | "net_config": {# Specify the actor architecture 43 | "policy": {"input_dim": [1, 3], 44 | "layers_info": [["flatten"], 45 | ["linear", 32, 1], 46 | ["linear", 32, 1], 47 | ["linear", 1, 1]], 48 | "output_act": "identity", 49 | "hidden_act": "relu", 50 | "learn_constant": 1 # whether to learn log std 51 | }, 52 | # Specify the critic architecture 53 | "value": {"input_dim": [1, 3], 54 | "layers_info": [["flatten"], 55 | ["linear", 32, 1], 56 | ["linear", 32, 1], 57 | ["linear", 1, 1]], 58 | "output_act": "identity", 59 | "hidden_act": "relu", 60 | "dropout": 0.0, 61 | "batch_norm": 0}} 62 | } 63 | -------------------------------------------------------------------------------- /EXPERIMENTS_PG/train_policy_gradients.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import torch 3 | 4 | from drl_toolbox.utils import DotDic 5 | from drl_toolbox.utils import set_random_seeds, load_config 6 | from drl_toolbox.dl import BodyBuilder, RecurrentBodyBuilder 7 | from drl_toolbox.dl import set_optimizer, DeepLogger 8 | 9 | from drl_toolbox.single_rl import make_parallel_env, ActorCriticBuilder 10 | from drl_toolbox.single_rl.agents import VPG_Agent, A2C_Agent, DDPG_Agent 11 | 12 | 13 | def main(train_config, net_config, log_config): 14 | """ Train & evaluate a policy gradient agent. """ 15 | # Set the training device & the random seed for the example run 16 | device_name = train_config.device_name 17 | device = torch.device(device_name) 18 | set_random_seeds(seed_id=train_config.seed_id, verbose=False) 19 | 20 | # Agent - Envs, Nets, Optimizer, Train Config, Logger 21 | train_log = DeepLogger(**log_config) 22 | env = make_parallel_env(train_config.env_name, 23 | train_config.seed_id, 24 | train_config.num_threads) 25 | 26 | if train_config.agent_name == "VPG": 27 | # Define the policy network architecture & optimizer 28 | policy_network = BodyBuilder(**net_config).to(device) 29 | optimizer = set_optimizer(network=policy_network, 30 | opt_type=train_config.opt_type, 31 | l_rate=train_config.l_rate) 32 | 33 | # Instantiate PG agent class with both the actor net (use full returns) 34 | pg_agent = VPG_Agent(env, train_config, train_log, 35 | policy_network, optimizer, device) 36 | 37 | elif train_config.agent_name == "A2C": 38 | # Define the actor-critic network architecture & optimizer 39 | actor_critic_network = ActorCriticBuilder(**net_config["value"]).to(device) 40 | optimizer = set_optimizer(network=actor_critic_network, 41 | opt_type=train_config.opt_type, 42 | l_rate=train_config.l_rate) 43 | 44 | # Instantiate A2C agent class with both the actor & critic networks 45 | pg_agent = A2C_Agent(env, train_config, train_log, 46 | actor_critic_network, optimizer, device) 47 | 48 | elif train_config.agent_name == "DDPG": 49 | # Define the actor-critic network architecture & optimizer 50 | actor_critic_network = ActorCriticBuilder(**net_config["value"]).to(device) 51 | value_optimizer = set_optimizer(network=actor_critic_network, 52 | opt_type=train_config.opt_type, 53 | l_rate=train_config.l_rate) 54 | 55 | # Define the target networks 56 | 57 | # Define the memory buffer system 58 | 59 | # Define the exploration strategy of DDPG Agent - OU process 60 | 61 | # Instantiate DDPG agent class w. networks/targets/buffer/exploration 62 | pg_agent = DDPG_Agent() 63 | 64 | # Run the training loop for the Policy Gradient agent 65 | pg_agent.run_learning_loop(num_train_updates=train_config.num_train_updates) 66 | return train_log, policy_network 67 | 68 | 69 | if __name__ == "__main__": 70 | def get_cmd_args(): 71 | """ Get env name, config file path & device to train from cmd line """ 72 | parser = argparse.ArgumentParser() 73 | parser.add_argument('-config', '--config_fname', action="store", 74 | default="configs/base_vpg.json", type=str, 75 | help='Filename from which to load config') 76 | return parser.parse_args() 77 | 78 | cmd_args = get_cmd_args() 79 | config = load_config(cmd_args.config_fname) 80 | config.log_config.config_fname = cmd_args.config_fname 81 | 82 | net_config = DotDic(config["net_config"]) 83 | train_config = DotDic(config["train_config"]) 84 | log_config = dict(config["log_config"]) 85 | main(train_config, net_config, log_config) 86 | -------------------------------------------------------------------------------- /EXPERIMENTS_PG/viz_results.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# A Tutorial on Policy Gradient Methods\n", 8 | "# Author: Robert T Lange" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": 13, 14 | "metadata": {}, 15 | "outputs": [], 16 | "source": [ 17 | "import gym" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 14, 23 | "metadata": {}, 24 | "outputs": [ 25 | { 26 | "name": "stderr", 27 | "output_type": "stream", 28 | "text": [ 29 | "/Users/rtl/anaconda2/envs/ma-vision/lib/python3.6/site-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n", 30 | " warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n" 31 | ] 32 | } 33 | ], 34 | "source": [ 35 | "env = gym.make(\"Pendulum-v0\")" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": 15, 41 | "metadata": {}, 42 | "outputs": [ 43 | { 44 | "data": { 45 | "text/plain": [ 46 | "array([-0.77561978, -0.63120041, 0.3537931 ])" 47 | ] 48 | }, 49 | "execution_count": 15, 50 | "metadata": {}, 51 | "output_type": "execute_result" 52 | } 53 | ], 54 | "source": [ 55 | "env.reset()" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 16, 61 | "metadata": {}, 62 | "outputs": [ 63 | { 64 | "data": { 65 | "text/plain": [ 66 | "(array([-0.77841584, -0.62774897, -0.08883786]), -6.056745576278789, False, {})" 67 | ] 68 | }, 69 | "execution_count": 16, 70 | "metadata": {}, 71 | "output_type": "execute_result" 72 | } 73 | ], 74 | "source": [ 75 | "import numpy as np\n", 76 | "action = np.array([0.205129])\n", 77 | "\n", 78 | "low = env.action_space.low\n", 79 | "high = env.action_space.high\n", 80 | "\n", 81 | "a = np.clip(action, low, high)\n", 82 | "env.step(a)" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": 17, 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [ 91 | "action = [np.array([0.2323]), np.array([0.34343])]" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": 18, 97 | "metadata": {}, 98 | "outputs": [ 99 | { 100 | "data": { 101 | "text/plain": [ 102 | "array([[-0.64023891, -0.76817585, 0.08822884],\n", 103 | " [ 0.97492677, -0.22252596, 0.84514401]])" 104 | ] 105 | }, 106 | "execution_count": 18, 107 | "metadata": {}, 108 | "output_type": "execute_result" 109 | } 110 | ], 111 | "source": [ 112 | "from drl_toolbox.single_rl import make_parallel_env\n", 113 | "env = make_parallel_env(\"Pendulum-v0\",\n", 114 | " 0, 2)\n", 115 | "env.reset()" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": 19, 121 | "metadata": {}, 122 | "outputs": [], 123 | "source": [ 124 | "for i in range(1000):\n", 125 | " env.step(action)" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 20, 131 | "metadata": {}, 132 | "outputs": [ 133 | { 134 | "data": { 135 | "text/plain": [ 136 | "[0, 1]" 137 | ] 138 | }, 139 | "execution_count": 20, 140 | "metadata": {}, 141 | "output_type": "execute_result" 142 | } 143 | ], 144 | "source": [ 145 | "a = np.array([0, 1])\n", 146 | "list(a)" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": 21, 152 | "metadata": {}, 153 | "outputs": [ 154 | { 155 | "ename": "SyntaxError", 156 | "evalue": "invalid syntax (, line 1)", 157 | "output_type": "error", 158 | "traceback": [ 159 | "\u001b[0;36m File \u001b[0;32m\"\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m env.step([)\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n" 160 | ] 161 | } 162 | ], 163 | "source": [ 164 | "env.step([)" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": 8, 170 | "metadata": {}, 171 | "outputs": [ 172 | { 173 | "data": { 174 | "text/plain": [ 175 | "[array([0.2323]), array([0.34343])]" 176 | ] 177 | }, 178 | "execution_count": 8, 179 | "metadata": {}, 180 | "output_type": "execute_result" 181 | } 182 | ], 183 | "source": [ 184 | "action" 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": 12, 190 | "metadata": {}, 191 | "outputs": [ 192 | { 193 | "data": { 194 | "text/plain": [ 195 | "[array(0), array(1)]" 196 | ] 197 | }, 198 | "execution_count": 12, 199 | "metadata": {}, 200 | "output_type": "execute_result" 201 | } 202 | ], 203 | "source": [ 204 | "[np.array(a_t) for a_t in a]" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 22, 210 | "metadata": {}, 211 | "outputs": [], 212 | "source": [ 213 | "import math\n", 214 | "import random\n", 215 | "\n", 216 | "import gym\n", 217 | "import numpy as np\n", 218 | "\n", 219 | "import torch\n", 220 | "import torch.nn as nn\n", 221 | "import torch.optim as optim\n", 222 | "import torch.nn.functional as F\n", 223 | "from torch.distributions import Normal" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": 23, 229 | "metadata": {}, 230 | "outputs": [], 231 | "source": [ 232 | "use_cuda = torch.cuda.is_available()\n", 233 | "device = torch.device(\"cuda\" if use_cuda else \"cpu\")" 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "execution_count": 24, 239 | "metadata": {}, 240 | "outputs": [], 241 | "source": [ 242 | "import numpy as np\n", 243 | "from multiprocessing import Process, Pipe\n", 244 | "\n", 245 | "def worker(remote, parent_remote, env_fn_wrapper):\n", 246 | " parent_remote.close()\n", 247 | " env = env_fn_wrapper.x()\n", 248 | " while True:\n", 249 | " cmd, data = remote.recv()\n", 250 | " if cmd == 'step':\n", 251 | " ob, reward, done, info = env.step(data)\n", 252 | " if done:\n", 253 | " ob = env.reset()\n", 254 | " remote.send((ob, reward, done, info))\n", 255 | " elif cmd == 'reset':\n", 256 | " ob = env.reset()\n", 257 | " remote.send(ob)\n", 258 | " elif cmd == 'reset_task':\n", 259 | " ob = env.reset_task()\n", 260 | " remote.send(ob)\n", 261 | " elif cmd == 'close':\n", 262 | " remote.close()\n", 263 | " break\n", 264 | " elif cmd == 'get_spaces':\n", 265 | " remote.send((env.observation_space, env.action_space))\n", 266 | " else:\n", 267 | " raise NotImplementedError\n", 268 | "\n", 269 | "class VecEnv(object):\n", 270 | " \"\"\"\n", 271 | " An abstract asynchronous, vectorized environment.\n", 272 | " \"\"\"\n", 273 | " def __init__(self, num_envs, observation_space, action_space):\n", 274 | " self.num_envs = num_envs\n", 275 | " self.observation_space = observation_space\n", 276 | " self.action_space = action_space\n", 277 | "\n", 278 | " def reset(self):\n", 279 | " \"\"\"\n", 280 | " Reset all the environments and return an array of\n", 281 | " observations, or a tuple of observation arrays.\n", 282 | " If step_async is still doing work, that work will\n", 283 | " be cancelled and step_wait() should not be called\n", 284 | " until step_async() is invoked again.\n", 285 | " \"\"\"\n", 286 | " pass\n", 287 | "\n", 288 | " def step_async(self, actions):\n", 289 | " \"\"\"\n", 290 | " Tell all the environments to start taking a step\n", 291 | " with the given actions.\n", 292 | " Call step_wait() to get the results of the step.\n", 293 | " You should not call this if a step_async run is\n", 294 | " already pending.\n", 295 | " \"\"\"\n", 296 | " pass\n", 297 | "\n", 298 | " def step_wait(self):\n", 299 | " \"\"\"\n", 300 | " Wait for the step taken with step_async().\n", 301 | " Returns (obs, rews, dones, infos):\n", 302 | " - obs: an array of observations, or a tuple of\n", 303 | " arrays of observations.\n", 304 | " - rews: an array of rewards\n", 305 | " - dones: an array of \"episode done\" booleans\n", 306 | " - infos: a sequence of info objects\n", 307 | " \"\"\"\n", 308 | " pass\n", 309 | "\n", 310 | " def close(self):\n", 311 | " \"\"\"\n", 312 | " Clean up the environments' resources.\n", 313 | " \"\"\"\n", 314 | " pass\n", 315 | "\n", 316 | " def step(self, actions):\n", 317 | " self.step_async(actions)\n", 318 | " return self.step_wait()\n", 319 | "\n", 320 | " \n", 321 | "class CloudpickleWrapper(object):\n", 322 | " \"\"\"\n", 323 | " Uses cloudpickle to serialize contents (otherwise multiprocessing tries to use pickle)\n", 324 | " \"\"\"\n", 325 | " def __init__(self, x):\n", 326 | " self.x = x\n", 327 | " def __getstate__(self):\n", 328 | " import cloudpickle\n", 329 | " return cloudpickle.dumps(self.x)\n", 330 | " def __setstate__(self, ob):\n", 331 | " import pickle\n", 332 | " self.x = pickle.loads(ob)\n", 333 | "\n", 334 | " \n", 335 | "class SubprocVecEnv(VecEnv):\n", 336 | " def __init__(self, env_fns, spaces=None):\n", 337 | " \"\"\"\n", 338 | " envs: list of gym environments to run in subprocesses\n", 339 | " \"\"\"\n", 340 | " self.waiting = False\n", 341 | " self.closed = False\n", 342 | " nenvs = len(env_fns)\n", 343 | " self.nenvs = nenvs\n", 344 | " self.remotes, self.work_remotes = zip(*[Pipe() for _ in range(nenvs)])\n", 345 | " self.ps = [Process(target=worker, args=(work_remote, remote, CloudpickleWrapper(env_fn)))\n", 346 | " for (work_remote, remote, env_fn) in zip(self.work_remotes, self.remotes, env_fns)]\n", 347 | " for p in self.ps:\n", 348 | " p.daemon = True # if the main process crashes, we should not cause things to hang\n", 349 | " p.start()\n", 350 | " for remote in self.work_remotes:\n", 351 | " remote.close()\n", 352 | "\n", 353 | " self.remotes[0].send(('get_spaces', None))\n", 354 | " observation_space, action_space = self.remotes[0].recv()\n", 355 | " VecEnv.__init__(self, len(env_fns), observation_space, action_space)\n", 356 | "\n", 357 | " def step_async(self, actions):\n", 358 | " for remote, action in zip(self.remotes, actions):\n", 359 | " remote.send(('step', action))\n", 360 | " self.waiting = True\n", 361 | "\n", 362 | " def step_wait(self):\n", 363 | " results = [remote.recv() for remote in self.remotes]\n", 364 | " self.waiting = False\n", 365 | " obs, rews, dones, infos = zip(*results)\n", 366 | " return np.stack(obs), np.stack(rews), np.stack(dones), infos\n", 367 | "\n", 368 | " def reset(self):\n", 369 | " for remote in self.remotes:\n", 370 | " remote.send(('reset', None))\n", 371 | " return np.stack([remote.recv() for remote in self.remotes])\n", 372 | "\n", 373 | " def reset_task(self):\n", 374 | " for remote in self.remotes:\n", 375 | " remote.send(('reset_task', None))\n", 376 | " return np.stack([remote.recv() for remote in self.remotes])\n", 377 | "\n", 378 | " def close(self):\n", 379 | " if self.closed:\n", 380 | " return\n", 381 | " if self.waiting:\n", 382 | " for remote in self.remotes: \n", 383 | " remote.recv()\n", 384 | " for remote in self.remotes:\n", 385 | " remote.send(('close', None))\n", 386 | " for p in self.ps:\n", 387 | " p.join()\n", 388 | " self.closed = True\n", 389 | " \n", 390 | " def __len__(self):\n", 391 | " return self.nenvs" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": 50, 397 | "metadata": {}, 398 | "outputs": [], 399 | "source": [ 400 | "num_envs = 3\n", 401 | "env_name = \"Pendulum-v0\"\n", 402 | "\n", 403 | "def make_env():\n", 404 | " def _thunk():\n", 405 | " env = gym.make(env_name)\n", 406 | " return env\n", 407 | "\n", 408 | " return _thunk\n", 409 | "\n", 410 | "envs = [make_env() for i in range(num_envs)]\n", 411 | "envs = SubprocVecEnv(envs)\n", 412 | "\n", 413 | "env = gym.make(env_name)" 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "execution_count": 51, 419 | "metadata": {}, 420 | "outputs": [], 421 | "source": [ 422 | "def init_weights(m):\n", 423 | " if isinstance(m, nn.Linear):\n", 424 | " nn.init.normal_(m.weight, mean=0., std=0.1)\n", 425 | " nn.init.constant_(m.bias, 0.1)\n", 426 | " \n", 427 | "\n", 428 | "class ActorCritic(nn.Module):\n", 429 | " def __init__(self, num_inputs, num_outputs, hidden_size, std=0.0):\n", 430 | " super(ActorCritic, self).__init__()\n", 431 | " \n", 432 | " self.critic = nn.Sequential(\n", 433 | " nn.Linear(num_inputs, hidden_size),\n", 434 | " nn.ReLU(),\n", 435 | " nn.Linear(hidden_size, 1)\n", 436 | " )\n", 437 | " \n", 438 | " self.actor = nn.Sequential(\n", 439 | " nn.Linear(num_inputs, hidden_size),\n", 440 | " nn.ReLU(),\n", 441 | " nn.Linear(hidden_size, num_outputs),\n", 442 | " )\n", 443 | " self.log_std = nn.Parameter(torch.ones(1, num_outputs) * std)\n", 444 | " \n", 445 | " self.apply(init_weights)\n", 446 | " \n", 447 | " def forward(self, x):\n", 448 | " value = self.critic(x)\n", 449 | " mu = self.actor(x)\n", 450 | " std = self.log_std.exp().expand_as(mu)\n", 451 | " dist = Normal(mu, std)\n", 452 | " return dist, value" 453 | ] 454 | }, 455 | { 456 | "cell_type": "code", 457 | "execution_count": 94, 458 | "metadata": {}, 459 | "outputs": [], 460 | "source": [ 461 | "def compute_gae(next_value, rewards, masks, values, gamma=0.99, tau=0.95):\n", 462 | " values = values + [next_value]\n", 463 | " gae = 0\n", 464 | " returns = []\n", 465 | " for step in reversed(range(len(rewards))):\n", 466 | " delta = rewards[step] + gamma * values[step + 1] * masks[step] - values[step]\n", 467 | " gae = delta + gamma * tau * masks[step] * gae\n", 468 | " returns.insert(0, gae + values[step])\n", 469 | " return returns\n", 470 | "\n", 471 | "def ppo_iter(mini_batch_size, states, actions, log_probs, returns, advantage):\n", 472 | " batch_size = states.size(0)\n", 473 | " for _ in range(batch_size // mini_batch_size):\n", 474 | " rand_ids = np.random.randint(0, batch_size, mini_batch_size)\n", 475 | " yield states[rand_ids, :], actions[rand_ids, :], log_probs[rand_ids, :], returns[rand_ids, :], advantage[rand_ids, :]\n", 476 | " \n", 477 | " \n", 478 | "\n", 479 | "def ppo_update(ppo_epochs, mini_batch_size, states, actions, log_probs, returns, advantages, clip_param=0.2):\n", 480 | " for _ in range(ppo_epochs):\n", 481 | " for state, action, old_log_probs, return_, advantage in ppo_iter(mini_batch_size, states, actions, log_probs, returns, advantages):\n", 482 | " dist, value = model(state)\n", 483 | " entropy = dist.entropy().mean()\n", 484 | " new_log_probs = dist.log_prob(action)\n", 485 | "\n", 486 | " ratio = (new_log_probs - old_log_probs).exp()\n", 487 | " surr1 = ratio * advantage\n", 488 | " surr2 = torch.clamp(ratio, 1.0 - clip_param, 1.0 + clip_param) * advantage\n", 489 | "\n", 490 | " actor_loss = - torch.min(surr1, surr2).mean()\n", 491 | " critic_loss = (return_ - value).pow(2).mean()\n", 492 | "\n", 493 | " loss = 0.5 * critic_loss + actor_loss - 0.001 * entropy\n", 494 | "\n", 495 | " optimizer.zero_grad()\n", 496 | " loss.backward()\n", 497 | " optimizer.step()" 498 | ] 499 | }, 500 | { 501 | "cell_type": "code", 502 | "execution_count": 95, 503 | "metadata": {}, 504 | "outputs": [], 505 | "source": [ 506 | "num_inputs = envs.observation_space.shape[0]\n", 507 | "num_outputs = envs.action_space.shape[0]\n", 508 | "\n", 509 | "#Hyper params:\n", 510 | "hidden_size = 256\n", 511 | "lr = 3e-4\n", 512 | "num_steps = 10\n", 513 | "mini_batch_size = 5\n", 514 | "ppo_epochs = 4\n", 515 | "threshold_reward = -200\n", 516 | "\n", 517 | "model = ActorCritic(num_inputs, num_outputs, hidden_size).to(device)\n", 518 | "optimizer = optim.Adam(model.parameters(), lr=lr)\n", 519 | "\n", 520 | "max_frames = 10\n", 521 | "frame_idx = 0\n", 522 | "test_rewards = []" 523 | ] 524 | }, 525 | { 526 | "cell_type": "code", 527 | "execution_count": 96, 528 | "metadata": {}, 529 | "outputs": [ 530 | { 531 | "name": "stdout", 532 | "output_type": "stream", 533 | "text": [ 534 | "torch.Size([3, 1])\n", 535 | "[tensor([[-50.4879],\n", 536 | " [-30.6074],\n", 537 | " [ -9.5683]], grad_fn=), tensor([[-48.6792],\n", 538 | " [-31.0566],\n", 539 | " [-10.0888]], grad_fn=), tensor([[-46.8133],\n", 540 | " [-31.5220],\n", 541 | " [-10.5692]], grad_fn=), tensor([[-44.7081],\n", 542 | " [-31.7268],\n", 543 | " [-10.9529]], grad_fn=), tensor([[-42.1678],\n", 544 | " [-31.4641],\n", 545 | " [-11.1314]], grad_fn=), tensor([[-38.8040],\n", 546 | " [-30.6559],\n", 547 | " [-11.1336]], grad_fn=), tensor([[-34.3036],\n", 548 | " [-28.9670],\n", 549 | " [-10.6924]], grad_fn=), tensor([[-28.3502],\n", 550 | " [-25.7622],\n", 551 | " [ -9.7362]], grad_fn=), tensor([[-20.6461],\n", 552 | " [-20.4984],\n", 553 | " [ -8.1192]], grad_fn=), tensor([[-11.0852],\n", 554 | " [-12.5355],\n", 555 | " [ -5.4365]], grad_fn=)]\n", 556 | "torch.Size([30, 1])\n", 557 | "torch.Size([30, 1])\n", 558 | "torch.Size([30, 1])\n", 559 | "tensor([[-50.4329],\n", 560 | " [-30.7752],\n", 561 | " [ -9.3755],\n", 562 | " [-48.6794],\n", 563 | " [-31.0111],\n", 564 | " [ -9.8513],\n", 565 | " [-46.8776],\n", 566 | " [-31.2014],\n", 567 | " [-10.2692],\n", 568 | " [-44.8238],\n", 569 | " [-31.2528],\n", 570 | " [-10.5405],\n", 571 | " [-42.3125],\n", 572 | " [-30.9059],\n", 573 | " [-10.6627],\n", 574 | " [-38.9773],\n", 575 | " [-29.9845],\n", 576 | " [-10.5372],\n", 577 | " [-34.5070],\n", 578 | " [-28.1098],\n", 579 | " [-10.0100],\n", 580 | " [-28.5870],\n", 581 | " [-24.7333],\n", 582 | " [ -8.9654],\n", 583 | " [-20.9148],\n", 584 | " [-19.3070],\n", 585 | " [ -7.2215],\n", 586 | " [-11.3773],\n", 587 | " [-11.2190],\n", 588 | " [ -4.3901]])\n" 589 | ] 590 | } 591 | ], 592 | "source": [ 593 | "state = envs.reset()\n", 594 | "early_stop = False\n", 595 | "\n", 596 | "while frame_idx < max_frames and not early_stop:\n", 597 | "\n", 598 | " log_probs = []\n", 599 | " values = []\n", 600 | " states = []\n", 601 | " actions = []\n", 602 | " rewards = []\n", 603 | " masks = []\n", 604 | " entropy = 0\n", 605 | "\n", 606 | " for _ in range(num_steps):\n", 607 | " state = torch.FloatTensor(state).to(device)\n", 608 | " dist, value = model(state)\n", 609 | "\n", 610 | " action = dist.sample()\n", 611 | " next_state, reward, done, _ = envs.step(action.cpu().numpy())\n", 612 | "\n", 613 | " log_prob = dist.log_prob(action)\n", 614 | " entropy += dist.entropy().mean()\n", 615 | " # print(log_prob.size())\n", 616 | " log_probs.append(log_prob)\n", 617 | " # print(log_prob.size())\n", 618 | " values.append(value)\n", 619 | " rewards.append(torch.FloatTensor(reward).unsqueeze(1).to(device))\n", 620 | " masks.append(torch.FloatTensor(1 - done).unsqueeze(1).to(device))\n", 621 | " \n", 622 | " states.append(state)\n", 623 | " actions.append(action)\n", 624 | " \n", 625 | " state = next_state\n", 626 | " frame_idx += 1\n", 627 | " \n", 628 | " next_state = torch.FloatTensor(next_state).to(device)\n", 629 | " _, next_value = model(next_state)\n", 630 | " print(next_value.size())\n", 631 | " #print(len(rewards))\n", 632 | " # print(len(masks))\n", 633 | " returns = compute_gae(next_value, rewards, masks, values)\n", 634 | " print(returns)\n", 635 | " returns = torch.cat(returns).detach()\n", 636 | " log_probs = torch.cat(log_probs).detach()\n", 637 | " print(log_probs.size())\n", 638 | " values = torch.cat(values).detach()\n", 639 | " states = torch.cat(states)\n", 640 | " actions = torch.cat(actions)\n", 641 | " print(values.size())\n", 642 | " print(returns.size())\n", 643 | " advantage = returns - values\n", 644 | " print(advantage)\n", 645 | " ppo_update(ppo_epochs, mini_batch_size, states, actions, log_probs, returns, advantage)\n" 646 | ] 647 | }, 648 | { 649 | "cell_type": "code", 650 | "execution_count": 87, 651 | "metadata": {}, 652 | "outputs": [], 653 | "source": [ 654 | "# Sum mean entropy (over threads) for all timesteps" 655 | ] 656 | }, 657 | { 658 | "cell_type": "code", 659 | "execution_count": null, 660 | "metadata": {}, 661 | "outputs": [], 662 | "source": [] 663 | }, 664 | { 665 | "cell_type": "code", 666 | "execution_count": null, 667 | "metadata": {}, 668 | "outputs": [], 669 | "source": [] 670 | } 671 | ], 672 | "metadata": { 673 | "kernelspec": { 674 | "display_name": "Python3 (ma-vision)", 675 | "language": "python", 676 | "name": "ma-vision" 677 | }, 678 | "language_info": { 679 | "codemirror_mode": { 680 | "name": "ipython", 681 | "version": 3 682 | }, 683 | "file_extension": ".py", 684 | "mimetype": "text/x-python", 685 | "name": "python", 686 | "nbconvert_exporter": "python", 687 | "pygments_lexer": "ipython3", 688 | "version": "3.6.9" 689 | } 690 | }, 691 | "nbformat": 4, 692 | "nbformat_minor": 4 693 | } 694 | -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | # A Tutorial Series on Deep Reinforcement Learning 2 | ## Author: Robert Tjarko Lange (TU Berlin) 3 | 4 | This repository contains a series of tutorials on Deep Reinforcement Learning (DRL). This includes slides as well as experiments. Going forward I plan on adding exercises as well as complementary blog posts. So stay tuned! 5 | 6 | ### Deep Q-Learning (July 2019) 7 | 8 | 9 | 10 | * [Slides](Deep_Q_Learning.pdf): Includes DQN, Double DQN, Prioritized Experience Replay & Dueling DQNs 11 | * [Experiments](EXPERIMENTS_DQL): Provides code to implement all of the above. 12 | * [Blog Post I/II](https://roberttlange.github.io/posts/2019/08/blog-post-5/): Covering all algorithms from Fitted Q-Learning to Categorical DQNs. 13 | * Replicating the experiments: 14 | 1. Create & activate a virtual env. Install the requirements. 15 | 2. Afterwards you can run all experiments by executing: 16 | 17 | ``` 18 | time bash run_experiments_dqn.sh dqn 19 | time bash run_experiments_dqn.sh double-dqn 20 | time bash run_experiments_dqn.sh per-dqn 21 | time bash run_experiments_dqn.sh dueling-dqn 22 | ``` 23 | 3. The visualizations for the different experiments as well as the mini double DQN illustration can be replicated by executing the notebook: 24 | 25 | ``` 26 | jupyter notebook viz_results.ipynb 27 | ``` 28 | 29 | 4. Finally, in order to visualize an episode rollout of a DQN agent at different stages do the following: 30 | 31 | ``` 32 | python train_dqn.py --SAVE_AGENT 33 | python enjoy_dense.py --AGENT 5000_MLP-DQN --TITLE 5000 34 | python enjoy_dense.py --AGENT 40000_MLP-DQN --TITLE 40000 35 | python enjoy_dense.py --AGENT 500000_MLP-DQN --TITLE 500000 36 | ``` 37 | 38 | 39 | 40 | 41 | ### Deep Policy Gradients (to be continued) 42 | --------------------------------------------------------------------------------