├── MDP_RL_Lecture1.pdf ├── MDP_RL_Lecture2.pdf ├── README.md ├── frameworks_demo └── Gym │ ├── images │ ├── algo.png │ ├── atari.png │ ├── cartpole.jpg │ ├── go.png │ └── mujoco.png │ ├── lua │ ├── README.MD │ ├── dqn.lua │ ├── gym_http_client.lua │ ├── layers.lua │ ├── main.lua │ ├── replay_buffer.lua │ └── utils.lua │ ├── main.ipynb │ └── python │ ├── README.MD │ ├── dqn.py │ ├── main.py │ └── replay_buffer.py └── mb_demo.ipynb /MDP_RL_Lecture1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cilvrRG/RL/3d1a68624d14b29deb80bce23bcab6379f507fc8/MDP_RL_Lecture1.pdf -------------------------------------------------------------------------------- /MDP_RL_Lecture2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cilvrRG/RL/3d1a68624d14b29deb80bce23bcab6379f507fc8/MDP_RL_Lecture2.pdf -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # RL 2 | ##Reading Group on Reinforcement Learning topics 3 | ##NYU, Fall 2016 4 | 5 | ###Logistics 6 | - Meetings run every **Wednesday at 9h30** (before the CILVR lab meeting), at the large conference room at 715/719 Broadway 12th floor. Breakfast will be provided. 7 | - Paper discussion + Paper review plan: Each week we will assign one or two papers to volunteers who will present it following week. During the reading/presentation, we will edit a review of the paper which will be posted here. [also subject to change]. 8 | - Guest speakers. We will try to invite RL experts (e.g. G. Tesauro) with some frequency. 9 | - Other communication channels ( Facebook groups?, Slack? ) [TBD]. 10 | 11 | ###Organization 12 | The RG is initially organized by J.Bruna, K. Cho, S. Sukhbaatar, K. Ross, D. Sontag, with help from the rest of the CILVR group. 13 | 14 | ### Tentative Agenda 15 | 16 | - 9/21: [Tutorial on MDPs, Policy Gradient (part 1)](MDP_RL_Lecture1.pdf). [**Keith Ross**] 17 | - Markov Decision Process Paradigm 18 | - Discounted and average cost criteria 19 | - Model-free Reinforcement Learning Paradigm 20 | - Policy Gradient: parameterized policies; policy gradient theorem; Monte Carlo Policy Gradient (REINFORCE) 21 | - Using Policy Gradient and deep neural networks to learn the Atari game "pong". 22 | 23 | - 9/28: [Tutorial on MDPs, Policy Gradient (part 2)](MDP_RL_Lecture2.pdf). [**Keith**] 24 | - Dynamic Programming equations for MDPs 25 | - Policy iteration 26 | - Value iteration 27 | - Monte Carlo methods for RL 28 | - Q-learning for RL 29 | 30 | - 10/5 and 10/12: Actor-Critic. [**Martin**] 31 | - Deterministic Policy Gradient 32 | - Off-Policy variants 33 | - Relevant Papers: 34 | - [Policy Gradient and Actor Critic](https://webdocs.cs.ualberta.ca/~sutton/papers/SMSM-NIPS99.pdf) 35 | - [Deterministic Policy Gradients](http://jmlr.org/proceedings/papers/v32/silver14.pdf) 36 | - [Off-policy actor critic](https://webdocs.cs.ualberta.ca/~sutton/papers/Degris-OffPAC-ICML-2012.pdf) 37 | 38 | - 10/19: Tutorial on OpenAI Gym and Mazebase. Also, Twitter's new [twrl](https://github.com/twitter/torch-twrl) [**Sainaa and Ilya**] 39 | - MazeBase: https://github.com/facebook/MazeBase 40 | - 10/26: [Apprenticeship Learning via Inverse Reinforcement Learning](http://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf) and [Model-Free Imitation Learning with Policy Optimization](https://arxiv.org/abs/1605.08478) [**Arthur**] 41 | - 10/31: Trust region policy optimization (TRPO) [**Elman, Ilya**] 42 | 43 | 44 | 45 | 46 | ### Pool of Papers [please fill] 47 | 48 | - Guided Policy Search 49 | - Value Iteration Networks 50 | - TRPO [Elman, early November] 51 | - Review of recent hierarchical reinforcement learning papers [Sainaa] 52 | - Intrinsically Motivated Reinforcement Learning [Martin?]: 53 | - [Intrinsically Motivated Reinforcement Learning](https://web.eecs.umich.edu/~baveja/Papers/FinalNIPSIMRL.pdf) 54 | - [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://arxiv.org/pdf/1509.08731v1.pdf) 55 | - [Bayesian Surprise Attracts Human Attention](https://papers.nips.cc/paper/2822-bayesian-surprise-attracts-human-attention.pdf) 56 | - [Variational Information Maximizing Exploration](https://arxiv.org/abs/1605.09674) 57 | - [Unifying Count-Based Exploration and Intrinsic Motivation](https://arxiv.org/pdf/1606.01868v2.pdf) 58 | - High dimensional action spaces: 59 | - [Reinforcement Learning with Factored States and Actions](http://www.jmlr.org/papers/volume5/sallans04a/sallans04a.pdf) 60 | - [Deep Reinforcement Learning in Large Discrete Action Spaces](https://arxiv.org/pdf/1512.07679.pdf) 61 | - [Learning Multiagent Communication with Backpropagation](https://arxiv.org/pdf/1605.07736.pdf) 62 | - Stability in RL (these 4 papers shouldn't take more than 1 or 2 lectures): 63 | - [Playing Atari with Deep Reinforcement Learning](https://arxiv.org/abs/1312.5602) 64 | - [Human-level control through deep reinforcement learning](http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html) 65 | - [Double Q-learning](http://papers.nips.cc/paper/3964-double-q-learning.pdf). Also consider reading about [the optimizer's curse](https://faculty.fuqua.duke.edu/~jes9/bio/Optimizers_Curse.pdf) to make the reading simpler. 66 | - [Prioritized Experience Replay](https://arxiv.org/abs/1511.05952) 67 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/images/algo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cilvrRG/RL/3d1a68624d14b29deb80bce23bcab6379f507fc8/frameworks_demo/Gym/images/algo.png -------------------------------------------------------------------------------- /frameworks_demo/Gym/images/atari.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cilvrRG/RL/3d1a68624d14b29deb80bce23bcab6379f507fc8/frameworks_demo/Gym/images/atari.png -------------------------------------------------------------------------------- /frameworks_demo/Gym/images/cartpole.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cilvrRG/RL/3d1a68624d14b29deb80bce23bcab6379f507fc8/frameworks_demo/Gym/images/cartpole.jpg -------------------------------------------------------------------------------- /frameworks_demo/Gym/images/go.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cilvrRG/RL/3d1a68624d14b29deb80bce23bcab6379f507fc8/frameworks_demo/Gym/images/go.png -------------------------------------------------------------------------------- /frameworks_demo/Gym/images/mujoco.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cilvrRG/RL/3d1a68624d14b29deb80bce23bcab6379f507fc8/frameworks_demo/Gym/images/mujoco.png -------------------------------------------------------------------------------- /frameworks_demo/Gym/lua/README.MD: -------------------------------------------------------------------------------- 1 | 1. Install server: https://github.com/openai/gym-http-api 2 | 2. Launch server: 3 | ```bash 4 | python gym_http_server.py 5 | ``` 6 | 3. Run: 7 | ```bash 8 | th main.lua 9 | ``` 10 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/lua/dqn.lua: -------------------------------------------------------------------------------- 1 | require 'nn' 2 | require 'optim' 3 | 4 | local class = require 'class' 5 | 6 | DQN = class('DQN') 7 | 8 | function DQN:__init(model, discount, optimConfig) 9 | self.model = model 10 | self.parameters, self.gradParameters = self.model:getParameters() 11 | 12 | self.criterion = nn.MSECriterion() 13 | 14 | self.optimConfig = optimConfig or { 15 | optimizer = "rmsprop", 16 | learningRate = 0.001, 17 | } 18 | 19 | self.targets = torch.Tensor() --A buffer for targets 20 | self.discount = discount or 0.99 21 | end 22 | 23 | function DQN:act(observation) 24 | local qvalues = self.model:forward(observation) 25 | local _, indices = torch.max(qvalues, 2) 26 | return indices 27 | end 28 | 29 | function DQN:create_targets(rewards, next_observations, mask) 30 | local values = torch.max(self.model:forward(next_observations), 2) 31 | return torch.cmul(mask, values) * self.discount + rewards 32 | end 33 | 34 | function DQN:update_params(observations, actions, qtargets) 35 | local feval = function(x) 36 | 37 | -- reset gradients 38 | self.gradParameters:zero() 39 | 40 | local qvalues = self.model:forward(observations) 41 | 42 | self.targets:resizeAs(qvalues) 43 | self.targets:copy(qvalues) 44 | 45 | -- A simple trick to avoid using a mask for other actions. 46 | -- Set targets only for taken actions, zero gradient for others. 47 | for i=1,actions:size(1) do 48 | self.targets[i][actions[i][1]] = qtargets[i] 49 | end 50 | 51 | local error = self.criterion:forward(qvalues, self.targets) 52 | local df_do = self.criterion:backward(qvalues, self.targets) 53 | 54 | self.model:backward(observations, df_do) 55 | 56 | return error, self.gradParameters 57 | end 58 | 59 | optim[self.optimConfig.optimizer](feval, self.parameters, self.optimConfig) 60 | end 61 | 62 | function DQN:update(observations, actions, rewards, next_observations, mask) 63 | local qtargets = self:create_targets(rewards, next_observations, mask) 64 | self:update_params(observations, actions, qtargets) 65 | end 66 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/lua/gym_http_client.lua: -------------------------------------------------------------------------------- 1 | local HttpClient = require("httpclient") 2 | local json = require("dkjson") 3 | local os = require 'os' 4 | 5 | local GymClient = {} 6 | local m = {} 7 | 8 | function m.new(remote_base) 9 | local self = {} 10 | self.remote_base = remote_base 11 | self.http = HttpClient.new() 12 | setmetatable(self, {__index = GymClient}) 13 | return self 14 | end 15 | 16 | function GymClient:parse_server_error_or_raise_for_status(resp) 17 | local resp_data, pos, err = {} 18 | if resp.err then 19 | err = resp.err 20 | -- print('Response error: ' .. err) 21 | else 22 | if resp.code ~= 200 then 23 | err = resp.status_line 24 | -- Descriptive message from the server side 25 | -- print('Response: ' .. err) 26 | end 27 | resp_data, pos, err = json.decode(resp.body) 28 | end 29 | return resp_data, pos, err 30 | end 31 | 32 | function GymClient:get_request(route) 33 | url = self.remote_base .. route 34 | options = {} 35 | options.content_type = 'application/json' 36 | resp = self.http:get(url, options) 37 | return self:parse_server_error_or_raise_for_status(resp) 38 | end 39 | 40 | function GymClient:post_request(route, req_data) 41 | url = self.remote_base .. route 42 | options = {} 43 | options.content_type = 'application/json' 44 | json_str = json.encode(req_data) 45 | resp = self.http:post(url, json_str, options) 46 | return self:parse_server_error_or_raise_for_status(resp) 47 | end 48 | 49 | function GymClient:env_create(env_id) 50 | route = '/v1/envs/' 51 | req_data = {env_id = env_id} 52 | resp_data = self:post_request(route, req_data) 53 | return resp_data['instance_id'] 54 | end 55 | 56 | function GymClient:env_list_all() 57 | route = '/v1/envs/' 58 | resp_data = self:get_request(route) 59 | return resp_data['all_envs'] 60 | end 61 | 62 | function GymClient:env_reset(instance_id) 63 | route = '/v1/envs/'..instance_id..'/reset/' 64 | resp_data = self:post_request(route, '') 65 | return resp_data['observation'] 66 | end 67 | 68 | function GymClient:env_step(instance_id, action, render, video_callable) 69 | render = render or false 70 | video_callable = video_callable or false 71 | route = '/v1/envs/'..instance_id..'/step/' 72 | req_data = {action = action, render = render, video_callable = video_callable} 73 | resp_data = self:post_request(route, req_data) 74 | obs = resp_data['observation'] 75 | reward = resp_data['reward'] 76 | done = resp_data['done'] 77 | info = resp_data['info'] 78 | return obs, reward, done, info 79 | end 80 | 81 | function GymClient:env_action_space_info(instance_id) 82 | route = '/v1/envs/'..instance_id..'/action_space/' 83 | resp_data = self:get_request(route) 84 | return resp_data['info'] 85 | end 86 | 87 | function GymClient:env_action_space_sample(instance_id) 88 | route = '/v1/envs/'..instance_id..'/action_space/sample' 89 | resp_data = self:get_request(route) 90 | action = resp_data['action'] 91 | return action 92 | end 93 | 94 | function GymClient:env_action_space_contains(instance_id) 95 | route = '/v1/envs/'..instance_id..'/action_space/contains' 96 | resp_data = self:get_request(route) 97 | member = resp['member'] 98 | return member 99 | end 100 | 101 | function GymClient:env_observation_space_info(instance_id) 102 | route = '/v1/envs/'..instance_id..'/observation_space/' 103 | resp_data = self:get_request(route) 104 | return resp_data['info'] 105 | end 106 | 107 | function GymClient:env_monitor_start(instance_id, directory, force, resume, video_callable) 108 | if not force then force = false end 109 | if not resume then resume = false end 110 | req_data = {directory = directory, 111 | force = tostring(force), 112 | resume = tostring(resume), 113 | video_callable = video_callable} 114 | route = '/v1/envs/'..instance_id..'/monitor/start/' 115 | resp_data = self:post_request(route, req_data) 116 | end 117 | 118 | function GymClient:env_monitor_close(instance_id) 119 | route = '/v1/envs/'..instance_id..'/monitor/close/' 120 | resp_data = self:post_request(route, '') 121 | end 122 | 123 | function GymClient:upload(training_dir, algorithm_id, api_key) 124 | if not api_key then 125 | api_key = os.getenv('OPENAI_GYM_API_KEY') 126 | end 127 | if not algorithm_id then algorithm_id = '' end 128 | req_data = {training_dir = training_dir, 129 | algorithm_id = algorithm_id, 130 | api_key = api_key} 131 | route = '/v1/upload/' 132 | resp = self:post_request(route, req_data) 133 | return resp 134 | end 135 | 136 | function GymClient:shutdown_server() 137 | route = '/v1/shutdown/' 138 | self:post_request(route, '') 139 | end 140 | 141 | return m 142 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/lua/layers.lua: -------------------------------------------------------------------------------- 1 | require 'nngraph' 2 | 3 | --[[ 4 | Copied from: 5 | https://raw.githubusercontent.com/ryankiros/layer-norm/master/torch_modules/LayerNormalization.lua 6 | --]] 7 | 8 | function nn.LayerNormalization(nOutput, bias, eps, affine) 9 | local eps = eps or 1e-5 10 | local affine = affine or true 11 | local bias = bias or nil 12 | 13 | local input = nn.Identity()() 14 | local mean = nn.Mean(2)(input) 15 | local mean_rep = nn.Replicate(nOutput,2)(mean) 16 | 17 | local input_center = nn.CSubTable()({input, mean_rep}) 18 | local std = nn.Sqrt()(nn.Mean(2)(nn.Square()(input_center))) 19 | local std_rep = nn.AddConstant(eps)(nn.Replicate(nOutput,2)(std)) 20 | local output = nn.CDivTable()({input_center, std_rep}) 21 | 22 | if affine == true then 23 | local biasTransform = nn.Add(nOutput, false) 24 | if bias ~=nil then 25 | biasTransform.bias:fill(bias) 26 | end 27 | local gainTransform = nn.CMul(nOutput) 28 | gainTransform.weight:fill(1.) 29 | output = biasTransform(gainTransform(output)) 30 | end 31 | return nn.gModule({input},{output}) 32 | end 33 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/lua/main.lua: -------------------------------------------------------------------------------- 1 | require 'dqn' 2 | require 'replay_buffer' 3 | local utils = require 'utils' 4 | local GymClient = require("gym_http_client") 5 | local HttpClient = require("httpclient") 6 | 7 | client = GymClient.new('http://127.0.0.1:5000') 8 | 9 | -- Parameters 10 | BATCH_SIZE = 32 11 | NUM_EPISODES = 50000 12 | MIN_SAMPLES = 2 * BATCH_SIZE -- Minimal number of samples for an update 13 | EVAL_EPISODE = 10 -- Perform evaluation each EVAL_EPISODE number of episodes 14 | NUM_STEPS = 200 15 | NUM_EVALUATIONS = 10 -- Number of times we run evaluation 16 | INITIAL_EPSILON = 0.9 17 | MIN_EPSILON = 0.1 18 | EPSILON_DECAY = 0.999 19 | BUFFER_SIZE = 100000 20 | 21 | -- Set up environment 22 | env_id = 'CartPole-v0' 23 | instance_id = client:env_create(env_id) 24 | 25 | ndim_action = client:env_action_space_info(instance_id).n 26 | ndim_obs = client:env_observation_space_info(instance_id).shape[1] 27 | epsilon = INITIAL_EPSILON 28 | 29 | agent = DQN(utils.mlp({ndim_obs, 128, 128, ndim_action})) 30 | replayBuffer = ReplayBuffer(BUFFER_SIZE) 31 | 32 | for i = 1, NUM_EPISODES do 33 | -- Training 34 | observation = torch.Tensor(client:env_reset(instance_id)) 35 | for j = 1, NUM_STEPS do 36 | -- Epsilon greedy exploration 37 | if torch.uniform() < epsilon then 38 | action = torch.random(2) 39 | else 40 | action = agent:act(observation:view(1, -1))[1][1] 41 | end 42 | 43 | -- Perform an action, observe next state and reward 44 | next_observation, reward, done, info = client:env_step(instance_id, action-1, true) 45 | next_observation = torch.Tensor(next_observation) 46 | 47 | -- Insert it to replay buffer 48 | replayBuffer:insert(observation, action, reward, done, next_observation) 49 | 50 | if done == true then 51 | break 52 | else 53 | observation = next_observation 54 | end 55 | 56 | if #replayBuffer.replayBuffer > MIN_SAMPLES then 57 | -- Decay epsilon 58 | epsilon = epsilon * EPSILON_DECAY 59 | epsilon = math.max(epsilon, MIN_EPSILON) 60 | 61 | -- Sample a batch of samples and then update 62 | observations, actions, rewards, next_observations, mask = replayBuffer:sample(BATCH_SIZE) 63 | agent:update(observations, actions, rewards, next_observations, mask) 64 | end 65 | end 66 | 67 | -- Evaluation 68 | if i % EVAL_EPISODE == 0 then 69 | local total_reward = 0 70 | for k = 1, NUM_EVALUATIONS do 71 | observation = torch.Tensor(client:env_reset(instance_id)) 72 | 73 | for j = 1, NUM_STEPS do 74 | action = agent:act(observation:view(1, -1))[1][1] 75 | next_observation, reward, done, info = client:env_step(instance_id, action-1, true) 76 | next_observation = torch.Tensor(next_observation) 77 | total_reward = total_reward + reward 78 | 79 | if done == true then 80 | break 81 | else 82 | observation = next_observation 83 | end 84 | end 85 | end 86 | print("Eval after episode #".. i .. ", average reward: " .. total_reward/NUM_EVALUATIONS) 87 | end 88 | end 89 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/lua/replay_buffer.lua: -------------------------------------------------------------------------------- 1 | require 'nn' 2 | local class = require 'class' 3 | 4 | ReplayBuffer = class('ReplayBuffer') 5 | 6 | function ReplayBuffer:__init(maxSize) 7 | self.maxSize = maxSize 8 | self.currentIndex = 1 9 | self.replayBuffer = {} 10 | self.observations = torch.Tensor() 11 | self.actions = torch.LongTensor() 12 | self.rewards = torch.Tensor() 13 | self.next_observations = torch.Tensor() 14 | self.mask = torch.Tensor() 15 | end 16 | 17 | function ReplayBuffer:insert(observation, action, reward, done, next_observation) 18 | self.replayBuffer[self.currentIndex] = { 19 | observation = observation:clone(), 20 | action = action, 21 | next_observation = next_observation:clone(), 22 | reward = reward, 23 | done = done 24 | } 25 | 26 | self.currentIndex = self.currentIndex + 1 27 | if self.currentIndex > self.maxSize then 28 | self.currentIndex = 1 29 | end 30 | end 31 | 32 | function ReplayBuffer:sample(nSamples) 33 | local observationSize = self.replayBuffer[1].observation:size(1) 34 | self.observations:resize(nSamples, observationSize) 35 | self.actions:resize(nSamples, 1) 36 | self.rewards:resize(nSamples, 1) 37 | self.next_observations:resize(nSamples, observationSize) 38 | self.mask:resize(nSamples, 1) 39 | 40 | for i=1,nSamples do 41 | local index = torch.random(#self.replayBuffer) 42 | self.observations[i] = self.replayBuffer[index].observation 43 | self.actions[i] = self.replayBuffer[index].action 44 | self.rewards[i] = self.replayBuffer[index].reward 45 | self.next_observations[i] = self.replayBuffer[index].next_observation 46 | self.mask[i] = self.replayBuffer[index].done and 0 or 1 47 | end 48 | 49 | return self.observations, self.actions, self.rewards, self.next_observations, self.mask 50 | end 51 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/lua/utils.lua: -------------------------------------------------------------------------------- 1 | require 'layers' 2 | 3 | local utils = {} 4 | 5 | function utils.mlp(dims) 6 | local model = nn.Sequential() 7 | 8 | for i=1,#dims-2 do 9 | model:add(nn.Linear(dims[i], dims[i+1])) 10 | model:add(nn.LayerNormalization(dims[i+1])) 11 | model:add(nn.ReLU()) 12 | end 13 | 14 | model:add(nn.Linear(dims[#dims-1], dims[#dims])) 15 | return model 16 | end 17 | 18 | return utils 19 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/python/README.MD: -------------------------------------------------------------------------------- 1 | # Run 2 | ```bash 3 | python main.py 4 | ``` 5 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/python/dqn.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | from tensorflow.contrib import slim 4 | 5 | 6 | class dqn(object): 7 | 8 | def __init__(self, ndim_input, num_actions, discount=0.99, model_path=None): 9 | self.num_actions = num_actions 10 | 11 | self.discount = discount 12 | self.sess = tf.Session() 13 | 14 | self.inputs_tf = tf.placeholder(tf.float32, [None, ndim_input]) 15 | self.q_values_tf = self.get_q_net(self.inputs_tf, num_actions) 16 | 17 | self.action_mask_tf = tf.placeholder(tf.float32, [None, num_actions]) 18 | self.target_tf = tf.placeholder(tf.float32, [None, 1]) 19 | 20 | loss = tf.reduce_mean( 21 | tf.pow(self.q_values_tf - self.target_tf, 2) * self.action_mask_tf) 22 | 23 | optim = tf.train.RMSPropOptimizer(1e-3) 24 | 25 | self.train_op = slim.learning.create_train_op(loss, optim) 26 | 27 | self.sess.run(tf.initialize_all_variables()) 28 | self.saver = tf.train.Saver() 29 | 30 | if model_path is not None: 31 | self.saver.restore(self.sess, model_path) 32 | 33 | def get_q_net(self, inputs, num_actions): 34 | with slim.arg_scope([slim.fully_connected], 35 | activation_fn=tf.nn.relu, 36 | normalizer_fn=slim.layer_norm): 37 | net = slim.fully_connected(self.inputs_tf, 128, scope="l1") 38 | net = slim.fully_connected(net, 128, scope="l2") 39 | return slim.fully_connected( 40 | net, num_actions, normalizer_fn=None, activation_fn=None, scope="l3") 41 | 42 | def act(self, obs): 43 | q_values_np = self.sess.run(self.q_values_tf, {self.inputs_tf: obs}) 44 | return np.argmax(q_values_np, 1) 45 | 46 | def get_targets(self, reward, newobs, mask): 47 | q_values_np = self.sess.run( 48 | self.q_values_tf, {self.inputs_tf: newobs}) 49 | 50 | return np.max(q_values_np, 1, keepdims=True) * mask * self.discount + reward 51 | 52 | def update(self, obs, action, reward, newobs, mask): 53 | q_targets = self.get_targets(reward, newobs, mask) 54 | 55 | onehot_actions = np.zeros((action.shape[0], self.num_actions)) 56 | onehot_actions[np.arange(action.shape[0]), 57 | action.astype('int')[:, 0]] = 1 58 | 59 | self.sess.run(self.train_op, {self.inputs_tf: obs, 60 | self.target_tf: q_targets, 61 | self.action_mask_tf: onehot_actions}) 62 | 63 | def save(self, save_path="/tmp/model.ckpt"): 64 | self.saver.save(self.sess, save_path) 65 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/python/main.py: -------------------------------------------------------------------------------- 1 | from __future__ import division 2 | from __future__ import absolute_import 3 | 4 | import gym 5 | import tensorflow as tf 6 | import numpy as np 7 | from dqn import dqn 8 | from replay_buffer import replay_buffer 9 | 10 | # Parameters 11 | BATCH_SIZE = 32 12 | NUM_EPISODES = 100000 13 | MIN_SAMPLES = 2 * BATCH_SIZE # Minimal number of samples for an update 14 | EVAL_EPISODE = 10 # Perform evaluation each EVAL_EPISODE number of episodes 15 | NUM_STEPS = 200 16 | NUM_EVALUATIONS = 10 # Number of times we run evaluation 17 | INITIAL_EPSILON = 0.9 18 | MIN_EPSILON = 0.1 19 | EPSILON_DECAY = 0.999 20 | BUFFER_SIZE = 100000 21 | 22 | env = gym.make('CartPole-v0') 23 | 24 | ndim_action = env.action_space.n 25 | ndim_obs = env.observation_space.shape[0] 26 | epsilon = INITIAL_EPSILON 27 | 28 | agent = dqn(ndim_obs, ndim_action) 29 | replay = replay_buffer(BUFFER_SIZE) 30 | 31 | for i in range(NUM_EPISODES): 32 | # Training 33 | obs = env.reset() 34 | for j in range(NUM_STEPS): 35 | # Epsilon greedy exploration 36 | if np.random.uniform() < epsilon: 37 | action = env.action_space.sample() 38 | else: 39 | action = agent.act(np.expand_dims(obs, 0))[0] 40 | 41 | # Perform an action, observe next state and reward 42 | newobs, reward, done, info = env.step(action) 43 | 44 | # Insert it to replay buffer 45 | replay.insert(obs, action, reward, newobs, 0 if done == True else 1) 46 | 47 | if done == True: 48 | break 49 | else: 50 | obs = newobs 51 | 52 | if len(replay.deque) >= MIN_SAMPLES: 53 | # Decay epsilon 54 | epsilon = epsilon * EPSILON_DECAY 55 | epsilon = max(epsilon, MIN_EPSILON) 56 | 57 | # Sample a batch of samples and then update 58 | obserbation_batch, action_batch, reward_batch, next_obserbation_batch, mask_batch = replay.sample( 59 | BATCH_SIZE) 60 | agent.update(obserbation_batch, action_batch, reward_batch, 61 | next_obserbation_batch, mask_batch) 62 | 63 | agent.save() 64 | 65 | # Evaluation 66 | if i % EVAL_EPISODE == 0: 67 | total_reward = 0 68 | for k in range(NUM_EVALUATIONS): 69 | obs = env.reset() 70 | for j in range(NUM_STEPS): 71 | env.render() 72 | 73 | action = agent.act(np.expand_dims(obs, 0))[0] 74 | newobs, reward, done, info = env.step(action) 75 | total_reward += reward 76 | 77 | if done: 78 | break 79 | else: 80 | obs = newobs 81 | 82 | print("Eval after episode #{0}, average reward: {1}".format( 83 | i, total_reward / NUM_EVALUATIONS)) 84 | -------------------------------------------------------------------------------- /frameworks_demo/Gym/python/replay_buffer.py: -------------------------------------------------------------------------------- 1 | from collections import deque 2 | import numpy as np 3 | 4 | 5 | class replay_buffer(object): 6 | 7 | def __init__(self, max_size): 8 | self.max_size = max_size 9 | self.deque = deque() 10 | 11 | self.obserbation_batch = np.array([]) 12 | self.action_batch = np.array([]) 13 | self.reward_batch = np.array([]) 14 | self.next_obserbation_batch = np.array([]) 15 | self.mask_batch = np.array([]) 16 | 17 | def insert(self, obs, action, reward, newobs, mask): 18 | self.deque.append([obs, action, reward, newobs, mask]) 19 | if len(self.deque) > self.max_size: 20 | self.deque.popleft() 21 | 22 | def sample(self, batch_size): 23 | indices = np.random.choice(len(self.deque), batch_size) 24 | 25 | self.obserbation_batch.resize(batch_size, self.deque[0][0].shape[0]) 26 | self.action_batch.resize(batch_size, 1) 27 | self.reward_batch.resize(batch_size, 1) 28 | self.next_obserbation_batch.resize( 29 | batch_size, self.deque[0][0].shape[0]) 30 | self.mask_batch.resize(batch_size, 1) 31 | 32 | for b in range(batch_size): 33 | other_b = indices[b] 34 | self.obserbation_batch[b] = self.deque[other_b][0] 35 | self.action_batch[b] = self.deque[other_b][1] 36 | self.reward_batch[b] = self.deque[other_b][2] 37 | self.next_obserbation_batch[b] = self.deque[other_b][3] 38 | self.mask_batch[b] = self.deque[other_b][4] 39 | 40 | return self.obserbation_batch, self.action_batch, self.reward_batch, self.next_obserbation_batch, self.mask_batch 41 | -------------------------------------------------------------------------------- /mb_demo.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "mb = require'mazebase'" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 5, 17 | "metadata": { 18 | "collapsed": true 19 | }, 20 | "outputs": [], 21 | "source": [ 22 | "g_opts = {games_config_path='/home/sainbar/MazeBase-public/lua/mazebase/config/game_config.lua'}" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 6, 28 | "metadata": { 29 | "collapsed": false 30 | }, 31 | "outputs": [], 32 | "source": [ 33 | "mb.init_vocab()\n", 34 | "mb.init_game()" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": 42, 40 | "metadata": { 41 | "collapsed": true 42 | }, 43 | "outputs": [], 44 | "source": [ 45 | "g_opts.game = 'MultiGoals'\n", 46 | "g = mb.new_game()" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 53, 52 | "metadata": { 53 | "collapsed": false 54 | }, 55 | "outputs": [ 56 | { 57 | "data": { 58 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAKAAAAFACAIAAAC0nuBMAAAfpUlEQVR4nO2de5RcVZX/v/uce289uqo7nU53J+nOg9ArtNCQCPmRCIGAAyLLiLw1AgMsEIf1MywURdFBZZjx7cgv66cooigMMiI6o46PkYckoGT4IZAgJsQgTZ4knYQk1fW4955z9u+Pe6vTeZCQ2JU2Z53PqgVVp2+d3LrfOnufXXefs4mZ4bAXMdon4GgsTmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOV4y59/7j3z33X0lK69/7Z95+CY5sIht2/dUfrlw78d2PRaQ/ufMGHCvj7XEcOy5S9ceNH7stkcgGLzmD3+Wtq5/ZDbd2zf+tgjv/QWfe1f+9dv7F+/ccRPPSPpoYceWvbM0w3tf+HChSPe8+Hkzju/XW2+RHTOBbB17z+34JDbB7ff9+CDD3q5XHbRKVjYlx5Ad4Gv28d5HEL7B47PAmh0/0c6RCSD1pZJZyUv/3AXATjpuj23XTiEdiEzwInOB48+65bemDz5w1100nV80nWcyJb895DbJx7/QQDe4f0sjn3QPeeOPQZi8mRIuUNuB+DVauENv8cNv9/179Fd+z6Pg22/+4Xal4BG928He9vYkWr3stkMMGJ+cXj7wqezABrdv2WMtA/+ovPBo4/zwZbTcB+MkfOLw9uzMvXBDe3fDhrrgxsUp157fOqDG9q/ZTgfbCHOB1uOi4P3178duDj4Dfu3DOeDLcT5YMtxcfD++rcDFwe/Yf+W4XywhTgfbDkuDt5f/3bg4uA37N8ynA+2EOeDLcfFwfvr3w5cHPyG/VuG88EW4nyw5bg4eH/920EDffDkKZMB/F3PnuvUADyzbnBW9z5W+b3J9kdXb584ceL2LY3tf+/Djiw6x7evW3qj2fLz5OWWdc8AGNc9a4/DDqE9qu3o6voRRVG0ePHiRpx6EASnn356HMcN7b8RPR9OlFKPP/54I3r2fX/evHm07Lln3fpgm7nm7y9vUM8ZSYsWLWp0/+zYL259sOW4ONhynMC2c+2VVzSo56ykRYsWNbr/0fZxf+u4+8GW40y05TiBLcfdD7Ycdz/YcpyJthwnsO24ONhuXBxsOc5EW44T2HJcHGw5Lg62HGeiLccJbDsuDrYbFwdbjjPRluMEthwXB1uOi4Mtx5loy3EC246Lg+3GrQ+2HLc+2HKIed+bAYwUja5P7NYHH4BG+wC3Pnh0afguO2598OjiwiTLcQJbTsNNtNsna3RpuMDufvDo4ky05TiBLedw+GC4+8Gjx+Hwwe5+8CjiTLTlOIEtx8XBluPiYMtxJtpynMCW4+Jgy3FxsOU4E205TmDLcXGw5bg42HKcibYcJ7DluDjYclwcbDnORFuOE9hyXBxsOQ0XuNHrj0fuTO2k4ctH3frg0aXhAjtGFzfJshwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcrxly1+48KL3ZbM5AMXmPdfhl3ZuP+T2Hdu3PvbILwc2vebqB48i3p13frvafInonAtg695/b8Ehtw9uv+/BBx984dln+tdv7F+/caTOeIiMpIceemjhwoUj3rNNeEQkg9aWSWclr/9wFwE46bo9l/0fQruQGeBEVz94dBEA1i29MXnxh7vopOv4pOs4kS357yG3Tzz+g4f3szj2gQege84dewzE5MmQcofc7hh10m2U3kiSv77d7ZM1uuy5T9ZI++Avuv2iRxfngy3H+WDLORw+GG6/6NHjcPhgt1/0KOJ8sOU4H2w5Lg62HBcHW47zwZbjfLDluDjYclwcbDnOB1uO88GW4+Jgy3FxsOU4H2w5zgdbjouDLcfFwZbjfLDlOB9sOS4Othyvc3z7uqU3mi0/T15vWfcMgHHds/Y47hDao9qOrq4f7dw2AFc/ePSgOI4ff/zxRnTt+/68efNc/eDRpeH1g5c//5xbHzyacIO55u8vb9CZZyQtWrSo0ed/pLNnHDziuPXBo4vbo8NynMCW03AT7eLg0aXhArv7waOLM9GW4wS2nMPhg+HuB48eh8MHu/vBo4gz0ZbjBLYcFwdbjouDLceZaMtxAluOi4Mtx8XBluNMtOU4gS3HxcGW4+Jgy3Em2nKcwJbj4mDLcXGw5TgTbTlOYMtxcbDlNFzgyVMmw60PHj0avnzUrQ8eXRou8N8Ig4ODq1evHhgYKJfLzc3N48eP7+npCYLgwO/UPogBA8EMYoAAQJIOAIAiCA2WhtjAAJCQABQ0gb1dl1bCZAAAIYRWBAWSkIDwEY78px1Gw0306LJq1ar77733Fz/84Utr1kzL5TqIikQl5jVarw/Dt77lLedffvn7L7vswKvIabf/EzPIpE0sAKb6fJWYASISAgYAGBBggKj+kkkQJECHZ4o72guUG8WqVasuOOec8fn8RzOZJwCdXNthjxB4GLg6nx+by117+eUbN27cd0daspFsiBnMtOth5LAHMZNhMIM1sZJsAmMC1h4bT3MQsRezNCxZS9bSpM891l6jr4OFAiul/uW229rz+TukDPfSde/H68CnMpn2QuG+e+/dR3dGMtfVTeRMXw5rMZQ+NLF6I4FJJ/Kn3xiPtcfKCXyQhGH4rjPPnNvUtPFNSDv8sQLoLRQ+9IEP7NGhYTJ7CJyOZpihMW2IjWQVsApYSzaUvitRnZOXYAYbsK4bAB2wChp9QayaZGmte48+2lu//o9KyYN/+yBQBK6/6qpv3HPPUKOBACBYAImrBUgnnpaBxCkLACxgJADImMHJXIzSS0ugxFUDbEBgYoCIJSBAjZ1k2fNLFjNfMn++WL9+xSGpC6AAlIDlDz30yY99bFizACTYB2eG5qSJugakITWkAZgMZAwZG7AGTCI/EUBgCeODfbAP+IBgEIMNaab4r/7cB8CeWfQ3v/719U8++aJSf00nBeBng4Mz77zzrHPPffvb35400m6HCJBmSI2c4SxzEwCDWFAoUQVFBnroUIYEZ2Hy4AwjABTkIFAFIkATDofttMREb9q0qa+nZ8ng4FtGorf/Am7q7n7h5ZeDIDBMggKjY2PY83MAA3GoA6MnECayaVcxfD8isVX4A4Y3C654kpRRvshFWggaa1SnjoqEgpEVyLXZoGSwg1XF9wiaIRt8/Rvt5A8PNy1c+OFs9qBmVft/nFss3n333czMTEqBWRiWzH4UNVcrU9eunfqFz+Fd78TYMWgbg3FjMX8+Pv1paHO6MVO1KTI3xXELc9+KFVP+4VqceALa2nDCTFx3PV5a3VUNJzK3MDexoUZfGRtG8ODg4OTOzpWVSsfI9fl74AOTJ/+xv584C8GAjE3NF/lYTfzR90o3LtzEErNPwbTpmNCNFX/Cihfw8kvo6sKiO5vnndmh9EYyba/8eeJlC5a+2o9Zb8OsuVi3Hj/+d0zowG8eOXZC1yDz1iBQQG3kznpfNPobdBi4//77z2tpGcHhmzymF4vPPvssK585G8dScXNVdX/3OxiXRd9k/PYXc2qliyrlOZGaXQ3P2Lb14ts/1dFexIxjUS7NZp7I8fQbrkZnDv/0CbB6b82cUlbnfvubM1qbcO2VqFVmGO6MueFhkg0CX3Xppd8YaXUZWJjJfPELX+AYzIHmoFSbsnrVKa1FTGjDSy+cHUVzwmhiGLbEptXw+KqaXovnf+AadIzFffegFk6Iw67zz8HMHphodqXaWdJjdkTH7Ch9sLkZc05CWD2zqtsrjb/+Nsyin37qqZsa0O1pYfjDxx7Dx6WKI89vymXa7v3uMzC49kOYOH2n8P9CPCCIAFSqDP91otrNn+oiWt/RmQs8EMU/+enptSgkf4PvV2AKntf5l79sE4RxbWBThYk9QQc8jb+WRn+DGo0xJvC8qAEjeDnQN2UKa2m0qGjU1Kx3nIHmHP70l94St4XGYw02YAXmAnPW8JhSXKxFE5m7WeVN5GseX9WdpTgfc0+l9p6f/WxO73QU8/j+d8F6sjZ5zQ2fZB3xI7hcLvtC+A3oeQywvVSCkDAqK7zBqlq3BgR0TMgG2OGRBgPIQuV1tZX9Qo1CGXislDFVMCAUYwsEZ0Th+z9Y/bGFqzlGFOGjH5l62eUzq+HSXEYAAg0ew0e8wHEcZ+Wh/XJ1ADKA0hoqIkkAE+TG9Zg0CUTCwBiwEBmowuOPb3l66bbIdJRNNch5plbOZKNbbpkqaA3DAKSgpY+PfBz9q/HUEnzt//Sv3dz/rW+8A2YFc0iN+G4Op9EmotHEcRxIOeL2mYE/Az0TJrAGaxgWkZrd6qMlQCk+tWSEZhnHklXnT36A9gJa8iiMQaYZTUVMPRrVaEakJHM+5qYyN9W4PVY9pZ0nl3ZcdfbfobkZt30WtdLRzIVGX58j/rdoz/M6Wlr6G9DzamBqV5fRgCCGVCqePRsC2PDKQIE62LDn+bo6cMGC7o2bezdvnTkwcNyWLReHBiBILvpojqsZT+Xy8AIue3pDoWldofjM1785R8d48gnIXIvhSgNOfDeOeIEB9PX2Lm9At8uFOH7OHOED8Go6Jhmeelo2n8WD96+qDHZIatNRTRZyiDfI7Mue/0ePXobZIAEwfC/Pqul/lrz+zJMlFTXBaHgK0UAUbegc3+T5WLsOlWpIQ2khDcMGgc8877xHcrkR7/bR5uYzzj4bEBHiQArPK1/y/hmRwn/8Gwb6p0F1yqAprJThA8RC5qVu8+N2aSAJg5VtFHRcej7+4ZowNpNYtkD4yLb6mcm/X/pSqDH5KAgRNz5IOvJ9MDO/8sor43K5kY2U1gKt+XytVosNIg3mbGRay+WZD9ybbc+hbyp+86uxsZ5TCfs0z6yGfTt3znzisTHvOhtNOfT2ohYdWy71LLwSE4v49D9hezi3VJlVqpy67IW+c9+NphY8/NiMKO5h9ht9cWwQmJnfOXfud4lGUOCbM5kbr7+emZmzzH65AuZiqMeWqj33fB/H9aHQgp5jcfm1+PAthQVXofcE5FvQ1Iyz34Ennmit6nbFE1etPGH60Whpxlln4dZbJ111FY7pRctY3P5F7CydwGaSrmQafWUsEXjx4sVTC4U3k4H1Zh4bgbZ8fs2aNcysamDOMGdUDGapuKUUTX9x9Qn/eBvNOxvTjkW2FZkiju7FZVfiV7+ZUa2dGsbdoc4qbqnF0//8577P/iN6e1BsQqEZZ5+DXz8ytVw7zuh2jppZtTT6ythwNynhveedN+3hhz9fG4GbM+8pFN56442fvf12AGAPZJBeJAY8g5zmotFjGAEoBiC1LxmCGaLM3naICqAAAfaZm4wew8ho0kxMFBNVfewUVE3TPrzGpuzYI/DAwMDJfX2fHxh431/3ia70vFXHHbf46afrafFJLhUDBE5+eDJMACQDBE0AWMBQkgcNj5lApp69BYDAEAAxOEnmEUhsBQGAaOxE2h6BASxbtuzU2bO/H4YXHWoP/yzlrVpv3bp17NixaRPXp7oswRkAECGTrqfUMSVJdWnSnWABJgijkCTHE0AAS+gsCBBVpjSnJ03Jo8ZefxvCpCFmzJixcvXqi4GvikP5XLcGwfc6OrZt27ZLXSS60i6ZAaRZWsNbGTAgA5EMbgNgSF1OPTvqdh4M4lT5hmOVwAC6u7tXrlx577RpFzc1bXzT7/oTMLdY/H8nn/zUsmWtra27/Y0zgA8SIAMRQoSAAYOYiImS5UoMgEEa0EBM0GlqLXGSZKnIsFdlr2rIMMhAGkhAHA6NGz2LGxWq1ertn/lMWz6/MJd7fr8T5t8CCwqFjubmb9555777UnnW2fqCBkqXNRiwoWQdSpLpzmb3VHhOF7NohmaqP4ZeBsYE9Rz6xmKVD96DzZs3f33Ron+7+25TLp8FHDM42AkUgRLwqhAvFQqPKtUxfvxVH/rQNddeWywW992LSVYXJuaYEmucTqyY0ttxpAxpAxaASGZPJJhAnEy/E9+sGUaAwZSmyAs9NPwbh80CD7FixYolS5a89OKLA2vXlsvl5paWCUcd1XvccfPmzZs6deoB3pzMnFkwgSEBQ9AAiAVYggOQYVKaYg3IZMnokMDGAIDOgAAZMxkCwxC0BAFCJ1+Hhn52Wvbcs66+r824+r5247n6vnZjW5jk2AMnsOV4bh8ru/Hcfs5240y05TiBLcdz+znbjef2c7YbZ6ItxwlsOS4OthwXB1uOM9GW4wS2HBcHW46Lgy3HmWjLcQJbjouDLcfFwZbjTLTlOIEtx8XBluPiYMtxJtpynMCW4+Jgy/FcfV+7oSiKXH1fiyFmqi9WB5KiTiSYSIMZBunmQUh3/qlvOpOuTE/2UdYCAkYaA/aQLHFmAijZWchr+H6bRzYHU5+YAYlkg+k4rZDHBPaBtKRe0mWiZfIi2YHApEVud8Ei3SsGAO27RtfQ/iFiWFO6P7FJT5TcJO7N8ebqExtglyy7MMP2cpFpS33Iepw+Sw5KhEx28UqEE0Nv2O1ciJNWonSjJ5McnHybKDYwJGSD94CygmRHpeTprkYDDC9rmCjCMt0kZNjBqS1NbC0xp+N7SDKvvmMbdvtqMCgtq5luPTK0M1t9OCdmhQHIVGMh0r3gTL0PZnJD+EBQIifvsn/JS9LDLCLASQ3TYQcwkl24gMRSmiFfCqbUe4I9hl/f6kUBepd3ZUlDwzdVOS2QSgQYCQFOC2sbQEiWAEARwJRsJwNjXKB9IJiSwTe8qb7TGnhXYXiIXXvzDKmbVEZN3WvMNMzOctp7vV4qgSDqaiV/3F2a4adQH8KUfHHAoGReheQrQkxExHs5dsfe1PczlMDQDDZtH9qBRwxXbvgRNDSGh9xwsskeD72fDKclSyj9UnB6NEsmIJ1Lp6eRbr+X2mDBJAEQ4l3ful2mhlIr3vhd649oDDyAhPHqsygF0kycXDuGBCCh0yvPYth1Nom6DGFg6sNRACAYquuWlDgFACY5zAcIcJbZN/CTLaLqX41YUlmIOB23ECADFiANMJPHnCMWhBBcr+N7ODZkPLI5mPrEKplwUX3vSw3ByIEzxmQAgBQoBJUlKSIC4AFkNEuZFGmSAMUmIuGDC+A2mA6tMsaw8GCgyduuxMtZUSIYCAETp3M3DXhgFAZrHYEXZP3NiDfD96DYOeEDwJooMBzW6xMLQMQ6MLpjeH1iI7bCHzBmsxChBwUiQALGIFurtma9KRSOJfihHvAL24x81UNNx7HMFIiNYG3I8wE/1rE2XtYfUwszr63Fg//e/8ijeP659IeQOW/D7Dm4+ZOnM9b4ZrP0FKtIMzzf0xoGzaGeePPH/rh4Mf7nyY5CYRvHinyBYRWxHftCaM1SCgYRRBzntBq7ZSvuv6//iSV4aikIIIE5p+DEE/GZ2+YqftXoTT7iao2bCp2axz1w/4sP/gBP/w6BRNdkzL8QN31yBtHmDFVMPAg2AXPA7FeqUOwzd5fLs35wT7Yzj/Y8LjgXC6/Hl76Iq65G33FoKeKYY/DTn41nnqY0mD1mXzEUd0Tx6bfcgkwTpvUi0ifGCsxZpezc7HQk0QGzz5yNNJjzUdxz/7c727MY14R3nY2F/xuf+zyuuAKzZqI1h74e/OpXGeZjmVuYx+va3M/dhrFF9PXiK5/Dv3wWx05DZwvOeQeqlflaT2HOgiPBnAtrYM4pzpYqU753V9Cex7GT8cRvZpe2n1OunFyqzKyGp9cqF99+G1pb0NsLzTMijZpBzIUwnlytnfLB69BShJ/F1B5UwlOYg0gn9Uoc++Ug6xOfNBM7tp8UVcdyOGnJrye0F3D9dSiV3rtz54kmPsVEl119BdqKuO1WhOExmj1wmBSdkDEHEbeuWNU1LofxBbzy8onV6GjF3cwTQz02VO2ap+4szb76GrS04Dv3QHFTxLKmJ/33z7tn9qKzGWfNg+dhWg/i+ExjgoiFZjna1+9vnoOsTzy2Gfd9B2x6uDLlq7ehvRkrVk+q8lTDnZFqK9WmrPrzrOYszjkTsZoVMnkIsqxqLCEoX4mL//kfa4zBwpvauo8SNbyc/PjtC8nMbLYUCvj4J072vKffMh06jqUvDItf/OJVFeHLX24+/6IbJnT/MzFUrYqM8n1ZQ+zSdg6Ad1D1iScxrZ16dIcqb/f84LTTO8tm05SjtcK6APBkliEmTWkHo/8VkPCMYXDUZCKfOVOJWiI9953nYlwL/rT83aVqn+IWZTJGC2aPYzBnVZjfUe423MummY3U7Ctu/c2j0OoEHfZUd85vzmH6ZHBtDrNkN4LfDAdVn1i1lE1XzYw3YYZNMaqOi/T4iItlJm0k63FxdML3viOKOVx9Jcphr2byYAxl8trslP6YOGzqX4MYmNIzxgsQ6zgrfW0IYU76Y+NBTwbFQi5fGtzcFOSlB3BMonrW25sYK4U0nhnDBn4AFVY9X0McprIERzYHU59YBJ7RyqMaBRmYkpcpgaBAAQJBhbA2bm1/4StfNsLHgiuOJSoR2INUiGvS96oRgzMbN6B7EmIeJCoRVRk+U/PiJ7Y89eTrhosqbiZfCvGaL6Jbbu4V/g6OtiMIwQokDDgWCBnS8wEBJpH8Qu7YDwdTnzibCTiseaL8qVunkVchrQD40jMmq0z7mjWF88/7/br1eO9VeNu8IOMNEuBBGDBrjnNBEEbMjHVrEIiCYOMLRHGU8cXmjbhjEZQqxbqkASkwfQo++fEONbjVa/YArQyEhEnu8VN6MyPZ+Hy0r9/fPAJgZuJM4MFg7RoEMqcYAUmllEfy9W34ylcQmc06gNLIaUyZjI98NMgXASFMyEK2VcKWNf3ZSy9+Zv06XPJeLLpjLtGLOt4hJAQiDekRIVZlSYNnnAo/wvqXqhmVF4yM7yPetuD97RvWTdy2ve+1LWdu2naDCKAUTOR7TT5MWenIlxlA1u82w9DwHBDH/jio+sQDWy+oAhWFXLbZhAokRHbclsG2pc/RJZc8v7YfCxbgW3eenqH+jKz6fgCGQFZGkRIQ8GKmTXPmIJD46Q+XxNUW8FhoD56E3BHkB7T6Yy73ai1aKyU8Dwa+UQpCCImQQ+x9wyO90+nYHwdVn1joTZCIgDDUQgbAmDBuf+gnL86fv/LVfnziE5l/XXSy9FeSv41VLa7WIIUw0AigwRJxiJcvvXxqpPDd723e9FpWR51K5eMIcchAVsomHWfzfgcrxCHAUnhZo7WAFARAg/SuaRUhqSMzmhfvyOAg6hML1WY0GJD5nPHGqLj7gfv/dNNCeAY//dnUD3+sL1fYwHK7Ro088nMBR0ZoAx/CgAWinBdP6BJf+78dW0uY/54ljy/ZIoP/JYJZoT6upo4xeOvixRsuuuBb0Mg1QWMrRCykF8baR54BYpYG0uye0eXYL4oNGXgImHdMmhYu+mZ27QbMf/d/PvzoBmWON16fwQm1qLdUmva735UvvOjnngRJxDxYiTKrXil86AaA8cB975t7yjyjvLA2RnFPNZ4eYYrRPvnk+eQDJONI+EQMg4EF1xwVy81f/SouuHhzW+evTz0Nk6e29ve//rslGBiAAM44Ax++AZniWpjXjdGezCltPCkAkmZ4sUXh5tAHxKMsSFeqtXzO5+ya+Zd013j1V76MC9+3bXzX0jmnoL298NrGweeexZpXQQazZ+Ezt00gsTVSY7/01SdrMd56Ip566i9PP/laJghjVLRQfiar4tc/eXN3LhcTqwCsQAaCQBmtlZGB1hNeWd364x8vf/S30dp1WL8Rnofx43HqyVhwWe9pp02Q9JIntkgJKAWvReuqlCoOezrbVnZ1YdnyY0VmJZLMjt2Sxxx7okOSmQwArULpSY1CNe5csyb7wP0vPPEkr12PDRvBCt1dmDMbVyw48azT2ki8EnmbyZvWNuZ5ZjDDY/ga2QBlDfIRa0yajBeX9xizmmL2BJjSREtiCIY0nNNcZM4zB4AkxMRIanlpWSGq+SgLTlL36umYZIxpjkwXgIxYS1RCGgQ7W71fDqo+MaoQ20FViBhoUqaTOQdoySxYJTkYhqCJjCxLuUVgkGKWIo1dh3JpBQAmk6RjiyQFJCnZRcIQDEiyRwywAZmkECOYmKAgAOOjXnAXbiJ9QA6mPjGjXpNYpEGKqOfxcD2frp4hyQQGPI/rKVeMobxoJLNgAtWfgwmQMIKklkgKuwkgGv5eAkkQQYJ1ejyLemzseAOGSguyAGfAgAiBpJBpPX2SkoEmwfWcOjIQYLBGPduRpAQJaDAn7yOTIRJemoC5W4p6PRU7zY6uJ9azQJoNNJRNZ+o5vSkizeNlZ5nfLCz3vlYE1BckpAcBBlSfsxLSkTeUy5xaS8HQNDwF0+zKqjTD1qcMy4809QFIuwY3gH2seEjqXNdXNgDameg3hW4CKZBO0ssB1K/q8BVGCYkX5froIUAKCMEaAEySGK9NfciSEWB49STaNNk5ScKtH54wlP4+lHKNYYm4VD8ZBpm6DXGiHhQC0MMmpPWLPLSsIV3rwGmm866Ec5EuITKAEcl4rn9T2IcBw0vWOTEYIAMfECJVN7HmyQRKJv84JbnRTEmd3OQfJwiwSUrTEyWeA0PWwwXCB4CiutsTQ3npREMp0MnKhLQ+MYMIUgLEGgCTTt9B6ahmcLKqyQP5EGCkS4kSC54+2Bv23Rl657CbB0NjN5n4seD0vbvM+DDD7dgvpIet5hI8bA1gqgUEQxigfpwHFjACzPUVgEk/PHyenD5j+v90dLQYASR+BwAAAABJRU5ErkJggg==", 59 | "text/plain": [ 60 | "Console does not support images" 61 | ] 62 | }, 63 | "metadata": { 64 | "image/png": { 65 | "height": 320, 66 | "width": 160 67 | } 68 | }, 69 | "output_type": "display_data" 70 | } 71 | ], 72 | "source": [ 73 | "g:act(1)\n", 74 | "itorch.image(g.map:to_image())" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 30, 80 | "metadata": { 81 | "collapsed": false 82 | }, 83 | "outputs": [ 84 | { 85 | "data": { 86 | "text/plain": [ 87 | "-0.3\t\n" 88 | ] 89 | }, 90 | "execution_count": 30, 91 | "metadata": {}, 92 | "output_type": "execute_result" 93 | } 94 | ], 95 | "source": [ 96 | "g:get_reward()" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": 37, 102 | "metadata": { 103 | "collapsed": false 104 | }, 105 | "outputs": [ 106 | { 107 | "data": { 108 | "text/plain": [ 109 | "{\n" 110 | ] 111 | }, 112 | "execution_count": 37, 113 | "metadata": {}, 114 | "output_type": "execute_result" 115 | }, 116 | { 117 | "data": { 118 | "text/plain": [ 119 | " 1 : up\n", 120 | " 2 : down\n", 121 | " 3 : left\n", 122 | " 4 : right\n", 123 | " 5 : stop\n", 124 | " 6 : toggle\n", 125 | " 7 : push_up\n", 126 | " 8 : push_down\n", 127 | " 9 : push_left\n", 128 | " 10 : push_right\n", 129 | "}\n" 130 | ] 131 | }, 132 | "execution_count": 37, 133 | "metadata": {}, 134 | "output_type": "execute_result" 135 | } 136 | ], 137 | "source": [ 138 | "g.agent.action_names" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": 45, 144 | "metadata": { 145 | "collapsed": false 146 | }, 147 | "outputs": [ 148 | { 149 | "data": { 150 | "text/plain": [ 151 | "{\n", 152 | " 1 : obj2\n", 153 | " 2 : info\n", 154 | " 3 : goal3\n", 155 | "}\n" 156 | ] 157 | }, 158 | "execution_count": 45, 159 | "metadata": {}, 160 | "output_type": "execute_result" 161 | } 162 | ], 163 | "source": [ 164 | "g.items_bytype['info'][2]:to_sentence()" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": 47, 170 | "metadata": { 171 | "collapsed": false 172 | }, 173 | "outputs": [ 174 | { 175 | "data": { 176 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAKAAAAFACAIAAAC0nuBMAAAgAElEQVR4nO2de5xVxZXvf6tq730efU433dDdPBroIBc72AYiXEEkoBmN+pFxYtQo0Qzxo2Nu7g1eE2PM4+bhOHmqM07PJ9FR89KYh/FmJsnNU5P4jAxjNGpUJKhA8xAaEDh9+py9d1Wt+8c+p2ke0oK9u8f61PdzPkhX71MW53eqVq29V61FzAyHvYixHoAjXZzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDL8Z7+05N/s+zsY6ZPOfh3u/f2j2ssHHX7zj2lX9z3+75tr6Ta/6RJkw7173rT8NTTz7znvIuy2RyAYuO4A35b2rv7qNv37N75u/t/4fX80z+u37x1/eatIz70jKR77733qcdXp9r/ypUrR7zn0eSWW26vNF4g2hcD2Hnwr5tw1O39u++65557vFwu27MIK7trF9Bt4CsOMY6jaP+747MA0u7/zQ4RyaC5aeppyY9/vI0AzLviwLQLR9EuZAY4wdngsWfTqquSv/zxNpp3Bc+7ghPZkj+Pun3y8R8E4I3uv8VxCDoW3nzAREz+MqjcUbcD8KrV8Mo/4Mo/7Pv/0W2HHseRtt/xTPWrQNr928HBa+xItXvZbAYYMbs4tH3l6iyAtPu3jJG2wV9xNnjscTbYclK3wRg5uzi0PStrNjjV/u0gXRuckp96+fE1G5xq/5bhbLCFOBtsOc4PPlz/duD84Nfs3zKcDbYQZ4Mtx/nBh+vfDpwf/Jr9W4azwRbibLDlOD/4cP3bgfODX7N/y3A22EKcDbYc5wcfrn87cH7wa/ZvGc4GW4izwZbj/ODD9W8HKdrgadOnAfirmQeeUwPw+Kb++R2HOOX3Ott/u2735MmTd+9It/+DL3tz0T6xddOqq8yOnyU/7tj0OIAJHfMPuOwo2qPqnilTfkRRFD344INpDD0IgiVLlsRxnGr/afQ8miilHnjggTR69n1/6dKl9NSTT7jzwTZz2d9eklLPGUk9PT1p98+Ow+LOB1uO84MtxwlsO5eveH9KPWcl9fT0pN3/WNu4/+q458GW45Zoy3ECW457Hmw57nmw5bgl2nKcwLbj/GC7cX6w5bgl2nKcwJbj/GDLcX6w5bgl2nKcwLbj/GC7cX6w5bgl2nKcwJbj/GDLcX6w5bgl2nKcwLbj/GC7ceeDLcedD7YcYj50MoCRIu36xO588DCkbQPc+eCxJfUsO+588Nji3CTLcQJbTupLtMuTNbakLrB7Hjy2uCXacpzAljMaNhjuefDYMRo22D0PHkPcEm05TmDLcX6w5Tg/2HLcEm05TmDLcX6w5Tg/2HLcEm05TmDLcX6w5Tg/2HLcEm05TmDLcX6w5Tg/2HLcEm05TmDLcX6w5aQucNrnj0dupHaS+vFRdz54bEldYMfY4jZZluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjhPYcpzAluMEthwnsOU4gS3HCWw5TmDLcQJbjvfU08+857yLstkcgGLjgefwS3t3H3X7nt07f3f/L/q2veLqB48h3i233F5pvEC0Lwaw8+DfN+Go2/t333XPPfc888Tj6zdvXb9560iNeJCMpHvvvXflypUj3rNNeEQkg+amqaclP//xNgIw74oDj/0fRbuQGeAEVz94bBEANq26Kvnhj7fRvCt43hWcyJb8edTtk4//4Oj+WxyHwAPQsfDmAyZi8pdB5Y663THm1NIovZYkb7zd5ckaWw7MkzXSNvgrLl/02OJssOU4G2w5o2GD4fJFjx2jYYNdvugxxNlgy3E22HKcH2w5zg+2HGeDLcfZYMtxfrDlOD/YcpwNthxngy3H+cGW4/xgy3E22HKcDbYc5wdbjvODLcfZYMtxNthynB9sOV77xNZNq64yO36W/Lxj0+MAJnTMP+C6o2iPqnumTPnR3l19cPWDxw6K4/iBBx5Io2vf95cuXerqB48tqdcPfvpPT7rzwWMJp8xlf3tJSiPPSOrp6Ul7/G92DvSDRxx3PnhscTk6LMcJbDmpL9HODx5bUhfYPQ8eW9wSbTlOYMsZDRsM9zx47BgNG+yeB48hbom2HCew5Tg/2HKcH2w5bom2HCew5Tg/2HKcH2w5bom2HCew5Tg/2HKcH2w5bom2HCew5Tg/2HKcH2w5bom2HCew5Tg/2HJSF3ja9Glw54PHjtSPj7rzwWNL6gK/6dE+iAEDwQxigABAkg4AgCIIDZaG2MAAkJAAFDSBvX0frYTJAABCCK0ICiQhAeEjTHX4qS/RlkD7/ZeYQabWxAJgqu9XiRkgIiFgAIABAQaI6j8yCYIEaFS2uG4GD4fxQADMvtmbwEPVMSAwmAAYAgsIyQCxAcGQ0DAElqh9K5K3kiEAEHGqw3czeDgokYUBGlQGYJDe1wKAQSAw8JrzxTCYQIAgRu27kv7kcgIPA5PBfjO3pjUABu9beCFgJACQBhkeXMABAZ1cTgCIwQaUzGI5CuN3Ag9DMseIJVDTaHDuDU5XQUPmojAM3m81ZyICaj3UFnOQIRAodSvsBB4WARDYAwRgAIW6tAbEtVmoiQxJA8CAzeDbKFm0JVjUd1QG0AwDMJOm2uROESfw8OyvgQBphtTIGc4yNwAwiAWFEhVQZKAHL2VIcBYmD84wAkBB9gMVIAI0jYIFdgIPD2uiwHBoDHt+DhCAiHVgdBthMptWFcP3IyN2wu8zZrvgAU+SMkqIXKSFoGZj2nVUJBSMHAD3ZoOSwR5WA75H0Jy2IXZu0rAIrVlKwSCCiOOcVi07duLuu9Y//BAeWwUCSGDhIpxwAj73+SWEjYydgoxSnudNXbOm9M83bVi9Ghs2Y8pULDwJV189ZdpUzgZlQIEHMGQ7lgZO4OEwGQgGZGyqvsjHavKPvl26auU2lliwCDNmYVIHnn8Ozz+DF1/AlCnouaVx6altSm8lM/7lv0y+ePmqDesx/yTMX4xNm/F/f4BJbfjN/bMnTeln3hkECqimO/4xTcP2ZkD5zNk4loobK6rjm9/AhCy6p+H3P19YLZ03UF4YqQWV8JRdO8+//tNtrUXMmY1yaQHzZI5nXXkp2nP4+0+A1YVVs6iszrr91jnNDbh8BaoDcwy3xxykPXwn8HDEYA40B6Xq9HVrFzUXMWk8Xnjm9ChaGEaTw7ApNs2GJ1bUrGq87O8uQ1sL7voWquGkOJzy7jMwdyZMtGCg0l7S4/ZEx+4pfbCxEQvnIaycWtGtA+l//m6TNRyeVHHk+Q25zPg7v/k4DC7/MCbP2iv8l4j7BBGAgQrDf5Wo+vFPTyHa3NaeCzwQxT/+yZJqFJK/xfcHYAqe1/7SS7sEYcJ4sKnAxJ5I20tyu+hhMZBSVExZsFi1KtIGy1d0IVineU/AABgG+VwBUIy9E6fqnq9PzvgCehfrmDNryed+Vcp6kxEfd99vtn38Yz/SMS64CJnsFkPRKAzfCTwcQsKorPD6K2rTRhDQNikbYI9HGgwgC5XXlWb2C1UKZeCxUsZUwIBQjB0QnBGF73xv3TUr13GMKMLHPtp58SVzK+GqXEYAAinPYSfwcKiIJAFMkFs3Y+pUEAkDY8BCZKAKDzywY/WqXZFpK5tKkPNMtZzJRp/8ZKegjQwDkIKWPj56Ldavw2MP4Z/+eX3v9vX/+vV3wTzPHJKf8vjTNvJvejRYw7CI1IJmH00BSvHJJSM0yziWrNp//D20FtCUR2EcMo1oKKLzGFSiOZGSzPmYG8rcUOXWWM0s7T2xtOcDp/8VGhtx3edRLR3DXEh7+C6qchiMTp7vSaXiBQsggC0v9xWojQ17nq8rfecu79i6vWv7zrl9fcft2HF+aACC5KKPxriS8VQuDy/gsqe3FBo2FYqPf+3WhTrGIw9D5poMD6Q9fifwMAgfgFfVMcnw5Hdk81ncc/fagf42SeN1VJWFHOItMvui5//ZoxdhtkgADN/Ls2r4j4deffyRkooaYDQ8hagvira0T2zwfPRuwkAlpJRvY8EJ/DoQEeJACs8rX/C+OZHCv30XfetnQLXLoCEcKMMHiIXMSz3ej1ulgST0D+yioO2978b/uCyMzVSWTRA+ss1+ZtofVr0Qakx7C4SIU3eSnMDDotiQgYeAec/UGWHPrdneLVj21/9+32+3KHO88boN3laNukqlGY8+Wj7vPT8hwPfgZwYGynvPvxBb+/DlG/6jFM3qr7y1v/Lfnvmz6unZJD1c8/E5mcAAaW+x3L3o4ckBeqAS53PFyPhR1HLvPetuvAEbejFxChYuQmtr4ZWt/U8+gY0bQIxFC/HZzzTPX+T5wn/phQnLzn56Wx/++4k46aSpvb29j63CK9vxsWvxv//n24oNr5rqdpFL9160E3gYdEgykwGgVSg9qVGoxO0bN2a/f/czDz/CvZuxZStYoWMKFi7AJRfPOWVJQcgNEDukyCjV3rs+uPs7f/7BD7B5K1jipJNw9TWd71jckPO3kw4hCHJ3quN3Ag8HeyBTfzbPgGeQ01w0ehwjAMUApPYlQzBDlNnbDTEAKECAfeYGo8cxMpo0ExPFRBUfewVVkITyeenGRTuBh0UAqEVZcXLjyTABkAzUYm5YwFASBw2PmeohsaQBgMAQSbwdYGpRs1yP8BLpbqTdnazhGJwALMAZMCBCQAMaAIORBLWLJEpScC3qUtXemAjNBJ0lAkSFSSdRsy5k578GLIEDJxmhHuZevwgwIAKBSdSuJwNCPSCea1fVriaMVmC0W6KHQzeAFEgDZkhk5GBc7NBPjwBhCJScaQAnwiZqSyQLcxJ5KQF4nHwt3BI99ghA1w6wAPWFecjpldpZB2bo2uOh2hRNIuIJNWlrQdaidrhBI32cwMNBESh58CvqmhmqHU2g+geoDGmTNHFyQkUwgdgQo2abhWYYkXw5jAEAkZx3Shd3J2s4SA+uogzBtemYIJKAeIYwgD54tU2u0x6MIK4/+eX6Gj8qtpGeevIJV9/XZlx9X7vxXH1fu3E22HKcwJbjuTxWduO5fM5245Zoy3ECW47n8jnbjefyOduNW6ItxwlsOc4PthznB1uOW6ItxwlsOc4PthznB1uOW6ItxwlsOc4PthznB1uOW6ItxwlsOc4PthznB1uOW6ItxwlsOc4PthzP1fe1G4qiyNX3tRhiJrCoZxRJ0v4IJtLgJKEEDZbtShKJ1NLG1NNUMEELCBhpDNgDgWGIKamuaQAv9Xybr4f+/v5169b19fWVy+XGxsaJEyfOnDkzCIKxHteR1SdmQNayH8YEpiQXE/sAION6Zon9sod4QJJiQux/4JzFvtwydOiEP4NJZsSQplraClMbaPq1+Q7P2rVr777zzp//8IcvbNw4I5drIyoSlZg3ar05DN/+1re++5JL3nfxxWN/Sv311Sc2wMEpf5LitnVkrWUwCVBSSpENsSE2YAM2xEyGSTNplpqlYWJOfgXDVPuRoRmGiY1kI2OmmCXrLOusZhkzGfbYeGN18Hnt2rXnnnHGxHz+Y5nMw4AeUk4yeYXAfcCl+XxLLnf5JZds3bp1bAaqJRvJJvlIad/LyCGvRJHkw5dmf0VYS9a1dxmTNSZbe0tSRjF5w9Cmmti1vjzDHjOxAWtKOjJc+zNmxEzJ22OWmj1W+UGBddIy6iilvnDdda35/M1ShgfpevDrVeDTmUxroXDXnXeO/mjZSOa6uomc9fkzRGaqv4ZckCilqfYVYWIjjckbk2ft1X8F0pylZO2HAnQ97wsluWHqK4AG7ytcDQKMZAEFneRyJBZJEb5afihhGJws/d4hlpQUiaLoPWeeuWf16h+VyxOP5I1rgHMLhdOWL/+X217D7UsHhkCSLm2Igavn9QHtS5dWr08sdE2OpEY812WimAngDADiqLad4noaJSYQxJAMMDjwHsjQ3NVJjj4G1bK6MSjZVyH5ihATEfFBhj1ttNbHzZrlbd78Z6WOtOZjF/Cf/f3F22/Xcfz1b30rlfEdiiOuTzz0ilomPQwxw0naNR58vyDEoBgUA7r2pah9bxJrq5MMUAZsksLWtf+ZBrNkKdknJjDX0jdS/ZtoJLEQQ9P9pQwzX7Bsmdi8+fkjVzehAJSAp++991PXXDPCgzscApBgH5wZTFs2WJ9YQ2pIkxQilzFEUlde1tKrJXaSYk0xI7HRMVPMZOp+DiclTgGASQ6pUS/AWWbfwAcnMiVfjVhSWYi4Nm8hQAYsQBpgJo85RywIIVjVxj9aEt/6ta9tfuSRZ5Ua/tLXpgD8tL9/7i23nHbWWe985ztHamyH50jqEyvAEGqGlAENwciBM8ZkAIAUKASVJSmipKY8U1I+FZCAB1BsBkhkoceBx8O0aZUxhoUHA03ebhYvZkWJjAFJmCQDp2Ft4IExrr/SFnhB1t+OeDt8D0rBG41Vetu2bd0zZz7U3//Wkejt/wFXd3Q88+KLo+AoGyZBgdFxvT4xA3GoA6MnDa1PTGKn8PsMbxci9KBqCTFhYuSqlZasN12ELQQ/1H1+YZeRGzKo6jCWmQLYCBOD2WfOR8qvRDnmSZVq58t/6fzK9Tj9FLQ2YUIzxrfg7LPx99ejqpdUdKeK88yBiRFHYPaU8iLVUgq7P3QlZs9BqdTG7JkIzGJ0tqJXr1z5kWx22A3z63+dVSzecccdozJ2UirZl0pmP4oaKwOdvb2dX/4izj4TLeMwfhwmtGDZMnz2s9C8ODRTK3GgYiqVYLg9NsfdeReWnYW2RnQ0Y8EcXH8dBuI5FTXJ6CZVlWATMAfM/kAFin3mjnJ5/ve+lW3PozWPc8/Cyg/hq1/BBy5F93FoKuLYY/GTn05knqE0mD1mXzEUt0Xxkk9+EpkGzOhCpE+IFZizSo1G4a1SqdScz28bOXUZeBSYPW2aMSb10euA2WfORhrM+Sieefft7a1ZTGjA2adj5f/CF7+E978f8+eiOYfumfjlLzPMs5mbmCfq6uIvXoeWIrq7cOMX8YXPY/YMtDfhjHehMrBM6+nMWXAkmHNhFcw5xdnSwPRv3xa05jF7Gh7+zYLS7jPKAyeWBuZWwiXVgfOvvw7NTejqguY5kUbVIOZCGE+rVBd98Ao0FeFn0TkTA+Ei5iDSiPRoCHz33Xef09Q0guomr1nF4hNPPJH66I+wPvG8udize15UaeFw6kO/mtRawIeuQKl04d69J5h4kYkuvvT9GF/EdZ9BGB6r2QOHYM4wy5iDiJufXztlQg4TC3j5xRMq0TGKO5gnh7olVK2aO/eWFlx6GZqa8I1vQXFDxLKqp/76Zx1zu9DeiNOWwvMwYybi+FRjgoiFZpn6B8T8gfe+9+sjrS4DKzOZr3z5y6mP/gjrE7c04q5vgM1MHph+03VobcTz66ZWuNNwe6TGl6rT1/5lfmMWZ5yKWM0PmTwEWVZVlhCUH4iL//5vG43ByqvHd7xFVPFi4vX4QjIzmx2FAq79xImet/qts6DjWPrCsPj5zzeoCDfc0Pju866c1PEPxFDVCjLK92UV8SiE7ax+7LGrU+j2HWH4w9/9Dtdem0LfQziy+sRTmXo7j2lT5d2eH7xjSXvZbJt+jFbYFACezDLE1OmtYKx/GSQ8YxgcNZjIZ84MRE2RXnzmWZjQhOee/utSpVtxkzIZowWzxzGYsyrM7yl3GO5i08hGavYVN//mt9DqbTqcWdm7rDGHWdPA1YXMkkdlBhtjAs+LUpjBTwPd06enPX7W0mgxoFFV8991ChpzeO6lrhKPD43HGmzACswF5qzhcSXVVDZTqmaiCTNsilFlQqQnRlwsM2kjWU+Io7d9+xuimMOlK1AOuzSTB2Mok9dmr/THxWHD+o2Igekzx3kBYh1npa8NIcxJvyXu92RQLOTypf7tDUFeegDHJCqnvbOBsUZI45lxbOAHUGHF8zUEjYIXXC6XfSHSKCA2DthdKqXQ8f4cSX1iEXhGK4+qFGRgSl6mBIICBQgEFcLqhN71hRtvMMLH8vfPJioR2INUiKvS9yoRgzNbt6BjKmLuJyoRVRg+U+ODD+947JFXDRdV3Ei+FOIVX0Sf/HiX8PdwtBtBCFYgYcCxQMiQng8IMIn6zdT0iOM4K4/uztUwZACl08+6fyT1ibOZgMOqJ8qf/swM8gZIKwC+9IzJKtO6cWPh3ef8YdNmXPgBnLQ0yHj9BHgQBsya41wQhBEzY9NGBKIg2PgCURxlfLF9K27ugVKlWJc0IAVmTcenrm1T/Tu9Rg/QykBImOQZPyV3tpFktk/78ykWi6UolVrpe4FCPp9Gz/shAGYmzgQeDHo3IpA5xQhIKqU8kq/uwo03IjLbdQClkdOYPg0f/ViQLwJCmJCFHD8QNm1cn33v+Y9v3oQLLkTPzYuJntXxHiEhEGlIjwixKkvqP+Vk+BE2v1DJqLxgZHwf8a7l72vdsmnyrt3dr+w4dduuK0UApWAi32vwYcpKR77MALL+tBmGhsaApIvneW1NTetT6Hkd0DnlEInwR5Yjqk/ct/PcCjCgkMs2mlCBhMhO2NE/ftWTdMEFf+pdj+XL8a+3LMnQ+oys+H4AhkBWRpESEPBipm0LFyKQ+MkPH4orTeAWaA+ehNwT5Pu0+nMut6Ea9UoJz4OBb5SCEEIi5BAHyJnY3/Tr4wLo7up6OoVunxbi+IULU+h4P46oPrHQ2yARAWGohQyAcWHceu+Pn122bM2G9fjEJzL/2HOi9NeQv4tVNa5UIYUw0AigwRJxiBffe0lnpPDNb2/f9kpWR+1K5eMIcchAVsoGHWfzfhsrxCHAUnhZo7WAFARAg/S+bRWBSfOoVI459Zxz7s/lRrzb3zY2nnL66SPe7UEcQX1iocYbDQZkPme8cSru+P7dz129Ep7BT37a+ZFrunOFLSx3a1TJIz8XcGQQJQXqGYopZFGOZtx1e1t7AXOOxf2/bjPmTGVOLpXnVuL5sVr86/vGnX4mGhsxfw6qexeymcTsVyMw5w2LSmVePotZMxAPLGQWSdBP6m4G88svvzwhlxtZT6kXaM7nq9Vq2oOPTXK/LxuZ5nJ57vfvzLbm0N2J3/yyJdYLB8JuzXMrYffevXMf/t24s96FfBGdx6Ks5pbCqc++sLihiHwRP/3pRdXKinJpQbXSHenZpagr5E6tGtiQ55MPkIwj4RMxDPqWX/aWWG6/6Sace/728e2/OvkdmNbZvH79q48+hL4+COCUU/CRK5Ep9sK8aoz2ZE5p40kBkDRDiy2K1PfQAIDOzs758+Z999FHLx25Km7/ksmsWLEik8mMVIevhUdZkB6oVPM5n7Mbl13QUeV1N96A91y0a+KUVQfWJzZYMB+fu24SiZ2RavnqTY9UY7z9BDz22EurH3klE4QxBrRQfiar4lc/9fGOXC4Gq4BjwSqJtcsq5UWcr6hjnlsz//ovBKechmO6kG1CYTxmHocVl+JX93eVw1Or0WSlAuaAY8HcrFSW2YuqXc0N6J4FXZ3NLNhIM1oxWQ8++GBnofB6IrBez2srMD6f37hx4yiMXFWTW8UZFYNZKm4qRbOeXfe2/3MdLT0dM2Yj24xMEcd04eIV+NWvTlDl03VlZiVurPLchibkG5ErolhASw6Tm9BUwLhmNDSiqxvazIwVKGZPgKkWaEkMwZCGc5qLzHnmAJCEmBhJLS8tB4iqPsqCY8DUg0UYZIxpjMwUABnRS1QajAxJexIkXHjOOTPuu+9L1REoqP03hcLbr7rq89df/8a7Gp4jqk+MCsRuUAUiBhqUaWfOAVoyC1YAg2AImsjIspQ7BPopZilqvivqsbQCAJNhICmbmji3MAQShmBAkj1igA3IgDSIwcQEBQEYH6gV3MUobaQB9PX1ndjd/aW+vove2EK9wvPWHnfcg6tXj1ZY/JHUJ2bUaxLXQnYgUCuyl5S6xGBlU5GUxiQ29YiufR9L/fACDV6elEiVYMFSA0QmAAsgqld+YwAgMpAEECe32SRYQKZb4XooTz311MkLFnwnDM872h7+QcrPaL1z586WlpaRHNlhGAxbY5nEREKETHowLIeSlZAFjASbWuAbEQQYrEH1oopCwgjoWt3ipDcSAgwcGBrHgzE/NbcnCaxngUT3Wl1UBTL71AUACBZUP+ky+syZM2fNunXnAzeJoxnAZ4Lg221tu3btGj11kShBB0iQRE0OaeVaIdrkoxWAGJytYshbBAP77WsNBAvBQjDRkDdTUusYRDASRoLrEbli0O4msZh6n7pJQU4W9cg9BiXXjyodHR1r1qy5c8aM8xsatr7udz0HLC4W//PEEx976qnm5uYUx3cwnAF8kAAZiBAiBAwYxERMtZB1BsAgDdIs2BBUcmgBUkD6DJ/Z00YYQ4k8BBYGFIIqMOxpljq5K8Gk952MIGZi7bEOWAdsPB56XMIMPWRB+2Lzk4t56EGMMaBSqVz/uc+Nz+dX5nJ/OuyG+ffA8kKhrbHx1ltuGZOhJmdB6ucVaP9TC7L2Sg6dMAxDM8X7xAqMCdgQK+JIciyZKWJUGRGDNXFMxEYyDR5E8AGRGHRKVvPkgQFLkDGkh+zFBkOeiVgAhkkDIK5t0ACgFrP/huJY3wjbt2//Wk/Pd++4w5TLpwHH9ve3A0WgBGwQ4oVC4bdKtU2c+IEPf/iyyy8vFotjM0rj184x7KspbWobK67XJ6ZafWIGEaRMdjlIysmjJr4mENjjCIgBD5Q1AhrEXLO3DFkTmEFQRHHNXIPAPhOYYtTXX7BgSiyFIBac7KWhiWWyCWQkRyWIMNqr9ME8//zzDz300AvPPtvX21sulxubmia95S1dxx23dOnSzs7OMR5csnPmZNMrkUytZKqwBAcgw6Q0xcldX4mMZENGgzRIMhPEzeIAAAAVSURBVBli1I/6gj2EgAIkKKcFNP1/dVEf00oBhCMAAAAASUVORK5CYII=", 177 | "text/plain": [ 178 | "Console does not support images" 179 | ] 180 | }, 181 | "metadata": { 182 | "image/png": { 183 | "height": 320, 184 | "width": 160 185 | } 186 | }, 187 | "output_type": "display_data" 188 | } 189 | ], 190 | "source": [ 191 | "g:place_item({type='block'},3,1)\n", 192 | "itorch.image(g.map:to_image())" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": null, 198 | "metadata": { 199 | "collapsed": true 200 | }, 201 | "outputs": [], 202 | "source": [] 203 | } 204 | ], 205 | "metadata": { 206 | "kernelspec": { 207 | "display_name": "iTorch", 208 | "language": "lua", 209 | "name": "itorch" 210 | }, 211 | "language_info": { 212 | "name": "lua", 213 | "version": "5.1" 214 | } 215 | }, 216 | "nbformat": 4, 217 | "nbformat_minor": 1 218 | } 219 | --------------------------------------------------------------------------------