41 | 
44 | 
](https://www.slideshare.net/ssuserbd7730/pyconkr-2018-rladventureto-the-rainbow)
8 |
9 | ## 발표내용
10 | - 파트 1 : DQN, Double & Dueling DQN - 성태경
11 | - 파트 2 : PER and NoisyNet - 양홍선
12 | - 파트 3 : Distributed RL - 이의령
13 | - 파트 4 : RAINBOW - 김예찬
14 |
15 |
16 | ## 실험환경
17 | **openai gym** : atari
18 | - CartPole-v1
19 | - PongNoFrameskip-v4
20 |
21 |
22 | ## 구현체
23 | - DQN
24 | - DDQN
25 | - Dueling DQN
26 | - PER
27 | - Noisy DQN
28 | - Categorical DQN
29 | - Rainbow
30 |
31 |
32 | ## 실험결과 확인
33 | | CartPole-v1 | 36 |PongNoFrameskip-v4 | 37 |
|
40 |
41 | |
42 |
43 |
44 | |
45 |
9 |
10 | This is easy-to-follow step-by-step Deep Q Learning tutorial with clean readable code.
11 |
12 | The deep reinforcement learning community has made several independent improvements to the DQN algorithm. This tutorial presents latest extensions to the DQN algorithm in the following order:
13 |
14 | 1. Playing Atari with Deep Reinforcement Learning [[arxiv]](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) [[code]](https://github.com/higgsfield/RL-Adventure/blob/master/1.dqn.ipynb)
15 | 2. Deep Reinforcement Learning with Double Q-learning [[arxiv]](https://arxiv.org/abs/1509.06461) [[code]](https://github.com/higgsfield/RL-Adventure/blob/master/2.double%20dqn.ipynb)
16 | 3. Dueling Network Architectures for Deep Reinforcement Learning [[arxiv]](https://arxiv.org/abs/1511.06581) [[code]](https://github.com/higgsfield/RL-Adventure/blob/master/3.dueling%20dqn.ipynb)
17 | 4. Prioritized Experience Replay [[arxiv]](https://arxiv.org/abs/1511.05952) [[code]](https://github.com/higgsfield/RL-Adventure/blob/master/4.prioritized%20dqn.ipynb)
18 | 5. Noisy Networks for Exploration [[arxiv]](https://arxiv.org/abs/1706.10295) [[code]](https://github.com/higgsfield/RL-Adventure/blob/master/5.noisy%20dqn.ipynb)
19 | 6. A Distributional Perspective on Reinforcement Learning [[arxiv]](https://arxiv.org/pdf/1707.06887.pdf) [[code]](https://github.com/higgsfield/RL-Adventure/blob/master/6.categorical%20dqn.ipynb)
20 | 7. Rainbow: Combining Improvements in Deep Reinforcement Learning [[arxiv]](https://arxiv.org/abs/1710.02298) [[code]](https://github.com/higgsfield/RL-Adventure/blob/master/7.rainbow%20dqn.ipynb)
21 |
22 |
23 | # Environments
24 | First, I recommend to use small test problems to run experiments quickly. Then, you can continue on environments with large observation space.
25 |
26 | - **CartPole** - classic RL environment can be solved on a single cpu
27 | - **Atari Pong** - the easiest atari environment, only takes ~ 1 million frames to converge, comparing with other atari games that take > 40 millions
28 | - **Atari others** - change hyperparameters, target network update frequency=10K, replay buffer size=1M
29 |
30 | # If you get stuck…
31 | - Remember you are not stuck unless you have spent more than a week on a single algorithm. It is perfectly normal if you do not have all the required knowledge of mathematics and CS. For example, you will need knowledge of the fundamentals of measure theory and statistics, especially the [Wasserstein metric](https://en.wikipedia.org/wiki/Wasserstein_metric) and [quantile regression](https://en.wikipedia.org/wiki/Quantile_regression). Statistical inference: [importance sampling](https://en.wikipedia.org/wiki/Importance_sampling). Data structures: [Segment Tree](https://leetcode.com/tag/segment-tree/) and [K-dimensional Tree](https://en.wikipedia.org/wiki/K-d_tree).
32 | - Carefully go through the paper. Try to see what is the problem the authors are solving. Understand a high-level idea of the approach, then read the code (skipping the proofs), and after go over the mathematical details and proofs.
33 |
34 | # Best RL courses
35 | - David Silver's course [link](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html)
36 | - Berkeley deep RL [link](http://rll.berkeley.edu/deeprlcourse/)
37 | - Practical RL [link](https://github.com/yandexdataschool/Practical_RL)
38 |
--------------------------------------------------------------------------------
/code/common/__init__.py:
--------------------------------------------------------------------------------
1 | import layers
2 | import wrappers
3 | import replay_buffer
--------------------------------------------------------------------------------
/code/common/layers.py:
--------------------------------------------------------------------------------
1 | import math
2 | import torch
3 | import torch.nn as nn
4 | import torch.nn.functional as F
5 | from torch.autograd import Variable
6 |
7 | class NoisyLinear(nn.Module):
8 | def __init__(self, in_features, out_features, use_cuda, std_init=0.4):
9 | super(NoisyLinear, self).__init__()
10 |
11 | self.use_cuda = use_cuda
12 | self.in_features = in_features
13 | self.out_features = out_features
14 | self.std_init = std_init
15 |
16 | self.weight_mu = nn.Parameter(torch.FloatTensor(out_features, in_features))
17 | self.weight_sigma = nn.Parameter(torch.FloatTensor(out_features, in_features))
18 | self.register_buffer('weight_epsilon', torch.FloatTensor(out_features, in_features))
19 |
20 | self.bias_mu = nn.Parameter(torch.FloatTensor(out_features))
21 | self.bias_sigma = nn.Parameter(torch.FloatTensor(out_features))
22 | self.register_buffer('bias_epsilon', torch.FloatTensor(out_features))
23 |
24 | self.reset_parameters()
25 | self.reset_noise()
26 |
27 | def forward(self, x):
28 | if self.use_cuda:
29 | weight_epsilon = self.weight_epsilon.cuda()
30 | bias_epsilon = self.bias_epsilon.cuda()
31 | else:
32 | weight_epsilon = self.weight_epsilon
33 | bias_epsilon = self.bias_epsilon
34 |
35 | if self.training:
36 | weight = self.weight_mu + self.weight_sigma.mul(Variable(weight_epsilon))
37 | bias = self.bias_mu + self.bias_sigma.mul(Variable(bias_epsilon))
38 | else:
39 | weight = self.weight_mu
40 | bias = self.bias_mu
41 |
42 | return F.linear(x, weight, bias)
43 |
44 | def reset_parameters(self):
45 | mu_range = 1 / math.sqrt(self.weight_mu.size(1))
46 |
47 | self.weight_mu.data.uniform_(-mu_range, mu_range)
48 | self.weight_sigma.data.fill_(self.std_init / math.sqrt(self.weight_sigma.size(1)))
49 |
50 | self.bias_mu.data.uniform_(-mu_range, mu_range)
51 | self.bias_sigma.data.fill_(self.std_init / math.sqrt(self.bias_sigma.size(0)))
52 |
53 | def reset_noise(self):
54 | epsilon_in = self._scale_noise(self.in_features)
55 | epsilon_out = self._scale_noise(self.out_features)
56 |
57 | self.weight_epsilon.copy_(epsilon_out.ger(epsilon_in))
58 | self.bias_epsilon.copy_(self._scale_noise(self.out_features))
59 |
60 | def _scale_noise(self, size):
61 | x = torch.randn(size)
62 | x = x.sign().mul(x.abs().sqrt())
63 | return x
--------------------------------------------------------------------------------
/code/common/replay_buffer.py:
--------------------------------------------------------------------------------
1 | #code from openai
2 | #https://github.com/openai/baselines/blob/master/baselines/deepq/replay_buffer.py
3 |
4 | import numpy as np
5 | import random
6 |
7 | import operator
8 |
9 |
10 | class SegmentTree(object):
11 | def __init__(self, capacity, operation, neutral_element):
12 | """Build a Segment Tree data structure.
13 | https://en.wikipedia.org/wiki/Segment_tree
14 | Can be used as regular array, but with two
15 | important differences:
16 | a) setting item's value is slightly slower.
17 | It is O(lg capacity) instead of O(1).
18 | b) user has access to an efficient `reduce`
19 | operation which reduces `operation` over
20 | a contiguous subsequence of items in the
21 | array.
22 | Paramters
23 | ---------
24 | capacity: int
25 | Total size of the array - must be a power of two.
26 | operation: lambda obj, obj -> obj
27 | and operation for combining elements (eg. sum, max)
28 | must for a mathematical group together with the set of
29 | possible values for array elements.
30 | neutral_element: obj
31 | neutral element for the operation above. eg. float('-inf')
32 | for max and 0 for sum.
33 | """
34 | assert capacity > 0 and capacity & (capacity - 1) == 0, "capacity must be positive and a power of 2."
35 | self._capacity = capacity
36 | self._value = [neutral_element for _ in range(2 * capacity)]
37 | self._operation = operation
38 |
39 | def _reduce_helper(self, start, end, node, node_start, node_end):
40 | if start == node_start and end == node_end:
41 | return self._value[node]
42 | mid = (node_start + node_end) // 2
43 | if end <= mid:
44 | return self._reduce_helper(start, end, 2 * node, node_start, mid)
45 | else:
46 | if mid + 1 <= start:
47 | return self._reduce_helper(start, end, 2 * node + 1, mid + 1, node_end)
48 | else:
49 | return self._operation(
50 | self._reduce_helper(start, mid, 2 * node, node_start, mid),
51 | self._reduce_helper(mid + 1, end, 2 * node + 1, mid + 1, node_end)
52 | )
53 |
54 | def reduce(self, start=0, end=None):
55 | """Returns result of applying `self.operation`
56 | to a contiguous subsequence of the array.
57 | self.operation(arr[start], operation(arr[start+1], operation(... arr[end])))
58 | Parameters
59 | ----------
60 | start: int
61 | beginning of the subsequence
62 | end: int
63 | end of the subsequences
64 | Returns
65 | -------
66 | reduced: obj
67 | result of reducing self.operation over the specified range of array elements.
68 | """
69 | if end is None:
70 | end = self._capacity
71 | if end < 0:
72 | end += self._capacity
73 | end -= 1
74 | return self._reduce_helper(start, end, 1, 0, self._capacity - 1)
75 |
76 | def __setitem__(self, idx, val):
77 | # index of the leaf
78 | idx += self._capacity
79 | self._value[idx] = val
80 | idx //= 2
81 | while idx >= 1:
82 | self._value[idx] = self._operation(
83 | self._value[2 * idx],
84 | self._value[2 * idx + 1]
85 | )
86 | idx //= 2
87 |
88 | def __getitem__(self, idx):
89 | assert 0 <= idx < self._capacity
90 | return self._value[self._capacity + idx]
91 |
92 |
93 | class SumSegmentTree(SegmentTree):
94 | def __init__(self, capacity):
95 | super(SumSegmentTree, self).__init__(
96 | capacity=capacity,
97 | operation=operator.add,
98 | neutral_element=0.0
99 | )
100 |
101 | def sum(self, start=0, end=None):
102 | """Returns arr[start] + ... + arr[end]"""
103 | return super(SumSegmentTree, self).reduce(start, end)
104 |
105 | def find_prefixsum_idx(self, prefixsum):
106 | """Find the highest index `i` in the array such that
107 | sum(arr[0] + arr[1] + ... + arr[i - i]) <= prefixsum
108 | if array values are probabilities, this function
109 | allows to sample indexes according to the discrete
110 | probability efficiently.
111 | Parameters
112 | ----------
113 | perfixsum: float
114 | upperbound on the sum of array prefix
115 | Returns
116 | -------
117 | idx: int
118 | highest index satisfying the prefixsum constraint
119 | """
120 | assert 0 <= prefixsum <= self.sum() + 1e-5
121 | idx = 1
122 | while idx < self._capacity: # while non-leaf
123 | if self._value[2 * idx] > prefixsum:
124 | idx = 2 * idx
125 | else:
126 | prefixsum -= self._value[2 * idx]
127 | idx = 2 * idx + 1
128 | return idx - self._capacity
129 |
130 |
131 | class MinSegmentTree(SegmentTree):
132 | def __init__(self, capacity):
133 | super(MinSegmentTree, self).__init__(
134 | capacity=capacity,
135 | operation=min,
136 | neutral_element=float('inf')
137 | )
138 |
139 | def min(self, start=0, end=None):
140 | """Returns min(arr[start], ..., arr[end])"""
141 |
142 | return super(MinSegmentTree, self).reduce(start, end)
143 |
144 |
145 | class ReplayBuffer(object):
146 | def __init__(self, size):
147 | """Create Replay buffer.
148 | Parameters
149 | ----------
150 | size: int
151 | Max number of transitions to store in the buffer. When the buffer
152 | overflows the old memories are dropped.
153 | """
154 | self._storage = []
155 | self._maxsize = size
156 | self._next_idx = 0
157 |
158 | def __len__(self):
159 | return len(self._storage)
160 |
161 | def push(self, state, action, reward, next_state, done):
162 | data = (state, action, reward, next_state, done)
163 |
164 | if self._next_idx >= len(self._storage):
165 | self._storage.append(data)
166 | else:
167 | self._storage[self._next_idx] = data
168 | self._next_idx = (self._next_idx + 1) % self._maxsize
169 |
170 | def _encode_sample(self, idxes):
171 | obses_t, actions, rewards, obses_tp1, dones = [], [], [], [], []
172 | for i in idxes:
173 | data = self._storage[i]
174 | obs_t, action, reward, obs_tp1, done = data
175 | obses_t.append(np.array(obs_t, copy=False))
176 | actions.append(np.array(action, copy=False))
177 | rewards.append(reward)
178 | obses_tp1.append(np.array(obs_tp1, copy=False))
179 | dones.append(done)
180 | return np.array(obses_t), np.array(actions), np.array(rewards), np.array(obses_tp1), np.array(dones)
181 |
182 | def sample(self, batch_size):
183 | """Sample a batch of experiences.
184 | Parameters
185 | ----------
186 | batch_size: int
187 | How many transitions to sample.
188 | Returns
189 | -------
190 | obs_batch: np.array
191 | batch of observations
192 | act_batch: np.array
193 | batch of actions executed given obs_batch
194 | rew_batch: np.array
195 | rewards received as results of executing act_batch
196 | next_obs_batch: np.array
197 | next set of observations seen after executing act_batch
198 | done_mask: np.array
199 | done_mask[i] = 1 if executing act_batch[i] resulted in
200 | the end of an episode and 0 otherwise.
201 | """
202 | idxes = [random.randint(0, len(self._storage) - 1) for _ in range(batch_size)]
203 | return self._encode_sample(idxes)
204 |
205 |
206 | class PrioritizedReplayBuffer(ReplayBuffer):
207 | def __init__(self, size, alpha):
208 | """Create Prioritized Replay buffer.
209 | Parameters
210 | ----------
211 | size: int
212 | Max number of transitions to store in the buffer. When the buffer
213 | overflows the old memories are dropped.
214 | alpha: float
215 | how much prioritization is used
216 | (0 - no prioritization, 1 - full prioritization)
217 | See Also
218 | --------
219 | ReplayBuffer.__init__
220 | """
221 | super(PrioritizedReplayBuffer, self).__init__(size)
222 | assert alpha > 0
223 | self._alpha = alpha
224 |
225 | it_capacity = 1
226 | while it_capacity < size:
227 | it_capacity *= 2
228 |
229 | self._it_sum = SumSegmentTree(it_capacity)
230 | self._it_min = MinSegmentTree(it_capacity)
231 | self._max_priority = 1.0
232 |
233 | def push(self, *args, **kwargs):
234 | """See ReplayBuffer.store_effect"""
235 | idx = self._next_idx
236 | super(PrioritizedReplayBuffer, self).push(*args, **kwargs)
237 | self._it_sum[idx] = self._max_priority ** self._alpha
238 | self._it_min[idx] = self._max_priority ** self._alpha
239 |
240 | def _sample_proportional(self, batch_size):
241 | res = []
242 | for _ in range(batch_size):
243 | # TODO(szymon): should we ensure no repeats?
244 | mass = random.random() * self._it_sum.sum(0, len(self._storage) - 1)
245 | idx = self._it_sum.find_prefixsum_idx(mass)
246 | res.append(idx)
247 | return res
248 |
249 | def sample(self, batch_size, beta):
250 | """Sample a batch of experiences.
251 | compared to ReplayBuffer.sample
252 | it also returns importance weights and idxes
253 | of sampled experiences.
254 | Parameters
255 | ----------
256 | batch_size: int
257 | How many transitions to sample.
258 | beta: float
259 | To what degree to use importance weights
260 | (0 - no corrections, 1 - full correction)
261 | Returns
262 | -------
263 | obs_batch: np.array
264 | batch of observations
265 | act_batch: np.array
266 | batch of actions executed given obs_batch
267 | rew_batch: np.array
268 | rewards received as results of executing act_batch
269 | next_obs_batch: np.array
270 | next set of observations seen after executing act_batch
271 | done_mask: np.array
272 | done_mask[i] = 1 if executing act_batch[i] resulted in
273 | the end of an episode and 0 otherwise.
274 | weights: np.array
275 | Array of shape (batch_size,) and dtype np.float32
276 | denoting importance weight of each sampled transition
277 | idxes: np.array
278 | Array of shape (batch_size,) and dtype np.int32
279 | idexes in buffer of sampled experiences
280 | """
281 | assert beta > 0
282 |
283 | idxes = self._sample_proportional(batch_size)
284 |
285 | weights = []
286 | p_min = self._it_min.min() / self._it_sum.sum()
287 | max_weight = (p_min * len(self._storage)) ** (-beta)
288 |
289 | for idx in idxes:
290 | p_sample = self._it_sum[idx] / self._it_sum.sum()
291 | weight = (p_sample * len(self._storage)) ** (-beta)
292 | weights.append(weight / max_weight)
293 | weights = np.array(weights)
294 | encoded_sample = self._encode_sample(idxes)
295 | return tuple(list(encoded_sample) + [weights, idxes])
296 |
297 | def update_priorities(self, idxes, priorities):
298 | """Update priorities of sampled transitions.
299 | sets priority of transition at index idxes[i] in buffer
300 | to priorities[i].
301 | Parameters
302 | ----------
303 | idxes: [int]
304 | List of idxes of sampled transitions
305 | priorities: [float]
306 | List of updated priorities corresponding to
307 | transitions at the sampled idxes denoted by
308 | variable `idxes`.
309 | """
310 | assert len(idxes) == len(priorities)
311 | for idx, priority in zip(idxes, priorities):
312 | assert priority > 0
313 | assert 0 <= idx < len(self._storage)
314 | self._it_sum[idx] = priority ** self._alpha
315 | self._it_min[idx] = priority ** self._alpha
316 |
317 | self._max_priority = max(self._max_priority, priority)
--------------------------------------------------------------------------------
/code/common/wrappers.py:
--------------------------------------------------------------------------------
1 | import numpy as np
2 | from collections import deque
3 | import gym
4 | from gym import spaces
5 | import cv2
6 | cv2.ocl.setUseOpenCL(False)
7 |
8 | class NoopResetEnv(gym.Wrapper):
9 | def __init__(self, env, noop_max=30):
10 | """Sample initial states by taking random number of no-ops on reset.
11 | No-op is assumed to be action 0.
12 | """
13 | gym.Wrapper.__init__(self, env)
14 | self.noop_max = noop_max
15 | self.override_num_noops = None
16 | self.noop_action = 0
17 | assert env.unwrapped.get_action_meanings()[0] == 'NOOP'
18 |
19 | def reset(self, **kwargs):
20 | """ Do no-op action for a number of steps in [1, noop_max]."""
21 | self.env.reset(**kwargs)
22 | if self.override_num_noops is not None:
23 | noops = self.override_num_noops
24 | else:
25 | noops = self.unwrapped.np_random.randint(1, self.noop_max + 1) #pylint: disable=E1101
26 | assert noops > 0
27 | obs = None
28 | for _ in range(noops):
29 | obs, _, done, _ = self.env.step(self.noop_action)
30 | if done:
31 | obs = self.env.reset(**kwargs)
32 | return obs
33 |
34 | def step(self, ac):
35 | return self.env.step(ac)
36 |
37 | class FireResetEnv(gym.Wrapper):
38 | def __init__(self, env):
39 | """Take action on reset for environments that are fixed until firing."""
40 | gym.Wrapper.__init__(self, env)
41 | assert env.unwrapped.get_action_meanings()[1] == 'FIRE'
42 | assert len(env.unwrapped.get_action_meanings()) >= 3
43 |
44 | def reset(self, **kwargs):
45 | self.env.reset(**kwargs)
46 | obs, _, done, _ = self.env.step(1)
47 | if done:
48 | self.env.reset(**kwargs)
49 | obs, _, done, _ = self.env.step(2)
50 | if done:
51 | self.env.reset(**kwargs)
52 | return obs
53 |
54 | def step(self, ac):
55 | return self.env.step(ac)
56 |
57 | class EpisodicLifeEnv(gym.Wrapper):
58 | def __init__(self, env):
59 | """Make end-of-life == end-of-episode, but only reset on true game over.
60 | Done by DeepMind for the DQN and co. since it helps value estimation.
61 | """
62 | gym.Wrapper.__init__(self, env)
63 | self.lives = 0
64 | self.was_real_done = True
65 |
66 | def step(self, action):
67 | obs, reward, done, info = self.env.step(action)
68 | self.was_real_done = done
69 | # check current lives, make loss of life terminal,
70 | # then update lives to handle bonus lives
71 | lives = self.env.unwrapped.ale.lives()
72 | if lives < self.lives and lives > 0:
73 | # for Qbert sometimes we stay in lives == 0 condtion for a few frames
74 | # so its important to keep lives > 0, so that we only reset once
75 | # the environment advertises done.
76 | done = True
77 | self.lives = lives
78 | return obs, reward, done, info
79 |
80 | def reset(self, **kwargs):
81 | """Reset only when lives are exhausted.
82 | This way all states are still reachable even though lives are episodic,
83 | and the learner need not know about any of this behind-the-scenes.
84 | """
85 | if self.was_real_done:
86 | obs = self.env.reset(**kwargs)
87 | else:
88 | # no-op step to advance from terminal/lost life state
89 | obs, _, _, _ = self.env.step(0)
90 | self.lives = self.env.unwrapped.ale.lives()
91 | return obs
92 |
93 | class MaxAndSkipEnv(gym.Wrapper):
94 | def __init__(self, env, skip=4):
95 | """Return only every `skip`-th frame"""
96 | gym.Wrapper.__init__(self, env)
97 | # most recent raw observations (for max pooling across time steps)
98 | self._obs_buffer = np.zeros((2,)+env.observation_space.shape, dtype=np.uint8)
99 | self._skip = skip
100 |
101 | def reset(self):
102 | return self.env.reset()
103 |
104 | def step(self, action):
105 | """Repeat action, sum reward, and max over last observations."""
106 | total_reward = 0.0
107 | done = None
108 | for i in range(self._skip):
109 | obs, reward, done, info = self.env.step(action)
110 | if i == self._skip - 2: self._obs_buffer[0] = obs
111 | if i == self._skip - 1: self._obs_buffer[1] = obs
112 | total_reward += reward
113 | if done:
114 | break
115 | # Note that the observation on the done=True frame
116 | # doesn't matter
117 | max_frame = self._obs_buffer.max(axis=0)
118 |
119 | return max_frame, total_reward, done, info
120 |
121 | def reset(self, **kwargs):
122 | return self.env.reset(**kwargs)
123 |
124 | class ClipRewardEnv(gym.RewardWrapper):
125 | def __init__(self, env):
126 | gym.RewardWrapper.__init__(self, env)
127 |
128 | def reward(self, reward):
129 | """Bin reward to {+1, 0, -1} by its sign."""
130 | return np.sign(reward)
131 |
132 | class WarpFrame(gym.ObservationWrapper):
133 | def __init__(self, env):
134 | """Warp frames to 84x84 as done in the Nature paper and later work."""
135 | gym.ObservationWrapper.__init__(self, env)
136 | self.width = 84
137 | self.height = 84
138 | self.observation_space = spaces.Box(low=0, high=255,
139 | shape=(self.height, self.width, 1), dtype=np.uint8)
140 |
141 | def observation(self, frame):
142 | frame = cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)
143 | frame = cv2.resize(frame, (self.width, self.height), interpolation=cv2.INTER_AREA)
144 | return frame[:, :, None]
145 |
146 | class FrameStack(gym.Wrapper):
147 | def __init__(self, env, k):
148 | """Stack k last frames.
149 | Returns lazy array, which is much more memory efficient.
150 | See Also
151 | --------
152 | baselines.common.atari_wrappers.LazyFrames
153 | """
154 | gym.Wrapper.__init__(self, env)
155 | self.k = k
156 | self.frames = deque([], maxlen=k)
157 | shp = env.observation_space.shape
158 | self.observation_space = spaces.Box(low=0, high=255, shape=(shp[0], shp[1], shp[2] * k), dtype=np.uint8)
159 |
160 | def reset(self):
161 | ob = self.env.reset()
162 | for _ in range(self.k):
163 | self.frames.append(ob)
164 | return self._get_ob()
165 |
166 | def step(self, action):
167 | ob, reward, done, info = self.env.step(action)
168 | self.frames.append(ob)
169 | return self._get_ob(), reward, done, info
170 |
171 | def _get_ob(self):
172 | assert len(self.frames) == self.k
173 | return LazyFrames(list(self.frames))
174 |
175 | class ScaledFloatFrame(gym.ObservationWrapper):
176 | def __init__(self, env):
177 | gym.ObservationWrapper.__init__(self, env)
178 |
179 | def observation(self, observation):
180 | # careful! This undoes the memory optimization, use
181 | # with smaller replay buffers only.
182 | return np.array(observation).astype(np.float32) / 255.0
183 |
184 | class LazyFrames(object):
185 | def __init__(self, frames):
186 | """This object ensures that common frames between the observations are only stored once.
187 | It exists purely to optimize memory usage which can be huge for DQN's 1M frames replay
188 | buffers.
189 | This object should only be converted to numpy array before being passed to the model.
190 | You'd not believe how complex the previous solution was."""
191 | self._frames = frames
192 | self._out = None
193 |
194 | def _force(self):
195 | if self._out is None:
196 | self._out = np.concatenate(self._frames, axis=2)
197 | self._frames = None
198 | return self._out
199 |
200 | def __array__(self, dtype=None):
201 | out = self._force()
202 | if dtype is not None:
203 | out = out.astype(dtype)
204 | return out
205 |
206 | def __len__(self):
207 | return len(self._force())
208 |
209 | def __getitem__(self, i):
210 | return self._force()[i]
211 |
212 | def make_atari(env_id):
213 | env = gym.make(env_id)
214 | assert 'NoFrameskip' in env.spec.id
215 | env = NoopResetEnv(env, noop_max=30)
216 | env = MaxAndSkipEnv(env, skip=4)
217 | return env
218 |
219 | def wrap_deepmind(env, episode_life=True, clip_rewards=True, frame_stack=False, scale=False):
220 | """Configure environment for DeepMind-style Atari.
221 | """
222 | if episode_life:
223 | env = EpisodicLifeEnv(env)
224 | if 'FIRE' in env.unwrapped.get_action_meanings():
225 | env = FireResetEnv(env)
226 | env = WarpFrame(env)
227 | if scale:
228 | env = ScaledFloatFrame(env)
229 | if clip_rewards:
230 | env = ClipRewardEnv(env)
231 | if frame_stack:
232 | env = FrameStack(env, 4)
233 | return env
234 |
235 |
236 |
237 | class ImageToPyTorch(gym.ObservationWrapper):
238 | """
239 | Image shape to num_channels x weight x height
240 | """
241 | def __init__(self, env):
242 | super(ImageToPyTorch, self).__init__(env)
243 | old_shape = self.observation_space.shape
244 | self.observation_space = gym.spaces.Box(low=0.0, high=1.0, shape=(old_shape[-1], old_shape[0], old_shape[1]), dtype=np.uint8)
245 |
246 | def observation(self, observation):
247 | return np.swapaxes(observation, 2, 0)
248 |
249 |
250 | def wrap_pytorch(env):
251 | return ImageToPyTorch(env)
--------------------------------------------------------------------------------
/code/play_rainbow_dqn_CartPole-v1.py:
--------------------------------------------------------------------------------
1 | from common.layers import NoisyLinear
2 |
3 | import torch
4 | import torch.nn as nn
5 | import torch.nn.functional as F
6 | import torch.autograd as autograd
7 | import imageio
8 | import gym
9 |
10 |
11 | class RainbowDQN(nn.Module):
12 | def __init__(self, num_inputs, num_actions, num_atoms, Vmin, Vmax):
13 | super(RainbowDQN, self).__init__()
14 |
15 | self.num_inputs = num_inputs
16 | self.num_actions = num_actions
17 | self.num_atoms = num_atoms
18 | self.Vmin = Vmin
19 | self.Vmax = Vmax
20 |
21 | self.linear1 = nn.Linear(num_inputs, 32)
22 | self.linear2 = nn.Linear(32, 64)
23 |
24 | self.noisy_value1 = NoisyLinear(64, 64, use_cuda=USE_CUDA)
25 | self.noisy_value2 = NoisyLinear(64, self.num_atoms, use_cuda=USE_CUDA)
26 |
27 | self.noisy_advantage1 = NoisyLinear(64, 64, use_cuda=USE_CUDA)
28 | self.noisy_advantage2 = NoisyLinear(64, self.num_atoms * self.num_actions, use_cuda=USE_CUDA)
29 |
30 | def forward(self, x):
31 | batch_size = x.size(0)
32 |
33 | x = F.relu(self.linear1(x))
34 | x = F.relu(self.linear2(x))
35 |
36 | value = F.relu(self.noisy_value1(x))
37 | value = self.noisy_value2(value)
38 |
39 | advantage = F.relu(self.noisy_advantage1(x))
40 | advantage = self.noisy_advantage2(advantage)
41 |
42 | value = value.view(batch_size, 1, self.num_atoms)
43 | advantage = advantage.view(batch_size, self.num_actions, self.num_atoms)
44 |
45 | x = value + advantage - advantage.mean(1, keepdim=True)
46 | x = F.softmax(x.view(-1, self.num_atoms)).view(-1, self.num_actions, self.num_atoms)
47 |
48 | return x
49 |
50 | def reset_noise(self):
51 | self.noisy_value1.reset_noise()
52 | self.noisy_value2.reset_noise()
53 | self.noisy_advantage1.reset_noise()
54 | self.noisy_advantage2.reset_noise()
55 |
56 | def action(self, state):
57 | state = Variable(torch.FloatTensor(state).unsqueeze(0), volatile=True)
58 | dist = self.forward(state).data.cpu()
59 | dist = dist * torch.linspace(self.Vmin, self.Vmax, self.num_atoms)
60 | action = dist.sum(2).max(1)[1].numpy()[0]
61 | return action
62 |
63 |
64 | USE_CUDA = torch.cuda.is_available() # False
65 | Variable = lambda *args, **kwargs: autograd.Variable(*args, **kwargs).cuda() if USE_CUDA else autograd.Variable(*args,
66 | **kwargs)
67 |
68 | env_id = "CartPole-v1"
69 | env = gym.make(env_id)
70 |
71 | num_atoms = 51
72 | Vmin = -10
73 | Vmax = 10
74 |
75 | current_model = RainbowDQN(env.observation_space.shape[0], env.action_space.n, num_atoms, Vmin, Vmax)
76 | current_model.load_state_dict(torch.load('./save_model/27586-CartPole-v1_RainbowDQN.pkl'))
77 |
78 | if USE_CUDA:
79 | current_model = current_model.cuda()
80 |
81 | EPISODE = 10
82 | rendering = True
83 | frames = []
84 |
85 | for episode in range(1, EPISODE + 1):
86 | done = False
87 | score = 0
88 | state = env.reset()
89 |
90 | # for gif
91 | obs = env.render(mode='rgb_array')
92 | frames.append(obs)
93 |
94 | while True:
95 | if rendering:
96 | env.render()
97 |
98 | frames.append(env.render(mode='rgb_array'))
99 | action = current_model.action(state)
100 |
101 | next_state, reward, done, _ = env.step(action)
102 |
103 | state = next_state
104 | score += reward
105 | print(len(frames))
106 |
107 | if done:
108 | break
109 |
110 | string = '{}_{}_{}.gif'
111 | imageio.mimsave('./save_gif/' + string.format(env_id, "RainbowDQN", episode), frames, duration=0.0286)
112 |
113 | env.close()
114 |
--------------------------------------------------------------------------------
/code/play_rainbow_dqn_PongNoFrameskip-v4.py:
--------------------------------------------------------------------------------
1 | from common.wrappers import make_atari, wrap_deepmind, wrap_pytorch
2 | from common.layers import NoisyLinear
3 |
4 | import math
5 | import numpy as np
6 | import torch
7 | import torch.nn as nn
8 | import torch.nn.functional as F
9 | import torch.autograd as autograd
10 | import imageio
11 |
12 |
13 | # model은 학습모형에 맞는 모델을 class로 설계해주시면 됩니
14 | class RainbowDQN(nn.Module):
15 | def __init__(self, input_shape, num_actions, num_atoms, Vmin, Vmax):
16 | super(RainbowDQN, self).__init__()
17 |
18 | self.input_shape = input_shape
19 | self.num_actions = num_actions
20 | self.num_atoms = num_atoms
21 | self.Vmin = Vmin
22 | self.Vmax = Vmax
23 |
24 | self.features = nn.Sequential(
25 | # ((84 - 8 - 2*0) / 4) + 1 = 20
26 | nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4), # batch_size x 32 x 20 x 20
27 | nn.ReLU(),
28 |
29 | # ((20 - 4 - 2*0) / 2) + 1 = 9
30 | nn.Conv2d(32, 64, kernel_size=4, stride=2), # batch_size x 64 x 9 x 9
31 | nn.ReLU(),
32 |
33 | # ((9 - 3 - 2*0) / 2) + 1 = 4
34 | nn.Conv2d(64, 64, kernel_size=3, stride=1), # batch_size x 64 x 4 x 4
35 | nn.ReLU()
36 | )
37 |
38 | self.noisy_value1 = NoisyLinear(self.feature_size(), 512, use_cuda=USE_CUDA)
39 | self.noisy_value2 = NoisyLinear(512, self.num_atoms, use_cuda=USE_CUDA)
40 |
41 | self.noisy_advantage1 = NoisyLinear(self.feature_size(), 512, use_cuda=USE_CUDA)
42 | self.noisy_advantage2 = NoisyLinear(512, self.num_atoms * self.num_actions, use_cuda=USE_CUDA)
43 |
44 |
45 | def forward(self, x):
46 | batch_size = x.size(0)
47 |
48 | x = x / 255.
49 | x = self.features(x)
50 | x = x.view(batch_size, -1)
51 |
52 | value = F.relu(self.noisy_value1(x))
53 | value = self.noisy_value2(value)
54 |
55 | advantage = F.relu(self.noisy_advantage1(x))
56 | advantage = self.noisy_advantage2(advantage)
57 |
58 | value = value.view(batch_size, 1, self.num_atoms)
59 | advantage = advantage.view(batch_size, self.num_actions, self.num_atoms)
60 |
61 | x = value + advantage - advantage.mean(1, keepdim=True)
62 | x = F.softmax(x.view(-1, self.num_atoms)).view(-1, self.num_actions, self.num_atoms)
63 |
64 | return x
65 |
66 | def reset_noise(self):
67 | self.noisy_value1.reset_noise()
68 | self.noisy_value2.reset_noise()
69 | self.noisy_advantage1.reset_noise()
70 | self.noisy_advantage2.reset_noise()
71 |
72 | def feature_size(self):
73 | return self.features(autograd.Variable(torch.zeros(1, *self.input_shape))).view(1, -1).size(1)
74 |
75 | def act(self, state):
76 | state = Variable(torch.FloatTensor(np.float32(state)).unsqueeze(0), volatile=True)
77 | dist = self.forward(state).data.cpu()
78 | dist = dist * torch.linspace(self.Vmin, self.Vmax, self.num_atoms)
79 | action = dist.sum(2).max(1)[1].numpy()[0]
80 | return action
81 |
82 |
83 | USE_CUDA = torch.cuda.is_available() # False
84 | Variable = lambda *args, **kwargs: autograd.Variable(*args, **kwargs).cuda() if USE_CUDA else autograd.Variable(*args,
85 | **kwargs)
86 |
87 | env_id = "PongNoFrameskip-v4"
88 | env = make_atari(env_id)
89 | env = wrap_deepmind(env)
90 | env = wrap_pytorch(env)
91 |
92 | num_atoms = 51
93 | Vmin = -10
94 | Vmax = 10
95 |
96 |
97 | # 모델을 초기화하고 load하는 부분
98 | current_model = RainbowDQN(env.observation_space.shape, env.action_space.n, num_atoms, Vmin, Vmax)
99 | current_model.load_state_dict(torch.load('./save_model/999398-PongNoFrameskip-v4_RainbowDQN.pkl'))
100 |
101 | if USE_CUDA:
102 | current_model = current_model.cuda()
103 |
104 |
105 | EPISODE = 3
106 | rendering = True
107 | frames = []
108 |
109 | for episode in range(1, EPISODE + 1):
110 | done = False
111 | score = 0
112 | state = env.reset()
113 |
114 | # for gif
115 | obs = env.render(mode='rgb_array')
116 | frames.append(obs)
117 |
118 | while True:
119 | if rendering:
120 | env.render()
121 |
122 | frames.append(env.render(mode='rgb_array'))
123 | action = current_model.act(state)
124 |
125 | next_state, reward, done, _ = env.step(action)
126 |
127 | state = next_state
128 | score += reward
129 |
130 | if done:
131 | break
132 |
133 | string = '{}_{}_{}.gif'
134 | imageio.mimsave('./save_gif/' + string.format(env_id, "RainbowDQN", episode), frames, duration=0.0286)
135 |
136 | env.close()
--------------------------------------------------------------------------------
/code/pytorch0.4.1/4.prioritized dqn.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "import math, random\n",
10 | "\n",
11 | "import gym\n",
12 | "import numpy as np\n",
13 | "\n",
14 | "import torch\n",
15 | "import torch.nn as nn\n",
16 | "import torch.optim as optim\n",
17 | "import torch.nn.functional as F"
18 | ]
19 | },
20 | {
21 | "cell_type": "code",
22 | "execution_count": null,
23 | "metadata": {},
24 | "outputs": [],
25 | "source": [
26 | "from IPython.display import clear_output\n",
27 | "import matplotlib.pyplot as plt\n",
28 | "%matplotlib inline"
29 | ]
30 | },
31 | {
32 | "cell_type": "markdown",
33 | "metadata": {},
34 | "source": [
35 | "Prioritized Experience Replay: https://arxiv.org/abs/1511.05952
" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "class NaivePrioritizedBuffer(object):\n", 69 | " def __init__(self, capacity, prob_alpha=0.6):\n", 70 | " self.prob_alpha = prob_alpha\n", 71 | " self.capacity = capacity\n", 72 | " self.buffer = []\n", 73 | " self.pos = 0\n", 74 | " self.priorities = np.zeros((capacity,), dtype=np.float32)\n", 75 | " \n", 76 | " def push(self, state, action, reward, next_state, done):\n", 77 | " assert state.ndim == next_state.ndim\n", 78 | " state = np.expand_dims(state, 0)\n", 79 | " next_state = np.expand_dims(next_state, 0)\n", 80 | " \n", 81 | " max_prio = self.priorities.max() if self.buffer else 1.0\n", 82 | " \n", 83 | " if len(self.buffer) < self.capacity:\n", 84 | " self.buffer.append((state, action, reward, next_state, done))\n", 85 | " else:\n", 86 | " self.buffer[self.pos] = (state, action, reward, next_state, done)\n", 87 | " \n", 88 | " self.priorities[self.pos] = max_prio\n", 89 | " self.pos = (self.pos + 1) % self.capacity\n", 90 | " \n", 91 | " def sample(self, batch_size, beta=0.4):\n", 92 | " if len(self.buffer) == self.capacity:\n", 93 | " prios = self.priorities\n", 94 | " else:\n", 95 | " prios = self.priorities[:self.pos]\n", 96 | " \n", 97 | " probs = prios ** self.prob_alpha\n", 98 | " probs /= probs.sum()\n", 99 | " \n", 100 | " indices = np.random.choice(len(self.buffer), batch_size, p=probs)\n", 101 | " samples = [self.buffer[idx] for idx in indices]\n", 102 | " \n", 103 | " total = len(self.buffer)\n", 104 | " weights = (total * probs[indices]) ** (-beta)\n", 105 | " weights /= weights.max()\n", 106 | " weights = np.array(weights, dtype=np.float32)\n", 107 | " \n", 108 | " batch = list(zip(*samples))\n", 109 | " states = np.concatenate(batch[0])\n", 110 | " actions = batch[1]\n", 111 | " rewards = batch[2]\n", 112 | " next_states = np.concatenate(batch[3])\n", 113 | " dones = batch[4]\n", 114 | " \n", 115 | " return states, actions, rewards, next_states, dones, indices, weights\n", 116 | " \n", 117 | " def update_priorities(self, batch_indices, batch_priorities):\n", 118 | " for idx, prio in zip(batch_indices, batch_priorities):\n", 119 | " self.priorities[idx] = prio\n", 120 | "\n", 121 | " def __len__(self):\n", 122 | " return len(self.buffer)" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": null, 128 | "metadata": {}, 129 | "outputs": [], 130 | "source": [ 131 | "beta_start = 0.4\n", 132 | "beta_frames = 1000 \n", 133 | "beta_by_frame = lambda frame_idx: min(1.0, beta_start + frame_idx * (1.0 - beta_start) / beta_frames)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "metadata": {}, 140 | "outputs": [], 141 | "source": [ 142 | "plt.plot([beta_by_frame(i) for i in range(10000)])" 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "metadata": {}, 148 | "source": [ 149 | "