24 |
25 | Q-Learning with Eligibility Traces
26 | ----------------------------------
27 |
28 | ``` r
29 | env$resetEverything()
30 | #> [1] 30
31 | policy = makePolicy("epsilon.greedy", epsilon = 0.1)
32 | alg = makeAlgorithm("qlearning", lambda = 0.8, traces = "accumulate")
33 | agent = makeAgent(policy, "table", alg)
34 |
35 | res = interact(env, agent, n.episodes = 500L)
36 | ```
37 |
38 | 
39 |
40 | Q-Learning with Experience replay
41 | ---------------------------------
42 |
43 | ``` r
44 | env$resetEverything()
45 | #> [1] 30
46 | policy = makePolicy("epsilon.greedy", epsilon = 0.1)
47 | mem = makeReplayMemory(size = 10L, batch.size = 10L)
48 | agent = makeAgent(policy, "table", "qlearning", experience.replay = mem)
49 |
50 | res = interact(env, agent, n.episodes = 500L)
51 | ```
52 |
53 | 
54 |
55 | Q-Learning with neural network and experience replay
56 | ----------------------------------------------------
57 |
58 | ``` r
59 | env$resetEverything()
60 | #> [1] 30
61 | library(keras)
62 | model = keras_model_sequential() %>%
63 | layer_dense(units = env$n.actions, activation = "linear",
64 | input_shape = c(env$n.states), kernel_initializer = initializer_zeros(),
65 | use_bias = FALSE) %>%
66 | compile(loss = "mae", optimizer = optimizer_sgd(lr = 1))
67 | mem = makeReplayMemory(size = 2L, batch.size = 2L)
68 | val = makeValueFunction("neural.network", model = model)
69 | policy = makePolicy("epsilon.greedy", epsilon = 0.1)
70 | preprocess = function(x) to_categorical(x, num_classes = env$n.states)
71 | agent = makeAgent(policy, val, "qlearning",
72 | preprocess = preprocess, experience.replay = mem)
73 |
74 | res = interact(env, agent, n.episodes = 500L)
75 | ```
76 |
77 | 
78 |
--------------------------------------------------------------------------------
/man/tilecoding.Rd:
--------------------------------------------------------------------------------
1 | % Generated by roxygen2: do not edit by hand
2 | % Please edit documentation in R/tiles.R
3 | \name{tiles}
4 | \alias{tiles}
5 | \alias{iht}
6 | \title{Tile Coding}
7 | \usage{
8 | tiles(iht, n.tilings, state, action = integer(0))
9 |
10 | iht(max.size)
11 | }
12 | \arguments{
13 | \item{iht}{[\code{IHT}] \cr A hash table created with \code{iht}.}
14 |
15 | \item{n.tilings}{[\code{integer(1)}] \cr Number of tilings.}
16 |
17 | \item{state}{[\code{vector(2)}] \cr A two-dimensional state observation.
18 | Make sure to scale the observation to unit variance before.}
19 |
20 | \item{action}{[\code{integer(1)}] \cr Optional: If supplied the action space
21 | will also be tiled. All distinct actions will result in different tile numbers.}
22 |
23 | \item{max.size}{[\code{integer(1)}] \cr Maximal size of hash table.}
24 | }
25 | \value{
26 | \code{iht} creates a hash table, which can then be passed on to \code{tiles}.
27 | \code{tiles} returns an integer vector of size \code{n.tilings} with the active tile numbers.
28 | }
29 | \description{
30 | Implementation of Sutton's tile coding software version 3.
31 | }
32 | \details{
33 | Tile coding is a way of representing the values of a vector of continuous variables as a large
34 | binary vector with few 1s and many 0s. The binary vector is not represented explicitly,
35 | but as a list of the components that are 1s. The main step is to partition, or tile,
36 | the continuous space multiple times and select one tile from each tiling, that corresponding
37 | the the vector's value. Each tile is converted to an element in the big binary vector,
38 | and the list of the tile (element) numbers is returned as the representation of the vector's value.
39 | Tile coding is recommended as a way of applying online learning methods to domains with continuous
40 | state or action variables. [copied from manual]
41 |
42 | See detailed manual on the web.
43 | In comparison to the Python implementation indices start with 1 instead of 0. The hash table is
44 | implemented as an environment, which is an attribute of an R6 class.
45 |
46 | Make sure that the size of the hash table is large enough, else an error will be triggered,
47 | when trying to assign a value to a full hash table.
48 | }
49 | \examples{
50 | # Create hash table
51 | hash = iht(1024)
52 |
53 | # Partition state space using 8 tilings
54 | tiles(hash, n.tilings = 8, state = c(3.6, 7.21))
55 | tiles(hash, n.tilings = 8, state = c(3.7, 7.21))
56 | tiles(hash, n.tilings = 8, state = c(4, 7))
57 | tiles(hash, n.tilings = 8, state = c(- 37.2, 7))
58 |
59 | }
60 | \references{
61 | Sutton and Barto (Book draft 2017): Reinforcement Learning: An Introduction
62 | }
63 |
--------------------------------------------------------------------------------
/session_info.txt:
--------------------------------------------------------------------------------
1 | - Session info ----------------------------------------------------------
2 | setting value
3 | version R version 3.4.3 (2017-11-30)
4 | os Windows 10 x64
5 | system x86_64, mingw32
6 | ui RTerm
7 | language (EN)
8 | collate English_Germany.1252
9 | tz Europe/Berlin
10 | date 2017-12-23
11 |
12 | - Packages --------------------------------------------------------------
13 | package * version date source
14 | assertthat 0.2.0 2017-04-11 CRAN (R 3.4.2)
15 | backports 1.1.1 2017-09-25 CRAN (R 3.4.1)
16 | cli 1.0.0 2017-12-22 Github (r-lib/cli@ab1c3aa)
17 | clisymbols 1.2.0 2017-05-21 CRAN (R 3.4.3)
18 | crayon 1.3.4 2017-09-16 CRAN (R 3.4.2)
19 | desc 1.1.1 2017-08-03 CRAN (R 3.4.2)
20 | devtools 1.13.3.9000 2017-12-22 Github (hadley/devtools@0bcfd6e)
21 | digest 0.6.13 2017-12-14 CRAN (R 3.4.3)
22 | evaluate 0.10.1 2017-06-24 CRAN (R 3.4.2)
23 | htmltools 0.3.6 2017-04-28 CRAN (R 3.4.2)
24 | knitr 1.17 2017-08-10 CRAN (R 3.4.2)
25 | magrittr 1.5 2014-11-22 CRAN (R 3.4.2)
26 | memoise 1.1.0 2017-04-21 CRAN (R 3.4.2)
27 | pkgbuild 0.0.0.9000 2017-12-22 Github (r-lib/pkgbuild@ce7f6d1)
28 | pkgload 0.0.0.9000 2017-12-22 Github (r-lib/pkgload@70eaef8)
29 | R6 2.2.2 2017-06-17 CRAN (R 3.4.2)
30 | Rcpp 0.12.13 2017-09-28 CRAN (R 3.4.2)
31 | rlang 0.1.4.9000 2017-12-22 Github (tidyverse/rlang@cc7587c)
32 | rmarkdown 1.8 2017-11-17 CRAN (R 3.4.2)
33 | rprojroot 1.3-1 2017-12-18 CRAN (R 3.4.3)
34 | sessioninfo 1.0.1.9000 2017-12-22 Github (r-lib/sessioninfo@c871d01)
35 | stringi 1.1.6 2017-11-17 CRAN (R 3.4.2)
36 | stringr 1.2.0 2017-02-18 CRAN (R 3.4.2)
37 | testthat 2.0.0 2017-12-13 CRAN (R 3.4.3)
38 | usethis 1.1.0.9000 2017-12-22 Github (r-lib/usethis@973bcab)
39 | withr 2.1.1 2017-12-19 CRAN (R 3.4.3)
40 | yaml 2.1.16 2017-12-12 CRAN (R 3.4.3)
41 |
--------------------------------------------------------------------------------
/docs/jquery.sticky-kit.min.js:
--------------------------------------------------------------------------------
1 | /*
2 | Sticky-kit v1.1.2 | WTFPL | Leaf Corcoran 2015 | http://leafo.net
3 | */
4 | (function(){var b,f;b=this.jQuery||window.jQuery;f=b(window);b.fn.stick_in_parent=function(d){var A,w,J,n,B,K,p,q,k,E,t;null==d&&(d={});t=d.sticky_class;B=d.inner_scrolling;E=d.recalc_every;k=d.parent;q=d.offset_top;p=d.spacer;w=d.bottoming;null==q&&(q=0);null==k&&(k=void 0);null==B&&(B=!0);null==t&&(t="is_stuck");A=b(document);null==w&&(w=!0);J=function(a,d,n,C,F,u,r,G){var v,H,m,D,I,c,g,x,y,z,h,l;if(!a.data("sticky_kit")){a.data("sticky_kit",!0);I=A.height();g=a.parent();null!=k&&(g=g.closest(k));
5 | if(!g.length)throw"failed to find stick parent";v=m=!1;(h=null!=p?p&&a.closest(p):b(""))&&h.css("position",a.css("position"));x=function(){var c,f,e;if(!G&&(I=A.height(),c=parseInt(g.css("border-top-width"),10),f=parseInt(g.css("padding-top"),10),d=parseInt(g.css("padding-bottom"),10),n=g.offset().top+c+f,C=g.height(),m&&(v=m=!1,null==p&&(a.insertAfter(h),h.detach()),a.css({position:"",top:"",width:"",bottom:""}).removeClass(t),e=!0),F=a.offset().top-(parseInt(a.css("margin-top"),10)||0)-q,
6 | u=a.outerHeight(!0),r=a.css("float"),h&&h.css({width:a.outerWidth(!0),height:u,display:a.css("display"),"vertical-align":a.css("vertical-align"),"float":r}),e))return l()};x();if(u!==C)return D=void 0,c=q,z=E,l=function(){var b,l,e,k;if(!G&&(e=!1,null!=z&&(--z,0>=z&&(z=E,x(),e=!0)),e||A.height()===I||x(),e=f.scrollTop(),null!=D&&(l=e-D),D=e,m?(w&&(k=e+u+c>C+n,v&&!k&&(v=!1,a.css({position:"fixed",bottom:"",top:c}).trigger("sticky_kit:unbottom"))),e
41 |
42 |
89 |
90 |
91 |
92 |
93 |
122 |
123 |
124 |
125 |
--------------------------------------------------------------------------------
/docs/news/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
94 |
96 |
97 | Articles version 0.1.0
95 |
98 |
110 |
111 |
121 |
99 |
109 |
100 |
108 | All vignettes
101 | 102 | 103 |-
104 |
- Agents 105 |
- Environments 106 |
41 |
42 |
89 |
90 |
91 |
92 |
93 |
125 |
126 |
127 |
128 |
--------------------------------------------------------------------------------
/R/policy.R:
--------------------------------------------------------------------------------
1 | #' Create policy.
2 | #'
3 | #' Reinforcement learning policies.
4 | #'
5 | #' @param class \[`character(1)`] \cr
6 | #' Class of policy. One of `c("random", "epsilon.greedy", "greedy", "softmax")`.
7 | #' @param ... \[`any`] \cr Optional named arguments passed on to the subclass. Alternatively
8 | #' these can be given using the `args` argument.
9 | #' @param args \[`list`] \cr Optional list of named arguments passed on to the
10 | #' subclass. The arguments in ... take precedence over values in this list.
11 | #' We strongly encourage you to use one or the other to pass arguments
12 | #' to the function but not both.
13 | #'
14 | #' @return \[`list(name, args)`] List with the name and optional args.
15 | #' This list can then be passed onto [makeAgent], which will construct the
16 | #' policy accordingly.
17 | #'
18 | #' @md
19 | #' @aliases Policy
20 | #'
21 | #' @section Policies:
22 | #' * [RandomPolicy]
23 | #' * [GreedyPolicy]
24 | #' * [EpsilonGreedyPolicy]
25 | #' * [SoftmaxPolicy]
26 | #'
27 | #' @export
28 | #' @examples
29 | #' policy = makePolicy("random")
30 | #' policy = makePolicy("epsilon.greedy", epsilon = 0.1)
31 | makePolicy = function(class = "random", args = list(), ...) {
32 | checkmate::assertChoice(class,
33 | c("random", "epsilon.greedy", "greedy", "softmax")) #, "gaussian"))
34 | checkmate::assertList(args, names = "unique")
35 | args = append(list(...), args)
36 | # remove duplicate entries in args list
37 | args = args[unique(names(args))]
38 |
39 | # fixme: check arguments of policy here
40 | x = list(name = class, args = args)
41 | class(x) = "Policy"
42 | x
43 | }
44 |
45 |
46 | Policy = R6::R6Class("Policy",
47 | public = list(
48 | sampleAction = function(policy) {
49 | action = sample(seq_along(policy), prob = policy,
50 | size = 1, replace = TRUE) - 1L
51 | action
52 | }
53 | )
54 | )
55 |
56 | #' Epsilon Greedy Policy
57 | #'
58 | #' @aliases GreedyPolicy
59 | #' @export
60 | #' @section Usage:
61 | #' \code{makePolicy("epsilon.greedy", epsilon = 0.1)} \cr
62 | #' \code{makePolicy("greedy")}
63 | #'
64 | #' @param epsilon [\code{numeric(1) in [0, 1]}] \cr
65 | #' Ratio of random exploration in epsilon-greedy action selection.
66 | #'
67 | #' @name EpsilonGreedyPolicy
68 | #' @examples
69 | #' policy = makePolicy("epsilon.greedy", epsilon = 0.1)
70 | NULL
71 |
72 | EpsilonGreedyPolicy = R6::R6Class("EpsilonGreedyPolicy",
73 | inherit = Policy,
74 | public = list(
75 | epsilon = NULL,
76 | getActionProbs = function(Q, n.actions) { # fixme: break ties
77 | greedy.action = nnet::which.is.max(Q)
78 | policy = matrix(0, nrow = 1, ncol = n.actions)
79 | policy[, greedy.action] = 1 - self$epsilon
80 | policy = policy + self$epsilon / n.actions
81 | policy
82 | },
83 | initialize = function(epsilon = 0.1) {
84 | checkmate::assertNumber(epsilon, lower = 0, upper = 1)
85 | self$epsilon = epsilon
86 | }
87 | )
88 | )
89 |
90 | GreedyPolicy = R6::R6Class("GreedyPolicy",
91 | # inherit = EpsilonGreedyPolicy,
92 | public = list(
93 | getActionProbs = function(Q, n.actions) {
94 | greedy.action = nnet::which.is.max(Q) # this is duplicate code!
95 | policy = matrix(0, nrow = 1, ncol = n.actions)
96 | policy[, greedy.action] = 1
97 | policy
98 | }
99 | )
100 | )
101 |
102 | #' Random Policy
103 | #'
104 | #' @export
105 | #' @section Usage:
106 | #' \code{makePolicy("random")}
107 | #'
108 | #' @name RandomPolicy
109 | #' @examples
110 | #' pol = makePolicy("random")
111 | NULL
112 |
113 | RandomPolicy = R6::R6Class("RandomPolicy",
114 | inherit = Policy,
115 | public = list(
116 | getActionProbs = function(Q, n.actions) {
117 | policy = matrix(1 / n.actions, nrow = 1, ncol = n.actions)
118 | policy
119 | }
120 | )
121 | )
122 |
123 | # GaussianPolicy = R6::R6Class("GaussianPolicy",
124 | # inherit = Policy,
125 | # public = list(
126 | # sampleAction = function(mean, sd) {
127 | # rnorm(1L, mean, sd)
128 | # }
129 | # )
130 | # )
131 |
132 | #' Softmax Policy
133 | #'
134 | #' @export
135 | #' @section Usage:
136 | #' \code{makePolicy("softmax")}
137 | #'
138 | #' @name SoftmaxPolicy
139 | #' @examples
140 | #' pol = makePolicy("softmax")
141 | NULL
142 |
143 | SoftmaxPolicy = R6::R6Class("SoftmaxPolicy",
144 | inherit = Policy,
145 | public = list(
146 | getActionProbs = function(Q, n.actions) {
147 | policy = exp(Q) / rowSums(exp(Q))
148 | policy
149 | }
150 | )
151 | )
152 |
--------------------------------------------------------------------------------
/man/gridworld.Rd:
--------------------------------------------------------------------------------
1 | % Generated by roxygen2: do not edit by hand
2 | % Please edit documentation in R/environment_gridworld.R
3 | \name{Gridworld}
4 | \alias{Gridworld}
5 | \title{Gridworld}
6 | \arguments{
7 | \item{shape}{[\code{integer(2)}] \cr
8 | Shape of the gridworld (number of rows x number of columns).}
9 |
10 | \item{goal.states}{[\code{integer}] \cr
11 | Goal states in the gridworld.}
12 |
13 | \item{cliff.states}{[\code{integer}] \cr
14 | Cliff states in the gridworld.}
15 |
16 | \item{reward.step}{[\code{integer(1)}] \cr
17 | Reward for taking a step.}
18 |
19 | \item{cliff.transition.states}{[\code{integer}] \cr
20 | States to which the environment transitions if stepping into the cliff.
21 | If it is a vector, all states will have equal probability.
22 | Only used when \code{cliff.transition.done == FALSE},
23 | else specify the \code{initial.state} argument.}
24 |
25 | \item{reward.cliff}{[\code{integer(1)}] \cr
26 | Reward for taking a step in the cliff state.}
27 |
28 | \item{diagonal.moves}{[\code{logical(1)}] \cr
29 | Should diagonal moves be allowed?}
30 |
31 | \item{wind}{[\code{integer}] \cr
32 | Strength of the upward wind in each cell.}
33 |
34 | \item{cliff.transition.done}{[\code{logical(1)}] \cr
35 | Should the episode end after stepping into the cliff?}
36 |
37 | \item{stochasticity}{[\code{numeric(1)}] \cr
38 | Probability of random transition to any of the neighboring states when taking any action.}
39 |
40 | \item{...}{[\code{any}] \cr Arguments passed on to \link{makeEnvironment}.}
41 | }
42 | \description{
43 | Creates gridworld environments.
44 | }
45 | \details{
46 | A gridworld is an episodic navigation task, the goal is to get from start state to goal state.
47 |
48 | Possible actions include going left, right, up or down. If \code{diagonal.moves = TRUE} diagonal
49 | moves are also possible, leftup, leftdown, rightup and rightdown.
50 |
51 | When stepping into a cliff state you get a reward of \code{reward.cliff},
52 | usually a high negative reward and transition to a state specified by \code{cliff.transition.states}.
53 |
54 | In each column a deterministic wind specified via \code{wind} pushes you up a specific number of
55 | grid cells (for the next action).
56 |
57 | A stochastic gridworld is a gridworld where with probability \code{stochasticity} the next state
58 | is chosen at random from all neighbor states independent of the actual action.
59 |
60 | If an action would take you off the grid, the new state is the nearest cell inside the grid.
61 | For each step you get a reward of \code{reward.step}, until you reach a goal state,
62 | then the episode is done.
63 |
64 | States are enumerated row-wise and numeration starts with 0.
65 | Here is an example 4x4 grid:
66 | \tabular{rrrr}{
67 | 0 \tab 1 \tab 2 \tab 3 \cr
68 | 4 \tab 5 \tab 6 \tab 7 \cr
69 | 8 \tab 9 \tab 10 \tab 11 \cr
70 | 12 \tab 13 \tab 14 \tab 15 \cr
71 | }
72 | So a board position could look like this (G: goal state, x: current state, C: cliff state):
73 | \tabular{rrrr}{
74 | G \tab o \tab o \tab o \cr
75 | o \tab o \tab o \tab o \cr
76 | o \tab x \tab o \tab o \cr
77 | o \tab o \tab o \tab C \cr
78 | }
79 | }
80 | \section{Usage}{
81 |
82 | \code{makeEnvironment("gridworld", shape = NULL, goal.states = NULL, cliff.states = NULL, reward.step = -1, reward.cliff = -100, diagonal.moves = FALSE, wind = rep(0, shape[2]), cliff.transition.states = NULL, cliff.transition.done = FALSE, stochasticity = 0, ...)}
83 | }
84 |
85 | \section{Methods}{
86 |
87 | \itemize{
88 | \item \code{$step(action)} \cr
89 | Take action in environment.
90 | Returns a list with \code{state}, \code{reward}, \code{done}.
91 | \item \code{$reset()} \cr
92 | Resets the \code{done} flag of the environment and returns an initial state.
93 | Useful when starting a new episode.
94 | \item \code{$visualize()} \cr
95 | Visualizes the environment (if there is a visualization function).
96 | }
97 | }
98 |
99 | \examples{
100 | # Gridworld Environment (Sutton & Barto Example 4.1)
101 | env1 = makeEnvironment("gridworld", shape = c(4L, 4L), goal.states = 0L,
102 | initial.state = 15L)
103 | env1$reset()
104 | env1$visualize()
105 | env1$step(0L)
106 | env1$visualize()
107 |
108 | # Windy Gridworld (Sutton & Barto Example 6.5)
109 | env2 = makeEnvironment("gridworld", shape = c(7, 10), goal.states = 37L,
110 | reward.step = -1, wind = c(0, 0, 0, 1, 1, 1, 2, 2, 1, 0),
111 | initial.state = 30L)
112 |
113 | # Cliff Walking (Sutton & Barto Example 6.6)
114 | env3 = makeEnvironment("gridworld", shape = c(4, 12), goal.states = 47L,
115 | cliff.states = 37:46, reward.step = -1, reward.cliff = -100,
116 | cliff.transition.states = 36L, initial.state = 36L)
117 | }
118 |
--------------------------------------------------------------------------------
/benchmark/benchmark_windy_gridworld.Rmd:
--------------------------------------------------------------------------------
1 | ---
2 | title: "Benchmark Algorithms on Windy Gridworld Task"
3 | author: "Markus Dumke"
4 | date: "`r Sys.Date()`"
5 | output: github_document
6 | ---
7 |
8 | ```{r setup, include = FALSE}
9 | knitr::opts_chunk$set(comment = "#>", collapse = FALSE, message = FALSE)
10 | knitr::opts_chunk$set(fig.path = 'Images/', eval = TRUE, cache = FALSE,
11 | size = "footnotesize", fig.asp = 0.618, fig.width = 4.5, fig.align = "center",
12 | message = FALSE, comment = "#>", collapse = TRUE, echo = TRUE)
13 | ```
14 |
15 |
16 | ```{r}
17 | library(reinforcelearn)
18 | env = makeEnvironment("windy.gridworld")
19 | ```
20 |
21 | The optimal solution is 15 steps.
22 |
23 | ## Simple Q-Learning
24 |
25 | ```{r}
26 | policy = makePolicy("epsilon.greedy", epsilon = 0.1)
27 | agent = makeAgent(policy, "table", "qlearning")
28 |
29 | res = interact(env, agent, n.episodes = 500L)
30 | ```
31 |
32 | ```{r qlearning_windygrid, echo = FALSE, fig.align = "center"}
33 | library(ggplot2)
34 | df = data.frame(episode = seq_along(res$steps),
35 | steps = res$steps)
36 |
37 | ggplot(df, aes(episode, steps), col = "brown1") +
38 | geom_point(alpha = 0.2) +
39 | theme_bw() +
40 | labs(
41 | title = "Q-Learning",
42 | x = "Episode",
43 | y = "Steps per episode"
44 | ) +
45 | coord_cartesian(ylim = c(0, 200)) +
46 | geom_smooth(se = FALSE, size = 1) +
47 | geom_hline(yintercept = 15, size = 1, col = "black", lty = 2)
48 | ```
49 |
50 | ## Q-Learning with Eligibility Traces
51 |
52 | ```{r}
53 | env$resetEverything()
54 | policy = makePolicy("epsilon.greedy", epsilon = 0.1)
55 | alg = makeAlgorithm("qlearning", lambda = 0.8, traces = "accumulate")
56 | agent = makeAgent(policy, "table", alg)
57 |
58 | res = interact(env, agent, n.episodes = 500L)
59 | ```
60 |
61 | ```{r qlearning_windygrid_elig, echo = FALSE, fig.align = "center"}
62 | library(ggplot2)
63 | df = data.frame(episode = seq_along(res$steps),
64 | steps = res$steps)
65 |
66 | ggplot(df, aes(episode, steps), col = "brown1") +
67 | geom_point(alpha = 0.2) +
68 | theme_bw() +
69 | labs(
70 | title = "Q-Learning",
71 | subtitle = "Eligibility traces",
72 | x = "Episode",
73 | y = "Steps per episode"
74 | ) +
75 | coord_cartesian(ylim = c(0, 200)) +
76 | geom_smooth(se = FALSE, size = 1) +
77 | geom_hline(yintercept = 15, size = 1, col = "black", lty = 2)
78 | ```
79 |
80 | ## Q-Learning with Experience replay
81 |
82 | ```{r}
83 | env$resetEverything()
84 | policy = makePolicy("epsilon.greedy", epsilon = 0.1)
85 | mem = makeReplayMemory(size = 10L, batch.size = 10L)
86 | agent = makeAgent(policy, "table", "qlearning", replay.memory = mem)
87 |
88 | res = interact(env, agent, n.episodes = 500L)
89 | ```
90 |
91 | ```{r qlearning_windygrid_expreplay, echo = FALSE, fig.align = "center"}
92 | library(ggplot2)
93 | df = data.frame(episode = seq_along(res$steps),
94 | steps = res$steps)
95 |
96 | ggplot(df, aes(episode, steps), col = "brown1") +
97 | geom_point(alpha = 0.2) +
98 | theme_bw() +
99 | labs(
100 | title = "Q-Learning",
101 | subtitle = "Experience replay",
102 | x = "Episode",
103 | y = "Steps per episode"
104 | ) +
105 | coord_cartesian(ylim = c(0, 200)) +
106 | geom_smooth(se = FALSE, size = 1) +
107 | geom_hline(yintercept = 15, size = 1, col = "black", lty = 2)
108 | ```
109 |
110 | ## Q-Learning with neural network and experience replay
111 |
112 | ```{r}
113 | env$resetEverything()
114 | library(keras)
115 | model = keras_model_sequential() %>%
116 | layer_dense(units = env$n.actions, activation = "linear",
117 | input_shape = c(env$n.states), kernel_initializer = initializer_zeros(),
118 | use_bias = FALSE) %>%
119 | compile(loss = "mae", optimizer = optimizer_sgd(lr = 1))
120 | mem = makeReplayMemory(size = 2L, batch.size = 2L)
121 | val = makeValueFunction("neural.network", model = model)
122 | policy = makePolicy("epsilon.greedy", epsilon = 0.1)
123 | preprocess = function(x) to_categorical(x, num_classes = env$n.states)
124 | agent = makeAgent(policy, val, "qlearning",
125 | preprocess = preprocess, replay.memory = mem)
126 |
127 | res = interact(env, agent, n.episodes = 500L)
128 | ```
129 |
130 | ```{r qlearning_windygrid_neuralnetwork, echo = FALSE, fig.align = "center"}
131 | library(ggplot2)
132 | df = data.frame(episode = seq_along(res$steps),
133 | steps = res$steps)
134 |
135 | ggplot(df, aes(episode, steps), col = "brown1") +
136 | geom_point(alpha = 0.2) +
137 | theme_bw() +
138 | labs(
139 | title = "Q-Learning",
140 | subtitle = "Experience replay and neural network",
141 | x = "Episode",
142 | y = "Steps per episode"
143 | ) +
144 | coord_cartesian(ylim = c(0, 200)) +
145 | geom_smooth(se = FALSE, size = 1) +
146 | geom_hline(yintercept = 15, size = 1, col = "black", lty = 2)
147 | ```
148 |
--------------------------------------------------------------------------------
/docs/LICENSE.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
94 |
95 |
113 |
114 |
124 |
96 |
103 |
104 |
111 |
112 |
97 |
99 |
100 | Change log All releases
98 |
101 |
102 |
41 |
42 |
89 |
90 |
91 |
92 |
93 |
125 |
126 |
127 |
128 |
--------------------------------------------------------------------------------
/R/interact.R:
--------------------------------------------------------------------------------
1 | #' Interaction between agent and environment.
2 | #'
3 | #' Run interaction between agent and environment for specified number of steps
4 | #' or episodes.
5 | #'
6 | #' @param env \[`Environment`] \cr Reinforcement learning environment created by [makeEnvironment].
7 | #' @param agent \[`Agent`] \cr Agent created by [makeAgent].
8 | #' @param n.steps \[`integer(1)`] \cr Number of steps to run.
9 | #' @param n.episodes \[`integer(1)`] \cr Number of episodes to run.
10 | #' @param max.steps.per.episode \[`integer(1)`] \cr Maximal number of steps allowed per episode.
11 | #' @param learn \[`logical(1)`] \cr Should the agent learn?
12 | #' @param visualize \[`logical(1)`] \cr Visualize the interaction between agent and environment?
13 | #'
14 | #' @return \[`list`] Return and number of steps per episode.
15 | #'
16 | #' @md
17 | #'
18 | #' @export
19 | #' @examples
20 | #' env = makeEnvironment("windy.gridworld")
21 | #' agent = makeAgent("softmax", "table", "qlearning")
22 | #' interact(env, agent, n.episodes = 10L)
23 | interact = function(env, agent, n.steps = Inf, n.episodes = Inf,
24 | max.steps.per.episode = Inf, learn = TRUE, visualize = FALSE) {
25 |
26 | checkmate::assertClass(env, "Environment")
27 | checkmate::assertClass(agent, "Agent")
28 | if (!is.infinite(n.steps)) checkmate::assertInt(n.steps, lower = 1)
29 | if (!is.infinite(n.episodes)) checkmate::assertInt(n.episodes, lower = 1)
30 | if (!is.infinite(max.steps.per.episode)) checkmate::assertInt(max.steps.per.episode, lower = 1)
31 | checkmate::assertFlag(learn)
32 | checkmate::assertFlag(visualize)
33 |
34 | # one of steps / episodes must be finite!
35 | if (is.infinite(n.steps) && is.infinite(n.episodes)) {
36 | stop("Specify finite number of steps or finite number of episodes!")
37 | }
38 |
39 | # preallocation if number of episodes | steps is known in advance else append to list
40 | if (n.episodes < Inf) {
41 | episode.returns = rep(NA_real_, n.episodes)
42 | } else {
43 | episode.returns = vector(mode = "double")
44 | }
45 | if (n.episodes < Inf) {
46 | episode.steps = rep(NA_integer_, n.episodes)
47 | } else {
48 | episode.steps = vector(mode = "integer")
49 | }
50 |
51 | # index to fill in
52 | episode = 0L
53 |
54 | # get episode | step number of when to stop
55 | stop.step = env$n.step + n.steps
56 | stop.episode = env$episode + n.episodes
57 |
58 | # # check if environment has been resetted, if not reset else get current state
59 | # if (is.null(env$state)) {
60 | # message("Reset environment.")
61 | # state = env$reset()
62 | # if (visualize) {
63 | # env$visualize()
64 | # }
65 | # } else {
66 | state = env$state
67 | #}
68 |
69 | agent$n.actions = env$n.actions
70 |
71 | if (agent$initialized == FALSE) {
72 | agent$init(env) # if e.g. value fun has not been initialized do this here
73 | agent$initialized = TRUE
74 | }
75 |
76 | while (TRUE) {
77 | # print(paste0("episode: ", env$episode, "; step: ", env$n.step))
78 | # # agent$observeBeforeAct() # observe before act
79 | action = agent$act(state) # fixme: store action also in agent attribute
80 | res = env$step(action)
81 |
82 | if (visualize) {
83 | env$visualize()
84 | }
85 |
86 | # # keep track of visited states, actions, rewards
87 | # agent$history = append(agent$history, list(list(state = state, action = action,
88 | # reward = res$reward, episode = env$episode + 1L)))
89 |
90 | # observe: e.g. add observation to replay memory
91 | agent$observe(state, action, res$reward, res$state, env)
92 |
93 | # optional learning (check whether to learn maybe as agent method)
94 | if (learn) {
95 | #browser()
96 | agent$learn(env, learn)
97 | }
98 |
99 | state = res$state # set state to next state for new iteration
100 |
101 | # when episode is finished print out information and reset environment
102 | if (res$done || env$episode.step == max.steps.per.episode) {
103 | if (!res$done) {
104 | env$episode = env$episode + 1L
105 | }
106 | message(paste("Episode", env$episode, "finished after",
107 | env$episode.step, "steps with a return of", env$episode.return)) # let this be customizable by having his in a function argument
108 | episode = episode + 1L
109 | episode.returns[episode] = env$episode.return
110 | episode.steps[episode] = env$episode.step
111 | state = env$reset()
112 | # if (visualize) {
113 | # env$visualize()
114 | # }
115 | agent$reset()
116 | }
117 |
118 | # stop criteria
119 | if (env$n.step == stop.step || env$episode == stop.episode) {
120 | break
121 | }
122 | }
123 | # return information about returns, steps
124 | list(returns = episode.returns, steps = episode.steps) # return history
125 | }
126 | # fixme: logging
127 | # fixme: control when to learn
128 | # fixme: print out average return of last n episodes ...
129 | # fixme: maybe return training time, history ...
130 | # make message after done configurable as function argument
131 |
--------------------------------------------------------------------------------
/R/tiles.R:
--------------------------------------------------------------------------------
1 | #' Tile Coding
2 | #'
3 | #' Implementation of Sutton's tile coding software version 3.
4 | #'
5 | #' @param iht \[`IHT`] \cr A hash table created with `iht`.
6 | #' @param n.tilings \[`integer(1)`] \cr Number of tilings.
7 | #' @param state \[`vector(2)`] \cr A two-dimensional state observation.
8 | #' Make sure to scale the observation to unit variance before.
9 | #' @param action \[`integer(1)`] \cr Optional: If supplied the action space
10 | #' will also be tiled. All distinct actions will result in different tile numbers.
11 | #'
12 | #' @return `iht` creates a hash table, which can then be passed on to `tiles`.
13 | #' `tiles` returns an integer vector of size `n.tilings` with the active tile numbers.
14 | #'
15 | #' @md
16 | #'
17 | #' @details
18 | #' Tile coding is a way of representing the values of a vector of continuous variables as a large
19 | #' binary vector with few 1s and many 0s. The binary vector is not represented explicitly,
20 | #' but as a list of the components that are 1s. The main step is to partition, or tile,
21 | #' the continuous space multiple times and select one tile from each tiling, that corresponding
22 | #' the the vector's value. Each tile is converted to an element in the big binary vector,
23 | #' and the list of the tile (element) numbers is returned as the representation of the vector's value.
24 | #' Tile coding is recommended as a way of applying online learning methods to domains with continuous
25 | #' state or action variables. \[copied from manual]
26 | #'
27 | #' See detailed manual on the web.
28 | #' In comparison to the Python implementation indices start with 1 instead of 0. The hash table is
29 | #' implemented as an environment, which is an attribute of an R6 class.
30 | #'
31 | #' Make sure that the size of the hash table is large enough, else an error will be triggered,
32 | #' when trying to assign a value to a full hash table.
33 | #'
34 | #' @references Sutton and Barto (Book draft 2017): Reinforcement Learning: An Introduction
35 | #' @rdname tilecoding
36 | #' @export
37 | #' @examples
38 | #' # Create hash table
39 | #' hash = iht(1024)
40 | #'
41 | #' # Partition state space using 8 tilings
42 | #' tiles(hash, n.tilings = 8, state = c(3.6, 7.21))
43 | #' tiles(hash, n.tilings = 8, state = c(3.7, 7.21))
44 | #' tiles(hash, n.tilings = 8, state = c(4, 7))
45 | #' tiles(hash, n.tilings = 8, state = c(- 37.2, 7))
46 | #'
47 | tiles = function(iht, n.tilings, state, action = integer(0)) {
48 | checkmate::assertClass(iht, "IHT")
49 | checkmate::assertInt(n.tilings)
50 | checkmate::assertVector(state)
51 | checkmate::assertIntegerish(action, max.len = 1)
52 |
53 | qfloats = floor(state * n.tilings)
54 | active.tiles = rep(0, n.tilings)
55 | coords = rep(0, length(state) + 1)
56 |
57 | for (tiling in seq_len(n.tilings)) {
58 | tiling = tiling - 1
59 | tiling2 = tiling * 2
60 | coords[1] = tiling
61 | b = tiling
62 | for (q in seq_along(qfloats)) {
63 | coords[q + 1] = (qfloats[q] + b) %/% n.tilings
64 | b = b + tiling2
65 | }
66 | coords = append(coords, action)
67 | active.tiles[tiling + 1] = hashcoords(paste(coords, collapse = ""), iht)
68 | }
69 |
70 | return(active.tiles)
71 | }
72 |
73 | hashcoords = function(coords, iht) {
74 | iht$add2Env(coords)
75 | iht$checkFull()
76 | iht$getIndex(coords)
77 | }
78 |
79 | #' @rdname tilecoding
80 | #' @param max.size \[`integer(1)`] \cr Maximal size of hash table.
81 | #' @export
82 | #' @md
83 | iht = function(max.size) {
84 | checkmate::assertInt(max.size)
85 | IHTClass$new(max.size)
86 | }
87 |
88 | IHTClass = R6::R6Class("IHT",
89 | public = list(
90 | i = 0,
91 | max.size = NULL,
92 | e = NULL,
93 |
94 | initialize = function(max.size) {
95 | self$max.size = max.size
96 | self$e = new.env(size = max.size)
97 | },
98 |
99 | checkFull = function() {
100 | if (length(self$e) > self$max.size) {
101 | stop("Tile Coding failed because hash table IHT is full!")
102 | }
103 | },
104 |
105 | add2Env = function(coords) {
106 | if (!exists(coords, envir = self$e, inherits = FALSE)) {
107 | self$i = self$i + 1
108 | self$checkFull()
109 | self$e[[coords]] = self$i
110 | }
111 | },
112 |
113 | getIndex = function(coords) {
114 | return(self$e[[coords]])
115 | }
116 | )
117 | )
118 |
119 | #' Make n hot vector.
120 | #'
121 | #' @param x \[`integer`] \cr Which features are active?
122 | #' @param len \[`integer(1)`] \cr Length of the feature vector.
123 | #' @param out \[`character(1)`] \cr Format of the output. Can be a vector or a matrix.
124 | #'
125 | #' @return \[`matrix(1, len)`] A one-row matrix with `len` columns with every
126 | #' entry 0 except the columns specified by `x` which are 1.
127 | #'
128 | #' @md
129 | #'
130 | #' @export
131 | #' @examples
132 | #' nHot(c(1, 3), 5)
133 | #' nHot(c(1, 3), 5, out = "vector")
134 | nHot = function(x, len, out = "matrix") {
135 | checkmate::assertIntegerish(x, max.len = len)
136 | checkmate::assertInt(len)
137 | if (out == "matrix") {
138 | m = matrix(rep(0, len), nrow = 1)
139 | m[1, x] = 1
140 | } else {
141 | m = rep(0, len)
142 | m[x] = 1
143 | }
144 | m
145 | }
146 |
--------------------------------------------------------------------------------
/README.Rmd:
--------------------------------------------------------------------------------
1 | ---
2 | output: github_document
3 | ---
4 |
5 | ```{r, echo = FALSE}
6 | knitr::opts_chunk$set(
7 | collapse = TRUE,
8 | comment = "#>",
9 | message = FALSE,
10 | fig.path = "README-"
11 | )
12 | ```
13 |
14 | # Reinforcement Learning in R
94 |
112 |
113 |
114 |
124 |
95 |
110 |
111 |
96 |
98 |
99 | License
97 |YEAR: 2017 100 | COPYRIGHT HOLDER: Markus Dumke 101 | 102 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 103 | 104 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 105 | 106 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 107 |108 | 109 |
15 |
16 | [](https://travis-ci.org/markusdumke/reinforcelearn)
17 | [](https://cran.r-project.org/package=reinforcelearn)
18 | [](https://codecov.io/github/markusdumke/reinforcelearn?branch=master)
19 |
20 | ```{r, include = FALSE}
21 | writeLines(capture.output(devtools::session_info()), "session_info.txt")
22 | ```
23 |
24 |
25 | ### Documentation
26 |
27 | [Website](https://markusdumke.github.io/reinforcelearn)
28 |
29 | ----
30 |
31 | ### Installation
32 |
33 | ```{r, eval = FALSE}
34 | # Install from CRAN.
35 | install.packages("reinforcelearn")
36 |
37 | # Install development version from github.
38 | devtools::install_github("markusdumke/reinforcelearn")
39 | ```
40 |
41 | ----
42 |
43 | ### Get started
44 |
45 | Reinforcement Learning with the package `reinforcelearn` is as easy as
46 |
47 | ```{r}
48 | library(reinforcelearn)
49 |
50 | env = makeEnvironment("windy.gridworld")
51 | agent = makeAgent("softmax", "table", "qlearning")
52 |
53 | # Run interaction for 10 episodes.
54 | interact(env, agent, n.episodes = 10L)
55 | ```
56 |
57 | ----
58 |
59 | ### Environments
60 |
61 | With `makeEnvironment` you can create reinforcement learning environments.
62 |
63 | ```{r}
64 | # Create environment.
65 | step = function(self, action) {
66 | state = list(mean = action + rnorm(1), sd = runif(1))
67 | reward = rnorm(1, state[[1]], state[[2]])
68 | done = FALSE
69 | list(state, reward, done)
70 | }
71 |
72 | reset = function(self) {
73 | state = list(mean = 0, sd = 1)
74 | state
75 | }
76 |
77 | env = makeEnvironment("custom", step = step, reset = reset)
78 | ```
79 |
80 | The environment is an `R6` class with a set of attributes and methods.
81 | You can interact with the environment via the `reset` and `step` method.
82 |
83 | ```{r}
84 | # Reset environment.
85 | env$reset()
86 |
87 | # Take action.
88 | env$step(100)
89 | ```
90 |
91 | There are some predefined environment classes, e.g. `MDPEnvironment`, which allows you to create a Markov Decision Process by passing on state transition array and reward matrix, or `GymEnvironment`, where you can use toy problems from [OpenAI Gym](https://gym.openai.com/).
92 |
93 | ```{r, eval = FALSE}
94 | # Create a gym environment.
95 | # Make sure you have Python, gym and reticulate installed.
96 | env = makeEnvironment("gym", gym.name = "MountainCar-v0")
97 |
98 | # Take random actions for 200 steps.
99 | env$reset()
100 | for (i in 1:200) {
101 | action = sample(0:2, 1)
102 | env$step(action)
103 | env$visualize()
104 | }
105 | env$close()
106 | ```
107 |
108 | This should open a window showing a graphical visualization of the environment during interaction.
109 |
110 | For more details on how to create an environment have a look at the vignette: [Environments](https://markusdumke.github.io/reinforcelearn/articles/environments.html)
111 |
112 | ----
113 |
114 | ### Agents
115 |
116 | With `makeAgent` you can set up a reinforcement learning agent to solve the environment, i.e. to find the best action in each time step.
117 |
118 | The first step is to set up the policy, which defines which action to choose. For example we could use a uniform random policy.
119 |
120 | ```{r}
121 | # Create the environment.
122 | env = makeEnvironment("windy.gridworld")
123 |
124 | # Create agent with uniform random policy.
125 | policy = makePolicy("random")
126 | agent = makeAgent(policy)
127 |
128 | # Run interaction for 10 steps.
129 | interact(env, agent, n.steps = 10L)
130 | ```
131 |
132 | In this scenario the agent chooses all actions with equal probability and will not learn anything from the interaction. Usually we want the agent to be able to learn something. Value-based algorithms learn a value function from interaction with the environment and adjust the policy according to the value function. For example we could set up Q-Learning with a softmax policy.
133 |
134 | ```{r}
135 | # Create the environment.
136 | env = makeEnvironment("windy.gridworld")
137 |
138 | # Create qlearning agent with softmax policy and tabular value function.
139 | policy = makePolicy("softmax")
140 | values = makeValueFunction("table", n.states = env$n.states, n.actions = env$n.actions)
141 | algorithm = makeAlgorithm("qlearning")
142 | agent = makeAgent(policy, values, algorithm)
143 |
144 | # Run interaction for 10 steps.
145 | interact(env, agent, n.episodes = 10L)
146 | ```
147 |
148 | ----
149 |
150 | ### Vignettes
151 |
152 | Also have a look at the vignettes for further examples.
153 |
154 | - [Environments](https://markusdumke.github.io/reinforcelearn/articles/environments.html)
155 | - [Agents](https://markusdumke.github.io/reinforcelearn/articles/agents.html)
156 |
157 | ----
158 |
159 | Logo is a modification of https://www.r-project.org/logo/.
160 |
--------------------------------------------------------------------------------
/R/valuefunction.R:
--------------------------------------------------------------------------------
1 | #' Value Function Representation
2 | #'
3 | #' A representation of the value function.
4 | #'
5 | #' @param class \[`character(1)`] \cr Class of value function approximation.
6 | #' One of `c("table", "neural.network")`.
7 | #' @inheritParams makePolicy
8 | #'
9 | #' @return \[`list(name, args)`] List with the name and optional args.
10 | #' This list can then be passed onto [makeAgent], which will construct the
11 | #' value function accordingly.
12 | #'
13 | #' @md
14 | #'
15 | #' @section Representations:
16 | #' * [ValueTable]
17 | #' * [ValueNetwork]
18 | #'
19 | #' @export
20 | #' @examples
21 | #' val = makeValueFunction("table", n.states = 16L, n.actions = 4L)
22 | #' # If the number of states and actions is not supplied, the agent will try
23 | #' # to figure these out from the environment object during interaction.
24 | #' val = makeValueFunction("table")
25 | makeValueFunction = function(class, args = list(), ...) {
26 | checkmate::assertChoice(class, c("table", "neural.network")) #, "keras.neural.network", "mxnet.neural.network"))
27 | # fixme: check arguments here
28 | checkmate::assertList(args, names = "unique")
29 | args = append(list(...), args)
30 | # remove duplicate entries in args list
31 | args = args[unique(names(args))]
32 |
33 | x = list(name = class, args = args)
34 | class(x) = "ValueFunction"
35 | x
36 | }
37 | # comment: this could also be used for policy params -> better name?
38 |
39 |
40 | #' Value Table
41 | #'
42 | #' Table representing the action value function Q.
43 | #'
44 | #' You can specify the shape of the value table. If omitted the agent will try
45 | #' to configure these automatically from the environment during interaction
46 | #' (therefore the environment needs to have a `n.states` and `n.actions` attribute).
47 | #'
48 | #' @section Usage:
49 | #' `makeValueFunction("table", n.states = NULL, n.actions = 1L,
50 | #' step.size = 0.1, initial.value = NULL)`
51 | #'
52 | #' @param n.states \[`integer(1)`] \cr Number of states (rows in the value function).
53 | #' @param n.actions \[`integer(1)`] \cr Number of actions (columns in the value function).
54 | #' @param step.size \[`numeric(1)`] \cr Step size (learning rate) for gradient descent update.
55 | #'
56 | #' @name ValueTable
57 | #' @aliases table
58 | #' @md
59 | #'
60 | #' @examples
61 | #' val = makeValueFunction("table", n.states = 20L, n.actions = 4L)
62 | NULL
63 |
64 | ValueTable = R6::R6Class("ValueTable",
65 | public = list(
66 | Q = NULL,
67 | step.size = NULL,
68 |
69 | # fixme: get number of states and actions automatically from environment
70 | # fixme: custom initializer, e.g. not to 0
71 | initialize = function(n.states = NULL, n.actions = 1L, step.size = 0.1,
72 | initial.value = NULL) {
73 |
74 | checkmate::assertInt(n.states, lower = 1)
75 | checkmate::assertInt(n.actions, lower = 1)
76 | checkmate::assertNumber(step.size, lower = 0)
77 | checkmate::assertMatrix(initial.value, null.ok = TRUE)
78 |
79 | # state or action value function
80 | if (!is.null(initial.value)) {
81 | self$Q = initial.value
82 | } else {
83 | self$Q = matrix(0, nrow = n.states, ncol = n.actions)
84 | }
85 | self$step.size = step.size
86 | },
87 |
88 | predictQ = function(state) {
89 | self$Q[state + 1L, , drop = FALSE]
90 | },
91 |
92 | # fixme: make this vectorised -> ok
93 | # caveat: states must be unique!
94 | train = function(state, target, step.size = self$step.size) {
95 | self$Q[state + 1L, ] = self$Q[state + 1L, ] + step.size * (target - self$Q[state + 1L, ]) # drop = FALSE ?
96 | },
97 |
98 | # train with td error and eligibility traces
99 | trainWithError = function(eligibility, error, step.size = self$step.size) {
100 | self$Q = self$Q + step.size * error * eligibility
101 | },
102 |
103 | processBatch = function(batch) {
104 | data = data.frame(state = unlist(batch[["state"]]), action = unlist(batch[["action"]]),
105 | reward = unlist(batch[["reward"]]), next.state = unlist(batch[["next.state"]]))
106 | data
107 | },
108 |
109 | getWeights = function() {
110 | self$Q
111 | }
112 | )
113 | )
114 |
115 | #' Value Network
116 | #'
117 | #' Neural network representing the action value function Q.
118 | #'
119 | #' @section Usage:
120 | #' `makeValueFunction("neural.network", model)`
121 | #'
122 | #' @param model \[`keras model`] \cr A keras model.
123 | #' Make sure that the model has been compiled.
124 | #'
125 | #' @name ValueNetwork
126 | #' @aliases neural.network
127 | #' @md
128 | #'
129 | #' @examples
130 | #' \dontrun{
131 | #' library(keras)
132 | #' model = keras_model_sequential()
133 | #' model %>% layer_dense(20, input_shape = 10, activation = "relu")
134 | #' model %>% layer_dense(4, activation = "softmax")
135 | #' keras::compile(model, loss = "mae", optimizer = keras::optimizer_sgd(lr = 0.4))
136 | #'
137 | #' val = makeValueFunction("neural.network", model = model)
138 | #' }
139 | NULL
140 |
141 | ValueNetwork = R6::R6Class("ValueNetwork",
142 | public = list(
143 | model = NULL,
144 |
145 | # keras model # fixme: add support for mxnet
146 | initialize = function(model) {
147 | checkmate::assertClass(model, "keras.models.Sequential")
148 | self$model = model
149 | },
150 |
151 | predictQ = function(state) {
152 | predict(self$model, state) # another function?
153 | },
154 |
155 | train = function(state, target) { # add ... argument to pass on arguments to fit
156 | keras::fit(self$model, state, target, verbose = 0L)
157 | },
158 |
159 | processBatch = function(batch) {
160 | data = list(
161 | state = do.call(rbind, batch[["state"]]), # problematic for matrix with many columns, purrr::reduce
162 | action = unlist(batch[["action"]]),
163 | reward = unlist(batch[["reward"]]),
164 | next.state = purrr::reduce(batch[["next.state"]], rbind)
165 | )
166 | data
167 | },
168 |
169 | getWeights = function() {
170 | self$model %>% get_weights()
171 | }
172 | )
173 | )
174 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 | Reinforcement Learning in R
3 | =======================================================================================
4 |
5 | [](https://travis-ci.org/markusdumke/reinforcelearn) [](https://cran.r-project.org/package=reinforcelearn) [](https://codecov.io/github/markusdumke/reinforcelearn?branch=master)
6 |
7 | WARNING: This package is not maintained anymore!
8 |
9 | ### Documentation
10 |
11 | [Website](https://markusdumke.github.io/reinforcelearn)
12 |
13 | ------------------------------------------------------------------------
14 |
15 | ### Installation
16 |
17 | ``` r
18 | # Install from CRAN.
19 | install.packages("reinforcelearn")
20 |
21 | # Install development version from github.
22 | devtools::install_github("markusdumke/reinforcelearn")
23 | ```
24 |
25 | ------------------------------------------------------------------------
26 |
27 | ### Get started
28 |
29 | Reinforcement Learning with the package `reinforcelearn` is as easy as
30 |
31 | ``` r
32 | library(reinforcelearn)
33 |
34 | env = makeEnvironment("windy.gridworld")
35 | agent = makeAgent("softmax", "table", "qlearning")
36 |
37 | # Run interaction for 10 episodes.
38 | interact(env, agent, n.episodes = 10L)
39 | #> $returns
40 | #> [1] -3244 -2335 -1734 -169 -879 -798 -216 -176 -699 -232
41 | #>
42 | #> $steps
43 | #> [1] 3244 2335 1734 169 879 798 216 176 699 232
44 | ```
45 |
46 | ------------------------------------------------------------------------
47 |
48 | ### Environments
49 |
50 | With `makeEnvironment` you can create reinforcement learning environments.
51 |
52 | ``` r
53 | # Create environment.
54 | step = function(self, action) {
55 | state = list(mean = action + rnorm(1), sd = runif(1))
56 | reward = rnorm(1, state[[1]], state[[2]])
57 | done = FALSE
58 | list(state, reward, done)
59 | }
60 |
61 | reset = function(self) {
62 | state = list(mean = 0, sd = 1)
63 | state
64 | }
65 |
66 | env = makeEnvironment("custom", step = step, reset = reset)
67 | ```
68 |
69 | The environment is an `R6` class with a set of attributes and methods. You can interact with the environment via the `reset` and `step` method.
70 |
71 | ``` r
72 | # Reset environment.
73 | env$reset()
74 | #> $mean
75 | #> [1] 0
76 | #>
77 | #> $sd
78 | #> [1] 1
79 |
80 | # Take action.
81 | env$step(100)
82 | #> $state
83 | #> $state$mean
84 | #> [1] 99.56104
85 | #>
86 | #> $state$sd
87 | #> [1] 0.5495179
88 | #>
89 | #>
90 | #> $reward
91 | #> [1] 99.40968
92 | #>
93 | #> $done
94 | #> [1] FALSE
95 | ```
96 |
97 | There are some predefined environment classes, e.g. `MDPEnvironment`, which allows you to create a Markov Decision Process by passing on state transition array and reward matrix, or `GymEnvironment`, where you can use toy problems from [OpenAI Gym](https://gym.openai.com/).
98 |
99 | ``` r
100 | # Create a gym environment.
101 | # Make sure you have Python, gym and reticulate installed.
102 | env = makeEnvironment("gym", gym.name = "MountainCar-v0")
103 |
104 | # Take random actions for 200 steps.
105 | env$reset()
106 | for (i in 1:200) {
107 | action = sample(0:2, 1)
108 | env$step(action)
109 | env$visualize()
110 | }
111 | env$close()
112 | ```
113 |
114 | This should open a window showing a graphical visualization of the environment during interaction.
115 |
116 | For more details on how to create an environment have a look at the vignette: [Environments](https://markusdumke.github.io/reinforcelearn/articles/environments.html)
117 |
118 | ------------------------------------------------------------------------
119 |
120 | ### Agents
121 |
122 | With `makeAgent` you can set up a reinforcement learning agent to solve the environment, i.e. to find the best action in each time step.
123 |
124 | The first step is to set up the policy, which defines which action to choose. For example we could use a uniform random policy.
125 |
126 | ``` r
127 | # Create the environment.
128 | env = makeEnvironment("windy.gridworld")
129 |
130 | # Create agent with uniform random policy.
131 | policy = makePolicy("random")
132 | agent = makeAgent(policy)
133 |
134 | # Run interaction for 10 steps.
135 | interact(env, agent, n.steps = 10L)
136 | #> $returns
137 | #> numeric(0)
138 | #>
139 | #> $steps
140 | #> integer(0)
141 | ```
142 |
143 | In this scenario the agent chooses all actions with equal probability and will not learn anything from the interaction. Usually we want the agent to be able to learn something. Value-based algorithms learn a value function from interaction with the environment and adjust the policy according to the value function. For example we could set up Q-Learning with a softmax policy.
144 |
145 | ``` r
146 | # Create the environment.
147 | env = makeEnvironment("windy.gridworld")
148 |
149 | # Create qlearning agent with softmax policy and tabular value function.
150 | policy = makePolicy("softmax")
151 | values = makeValueFunction("table", n.states = env$n.states, n.actions = env$n.actions)
152 | algorithm = makeAlgorithm("qlearning")
153 | agent = makeAgent(policy, values, algorithm)
154 |
155 | # Run interaction for 10 steps.
156 | interact(env, agent, n.episodes = 10L)
157 | #> $returns
158 | #> [1] -1524 -3496 -621 -374 -173 -1424 -1742 -468 -184 -39
159 | #>
160 | #> $steps
161 | #> [1] 1524 3496 621 374 173 1424 1742 468 184 39
162 | ```
163 |
164 | ------------------------------------------------------------------------
165 |
166 | ### Vignettes
167 |
168 | Also have a look at the vignettes for further examples.
169 |
170 | - [Environments](https://markusdumke.github.io/reinforcelearn/articles/environments.html)
171 | - [Agents](https://markusdumke.github.io/reinforcelearn/articles/agents.html)
172 |
173 | ------------------------------------------------------------------------
174 |
175 | Logo is a modification of
41 |
42 |
89 |
90 |
91 |
92 |
93 |
373 |
374 |
375 |
376 |
--------------------------------------------------------------------------------
94 |
361 |
362 |
372 |
95 |
346 |
347 |
360 |
96 |
101 |
102 | 97 | Reference 98 | version 0.1.0 99 |
100 |
103 |
345 |