Q-learning

Q-learning is a model-free reinforcement learning technique. Specifically, Q-learning can be used to find an optimal action-selection policy for any given (finite) Markov decision process (MDP). It works by learning an action-value function that ultimately gives the expected utility of taking a given action in a given state and following the optimal policy thereafter. A policy is a rule that the agent follows in selecting actions, given the state it is in. When such an action-value function is learned, the optimal policy can be constructed by simply selecting the action with the highest value in each state. One of the strengths of Q-learning is that it is able to compare the expected utility of the available actions without requiring a model of the environment. Additionally, Q-learning can handle problems with stochastic transitions and rewards, without requiring any adaptations. It has been proven that for any finite MDP, Q-learning eventually finds an optimal policy, in the sense that the expected value of the total reward return over all successive steps, starting from the current state, is the maximum achievable.

Algorithm

The problem model consists of an agent, states ⁠ $S$ ⁠ and a set of actions per state ⁠ $A$ ⁠. By performing an action $a\in A$ , the agent can move from state to state. Executing an action in a specific state provides the agent with a reward (a numerical score). The goal of the agent is to maximize its total reward. It does this by learning which action is optimal for each state. The action that is optimal for each state is the action that has the highest long-term reward. This reward is a weighted sum of the expected values of the rewards of all future steps starting from the current state, where the weight for a step from a state $\Delta t$ steps into the future is calculated as $\gamma ^{\Delta t}$ . Here, $\gamma$ is a number between 0 and 1 ( $0\leq \gamma \leq 1$ ) called the discount factor and trades off the importance of sooner versus later rewards. $\gamma$ may also be interpreted as the likelihood to succeed (or survive) at every step $\Delta t$ .

The algorithm therefore has a function that calculates the Quantity of a state-action combination:

Q:S\times A\to \mathbb {R}

Before learning has started, ⁠ $Q$ ⁠ returns an (arbitrary) fixed value, chosen by the designer. Then, each time the agent selects an action, and observes a reward and a new state that may depend on both the previous state and the selected action, $Q$ is updated. The core of the algorithm is a simple value iteration update. It assumes the old value and makes a correction based on the new information.

Q(s_{t},a_{t})\leftarrow \underbrace {Q(s_{t},a_{t})} _{\rm {old~value}}+\underbrace {\alpha } _{\rm {learning~rate}}\cdot \left(\overbrace {\underbrace {r_{t+1}} _{\rm {reward}}+\underbrace {\gamma } _{\rm {discount~factor}}\cdot \underbrace {\max _{a}Q(s_{t+1},a)} _{\rm {estimate~of~optimal~future~value}}} ^{\rm {learned~value}}-\underbrace {Q(s_{t},a_{t})} _{\rm {old~value}}\right)

where $r_{t+1}$ is the reward observed after performing $a_{t}$ in $s_{t}$ , and where $\alpha _{t}(s,a)$ ( $0<\alpha \leq 1$ ) is the learning rate (may be the same for all pairs).

An episode of the algorithm ends when state $s_{t+1}$ is a final state (or, "absorbing state"). However, Q-learning can also learn in non-episodic tasks. If the discount factor is lower than 1, the action values are finite even if the problem can contain infinite loops.

Note that for all final states $s_{f}$ , $Q(s_{f},a)$ is never updated but is set to the reward value $r$ . In most cases, $Q(s_{f},a)$ can be taken to be equal to zero.

Influence of variables on the algorithm

Learning rate

The learning rate or step size determines to what extent the newly acquired information will override the old information. A factor of 0 will make the agent not learn anything, while a factor of 1 would make the agent consider only the most recent information. In fully deterministic environments, a learning rate of $\alpha _{t}(s,a)=1$ is optimal. When the problem is stochastic, the algorithm still converges under some technical conditions on the learning rate, that require it to decrease to zero. In practice, often a constant learning rate is used, such as $\alpha _{t}(s,a)=0.1$ for all $t$ .^[1]

Discount factor

The discount factor ⁠ $\gamma$ ⁠ determines the importance of future rewards. A factor of 0 will make the agent "myopic" (or short-sighted) by only considering current rewards, while a factor approaching 1 will make it strive for a long-term high reward. If the discount factor meets or exceeds 1, the action values may diverge. For ⁠ $\gamma =1$ ⁠, without a terminal state, or if the agent never reaches one, all environment histories will be infinitely long, and utilities with additive, undiscounted rewards will generally be infinite.^[2] Even with a discount factor only slightly lower than 1, the Q-function learning leads to propagation of errors and instabilities when the value function is approximated with an artificial neural network.^[3] In that case, it is known that starting with a lower discount factor and increasing it towards its final value yields accelerated learning.^[4]

Initial conditions (Q₀)

Since Q-learning is an iterative algorithm, it implicitly assumes an initial condition before the first update occurs. High initial values, also known as "optimistic initial conditions",^[5] can encourage exploration: no matter what action is selected, the update rule will cause it to have lower values than the other alternative, thus increasing their choice probability. Recently, it was suggested that the first reward $r$ could be used to reset the initial conditions.^{[citation needed]} According to this idea, the first time an action is taken the reward is used to set the value of $Q$ . This will allow immediate learning in case of fixed deterministic rewards. Surprisingly, this resetting-of-initial-conditions (RIC) approach seems to be consistent with human behaviour in repeated binary choice experiments.^[6]

Implementation

Q-learning at its simplest uses tables to store data. This very quickly loses viability with increasing sizes of state/action space of the system it is monitoring/controlling. One answer to this problem is to use an (adapted) artificial neural network as a function approximator, as demonstrated by Tesauro in his Backgammon playing temporal difference learning research.^[7]

More generally, Q-learning can be combined with function approximation.^[8] This makes it possible to apply the algorithm to larger problems, even when the state space is continuous, and therefore infinitely large. Additionally, it may speed up learning in finite problems, due to the fact that the algorithm can generalize earlier experiences to previously unseen states.

Early study

Q-learning was first introduced by Watkins^[9] in 1989. The convergence proof was presented later by Watkins and Dayan^[10] in 1992.

Variants

A recent application of Q-learning to deep learning, by Google DeepMind, titled "deep reinforcement learning" or "deep Q-networks", has been successful at playing some Atari 2600 games at expert human levels. Preliminary results were presented in 2014, with a paper published in February 2015 in Nature.^[11]

Because the maximum approximated action value is used in the Q-learning update, in noisy environments Q-learning can sometimes overestimate the actions values, slowing the learning. A recent variant called Double Q-learning was proposed to correct this. ^[12] This algorithm was later combined with deep learning, as in the DQN algorithm (see above), resulting in Double DQN which was shown to outperform the original DQN algorithm. ^[13]

Delayed Q-learning is an alternative implementation of the online Q-learning algorithm, with Probably approximately correct learning (PAC).^[14]

Greedy GQ is a variant of Q-learning to use in combination with (linear) function approximation.^[15] The advantage of Greedy GQ is that convergence guarantees can be given even when function approximation is used to estimate the action values.

Q-learning may suffer from slow rate of convergence, especially when the discount factor $\gamma$ is close to one.^[16] Speedy Q-learning, a new variant of Q-learning algorithm, deals with this problem and achieves a probably same rate of convergence as model-based methods such as value iteration.^[17]

References

^ Reinforcement Learning: An Introduction. Richard Sutton and Andrew Barto. MIT Press, 1998.
^ Stuart J. Russell; Peter Norvig (2010). Artificial Intelligence: A Modern Approach (Third ed.). Prentice Hall. p. 649. ISBN 978-0136042594.
^ Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. ICML, pages 30–37, 1995
^ François-Lavet Vincent, Raphael Fonteneau, Damien Ernst. "How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies". NIPS, Deep RL workshop 2015.
^ http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node21.html
^ Shteingart, H; Neiman, T; Loewenstein, Y (May 2013). "The Role of First Impression in Operant Learning". J Exp Psychol Gen. 142 (2): 476–88. doi:10.1037/a0029550. PMID 22924882.
^ Tesauro, Gerald (March 1995). "Temporal Difference Learning and TD-Gammon". Communications of the ACM. 38 (3). doi:10.1145/203330.203343. Retrieved 2010-02-08. {{cite journal}}: Cite has empty unknown parameter: |coauthors= (help)
^ Hado van Hasselt. Reinforcement Learning in Continuous State and Action Spaces. In: Reinforcement Learning: State of the Art, Springer, pages 207-251, 2012
^ Watkins, C.J.C.H., (1989), Learning from Delayed Rewards. Ph.D. thesis, Cambridge University.
^ Watkins and Dayan, C.J.C.H., (1992), 'Q-learning.Machine Learning'
^ Mnih, Volodymyr; et al. (2015). "Human-level control through deep reinforcement learning" (PDF). 518: 529–533. {{cite journal}}: Cite journal requires |journal= (help)
^ van Hasselt, Hado (2011). "Double Q-learning" (PDF). Advances in Neural Information Processing Systems. 23: 2613–2622.
^ van Hasselt, Hado; Guez, Arthur; Silver, David (2015). "Deep reinforcement learning with double Q-learning" (PDF). AAAI Conference on Artificial Intelligence: 2094–2100.
^ Alexander L. Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L. Littman. Pac model-free reinforcement learning. In Proc. 22nd ICML 2006, pages 881–888, 2006.
^ Hamid Maei, and Csaba Szepesvári, Shalabh Bhatnagar and Richard Sutton. Toward off-policy learning control with function approximation. In proceedings of the 27th International Conference on Machine Learning, pages 719-726, 2010.
^ Csaba Szepesva ́ri. The asymptotic convergence-rate of Q-learning. Advances in Neural Information Processing Systems 10, Denver, Colorado, USA, 1997.
^ Gheshlaghi Azar, Mohammad; Munos, Remi; Ghavamzadeh, Mohammad; Kappen, Hilbert J. (2011). "Speedy Q-Learning" (PDF). Advances in Neural Information Processing Systems. 24: 2411–2419.

External links

Watkins, C.J.C.H. (1989). Learning from Delayed Rewards. PhD thesis, Cambridge University, Cambridge, England.
Strehl, Li, Wiewiora, Langford, Littman (2006). PAC model-free reinforcement learning
Reinforcement Learning: An Introduction by Richard Sutton and Andrew S. Barto, an online textbook. See "6.5 Q-Learning: Off-Policy TD Control".
Piqle: a Generic Java Platform for Reinforcement Learning
Reinforcement Learning Maze, a demonstration of guiding an ant through a maze using Q-learning.
Q-learning work by Gerald Tesauro
Q-learning work by Tesauro Citeseer Link
Q-learning algorithm implemented in processing.org language
Solution for the pole balancing problem with Q(lambda) / SARSA(lambda) and the fourier basis in javascript

[1] Reinforcement Learning: An Introduction. Richard Sutton and Andrew Barto. MIT Press, 1998.

[2] Stuart J. Russell; Peter Norvig (2010). Artificial Intelligence: A Modern Approach (Third ed.). Prentice Hall. p. 649. ISBN 978-0136042594.

[3] Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. ICML, pages 30–37, 1995

[4] François-Lavet Vincent, Raphael Fonteneau, Damien Ernst. "How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies". NIPS, Deep RL workshop 2015.

[5] ttp://webdocs.cs.ualberta.ca/~sutton/book/ebook/node21.html

[6] Shteingart, H; Neiman, T; Loewenstein, Y (May 2013). "The Role of First Impression in Operant Learning". J Exp Psychol Gen. 142 (2): 476–88. doi:10.1037/a0029550. PMID 22924882.

[CACM-7] Tesauro, Gerald (March 1995). "Temporal Difference Learning and TD-Gammon". Communications of the ACM. 38 (3). doi:10.1145/203330.203343. Retrieved 2010-02-08. {{cite journal}}: Cite has empty unknown parameter: |coauthors= (help)

[8] Hado van Hasselt. Reinforcement Learning in Continuous State and Action Spaces. In: Reinforcement Learning: State of the Art, Springer, pages 207-251, 2012

[9] Watkins, C.J.C.H., (1989), Learning from Delayed Rewards. Ph.D. thesis, Cambridge University.

[10] Watkins and Dayan, C.J.C.H., (1992), 'Q-learning.Machine Learning'

[DQN-11] Mnih, Volodymyr; et al. (2015). "Human-level control through deep reinforcement learning" (PDF). 518: 529–533. {{cite journal}}: Cite journal requires |journal= (help)

[12] van Hasselt, Hado (2011). "Double Q-learning" (PDF). Advances in Neural Information Processing Systems. 23: 2613–2622.

[13] van Hasselt, Hado; Guez, Arthur; Silver, David (2015). "Deep reinforcement learning with double Q-learning" (PDF). AAAI Conference on Artificial Intelligence: 2094–2100.

[14] Alexander L. Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L. Littman. Pac model-free reinforcement learning. In Proc. 22nd ICML 2006, pages 881–888, 2006.

[15] Hamid Maei, and Csaba Szepesvári, Shalabh Bhatnagar and Richard Sutton. Toward off-policy learning control with function approximation. In proceedings of the 27th International Conference on Machine Learning, pages 719-726, 2010.

[16] Csaba Szepesva ́ri. The asymptotic convergence-rate of Q-learning. Advances in Neural Information Processing Systems 10, Denver, Colorado, USA, 1997.

[17] Gheshlaghi Azar, Mohammad; Munos, Remi; Ghavamzadeh, Mohammad; Kappen, Hilbert J. (2011). "Speedy Q-Learning" (PDF). Advances in Neural Information Processing Systems. 24: 2411–2419.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]