Email updates

Keep up to date with the latest news and content from BMC Neuroscience and BioMed Central.

This article is part of the supplement: Twentieth Annual Computational Neuroscience Meeting: CNS*2011

Open Access Poster presentation

An abstract model of the basal ganglia, reward learning and action selection

Pierre Berthet12* and Anders Lansner123

Author Affiliations

1 Department of Numerical Analysis and Computer Science, Stockholm University, Stockholm, 106 91, Sweden

2 Stockholm Brain Institute, Karolinska Institutet Stockholm, 171 77, Sweden

3 Department of Computational Biology, Royal Institute of Technology (KTH), Stockholm, 106 91, Sweden

For all author emails, please log on.

BMC Neuroscience 2011, 12(Suppl 1):P189  doi:10.1186/1471-2202-12-S1-P189

The electronic version of this article is the complete one and can be found online at: http://www.biomedcentral.com/1471-2202/12/S1/P189


Published:18 July 2011

© 2011 Berthet and Lansner; licensee BioMed Central Ltd.

This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Poster presentation

Learning is of major interest for various fields, from human pathologies to artificial intelligence. Animal and biology experiments have provided a great amount of data in this area, especially in classical and operant conditioning. Reinforcement learning is largely used in computational models in order to reproduce and explain these observations. This method enables to model a wide range of phenomena from the neuronal level to the behavior of a whole organism. Studying how information about reward or punishment is processed in the brain is crucial in the understanding of the action selection and decision-making in normal and pathological conditions. Basal Ganglia are a group of nuclei playing a major role in processing motor, associative and limbic information [1] and could be specialized in resolving conflicts between these sub-systems that compete for access to limited cognitive ressources [2]. Dopaminergic neurons activity is believed to be related to the reward prediction error and is involved in long term potentiation (LTP) and depression (LTD) in the striatum [3].

We present here an abstract computational model of the Basal Ganglia using reinforcement learning and Bayesian inference. It is based on a dual pathway architecture similar to the direct / Go and indirect / NoGo pathways that have been found in biology [4]. The units could be seen as grandmother cells, with inputs consisting of states and outputs being actions. Thus as a result of the activity in both pathways, an action will be selected based on the state and the previous outcomes of the different actions when dealing with the same state, that is if a reward has been obtained or not. One aim of our model is to be biologically plausible, weights can be updated via a triple activation Hebbian learning rule, similar to pre-synaptic activity, post-synaptic depolarisation and dopamine level [5]. The update equation is based on the probability of co-activity of the different active units. In its current form, the model uses trace activation and a simple delay mechanism. Basically, when a reward occurs, the weight between the active units is increased in the Go projection while it is decreased in the NoGo connection.

In a one to one mapping learning scheme (only one specific action for a given state triggers a reward), the simulation shows good results in both learning and re-learning, i.e. the mapping is then shuffled.

One interesting feature is the homeostasis property of the units: the variations of the sum of the outgoing and ingoing weights and bias, of a particular unit, are very small. In constrained set up (low numbers of possible actions and states), similar to experimental design, learning in stochastic reward occurrence is handled and the results are similar to the data. The response of midbrain dopamine neurons is positively correlated with the number of unrewarded trials [6] and our model produces a similar result with the prediction-error value. Future works will focus on reproducing conditioning phenomena and implementing spiking neurons.

Acknowledgements

FACETS-ITN 237955, Swedish Scientific Counsel

References

  1. Mink JW: The basal ganglia: focused selection and inhibition of competing motor programs.

    Progress in neurobiology 1996, 50(4):381-425. PubMed Abstract | Publisher Full Text OpenURL

  2. Redgrave P, Prescott T, Gurney K: The basal ganglia: a vertebrate solution to the selection problem?

    Neuroscience 1999, 89(4):1009-1023. PubMed Abstract | Publisher Full Text OpenURL

  3. Cohen MX, Frank MJ: Neurocomputational models of basal ganglia function in learning, memory and choice.

    Behavioural brain research 2009, 199(1):141-156. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  4. Surmeier DJ, Ding J, Day M, Wang Z, Shen W: D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons.

    Trends in neurosciences 2007, 30(5):228-235. PubMed Abstract | Publisher Full Text OpenURL

  5. Reynolds JNJ, Wickens JR: Dopamine-dependent plasticity of corticostriatal synapses.

    Neural Networks 2002, 15(4-6):507-521. PubMed Abstract | Publisher Full Text OpenURL

  6. Nakahara H, Itoh H, Kawagoe R, Takikawa Y, Hikosaka O: Dopamine neurons can represent context-dependent prediction error.

    Neuron 2004, 41(2):269-280. PubMed Abstract | Publisher Full Text OpenURL