The Brains Blog

There were some really good papers last month. The three I picked to summarize all involve error-based learning on fast time-scales. One involves the cerebellum in monkeys, the other involves the songbird system in…songbirds. One reason I like these examples is because they illustrate how deeply error-sensitivity is knitted into basic sensorimotor loops in non-human species. This is not really news in neuroscience, but superficially contradicts some philosophers who seem to be providing transcendental arguments that discursive linguistic practices are a necessary condition for the possibility of error. Anyway, let’s check out the science.

Singin’ in the dopamine! Birdsong plasticity

Hisey, E, Kearney, MG, and Mooney, R (2018) A common neural circuit mechanism for internally guided and externally reinforced forms of motor learning. Nat Neurosci 21: 589-597. [Pubmed]

Xiao, L, Chattree, G, Oscos, FG, Cao, M, Wanat, MJ, and Roberts, TF (2018) A Basal Ganglia Circuit Sufficient to Guide Birdsong Learning. Neuron 98: 208-221. [Pubmed]

Here we have two elegant papers showing that birds can learn to adjust individual notes in their songs in response to brief pulses of dopamine. While many of us tend to think of the dopaminergic system as an extremely course, slow reinforcement signal, these papers suggest that it can act very quickly to reinforce specific actions embedded in complex behavioral contexts.

Note if you don’t like words, there is a nice video explaining the basic results posted at the bottom of this summary.

For decades, the birdsong system has been a workhorse for the study of socially acquired vocal behavior. This vocal learning occurs in two main stages: first, during the sensory learning phase, a fledgling male memorizes a tutor song from a conspecific adult male. Then, in the sensorimotor learning phase, he will slowly come to reproduce that tutor song himself. He will start by generating discordant, uncoordinated songs, and slowly shaping his vocalizations until the song closely resembles the tutor song. A typical song will have a rich internal structure like that in the figure.

Sensorimotor learning requires auditory feedback, as the animal shapes its behavior by comparing its its current song to the memorized tutor song (Mooney, 2009). Consider what it is like when you are singing a tune and need to hit B-flat. You can hear how close you are to the target. For instance, if you are coming in a bit sharp, you will try to push your pitch down a bit for that particular note.

Similarly, birds are extremely sensitive to errors in individual notes in their songs. If you add annoying white noise (WN) when one of their notes is below a certain pitch, songbirds adjust their pitch upward on that note in order to escape the noise. In the figure below, ‘Pitch-dependent auditory feedback’, you can see this in action. When note ‘d’ is above a certain pitch, that is an escape trial, and the bird is left alone. But when it is below that threshold, we have a ‘hit’ trial and WN is played. Over the course of two days (WN1 and WN2, compared to baseline day B1), the animal slowly increases the pitch of that individual note, to avoid the annoying WN pips.

Dopamine as an internal reinforcer for song plasticity

What is the underlying neural mechanism for such spectral plasticity? One nice thing about the birdsong system is that it has been relatively well mapped anatomically. For instance, there is a dopaminergic region, the ventral tegmental area (VTA), and neurons in this region are modulated by how close notes are to their target notes (Gadagkar et al, 2016). Further, VTA projects to Area X, a nucleus in the basal ganglia that is necessary for song learning (Scharff and Nottebohm, 1991).

As dopamine is a known driver of reinforcement learning, it was hypothesized that the dopaminergic signal from VTA to Area X could drive pitch learning. To test this, Hisey et al (2018) came up with a molecular-genetic method to ablate those VTA neurons that projected to Area X, and found a significant reduction in pitch learning in those animals.

They didn’t stop there. To determine if the internal dopaminergic feedback loop was, by itself, sufficient to drive pitch learning, they stimulated the dopamine pathway in the absence of external stimuli such as white noise. That is, manipulate the dopamine levels on a trial-by-trial basis, and see if the animal adjusts its behavior accordingly.

Both papers carried out this experiment using optogenetics, tagging VTA neurons with channelrhodopsin, and activating their axons in Area X during individual notes (see ‘Pitch-dependent dopamine feedback’). Impressively, if the VTA axons were activated when a note’s pitch was above threshold, this caused the animals to shift the pitch of that note upward over multiple days (see days L1…L4 in the Figure).

Xiao et al. also did the converse experiment. They inserted an inhibitory channel into the VTA neurons, and showed that pitch-dependent inhibition of dopamine in Area X pushed birds away from the inhibited pitch, similar to white noise.

There are many details of both studies I am omitting, but it is nice to see two studies aligning so nicely in terms of methods and results. While many of us tend to think of the dopaminergic system as a sluggish, stupid reinforcement signal, these papers suggest that it can actually act quite fast to target very specific actions embedded in complex behavioral contexts. Eat your heart out, Chomsky.

Video explaining results, from Xiao et al. from the online Neuron paper.

Learning and timing in the cerebellum

Herzfeld, DJ, Jojima, Y, Soetedjo, R, and Shadmehr, R (2018) Encoding of error and learning to correct that error by the Purkinje cells of the cerebellum. Nat Neurosci 21: 736: 743. [Pubmed]

This paper suggests that the cerebellum relies on an ingenious temporal coding scheme to minimize errors in the eye movement system.

The cerebellum, or the “little brain”, actually contains significantly more neurons than the cerebral cortex, and is a key driver of motor control and learning. This paper is an investigation of the main output neurons of the cerebellar cortex, those Purkinje cells whose dendritic branches are stacked like flattened trees along its lobes (see Figure).

Simple and complex spikes: kinematics and error fields

Purkinje cells are very interesting physiologically: they generate two very different types of spikes: simple spikes (SSs) which are garden-variety action potentials, and complex spikes (CSs), which are much more complicated creatures (see Figure). CSs consist of a single, almost normal-looking spike followed by a high frequency burst of spikelets which fire at around six-hundred hertz.

This paper is the second of a beautiful two-part study of the properties of SSs and CSs during saccades. In a clever task design (see Figure), monkeys are instructed to saccade to a target. However, on a certain percentage of trials, the target location is surreptitiously switched in the middle of the saccade in order to generate an error signal (see Figure).

By examining the patterns of SSs and CSs in the context of this simple protocol, Herzfeld et al. seem to have made significant inroads into understanding the roles of these two firing modes. First, in (Herzfeld et al, 2015) they found that the firing rate of SSs predicts the kinematic features of eye-movements, such as their speed. This is what you might expect in a brain area that is directly involved in eye movement control (Ritchie, 1976).

But what about the CSs? This is where things get interesting. CSs display a beautiful directional tuning for saccade error. On each trial there tends to be at most one CS, and the probability of a CS depends on the direction of the saccade error (see ‘Error Field’ figure). For instance, some cells fire CSs when the eyes erroneously land above the target, some when the eyes land southwest, etc..

What about the magnitude of errors? Does a larger saccade error increase the probability of a CS? No. Instead, what they found was that as the error magnitude increases, this had a significant influence on the onset time, or latency, of the CS. In particular, as the error magnitude increased, the CS became much more likely to fall into a little time window about 100ms after the saccade ended (see ‘CS Timing and Error Strength’). For small errors, however, the CS latency was basically described by a uniform distribution. The potential significance of this result will be clear shortly.

We can imagine all sorts of standard “motor controlly” reasons that SSs might predict eye movement parameters, but why would you want those same cells to be error tuned? It is known that the cerebellum is involved in motor learning, so maybe such learning is driven by the error signals embodied in the CSs. To explore this possibility, Hertzfeld et al. examined how the monkeys behaved on trials after those in which there was a CS. They found significantly more behavioral change (i.e., change in saccade amplitude in the preferred error direction) after trials with a CS than trials without. Remarkably, this behavioral change was largest when that CS occured in the time window about 120 ms after saccade offset on the previous trial (see ‘CS Timing and Learning’)!

Next steps?

Cumulatively, these results (and many others I simply don’t have space to go over) suggest that the saccade system in the cerebellum monitors, on a trial-by-trial basis, the error vector between the intended and actual saccadic target. Further, when the error is larger, the temporal window within which the error signal (the CS) is generated is optimized so as to minimize errors on subsequent trials.

The Herzfeld papers provide a unique empirical window into how the brain can update itself using multiplexed signals in the cerebellum. There are lots of cool directions they could take this research. Just to mention a couple: they recorded from one neuron at a time, and it would be great to see data from many cells recorded at the same time with lots of different error preferences. If I’m pulling ridiculously hard experiments out of thin air, wouldn’t it be great to have optogenetic control of subsets of neurons tuned to a particular error? This may sound crazy, but we can tag neurons that are active during particular tasks (Liu et al, 2012; Want et al, 2017). It probably won’t happen in monkeys, but maybe someone could pull it off in mice. This would let us more directly get at causality in the system.

Additional References

Berwick et al. (2011) Songs to syntax: the linguistics of birdsong. Trends Cog Sci 15: 113-21.

Burroughs et al. (2016) The dynamic relationship between cerebellar Purkinje cell simple spikes and the spikelet number of complex spikes. J Physiol 595: 283-289.

Herzfeld et al, 2015 Encoding of action by the Purkinje cells of the cerebellum. Nature 526: 439-442.

Liu et a. (2012) Optogenetic stimulation of a hippocampal engram activates fear memory recall. Nature 484: 381–385 (2012).

Mooney, R (2009) Neural mechanisms for birdsong learning. Learning and Memory 16: 655-669.

Ritchie, L (1976) Effects of Cerebellar Lesions on Saccadic Eye Movements. 39: 1246-1256.

Scharff, C, and Nottebohm, F (1991) A comparative study of the behavioral deficits following lesions of various parts of the zebra finch song system: implications for vocal learning. J Neurosci 11: 2896-2913.

Tumer, E. C. & Brainard, M. S. Performance variability enables adaptive plasticity of ‘crystallized’ adult birdsong. Nature 450, 1240–1244 (2007).

Wang et al. (2017) A light- and calcium-gated transcription factor for imaging and manipulating activated neurons. Nat Biotechnol. 35: 864-871.