Coloured-hearing synesthesia (CHS) is a perceptual phenomenon in which auditory stimuli cause additional colour experiences. To date, different forms of CHS have been reported, comprising tone-colour 1 2 , spoken word-colour 1 , music-colour 3 , or general auditory-colour synesthesia 4 . The common denominator of all these synesthesia variants is a close relationship between auditory and visual perceptual representations. Furthermore, these synesthetic experiences (as all other synesthetic forms) are fast, non-suppressible, and mostly unidirectional 5 , although cases of bidirectional synesthesia have been reported 6 7 .

Two different models^a are mainly discussed to explain the neurophysiological underpinnings of all variants of synesthesia: the two-stage cross-activation / hyper-binding model 8 , and the disinhibited feedback model 9 10 . The two-stage cross-activation / hyper-binding model was proposed on the basis of fMRI studies conducted with grapheme-colour synesthetes, and primarily relies on the physical closeness between the involved processing areas (e.g., grapheme and colour areas) 8 . According to this framework, the grapheme and colour processing areas (V4) are functionally and/or anatomically strongly interconnected. Consequently, this aberrant connectivity should result in co-activation of these areas during grapheme processing. Both perceptions are then bound together by parietal regions, resulting in hyper-binding. For CH synesthesia this would imply strong anatomical and/or functional connections between auditory and visual (as well as parietal) areas, and therefore rely on cross-activation across a quite long distance. This specific model has received some support from grapheme-colour as well as from CHS 11 12 13 14 15 16 17 18 . Otherwise, the disinhibited feedback model is based on studies demonstrating specific forms of acquired and congenital synesthesia rather than on brain imaging data (first mentioned by Armel and Ramachandran 9 and summarized by Grossenbacher & Lovelace 10 ). The disinhibited feedback model suggests that synesthesia results from disinhibited feedback from higher-level cortical areas in the processing hierarchy. With respect to CHS, this would imply that higher-level cortical areas collect information transmitted from the auditory cortex, and project this information to the brain areas eliciting the concurrent perception (for CHS to the colour area V4). On the basis of the two models, different predictions can be drawn with respect to the timing of neural activations associated with the concurrent perception. The cross-activation / hyper-binding model proposes simultaneous activation of the brain areas involved in processing the inducer and the concurrent. By contrast, the disinhibition model proposes that the neural activation associated with the processing of the inducer should precede that associated with the processing of the concurrent 19 . However, none of these models makes any assumptions about the activation of the brain areas processing the inducer.

CHS (as all other forms of synesthesia) can be seen as a special variant of audiovisual (AV) integration. In fact, even though visual stimuli are not physically present, they are perceived during auditory stimulation. Recent research in non-synesthetes has been dedicated to identify the time point during perception at which cross-modal stimuli are merged to a single percept. However, it is currently debated whether cross-modal integration happens at very early or rather later processing stages. One idea is that AV stimuli would impact primary sensory areas due to feedback loops from higher-order multisensory areas 20 . Other researchers rather proposed that AV stimuli would influence the early stages of sensory processing by feedforward inputs and lateral connections between the involved primary sensory areas 21 . Early and automatic AV integration mechanisms have been demonstrated in non-synesthetes in the context of the McGurk illusion effect 22 23 24 or even in skilled readers while processing graphemes 25 . Of particular interest for the present work is that these previous studies investigated AV interactions by evaluating a pre-attentive component of the event-related potential (ERP), namely the mismatch negativity (MMN). The MMN indicates an automatic and pre-attentive auditory deviance, and one of its major sources is located in the auditory cortex (primary and secondary). However, also sources extending into brain areas outside the auditory cortex (superior temporal sulcus, temporal and parietal areas) have been reported 26 27 28 . The MMN is evoked between 100 and 250 ms after stimulus onset when a rarely presented sound deviates (the deviant) from a frequently presented standard sound (the standard) in one or more dimensions. It is important to mention that the MMN is even evoked when the subjects focus attention on other aspects than the auditory stimuli 29 30 . Therefore, it is assumed that the MMN reflects pre-attentive processing. Previous studies which made use of the MMN for examining AV interactions have repeatedly shown that the amplitude of the MMN is larger in response to auditory stimuli which are automatically integrated with visual features. For example, skilled readers with several years of practise demonstrate enhanced MMN amplitudes in response to phonemes when they are presented simultaneously with the corresponding graphemes 25 .

Since none of the previous synesthesia studies made any assumptions about the activation of the brain areas involved in processing the inducer, in the present work we used the MMN in order to examine whether tone-colour associations modulate brain activity in the auditory-related cortex of CH synesthetes. In other words, we were interested in examining whether synesthetic AV processing happens pre-attentively at very early stages of auditory processing. This is of particular interest for the question whether synesthetic experiences are driven by early (perceptual, bottom-up) or late (cognitive, top-down) processing steps. In this context, recent work has proposed that both bottom-up and top-down processes might contribute to synesthesia 31 . For example, Ramachandran and Hubbard 32 classified grapheme-colour synesthetes as “higher” synesthetes for whom the grapheme “concept” is critical, and “lower” synesthetes for whom the “percept” of the physical grapheme is necessary to elicit synes-thetic experience. According to this classification, “lower” synesthetes would show greater neurophysiological modulation in early processing stages, whereas “higher” synesthetes would show greater neurophysiological modulation in later ones.

Based on previous work showing that AV double deviations (i.e., the simultaneously presented visual and auditory stimuli deviate from the simultaneously presented visual and auditory standards) result in larger MMN 25 , we expect to find larger MMN responses to tones in CH synesthetes compared to non-synesthetes. In fact, for CH synesthetes each acoustic deviant is automatically processed as double deviation. This means that tones deviating from the standard will automatically induce the perception of a different colour (double deviation: tone and concurrent colour of the deviant diverge from the standard). Hence, the larger the double deviation is, the larger the MMN should be. However, small frequency differences between the standard and the deviant often induce the same colour experience in CH synesthetes. In this specific case, the MMN should not differ between CH synesthetes and non-synesthetes since the deviant only differs in the auditory dimension while the colour perception does not change.

To summarize, in the present work we used the MMN to investigate the timing characteristics as well as the automaticity of tone-colour associations in CH synesthetes. Our hypothesis was that CH synesthetes will elicit increased MMN amplitudes in response to the tones inducing a change in colour perception. In line with our hypothesis, we found larger MMN amplitudes in CH synesthetes, compared to non-synesthetes, in response to the deviants differing at least 1 semitone from the standard (i.e., 416 and 264 Hz tones). These deviants consistently induced different concurrent colours, as provided by the behavioural data. Otherwise, the deviant tone differing from the standard in 1/10 semitone (i.e., 438 Hz) did not elicit larger MMN amplitudes in the CH synesthetes. This is indeed not surprising. In fact, for most synesthetes this tone induced similar colour experiences as those elicited in response to the standard tone.

The differential MMN responses we revealed between the two groups in the time range between 100 and 150 ms after stimulus onset indicate that the concurrent colour perception occurs early in the processing stream. Since the MMN is known to be evoked automatically, pre-attentively, and with no task demands, our findings strongly support the view that tone-colour coupling in CH synesthetes occurs (or starts) automatically and pre-attentively. In both groups, the MMN source estimation (LORETA) revealed strong current densities originating from bilateral perisylvian brain regions, comprising the primary and secondary auditory cortex, the superior temporal sulcus, and the middle temporal gyrus. We also estimated MMN sources originating from the inferior and superior parietal lobules. In addition, voxel-based statistical comparisons between the two groups revealed stronger current densities in right-sided auditory-related brain regions as well as in the inferior and superior parietal lobule of CH synesthetes. The same statistical analysis also yielded stronger bilateral current densities in CH synesthetes in the extrastriate cortex, precuneus, cuneus, and ventral part of the extrastriate cortex. Thus, 100–150 ms after tone presentation, we identified a distributed network which was significantly differently activated in CH synesthetes compared to non-synesthetes. This network comprises brain areas which have previously been identified to be involved in synesthetic experiences, like the parietal and the extrastriate cortex 11 12 13 15 . Furthermore, since a major source of the MMN was estimated in the auditory cortex (primary and secondary areas), it is possible that the coupling between auditory and visual experiences is driven by brain activity originating from the auditory-related cortex.

To date, there is only meagre evidence for a specific involvement of the auditory system in CHS 1 2 15 35 36 . Whereas visual inputs into the auditory cortex have been described in humans 15 37 38 39 , non-human primates 40 41 , and ferrets 42 , a modulation of the auditory cortex may also be mediated by thalamo-cortical interactions 43 and/or feedback loops from multisensory brain areas to the auditory cortex 5 , or even by multimodal response properties of neurons situated in the primary auditory cortex and posterior belt areas 44 . The few brain-imaging 3 15 35 45 46 and electrophysiological 1 36 47 studies dedicated to investigate the neural underpinnings of CHS have reported conflicting findings with respect to brain responses in the auditory cortex. In fact, some authors provided evidence for a modulation of auditory-related brain regions in CH synesthetes 1 15 35 36 , whereas others could not confirm this finding 3 45 46 47 .

Those studies which have used fMRI techniques to study hemodynamic responses in the auditory cortex of CH synesthetes 3 15 35 are partly problematic for several reasons. First of all, it should be mentioned that the fMRI environment is relatively loud (even when using less loud FLASH sequences). This loudness not only disturbs the subjects, but also causes substantial activation in the auditory cortex, and therefore contaminates the stimuli-related activations 48 49 50 51 . In addition, the scanner noise will (at least partly) induce concurrent colour perceptions. Consequently, it results unclear whether the identified hemodynamic responses in the auditory cortex are induced by the stimuli or rather by the noise of the scanner. In contrast, the EEG technique is particular fruitful in that it permits to measure brain responses originating from the auditory cortex which are uncontaminated by noise. In previous EEG studies performed with CH synesthetes, differences in the early N1 component of the auditory evoked potential have been identified 1 36 . The associated intracerebral sources were located in the primary and secondary auditory cortex. Together with our data, these previous studies support the pivotal role of the auditory cortex in generating the concurrent colour perception in CH synesthetes. These results are even in line with a previous study of Hanggi and colleagues 2 who analysed the specific anatomical features of the auditory cortex in the multiple synesthete E.S. who is a tone-colour synesthete. Probabilistic fibre tractography revealed hyperconnectivity especially between the auditory and insular cortices. Thus, we may speculate whether the auditory cortex of CH synesthetes is anatomically and functionally stronger connected with the adjacently located insula, and via the insula with higher-order integration centres. This perspective is further supported by a recent resting state EEG study of our group conducted with CH synesthetes 13 . This specific study identified a strong hub in the auditory cortex of CH synesthetes and emphasize that this area is functionally strongly interconnected with other brain areas 13 .

In the present work, we also revealed stronger P3a amplitudes at frontal scalp sites in CH synesthetes compared to non-synesthetes. These larger P3a amplitudes possibly indicate that the bottom-up attention network is stronger activated in CH synesthetes. In fact, a P3a response is thought to indicate the actual orienting of attention to a MMN-eliciting sound change occurring outside the current focus of attention 52 . Several authors have located the source (using MEG, EEG, and intracranial recordings) of the P3a response to deviant tones and novel sounds within the auditory cortex. However, also sources in the frontal cortex, parietal cortex, parahippocampal gyrus, anterior cingulate gyrus, and temporoparietal junction have been reported 53 54 55 56 57 . Statistical voxel-based analyses between the two groups indicated stronger current densities related to the P3a in CH synesthetes within the left-sided posterior superior and middle temporal gyrus, and at the posterior end of the superior temporal sulcus in the vicinity of the temporoparietal junction (in the vicinity of the temporoparietal junction). In addition, we observed a spot of stronger current density in the right-sided inferior parietal cortex in CH synesthetes. These brain regions are known to be engaged in the perception of complex sounds 58 , audiovisual processing of speech stimuli, and audiovisual integration in general 38 39 59 60 .

In the last decades, a vast amount of work has been dedicated to analyse the spatiotemporal dynamics of crossmodal processing in the brain of non-synesthetes (for an overview see 61 ). However, the results arising from these previous studies were shown to be heavily dependent on the experimental paradigms used, the nature of the information being combined, the modalities under investigation, as well as on the analytic strategies adopted by the subjects 61 . Nevertheless, meanwhile an increasing number of reports suggests that sensory brain regions are fundamentally involved in integrating visual and auditory information 62 63 64 65 . In addition, recent neuroimaging and electrophysiological work conducted with humans 27 66 67 68 69 and animals 40 70 highlighted that the synthesis of auditory and visual information can likewise occur in brain regions thought to be sensory-specific. Currently, only a few EEG studies have investigated the spatiotemporal dynamics associated with the crossmodal processing of elementary AV stimuli, such as disks, flashlights, checkerboards, colours, tones, or noise bursts 27 60 65 68 69 71 72 . These previous studies are important in that they provide evidence for the integration of non-linguistic material in sensory as well as associative brain regions of non-synesthetes. Of particular relevance for our work are two previous EEG studies 27 72 , which investigated the crossmodal processing of visually presented disks and pure tones and reported early AV interaction effects between 90 and 140 ms post stimulus onset. In particular, the authors could show that the brain responses elicited by the bimodal stimuli corresponded in latency, polarity, and topography to the N1 component of auditory-evoked ERPs, hence indicating a contribution of auditory-related brain regions to the fusion of elementary visual and auditory information.

Our results replicate the previous findings of the few published neuroimaging 3 15 35 and EEG 1 13 36 studies indicating a modulation of brain responses in the auditory-related cortex of CH synesthetes. Otherwise, in contrast to the two EEG studies of Beeli et al. 1 and Goller and colleagues 36 , in the present work we revealed that CHS is associated with increased and not reduced auditory-evoked ERP amplitudes. Although the question whether synesthetic experiences are associated with increased or decreased auditory-related brain responses remains to be fully explored, all EEG studies performed with CH synesthetes have in common a relatively early (between 100 and 200 ms) modulation of auditory-evoked ERPs as a consequence of the crossmodal linkage of auditory and visual information. In contrast to these earlier papers, we adopted a passive MMN paradigm which enables to measure brain responses that are more or less uninfluenced by attention. The two previous EEG studies examining CH synesthetes so far used active auditory discrimination tasks. Thus, it could be that the amount of attentional demands necessary to deal with the inducer may have an influence on the depression or enhancement of brain responses in the auditory-related cortex. In fact, whereas we revealed increased bottom-up driven brain responses in CH synesthetes while passively listening to auditory stimuli, Beeli et al. 1 as well as Goller and co-workers 36 found the reversed pattern; namely decreased auditory-evoked ERP amplitudes during tasks in which the participants overtly attended to the auditory stimuli. Consequently, it is possible that increased brain responses in the auditory-related cortex of synesthetes can be attenuated by the engagement of attention to the inducing stimulus. Furthermore, it is plausible to assume that the physical properties of the stimuli, or even the semantic and overlearned contents of the stimulus material, may have an influence on brain responses in the auditory cortex of CH synesthetes 73 74 75 . Finally, it is also conceivable that increased or decreased auditory-evoked ERPs may be driven by an additional superimposition of positive- or negative-going deflections, as previously suggested by Goller and colleagues 36 .

We found strong differences in the MMN amplitudes between CH synesthetes and non-synesthetes. This result indicates that in CH synesthetes tones and concurrent colour perceptions are processed early and automatically as compound stimuli. These early different electrophysiological responses in CH synesthetes are accompanied by stronger intracerebral activations in the right-sided auditory-related regions, the inferior and superior parietal lobules, and ventral parts of the occipital lobe. We also revealed stronger P3a amplitudes in the CH synesthetes, this result indicating a more pronounced implicit attention orientation to the deviating tones and their concurrent perceptions. These implicit orienting responses are accompanied by stronger current densities in the auditory-related cortex and the superior temporal sulcus region. Thus, we provide objective electrophysiological evidence indicating that changes in colour experiences in response to tones are accompanied, at least in part, by a modulation of early auditory processing steps in CH synesthetes.

^aThere are two additional models, which are currently not intensively discussed. The first is the limbic mediation hypothesis, first proposed by Richard Cytowic and Frank Wood 86 , proposing that synesthesia is mediated by the limbic system and especially the hippocampus, on which multiple sensory signals converge. The second is a hybrid model, the so-called re-entrant processing model sharing with the cross-activation model the notion of hyper-connectivity between form and colour processing areas in the fusiform gyrus, and suggests, like the disinhibited feedback model, that synesthetic colours require feedback of neural activity that originates in higher-level areas (e.g., anterior inferior temporal and posterior inferior temporal) to V4 87 .

The authors declare that they have no competing interests.

LJ conceived the study, the design, and formulated the hypotheses. SE and LJ drafted this manuscript together. SE coordinated the study and contributed to the study’s hypotheses, design, results analysis, and discussion. LR performed the EEG measurements and evaluated the data together with SE LR also contributed to the study’s hypothesis, design, results analysis, and discussion. MM was involved in drafting and commenting the manuscript. All authors read and approved the final version of this manuscript.