Multisensory integration, also known as multimodal integration, is the study of how information from the different sensory modalities, such as sight, sound, touch, smell, self-motion and taste, may be integrated by the nervous system. A coherent representation of objects combining modalities enables us to have meaningful perceptual experiences. Indeed, multisensory integration is central to adaptive behavior because it allows us to perceive a world of coherent perceptual entities. Multisensory integration also deals with how different sensory modalities interact with one another and alter each other’s processing.
- 1 General introduction
- 2 Theories and approaches
- 3 Principles of multisensory integration
- 4 Neural mechanisms
- 5 Development of multisensory operations
- 6 See also
- 7 References
- 8 Further reading
- 9 External links
Multi-modal perception is a scientific term that describes how humans form coherent, valid, and robust perception by processing sensory stimuli from various modalities. Surrounded by multiple objects and receiving multiple sensory stimulations, the brain is faced with the decision of how to categorize the stimuli resulting from different objects or events in the physical world. The nervous system is thus responsible for whether to integrate or segregate certain groups of temporally coincident sensory signals based on the degree of spatial and structural congruence of those stimulations. Multi-modal perception has been widely studied in cognitive science, behavioral science, and neuroscience.
Stimuli and sensory modalities
There are four attributes of stimulus: modality, intensity, location, and duration. Neo-cortex in the mammalian brain has parcellations that primarily process sensory input from one modality. For example primary visual area, V1, or primary somatosensory area, S1. These areas mostly deal with low-level stimulus features such as brightness, orientation, intensity, etc. These areas have extensive connections to each other as well as to higher association areas that further process the stimuli and are believe to integrate sensory input from various modalities. However, recently multi-sensory effects have been shown to occur in primary sensory areas as well.
The relationship between the binding problem and multisensory perception can be thought of as a question - the binding problem, and potential solution - multisensory perception. The binding problem stemmed from unanswered questions about how mammals (particularly higher primates) generate a unified, coherent perception of their surroundings from the cacophony of electromagnetic waves, chemical interactions, and pressure fluctuations that forms the physical basis of the world around us. It was investigated initially in the visual domain (colour, motion, depth, and form), then in the auditory domain, and recently in the multi cortical areas. It can be said therefore, that binding problem is central to multi sensory perception.
In the visual domain, if color, motion, depth, and form, are processed independently, where does the unified coherent conscious experience of the visual world come in? This is known as the binding problem and is usually studied entirely within visual processes, however it is clear that the binding problem is central to multisensory perception.
However, considerations of how unified conscious representations are formed are not the full focus of multisensory Integration research. It is obviously important for the senses to interact in order to maximize how efficiently people interact with the environment. For perceptual experience and behavior to benefit from the simultaneous stimulation of multiple sensory modalities, integration of the information from these modalities is necessary. Some of the mechanisms mediating this phenomenon and its subsequent effects on cognitive and behavioural processes will be examined hereafter. Perception is often defined as one's conscious experience, and thereby combines inputs from all relevant senses and prior knowledge. Perception is also defined and studied in terms of feature extraction, which is several hundred milliseconds away from conscious experience. Notwithstanding the existence of Gestalt psychology schools that advocate a holistic approach to the operation of the brain, the physiological processes underlying the formation of percepts and conscious experience have been vastly understudied. Nevertheless, burgeoning neuroscience research continues to enrich our understanding of the many details of the brain, including neural structures implicated in multisensory integration such as the superior colliculus (SC) and various cortical structures such as the superior temporal gyrus (GT) and visual and auditory association areas. Although the structure and function of the SC are well known, the cortex and the relationship between its constituent parts are presently the subject of much investigation. Concurrently, the recent impetus on integration has enabled investigation into perceptual phenomena such as the ventriloquism effect, rapid localization of stimuli and the McGurk effect; culminating in a more thorough understanding of the human brain and its functions.
Studies of sensory processing in humans and other animals has traditionally been performed one sense at a time, and to the present day, numerous academic societies and journals are largely restricted to considering sensory modalities separately ('Vision Research', 'Hearing Research' etc.). However, while there is also a long and parallel history of multisensory research (e.g., as early as Stratton's (1897) experiments on the somatosensory effects of wearing vision-distorting prism glasses), this research field has recently gained enormous interest and popularity.
Example of spatial congruent and structural congruent
When we hear a car honk, we would determine which car triggers the honk by which car we see is the spatially closest to the honk. It’s a spacial congruent example by combining visual and auditorial stimuli. On the other hand, the sound and the pictures of a TV program would be integrated as structural congruent by combining visual and auditorial stimuli. However, if the sound and the pictures were not meaningfully fit, we would segregate the two stimuli. Therefore, whether spacial or structural congruent should not only combine the stimuli but also be determined by understanding.
Theories and approaches
Literature in spatial crossmodal biases suggest that visual modality often biases information from other senses. Ladan Shams and Ulrik B. study conclude that when varying the degree of spatial congruency vision dominates what we hear. This behavior is also known as the ventriloquist effect. Further literature describes that in cases of visual and haptic integration, children younger that 8 years of age show visual dominance when required to identify object orientation. Alternatively, haptic dominance was observed when the factor to identify was object size.
Welch and Warren (1980) asserted that multisensory processes followed a modality appropriateness hypothesis, in which due to visual dominance of spatial tasks, also known as visual capture, one will always depend on vision over audition or tactition to solve spatial problems. Thus, auditory stimuli can not at all influence ones perception of the location of a visual stimulus. Concurrently, audition was considered dominant toward temporal tasks.
However, more recent studies have generated results that contradict this hypothesis. Alais and Burr (2004), found that following progressive degradation in the quality of a visual stimulus, participants’ perception of spatial location was determined progressively more by a simultaneous auditory cue. reached a similar finding. However, they also progressively changed the temporal uncertainty of the auditory cue; eventually concluding that it is the uncertainty of individual modalities that determine to what extent information from each modality is considered when forming a percept. This conclusion is similar in some respects to the ‘inverse effectiveness rule’. The extent to which multisensory integration occurs may vary according to the ambiguity of the relevant stimuli.
The theory of Bayesian integration is based on the fact that the brain must deal with a number of inputs, which vary in reliability. In dealing with these inputs, it must construct a coherent representation of the world that corresponds to reality. The Bayesian integration view is that the brain uses a form of Bayesian inference. This view has been backed up by computational modeling of such a Bayesian inference from signals to coherent representation, which shows similar characteristics to integration in the brain.
Cue combination vs. causal inference models
With the assumption of independence between various sources, traditional cue combination model is successful in modality integration. However, depending on the discrepancies between modalities, there might be different forms of stimuli fusion: integration, partial integration, and segregation. To fully understand the other two types, we have to use causal inference model without the assumption as cue combination model. This freedom gives us general combination of any numbers of signals and modalities by using Bayes’ rule to make causal inference of sensory signals.
The hierarchical vs. non-hierarchical models
The difference between two models is that hierarchical model can explicitly make causal inference to predict certain stimulus while non-hierarchical model can only predict joint probability of stimuli. However, hierarchical model is actually a special case of non-hierarchical model by setting joint prior as a weighted average of the prior to common and independent causes, each weighted by their prior probability. Based on the correspondence of these two models, we can also say that hierarchical is a mixture modal of non-hierarchical model.
Independence of likelihoods and priors
For Bayesian model, the prior and likelihood generally represent the statistics of the environment and the sensory representations. The independence of priors and likelihoods is not assured since the prior may vary with likelihood only by the representations. However, the independence has been proved by Shams with series of parameter control in multi sensory perception experiment.
Principles of multisensory integration
The contributions of Barry Stein, Alex Meredith, and their colleagues (e.g."The merging of the senses" 1993,) are widely considered to be the groundbreaking work in the modern field of multisensory integration. Through detailed long-term study of the neurophysiology of the superior colliculus, they distilled three general principles by which multisensory integration may best be described.
- The spatial rule states that multisensory integration is more likely or stronger when the constituent unisensory stimuli arise from approximately the same location.
- The temporal rule states that multisensory integration is more likely or stronger when the constituent unisensory stimuli arise at approximately the same time.
- The principle of inverse effectiveness states that multisensory integration is more likely or stronger when the constituent unisensory stimuli evoke relatively weak responses when presented in isolation.
Perceptual and behavioral consequences
A unimodal approach dominated scientific literature until the beginning of this century. Although this enabled rapid progression of neural mapping, and an improved understanding of neural structures, the investigation of perception remained relatively stagnant. The recent revitalized enthusiasm into perceptual research is indicative of a substantial shift away from reductionism and toward gestalt methodologies. Gestalt theory, dominant in the late 19th and early 20th centuries espoused two general principles: the ‘principle of totality’ in which conscious experience must be considered globally, and the ‘principle of psychophysical isomorphism’ which states that perceptual phenomena are correlated with cerebral activity. These ideas are particularly relevant in the current climate and have driven researchers to investigate the behavioural benefits of multisensory integration.
Decreasing sensory uncertainty
It has been widely acknowledged that uncertainty in sensory domains results in an increased dependence of multisensory integration. Hence, it follows that cues from multiple modalities that are both temporally and spatially synchronous are viewed neurally and perceptually as emanating from the same source. The degree of synchrony that is required for this ‘binding’ to occur is currently being investigated in a variety of approaches. It should be noted here that the integrative function only occurs to a point beyond which the subject can differentiate them as two opposing stimuli. Concurrently, a significant intermediate conclusion can be drawn from the research thus far. Multisensory stimuli that are bound into a single percept, are also bound on the same receptive fields of multisensory neurons in the SC and cortex.
Decreasing reaction time
Responses to multiple simultaneous sensory stimuli can be faster than responses to the same stimuli presented in isolation. Hershenson (1962) presented a light and tone simultaneously and separately, and asked human participants to respond as rapidly as possible to them. As the asynchrony between the onsets of both stimuli was varied, it was observed that for certain degrees of asynchrony, reaction times were decreased. These levels of asynchrony were quite small, perhaps reflecting the temporal window that exists in multisensory neurons of the SC. Further studies have analysed the reaction times of saccadic eye movements; and more recently correlated these findings to neural phenomena.
Redundant target effects
The redundant target effect is the observation that people typically respond faster to double targets (two targets presented simultaneously) than to either of the targets presented alone. This difference in latency is termed the redundancy gain (RG).
In a study done by B. Forster, C. Cavina-Pratesi, S. M. Agolioti, and G. Berlucchi(2001), normal observers responded faster to simultaneous visual and tactile stimuli than to single visual or tactile stimuli. RT to simultaneous visual and tactile stimuli was also faster than RT to simultaneous dual visual or tactile stimuli. The advantage for RT to combined visual-tactile stimuli over RT to the other types of stimulation could be accounted for by intersensory neural facilitation rather than by probability summation. These effects can be ascribed to the convergence of tactile and visual inputs onto neural centers which contain flexible multisensory representations of body parts.
It has been found that two converging bimodal stimuli can produce a perception that is not only different in magnitude than the sum of its parts, but also quite different in quality. In a classic study labeled the McGurk effect, a person’s phoneme production was dubbed with a video of that person speaking a different phoneme. The end result was the perception of a third, different phoneme. McGurk and MacDonald (1976) explained that phonemes such as ba, da, ka, ta, ga and pa can be divided into four groups, those that can be visually confused, i.e. (da, ga, ka, ta) and (ba and pa), and those that can be audibly confused. Hence, when ba – voice and ga lips are processed together, the visual modality sees ga or da, and the auditory modality hears ba or da, combining to form the percept da.
Ventriloquism has been used as the evidence for the modality appropriateness hypothesis. Ventriloquism describes the situation in which auditory location perception is shifted toward a visual cue. The original study describing this phenomenon was conducted by Howard and Templeton, (1966) after which several studies have replicated and built upon the conclusions they reached. In conditions in which the visual cue is unambiguous, visual capture reliably occurs. Thus to test the influence of sound on perceived location, the visual stimulus must be progressively degraded. Furthermore, given that auditory stimuli are more attuned to temporal changes, recent studies have tested the ability of temporal characteristics to influence the spatial location of visual stimuli. Some types of EVP - Electronic voice phenomenon, mainly the ones using sound bubles are considered a kind of modern ventriloquism technique and is played by the use of sophisticated software, computers and sound equipment.
The double flash illusion was reported as the first illusion to show that visual stimuli can be qualitatively altered by audio stimuli. In the standard paradigm participants are presented combinations of one to four flashes accompanied by zero to 4 beeps. They were then asked to say how many flashes they perceived. Participants perceived illusory flashes when there were more beeps than flashes. fMRI studies have shown that there is crossmodal activation in early, low level visual areas, which was qualitatively similar to the perception of a real flash. This suggests that the illusion reflects subjective perception of the extra flash. Further, studies suggest that timing of multisensory activation in unisensory cortexes is too fast to be mediated by a higher order integration suggesting feed forward or lateral connections. One study has revealed the same effect but from vision to audition, as well as fission rather than fusion effects, although the level of the auditory stimulus was reduced to make it less salient for those illusions affecting audition.
Rubber hand illusion
In the rubber hand illusion (RHI)(Botvinick & Cohen, 1999), human participants view a dummy hand being stroked with a paintbrush, while they feel a series of identical brushstrokes applied to their own hand, which is hidden from view. If this visual and tactile information is applied synchronously, and if the visual appearance and position of the dummy hand is similar to one's own hand, then people may feel that the touches on their own hand are coming from the dummy hand, and even that the dummy hand is, in some way, their own hand. This is an early form of body transfer illusion. The RHI is an illusion of vision, touch, and posture (proprioception), but a similar illusion can also be induced with touch and proprioception. The very first report of this kind of illusion may have been as early as 1937 (Tastevin, 1937).
Body transfer illusion
Body transfer illusion involves the use of typically, virtual reality devices to induce the illusion in the subject that the body of another person or being is the subject's own body.
The superior colliculus (SC) or Optic tectum (OT) is part of the tectum, located in the midbrain, superior to the brainstem and inferior to the thalamus. It contains seven layers of alternating white and grey matter, of which the superficial contain topographic maps of the visual field; and deeper layers contain overlapping spatial maps of the visual, auditory and somatosensory modalities. The structure receives afferents directly from the retina, as well as from various regions of the cortex (primarily the occipital lobe), the spinal cord and the inferior colliculus. It sends efferents to the spinal cord, cerebellum, thalamus and occipital lobe via the lateral geniculate nucleus (LGN). The structure contains a high proportion of multisensory neurons and plays a role in the motor control of orientation behaviours of the eyes, ears and head.
Receptive fields from somatosensory, visual and auditory modalities converge in the deeper layers to form a two-dimensional multisensory map of the external world. Here, objects straight ahead are represented caudally and objects on the periphery are represented rosterally. Similarly, locations in superior sensory space are represented medially, and inferior locations are represented laterally.
However, in contrast to simple convergence, the SC integrates information to create an output that differs from the sum of its inputs. Following a phenomenon labelled the ‘spatial rule’, neurons are excited if stimuli from multiple modalities fall on the same or adjacent receptive fields, but are inhibited if the stimuli fall on disparate fields. Excited neurons may then proceed to innervate various muscles and neural structures to orient an individual’s behaviour and attention toward the stimulus. Neurons in the SC also adhere to the ‘temporal rule’, in which stimulation must occur within close temporal proximity to excite neurons. However, due to the varying processing time between modalities and the relatively slower speed of sound to light, it has been found the neurons may be optimally excited when stimulated some time apart.
Single neurons in the macaque putamen have been shown to have visual and somatosensory responses closely related to those in the polysensory zone of the premotor cortex and area 7b in the parietal lobe.
Multisensory neurons exist in a large number of locations, often integrated with unimodal neurons. They have recently been discovered in areas previously thought to be modality specific, such as the somatosensory cortex; as well as in clusters at the borders between the major cerebral lobes, such as the occipito-parietal space and the occipito-temporal space.
However, in order to undergo such physiological changes, there must exist continuous connectivity between these multisensory structures. It is generally agreed that information flow within the cortex follows a hierarchical configuration. Hubel and Wiesel showed that receptive fields and thus the function of cortical structures, as one proceeds out from V1 along the visual pathways, become increasingly complex and specialized. From this it was postulated that information flowed outwards in a feed forward fashion; the complex end products eventually binding to form a percept. However, via fMRI and intracranial recording technologies, it has been observed that the activation time of successive levels of the hierarchy does not correlate with a feed forward structure. That is, late activation has been observed in the striate cortex, markedly after activation of the prefrontal cortex in response to the same stimulus.
Complementing this, afferent nerve fibres have been found that project to early visual areas such as the lingual gyrus from late in the dorsal (action) and ventral (perception) visual streams, as well as from the auditory association cortex. Feedback projections have also been observed in the opossum directly from the auditory association cortex to V1. This last observation currently highlights a point of controversy within the neuroscientific community. Sadato et al. (2004) concluded, in line with Bernstein et al. (2002), that the primary auditory cortex (A1) was functionally distinct from the auditory association cortex, in that it was void of any interaction with the visual modality. They hence concluded that A1 would not at all be effected by cross modal plasticity. This concurs with Jones and Powell’s (1970) contention that primary sensory areas are connected only to other areas of the same modality.
In contrast, the dorsal auditory pathway, projecting from the temporal lobe is largely concerned with processing spatial information, and contains receptive fields that are topographically organized. Fibers from this region project directly to neurons governing corresponding receptive fields in V1. The perceptual consequences of this have not yet been empirically acknowledged. However, it can be hypothesized that these projections may be the precursors of increased acuity and emphasis of visual stimuli in relevant areas of perceptual space. Consequently, this finding rejects Jones and Powell’s (1970) hypothesis and thus is in conflict with Sadato et al.’s (2004) findings. A resolution to this discrepancy includes the possibility that primary sensory areas can not be classified as a single group, and thus may be far more different from what was previously thought. Regardless, further research is necessary for a definitive resolution.
||This section provides insufficient context for those unfamiliar with the subject. (July 2013)|
Area F4 in macaques
Polysensory zone of premotor cortex (PZ) in macaques
||This section provides insufficient context for those unfamiliar with the subject. (July 2013)|
Primary visual cortex (V1)
Lingual gyrus in humans
Lateral occipital complex (LOC), including lateral occipital tactile visual area (LOtv)
||This section provides insufficient context for those unfamiliar with the subject. (July 2013)|
Ventral intraparietal sulcus (VIP) in macaques
Lateral intraparietal sulcus (LIP) in macaques
Area 7b in macaques
Second somatosensory cortex (SII)
||This section provides insufficient context for those unfamiliar with the subject. (July 2013)|
Primary auditory cortex (A1)
Superior temporal cortex (STG/STS/PT) Audio visual cross modal interactions are known to occur in the auditory association cortex which lies directly inferior to the Sylvian fissure in the temporal lobe. Plasticity was observed in the superior temporal gyrus (STG) by Petitto et al. (2000). Here, it was found that the STG was more active during stimulation in native deaf signers compared to hearing non signers. Concurrently, further research has revealed differences in the activation of the Planum temporale (PT) in response to non linguistic lip movements between the hearing and deaf; as well as progressively increasing activation of the auditory association cortex as previously deaf participants gain hearing experience via a cochlear implant.
Rostral lateral suprasylvian sulcus (rLS) in cats
The most significant interaction between these two systems (corticotectal interactions) is the connection between the anterior ectosylvian sulcus (AES), which lies at the junction of the parietal, temporal and frontal lobes, and the SC. The AES is divided into three unimodal regions with multisensory neurons at the junctions between these sections. (Jiang & Stein, 2003). Neurons from the unimodal regions project to the deep layers of the SC and influence the multiplicative integration effect. That is, although they can receive inputs from all modalities as normal, the SC can not enhance or depress the effect of multisensory stimulation without input from the AES.
Concurrently, the multisensory neurons of the AES, although also integrally connected to unimodal AES neurons, are not directly connected to the SC. This pattern of division is reflected in other areas of the cortex, resulting in the observation that cortical and tectal multisensory systems are somewhat dissociated. Stein, London, Wilkinson and Price (1996) analysed the perceived luminance of an LED in the context of spatially disparate auditory distracters of various types. A significant finding was that a sound increased the perceived brightness of the light, regardless of their relative spatial locations, provided the light’s image was projected onto the fovea. Here, the apparent lack of the spatial rule, further differentiates cortical and tectal multisensory neurons. Little empirical evidence exists to justify this dichotomy. Nevertheless, cortical neurons governing perception, and a separate sub cortical system governing action (orientation behavior) is synonymous with the perception action hypothesis of the visual stream. Further investigation into this field is necessary before any substantial claims can be made.
Dual "what" and "where" multisensory routes
Research suggests the existence of two multisensory routes for "what" and "where". The "what" route identifying the identity of things involving area Brodmann area 9 in the right inferior frontal gyrus and right middle frontal gyrus, Brodmann area 13 and Brodmann area 45 in the right insula-inferior frontal gyrus area, and Brodmann area 13 bilaterally in the insula. The "where" route detecting their spatial attributes involving the Brodmann area 40 in the right and left inferior parietal lobule and the Brodmann area 7 in the right precuneus-superior parietal lobule and Brodmann area 7 in the left superior parietal lobule.
Development of multisensory operations
All species equipped with multiple sensory systems, utilize them in an integrative manner to achieve action and perception. However, in most species, especially higher mammals, the ability to integrate develops in parallel with physical and cognitive maturity. Classically, two opposing views that are principally modern manifestations of the nativist/empiricist dichotomy have been put forth. The integration (empiricist) view states that at birth, sensory modalities are not at all connected. Hence, it is only through active exploration that plastic changes can occur in the nervous system to initiate holistic perceptions and actions. Conversely, the differentiation (nativist) perspective asserts that the young nervous system is highly interconnected; and that during development, modalities are gradually differentiated as relevant connections are rehearsed and the irrelevant are discarded.
Using the SC as a model, the nature of this dichotomy can be analysed. In the newborn cat, deep layers of the SC contain only neurons responding to the somatosensory modality. Within a week, auditory neurons begin to occur, but it is not until two weeks after birth that the first multisensory neurons appear. Further changes continue, with the arrival of visual neurons after three weeks, until the SC has achieved its fully mature structure after three to four months. Concurrently in species of monkey, newborns are endowed with a significant complement of multisensory cells; however, along with cats there is no integration effect apparent until much later. This delay is thought to be the result of the relatively slower development of cortical structures including the AES; which as stated above, is essential for the existence of the integration effect.
Furthermore, it was found by Wallace (2004) that cats raised in a light deprived environment had severely underdeveloped visual receptive fields in deep layers of the SC. Although, receptive field size has been shown to decrease with maturity, the above finding suggests that integration in the SC is a function of experience. Nevertheless, the existence of visual multisensory neurons, despite a complete lack of visual experience, highlights the apparent relevance of nativist viewpoints. Multisensory development in the cortex has been studied to a lesser extent, however a similar study to that presented above was performed on cats whose optic nerves had been severed. These cats displayed a marked improvement in their ability to localize stimuli through audition; and consequently also showed increased neural connectivity between V1 and the auditory cortex. Such plasticity in early childhood allows for greater adaptability, and thus more normal development in other areas for those with a sensory deficit.
In contrast, following the initial formative period, the SC does not appear to display any neural plasticity. Despite this, habituation and sensititisation over the long term is known to exist in orientation behaviors. This apparent plasticity in function has been attributed to the adaptability of the AES. That is, although neurons in the SC have a fixed magnitude of output per unit input, and essentially operate an all or nothing response, the level of neural firing can be more finely tuned by variations in input by the AES.
Although there is evidence for either perspective of the integration/differentiation dichotomy, a significant body of evidence also exists for a combination of factors from either view. Thus, analogous to the broader nativist/empiricist argument, it is apparent that rather than a dichotomy, there exists a continuum, such that the integration and differentiation hypotheses are extremes at either end.
- Stein, BE.; Stanford, TR.; Rowland, BA. (Dec 2009). "The neural basis of multisensory integration in the midbrain: its organization and maturation.". Hear Res 258 (1-2): 4–15. doi:10.1016/j.heares.2009.03.012. PMC 2787841. PMID 19345256.
- Lewkowicz DJ, Ghazanfar AA (November 2009). "The emergence of multisensory systems through perceptual narrowing". Trends Cogn. Sci. (Regul. Ed.) 13 (11): 470–8. doi:10.1016/j.tics.2009.08.004. PMID 19748305.
- Lemus L, Hernández A, Luna R, Zainos A, Romo R (July 2010). "Do sensory cortices process more than one sensory modality during perceptual judgments?". Neuron 67 (2): 335–48. doi:10.1016/j.neuron.2010.06.015. PMID 20670839.
- Zmigrod, S.; Hommel, B. (Jan 2010). "Temporal dynamics of unimodal and multimodal feature binding.". Atten Percept Psychophys 72 (1): 142–52. doi:10.3758/APP.72.1.142. PMID 20045885.
- Wagemans, J.; Elder, JH.; Kubovy, M.; Palmer, SE.; Peterson, MA.; Singh, M.; von der Heydt, R. (Nov 2012). "A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization.". Psychol Bull 138 (6): 1172–217. doi:10.1037/a0029333. PMID 22845751.
- Wagemans, J.; Feldman, J.; Gepshtein, S.; Kimchi, R.; Pomerantz, JR.; van der Helm, PA.; van Leeuwen, C. (Nov 2012). "A century of Gestalt psychology in visual perception: II. Conceptual and theoretical foundations.". Psychol Bull 138 (6): 1218–52. doi:10.1037/a0029334. PMID 22845750.
- Stein, BE.; Rowland, BA. (2011). "Organization and plasticity in multisensory integration: early and late experience affects its governing principles.". Prog Brain Res 191: 145–63. doi:10.1016/B978-0-444-53752-2.00007-2. PMC 3245961. PMID 21741550.
- Recanzone, GH. (Dec 2009). "Interactions of auditory and visual stimuli in space and time.". Hear Res 258 (1-2): 89–99. doi:10.1016/j.heares.2009.04.009. PMC 2787663. PMID 19393306.
- Smith, E.; Duede, S.; Hanrahan, S.; Davis, T.; House, P.; Greger, B. (2013). "Seeing is believing: neural representations of visual stimuli in human auditory cortex correlate with illusory auditory perceptions.". PLoS ONE 8 (9): e73148. doi:10.1371/journal.pone.0073148. PMC 3762867. PMID 24023823.
- Fodor, Jerry A. (1983). Modularity of mind: an essay on faculty psychology. Cambridge, Mass: MIT Press. ISBN 0-262-06084-1. OCLC 551957787.
- Stratton, George M. (1896). "Some preliminary experiments on vision without inversion of the retinal image.". Psychological Review 3 (6): 611–617. doi:10.1037/h0072918.
- Witten, IB.; Knudsen, EI. (Nov 2005). "Why seeing is believing: merging auditory and visual worlds.". Neuron 48 (3): 489–96. doi:10.1016/j.neuron.2005.10.020. PMID 16269365.
- Shams, L.; Beierholm, UR. (Sep 2010). "Causal inference in perception.". Trends Cogn Sci 14 (9): 425–32. doi:10.1016/j.tics.2010.07.001. PMID 20705502.
- Gori, M.; Del Viva, M.; Sandini, G.; Burr, DC. (May 2008). "Young children do not integrate visual and haptic form information.". Curr Biol 18 (9): 694–8. doi:10.1016/j.cub.2008.04.036. PMID 18450446.
- Gori, M.; Sandini, G.; Burr, D. (2012). "Development of visuo-auditory integration in space and time.". Front Integr Neurosci 6: 77. doi:10.3389/fnint.2012.00077. PMC 3443931. PMID 23060759.
- Welch RB, Warren DH (November 1980). "Immediate perceptual response to intersensory discrepancy". Psychol Bull 88 (3): 638–67. PMID 7003641.
- Alais D, Burr D (February 2004). "The ventriloquist effect results from near-optimal bimodal integration". Curr. Biol. 14 (3): 257–62. doi:10.1016/j.cub.2004.01.029. PMID 14761661.
- Deneve S, Pouget A (2004). "Bayesian multisensory integration and cross-modal spatial links". J. Physiol. Paris 98 (1-3): 249–58. doi:10.1016/j.jphysparis.2004.03.011. PMID 15477036.
- Pouget A, Deneve S, Duhamel JR (September 2002). "A computational perspective on the neural basis of multisensory spatial representations". Nature Reviews Neuroscience 3 (9): 741–7. doi:10.1038/nrn914. PMID 12209122.
- Vilares, I.; Kording, K. (Apr 2011). "Bayesian models: the structure of the world, uncertainty, behavior, and the brain.". Annals of the New York Academy of Sciences 1224: 22–39. doi:10.1111/j.1749-6632.2011.05965.x. PMID 21486294.
- Beierholm, UR.; Quartz, SR.; Shams, L. (2009). "Bayesian priors are encoded independently from likelihoods in human multisensory perception.". J Vis 9 (5): 23.1–9. doi:10.1167/9.5.23. PMID 19757901.
- Stein, Barry; Meredith, M. Alex (1993). The merging of the senses. Cambridge, Mass: MIT Press. ISBN 0-262-19331-0. OCLC 25869284.
- Meredith, MA.; Stein, BE. (Feb 1986). "Spatial factors determine the activity of multisensory neurons in cat superior colliculus.". Brain Res 365 (2): 350–4. doi:10.1016/0006-8993(86)91648-3. PMID 3947999.
- King AJ, Palmer AR (1985). "Integration of visual and auditory information in bimodal neurones in the guinea-pig superior colliculus". Exp Brain Res 60 (3): 492–500. PMID 4076371.
- Meredith, MA.; Nemitz, JW.; Stein, BE. (Oct 1987). "Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors.". J Neurosci 7 (10): 3215–29. PMID 3668625.
- Meredith MA, Stein BE (July 1983). "Interactions among converging sensory inputs in the superior colliculus". Science 221 (4608): 389–91. PMID 6867718.
- Meredith, MA.; Stein, BE. (Sep 1986). "Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration.". J Neurophysiol 56 (3): 640–62. PMID 3537225.
- Hershenson M (March 1962). "Reaction time as a measure of intersensory facilitation". J Exp Psychol 63: 289–93. PMID 13906889.
- Hughes, HC.; Reuter-Lorenz, PA.; Nozawa, G.; Fendrich, R. (Feb 1994). "Visual-auditory interactions in sensorimotor processing: saccades versus manual responses.". J Exp Psychol Hum Percept Perform 20 (1): 131–53. doi:10.1037/0096-1522.214.171.124. PMID 8133219.
- Wallace, Mark T. (2004). "The development of multisensory processes". Cognitive Processing 5 (2): 69–83. doi:10.1007/s10339-004-0017-z. ISSN 1612-4782.
- Ridgway N, Milders M, Sahraie A (May 2008). "Redundant target effect and the processing of colour and luminance". Exp Brain Res 187 (1): 153–60. doi:10.1007/s00221-008-1293-0. PMID 18264703.
- Forster B, Cavina-Pratesi C, Aglioti SM, Berlucchi G (April 2002). "Redundant target effect and intersensory facilitation from visual-tactile interactions in simple reaction time". Exp Brain Res 143 (4): 480–7. doi:10.1007/s00221-002-1017-9. PMID 11914794.
- McGurk H, MacDonald J (1976). "Hearing lips and seeing voices". Nature 264 (5588): 746–8. doi:10.1038/264746a0. PMID 1012311.
- Nath, AR.; Beauchamp, MS. (Jan 2012). "A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion.". Neuroimage 59 (1): 781–7. doi:10.1016/j.neuroimage.2011.07.024. PMC 3196040. PMID 21787869.
- Hairston WD, Wallace MT, Vaughan JW, Stein BE, Norris JL, Schirillo JA (January 2003). "Visual localization ability influences cross-modal bias". J Cogn Neurosci 15 (1): 20–9. doi:10.1162/089892903321107792. PMID 12590840.
- Shams L, Kamitani Y, Shimojo S (December 2000). "Illusions. What you see is what you hear". Nature 408 (6814): 788. doi:10.1038/35048669. PMID 11130706.
- Watkins S, Shams L, Josephs O, Rees G (August 2007). "Activity in human V1 follows multisensory perception". Neuroimage 37 (2): 572–8. doi:10.1016/j.neuroimage.2007.05.027. PMID 17604652.
- Shams L, Iwaki S, Chawla A, Bhattacharya J (April 2005). "Early modulation of visual cortex by sound: an MEG study". Neurosci. Lett. 378 (2): 76–81. doi:10.1016/j.neulet.2004.12.035. PMID 15774261.
- Andersen TS, Tiippana K, Sams M (November 2004). "Factors influencing audiovisual fission and fusion illusions". Brain Res Cogn Brain Res 21 (3): 301–308. doi:10.1016/j.cogbrainres.2004.06.004. PMID 15511646.
- Botvinick M, Cohen J (February 1998). "Rubber hands 'feel' touch that eyes see". Nature 391 (6669): 756. doi:10.1038/35784. PMID 9486643.
- Ehrsson HH, Holmes NP, Passingham RE (November 2005). "Touching a rubber hand: feeling of body ownership is associated with activity in multisensory brain areas". J. Neurosci. 25 (45): 10564–73. doi:10.1523/JNEUROSCI.0800-05.2005. PMC 1395356. PMID 16280594.
- Holmes NP, Crozier G, Spence C (June 2004). "When mirrors lie: "visual capture" of arm position impairs reaching performance". Cogn Affect Behav Neurosci 4 (2): 193–200. doi:10.3758/CABN.4.2.193. PMC 1314973. PMID 15460925.
- Bergman, Ronald A.; Afifi, Adel K. (2005). Functional neuroanatomy: text and atlas. New York: McGraw-Hill. ISBN 0-07-140812-6. OCLC 475017241.
- Giard MH, Peronnet F (September 1999). "Auditory-visual integration during multimodal object recognition in humans: a behavioral and electrophysiological study". J Cogn Neurosci 11 (5): 473–90. PMID 10511637.
- Miller LM, D'Esposito M (June 2005). "Perceptual fusion and stimulus coincidence in the cross-modal integration of speech". J. Neurosci. 25 (25): 5884–93. doi:10.1523/JNEUROSCI.0896-05.2005. PMID 15976077.
- Graziano MS, Gross CG (1993). "A bimodal map of space: somatosensory receptive fields in the macaque putamen with corresponding visual receptive fields". Exp Brain Res 97 (1): 96–109. doi:10.1007/BF00228820. PMID 8131835.
- Gentile, G.; Petkova, VI.; Ehrsson, HH. (Feb 2011). "Integration of visual and tactile signals from the hand in the human brain: an FMRI study.". J Neurophysiol 105 (2): 910–22. doi:10.1152/jn.00840.2010. PMC 3059180. PMID 21148091.
- Wallace MT, Ramachandran R, Stein BE (February 2004). "A revised view of sensory cortical parcellation". Proc. Natl. Acad. Sci. U.S.A. 101 (7): 2167–72. doi:10.1073/pnas.0305697101. PMC 357070. PMID 14766982.
- Clavagnier S, Falchier A, Kennedy H (June 2004). "Long-distance feedback projections to area V1: implications for multisensory integration, spatial awareness, and visual consciousness". Cogn Affect Behav Neurosci 4 (2): 117–26. doi:10.3758/CABN.4.2.117. PMID 15460918.
- Foxe JJ, Simpson GV (January 2002). "Flow of activation from V1 to frontal cortex in humans. A framework for defining "early" visual processing". Exp Brain Res 142 (1): 139–50. doi:10.1007/s00221-001-0906-7. PMID 11797091.
- Macaluso E, Frith CD, Driver J (August 2000). "Modulation of human visual cortex by crossmodal spatial attention". Science 289 (5482): 1206–8. doi:10.1126/science.289.5482.1206. PMID 10947990.
- Sadato N, Yamada H, Okada T, et al. (December 2004). "Age-dependent plasticity in the superior temporal sulcus in deaf humans: a functional MRI study". BMC Neurosci 5: 56. doi:10.1186/1471-2202-5-56. PMC 539237. PMID 15588277.
- Bernstein LE, Auer ET, Moore JK, Ponton CW, Don M, Singh M (March 2002). "Visual speech perception without primary auditory cortex activation". NeuroReport 13 (3): 311–5. PMID 11930129.
- Jones EG, Powell TP (1970). "An anatomical study of converging sensory pathways within the cerebral cortex of the monkey". Brain 93 (4): 793–820. doi:10.1093/brain/93.4.793. PMID 4992433.
- Grefkes, C.; Fink, GR. (Jul 2005). "The functional organization of the intraparietal sulcus in humans and monkeys.". J Anat 207 (1): 3–17. doi:10.1111/j.1469-7580.2005.00426.x. PMID 16011542.
- Murata, A.; Fadiga, L.; Fogassi, L.; Gallese, V.; Raos, V.; Rizzolatti, G. (Oct 1997). "Object representation in the ventral premotor cortex (area F5) of the monkey.". J Neurophysiol 78 (4): 2226–30. PMID 9325390.
- Smiley, JF.; Falchier, A. (Dec 2009). "Multisensory connections of monkey auditory cerebral cortex.". Hear Res 258 (1-2): 37–46. doi:10.1016/j.heares.2009.06.019. PMC 2788085. PMID 19619628.
- Sharma, J.; Dragoi, V.; Tenenbaum, JB.; Miller, EK.; Sur, M. (Jun 2003). "V1 neurons signal acquisition of an internal representation of stimulus location.". Science 300 (5626): 1758–63. doi:10.1126/science.1081721. PMID 12805552.
- Lacey, S.; Tal, N.; Amedi, A.; Sathian, K. (May 2009). "A putative model of multisensory object representation.". Brain Topogr 21 (3-4): 269–74. doi:10.1007/s10548-009-0087-4. PMC 3156680. PMID 19330441.
- Neal, JW.; Pearson, RC.; Powell, TP. (Jul 1990). "The ipsilateral cortico-cortical connections of area 7b, PF, in the parietal and temporal lobes of the monkey.". Brain Res 524 (1): 119–32. doi:10.1016/0006-8993(90)90500-B. PMID 1698108.
- Eickhoff, SB.; Schleicher, A.; Zilles, K.; Amunts, K. (Feb 2006). "The human parietal operculum. I. Cytoarchitectonic mapping of subdivisions.". Cereb Cortex 16 (2): 254–67. doi:10.1093/cercor/bhi105. PMID 15888607.
- Petitto LA, Zatorre RJ, Gauna K, Nikelski EJ, Dostie D, Evans AC (December 2000). "Speech-like cerebral activity in profoundly deaf people processing signed languages: implications for the neural basis of human language". Proc. Natl. Acad. Sci. U.S.A. 97 (25): 13961–6. doi:10.1073/pnas.97.25.13961. PMID 11106400.
- Meredith, MA.; Clemo, HR. (Nov 1989). "Auditory cortical projection from the anterior ectosylvian sulcus (Field AES) to the superior colliculus in the cat: an anatomical and electrophysiological study.". J Comp Neurol 289 (4): 687–707. doi:10.1002/cne.902890412. PMID 2592605.
- Jiang, W.; Wallace, MT.; Jiang, H.; Vaughan, JW.; Stein, BE. (Feb 2001). "Two cortical areas mediate multisensory integration in superior colliculus neurons.". J Neurophysiol 85 (2): 506–22. PMID 11160489.
- Wallace, MT.; Carriere, BN.; Perrault, TJ.; Vaughan, JW.; Stein, BE. (Nov 2006). "The development of cortical multisensory integration.". J Neuroscience 26 (46): 11844–9. doi:10.1523/JNEUROSCI.3295-06.2006. PMID 17108157.
- Jiang W, Stein BE (October 2003). "Cortex controls multisensory depression in superior colliculus". J. Neurophysiol. 90 (4): 2123–35. doi:10.1152/jn.00369.2003. PMID 14534263.
- Wallace MT, Meredith MA, Stein BE (June 1993). "Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus". J. Neurophysiol. 69 (6): 1797–809. PMID 8350124.
- Stein, Barry E.; London, Nancy; Wilkinson, Lee K.; Price, Donald D. (1996). "Enhancement of Perceived Visual Intensity by Auditory Stimuli: A Psychophysical Analysis". Journal of Cognitive Neuroscience 8 (6): 497–506. doi:10.1162/jocn.19126.96.36.1997.
- Goodale MA, Milner AD (January 1992). "Separate visual pathways for perception and action". Trends Neurosci. 15 (1): 20–5. doi:10.1016/0166-2236(92)90344-8. PMID 1374953.
- Renier LA, Anurova I, De Volder AG, Carlson S, VanMeter J, Rauschecker JP (September 2009). "Multisensory integration of sounds and vibrotactile stimuli in processing streams for "what" and "where"". J. Neurosci. 29 (35): 10950–60. doi:10.1523/JNEUROSCI.0910-09.2009. PMC 3343457. PMID 19726653.
- Lewkowicz, D; Kraebel, K (2004). "The value of multisensory redundancy in the development of intersensory perception". In Gemma Calvert; Charles Spence; Barry E Stein. The Handbook of multisensory processes (Cambridge, Mass: MIT Press, cop.). pp. 655–78. ISBN 9780262033213. OCLC 803222288.
- Kujala, T.; Alho, K.; Huotilainen, M.; Ilmoniemi, RJ.; Lehtokoski, A.; Leinonen, A.; Rinne, T.; et, al. (Mar 1997). "Electrophysiological evidence for cross-modal plasticity in humans with early- and late-onset blindness.". Psychophysiology 34 (2): 213–6. doi:10.1111/j.1469-8986.1997.tb02134.x. PMID 9090272.
- Pascual-Leone,, A.; Theoret,, H.; et, al. (2006). "The Role of Visual Cortex in Tactile Processing: A Metamodal Brain.". In Morton A Heller; Soledad Ballesteros. Touch and blindness : psychology and neuroscience (Mahwah, N.J.: Lawrence Erlbaum Associates,). ISBN 9780805847260. OCLC 6124743.
- Wallace, Mark; Murray, Micah Middelmann (2012). The Neural Bases of Multisensory Processes (Frontiers in Neuroscience). Boca Raton: CRC Press. ISBN 1-4398-1217-9. OCLC 707710852.