Portal de Periódicos da CAPES

Making sense of multisensory integration

2010; Wiley; Volume: 31; Issue: 10 Linguagem: Inglês

10.1111/j.1460-9568.2010.07238.x

1460-9568

Sophie Molholm, John J. Foxe,

Olfactory and Sensory Function Studies

A pseudo-aquatic parasite, voiceless as a fish, yet constructing within itself an instrument of voice against the time when it will talk. Organs of skin, ear, eye, nose, tongue, superfluous all of them in the watery dark where formed, yet each unhaltingly preparing to enter a daylight, airy, object-full manifold world which they will be wanted to report on. A great excresence at one end of a nerve-tube, an outrageously outsized brain, of no avail at the moment but where the learning of a world which is to be experienced will go forward. –Sherrington (1940) Apart from the obvious appeal of his prose, which must surely give rise to an element of regret at the rather stolid constraints of modern science writing, one hears in this quotation of Sherrington's his profound fascination for the very nature of sensory development in the human fetus. Elsewhere in his wonderful book 'Man on his Nature' he turns specifically to the issue of multisensory integration: "The naïve would have expected evolution in its course to have supplied us with more various sense organs for ampler perception of the world…The policy has rather been to bring by the nervous system the so called 'five' into closer touch with one another…A central clearing house of sense has grown up…Not new senses, but better liaison between old senses is what the developing nervous system has in this respect stood for". Clearly scientists have pondered the seamlessness of liaison between the senses since the very early days of modern neuroscience. And it is surely one of the great remaining scientific puzzles just how it is that signals from such completely separated and wholly dissimilar sensory epithelia as the haircells of the cochlea, the photoreceptors of the retina and the corpuscles of the skin can be integrated centrally to form such a seamless unitary perceptual world. Our own predecessors here at The Albert Einstein College of Medicine were already studying the development of intersensory abilities in pediatric populations in the 1960s (e.g. Birch & Lefford, 1963, 1967). Yet despite these early research efforts, it is also fair to say that multisensory research was not at the forefront of the scientific effort across the early decades of the 20th century. Given the complexities of working with isolated sensory systems, many surely turned away from these issues because of the seeming intractability of trying to control more than just one system. But multisensory research has evolved considerably over the past 25 years. For this we owe a large debt to a few stalwarts of the field who forged ahead with this work when there was little to go on and when there was as yet limited interest in elucidating multisensory influences on information processing (e.g. Stein & Arigbede, 1972; Seltzer & Pandya, 1980; King & Palmer, 1985; see Stein & Meredith, 1993). From their extensive work on the multisensory properties of cells in the superior colliculus, Stein and Meredith defined a set of governing principles of multisensory integration at the cellular level. By providing a set of hypotheses to test in a domain for which we are still considerably lacking in solid models, these principles have given important fodder to those examining multisensory integration under different stimulus and task configurations. Their substantial role in generating research questions is clearly signified not only by the extraordinary number of citations to their work but also by the many studies aimed at challenging these principles (see Foxe, 2008). Even before there was hard evidence of anatomical plausibility, we and others made the case that multisensory integration was not deferred to higher-order processing areas, but also occurred at very early stages of cortical information processing in a feedforward manner and in regions considered to be unisensory (Molholm et al., 2002; Schroeder & Foxe, 2002). This commenced a marked change in how multisensory influences on neural processing are conceptualized. Falchier et al. (2002) documented monosynaptic connectivity between hierarchically early unisensory cortical areas, supplying anatomical plausibility for this thesis (see also Rockland & Ojima, 2003), and a significant volume of psychophysical work, much from the laboratories of John Driver and of Charles Spence, but many others as well, has made absolutely clear the ubiquity of multisensory influences on perception (e.g. Driver & Spence, 2000; Spence & Squire, 2003). In this special issue we see that the question of when and where cortical multisensory integration first occurs remains a topic of considerable interest. Using magnetoencephalographic recordings, Raij et al. (2010) examined the temporal dynamics of cross-sensory activation and auditory–visual multisensory integration in human primary auditory and visual cortices. With this approach they find activation of visual cortex in response to auditory stimuli and activation of auditory cortex in response to visual stimuli. They show that these 'cross-sensory' activations are delayed by 30–40 ms relative to the onset of responses to the native inputs of these same sensory regions (i.e. auditory cortical responses to auditory input). Further, they show that multisensory integration occurs in sensory cortical areas only a few milliseconds following the onset of cross-sensory activity. Raij et al. (2010) make the case that this cross-sensory activity reflects transmission between the sensory cortical regions, as opposed to inputs from subcortical regions or higher-order association areas. However, this argument from timing, while compelling, cannot definitively resolve this issue, and the transmission pathways leading to early cross-sensory activation of sensory cortex remain to be firmly established. This may be addressed with the use of additional analytic techniques than have been applied thus far, or more probably require the use of methodologies that can reveal causal links combined with the timecourse of involvement, such as transcranial magnetic stimulation (TMS). In this issue we find two papers in which direct manipulation of neural processing in a cortical region is deployed to understand the functional architecture of a particular process. With the use of anodal transcranial direct current stimulation, Bolognini et al. (2010) show that inducing an increase in the excitability of neurons in right posterior parietal cortex facilitates spatial orienting toward both auditory and visual stimuli presented to the contralateral field. This is consistent with a key role for posterior parietal cortex in the deployment of so-called supramodal spatial attention. Using single-pulse TMS to produce quite the opposite effect, Pasalar et al. (2010) show that the posterior parietal cortex near the junction of the anterior intraparietal sulcus and the postcentral sulcus is involved in the integration of visual and tactile information. Disruption of neuronal activity in this region leads to a decrease in multisensory facilitation whilst leaving unisensory performance intact. In addition to neurophysiological and neuroanatomical studies spearheading changes in how we think about the multisensory sciences and the mediating neural mechanisms, behavioral studies have been key to gaining insight into the consequences of the multisensory brain. Welch & Warren (1980, 1986) proposed the modality-appropriateness hypothesis to explain why information conveyed in certain sensory modalities seems to exert a disproportionate influence on certain types of judgments. They argued that particular sensory systems best conveyed specific types of information and accordingly dominated related judgments under multisensory conditions, and even exerted an influence when ancillary to the task. For example, auditory stimuli are reliably mislocalized toward a concurrently presented but task-irrelevant visual stimulus, reflecting the dominance of visual stimulation in spatial judgment (see e.g. Warren et al., 1981). More recent work has generated an appealing explanation of sensory dominance that differs from this original thesis only in that the dominant sensory modality can be altered by manipulating the integrity, or quality, of the respective multisensory signals (Ernst & Banks, 2002). Here it is argued that it is the reliability of the estimates that the signal provides that affect its relative contribution to performance. Of course this fits well with the notion that there is sensory-specific dominance for some types of judgments as, for example, it is seldom that an auditory signal will provide better spatial information than a visual signal. This can be well-accounted for using Bayesian modeling or variants thereof. In this issue Fetsch, DeAngelis and Angelaki describe how Bayesian decision theory accounts for the contributions of visual and vestibular information to estimations of one's translation in space (Fetsch et al., 2010). Perception of heading is intrinsically multisensory insofar as presenting both visual and vestibular inputs tend to result in superior performance compared to presenting just one. In their review, Fetsch and colleagues describe corresponding cellular activity that acts in accordance with behavioral measures of visual–vestibular cue integration. Along a similar vein, in this issue van Wanrooij et al. (2010) demonstrate that implicit expectations impact the degree to which ancillary multisensory information is used to facilitate performance. In their study subjects made heading movements to the perceived location of a visual stimulus along the vertical plane in meridian space. In blocks of trials in which ancillary auditory stimuli were always spatially aligned with the visual stimuli, reaction times were speeded compared to visual-alone presentations. In contrast, in blocks of trials in which the auditory stimuli were misaligned on 50% or 90% of trials, reaction time facilitation was not seen; follow-up analysis of sequential effects showed that reaction times for spatially congruent trials were faster when preceded by a spatially congruent trial compared to when preceded by a spatially incongruent trial, and that there was no effect of the preceding trial on reaction time for the spatially incongruent trials. These data show a dynamic system that is continually updated as a function of the statistical probabilities of the local environment. This signifies an adaptive perceptual–cognitive system that continues to be highly flexible even into adulthood. Such a flexible system precludes the possibility that multisensory processes are fully automatic, nor would one expect this to uniformly be the case. The question remains, however, as to just how malleable multisensory processes are, and which processes are more or less susceptible to higher-order influences such as attention and learning. Most of the foundational physiological work on multisensory integration has been done in anesthetized animals and it is clear that significant multisensory integration proceeds in the absence of attention (Stein & Meredith, 1993). Yet data from human scalp electrical recordings (Talsma et al., 2007) suggest that even the earliest evidence of cortical audiovisual effects, which one might expect to proceed in a fully bottom-up manner, can be affected by attention. Attention effects have been demonstrated for a number of multisensory situations, including integration of seen and heard speech (e.g., Soto-Faraco et al., 2004; Alsius et al., 2005). In this issue, Heron et al. (2010) show that attention to the temporal relationship between lagged auditory and visual inputs affects the degree of temporal adaptation, and thus appear to resolve some discrepancies in the literature on multisensory temporal adaptation effects (Fujisaki et al., 2004; Vroomen et al., 2004; Keetels & Vroomen, 2007; Hanson et al., 2008a; Harrar & Harris, 2008). Subjects were exposed to audiovisual stimulus pairs with onsets that were consistently misaligned by 120 ms during an adaptation phase. When subjects engaged in a task in which they were to detect infrequent targets in which the temporal order of the stimuli differed, there was a substantial increase in the amount of temporal adaptation observed during test as compared to when subjects engaged in tasks in which they detected either infrequent changes in the luminance of a fixation cross or infrequent changes in the visual stimulus. Thus inconsistencies in the literature with regard to the magnitude of such recalibration effects may well be related to the task that is employed and whether or not it is related to the adapting feature. In the 'unisensory' visual attention literature there is strong evidence for an inherent bias to process all of the features of an object, even when attention is explicitly focused on just a single feature (e.g., Stroop, 1935; Egly et al., 1994; Martinez et al., 2006). Until recently this had only been demonstrated for visual features, and an obvious next question was whether this principle would also apply to the multisensory features of an object. A series of recent studies has indeed shown that, when attention is directed toward just a single feature of a multisensory object, there is a strong bias to process the unattended features of that object in other sensory modalities, even when those features are completely irrelevant to the task at hand (Busse et al., 2005; Molholm et al., 2007; Fiebelkorn et al., 2010a). In this issue, work from our own laboratory using event-related potentials (ERPs) suggests that this automatic spread of attention across sensory systems occurs only after the spatial boundaries of the visual object have first been resolved [Fiebelkorn et al., 2010a, 2010b; this volume]. It remains to be seen whether this will turn out to be a fundamental principle of object recognition under differing multisensory conditions. Kennett et al. (2001) made the observation that participants demonstrated consistently greater tactile acuity when they could view the stimulated body site, even though there was no explicit visual information available at that site. That is, it was the simple act of bringing the 'felt' receptive fields into the center of gaze that caused an improvement in tactile abilities. With the use of ERPs, they went on to show a corresponding and dramatic amplification of the somatosensory evoked potential (Taylor-Clarke et al., 2002) and suggested that this reflected feedback from tactile–visual multisensory neurons in posterior parietal cortex to primary somatosensory cortex. The implications of course are great, indicating that the ability to recruit visuotactile neurons actually influences how we experience touch. In this special issue we see a bevy of related papers, emphasizing that this is an area of keen interest for multisensory scientists. Work from the laboratory of Olaf Blanke in Lausanne (Aspell et al., 2010; this volume), for example, adds a particularly interesting twist to this visualization story. They asked their participants to judge the elevation of four vibratory stimulators that were attached to their backs while they played unrelated auditory stimuli from speakers placed both in front of and behind the observers. It is often the case that automatic multisensory integrations will cause some spatial shifting under such circumstances, a form of audiotactile ventriloquism if you will. However, the auditory stimuli had no effect on the tactile elevation judgments under these circumstances. Nevertheless, when the participants were provided with a real-time but completely uninformative view of their own back using a head-mounted display fed by a camera placed 2 m behind them, incongruent auditory spatial information began to interfere with the tactile elevation judgments: a trisensory effect. And so, the interaction of the senses is clearly profound but also complex. This area will not lend itself to very simple models or principles and there is much fertile ground to be tilled before we are likely to arrive at any true understanding. In closing, we would like to thank the many multisensory scientists who submitted their work to this special issue. As in past years, the competition for limited places was fierce and the quality ran deep. With the production of this issue, an annual tradition since the founding of the International Multisensory Research Forum (IMRF) conference, we hand the reins over to the very capable hands of Dr Georg Meyer who will chair the organizing committee of the 11th annual meeting. He will host this year's meeting from June 16th to June 19th 2010 at The University of Liverpool (http://imrf.mcmaster.ca/IMRF/ocs2/index.php/imrf/2010/). Finally, we wish to extend our heartfelt appreciation once again to the local organizing committee of IMRF 2009 in New York, who made the meeting such a tremendous success. We look forward to what is sure to be a most engaging and stimulating IMRF 2010, and to meeting in Liverpool with colleagues and colleagues-to-be who share our enthusiasm for the multisensory perspective. The work of S.M. and J.J.F. on multisensory integration is supported by a grant from the US National Institute of Mental Health (MH85322). The authors would like to thank Professor Herb Cohen of the CERC at Einstein for helping us rediscover the important work of Birch & Lefford (1963, 1967).