Experience appears to comprise distinct experiences of seeing, hearing, touching, tasting and smelling, yet neuroscience tells us that the brain integrates inputs from many more sources to create unified, multisensory experiences. Therefore, this project aims to answer the following overarching research question: how can our conscious perception of the sensory world be reconciled with recent neuroscientific accounts of multisensory processing in the brain?
This question locates a common interest for the humanities and the neurosciences: to understand the experiences of the human mind. In doing so, we seek a better understanding of the variety of conscious experiences, including the perceptual experiences of infants, people with sensory deficits and the nature of social perception.
Rethinking the Senses (RTS) opens up a new research agenda by studying perception from a multisensory perspective, where a perceiver’s experience depends on inputs form the external senses acting in concert with information about the perceiver’s bodily states and movements. For example, I see and feel a mug, know its weight, whether it is hot or cold, know its colour in a variety of luminance conditions. Here sensory information is not merely juxtaposed and assembled by the exercise of judgment, but presented simultaneously and cross-modally. For a variety of multisensory effects what is perceived, or processed in one sensory stream, is influenced by what is perceived or processed in another. The following provide concrete examples of this taking place: in speech perception seeing the lips of the speaker affects what we hear (‘Do We Hear with our Eyes?’, McGurk &McDonald, 1976); in the cinema we locate the voices of actors as coming from the screen although the sound does not comes from there (‘Ventriloquism Effect’, Alais & Burr, 2004); what we hear can affect what we see (‘Sound-induced Flash illusion’, Shams et. Al, 2000); smells and colours affect how sweet a food tastes (‘Sweetness Enhancement’, Stevenson, 2009); and sounds can change how rough or smooth one’s hands feel (‘Parchment Effect’, Jousmäki & Hari, 1998). In synaesthesia, presentation of an object in one sensory modality elicits an additional experience in another non-stimulated stream. This explains, for instance, why certain individuals ‘hear sounds in colour’ and perhaps why we all tend to associate high-pitched sounds with bright lights. Multisensory interactions are the rule, not the exception.
Given the diversity of inter-sensory interactions, we address a number of questions in turn:
Is there a useful taxonomy of multisensory interactions? A taxonomy should ask what kinds of cross-modal effects there are and whether they should be explained in the same way.
Why, and how, should we distinguish the senses? Is the distinction based merely on kinds of experience or are there deeper, theoretical reasons for the distinctions? Does it help to think of certain experiences as modally distinct (e.g. auditory, visual, etc.) and others as amodal (e.g. the experience of time, space)? And what is the difference between a multimodal and an amodal representation of space? Many scientists use the terms interchangeably. There are broader implications for the field, raising concerns about the use of the concept of a sense to characterize our various inner representations – the sense of body, or self – or what we feel when we move intentionally – the sense of agency and ownership over our limbs. What changes do technology and surgery introduce? For example, will equipping people with a vibrating, magnetic belt create a new ‘magnetic’ or spatial sense; do people with retinal implants who perceive light in infra-red ‘see’ water boiling on the stove; and can blind individuals ‘hear the shape’ of distant objects thanks to head-mounted cameras connected to headphones? Are these new senses or in fact extensions of the traditional senses of vision and hearing?
The multisensory perspective on perception puts the notion of integration at the centre of the research agenda. Integration involves attention, action, and an interaction of information derived from multiple sources. We use our senses actively, exploring how things respond when we poke, palpate, probe, and manipulate them. This exploits our ability to use the senses together in ways that have been studied rarely in philosophy or psychology. Sensory exploration demands the integration of information over time, contrary to the dominant ‘snapshot’ conception of the senses. This raises the question:
What is the role of attention, sensory exploration and short-term memory in multisensory integration? Progress here depends on theoretical clarifications of what is meant by attention, whether top-down or bottom-up, a single capacity, distributed between the various senses, or, as often described in the sciences, sense-specific with competing forms of attention, like visual, auditory ,and tactile attention.
Information gathering makes sense only if it then leads to a unified perception of the world: the cup we hold has tactile and visual properties, while our hand has a shape that we know both visually from the outside and from the inside by proprioception. Multisensory objects have many properties.
What binds the various sensory elements to give rise to unified perceptions and how do the conscious signatures of unimodal sensations vary when altered by cross-modal interaction, imagery, synaesthetic concurrent sensations and working memories? If the experience of tasting is unitary and distinctive, how can it also be a matter of cross-modal or multi-modal interaction? Is the notion of ‘feature’ used in the scientific literature on vision, to talk for instance about binding edges and colours, processed in different parts of the brain the correct explanation for multisensory integration? By contrast with vision, various senses can provide redundant or complementary information about the same object. Investigating various kinds of breakdowns in experience is the key here to a better understanding of these questions. In the rubber hand illusion, one’s proprioceptive sense of one’s hand is fooled by seeing a rubber hand being stroke synchronously with one’s real hand, opening room to study dissociation between inner and outer senses of one’s body. Should this be analyzed as one having two, hard-to-reconcile experiences of one’s hand or as an illusory shift in one’s proprioception? Philosophical distinctions can guide experimental study here.
How do different senses contribute to the perception and recognition of objects? Have current scientific results solved or intensified the problem that Molyneux raised for Locke about whether experiences from different modalities could lead to the recognition of a single object? Is information extracted about structure invariant across different senses and are stored representations of objects common across the senses, or are there different representations associated with each sense?
In addition to the senses influencing each other, there is increasing evidence that our perception of the world is influenced by prior experience and learning. It is well known that our judgments of what we perceive are influenced by our personal histories, expectations, and culture. Since Westerners combine vanilla with sugar in their cuisine, vanilla will smell sweet, but not to the Japanese. Similarly, the red colour of a drink can make us expect red fruits flavor, but fruits can be different colours in different countries. So are these sensory associations or higher level effects?
Do cognitive states have, a ‘top-down’ effect on the contents of perceptual experiences? There are cross-modal adaptations and perceptual learning effects that are not cognitive, such as the adaptation which occurs between vision and touch when one wears prismatic goggles. A fundamental issue to be addressed here concerns the veridical status of perception: does ‘prior knowledge’ make perception more optimal and accurate or introduce a bias that prevents us from perceiving things as they presently are?
By addressing the above questions, we can understand better the nature of conscious perceptual experiences and the importance of multisensory integration for theories of the human mind. This understanding feeds into larger research questions, as follow.
How, if at all, are the interactions between the senses reflected in our conscious experience? Do they result, for example, in a unified conscious experience of the world and, if so, what form does unification take? Why do we continue to characterize our conscious experiences in terms of the five senses?
If the evidence described above requires us to rethink the nature of the senses, then what consequences does this have for knowledge, memory, imagination, hedonic and aesthetic evaluation? We take ourselves to be capable of imagining in a distinctively sensory way: to imagine seeing, touching, hearing it, etc. How does multisensory perception influence episodic memory and how should we understand the ‘Proust effect’ where smells recall visual images, a phenomenon particularly acute and traumatic for individuals with PTSD (post-traumatic stress disorder)? What about our experiences of artworks and music, which are always multisensory? This broader understanding of the relation between multisensory perception and other states is fundamental if we want to address the question of whether multisensory integration, if it is influenced by attention, action and previous experiences, is most of the time illusory. Do we need to give up on the claim that perception informs us about proper objects or events, such as audition supposedly being used to inform us about sounds, or soundwaves? Or can we think of this influence producing more reliable and veridical perception than would happen in a single sense modality? Empirically-informed philosophy is needed to tackle these questions.
If perceptual experiences result from multisensory integration, should we not rethink our notion of a single sensory loss? Should we change the way we address certain conditions, such as loss of smell and taste associated with depression, body-image distortions in anorexia and eating disorders, and consider challenges for the ageing population? Our distinctions seem to have been exclusively reliant on the causal origin of the disorder in one modality, or the most obvious symptom, when in fact these would benefit from being thought of as affecting or originating in other senses.
RTS addresses these research questions through a series of seminars, workshops, and conferences. Experiments within the lab contribute towards answering these questions. Collaborations with our partners, both academic and non-academic, further our understanding of multisensory perception.