바로가기메뉴

본문 바로가기 주메뉴 바로가기

logo

ISSN : 1226-9654

Article Contents

Prev Next
e-Submission

Vol.20 No.4

Citation Share

The influence of visual mouth cue on auditory speech perception: Structural Equation modeling

The Korean Journal of Cognitive and Biological Psychology / The Korean Journal of Cognitive and Biological Psychology, (P)1226-9654; (E)2733-466X
2008, v.20 no.4, pp.223-246
https://doi.org/10.22172/cogbio.2008.20.4.002



& (2008). The influence of visual mouth cue on auditory speech perception: Structural Equation modeling. The Korean Journal of Cognitive and Biological Psychology, 20(4), 223-246, https://doi.org/10.22172/cogbio.2008.20.4.002

Abstract

Visual cues emitted from a speaker, such as mouth and lip movements, are critical aspects of sensory information that influence auditory speech perception in face to face conversation. The way in which these complex visual cues are processed remains to be elucidated. Here we apply fMRI to a cross-modal speech perception paradigm in order to determine how functional connectivity between brain areas changes depending on the type of visual cue. Data obtained from a previous fMRI study in which three conditions, closed mouth (CM), twitching mouth (TM), or speaking mouth (SM) were delivered during an auditory sentence perception task. Structural Equation Modeling (SEM) was used to estimate functional strengths of links in a network among the brain regions activated in these three conditions. The chosen brain areas, which were modeled only in the right hemisphere, consisted of visual areas (lingual gyrus and V5/MT), both anterior and posterior superior temporal sulci (aSTS and pSTS), and the area equivalent to left Broca’s area (rBroca’s). Significant differences were found for the functional linkages between V5/MT and rBroca’s area across the three conditions. The path from V5/MT to rBroca became positive both during the TM and SM conditions, whereas that in the CM condition was negative. However, the connection from rBroca to V5/MT became negative both during the TM and SM, whereas that in the CM condition was positive. Additional linguistic information, such as the mouth movements of the SM condition, resulted in increased statistical significance of these linkages relative to the TM condition. The dynamic changes in functional connectivity between V5/MT and Broca’s area in the right hemisphere may subserve the cross-modal processing of auditory speech by inhibiting irrelevant or interfering visual cues from mouth movement.

keywords
교차양상 처리, 구조방정식 모형 (SEM), 입술 읽기, 언어, fMRI, Cross-modal processing, Structural Equation Model (SEM), lip-reading, speech, fMRI

Reference

1.

강혜진 ; 이동수 ; 강은주 ; 이정모 ; 나덕렬 ; 나동규, (2005) 연합기억과제의 특성과 해마활성화에 대한 fMRI 연구, 한국심리학회지:실험

2.

Allison, T., (2000) Social perception from visual cues:role of the STS region, TrendsCognSci

3.

Beauchamp,M.S., (2005) See me, hear me, touch me: multisensory intergration in lateral occipital-temporal cortex, Curr Opin Neurobiol

4.

Bonda, E., (1996) Bonda, E, JNeurosci

5.

Buccino, G., (2001) Action observation activates premotor and parietal areas in a somatotopic manner: an fMRI study, EurJNeurosci

6.

Cabeza, R., (1997) Age-related differences in effective neural connectivity during encoding and recall, NeuroReport

7.

Callan, D.E., (2001) Multimodal contribution to speech perception revealed by independent component analysis: a single?sweep EEG case study, BrainResCognBrainRes

8.

Calvert, G.A, (2000) Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex, CurrBiol

9.

Calvert, G.A., (1997) Activation of auditory cortex during silent lipreading, Science

10.

Campbell, R., (1997) Speechreading in the akinetopsic patient, L.M. Brain

11.

Campbell, R., (2001) Cortical substrates for the perception of face actions: an fMRI study of the specificity of activation for seen speech and for meaningless lower face acts (gurning), BrainResCognBrainRes

12.

Freitag, C.M., (2008) Perception of biological motion in autism spectrum disorders, Neuropsychologia

13.

Friston, K.J., (1993) Functional connectivity: the principal?component analysis of large (PET) data sets, J Cereb Blood Flow Metab

14.

Frackowiak, R.S., (2004) Human brain function, Second edition,

15.

Gallese, V., (1996) Action recognition in the premotor cortex, Brain

16.

Grossman, E., (2000) Brain areas involved in perception of biological motion, J Cogn Neurosci

17.

Horwitz, B., (1995) Network analysis of brain cognitive function using metabolic and blood flow data, Behav Brain Res

18.

Howard, R.J, (1996) A direct demonstration of functional specialization within motion?related visual and auditory cortex of the human brain, Curr Biol

19.

Kang, E., (2006) The neural correlates of cross-modal interaction in speech perception during a semantic decision task on sentences:A PET study, Neuroimage

20.

Kang, E., (2002) Visual modulation on speech processing in higher auditory cortices, Society for Neuroscience

21.

Kim, H., (2002) Effects of biological motion on speech processing: an fMRI study, Society for Neuroscience

22.

Kilts, C.D., (2003) Dissociable neural pathways are involved in the recognition of emotion in static and dynamic facial expressions, Neuroimagei

23.

Krause, B.J., (1999) Network analysis in episodic encoding and retrieval of word-pair associates: a PET study, Eur J Neurosci

24.

Krause, J.B., (2000) Imaging and neural modelling in episodic and working memory processes, Neural Netw

25.

Knutson, K.M, (2008) Observing social gestures: an fMRI study, Exp Brain Res

26.

McGurk, H., (1976) Hearing lips and seeing voices, Nature

27.

McIntosh, A.R., (1994a) Structural equation modeling and its application to network analysis in functional brain imaging, Human brain mapping

28.

McIntosh, A.R., (1994b) Network analysis of cortical visual pathways mapped with PET, J Neurosci

29.

Mogatty, F.M., (2006) Systems level modeling of a neuronal network subserving intrinsic alertness, Neuroimage

30.

Nyberg, L., (1996) Network analysis of Positron Emisson Tomography regional cerebral blood flow data: ensemble inhibition during episodic memory retrieval, J Neurosci

31.

Pelphrey, K.A., (2005) Functional anatomy of biological motion perception in posterior temporal cortex: an FMRI study of eye, mouth and hand movements, Cereb Cortex

32.

Peuskens, H., (2005) Specificity of regions processing biological motion, Eur J Neurosci

33.

Protzner, A.B., (2006) Testing effective connectivity changes with structural equation modeling: what does a bad model tell us?, Human brain mapping

34.

Puce, A., (1998) Temporal cortex activation in humans viewing eye and mouth movements, J Neurosci

35.

Rees G, (1999) Inattentional blindness versus inattentional amnesia for fixated but ignored words, Science

36.

Rizzolatti, G., (1996) Premotor cortex and the recognition of motor actions, Brain Res Cogn Brain Res

37.

Sekiyama, K., (2003) Auditory visual speech perception examined by fMRI and PET, Neuro sci Res

38.

Sekuler, R., (1997) Sound alters visual motion perception, Nature

39.

Seminowicz, D.A., (2004) Limbic-frontal circuitry in major depression:a path modeling metanalysis, NeuroImage

40.

Shams,L., (2000) Illusion.What you see is what you hear, Nature

41.

Thompson, J.C, (2007) Common and distinct brain activation to viewing dynamic sequences of face and hand movements, Neuroimage

42.

Wheaton, K.J., (2004) Viewing the motion of human body parts activates different regions of premotor, temporal, and parietal cortex, Neuroimage

43.

Winder, R., (2007) Functional connectivity in fMRI:Amodeling approach for estimation and for relating to local circuits, Neuroimage

The Korean Journal of Cognitive and Biological Psychology

e-Submission OA Policy