- 권한신청
- P-ISSN1226-9654
- E-ISSN2733-466X
- KCI
학습의 진행과 두뇌 피드백 정보처리의 변화
Dynamic changes in feedback processing as learning progresses
김진희 (강원대학교)
강은주 (강원대학교)
초록
자극에 대한 반응에 주어지는 긍정적 또는 부정적 피드백(예, 보상, 처벌)은 동기적/쾌락적 속성뿐만 아니라 피드백의 현저성의 측면에서도 달라질 수 있다. 행동 변화의 인지 통제를 위해서는 이 두 가지 특성에 대한 정보처리가 모두 요구될 가능성이 있다. 본 연구는 피드백 현저성을 처리하는 두뇌 영역을 확인하고자 수행되었으며, 이를 위해 학습의 초기와 후기 정적 피드백과 부적 피드백의 활성화를 비교하였다. 조건적 연합학습(conditional associative learning) 과제를 이용하여 자극-반응 연합 규칙이 피드백에 근거한 시행착오를 통해 학습되도록 하였다. 자극에 대한 반응을 4개로 하여 학습자가 부적 피드백을 통해 직전 반응이 오류 반응임을 확인하여도 정답 반응을 바로 유추하기 어렵게 만들었다. 이는 학습 초에는 정적 피드백이 높은 비율로 경험되는 부적 피드백에 비하여 경험 빈도가 낮으면서 동시에 행동 조절에 더 유용하여 현저성이 높은 피드백으로 간주될 수 있는 반면, 학습이 진행된 후에는 부적 피드백이 낮은 빈도로 주어지면서 행동 변화와의 관련성이 높아 현저성이 높은 피드백이 될 수 있게 하기 위해서였다. 각각의 피드백에 대한 반응이 연속되는 4개의 run동안에 변화하는 두뇌 영역을 확인하기 위하여, 정상 성인(n =29)으로부터 학습과제 중에 fMRI자료를 획득, 분석하였다. 그 결과, 전측 도, 전대상회를 포함한 배내측 전전두 피질, 하전전두 영역, 하 두정피질, 소뇌 등에서 관찰된 정적 피드백에 대한 활성화가 학습이 진행하면서 감소하는 반면, 부적 피드백에 대한 활성화는 증가함을 발견하였다. 이런 활성화의 변화는 학습 과제의 초기에서 말기로 가면서 정적 피드백의 현저성 처리는 감소하고 부적 피드백의 현저성 처리는 증가한 것을 반영함을 시사한다고 볼 수 있다. 이와 대조적으로 쾌락가 정보처리와 관련된 것으로 알려진 복측 선조체와 복내측 전전두 피질의 활성화 양상(정적 > 부적 피드백)은 run에 따라 변화하지 않았다. 이런 결과는 학습에서 피드백의 쾌락가를 처리하는 신경망과 독립적으로, 피드백의 현저성 정보가 별개의 신경망에서 처리됨을 시사한다.
- keywords
- Feedback, Learning, Saliency, Relevancy, Reward, Insula, dmPFC, Anterior cingulate, 피드백, 학습, 현저가, 과제관련성, 보상, 도, 배내측 전두영역, 전측 대상회
Abstract
For cognitive control of behavioral adjustments in feedback learning, various processings are required, including evaluating the saliency (i.e., relevance to the task) and hedonic value of feedback information for future response selection. In this study, brain regions involved in processing feedback saliency were investigated by comparing activations for positive feedback (following correct responses) and negative feedback (following errors) for early and late phases of learning. A conditional associative learning task was used in which stimulus-response association rules were learned by trial and error, based on the feedback. Since there were four available responses to choose among for each stimulus, only positive feedback (i.e., reward) was relevant to behavioral adjustment during the early learning phase of learning, but negative feedback (e.g., penalty) became more relavant as learning progressed. fMRI data obtained from normal adults (n = 29) were analyzed to identify brain regions where responses to each feedback varied across the four consecutive runs. Activation for reward decreased as learning progressed, whereas activation for penalty increased in the following areas: anterior insula, dmPFC and anterior cingulate region, inferior PFC, inferior parietal cortex, and cerebellum. We interpret these results as reflecting the decreased saliency of positive feedback and increased saliency of negative feedback, between early and late phases of the learning task. In contrast, for two areas associated with processing of hedonic value, the ventral striatum and vmPFC, activations (positive > negative feedback) did not vary across the four consecutive runs. These observations suggest that the saliency of feedback for learning is processed in a network separate from that for the hedonic value of feedback.
- keywords
- Feedback, Learning, Saliency, Relevancy, Reward, Insula, dmPFC, Anterior cingulate, 피드백, 학습, 현저가, 과제관련성, 보상, 도, 배내측 전두영역, 전측 대상회
참고문헌
신연순, 한상훈 (2013). 목표지향적 학습과 기억. 감성과학, 16(3), 319-322.
Alexander, W. H., & Brown, J. W. (2011). Medial prefrontal cortex as an action-outcome predictor. Nature Neuroscience, 14(10), 1338- U1163.
Alexander, W. H., & Brown, J. W. (2014). A general role for medial prefrontal cortex in event prediction. Frontiers in Computational Neuroscience, 8, 69.
Amiez, C., Hadj-Bouziane, F., & Petrides, M. (2012). Response selection versus feedback analysis in conditional visuo-motor learning. Neuroimage, 59(4), 3723-3735.
Amiez, C., Sallet, J., Procyk, E., & Petrides, M. (2012). Modulation of feedback related activity in the rostral anterior cingulate cortex during trial and error exploration. Neuroimage, 63(3), 1078-1090.
Baird, L. C. (1993). Advantage updating: (No. WL-TR-93-1146). WRIGHT LAB WRIGHT PATTERSON AFB OH.
Behrens, T. E., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10(9), 1214-1221.
Bischoff-Grethe, A., Hazeltine, E., Bergren, L., Ivry, R. B., & Grafton, S. T. (2009). The influence of feedback valence in associative learning. Neuroimage, 44(1), 243-251.
Blair, R. J., Morris, J. S., Frith, C. D., Perrett, D. I., & Dolan, R. J. (1999). Dissociable neural responses to facial expressions of sadness and anger. Brain, 122 (Pt 5), 883-893.
Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S., & Cohen, J. D. (2001). Conflict monitoring and cognitive control. Psychological Review, 108(3), 624-652.
Botvinick, M. M., Cohen, J. D., & Carter, C. S. (2004). Conflict monitoring and anterior cingulate cortex: an update. Trends in Cognitive Sciences, 8(12), 539-546.
Braver, T. S., Barch, D. M., Gray, J. R., Molfese, D. L., & Snyder, A. (2001). Anterior cingulate cortex and response conflict: effects of frequency, inhibition and errors. Cerebral Cortex, 11(9), 825-836.
Bromberg-Martin, E. S., Matsumoto, M., & Hikosaka, O. (2010). Dopamine in motivational control: rewarding, aversive, and alerting. Neuron, 68(5), 815-834.
Chen, S. H., & Desmond, J. E. (2005). Temporal dynamics of cerebro-cerebellar network recruitment during a cognitive task. Neuropsychologia, 43(9), 1227-1237.
Chib, V. S., Rangel, A., Shimojo, S., & O'Doherty, J. P. (2009). Evidence for a common representation of decision values for dissimilar goods in human ventromedial prefrontal cortex. The Journal of neuroscience, 29(39), 12315-12320.
Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews: Neuroscience, 3(3), 201-215.
Critchley, H. D., Wiens, S., Rotshtein, P., Ohman, A., & Dolan, R. J. (2004). Neural systems supporting interoceptive awareness. Nature Neuroscience, 7(2), 189-195.
d'Acremont, M., Lu, Z. L., Li, X., Van der Linden, M., & Bechara, A. (2009). Neural correlates of risk prediction error during reinforcement learning in humans. Neuroimage, 47(4), 1929-1939.
Delgado, M. R. (2007). Reward-related responses in the human striatum. Annals of the New York Academy of Sciences, 1104(1), 70-88.
Desmond, J. E., & Fiez, J. A. (1998). Neuroimaging studies of the cerebellum: language, learning and memory. Trends in Cognitive Sciences, 2(9), 355-362.
Elliott, R., Dolan, R. J., & Frith, C. D. (2000). Dissociable Functions in the Medial and Lateral Orbitofrontal Cortex: Evidence from Human Neuroimaging Studies. Cerebral Cortex, 10(3), 308-317.
Elliott, R., Friston, K. J., & Dolan, R. J. (2000). Dissociable neural responses in human reward systems. Journal of Neuroscience, 20(16), 6159- 6165.
Elliott, R., Newman, J. L., Longe, O. A., & Deakin, W. J. F. (2003). Differential response patterns in the striatum and orbitofrontal cortex to financial reward in humans: a parametric functional magnetic resonance imaging study. Journal of Neuroscience, 23(1), 303-307. doi: 23/1/303 [pii]
Elliott, R., Rees, G., & Dolan, R. J. (1999). Ventromedial prefrontal cortex mediates guessing. Neuropsychologia, 37(4), 403-411.
Esber, G. R., & Haselgrove, M. (2011). Reconciling the influence of predictiveness and uncertainty on stimulus salience: a model of attention in associative learning. Proceedings: Biological Sciences, 278(1718), 2553-2561.
Goulden, N., Khusnulina, A., Davis, N. J., Bracewell, R. M., Bokde, A. L., McNulty, J. P., & Mullins, P. G. (2014). The salience network is responsible for switching between the default mode network and the central executive network: replication from DCM. Neuroimage, 99, 180-190.
Hare, T. A., Camerer, C. F., & Rangel, A. (2009). Self-control in decision-making involves modulation of the vmPFC valuation system. Science, 324(5927), 646-648.
Haruno, M., & Kawato, M. (2006). Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. Journal of Neurophysiology, 95(2), 948-959.
Hester, R., Foxe, J. J., Molholm, S., Shpaner, M., & Garavan, H. (2005). Neural mechanisms involved in error processing: A comparison of errors made with and without awareness. Neuroimage, 27(3), 602-608.
Holroyd, C. B., & Coles, M. G. (2002). The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109(4), 679-709.
Holroyd, C. B., Nieuwenhuis, S., Yeung, N., Nystrom, L., Mars, R. B., Coles, M. G., & Cohen, J. D. (2004). Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nature Neuroscience, 7(5), 497-498.
Jung, J., Jerbi, K., Ossandon, T., Ryvlin, P., Isnard, J., Bertrand, O., . . . Lachaux, J. P. (2010). Brain responses to success and failure: Direct recordings from human cerebral cortex. Human Brain Mapping, 31(8), 1217-1232.
Kahnt, T., Park, S. Q., Haynes, J. D., & Tobler, P. N. (2014). Disentangling neural representations of value and salience in the human brain. Proceedings of the National Academy of Sciences of the United States of America, 111(13), 5000-5005.
Kastner, S., & Ungerleider, L. G. (2000). Mechanisms of visual attention in the human cortex. Annual Review of Neuroscience, 23(1), 315-341.
Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001). Anticipation of increasing monetary reward selectively recruits nucleus accumbens. The Journal of neuroscience, 21(16), RC159.
Knutson, B., Westdorp, A., Kaiser, E., & Hommer, D. (2000). FMRI Visualization of Brain Activity during a Monetary Incentive Delay Task. Neuroimage, 12(1), 20-27.
Kringelbach, M. L. (2005). The human orbitofrontal cortex: linking reward to hedonic experience. Nature Reviews: Neuroscience, 6(9), 691-702.
Kringelbach, M. L., & Rolls, E. T. (2004). The functional neuroanatomy of the human orbitofrontal cortex: evidence from neuroimaging and neuropsychology. Progress in Neurobiology, 72(5), 341-372.
Lammel, S., Ion, D. I., Roeper, J., & Malenka, R. C. (2011). Projection-specific modulation of dopamine neuron synapses by aversive and rewarding stimuli. Neuron, 70(5), 855-862.
Lang, P. J., & Davis, M. (2006). Emotion, motivation, and the brain: reflex foundations in animal and human research. Progress in Brain Research, 156, 3-29.
Li, W., Piech, V., & Gilbert, C. D. (2004). Perceptual learning and top-down influences in primary visual cortex. Nature Neuroscience, 7(6), 651-657.
Lin, A., Adolphs, R., & Rangel, A. (2012). Social and monetary reward learning engage overlapping neural substrates. Social Cognitive and Affective Neuroscience, 7(3), 274-281.
Lin, S. C., & Nicolelis, M. A. (2008). Neuronal ensemble bursting in the basal forebrain encodes salience irrespective of valence. Neuron, 59(1), 138-149.
Liu, X., Powell, D. K., Wang, H., Gold, B. T., Corbly, C. R., & Joseph, J. E. (2007). Functional dissociation in frontal and striatal areas for processing of positive and negative reward information. Journal of Neuroscience, 27(17), 4587-4597.
Mackintosh, N. J. (1983). Conditioning and Associative Learning. Oxford: Clarendon Press.
Mars, R. B., Coles, M. G. H., Grol, M. J., Holroyd, C. B., Nieuwenhuis, S., Hulstijn, W., & Toni, I. (2005). Neural dynamics of error processing in medial frontal cortex. Neuroimage, 28(4), 1007-1013.
Matsumoto, M., & Hikosaka, O. (2009). Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature, 459(7248), 837-841.
Menon, V., & Uddin, L. Q. (2010). Saliency, switching, attention and control: a network model of insula function. Brain Struct Funct, 214(5-6), 655-667.
Metereau, E., & Dreher, J. C. (2013). Cerebral correlates of salient prediction error for different rewards and punishments. Cerebral Cortex, 23(2), 477-487.
Nieuwenhuis, S., Slagter, H. A., von Geusau, N. J., Heslenfeld, D. J., & Holroyd, C. B. (2005). Knowing good from bad: differential activation of human cortical areas by positive and negative outcomes. European Journal of Neuroscience, 21(11), 3161-3168.
O'Doherty, J. P., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304(5669), 452-454.
O'Doherty, J. P., Kringelbach, M. L., Rolls, E. T., Hornak, J., & Andrews, C. (2001). Abstract reward and punishment representations in the human orbitofrontal cortex. Nature Neuroscience, 4(1), 95-102.
Palminteri, S., Justo, D., Jauffret, C., Pavlicek, B., Dauta, A., Delmaire, C., . . . Pessiglione, M. (2012). Critical roles for anterior insula and dorsal striatum in punishment-based avoidance learning. Neuron, 76(5), 998-1009. doi: 10.1016/ j.neuron.2012.10.017
Paulus, M. P., Rogalsky, C., Simmons, A., Feinstein, J. S., & Stein, M. B. (2003). Increased activation in the right insula during risk-taking decision making is related to harm avoidance and neuroticism. Neuroimage, 19(4), 1439-1448.
Persichetti, A. S., Aguirre, G. K., & Thompson- Schill, S. L. (2014). Value Is in the Eye of the Beholder: Early Visual Cortex Codes Monetary Value of Objects during a Diverted Attention Task. Journal of Cognitive Neuroscience, 1-9.
Poldrack, R. A., Clark, J., Pare-Blagoev, E. J., Shohamy, D., Creso Moyano, J., Myers, C., & Gluck, M. A. (2001). Interactive memory systems in the human brain. Nature, 414(6863), 546-550.
Poldrack, R. A., Prabhakaran, V., Seger, C. A., & Gabrieli, J. D. (1999). Striatal activation during acquisition of a cognitive skill. Neuropsychology, 13(4), 564-574.
Preuschoff, K., Quartz, S. R., & Bossaerts, P. (2008). Human insula activation reflects risk prediction errors as well as risk. The Journal of neuroscience, 28(11), 2745-2752.
Quilodran, R., Rothe, M., & Procyk, E. (2008). Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron, 57(2), 314- 325.
Ridderinkhof, K. R., Ullsperger, M., Crone, E. A., & Nieuwenhuis, S. (2004). The role of the medial frontal cortex in cognitive control. Science, 306(5695), 443-447.
Rothkirch, M., Schmack, K., Schlagenhauf, F., & Sterzer, P. (2012). Implicit motivational value and salience are processed in distinct areas of orbitofrontal cortex. Neuroimage, 62(3), 1717- 1725.
Schultz, W., Apicella, P., Scarnati, E., & Ljungberg, T. (1992). Neuronal activity in monkey ventral striatum related to the expectation of reward. The Journal of neuroscience, 12(12), 4595-4610.
Seeley, W. W., Menon, V., Schatzberg, A. F., Keller, J., Glover, G. H., Kenna, H., . . . Greicius, M. D. (2007). Dissociable intrinsic connectivity networks for salience processing and executive control. Journal of Neuroscience, 27(9), 2349-2356.
Seger, C. A., & Cincotta, C. M. (2005). The roles of the caudate nucleus in human classification learning. The Journal of neuroscience, 25(11), 2941-2951.
Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., . . . Frackowiak, R. S. (2004). Temporal difference models describe higher-order learning in humans. Nature, 429(6992), 664-667.
Singer, T., Critchley, H. D., & Preuschoff, K. (2009). A common role of insula in feelings, empathy and uncertainty. Trends in Cognitive Sciences, 13(8), 334-340.
Skinner, B. F. (1938). The Behavior of Organisms New York: Appleton-Century-Crofts.
Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Vol. 1): Cambridge Univ Press.
Thorndike, E. L. (1911). Animal Intelligence: . London: Macmillan.
Tricomi, E., & Fiez, J. A. (2012). Information content and reward processing in the human striatum during performance of a declarative memory task. Cognitive, Affective & Behavioral Neuroscience, 12(2), 361-372.
Turken, A. U., & Swick, D. (1999). Response selection in the human anterior cingulate cortex. Nature Neuroscience, 2(10), 920-924.
Ullsperger, M., Harsay, H. A., Wessel, J. R., & Ridderinkhof, K. R. (2010). Conscious perception of errors and its relation to the anterior insula. Brain Structure and Function, 214(5-6), 629-643.
Weissman, D. H., Giesbrecht, B., Song, A. W., Mangun, G. R., & Woldorff, M. G. (2003). Conflict monitoring in the human anterior cingulate cortex during selective attention to global and local object features. Neuroimage, 19(4), 1361-1368.
Wheeler, E. Z., & Fellows, L. K. (2008). The human ventromedial frontal lobe is critical for learning from negative feedback. Brain, 131(Pt 5), 1323-1331.
Yacubian, J., Glascher, J., Schroeder, K., Sommer, T., Braus, D. F., & Buchel, C. (2006). Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. Journal of Neuroscience, 26(37), 9530-9537.
- 다운로드 수
- 조회수
- 0KCI 피인용수
- 0WOS 피인용수