- Log In/Sign Up
- P-ISSN1226-9654
- E-ISSN2733-466X
- KCI
ISSN : 1226-9654
Dynamic changes in feedback processing as learning progresses
Abstract
For cognitive control of behavioral adjustments in feedback learning, various processings are required, including evaluating the saliency (i.e., relevance to the task) and hedonic value of feedback information for future response selection. In this study, brain regions involved in processing feedback saliency were investigated by comparing activations for positive feedback (following correct responses) and negative feedback (following errors) for early and late phases of learning. A conditional associative learning task was used in which stimulus-response association rules were learned by trial and error, based on the feedback. Since there were four available responses to choose among for each stimulus, only positive feedback (i.e., reward) was relevant to behavioral adjustment during the early learning phase of learning, but negative feedback (e.g., penalty) became more relavant as learning progressed. fMRI data obtained from normal adults (n = 29) were analyzed to identify brain regions where responses to each feedback varied across the four consecutive runs. Activation for reward decreased as learning progressed, whereas activation for penalty increased in the following areas: anterior insula, dmPFC and anterior cingulate region, inferior PFC, inferior parietal cortex, and cerebellum. We interpret these results as reflecting the decreased saliency of positive feedback and increased saliency of negative feedback, between early and late phases of the learning task. In contrast, for two areas associated with processing of hedonic value, the ventral striatum and vmPFC, activations (positive > negative feedback) did not vary across the four consecutive runs. These observations suggest that the saliency of feedback for learning is processed in a network separate from that for the hedonic value of feedback.
- keywords
- Feedback, Learning, Saliency, Relevancy, Reward, Insula, dmPFC, Anterior cingulate, 피드백, 학습, 현저가, 과제관련성, 보상, 도, 배내측 전두영역, 전측 대상회
Reference
신연순, 한상훈 (2013). 목표지향적 학습과 기억. 감성과학, 16(3), 319-322.
Alexander, W. H., & Brown, J. W. (2011). Medial prefrontal cortex as an action-outcome predictor. Nature Neuroscience, 14(10), 1338- U1163.
Alexander, W. H., & Brown, J. W. (2014). A general role for medial prefrontal cortex in event prediction. Frontiers in Computational Neuroscience, 8, 69.
Amiez, C., Hadj-Bouziane, F., & Petrides, M. (2012). Response selection versus feedback analysis in conditional visuo-motor learning. Neuroimage, 59(4), 3723-3735.
Amiez, C., Sallet, J., Procyk, E., & Petrides, M. (2012). Modulation of feedback related activity in the rostral anterior cingulate cortex during trial and error exploration. Neuroimage, 63(3), 1078-1090.
Baird, L. C. (1993). Advantage updating: (No. WL-TR-93-1146). WRIGHT LAB WRIGHT PATTERSON AFB OH.
Behrens, T. E., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10(9), 1214-1221.
Bischoff-Grethe, A., Hazeltine, E., Bergren, L., Ivry, R. B., & Grafton, S. T. (2009). The influence of feedback valence in associative learning. Neuroimage, 44(1), 243-251.
Blair, R. J., Morris, J. S., Frith, C. D., Perrett, D. I., & Dolan, R. J. (1999). Dissociable neural responses to facial expressions of sadness and anger. Brain, 122 (Pt 5), 883-893.
Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S., & Cohen, J. D. (2001). Conflict monitoring and cognitive control. Psychological Review, 108(3), 624-652.
Botvinick, M. M., Cohen, J. D., & Carter, C. S. (2004). Conflict monitoring and anterior cingulate cortex: an update. Trends in Cognitive Sciences, 8(12), 539-546.
Braver, T. S., Barch, D. M., Gray, J. R., Molfese, D. L., & Snyder, A. (2001). Anterior cingulate cortex and response conflict: effects of frequency, inhibition and errors. Cerebral Cortex, 11(9), 825-836.
Bromberg-Martin, E. S., Matsumoto, M., & Hikosaka, O. (2010). Dopamine in motivational control: rewarding, aversive, and alerting. Neuron, 68(5), 815-834.
Chen, S. H., & Desmond, J. E. (2005). Temporal dynamics of cerebro-cerebellar network recruitment during a cognitive task. Neuropsychologia, 43(9), 1227-1237.
Chib, V. S., Rangel, A., Shimojo, S., & O'Doherty, J. P. (2009). Evidence for a common representation of decision values for dissimilar goods in human ventromedial prefrontal cortex. The Journal of neuroscience, 29(39), 12315-12320.
Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews: Neuroscience, 3(3), 201-215.
Critchley, H. D., Wiens, S., Rotshtein, P., Ohman, A., & Dolan, R. J. (2004). Neural systems supporting interoceptive awareness. Nature Neuroscience, 7(2), 189-195.
d'Acremont, M., Lu, Z. L., Li, X., Van der Linden, M., & Bechara, A. (2009). Neural correlates of risk prediction error during reinforcement learning in humans. Neuroimage, 47(4), 1929-1939.
Delgado, M. R. (2007). Reward-related responses in the human striatum. Annals of the New York Academy of Sciences, 1104(1), 70-88.
Desmond, J. E., & Fiez, J. A. (1998). Neuroimaging studies of the cerebellum: language, learning and memory. Trends in Cognitive Sciences, 2(9), 355-362.
Elliott, R., Dolan, R. J., & Frith, C. D. (2000). Dissociable Functions in the Medial and Lateral Orbitofrontal Cortex: Evidence from Human Neuroimaging Studies. Cerebral Cortex, 10(3), 308-317.
Elliott, R., Friston, K. J., & Dolan, R. J. (2000). Dissociable neural responses in human reward systems. Journal of Neuroscience, 20(16), 6159- 6165.
Elliott, R., Newman, J. L., Longe, O. A., & Deakin, W. J. F. (2003). Differential response patterns in the striatum and orbitofrontal cortex to financial reward in humans: a parametric functional magnetic resonance imaging study. Journal of Neuroscience, 23(1), 303-307. doi: 23/1/303 [pii]
Elliott, R., Rees, G., & Dolan, R. J. (1999). Ventromedial prefrontal cortex mediates guessing. Neuropsychologia, 37(4), 403-411.
Esber, G. R., & Haselgrove, M. (2011). Reconciling the influence of predictiveness and uncertainty on stimulus salience: a model of attention in associative learning. Proceedings: Biological Sciences, 278(1718), 2553-2561.
Goulden, N., Khusnulina, A., Davis, N. J., Bracewell, R. M., Bokde, A. L., McNulty, J. P., & Mullins, P. G. (2014). The salience network is responsible for switching between the default mode network and the central executive network: replication from DCM. Neuroimage, 99, 180-190.
Hare, T. A., Camerer, C. F., & Rangel, A. (2009). Self-control in decision-making involves modulation of the vmPFC valuation system. Science, 324(5927), 646-648.
Haruno, M., & Kawato, M. (2006). Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. Journal of Neurophysiology, 95(2), 948-959.
Hester, R., Foxe, J. J., Molholm, S., Shpaner, M., & Garavan, H. (2005). Neural mechanisms involved in error processing: A comparison of errors made with and without awareness. Neuroimage, 27(3), 602-608.
Holroyd, C. B., & Coles, M. G. (2002). The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109(4), 679-709.
Holroyd, C. B., Nieuwenhuis, S., Yeung, N., Nystrom, L., Mars, R. B., Coles, M. G., & Cohen, J. D. (2004). Dorsal anterior cingulate cortex shows fMRI response to internal and external error signals. Nature Neuroscience, 7(5), 497-498.
Jung, J., Jerbi, K., Ossandon, T., Ryvlin, P., Isnard, J., Bertrand, O., . . . Lachaux, J. P. (2010). Brain responses to success and failure: Direct recordings from human cerebral cortex. Human Brain Mapping, 31(8), 1217-1232.
Kahnt, T., Park, S. Q., Haynes, J. D., & Tobler, P. N. (2014). Disentangling neural representations of value and salience in the human brain. Proceedings of the National Academy of Sciences of the United States of America, 111(13), 5000-5005.
Kastner, S., & Ungerleider, L. G. (2000). Mechanisms of visual attention in the human cortex. Annual Review of Neuroscience, 23(1), 315-341.
Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001). Anticipation of increasing monetary reward selectively recruits nucleus accumbens. The Journal of neuroscience, 21(16), RC159.
Knutson, B., Westdorp, A., Kaiser, E., & Hommer, D. (2000). FMRI Visualization of Brain Activity during a Monetary Incentive Delay Task. Neuroimage, 12(1), 20-27.
Kringelbach, M. L. (2005). The human orbitofrontal cortex: linking reward to hedonic experience. Nature Reviews: Neuroscience, 6(9), 691-702.
Kringelbach, M. L., & Rolls, E. T. (2004). The functional neuroanatomy of the human orbitofrontal cortex: evidence from neuroimaging and neuropsychology. Progress in Neurobiology, 72(5), 341-372.
Lammel, S., Ion, D. I., Roeper, J., & Malenka, R. C. (2011). Projection-specific modulation of dopamine neuron synapses by aversive and rewarding stimuli. Neuron, 70(5), 855-862.
Lang, P. J., & Davis, M. (2006). Emotion, motivation, and the brain: reflex foundations in animal and human research. Progress in Brain Research, 156, 3-29.
Li, W., Piech, V., & Gilbert, C. D. (2004). Perceptual learning and top-down influences in primary visual cortex. Nature Neuroscience, 7(6), 651-657.
Lin, A., Adolphs, R., & Rangel, A. (2012). Social and monetary reward learning engage overlapping neural substrates. Social Cognitive and Affective Neuroscience, 7(3), 274-281.
Lin, S. C., & Nicolelis, M. A. (2008). Neuronal ensemble bursting in the basal forebrain encodes salience irrespective of valence. Neuron, 59(1), 138-149.
Liu, X., Powell, D. K., Wang, H., Gold, B. T., Corbly, C. R., & Joseph, J. E. (2007). Functional dissociation in frontal and striatal areas for processing of positive and negative reward information. Journal of Neuroscience, 27(17), 4587-4597.
Mackintosh, N. J. (1983). Conditioning and Associative Learning. Oxford: Clarendon Press.
Mars, R. B., Coles, M. G. H., Grol, M. J., Holroyd, C. B., Nieuwenhuis, S., Hulstijn, W., & Toni, I. (2005). Neural dynamics of error processing in medial frontal cortex. Neuroimage, 28(4), 1007-1013.
Matsumoto, M., & Hikosaka, O. (2009). Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature, 459(7248), 837-841.
Menon, V., & Uddin, L. Q. (2010). Saliency, switching, attention and control: a network model of insula function. Brain Struct Funct, 214(5-6), 655-667.
Metereau, E., & Dreher, J. C. (2013). Cerebral correlates of salient prediction error for different rewards and punishments. Cerebral Cortex, 23(2), 477-487.
Nieuwenhuis, S., Slagter, H. A., von Geusau, N. J., Heslenfeld, D. J., & Holroyd, C. B. (2005). Knowing good from bad: differential activation of human cortical areas by positive and negative outcomes. European Journal of Neuroscience, 21(11), 3161-3168.
O'Doherty, J. P., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304(5669), 452-454.
O'Doherty, J. P., Kringelbach, M. L., Rolls, E. T., Hornak, J., & Andrews, C. (2001). Abstract reward and punishment representations in the human orbitofrontal cortex. Nature Neuroscience, 4(1), 95-102.
Palminteri, S., Justo, D., Jauffret, C., Pavlicek, B., Dauta, A., Delmaire, C., . . . Pessiglione, M. (2012). Critical roles for anterior insula and dorsal striatum in punishment-based avoidance learning. Neuron, 76(5), 998-1009. doi: 10.1016/ j.neuron.2012.10.017
Paulus, M. P., Rogalsky, C., Simmons, A., Feinstein, J. S., & Stein, M. B. (2003). Increased activation in the right insula during risk-taking decision making is related to harm avoidance and neuroticism. Neuroimage, 19(4), 1439-1448.
Persichetti, A. S., Aguirre, G. K., & Thompson- Schill, S. L. (2014). Value Is in the Eye of the Beholder: Early Visual Cortex Codes Monetary Value of Objects during a Diverted Attention Task. Journal of Cognitive Neuroscience, 1-9.
Poldrack, R. A., Clark, J., Pare-Blagoev, E. J., Shohamy, D., Creso Moyano, J., Myers, C., & Gluck, M. A. (2001). Interactive memory systems in the human brain. Nature, 414(6863), 546-550.
Poldrack, R. A., Prabhakaran, V., Seger, C. A., & Gabrieli, J. D. (1999). Striatal activation during acquisition of a cognitive skill. Neuropsychology, 13(4), 564-574.
Preuschoff, K., Quartz, S. R., & Bossaerts, P. (2008). Human insula activation reflects risk prediction errors as well as risk. The Journal of neuroscience, 28(11), 2745-2752.
Quilodran, R., Rothe, M., & Procyk, E. (2008). Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron, 57(2), 314- 325.
Ridderinkhof, K. R., Ullsperger, M., Crone, E. A., & Nieuwenhuis, S. (2004). The role of the medial frontal cortex in cognitive control. Science, 306(5695), 443-447.
Rothkirch, M., Schmack, K., Schlagenhauf, F., & Sterzer, P. (2012). Implicit motivational value and salience are processed in distinct areas of orbitofrontal cortex. Neuroimage, 62(3), 1717- 1725.
Schultz, W., Apicella, P., Scarnati, E., & Ljungberg, T. (1992). Neuronal activity in monkey ventral striatum related to the expectation of reward. The Journal of neuroscience, 12(12), 4595-4610.
Seeley, W. W., Menon, V., Schatzberg, A. F., Keller, J., Glover, G. H., Kenna, H., . . . Greicius, M. D. (2007). Dissociable intrinsic connectivity networks for salience processing and executive control. Journal of Neuroscience, 27(9), 2349-2356.
Seger, C. A., & Cincotta, C. M. (2005). The roles of the caudate nucleus in human classification learning. The Journal of neuroscience, 25(11), 2941-2951.
Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., . . . Frackowiak, R. S. (2004). Temporal difference models describe higher-order learning in humans. Nature, 429(6992), 664-667.
Singer, T., Critchley, H. D., & Preuschoff, K. (2009). A common role of insula in feelings, empathy and uncertainty. Trends in Cognitive Sciences, 13(8), 334-340.
Skinner, B. F. (1938). The Behavior of Organisms New York: Appleton-Century-Crofts.
Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Vol. 1): Cambridge Univ Press.
Thorndike, E. L. (1911). Animal Intelligence: . London: Macmillan.
Tricomi, E., & Fiez, J. A. (2012). Information content and reward processing in the human striatum during performance of a declarative memory task. Cognitive, Affective & Behavioral Neuroscience, 12(2), 361-372.
Turken, A. U., & Swick, D. (1999). Response selection in the human anterior cingulate cortex. Nature Neuroscience, 2(10), 920-924.
Ullsperger, M., Harsay, H. A., Wessel, J. R., & Ridderinkhof, K. R. (2010). Conscious perception of errors and its relation to the anterior insula. Brain Structure and Function, 214(5-6), 629-643.
Weissman, D. H., Giesbrecht, B., Song, A. W., Mangun, G. R., & Woldorff, M. G. (2003). Conflict monitoring in the human anterior cingulate cortex during selective attention to global and local object features. Neuroimage, 19(4), 1361-1368.
Wheeler, E. Z., & Fellows, L. K. (2008). The human ventromedial frontal lobe is critical for learning from negative feedback. Brain, 131(Pt 5), 1323-1331.
Yacubian, J., Glascher, J., Schroeder, K., Sommer, T., Braus, D. F., & Buchel, C. (2006). Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. Journal of Neuroscience, 26(37), 9530-9537.
- Downloaded
- Viewed
- 0KCI Citations
- 0WOS Citations