바로가기메뉴

본문 바로가기 주메뉴 바로가기

ACOMS+ 및 학술지 리포지터리 설명회

  • 한국과학기술정보연구원(KISTI) 서울분원 대회의실(별관 3층)
  • 2024년 07월 03일(수) 13:30
 

한국심리학회지: 일반

심리학 연구에서 재현성 위기의 현황과 원인 및 대안 모색에 대한 개관

Replication crisis in psychology: A review of its causes and solutions

한국심리학회지: 일반 / Korean Journal of Psychology: General, (P)1229-067X; (E)2734-1127
2017, v.36 no.3, pp.359-396
https://doi.org/10.22257/kjp.2017.09.36.3.359
김빛나 (서울대학교)
최준원 (상명대학교)
고현석 (공군사관학교)
  • 다운로드 수
  • 조회수

초록

최근 심리학계에서는 심리학 연구의 재현성에 대한 치열한 논란이 전개되어 왔다. 점화 효과 등 학계 및 일반 대중에게 큰 관심을 끌었던 유명 연구들이 반복검증에 실패하였고, 뒤따라 착수된 대규모 반복검증 프로젝트들에서의 반복검증 성공률도 만족스럽지 못한 것으로 나타나 과학으로서 심리학의 위상이 위협받는 계기가 되었다. 이 과정에서 통계적 추론절차, 연구관행 그리고 출판편향 등이 재현성 위기의 원인들로서 조명되었으며, 그에 따라 심리학의 재현성을 증진하기 위한 해결책들(신뢰구간, 효과 크기의 사용, 메타분석, 베이지안 통계, 연구 자료와 방법론의 투명성 증가)도 제안되고 있다. 앞으로 어떻게 해결되는가에 따라 작금의 재현성 위기는 향후 심리학이 발전해 나가는 기회가 될 가능성도 가지고 있다. 이에 본 논문에서는 심리학 내 재현성 위기가 전개되어온 현황을 소개하고 현재까지 도출된 원인과 잠정적 해결책들을 종합적으로 개관함으로써, 국내에서 관련 논의가 촉진될 수 있는 기초를 마련하고자 한다.

keywords
replication, reproducibility, publication bias, methodology, 반복검증, 재현성, 출판편향, 방법론

Abstract

Recently, there has been an ongoing controversy surrounding replication crisis in psychology. Some of popular experiments such as priming effect failed to be replicated. And subsequent large-scale replication projects of psychological studies revealed that the success rate of replication was less than ideal, which threatened the status of psychology as science. Through the course of mass debate, problems such as null hypothesis statistical testing, questionable research practices, and publication bias were discussed as causes of current replication crisis. Accordingly, solutions (use of confidence interval, effect size, meta-analysis, Bayesian statistics, and efforts to increase transparency in methodology) were proposed in order to enhance reproducibility of psychological research. Depending on how it is resolved, the present crisis in replication also holds opportunity for future development of psychology. Therefore, we attempted to introduce how replication crisis unfolded in psychology and to comprehensively review causes and potential solutions in the hope of establishing basic ground for further discussion in Korea.

keywords
replication, reproducibility, publication bias, methodology, 반복검증, 재현성, 출판편향, 방법론

참고문헌

1.

교보문고 (2016). 종합 연간 베스트. http://www.kyobobook.co.kr/bestSellerNew/bestseller.laf?range=1&kind=3&orderClick=DAC&mallGb=KOR&linkClass=A에서 2016, 12, 13 자료 얻음.

2.

김용대, 김혜중, 오만숙, 오현숙, 정윤식 (2001). 베이지안 통계학의 과거ㆍ현재ㆍ미래. 한국통계학회논문집, 8, 47-64.

3.

김청택 (2011). 통계적 가설검증의 절차와 문제점 그리고 대안. 서울: 민속원.

4.

김청택 (2013). Bayesian 통계의 소개. 강의자료 (2013. 6. 2.)

5.

박준석 (2015). 영가설 유의성검증 절차의 문제점들에 대한 해결책으로서의 베이지언통계학: 심리학의 경우. 과학철학, 18(2), 135-147.

6.

박준석 (2016). ‘연구 사전등록제’, 재현성 위기의제도적 해법. 사이언스온. http://scienceon.hani.co.kr/media/414001에서 2017, 1, 7 자료얻음.

7.

한국금융신문 (2015). 또 한 번의 개가(凱歌). http://www.fntimes.com/paper/view.aspx?num=142353에서 2016, 12, 15 자료 얻음.

8.

한국연구재단 (2016). 한국학술지인용색인(Korea Citation Index). https://www.kci.go.kr/kciportal/po/search/poCitaSearList.kci?years=5&search=directory&year=2015&field=*&page=1에서 2016, 12, 13 자료 얻음.

9.

Albarracín, D. (2015) Editorial. Psychological Bulletin, 141(1), 1-5.

10.

Alogna, V. K., Attaya, M. K., Aucoin, P., Bahník, Š., Birch, S., Birt, A. R., ..., & Buswell, K. (2014). Registered replication report: Schooler and engstler-schooler (1990). Perspectives on Psychological Science, 9(5), 556-578.

11.

American Statistical Association (2016). American Statistical Association releases statement on statistical significance and p-values. https://www.amstat.org/search?search=American%20Statistical%20Association%20releases%20statement%20on%20statistical%20significance%20and%20p-values에서 2017, 7, 3 자료 얻음

12.

Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J., Fiedler, K., ..., & Wicherts, J. M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108-119.

13.

Ashworth, A. (2015). Veto on the use of null hypothesis testing and p intervals: Right or wrong? Taylor & Francis Editor. http://editorresources.taylorandfrancisgroup.com/veto-on-the-use-of-null-hypothesis-testing-and-p-intervals-right-or-wrong에서 2017, 7, 1 자료 얻음.

14.

Association for Psychological Science (2013). Registered replication reports. http://www.psychologicalscience.org/publications/replication 에서 2016, 12, 1 자료 얻음.

15.

Balluerka, N., Gómez, J., & Hidalgo, D. (2005). The controversy over null hypothesis significance testing revisited. Methodology, 1(2), 55-70.

16.

Banerjee, P., Chatterjee, P., & Sinha, J. (2012). Is it light or dark? Recalling moral behavior changes perception of brightness. Psychological Science, 23(4), 407-409.

17.

Bargh, J. A. (2012). Priming effects replicate just fine, thanks. Psychology Today, 11. https://www.psychologytoday.com/blog/the-natural-unconscious/201205/priming-effects-replicate-just-fine-thanks에서 2017, 1, 7 자료 얻음.

18.

Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology, 71(2), 230-244.

19.

Baumeister, R. F. (2016). Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. Journal of Experimental Social Psychology, 66, 153-158.

20.

Bayarri, M. J., & Mayoral, A. M. (2002). Bayesian design of “successful” replications. The American Statistician, 56(3), 207-214.

21.

Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531-533.

22.

Bem, D. J. (2011). Feeling the future:Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407-425.

23.

Berezow, A. B. (2012). Why psychology isn’t science. http://articles.latimes.com/2012/jul/13/news/la-olblowback-pscyhology-science-20120713에서2016, 7, 14 자료 얻음.

24.

Berger, J. O. (1985). Statistical decision theory and Bayesian analysis (2nd ed). New York: Spinger.

25.

Bhattacharjee, Y (2013). The mind of a con man. http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audacious-academic-fraud.html?pagewanted=all에서 2016, 12, 1 자료 얻음.

26.

Bissell, M. (2013). The risks of the replication drive. Nature, 503(7476), 333-334.

27.

Bollen, K., Cacioppo, J. T., Kaplan, R. M., Krosnick, J. A., & Olds, J. L. (2015). Social, behavioral, and economic sciences perspectives on robust and reliable science. www.nsf.gov/sbe/AC_Materials/SBE_Robust_and_Reliable_Research_Report.pdf에서 2016, 12, 1 자료 얻음.

28.

Brandt, M. J., IJzerman, H., & Blanken, I. (2014). Does recalling moral behavior change the perception of brightness? Social Psychology, 45(3), 246-252.

29.

Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., ..., & Van't Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217-224.

30.

Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014). Continuously cumulating meta-analysis and replicability. Perspectives on Psychological Science, 9(3), 333-342.

31.

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376.

32.

Carter, T. J., Ferguson, M. J., & Hassin, R. R. (2011). A single exposure to the American flag shifts support toward Republicanism up to 8 months later. Psychological Science, 22(8), 1011-1018.

33.

Caruso, E. M., Vohs, K. D., Baxter, B., & Waytz, A. (2013). Mere exposure to money increases endorsement of freemarket systems and social inequality. Journal of Experimental Psychology:General, 142(2), 301-306.

34.

Center for Open Science (2013). Our mission is to increase openness, integrity, and reproducibility of research. https://cos.io/about/mission/에서 2016, 1, 10 자료 얻음.

35.

Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9(1), 40-48.

36.

Chambers, C. (2014). Physics envy: Do ‘hard’ sciences hold the solution to the replication crisis in psychology? http://www.theguardian.com/science/head-quarters/2014/jun/10/physics-envy-do-hardsciences-hold-the-solution-to-the-replication-crisis-in-psychology에서 2016, 7, 14 자료 얻음.

37.

Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. (2014). Instead of "playing the game" it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1(1), 4-17.

38.

Cheung, I., Campbell, L., & LeBel, E. P., ... & Yong, J. C. (2016). Registered Replication Report: Study 1 from Finkel, Rusbult, Kumashiro, & Hannon (2002). Perspectives on Psychological Science, 11(5), 750-764.

39.

Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. The Journal of Abnormal and Social Psychology, 65(3), 145-153.

40.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). NJ: Lawrence Earlbaum Associates.

41.

Cohen, J. (1994). The earth is round (p<.05). American Psychologist, 49(12), 997-1003.

42.

Coyne, J. C. (2016). Replication initiatives will not salvage the trustworthiness of psychology. BMC Psychology, 4, 28.

43.

Crandall, C. S., & Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93-99.

44.

Crocker, J., & Cooper, M. L. (2011). Addressing scientific fraud. Science, 334(6060), 1182-1182.

45.

Crotty, D. (2014). When crises collide: The tension between null results and reproducibility. https://scholarlykitchen.sspnet.org/2014/09/10/when-crises-collide-the-tension-between-null-results-and-reproducibility/에서 2016, 7, 18 자료 얻음.

46.

Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. NY: Routledge.

47.

Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7-29.

48.

Cumming, G., & Finch, S. (2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60(2), 170-180.

49.

Dovidio, J. F. (2016). Commentary: A big problem requires a foundational change. Journal of Experimental Social Psychology, 66, 159-165.

50.

Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: It's all in the mind, but whose mind? PLoS One, 7(1), e29081.

51.

Driscoll, R., Davis, K. E., & Lipetz, M. E. (1972). Parental interference and romantic love: The Romeo and Juliet effect. Journal of Personality and Social Psychology, 24(1), 1-10.

52.

Dunn, D. S. (2015). The Oxford handbook of undergraduate psychology education . Oxford University Press.

53.

Eerland, A., Sherrill, A. M., Magliano, J. P., Zwaan, R. A., Arnal, J. D., Aucoin, P., ..., & Crocker, C. (2016). Registered replication report: Hart & Albarracín (2011). Perspectives on Psychological Science, 11(1), 158-171.

54.

Eich, E. (2014). Business not as usual. Psychological Science, 25(1), 3-6.

55.

Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629-634.

56.

Ellis, P. D. (2010). The essential guide to effect sizes:Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press.

57.

Etz, A., & Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project:Psychology. PLoS One, 11(2), e0149794.

58.

Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PLoS One, 5(4), e10068.

59.

Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175-191.

60.

Feinstein, A. R. (1995). Meta-analysis: Statistical alchemy for the 21st century. Journal of Clinical Epidemiology, 48(1), 71-79.

61.

Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015). Best research practices in psychology:Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology, 108(2), 275-297.

62.

Fraley, R. C., & Vazire, S. (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS One, 9(10), e109019.

63.

Francis, G. (2012a). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19(6), 975-991.

64.

Francis, G. (2012b). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7(6), 585-594.

65.

Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences:Unlocking the file drawer. Science, 345(6203), 1502-1505.

66.

Funder, D. C., Levine, J. M., Mackie, D. M., Morf, C. C., Vazire, S., & West, S. G. (2014). Improving the dependability of research in personality and social psychology recommendations for research and educational practice. Personality and Social Psychology Review, 18(1), 3-12.

67.

Galak, J., LeBoeuf, R. A., Nelson, L. D., & Simmons, J. P. (2012). Correcting the past:Failures to replicate psi. Journal of Personality and Social Psychology, 103(6), 933-948.

68.

Gelman, A. (2015). The connection between varying treatment effects and the crisis of unreplicable research: A Bayesian perspective. Journal of Management, 41(2), 632-643.

69.

Gelman, A., & Geurts, H. M. (2017). The statistical crisis in science: How is it relevant to clinical neuropsychology? The Clinical Neuropsychologist, 1-15.

70.

Gergen, K. J. (1973). Social psychology as history. Journal of Personality and Social Psychology, 26(2), 309-320.

71.

Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33(5), 587-606.

72.

Gigerenzer, G., Krauss, S., & Vitouch, O. (2004). The Null Ritual: What you always wanted to know about significance testing but were afraid to ask. In D. Kaplan (Ed.), The Sage book of quantitative methodology for the social sciences (pp. 391-408). SAGE publications.

73.

Gilbert, D. T. (2014). Some thoughts on shameless bullies. http://www.psychol.cam.ac.uk/cece/blog에서 2014, 9, 11 자료 얻음.

74.

Ginger-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7(6), 562-571.

75.

Giofrè, D., Cumming, G., Fresc, L., Boedker, I., & Tressoldi, P. (2017). The influence of journal submission guidelines on authors'reporting of statistics and use of open research practices. PloS One, 12(4), e0175583.

76.

Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31, 337-350.

77.

Hales, A. H. (2016). Does the conclusion follow from the evidence? Recommendations for improving research. Journal of Experimental Social Psychology, 66, 39-46.

78.

Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3), e1002106.

79.

Hedges, L. V. (1987). How hard is hard science, how soft is soft science? The empirical cumulativeness of research. American Psychologist, 42(5), 443-455.

80.

Hüffmeier, J., Mazei, J., & Schultze, T. (2016). Reconceptualizing replication as a sequence of different studies: A replication typology. Journal of Experimental Social Psychology, 66, 81-92.

81.

Husnu, S., & Crisp, R. J. (2010). Elaboration enhances the imagined contact effect. Journal of Experimental Social Psychology, 46(6), 943-950.

82.

Hyde, J. S. (2001). Reporting effect sizes: The roles of editors, textbook authors, and publication manuals. Educational and Psychological Measurement, 61(2), 225-228.

83.

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Med, 2(8), e124.

84.

Jasny, B. R., Chin, G., Chong, L., & Vignieri, S. (2011). Again, and again, and again…. Science, 334(6060), 1225-1225.

85.

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532.

86.

Johnson, D. J., Cheung, F., & Donnellan, M. B. (2014). Does cleanliness influence moral judgments? Social Psychology, 45(3), 209-215.

87.

Judd, C. M., & Gawronski, B. (2011). Editorial comment. Journal of Personality and Social Psychology, 100(3), 406.

88.

Kahneman, D. (2014). A new etiquette for replication. Social Psychology, 45(4), 310-311.

89.

Kail, R. V. (2012). Reflections on five years as editor. Observer, 25(9).

90.

Kelley, H. (1992). Common-sense psychology and scientific psychology. Annual Review of Psychology, 43(1), 1-23.

91.

Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196-217.

92.

Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L. S., ..., & Errington, T. M. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14(5), e1002456.

93.

Klein, R. A., Ratliff, K. A., Vianello, M., Adams Jr, R. B., Bahník, Š., Bernstein, M. J., ..., & Nosek, B. A. (2014). Investigating variation in replicability. Social Psychology, 45(3), 142-152.

94.

Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method. American Psychologist, 56(1), 16-26.

95.

Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4(863).

96.

Lakens, D., Hilgard, J., & Staaks, J. (2016). On the reproducibility of meta-analyses: Six practical recommendations. BMC Psychology, 4, 24.

97.

Laws, K. R. (2016). Psychology, replication &beyond. BMC Psychology, 4, 30.

98.

Levine, T. R., Weber, R., Hullett, C., Park, H. S., & Lindsey, L. L. M. (2008). A critical assessment of null hypothesis significance testing in quantitative communication research. Human Communication Research, 34(2), 171-187.

99.

Lilienfeld, S. O. (2012). Public skepticism of psychology: Why many people perceive the study of human behavior as unscientific. American Psychologist, 67(2), 111-129.

100.

Lindsay, D. S. (2015). Replication in psychological science. Psychological Science, 26(12), 1827-1832.

101.

Lindsay, D. S. (2017). Editor's perspective: What happened when Psychological Science began encouraging the use of the new statistics? Presentation in the 29th Association for Psychological Science (APS) Convention. Boston, MA. https://thenewstatistics.com/itns/wp-content/uploads/2017/05/APS-Editor%E2%80%99s-Perspective.pdf에서 2017, 7, 11 자료얻음.

102.

Lynott, D., Corker, K. S., Wortman, J., Connell, L., Donnellan, M. B., Lucas, R. E., & O’Brien, K. (2014). Replication of “Experiencing physical warmth promotes interpersonal warmth” by Williams and Bargh (2008). Social Psychology, 45(3), 216-222.

103.

Madden, C. S., Easley, R. W., & Dunn, M. G. (1995). How journal editors view replication research. Journal of Advertising, 24(4), 77-87.

104.

Makel, M. C., & Plucker, J. A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304-316.

105.

Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research:How often do they really occur? Perspectives on Psychological Science, 7(6), 537-542.

106.

Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487-498.

107.

McGrath, J. E. (1981). Dilemmatics: The study of research choices and dilemmas. American Behavioral Scientist, 25(2), 179-201.

108.

Moonesinghe, R., Khoury, M. J., & Janssens, A. C. J. (2007). Most published research findings are false: But a little replication goes a long way. PLoS Medicine, 4(2), e28.

109.

Murayama, K., Pekrun, R., & Fiedler, K. (2014). Research practices that can prevent an inflation of false-positive rates. Personality and Social Psychology Review, 18(2), 107-118.

110.

Neuliep, J. W., & Crandall, R. (1990). Editorial bias against replication research. Journal of Social Behavior and Personality, 5(4), 85-90.

111.

Neuliep, J. W., & Crandall, R. (1993). Reviewer bias against replication research. Journal of Social Behavior and Personality, 8(6), 21-29.

112.

Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016). Publication bias and the canonization of false facts. Elife, 5, e21451.

113.

Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45(3), 137-141.

114.

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615-631.

115.

Nuzzo, R. (2014). Statistical errors: P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume. Nature, 506(7487), 150-153.

116.

Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657-660.

117.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), 943-951.

118.

Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531-536.

119.

Pashler, H., Harris, C. R., & Coburn, N. (2011). Elderly-related words prime slow walking. http://www.PsychFileDrawer.org/replication.php?attempt=MTU%3D에서 2016, 12, 9 자료얻음.

120.

Pashler, H., & Wagenmakers, E. J. (2012). Editors introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528-530.

121.

Popper, K. R. (1959). The logic of scientific discovery. London: Hutchinson.

122.

Popper, K. R. (1963). Conjectures and refutations: The growth of scientific knowledge. London: Routledge & Kegan Paul.

123.

PsychFileDrawer (2012). Archives for the attempts at replication in experimental psychology. http://psychfiledrawer.org/에서 2016, 6, 27 자료 얻음.

124.

Ritchie, S. J., Wiseman, R., & French, C. C. (2012). Failing the future: Three unsuccessful attempts to replicate Bem's retroactive facilitation of recall effect. PLoS One, 7(3), e33423.

125.

Robinson, E. (2011). Not feeling the future: A failed replication of retroactive facilitation of memory recall. Journal of the Society for Psychical Research, 75(904), 142-147.

126.

Roediger, H. L. (2012). Psychology’s woes and a partial cure: The value of replication. APS Observer, 25 (2), 9. http://www.psychologicalscience.org/issue/february-12에서2016, 12, 26 자료 얻음.

127.

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638-641.

128.

Rouder, J. N., & Morey, R. D. (2011). A Bayes factor meta-analysis of Bem’s ESP claim. Psychonomic Bulletin & Review, 18(4), 682-689.

129.

Rossi, J. S. (1990). Statistical power of psychological research: What have we gained in 20 years? Journal of Consulting and Clinical Psychology, 58(5), 646-656.

130.

Sakaluk, J. K. (2016). Exploring small, confirming big: An alternative system to the new statistics for advancing cumulative and replicable psychological research. Journal of Experimental Social Psychology, 66, 47-54.

131.

Schaller, M. (2016). The empirical benefits of conceptual rigor: Systematic articulation of conceptual hypotheses can reduce the risk of non-replicable results (and facilitate novel discoveries too). Journal of Experimental Social Psychology, 66, 107-115.

132.

Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1(2), 115-129.

133.

Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90-100.

134.

Schnall, S. (2014). Simone Schnall on her experience with a Registered Replication Project. http://www.spsp.org/blog/simone-schnall-on-her-experience-with-a-registered-replication-project에서 2014, 9, 11 자료 얻음.

135.

Schnall, S., Benton, J., & Harvey, S. (2008). With a clean conscience cleanliness reduces the severity of moral judgments. Psychological Science, 19(12), 1219-1222.

136.

Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., ..., & Srinivasan, M. (2016). The pipeline project:Pre-publication independent replications of a single laboratory's research pipeline. Journal of Experimental Social Psychology, 66, 55-67.

137.

Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105(2), 309-316.

138.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366.

139.

Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9(1), 76-80.

140.

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534-547.

141.

Sinclair, H. C., Hood, K. B., & Wright, B. L. (2014). Revisiting the Romeo and Juliet effect (Driscoll, Davis, & Lipetz, 1972). Social Psychology, 45(3), 170-178.

142.

Spellman, B. A. (2012). Introduction to the special section on research practice. Perspectives on Psychological Science, 7(6), 655-656.

143.

Stangor, C., & Lemay, E. P. (2016). Introduction to the special issue on methodological rigor and replicability. Journal of Experimental Social Psychology, 66, 1-3.

144.

Stroebe, W. (2016). Are most published social psychological findings false? Journal of Experimental Social Psychology, 66, 134-144.

145.

Stroebe, W., Postmes, T., & Spears, R. (2012). Scientific misconduct and the myth of self-correction in science. Perspectives on Psychological Science, 7(6), 670-688.

146.

Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59-71.

147.

Taylor, A. E., & Munafò, M. R. (2016). Triangulating meta-analyses: The example of the serotonin transporter gene, stressful life events and major depression. BMC Psychology, 4, 23.

148.

Thaler, R. H. (2016). 똑똑한 사람들의 멍청한선택 (박세연 역). 서울: 리더스북. (원전은2015년에 출판).

149.

Trafimow, D., & Marks, M. (2015). Editorial. Basic and Applied Social Psychology, 37, 1-2.

150.

Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358(3), 252-260.

151.

Verhagen, J., & Wagenmakers, E. J. (2014). Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General, 143(4), 1457-1475.

152.

Wagenmakers, E. J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams Jr, R. B., ..., & Bulnes, L. C. (2016). Registered Replication Report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11(6), 917-928.

153.

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA's statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133.

154.

Wagenmakers, E. J., Lee, M., Lodewyckx, T., & Iverson, G. J. (2008). Bayesian versus frequentist inference. In H. Hoijtink, I. Klugkist, & P. A. Boelen (Eds.), Bayesian evaluation of informative hypotheses (pp. 181-207). New York: Spinger Verlag.

155.

Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7(6), 632-638.

156.

Williams, L. E., & Bargh, J. A. (2008). Experiencing physical warmth promotes interpersonal warmth. Science, 322(5901), 606-607.

157.

Wilson, T. (2012). Stop bullying the ‘soft’ sciences. http://articles.latimes.com/2012/jul/12/opinion/laoe-wilson-social-sciences-20120712에서 2016, 7, 14 자료 얻음.

158.

Yong, E. (2012). In the wake of high profile controversies, psychologists are facing up to problems with replication. Nature, 485(17), 298-300.

한국심리학회지: 일반