ISSN : 1229-067X
Online surveys are widely used not only in general survey research but also in organizational research, such as customer and employee surveys, due to the ease of implementation and the considerations of cost-effectiveness compared to conventional paper-pencil surveys. There has been much research devoted to showing the negative impact of careless responding on subsequent data analysis and to evaluating statistical methods for detecting the careless respondents embedded in the data collected via online surveys. However, it would be fair to say that there has been a paucity of empirical research evaluating the effectiveness of a preventive measure against careless responding that would operate during the data collection phase of online survey. In this study, inspired by the objective self-awareness theory in social psychology, we proposed a simple but novel method of inserting a mirror-image in the background of each online survey page as a preventive measure of careless responding to the survey items. We collected data under three different conditions of control(standard instructions only), warning(standard instructions plus warnings against careless responding), and the mirror-image condition. Statistical properties of several existing indicators developed for detecting various types of careless respondents were compared among the three conditions. Our results suggested that the simple mirror-image method tended to reduce the careless responding more effectively than the warning and control condition. We concluded by discussing implications and limitations of our study.
김병수, 김정현 (2013). 스트레스 척도 핸드북. 서울: 학지사
배성우, 신원식 (2005). CES-D 척도(The Center for Epidemiologic Studies-Depression Scale)의 요인구조 분석. 보건과 사회과학, 18, 165-190.
이양현, 임효덕, 이종영 (1996). 한국판 20 항목 Toronto 감정표현불능증 척도(TAS-20K)의 개발과 타당도. 신경정신의학, 35(4), 888-899.
최보라 (2007). 사회적 바람직성 척도 타당화. 이화여대석사학위논문.
Aronson, E., Fried, C., & Stone, J. (1991). Overcoming denial and increasing the intention to use condoms through the induction of hypocrisy. American Journal of Public Health, 81(12), 1636-1638.
Baer, R. A., Ballenger, J., Berry, D. T., & Wetter, M. W. (1997). Detection of random responding on the MMPI-A. Journal of Personality Assessment, 68, 139-151.
Bargh, J. A., Gollwitzer, P. M., Lee-Chai, A., Barndollar, K., & Trötschel, R. (2001). The automated will: Nonconscious activation and pursuit of behavioral goals. Journal of Personality and Social Psychology, 81, 1014-1027.
Beaman, A. L., Klentz, B., Diener, E., & Svanum, S. (1979). Self-awareness and transgression in children: Two field studies. Journal of Personality and Social Psychology, 37, 1835-1846.
Berkowitz, L., & LePage, A. (1967). Weapons as aggression-eliciting stimuli. Journal of Personality and Social Psychology, 7, 202-207.
Berry, D. T., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMPI-2random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340-345.
Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111, 218-229.
Breitsohl, H., & Steidelmüller, C. (2018). The impact of insufficient effort responding detection methods on substantive responses:Results from an experiment testing parameter invariance. Applied Psychology, 67(2), 284-308.
Bruehl, S., Lofland, K. R., Sherman, J. J., & Carlson, C. R. (1998). The Variable Responding Scale for detection of random responding on the Multidimensional Pain Inventory. Psychological Assessment, 10(1), 3-9.
Buechley, R., & Ball, H. (1952). A new test of “validity” for the group MMPI. Journal of Consulting Psychology, 16(4), 299-301.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3-5.
Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A., & Kaemmer, B. (1989). Minnesota multiphasic personality inventory-2(MMPI-2): Manual for administration and scoring. Minneapolis: University of Mennesota Press.
Clark, M. E., Gironda, R. J., & Young, R. W. (2003). Detection of back random responding:Effectiveness of MMPI-2 and Personality Assessment Inventory validity indices. Psychological Assessment, 15, 223-234.
Carver, C. S., & Scheier, M. S. (1981). Attention and self-regulation: A control theory approach to human behavior. New York: Springer-Verlag.
Costa, P. T., & McCrae, R. R. (2008). The revised neo personality inventory (neo-pi-r). The SAGE handbook of personality theory and assessment, 2, 179-198.
Credé, M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70(4), 596-612.
Curran, P. G., Kotrba, L., & Denison, D. (2010). Careless responding in surveys: Applying traditional techniques to organizational settings. In 25th annual conference of Society for Industrial and Organizational Psychology, Atlanta, GA.
DeSimone, J. A., DeSimone, A. J., Harms, P. D., & Wood, D. (2018). The differential impacts of two forms of insufficient effort responding. Applied Psychology, 67(2), 309-338.
DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171-181.
Devine, P. G. (1989). Stereotypes and prejudice:Their automatic and controlled components. Journal of personality and social psychology, 56(1), 5-18.
Diener, E., & Wallbom, M. (1976). Effects of self-awareness on antinormative behavior. Journal of Research in Personality, 10, 107-111.
Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: the tailored design method. Hoboken:John Wiley & Sons.
Dunn, A. M., Heggestad, E. D., Shanock, L. R., & Theilgard, N. (2018). Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences. Journal of Business and Psychology, 33(1), 105-121.
Duval, S., Duval, V. H., & Neely, R. (1979). Self-focus, felt responsibility, and helping behavior. Journal of Personality and Social Psychology, 37(10), 1769-1778.
Duval, S., & Wicklund, R. A. (1972). A theory of objective self awareness. New York: Academic Press.
Ehlers, C., Greene-Shortridge, T. M., Weekley, J. A., & Zajack, M. D. (2009). The exploration of statistical methods in detecting random responding. In Annual Meeting of the Society for Industrial/Organizational Psychology, Atlanta, GA.
Gibson, B., & Zielaskowski, K. (2013). Subliminal priming of winning images prompts increased betting in slot machine play. Journal of Applied Social Psychology, 43, 106-115.
Gonzales, A. L., & Hancock, J. T. (2011). Mirror, mirror on my Facebook wall: Effects of exposure to Facebook on self-esteem. Cyberpsychology, Behavior, and Social Networking, 14(1-2), 79-83.
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213-224.
Govern, J. M., & Marsch, L. A. (2001). Development and validation of the situational self-awareness scale. Consciousness and Cognition, 10(3), 366-378.
Greene, R. L. (1978). An empirically derived MMPI carelessness scale. Journal of Clinical Psychology, 34(2), 407-410.
Guay, R. P. Oh, I.-S. Choi, D., Mitchell, M. S., Mount, M. K., & Shin, K. (2013). The interactive effect of conscientiousness and agreeableness on job performance dimensions in South Korea. International Journal of Selection and Assessment, 21, 233-238.
Haertzen, C. A., & Hill, H. E. (1963). Assessing subjective effects of drugs: An index of carelessness and confusion for use with the Addiction Research Center Inventory (ARCI). Journal of Clinical Psychology, 19(4), 407-412.
Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1, 104-121.
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99-114.
Huang, J. L., Liu, M., & Bowling, N. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828-845.
Ickes, W. J., Wicklund, R. A., & Ferris, C. B. (1973). Objective self awareness and self esteem. Journal of Experimental Social Psychology, 9(3), 202-219.
Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39(1), 103-129.
Kanfer, F. H. (1970). Self-regulation: Research, issues, and speculations. Behavior Modification in Clinical Psychology, 74, 178-220.
Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254-284.
Kreuter, F., Yan, T., & Tourangeau, R. (2008). Good item or bad-can latent class analysis tell?: the utility of latent class analysis for the evaluation of survey questions. Journal of the Royal Statistical Society: Series A (Statistics in Society), 171, 723-738.
Lawson, G., Stedmon, A. W., Zhang, K., Eubanks, D. L., & Frumkin, L. A. (2013). The effects of self-awareness on body movement indicators of the intention to deceive. Applied Ergonomics, 44(5), 687-693.
Liu, M., Bowling, N. A., Huang, J. L., & Kent, T. A. (2013). Insufficient effort responding to surveys as a threat to validity: The perceptions and practices of SIOP members. The Industrial-Organizational Psychologist, 51, 32-38.
Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61-83.
McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136(3), 450-470.
Mead, G. H. (1934). Mind, self and society (Vol. 111). Chicago: University of Chicago Press.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437-455.
Morey, L. C. (1991). Personality assessment inventory. Odessa, FL: Psychological Assessment Resources.
Mueller-Hanson, R., Heggestad, E. D., & Thornton III, G. C. (2003). Faking and selection:Considering the use of personality from select-in and select-out perspectives. Journal of Applied Psychology, 88, 348-355.
Niessen, A. S. M., Meijer, R. R., & Tendeiro, J. N. (2016). Detecting careless respondents in web-based questionnaires: Which method to use?. Journal of Research in Personality, 63, 1-11.
Northcraft, G. B., Schmidt, A. M., & Ashford, S. J. (2011). Feedback and the rationing of time and effort among competing tasks. Journal of Applied Psychology, 96(5), 1076-1086.
Ones, D. S., Viswesvaran, C., & Reiss, A. D. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81, 660-679.
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks:Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872.
Payne, B. K., Brown-Iannuzzi, J. L., & Loersch, C. (2016). Replicable effects of primes on human behavior. Journal of Experimental Psychology:General, 145, 1269-1279.
Perdue, C. W., & Gurtman, M. B. (1990). Evidence for the automaticity of ageism. Journal of Experimental Social Psychology, 26(3), 199-216.
Pinsoneault, T. B. (2005). Detecting random, partially random, and nonrandom Minnesota Multiphasic Personality Inventory-Adolescent Protocols. Psychological Assessment, 17(4), 476-480.
Pratto, F., & Bargh, J. A. (1991). Stereotyping based on apparently individuating information:Trait and global components of sex stereotypes under attention overload. Journal of Experimental Social Psychology, 27(1), 26-47.
R Development Core Team. (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org.
Schultz, D. P. (1969). The human subject in psychological research. Psychological Bulletin, 72, 214-228.
Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93-105.
Silvia, P. J., & Duval, T. S. (2001). Objective self-awareness theory: Recent progress and enduring problems. Personality and Social Psychology Review, 5, 230-241.
Silvia, P. J., & Eichstaedt, J. (2004). A self-novelty manipulation of self-focused attention for Internet and laboratory experiments. Behavior Research Methods, Instruments, & Computers, 36(2), 325-330.
Silvia, P. J., & Phillips, A. G. (2013). Self-awareness without awareness? Implicit self-focused attention and behavioral self-regulation. Self and Identity, 12, 114-127.
Spelke, E., Hirst, W., & Neisser, U. (1976). Skills of divided attention. Cognition, 4, 215-230.
Stone, J., Aronson, E., Crain, A. L., Winslow, M. P., & Fried, C. B. (1994). Inducing hypocrisy as a means of encouraging young adults to use condoms. Personality and Social Psychology Bulletin, 20(1), 116-128.
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. New York:Cambridge University Press.
Uleman, J. S., & Bargh, J. A. (Eds.). (1989). Unintended thought. New York: Guilford Press.
Ward, M. K., & Meade, A. W. (2018). Applying social psychology to prevent careless responding during online surveys. Applied Psychology, 67, 231-263.
Ward, M. K., & Pond III, S. B. (2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Computers in Human Behavior, 48, 554-568.
Weingarten, E., Chen, Q., McAdams, M., Yi, J., Hepler, J., & Albarracín, D. (2016). From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words. Psychological Bulletin, 142, 472-497.
Woods, C. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 186-191.