ISSN : 1229-0696
본 연구는 평가센터를 적용한 신입사원(1,249) 선발자료와 재직자(관리자 105, 일반사원 200) 평가센터자료를 이용하여, 평가센터의 신뢰도 및 타당도를 검증하기 위해 실시되었다. 특히 본 연구의 평가센터는 평가과제마다 특정 평가차원이 지정되어 있는 소위, 내재설계된(nested designed) 평가센터로서, 구조화 면접에서는 3명의 평가자가 3개의 평가차원(책임감, 적극성, 혁신성)을, 프레젠테이션에서는 2명의 평가자가 2개의 평가차원(정보분석력, 문제해결력)을, 그룹토의에서는 2명의 평가자가 2개의 평가차원(대인친화력, 의사전달력)을 평가하도록 설계되어 있었다. 이러한 내재설계 평가센터는 평가자에 의한 편향(rater bias)이나 평가과제 효과(exercise effect)가 우려되기 때문에, 평가의 신뢰도 및 타당도 검증을 통해 평가센터의 활용가능성을 확인할 필요가 있다. 일반화가능도 결정연구(D study)결과, 내재설계 평가센터는 양호한 신뢰도를 보여주었으며, 일반화가능도 일반화연구(G study)결과, 평가자의 편향은 거의 나타나지 않았고, 평가과제 효과가 매우 크게 나타났으나 평가차원의 변별성 또한 어느 정도 확인할 수 있었다. 확인적 요인분석을 통한 구성개념타당도 검증결과에서는 평가과제와 평가차원을 모두 고려한 통합모형이 가장 적합한 것으로 나타나 평가차원의 변별성을 보여주었다. 재직자의 평가센터 점수와 준거와의 상관관계분석 결과에서는 각 평가과제에 내재되어 있는 평가차원들이 어느 정도 차별적인 준거예측력을 보여주었으며, 전반적으로 구조화 면접에 속한 평가차원이 가장 포괄적인 준거관련성을 지녔고, 프레젠테이션에 속한 평가차원이 준거와 가장 관련성이 적었다. 논의에서는 본 연구의 의의와 제한점을 제시하였다.
An Assessment Center(AC) consists of a standardized evaluation of behavior based on multiple evaluations including: job-related simulations, interviews and psychological tests. Recently, as many companies have been adopted AC as a selection tool, the importance of AC have been rapidly increased. The general frame work of AC used is called “nested model”. That is, selecting a competence for assessment identifies exercises. However, there have been little researches conducted on the validity of nested AC. Therefore, this study is intended to the validity of nested AC using generalizability theory with G and D study and confirmatory factory analysis. In this study, we use two types of data: AC scores of applicants(n=1,249) and AC scores of incumbents(team manager=105, team member=200). The AC of this study is designed of Structured Interview(SI), which three competencies(responsibility, activity, innovation) are rated by three assessors and Presentation(PT) and Group Discussion(GD), which two competencies(information processing, problem solving / harmony, communication skill) are rated respectively by two assessors. In the result, generalizability analyses indicated that the reliabilities of three exercise were acceptable and exercise effects existed, on the other hand, dimension effects were found especially in SI. But, rater effect was almost not existed. Confirmatory factor analyses results were consistent with the generalizability results. That is, the dimensions of AC could be discriminated, and showed a differential predictive validity. Implications for future research and practice are discussed.
강애남, 이규민 (2006). 학생들의 동료평가를 활용한 수행평가 결과의 일반화가능도 분석. 교육평가연구, 19, 107-121.
김성숙, 김양분 (2001). 일반화가능도 이론. 서울: 교육과학사.
남명호 (1996). 수행평가에 있어서 일반화가능도 이론의 활용. 교육평가연구, 9, 73-93.
남명호 (2002). 수행평가: 기술적 측면. 서울: 교육과학사.
성태제 (2002). 현대교육평가. 서울: 학지사.
이규민 (2003). 단위검사 개념의 적용: 일반화가능도 이론을 중심으로. 교육평가연구, 16, 53-70.
이규환 (2008). 인재선발과 AC(평가센터)기법. 한경비지니스. 2008년 4월 11일.
이영식, 신상근 (2004). 다변량 일반화가능도 이론에 의한 말하기 시험의 타당도와 신뢰도에 관한 연구. Foreign Languages Education, 11, 249-265.
임대열 (2004). Assessment Center 프로그램의 개발 및 운영. 한국 산업 및 조직심리학회 추계학술대회 발표집(p.340).
조재윤 (2009). 일반화가능도 이론을 이용한 쓰기 평가의 오차원 분석 및 신뢰도 추정 연구. 국어교육, 128, 325-357.
지은림 (1999). 사회과 보고서 수행평가를 위한 총체적 채점과 분석적 채점의 비교. 교육평가연구, 12, 11-24.
지은림, 김성숙 (2005). 초등학교 수행평가의 교육적 효과와 활용 방식. 교육평가연구, 12, 173-191.
Arthur Jr., W., Woehr, D. J., & Maldegen, R. (2000). Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical re-examination of the assessment center construct-related validity paradox. Journal of Management, 26, 813-835.
Borman, W. C. (1982). Validity of behavioral assessment for predicting recruiter performance. Journal of Applied Psychology, 67, 3-9.
Bowler, M. C., & Woehr, D. J. (2006). A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology, 91, 1114-1124.
Bowler, M. C., & Woehr, D. J. (2008). Evaluating assessment center construct-related validity via variance partitioning. In B. J. Hoffman (Chair), Reexamining Assessment Centers: Alternate Approaches. Paper presented at the 23rd annual meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.
Bowler, M. C., & Woehr, D. J. (2009). Assessment center construct related validity: Stepping beyond the MTMM matrix. Journal of Vocational Behavior, 75, 173-182.
Bray, D. W., Campbell, R. J. (1968). Selection of salesmen by means of an assessment center. Journal of Applied Psychology, 52, 36-41.
Brennan, R. L. (1992). Elements of generalizability theory. Iowa City, IA: American College Testing.
Brennan, R. L. (2001a). Generalizability theory. New York: Springer.
Byham, W. C. (1970). Assessment center for spotting future managers. Harvard Business Review, 48, 150-160, plus appendix.
Cascio, W. F. (2002). Changes in workers, work, and organizations. In W. C. Borman, D. R. Ilgen, & R. J. Klimoski (Eds.), Comprehensive handbook of psychology, 12 : Industrial and organizational psychology (pp. 107-130). New York: Wiley.
Cohen, B. M., Moses, J. L., & Byham, W. C. (1974). The validity of assessment centers: A literature review. Monograph II. Pittsburgh, PA: Development Dimensions Press.
Crick, J. E. & Brennan, R. L. (1983). Manual for GENOVA: A generalized analysis of variance system. ACT Technical Bulletin, 43, The American College Testing Program, IA.
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163.
Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., III, & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 74, 493-511.
Gibbons, A. M., Rupp, D. E., Baldwin, A., & Holub, S. A. (2005, April). Developmental assessment center validation: Evidence for DACs as effective training interventions. Paper presented at the 20th annual meeting of the Society for Industrial and Organizational Psychology, Los Angeles, CA.
Hardison, C. M., & Sackett, P. R. (2004). Assessment center criterion-related validity: A meta-analysis update. Paper presented at the 19th annual conference of the Society Industrial and Organizational Psychology, Chicago, Ill.
Howard, A. (1995). The Changing nature of work. San Francisco, CA: Jossey-Bass.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.
Jackson, D. J. R., Stillman, J. A., & Atkins, S. G. (2005). Rating tasks versus dimensions in assessment centers: A psychometric comparison. Human Performance, 18, 213-241.
International Task Force on Assessment Center Guidelines (2009). Guidelines and Ethical Considerations for Assessment Center Operations. International Journal of Selection and Assessment, 17, 233-253.
Lance, C. E., Lambert, T. A., Gewin, A. G., Lievens, F., & Conway, J. M. (2004). Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. Journal of Applied Psychology, 89, 377–.385.
Lievens, F. & Conway, J. M. (2001). Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait -multimethod studies. Journal of Applied Psychology, 86, 1202-1222.
Schmitt, N., Gooding, R. Z., Noe, R. A., & Kirsch, M. (1984). Meta-analyses of validity studies published between 1964 and 1982 and the investingation of study characteristics. Personnel Psychology, 37, 407-422.
Thornton, C. G., III, Byham, W. C. (1982). Assessment centers and managerial performance. New York: Academic Press.
Thornton, G. C., III, & Rupp, D. E. (2003). Simulations and assessment centers. In J. C. Thomas (Ed.), & M. Hersen (Series Ed.), Comprehensive handbook of psychological assessment, Vol. 4: Industrial and organizational assessment (pp. 318-344). New York:
Thornton, G. C., III, & Rupp, D. E. (2006). Assessment centers in human resource management: Strategies for prediction, diagnosis, and development. Mahwah, NJ: Erlbaum.
