ISSN : 1229-0696
본 연구는 대입선발을 목적으로 개발된 평가센터를 이용하여, 평가센터의 신뢰도와 타당도를 확인함으로써 대학장면에서의 평가센터 활용성을 검증하였다. 연구에 이용된 자료는 B 대학의 입학사정관제 모의전형에 응시한 지원자 60명에 대한 평가자 30명의 평가자료이다. 평가자 3인이 한 팀이 되어, 평가팀 별로 6명의 지원자에 대해 3가지 평가과제(구조화 면접, 프레젠테이션, 그룹토의)를 진행했다. 결정연구(D study)를 이용한 평가센터의 신뢰도는 양호하게 나타나 현재의 평가설계가 적절함을 보여주었다. 본 평가센터의 타당도 측면에서 재학생 대상 설문조사 결과 본 평가센터에서 평가하는 역량이 대학에서의 학업수행과 적응에 중요한 역량이라고 답하여 내용타당성을 보여주었으며, 평가센터 참가자 대상 설문조사 결과 평가의 공정성과 평가의 적절성 및 참여동기 측면에서 본 평가센터의 결과타당도의 일면을 보여주었다. 일반화 연구(G study)를 통해 각 평가과제에서 평가차원(21.1%)과 평가자(10.2%)의 영향을 비교한 결과 평가차원의 영향이 상대적으로 높게 나타났으며, MTMM 분석결과는 MTHM(.51)은 5개 상관 모두 유의했고, HTMM(.55)은 8개 중 7개 상관이 유의했고, HTHM(.36)은 23개 중 8개 상관만이 유의하게 나타나, 평가센터에 평가차원 효과(dimension effect)와 평가과제 효과(exercise effect)가 모두 존재함을 알 수 있었다. 또한 평가센터 점수와 참가자의 여러 관련 변인들의 상관관계를 분석한 결과, 학업역량 중 자기주도적 학습은 지원자의 국어 및 영어 내신성적과 상관을 보여주었으며, 토론능력은 대중불안과 부적상관을 보여주었다. 인성역량인 도전정신은 지원자의 외향성, 자기효능감, 교외수상 횟수와 정적상관, 사회불안의 하위항목인 ‘낯선 것에 대한 두려움’과는 부적상관을 보여주었고, 창의성은 외향성과 정적상관, 적응력은 정서적 안정성과 정적상관을 보여주었다. 그룹토의에서 평가한 대인관계 역량은 수행불안 및 대인불안과 부적상관을 자기효능감과 정적상관을 보여주었다. 논의에서는 본 연구의 제한점과 향후연구 방향을 제시하였다.
This study were preformed for verifying the applicability of Assessment Center(AC) for selection of university student. First, the results of decision study of generalizability theory(D study) has shown acceptable reliabilities. So, we could think that this AC was designed properly. Second, the enrolled students of the university have said that the competencies rated in this AC were important for performing study and adapting to school. And participants in this AC reported they have felt fairness and they could have done their best. It means that this AC had validity. Third, generalizability study(G study) has shown dimension effect(21.1%) was higher than rator effect(10.2%). And, in the MTMM analysis, it were found both dimension effect and exercise effect. Forth, in relation analysis between AC ratings and the various records of participants, ‘Self-led Study’ related with ‘Records of Language’ positively, ‘Discussion Skill’ related with ‘Public Anxiety’ negatively, ‘Challenge’ related with ‘Extraversion, Self Efficacy, and Record of Award off campus’ positively and with ‘Anxiety to unfamiliar’ negatively, ‘Creativity’ related with ‘Extraversion’ positively, ‘Adapting to Change’ related with ‘Emotional Stability’ positively, and ‘Interpersonal Competencies’ related with ‘Performance Anxiety and Public Anxiety’ negatively. In short, this AC has shown applicability as selection tool for university student. Finally, the implications and limitations were discussed.
교육과학기술부 (2007.6.14). 입학사정관제 지원계획 보도자료.
남명호 (1996). 수행평가에 있어서 일반화가능도 이론의 활용. 교육평가연구, 9(2), 73- 93.
남명호 (2002). 수행평가: 기술적 측면. 서울: 교육과학사.
문혜신, 오경자 (2002). 한국판 아동․청소년용 사회 불안 척도의 타당화 연구. 한국심리학회지: 임상, 21, 429-443.
유현주 (2009). 역할에 기초한 입학사정관 전문성 훈련 프로그램. 한국교육논단, 8, 131 -153.
이영식, 신상근 (2004). 다변량 일반화가능도 이론에 의한 말하기 시험의 타당도와 신뢰도에 관한 연구. Foreign Languages Education, 11, 249-265.
조재윤 (2009). 일반화가능도 이론을 이용한 쓰기 평가의 오차원 분석 및 신뢰도 추정 연구. 국어교육, 128, 325-357.
한국대학교육협의회 (2008). 대학입학사정관제 지원사업 실행계획.
한국대학교육협의회 (2009. 5. 4). 보도자료: 공정성 및 신뢰성 확보노력으로 입학사정관제 정착에 주력.
허창구, 신강현 (2010). 내재설계 평가센터의 신뢰도 및 타당도. 한국심리학회지: 산업 및 조직, 23, 225-249.
Bartels, L. K., Bommer, W. H., & Rubin, R. S. (2000). Student performance: assessment centers versus traditional classroom evaluation techniques. Journal of Education for business, 75, 198-201.
Bauer, T. N., Maertz, C. P., Dolen, M. R., & Campion, M. A. (1998). A longitudinal assessment of applicant reactions to an employment test. Journal of Applied Psychology, 83, 892-903.
Binning, J. F., & Barrett, G. V. (1989). Validity of personnel decisions: An examination of the inferential and evidential bases. Journal of Applied Psychology, 74, 478-494.
Borman, W. C. (1982). Validity of behavioral assessment for predicting recruiter performance. Journal of Applied Psychology, 67, 3-9.
Bowler, M. C., & Woehr, D. J. (2006). A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology, 91, 1114-1124.
Bowler, M. C., & Woehr, D. J. (2008). Evaluating assessment center construct-related validity via variance partitioning. In B. J. Hoffman (Chair), Reexamining Assessment Centers: Alternate Approaches. Paper presented at the 23rd annual meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.
Bray, D. W., Campbell, R. J. (1968). Selection of salesmen by means of an assessment center. Journal of Applied Psychology, 52, 36-41.
Byham, W. C. (1970). Assessment center for spotting future managers. Harvard Business Review, 48, 150-160, plus appendix.
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16. 137-163.
Cohen, B. M., Moses, J. L., & Byham, W. C. (1974). The validity of assessment centers: A literature review. Monograph Ⅱ. Pittsburgh, PA: Development Dimensions Press.
Donahue, L. M., Truxillo, D. M., Cornwell, J. M., & Gerrity, M. J. (1997). Assessment center construct validity and behavioral checklists: Some additional findings. Journal of Social Behavior and Personality, 12, 85-108.
Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006). The Mini-IPIP Scales: Tiny-Yet-Effective Measures of the Big Five Factors of Personality. Psychological Assesment. 18, 2, 192-203.
Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., Ⅲ, & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 74, 493-511.
Gibbons, A. M., Rupp, D. E., Baldwin, A., & Holub, S. A. (2005). Developmental assessment center validation: Evidence for DACs as effective training interventions. Paper presented at the 20th annual meeting of the Society for Industrial and Organizational Psychology, Los Angeles, CA.
Howard, A. (1997). A reassessment of assessment centers: Challenges for the 21st century. Journal of Social Behavior and Personality, 12, 13-52.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.
International Task Force on Assessment Center Guidelines (2000). Guidelines and ethical considerations for assessment center operations. Public Personnel Management, 29, 315-331.
Jackson, D, J., Stillman, J. A., & Englert, P. (2010). Task-Based Assessment Centers: Empirical support for a systems model. International Journal of Selection and Assessment, 18, 141-154.
Kottke, J. L., & Shultz, K. S. (1997). Using an assessment center as a developmental tool for graduate students: A demonstration. In R. E. Riggio & B. T. Mayes(Eds.), Perspectives on assessment centers[Special Issue]. Journal of Social Behavior and Personality, 12, 289-302.
Kudisch, J. D., Ladd, R. T., & Dobbins, G. H. (1997). New evidence on the construct validity of diagnostic assessment centers: The findings may not be so troubling after all. Journal of Social Behavior and Personality, 12, 129-144.
Lance, C. E., Foster, M. R., Thoresen, J. D., & Gentry, W. A. (2004). Assessor cognitive processes in an operational assessment center. Journal of Applied Psychology, 89, 22-35.
Lievens, F. & Conway, J. M. (2001). Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait -multimethod studies. Journal of Applied Psychology. 86, 1202-1222.
Loacker, B. (1991). Designing a national assessment system: Alveno's perspective. National Center for Education Statistics.
Louiselle, K. G. (1980). Confirmatory factor analysis of two assessment center rating procedures. Paper presented at the 17th Annual IO/BO Graduate Student Conference, Minneapolis, MN.
Luthans, F., Avolio, B., Avey, J. B. & Norman, S. M. (2006). Psychological capital: Measurement and relationship with performance and job satisfaction (Working Paper No. 2006-I). Gallup Leadership institute, University of Nebraska _Lincoln.
Mullin, R. F., Shaffer, P. L., & Grelle, M. J. (1991). A study of the assessment center method of teaching basic management skills. In J. D. Bigelow(Ed.), Managerial Skills: Explorations in practical knowledge, 116-153. Newbury Park, CA: Sage.
Riggio, R. E, Aguirre, Monica, & Mayes, Bronston. (1997). The use of assessment center methods for student outcome assessment. Journal of Social Behavior and Personality, 12, 237-288.
Sackett, P. R., & Tuzinski, K. (2001). The role of dimensions and exercises in assessment center judgments. In M. London (Ed.), How people evaluate others in organizations (pp.111-129). Mahwah, NJ: Lawrence Erlbaum Associates.
Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame for frame-of reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87, 735-346.
Schmitt, N., Gooding, R. Z., Noe, R. A., & Kirsch, M. (1984). Meta-analyses of validity studies published between 1964 and 1982 and the investingation of study characteristics. Personnel Psychology, 37, 407-422.
Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149-155.
Spencer, L. M. Jr. and Spencer, S. M. (1993). Competence at work: Models for superior performance. New York: John Wiley & Sons.
Thornton, G. C., III, & Rupp, D. E. (2006). Assessment centers in human resource management: Strategies for prediction, diagnosis, and development. Mahwah, NJ: Erlbaum.
