Factor structure and reliability of a testlet-based test.

이 연구는 다수의 소검사로 이루어진 검사에서의 요인구조 추출 및 신뢰도에 대한 것이다. 여기서 소검사라고 하면 검사제작, 시행, 또는 채점에서 하나의 단위로 취급되는 문항군으로서 이 글에서는 제작단계에 중점을 둔다. 제작시 형태로 보면, 모의(시뮬레이션)과제, 시나리오 또는 큰 문단이 있고 그 아래 한 묶음의 채점항목 또는 문항군이 있는 형식을 의미한다. 이러한 검사의 자료에서는 각 소검사내의 국지적 관련성이 높아지고, 검사 전체로서 측정오차의 독립성 및 국지적 독립성(local independence)이 위배되어 요인분석의 실시 및 신뢰도 산출에 어려움이 있다. 이러한 검사 자료는 과제나 시나리오가 고정효과인지 무선효과인지에 따라 요인해석도 다르고 신뢰도의 정의도 다르다. 요인구조의 추출을 위해서는 제작시 무엇을 잴 것인지의 설계에 기초한 확인적 요인분석이 유용하며 신뢰도의 산출을 위해서는 고전검사이론에서 가정하는 오차의 독립성이 위반되는 것을 허용하며, 나아가 문항간 국지적 의존성까지 감안하는 신뢰도 산출 방식이 필요하다.

keywords: testlet, factor structure, reliability, situational judgment test, method effect, 소검사, 요인구조, 신뢰도, 상황판단검사, 방법효과, testlet, factor structure, reliability, situational judgment test, method effect

Abstract

When a test comprises of multiple testlets, it is very difficult to estimate factor structure and reliability of the data collected by the test. A testlet is defined by a scenario or passage and response alternatives following it. A testlet-based test suffers difficulties of estimating factor structure because the method factors are confounded with substance factors and of estimating reliability because the measurement errors are not independent and local independence is severely challenged. In the present study, we propose an approach of applying a series of confirmatory factor analysis to analyze the data prepared in the form of multitrait-multimethod data. The approach varies depending on whether the situation effect is fixed or random. In the specification search, we adopt correlated uniqueness model to avoid estimation problems when both trait factor(s) and method factor(s) exist in a model. The final model can be transformed to a correlated uniqueness model or a method factor model depending on whether situation effect is of random effect or fixed effect. In this process, we can estimate item reliability and in turn test reliability.

keywords: testlet, factor structure, reliability, situational judgment test, method effect, 소검사, 요인구조, 신뢰도, 상황판단검사, 방법효과, testlet, factor structure, reliability, situational judgment test, method effect

참고문헌

(1994) Additive and multiplicative method effects in applied psychological research An empirical assessment of three models Journal of Management,

(1959) Convergent and Discriminant Validation by Multitrait- Multimethod Matrix,

(1997) Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions. ,

(1996) Analysis and design of multitrait-multirater performance appraisal studies,

(1998) Estimation and uses of the proportion of method variance for multitrait- multimethod data,

(2001) Interactive LISREL: User's Guide., Lincolnwood

(1986) Estimating the reliability of multiple true-false tests,

(1976) The University of Chicago Press,

(1992) Validity generalization in the context of situational models Journal of Applied Psychology,

10.

(1989) A guide to the program and applications, SPSS Inc

11.

(1979) Correlation and Causality, Wiley

12.

(1992) Analysis of the Multitrait-Multimethod Matrix by Confirmatory Factor Analysis,

13.

(2000) Assessment center exercises represent cross-situational specificity, not method bias.,

14.

(1992) Specification of the criterion construct space:An application of hierarchical confirmatory factor analyses,

15.

(2000) Incorporating the testlet concept in test score analyses,

16.

(2002) A Three-Step Approach to Factor Analysis on Data of Multiple Testlets, Springer Verlag

17.

(2001) Dimension and Exercise Variance in Assessment Center Scores: A large-scale evaluation of multitrait- multimethod studies,

18.

(1980) Applications of item response theory to practical testing problems,

19.

(1997) The assessment center process:New directions,

20.

Marsh,H.W, (1989) Confirmatory Factor Analysis of Multitrait-Multitrait Data:Many Problems & A Few Solutions,

21.

(1991) Confirmatory factor analyses of multitrait-multimethod data comparison of the behavior of alternative models,

22.

(1992) Overcoming Problems in Confirmatory Factor Analyses of MTMM data,

23.

(April,2003.) Situational Judgment Tests, Knowledge, Behavioral Tendency. ,

24.

(2001) Use of Situational Judgment Tests to Predict Job Performance A Clarification of the Literature ,

25.

(1990) An Alternative Selection Procedure:The low-fidelity simulation,

26.

(2002) Noncognitive predictors of college student success,

27.

(2002) Hierarchical Linear Models: Applications and Data Analysis Methods(2nd Ed.), Sage.

28.

(1989187-195) The psychometric properties of managerial assessment centres Dimensions into exercises won't go Journal of Occupational Psychology,

29.

(1992) An exercise design approach to understanding assessment center dimension and exercise constructs,

30.

(1999) Using confirmatory factor analysis of correlated uniqueness to estimate method variance in Multitrait-Multimethod matrces,

31.

(1991) On the reliability of testlet-based tests Journal of Educational Measurement,

32.

(1980) Tests for comparing elements of a correlation matrix,

33.

(1989) Trace lines for testlets A use of multiple- categorical models Journal of Educational Measurement,

34.

(1987) Item clusters and computerized adaptive testings:A case for testlets,

35.

(1990) Toward a psychometrics for testlets,

36.

(2001) Users Guide for SCORIGHT A Computer Program for Scoring Tests Built of Testlets,

37.

(1999) Further Studies of Situational Tests,

38.

(1985) Hierarchically nested covariance structure models for multitrait-multimethod data,

39.

(1993) Scaling performance assessments:Strategies for managing local item dependence,

바로가기메뉴

논문 상세

Vol.23 No.1

소검사(testlet)로 구성된 검사에서의 요인구조와 신뢰도

Factor structure and reliability of a testlet-based test.

초록

Abstract

참고문헌

한국심리학회지: 일반