Factor structure and reliability of a testlet-based test.

When a test comprises of multiple testlets, it is very difficult to estimate factor structure and reliability of the data collected by the test. A testlet is defined by a scenario or passage and response alternatives following it. A testlet-based test suffers difficulties of estimating factor structure because the method factors are confounded with substance factors and of estimating reliability because the measurement errors are not independent and local independence is severely challenged. In the present study, we propose an approach of applying a series of confirmatory factor analysis to analyze the data prepared in the form of multitrait-multimethod data. The approach varies depending on whether the situation effect is fixed or random. In the specification search, we adopt correlated uniqueness model to avoid estimation problems when both trait factor(s) and method factor(s) exist in a model. The final model can be transformed to a correlated uniqueness model or a method factor model depending on whether situation effect is of random effect or fixed effect. In this process, we can estimate item reliability and in turn test reliability.

keywords: testlet, factor structure, reliability, situational judgment test, method effect, 소검사, 요인구조, 신뢰도, 상황판단검사, 방법효과, testlet, factor structure, reliability, situational judgment test, method effect

Reference

(1994) Additive and multiplicative method effects in applied psychological research An empirical assessment of three models Journal of Management,

(1959) Convergent and Discriminant Validation by Multitrait- Multimethod Matrix,

(1997) Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions. ,

(1996) Analysis and design of multitrait-multirater performance appraisal studies,

(1998) Estimation and uses of the proportion of method variance for multitrait- multimethod data,

(2001) Interactive LISREL: User's Guide., Lincolnwood

(1986) Estimating the reliability of multiple true-false tests,

(1976) The University of Chicago Press,

(1992) Validity generalization in the context of situational models Journal of Applied Psychology,

10.

(1989) A guide to the program and applications, SPSS Inc

11.

(1979) Correlation and Causality, Wiley

12.

(1992) Analysis of the Multitrait-Multimethod Matrix by Confirmatory Factor Analysis,

13.

(2000) Assessment center exercises represent cross-situational specificity, not method bias.,

14.

(1992) Specification of the criterion construct space:An application of hierarchical confirmatory factor analyses,

15.

(2000) Incorporating the testlet concept in test score analyses,

16.

(2002) A Three-Step Approach to Factor Analysis on Data of Multiple Testlets, Springer Verlag

17.

(2001) Dimension and Exercise Variance in Assessment Center Scores: A large-scale evaluation of multitrait- multimethod studies,

18.

(1980) Applications of item response theory to practical testing problems,

19.

(1997) The assessment center process:New directions,

20.

Marsh,H.W, (1989) Confirmatory Factor Analysis of Multitrait-Multitrait Data:Many Problems & A Few Solutions,

21.

(1991) Confirmatory factor analyses of multitrait-multimethod data comparison of the behavior of alternative models,

22.

(1992) Overcoming Problems in Confirmatory Factor Analyses of MTMM data,

23.

(April,2003.) Situational Judgment Tests, Knowledge, Behavioral Tendency. ,

24.

(2001) Use of Situational Judgment Tests to Predict Job Performance A Clarification of the Literature ,

25.

(1990) An Alternative Selection Procedure:The low-fidelity simulation,

26.

(2002) Noncognitive predictors of college student success,

27.

(2002) Hierarchical Linear Models: Applications and Data Analysis Methods(2nd Ed.), Sage.

28.

(1989187-195) The psychometric properties of managerial assessment centres Dimensions into exercises won't go Journal of Occupational Psychology,

29.

(1992) An exercise design approach to understanding assessment center dimension and exercise constructs,

30.

(1999) Using confirmatory factor analysis of correlated uniqueness to estimate method variance in Multitrait-Multimethod matrces,

31.

(1991) On the reliability of testlet-based tests Journal of Educational Measurement,

32.

(1980) Tests for comparing elements of a correlation matrix,

33.

(1989) Trace lines for testlets A use of multiple- categorical models Journal of Educational Measurement,

34.

(1987) Item clusters and computerized adaptive testings:A case for testlets,

35.

(1990) Toward a psychometrics for testlets,

36.

(2001) Users Guide for SCORIGHT A Computer Program for Scoring Tests Built of Testlets,

37.

(1999) Further Studies of Situational Tests,

38.

(1985) Hierarchically nested covariance structure models for multitrait-multimethod data,

39.

(1993) Scaling performance assessments:Strategies for managing local item dependence,

바로가기메뉴

Article Contents

Vol.23 No.1

Factor structure and reliability of a testlet-based test.

Abstract

Reference

Korean Journal of Psychology: General