Measurement Invariance Testing Using Random Effects model for Many groups: Multilevel Confirmatory Factor Analysis (ML CFA) and Multilevel Factor Mixture Modeling (ML FMM)

& (2019). Measurement Invariance Testing Using Random Effects model for Many groups: Multilevel Confirmatory Factor Analysis (ML CFA) and Multilevel Factor Mixture Modeling (ML FMM). Korean Journal of Psychology: General, 38(2), 185-218, https://doi.org/10.22257/kjp.2019.6.38.2.185

copy

Abstract

In multi-group analysis, measurement invariance is a requirement for meaningful comparisons between groups. Multi-group confirmatory factor analysis (MG CFA) has been widely used for group comparisons. However, MG CFA is appropriate for comparisons with a small number of groups and is limited for a large number of groups, in particular, in cross-cultural comparative studies. To overcome the limitation of MG CFA, this study described alternative approaches: multilevel confirmatory factor analysis (ML CFA) and multilevel factor mixture modeling (ML FMM), which are effective for comparing more than 10 groups. In ML CFA, its advantages and disadvantages were described by introducing two models: random intercept models that estimate only intercepts as random effects and random intercept and loading models that estimate intercepts and factor loadings as random effects. Specifically, this study presented theoretical models for the two methods and procedures for testing measurement invariances. In addition, this study discussed advantages of ML CFA, relative to those of MG CFA, and several points that should be considered when applying ML CFA. And, as an example of applying ML CFA, this study conducted latent means analysis across countries for instrumental motivation of science and enjoyment perceived by students using the PISA 2015 data. Finally, implications of this study and future research directions were discussed.

keywords: 국가비교, 측정동일성, 잠재평균 비교, 다층 확인적요인분석, 다층 요인혼합모형, many group comparison, measurement invariance, latent means comparison, multiple group analysis, multilevel confirmatory analysis, multilevel factor mixture modeling

Reference

박찬호 (2017). A Comparative Study on TIMSS Mathematics Assessment of Korea, Japan, and the USA: A Differential Item Functioning Approach. 비교교육연구, 27, 1-19.

이종희, 김기연, 김수진 (2011). 수학 학업성취도가 높은 국가의 수학-정의적 영역 요인 분석 및 측정 동일성 검증. 학교수학, 13(2), 307-321.

홍세희 (2000). 구조 방정식 모형의 적합도 지수 선정기준과 그 근거. 한국심리학회지:임상, 19(1), 161-177.

홍세희 (2015). 위계적자료 분석을 위한 횡단 다층모형. 서울: 에스앤엠리서치그룹.

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716-723.

Allua, S., Stapleton, L. M., & Beretvas, S. N. (2008). Testing latent mean differences between observed and unobserved groups using multilevel factor mixture models. Educational and Psychological Measurement, 68(3), 357-378.

Asparouhov, T., & Muthén, B. (2012, July). General random effect latent variable modeling: Random subjects, items, contexts, and parameters. In annual meeting of the National Council on Measurement in Education, Vancouver, British Columbia.

Asparouhov, T., & Muthén, B. (2014). Multiple-group factor analysis alignment. Structural Equation Modeling: A Multidisciplinary Journal, 21(4), 495-508.

Barcikowski, R. S. (1981). Statistical power with group mean as the unit of analysis. Journal of Educational and Behavioral Statistics, 6(3), 267-285.

10.

Bogler, R., & Nir, A. E. (2012). The importance of teachers' perceived organizational support to job satisfaction: What's empowerment got to do with it?. Journal of Educational Administration, 50(3), 287-306.

11.

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Newbury Park, CA:Sage.

12.

Buzick, H. M. (2010). Testing for heterogeneous factor loadings using mixtures of confirmatory factor analysis models. Frontiers in psychology, 1, 1-9.

13.

Byrne, B. M., & Van de Vijver, F. J. (2010). Testing for measurement and structural equivalence in large-scale cross-cultural studies:Addressing the issue of nonequivalence. International Journal of Testing, 10(2), 107-132.

14.

Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14(3), 464-504.

15.

Cheung, G. W., & Rensvold, R. B. (2000). Assessing extreme and acquiescence response sets in cross-cultural research using structural equations modeling. Journal of Cross-Cultural Psychology, 31, 187-212.

16.

Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233-255.

17.

Chopik, W. J., O’Brien, E., & Konrath, S. H. (2017). Differences in empathic concern and perspective taking across 63 countries. Journal of Cross-Cultural Psychology, 48(1), 23-38.

18.

Da Costa, L. P., & Dias, J. G. (2014). Perceptions of poverty attributions in Europe: A multilevel mixture model approach. Quality & Quantity, 48(3), 1409-1419.

19.

Davidov, E., Dülmer, H., Schlüter, E., Schmidt, P., & Meuleman, B. (2012). Using a multilevel structural equation modeling approach to explain cross-cultural measurement noninvariance. Journal of Cross-Cultural Psychology, 43(4), 558-575.

20.

De Boeck, P. (2008). Random Item IRT Models. Psychometrika, 73, 533-559.

21.

De Jong, M. G., & Steenkamp, J. B. E. (2010). Finite mixture multilevel multidimensional ordinal IRT models for large scale cross-cultural research. Psychometrika, 75(1), 3-32.

22.

Frederickx, S., F. Tuerlinckx, P. De Boeck, and D. Magis. (2010). RIM: A Random Item Mixture Model to Detect Differential Item Functioning. Journal of Educational Measurement, 47, 432-457.

23.

Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72-91.

24.

Horn, J. L., & McArdle, J. J. (1992). A practical guide to measurement invariance in research on aging. Experimental Aging Research, 18, 117-144.

25.

Hox, J. J., & Maas, C. J. (2001). The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples. Structural equation modeling, 8(2), 157-174.

26.

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis:Conventional criteria versus new alternatives. Structural Equation Modeling: a Multidisciplinary Journal, 6(1), 1-55.

27.

Jak, S., Oort, F. J., & Dolan, C. V. (2013). A test for cluster bias: Detecting violations of measurement invariance across clusters in multilevel data. Structural Equation Modeling: A Multidisciplinary Journal, 20(2), 265-282.

28.

Jak, S., Oort, F. J., & Dolan, C. V. (2014). Measurement bias in multilevel data. Structural Equation Modeling: A Multidisciplinary Journal, 21(1), 31-39.

29.

Jang, S., Kim, E. S., Cao, C., Allen, T. D., Cooper, C. L., Lapierre, L. M., ..., & Abarca, N. (2017). Measurement invariance of the satisfaction with life scale across 26 countries. Journal of Cross-Cultural Psychology, 48(4), 560-576.

30.

Jöreskog, K. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36(2), 109-133.

31.

Jöreskog, K. G., & Goldberger, A. S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of the American Statistical Association, 70(351a), 631-639.

32.

Kaplan, D., Kim, J.-S., & Kim, S.-Y. (2009). Multilevel latent variable modeling: Current research and recent developments. In R. Millsap & A. Maydeu-Olivares (Eds.), Sage handbook of quantitative methods in psychology (pp. 592-612). Thousand Oaks, CA: Sage.

33.

Kim, E. S., & Cao, C. (2015). Testing group mean differences of latent variables in multilevel data using multiple-group multilevel CFA and multilevel MIMIC modeling. Multivariate behavioral research, 50(4), 436-456.

34.

Kim, E. S., & Wang, Y. (2018). Investigating Sources of Heterogeneity with Three-Step Multilevel Factor Mixture Modeling: Beyond Testing Measurement Invariance in Cross-National Studies. Structural Equation Modeling: A Multidisciplinary Journal, 1-17.

35.

Kim, E. S., Cao, C., Wang, Y., & Nguyen, D. T. (2017). Measurement invariance testing with many groups: A comparison of five approaches. Structural Equation Modeling: A Multidisciplinary Journal, 24(4), 524-544.

36.

Kim, E. S., Joo, S. H., Lee, P., Wang, Y., & Stark, S. (2016). Measurement invariance testing across between-level latent classes using multilevel factor mixture modeling. Structural Equation Modeling: A Multidisciplinary Journal, 23(6), 870-887.

37.

Kim, E. S., Yoon, M., Wen, Y., Luo, W., & Kwok, O. M. (2015). Within-level group factorial invariance with multilevel data:multilevel factor mixture and multilevel mimic models. Structural Equation Modeling: A Multidisciplinary Journal, 22(4), 603-616.

38.

Kreft, I. G. G., & De Leeuw, J. (1998). Introducing multilevel modeling. Newbury Park, CA: Sage.

39.

Little, T. D. (1997). Mean and covariance structures (MACS) analyses of cross-cultural data: Practical and theoretical issues. Multivariate Behavioral Research, 32(1), 53-76.

40.

Little, T. D. (2000). On the comparability of constructs in cross-cultural research: A critique of Cheung and Rensvold. Journal of Cross-cultural Psychology, 31(2), 213-219.

41.

Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal mixture. Biometrika, 88, 767-778.

42.

Lubke, G. H., & Muthén, B. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10(1), 21-39.

43.

Maas, C. J., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1(3), 86-92.

44.

Marsh, H. W., & Hau, K. T. (2003). Big-Fish--Little-Pond effect on academic self-concept: A cross-cultural (26-country) test of the negative effects of academically selective schools. American psychologist, 58(5), 364.

45.

McLachlan, G., & Peel, D. (2000). Finite mixture models. Hoboken, NJ: Wiley.

46.

Mehta, P. D., & Neale, M. C. (2005). People are variables too: Multilevel structural equations modeling. Psychological methods, 10(3), 259-284.

47.

Mellenbergh, G. J. (1989). Item bias and item response theory. International Journal of Educational Research, 13(2), 127-143.

48.

Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525-543.

49.

Meredith, W., & Millsap, R. E. (1992). On the misuse of manifest variables in the detection of measurement bias. Psychometrika, 57(2), 289-311.

50.

Millsap, R. E. (1997). Invariance in measurement and prediction: Their relationship in the single-factor case. Psychological Methods, 2(3), 248-260.

51.

Millsap, R. E., & Everson, H. (1991). Confirmatory measurement model comparisons using latent means. Multivariate Behavioral Research, 26(3), 479-497.

52.

Muthén, B. O. (1994). Multilevel covariance structure analysis. Sociological methods &Research, 22(3), 376-398.

53.

Muthén, B. O., and Asparouhov, T. (2013). BSEM Measurement Invariance Analysis. Mplus Web Notes: No. 17. Available online at:www.statmodel.com

54.

Muthén, B., & Asparouhov, T. (2018). Recent methods for the study of measurement invariance with many groups: alignment and random effects. Sociological Methods & Research, 47(4), 637-664.

55.

Peugh, J. L. (2010). A practical guide to multilevel modeling. Journal of school psychology, 48(1), 85-112.

56.

Ramaswamy, V., Desarbo, W. S., Reibstein, D. J., & Robinson, W. T. (1993). An empirical pooling approach for estimating marketing mix elasticities with PIMS data. Marketing Science, 12(1), 103-124.

57.

Redner, R. A., & Walker, H. F. (1984). Mixture densities, maximum likelihood and the EM algorithm. SIAM Review, 26, 195-239.

58.

Rutkowski, L., & Svetina, D. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74(1), 31-57.

59.

Satorra, A., & Bentler, P. M. (2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66, 507-514.

60.

Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461-464.

61.

Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333-343.

62.

Selig, J. P., Card, N. A., & Little, T. D. (2008). Latent variable structural equation modelling in crosscultural research: Multigroup and multilevel approaches. In F. J. R. van de Vijver, D. A., van Hemert, & Y. H. Poortinga (Eds.), Multilevel analysis of individuals and cultures (pp. 93-119). Mahwah, NJ: Erlbaum.

63.

Stapleton, L. M., Yang, J. S., & Hancock, G. R. (2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481-520.

64.

Steenkamp, J. B. E., & Baumgartner, H. (1998). Assessing measurement invariance in cross-national consumer research. Journal of Consumer Research, 25(1), 78-90.

65.

Tay, L., Woo, S. E., & Vermunt, J. K. (2014). A conceptual and methodological framework for psychometric isomorphism: Validation of multilevel construct measures. Organizational Research Methods, 17(1), 77-106.

66.

Van de Vijver, F. J., & Poortinga, Y. H. (2002). Structural equivalence in multilevel research. Journal of Cross-Cultural Psychology, 33(2), 141-156.

67.

Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4-70.

68.

Verhagen, A. J. and J. P. Fox. (2013). Bayesian Tests of Measurement Invariance. The British Journal of Mathematical and Statistical Psychology, 66, 383-401.

69.

Vermunt, J. K. (2007, August). Multilevel mixture item response theory models: An application in education testing. Paper presented at the 56th Session of the International Statistical Institute, Lisbon, Portugal.

70.

Yoon, M., & Millsap, R. E. (2007). Detecting violations of factorial invariance using databased specification searches: A Monte Carlo study. Structural Equation Modeling, 14(3), 435-463.

71.

Żemojtel‐Piotrowska, M., Piotrowski, J. P., Osin, E. N., Cieciuch, J., Adams, B. G., Ardi, R., ..., & de Clunie, G. T. (2018). The mental health continuum‐short form: The structure and application for cross‐cultural studies-A 38 nation study. Journal of clinical psychology, 74(6), 1034-1052.

72.

Zyphur, M. J., Kaplan, S. A., & Christian, M. S. (2008). Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: Problems and solutions. Group Dynamics: Theory, Research, and Practice, 12(2), 127-140.

바로가기메뉴

Article Contents

Vol.38 No.2

Measurement Invariance Testing Using Random Effects model for Many groups: Multilevel Confirmatory Factor Analysis (ML CFA) and Multilevel Factor Mixture Modeling (ML FMM)

Abstract

Reference

Korean Journal of Psychology: General