In education research, ordinal data is the norm but does not meet the assumptions of most statistical methods and thus is often analyzed inappropriately. Using a dataset typical of the field, this study compared four factor analytic methods: a traditional exploratory factor analysis (EFA), a full-information EFA, and two EFAs within the confirmatory factor analysis framework (E/CFA) conducted according to the Jöreskog method and the Gugiu method. Because an approach for handling cross-loaded items in multifactor models has not been clearly defined within the Gugiu method, two approaches were compared. The fixed-loadings approach involves forcing cross-loaded items to load onto only one factor, chosen based on the strongest theoretical justification. The delete-items approach deletes all cross-loaded items from the model. Both approaches were used to arrive at a starting model that was then modified according to the Gugiu method. Methods were compared on initial model fit, replication in a confirmatory factor analysis, and the stability, interpretability, and reliability of the models.
In terms of initial model fit, methods appropriate for ordinal data produced better models, the E/CFAs outperformed the EFAs, and the Gugiu method demonstrated greater model interpretability than the Jöreskog method. Both approaches to the Gugiu method produced well-fitting models, but the delete-items approach outperformed the fixed-loadings approach. However, contrary to the findings of a previous study, these results did not hold for model validation. In CFAs conducted on posttest data, the model fit of the E/CFAs was on par with or worse than the model fit of the EFAs. Additionally, the two approaches to the Gugiu method performed the worst where before they had performed the best, with the fixed-loadings approach faring particularly poorly. In the case of this data, the full-information EFA produced the best fitting models.
Examining characteristics of the data help to explain the unexpectedly poor performance of the E/CFA methods and help to clarify when these methods are appropriate to use. Diagonal weighted least squares (DWLS), the method of estimation employed by the full-information EFA and the two E/CFAs, may produce biased parameter estimates when used with small sample sizes, with factors defined by only a few items, and with items with high skewness. These biased parameter estimates are even more problematic when used to make model modification decisions, as they were for the Jöreskog and Gugiu E/CFA methods. Thus, the results of the current study suggest that the full-information EFA may be the most appropriate method to use with data with these problematic characteristics. Secondarily, the findings also provide evidence for the delete-items approach as the more appropriate way of dealing with cross-loaded items in the Gugiu E/CFA method.