This dissertation explores the validity of a measure designed to assess the extent to which two-year community and technical colleges have a culture of evidence. Culture of evidence refers to an institutional culture in which decisions related to student outcomes are based on evidence and data. The expected result of developing a culture of evidence is that colleges will become more successful at identifying “what works” in efforts to improve student success, which ultimately translates to better outcomes for students. The idea of developing a culture of evidence has gained prominence in higher education in recent years, prompted by increased calls for accountability in education in general and by the implementation of national initiatives specifically aimed at improving outcomes at two-year colleges.
While the idea of building a culture of evidence has received attention in the literature, little has been done to measure the concept quantitatively. This study addressed this gap by developing, pilot testing, and conducting a survey designed to measure the extent to which two-year colleges have a culture of evidence. The process used to develop the survey included a literature review, think-alouds with institutional researchers, and expert content review. The online survey was administered to 853 directors of institutional research or institutional effectiveness identified through a query of public, two-year, associate’s degree-granting institutions. A total of 369 individuals responded.
The framework for conducting the validation study was the unified conceptualization of validity from the American Psychological Association, American Educational Research Association, and National Council on Measurement in Education’s Standards for Educational and Psychological Testing (1999), in tandem with validation methods recommended by Allen and Yen (2002). Following this framework, five aspects of validity were assessed: (a) content, (b) response process/substantive, (c) internal structure, (d) predictive/relationships to other variables, and (e) consequential validity. Data analyses included descriptive statistics, reliability, analysis of variance, principal components analysis, exploratory and confirmatory factor analyses, and structural equation modeling.
The results indicated strong evidence of content, response process/substantive, and consequential validity. However, the results for internal structural and expected relationships validity were more mixed. The number of full-time equivalent (FTE) staff who work in an institutional research function and membership in the Achieving the Dream initiative had statistically significant and positive relationships to the culture of evidence, but expected relationships to other contextual variables such as institution size were not found. When treated as an aggregate measure, a college’s total culture of evidence score also predicted the extent to which colleges engaged in institutional data practices that are consistent with having a culture evidence such as using student outcomes data in decision-making processes and faculty requesting data for decisions.
A single study cannot determine validity-- validity should involve ongoing research to gather evidence for the interpretation of the scores for the intended use of a test or instrument (Cronbach and Meehl, 1955; Kane, 2006). This dissertation demonstrated a first attempt at gathering validity evidence for a new measure that could assist two-year colleges in their efforts to implement changes to improve student outcomes.