Doctor of Philosophy, University of Toledo, 2021, Educational Research and Measurements

Existing methods used to validate self-report questionnaires in foreign language teaching effectiveness have relied on Classical Test Theory (CTT). However, the use of CTT approaches limits the reliability and validity of self-report instruments. The Rasch Model, which is based on the principles of objective measurement, addresses some of the major limitations of CTT-based validation and provides a way to construct linear item and person measures. Thus, the Rasch Model was adopted in the current study as a part of a multi-phase methodological approach to validate the teaching effectiveness questionnaire used in the STARTALK-CHELER Chinese Leadership program. The multiple aspects of validity of the self-report questionnaire were investigated for making an evaluative judgment about the quality of the measure. The multi-phase strategy utilized qualitative, measurement, and statistical techniques. Phase I, content validity, was examined in aspects of relevance, representative, translation, and technical quality. The results showed that some methodological issues, such as content representation gaps, imprecisely translated Mandarin Chinese items, and misfitting items, limited the content validity. The examination of the construct validity regarding the substantive, structural, generalizability, responsiveness and interpretability aspects in Phase II also showed many of the methodological issues, such as person misfit, ceiling effect, and differential item functioning, limited the substantive, generalizability, and interpretability validity. However, the items on the questionnaire were confirmed to measure a single latent construct – foreign language teaching effectiveness in terms of instructional skills domain summarized in the literature. In Phase III, the results of the independent t-test showed that there were no significant differences between the certified and non-certified teachers on the teaching effectiveness. Finally, a proposed questionnaire in both English a (open full item for complete abstract)

Committee: Noela Haughton (Committee Chair); Christine Fox (Committee Member); An Chung Cheng (Committee Member); Mingli Xiao (Committee Member) Subjects: Education; Educational Tests and Measurements; Foreign Language

Doctor of Philosophy, University of Toledo, 2019, Foundations of Education - Research and Measurement

This study explores six item-reduction methodologies used to shorten an existing complex problem-solving non-objective test by evaluating how each shortened form performs across three sources of validity evidence (i.e., test content, internal structure, and relationships with other variables). Two concerns prompted the development of the present study. First, to address practical concerns in education while administering psychometrically sound assessments, shorter, yet rigorous assessments, measuring complex student processes including problem solving are desirable. Second, methods to shorten assessments within psychology and medicine have been explored to some extent, best practices for item-reduction methods within educational testing has largely been absent. The Problem Solving Measure 6 (PSM6) was designed to measure sixth-grade students' mathematical problem-solving ability aligned with the Common Core State Standards for Mathematics (CCSSM) (Bostic & Sondergeld, 2015). The original 15-item assessment was reduced using three qualitative models (content only, content plus difficulty, and content, difficulty, and point-biserial) and three quantitative item-reduction methods (Rasch measurement, factor analysis, and point-biserial). Each item-reduction method was used in an attempt to identify five-item and ten-item shortened versions of the PSM6. The shortened tests were then evaluated based on test content, internal structure, and relationships to other variables. Test content was evaluated through an expert panel rating each item on its essentiality to measure its associated mathematical domain. Internal structure was evaluated through a Rasch (1960, 1980) measurement analysis. Relationships to other variable was evaluated through gender and ability differences, measure score correlations between the original and each short form, and the consistency of student quartile placement across the original and short forms. Some limitations were encountered during the re (open full item for complete abstract)

Committee: Gregory Stone (Committee Chair); Noela Haughton (Committee Member); Toni Sondergeld (Committee Member); Jonathan Bostic (Committee Member) Subjects: Educational Tests and Measurements

PHD, Kent State University, 2018, College of Education, Health and Human Services / School of Foundations, Leadership and Administration

The need to create assessment literate and assessment confident teachers is increasing (Popham, 2009; 2011). Research has revealed that teachers are not well trained to use assessment in the classroom and are poorly trained in standardized testing (Zhang & Burry-Stock, 1997; Zhang & Burry-Stock, 2003). The purpose of this study was to: (1) evaluate the psychometric properties (i.e., reliability and validity) of an instrument that measures the assessment literacy and assessment confidence of pre-service teachers (i.e., the Classroom Assessment Literacy Inventory [CALI]), and (2) investigate the relationship between assessment literacy, assessment confidence, and scores on a performance-based assessment (edTPA). In the pilot testing phase, Rasch Analysis and Rasch Principal Components Analysis (PCA) were used to evaluate the psychometric properties (i.e., reliability and validity) of the assessment literacy and confidence measures (i.e., the CALI). The pilot sample (N = 165) consisted of sophomores and juniors in one teacher preparation program in the Midwestern United States (US). After the pilot testing phase, the instrument was revised and administered to a second sample of 112 pre-service teachers who were in their final semester of the same undergraduate teacher preparation. Confirmatory Factor Analysis (CFA) was used to provide evidence of the internal structure of the CALI. Following the CFA, controlling for other demographic and academic variables such as teacher education program (e.g., Early Childhood, Middle Childhood, Adolescent Education, etc.) and Grade Point Average (GPA), the impact of the second phase sample's assessment confidence on the relationship between assessment literacy and performance-based assessment scores was examined. Results indicated the limited range of the assessment-related content measured by the modified CALI, as well as the modified CALI's relative difficulty for this sample. Significant relationships were found between p (open full item for complete abstract)

Committee: Aryn Karpinski Dr. (Advisor) Subjects: Education; Educational Tests and Measurements; Teacher Education

Doctor of Philosophy, University of Toledo, 2016, Counselor Education and Supervision

Individuals infected and affected by the human immunodeficiency virus (HIV) and acquired immune deficiency syndrome (AIDS) have distinctive mental and emotional health issues (Acuff et al., 1999; Badiee et al., 2012; Hult et al., 2007). This study sought to create an instrument that measures the attitude element of competency with the development of the Mental Health Professionals' Attitude Towards People Living with HIV/AIDS (MHP-PLHIV-AS). After the MHP-PLHIV-AS's creation by a Content Evaluation Panel of HIV/AIDS Experts, it was piloted for calibration with a sample of mental health professionals (n = 43), then administered to a larger sample for validation (n = 454). The newly designed MHP-PLHIV-AS was analyzed through a Rasch Measurement Model (RMM; Rasch, 1960, 1980). RMM diagnostics and analyses provides evidence to support a two-dimensional (societal and personal dimensions) measurement of the attitude towards PLHIV construct. The authors provide background, processes, and results of the study, and implications not only for the use of the MHP-PLHIV-AS, but also of attitude being a two-dimensional construct.

Committee: John Laux (Committee Chair); Christine Fox (Committee Member); Christopher Roseman (Committee Member); Mojisola Tiamiyu (Committee Member); Caroline O'Hara (Committee Member) Subjects: Counseling Education; Counseling Psychology; Multicultural Education; Psychology; Public Health; Therapy

Doctor of Philosophy, The Ohio State University, 2016, Social Work

This dissertation examines measurement of interprofessional education (IPE) competencies set forth by the Interprofessional Education Collaborative (IPEC) Expert Panel. Background: There is reason to believe that effective, interprofessional, collaborative care can lead to more positive health outcomes and quality patient care. A Cochrane Collaborations Review suggests, however, that these professionals do not collaborate well together, but that IPE offers a potential way to improve interprofessional collaboration, patient care, and health outcomes. The IPEC Expert Panel identified the following four core competency domains for interprofessional collaborative practice 1) values/ethics for interprofessional practice, 2) roles/responsibilities, 3) interprofessional communication, and 4) teams and teamwork. Methods: This study aims to contribute to the field of measurement in IPE by utilizing Rasch analysis to examine the psychometric properties of the IPEC Core Competencies Survey Instrument. Rasch analysis allowed the researcher to (1) examine response category functioning; (2) assess item fit, retention, and reliability; (3) examine person separation, reliability, and validity; and (4) assess the unidimensionality of each measure. Results and Implications: Findings suggest it is a useful instrument and the researcher recommends minor revisions for improvement of the psychometric properties. These revisions include the removal of one item each from the interprofessional communication scale and the teams and teamwork scale. The structure of the values and ethics response scale was reduced from 5 response categories to 3 categories. Discussion of the role of social work to take a lead in IPE and future directions for IPE measurement research are included. The unique skill set of social work professionals and their natural role as facilitator, convener, and collaborator make them appropriate to take a leadership role in the national and global IPE conversat (open full item for complete abstract)

Committee: Mo Yee Lee PhD (Advisor); Holly Dabelko-Schoeny PhD (Committee Member); Keith Anderson PhD (Committee Member) Subjects: Education; Educational Tests and Measurements; Health Education; Health Sciences; Social Work

Doctor of Philosophy, University of Toledo, 2010, Curriculum and Instruction: Educational Technology

This paper adds to the literature on student-centered teaching in higher education, answering the question of what is expected of instructors for them to teach in a student-centered manner. Building upon the existing literature regarding student-centered teaching, this paper defines a construct of student-centered teaching. In addition, two versions of a data collection instrument have been developed to measure the student centeredness of higher education instructors' teaching approaches, utilizing self-reported data. This paper also adds to the discussion on whether student-centered teaching is related to the teaching context or teacher demographic variables. Results indicated that there was no practically significant relationship between student-centered teaching and any of the contextual or demographic variables; course discipline and instructor sex were, however, statistically significant. This study took place within the context of a cultural change initiative at The University of Toledo, in which the university is creating a culture of student-centeredness.

Committee: Berhane Teclehaimanot (Committee Chair); Robert Sullivan (Committee Member); Gregory Stone (Committee Member); Leigh Chiarelott (Committee Member) Subjects: Education; Higher Education; Teaching

Doctor of Philosophy, University of Toledo, 2009, Education and Research and Measurement

Previous research regarding the relationship between program quality and child outcomes has left unanswered questions regarding how Head Start program quality is affecting children's gains in school readiness following program participation. Variations in program quality have not been found to have a strong association with child outcomes. Issues with traditional measurement development and validation may be impacting the research findings regarding the relationship between program quality and child outcomes. The purpose of this study was to evaluate measures of Head Start program quality using an objective measurement approach (i.e., Rasch analysis) and to create a holistic measure of Head Start program quality from four individual instruments used to evaluate early childhood program quality. The unit of analysis for this study was Head Start classrooms. Rasch analysis of the four classroom quality instruments revealed various issues with the functioning of the instruments including a failure to adequately distinguish classrooms on the construct, problems with the use of the rating scale, misfitting items and classrooms and poor targeting of classrooms, especially at the low end of the measurement scale. Poor separation and targeting of classrooms on these instruments may be a possible explanation for the failure to find a relationship between classroom quality and children's gains in school readiness. This study's attempt to develop a holistic measure of program quality using the four individual instruments resulted in an instrument with greater ability to distinguish classrooms, with a high level of reliability. However, the combined measure of classroom quality was found to still have problems with adequate targeting of classrooms due to the individual instruments from which it was constructed. Future research should attempt to identify higher-level indicators of classroom quality to improve the differentiation and targeting of Head Start classrooms in regards t (open full item for complete abstract)

Committee: Gregory Stone PhD (Advisor); Svetlana Beltyukova PhD (Committee Member); Christine M. Fox PhD (Committee Member); Ruslan Slutsky PhD (Committee Member) Subjects: Education; Preschool Education

Doctor of Philosophy, University of Toledo, 2013, Judith Herb College of Education

Many studies have been conducted where a face-to-face training environment is compared to an online training environment. While some research has been conducted on the nature of online training in faith-based not-for-profit organizations, little to no research has been found on engagement. The purpose of this study was to determine if the level of engagement of participants in a training course for new staff and interns with Campus Crusade for Christ would be increased by conducting training online instead of face-to-face and by utilizing multiple forms of media. The survey that was utilized included questions adapted from the Student Course Engagement Questionnaire (Handelsman et al., 2005) and was analyzed utilizing the Rasch measurement model to understand whether the survey successfully met the requirements for measuring engagement. The Rasch measurement analysis revealed that the survey was weak and did not measure engagement, thus the results of the survey revealed no significant differences in the level of engagement. Further research is recommended with new questions being added to the survey that are considered to have a greater level of difficulty as well as research should be conducted that involve qualitative data collection.

Committee: Judy Lambert Ph.D. (Committee Chair); Gregory Stone Ph.D. (Committee Member); Tony Sanchez Ph.D. (Committee Member); Savilla Banister Ph.D. (Committee Member) Subjects: Adult Education; Business Education; Curriculum Development; Educational Evaluation; Educational Software; Educational Technology; Higher Education; Information Technology; Instructional Design; Teaching