| dc.description.abstract | Students often enter our classrooms with many preexisting beliefs about force and motion learned through a lifetime of hands-on experience. Unfortunately, many of these beliefs are not correct, leading to a wide variety of misconceptions about mechanics concepts. Such misconceptions are often highly resistant to traditional instruction and can seriously impede deep conceptual learning in mechanical engineering and other STEM fields; it is therefore essential to develop ways of understanding, measuring, and correcting these in our students. Previous efforts to address this problem have led to the development of research-based multiple-choice tests such as the Force Concept Inventory (FCI), a pioneering instrument whose incorrect answer choices ("distractors") were mined from student responses and implicitly encode many common misconceptions. While numerous studies have undertaken to explore the structure of the FCI by analyzing student responses, very few of these make full use of the information present in the students' specific choice of distractors, and those that do are limited in the types of associations they are able to find.
This thesis demonstrates that, by combining highly-flexible psychometric models, modern Bayesian inference methods, and traditional factor analysis techniques to analyze a large dataset of student responses to the FCI, it is possible to discover a much greater number of interpretable student misconceptions than previously found, and to do so without prior labeling of test content beyond identifying the correct answers. Our approach is based on the Multidimensional Nominal Categories IRT Model (MNCM) which leverages information present in both correct and incorrect student responses to identify a plausible set of latent (unobserved) student traits governing response behavior. To provide robustness to infrequent responses and guard against overfitting, we implement this as a Bayesian model with hierarchical priors; approximate variational inference methods enable this approach to scale to larger datasets, and standard factor rotation techniques facilitate the discovery of small subsets of distractors most likely to encode real psychological constructs.
We apply our method to a dataset comprising both pre- and post-instruction FCI submissions from ~17,000 students across eight North American colleges and universities. We take a principled approach to rotation, comparing both subjective interpretability and several indicators of solution quality, including similarity of our results across both schools and instruction as well as a metric of overall simplicity (the hyperplane count). Our results inform the generation of several candidate solutions, which we further examine for robustness via non-parametric bootstrapping and from which we choose a subset of 27 unique, partly-overlapping sets of distractors for additional investigation. By examining both the content of the most heavily-loaded distractors in these sets and the broader context of their associated questions, we find 22 partly-overlapping dimensions corresponding to misconceptions and misunderstandings. Most of these are consistent with prior research on specific misconceptions in introductory mechanics, and two appear to be novel. We further observe that many of our misconceptions correspond to previously-accepted historical theories about force and motion.
Finally, we present a simple method for assessing the prevalence of each misconception and apply it to our data both before and after instruction. We identify two major classes of misconceptions---standard and naive---which differ both in which students they affect and in how resistant they are to instruction. Our results support and extend prior findings about misconceptions by showing that (1) it is primarily the standard misconceptions which are most resistant to instruction, while naive misconceptions appear better remediated in our data; and that (2) current instruction appears to be noticeably less effective in dispelling misconceptions in learners with average or below-average pre-test scores, further underscoring the need for improved instructional techniques. Importantly, though, our work also provides educators and researchers alike with a new tool to address this challenge. By providing a catalog of existing misconceptions as well as the means to discover and measure new ones, we hope this research will lead to the development of focused interventions for specific misconceptions, aid in the evaluation of new instructional techniques, and provide teachers with a clearer lens through which to understand and adapt to the students in their classrooms. | |
