This was a data analysis and exploration project I did as part of a company “code-a-thon”, in which I looked at how concepts from Bayesian statistics could be applied to assessment results, in a not-quite-Bayesian context. If you are only interested in my straight UX work, you might want to hit the Back button now and look at a different example.
The slides mostly stand on their own, but a few initialisms and conventions may require explanation:
Initialisms:
- MC: multiple choice
- CR: constructed response (i.e., questions that ask for a written answer)
- MCAo: another question type. This is enough information to understand the slide, but I’ve included an explanation below for anyone who is interested in the details.
- 3.MD.A and 3.OA.A are Common Core math “clusters”, or groups of standards (i.e. learning objectives). Again, while additional detail isn’t needed to follow, it is provided below for anyone who is interested.
Whenever a value is given for “n”, this is a student count.
The notation P(X|Y) means the probability of X being true, given that Y is known to be true. Thus, with reference to 3.MD.A, P(K|1) is the (estimated) probability that a student who earned a score of 1 out of 1 (i.e., answered the question correctly) did so because she knew the correct answer, rather than by random guessing. Similarly, with reference to 3.OA.A, P(K|2) is the (estimated) probability that a student who earned a score of 2 out of 2 (i.e., answered both questions correctly) did so because she knew both correct answers, rather than random guessing having been responsible for either or both of her correct choices.
A Bayesish Perspective on MC Item Data
Appendix: Additional details
Multiple Correct Answer – open (MCAo)
This is a question type where a student is given a number of answer choices and asked to choose ALL those that are correct, without being told how many that is. For instance, a math question might offer a choice of six expressions, and ask the student to choose all those that always simplify to an even integer. To earn full credit (two points), the student must choose every correct answer, and none of the incorrect answers. This tends to be challenging, since with n answer choices, there are 2n possible responses, only one of which is fully correct. For instance, there are 64 possible responses to an MCAo with six answer choices. One point (half credit) is awarded for a partially correct score that passes a sufficient correctness threshold. (Just as only one response pattern is fully correct, only one response pattern is fully incorrect. Since the vast majority of possible responses are thus “partially correct”, it would not be desirable to award half credit for every partially correct response.)
Common Core Math Clusters
The Common Core math standards are organized in a hierarchy of “domains”, “clusters”, and “standards”. MD and OA are domains: “measurement and data” and “operations and algebraic thinking”, respectively. Within 3rd grade measurement and data, cluster 3.MD.A consists of the individual standards 3.MD.1 and 3.MD.2, which are summarized by the brief heading, “Solve problems involving measurement and estimation”. The standards proper are relatively verbose and specify precisely what students are required to learn, whereas the cluster headings are more concise and provide a intermediate-level framework for organizing and thinking about the standards.