Rule space: an approach for dealing with misconceptions based on item response theory

TLDR: In this paper, Tatsuoka et al. developed a model to fill the gap between the rule-assessing methods and the personal-index approach for diagnosing students' misconceptions.

Abstract: Several authors from various disciplines such as cognitive psychology, artificial intelligence, and psychometrics have developed rule assessment methods for diagnosing students' misconceptions. Siegler's (1976, 1978) binary decision tree method was illustrated in the context of the problem of balance scales and Anderson's (1974, 1981) functional method was developed for diagnosing errors occurring during the operation of integrating functional rules. A group of artificial intelligence researchers (Brown & Burton, 1978; Brown & VanLehn, 1980; VanLehn, 1981) developed a computer program "DEBUGGY" that can diagnose a number of erroneous rules resulting from misconceptions ("bugs") in whole number subtraction problems. These investigations were motivated by an interest in the basic foundations of knowledge structure and development in human cognition. Tatsuoka and her associates (Tatsuoka, Baillie, & Yamamoto, 1982; Tatsuoka, Birenbaum, Tatsuoka, & Baillie, 1980) also developed a computer program that can diagnose a number of erroneous rules in signed-number addition and subtraction problems, but they were motivated primarily by the exploration of psychometric properties of bugs such as changes in error types or the stability of misconceptions committed by a student throughout a test (Birenbaum & Tatsuoka, 1980; Tatsuoka, 1981; Tatsuoka & Tatsuoka, 1983). Analysis of misconceptions can provide useful information in evaluating instruction or instructional materials as well as specific prescriptions for planning remediation for a student. For example, the source of many of the misconceptions committed by students is often the ambiguity of explanations or the lack of precise, accurate instructions in teaching material. It is useful to have such computer programs in educational practice. However, constructing a DEBUGGY-type system for domains more general than arithmetic is extremely difficult and time consuming. On the other hand, personal indices that are designed to summarize response patterns can be used for detecting anomalous patterns (Drasgow, 1982; Harnisch & Linn, 1981; Levine & Rubin, 1979; Sato, 1975; Tatsuoka & Tatsuoka, 1982; Wright & Stone, 1979) which may result from applications of some erroneous rules, but they have only limited power. For instance, these indices cannot diagnose sources of misconceptions or provide prescriptive information for remediating them. Nevertheless, the use of personal indices is economical and applicable to general areas because such indices can first be used to spot candidates among students who may possibly possess misconceptions and may hence need personal attention from teachers. This procedure seems to be widely used in Japan (Sato, 1982). This study develops a model to fill the gap between the rule-assessing methods and the personal-index approach. The number of different erroneous rules already discovered in