https://science.sciencemag.org/content/sci/5/110/206.full.pdf

The american psychological association.

This article reviews the corpus of research on feedback, with a focus on formative feedback—defined as information communicated to the learner that is intended to modify his or her thinking or behavior to improve learning According to researchers, formative feedback should be nonevaluative, supportive, timely, and specific Formative feedback is usually presented as information to a learner in response to some action on the learner’s part It comes in a variety of types (eg, verification of response accuracy, explanation of the correct answer, hints, worked examples) and can be administered at various times during the learning process (eg, immediately following an answer, after some time has elapsed) Finally, several variables have been shown to interact with formative feedback’s success at promoting learning (eg, individual characteristics of the learner and aspects of the task) All of these issues are discussed This review concludes with guidelines for generating formative feedback

/pdf/focus-on-formative-feedback-3stra0yvxe.pdf

Focus on Formative Feedback

Education is a hot topic. From the stage of presidential debates to tonight's dinner table, it is an issue that most Americans are deeply concerned about. While there are many strategies for improving the educational process, we need a way to find out what works and what doesn't work as well. Educational assessment seeks to determine just how well students are learning and is an integral part of our quest for improved education. The nation is pinning greater expectations on educational assessment than ever before. We look to these assessment tools when documenting whether students and institutions are truly meeting education goals. But we must stop and ask a crucial question: What kind of assessment is most effective? At a time when traditional testing is subject to increasing criticism, research suggests that new, exciting approaches to assessment may be on the horizon. Advances in the sciences of how people learn and how to measure such learning offer the hope of developing new kinds of assessments-assessments that help students succeed in school by making as clear as possible the nature of their accomplishments and the progress of their learning. Knowing What Students Know essentially explains how expanding knowledge in the scientific fields of human learning and educational measurement can form the foundations of an improved approach to assessment. These advances suggest ways that the targets of assessment-what students know and how well they know it-as well as the methods used to make inferences about student learning can be made more valid and instructionally useful. Principles for designing and using these new kinds of assessments are presented, and examples are used to illustrate the principles. Implications for policy, practice, and research are also explored. With the promise of a productive research-based approach to assessment of student learning, Knowing What Students Know will be important to education administrators, assessment designers, teachers and teacher educators, and education advocates.

Knowing What Students Know: The Science and Design of Educational Assessment

This paper reviews the corpus of research on feedback, with a particular focus on formative feedback—defined as information communicated to the learner that is intended to modify the learner's thinking or behavior for the purpose of improving learning. According to researchers in the area, formative feedback should be multidimensional, nonevaluative, supportive, timely, specific, credible, infrequent, and genuine (e.g., Brophy, 1981; Schwartz & White, 2000). Formative feedback is usually presented as information to a learner in response to some action on the learner's part. It comes in a variety of types (e.g., verification of response accuracy, explanation of the correct answer, hints, worked examples) and can be administered at various times during the learning process (e.g., immediately following an answer, after some period of time has elapsed). Finally, there are a number of variables that have been shown to interact with formative feedback's success at promoting learning (e.g., individual characteristics of the learner and aspects of the task). All of these issues will be discussed in this paper. This review concludes with a set of guidelines for generating formative feedback.

https://onlinelibrary.wiley.com/doi/pdf/10.1002/j.2333-8504.2007.tb02053.x

Focus on formative feedback

This study investigated the utility of confirmatory factor analysis (CFA) and item response theory (IRT) models for testing the comparability of psychological measurements. Both procedures were used to investigate whether mood ratings collected in Minnesota and China were comparable. Several issues were addressed. The first issue was that of establishing a common measurement scale across groups, which involves full or partial measurement invariance of trait indicators. It is shown that using CFA or IRT models, test items that function differentially as trait indicators across groups need not interfere with comparing examinees on the same trait dimension. Second, the issue of model fit was addressed. It is proposed that person-fit statistics be used to judge the practical fit of IRT models. Finally, topics for future research are suggested. Much research and debate has been motivated by the question of how to establish that a test measures the same trait dimension, in the same way, when administered to two or more qualitatively distinct groups (e.g., men and women). The question can also be posed as follows: Are test scores for individuals who belong to different examinee populations comparable on the same measurement scale? The objectives of this study were to review linear confirmatory factor analysis' (CFA; Long, 1983) and item response theory (IRT; Lord, 1980) approaches to addressing this important question and to suggest, by way of real-data application, advantages and disadvantages of each approach.

/pdf/confirmatory-factor-analysis-and-item-response-theory-two-br5f7ybz2x.pdf

Confirmatory factor analysis and item response theory : two approaches for exploring measurement invariance

Several authors from various disciplines such as cognitive psychology, artificial intelligence, and psychometrics have developed rule assessment methods for diagnosing students' misconceptions. Siegler's (1976, 1978) binary decision tree method was illustrated in the context of the problem of balance scales and Anderson's (1974, 1981) functional method was developed for diagnosing errors occurring during the operation of integrating functional rules. A group of artificial intelligence researchers (Brown & Burton, 1978; Brown & VanLehn, 1980; VanLehn, 1981) developed a computer program "DEBUGGY" that can diagnose a number of erroneous rules resulting from misconceptions ("bugs") in whole number subtraction problems. These investigations were motivated by an interest in the basic foundations of knowledge structure and development in human cognition. Tatsuoka and her associates (Tatsuoka, Baillie, & Yamamoto, 1982; Tatsuoka, Birenbaum, Tatsuoka, & Baillie, 1980) also developed a computer program that can diagnose a number of erroneous rules in signed-number addition and subtraction problems, but they were motivated primarily by the exploration of psychometric properties of bugs such as changes in error types or the stability of misconceptions committed by a student throughout a test (Birenbaum & Tatsuoka, 1980; Tatsuoka, 1981; Tatsuoka & Tatsuoka, 1983). Analysis of misconceptions can provide useful information in evaluating instruction or instructional materials as well as specific prescriptions for planning remediation for a student. For example, the source of many of the misconceptions committed by students is often the ambiguity of explanations or the lack of precise, accurate instructions in teaching material. It is useful to have such computer programs in educational practice. However, constructing a DEBUGGY-type system for domains more general than arithmetic is extremely difficult and time consuming. On the other hand, personal indices that are designed to summarize response patterns can be used for detecting anomalous patterns (Drasgow, 1982; Harnisch & Linn, 1981; Levine & Rubin, 1979; Sato, 1975; Tatsuoka & Tatsuoka, 1982; Wright & Stone, 1979) which may result from applications of some erroneous rules, but they have only limited power. For instance, these indices cannot diagnose sources of misconceptions or provide prescriptive information for remediating them. Nevertheless, the use of personal indices is economical and applicable to general areas because such indices can first be used to spot candidates among students who may possibly possess misconceptions and may hence need personal attention from teachers. This procedure seems to be widely used in Japan (Sato, 1982). This study develops a model to fill the gap between the rule-assessing methods and the personal-index approach. The number of different erroneous rules already discovered in

https://www.jstor.org/stable/info/1434951

Rule space: an approach for dealing with misconceptions based on item response theory

Finding the sources of misconceptions possessed by students is a difficult task 
because it is impossible to see what is happening in their heads The only directly 
observable outcomes are the students' responses to the test items Studying their 
think-aloud protocols is one method for discovering how students solve or think 
through problems Several computer programs that are capable of diagnosing 
students' misconceptions have been developed in the past decade (Brown & 
Burton, 1978; Marshall, 1980; Ohlsson & Langley, 1985; Sleeman, 1984; Tatsuoka, Baillie, & Yamamoto, 1982; VanLehn, 1983) The common ground for 
these cognitive diagnostic systems is that they infer the unobservable cognitive 
processes from logical interrelationships among cognitive tasks, subtasks, and 
goals involved in problems representing the domain of interest It is important 
that we be able to retrieve invisible things from the "black box" and put them 
into a useful form so that valuable information can be obtained for improving 
educational quality

Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis.

This paper introduces a probabilistic approach to the classification and diagnosis of erroneous rules of operations that result from misconceptions (“bugs”) in a procedural domain of arithmetic. The model is different from the usual deterministic strategies common in the field of artificial intelligence because variability of response errors is explicitly treated through item response theory. As a concrete example, we analyze a dataset that reflects the use of erroneous rules of operation in problems of signed-number subtraction. The same approach, however, is applicable to the classification of several different groups of response patterns caused by a variety of different underlying misconceptions, different backgrounds of knowledge, or treatment.

A Probabilistic Model for Diagnosing Misconceptions By The Pattern Classification Approach

A new family of indices was introduced earlier as a link between two approaches: One based on item response theory and the other on sample statistics. In this study, the statistical properties of these indices are investigated and then the relationships to Guttman Scales, and to item and person response curves are discussed. Further, these indices are standardized, and an example of their potential usefulness for diagnosing students' misconceptions is shown.

Caution indices based on item response theory

This study uses the rule-space methodology to explore the cognitive and linguistic attributes that underlie performance on an open-ended, short-answer, listening comprehension test. This is the first application of the methodology to language testing. The rule-space methodology is an adaptation of statistical pattern recognition techniques applied to the problem of diagnosing the cognitive attributes (knowledge, skills, abilities, strategies, etc.) underlying test performance. The methodology provides diagnostic information about the individual test-takers on each of these attributes. Based on a literature search, attribute candidates were identified which seemed likely to explain performance on the listening test. Two rule-space analyses were carried out, and the final attribute list had 15 attributes, with 14 interactions. With this, 96% of the test-takers were successfully classified into their respective latent knowledge states, and given scores on each attribute. This result suggests that the rule-sp...

Kikumi K. Tatsuoka

Papers

Rule space: an approach for dealing with misconceptions based on item response theory

Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis.

A Probabilistic Model for Diagnosing Misconceptions By The Pattern Classification Approach

Caution indices based on item response theory

Application of the rule-space procedure to language testing: examining attributes of a free response listening test: