Objectives: Lao Juan (LJ, 劳倦) is a syndrome described in Chinese medicine (CM) that manifests with : Lao Juan (LJ, 劳倦) is a syndrome described in Chinese medicine (CM) that manifests with fatigue, fever, spontaneous sweating, indigestion, work-induced pain, weakness of the limbs, and shortness of breath. fatigue, fever, spontaneous sweating, indigestion, work-induced pain, weakness of the limbs, and shortness of breath. The present study was conducted to examine the reliability and validity of a Lao Juan Questionnaire (LJQ). The present study was conducted to examine the reliability and validity of a Lao Juan Questionnaire (LJQ). Methods: A total of 151 outpatients and 73 normal subjects were asked to complete the LJQ. Seventy-three normal subjects A total of 151 outpatients and 73 normal subjects were asked to complete the LJQ. Seventy-three normal subjects were additionally asked to complete the Chalder Fatigue Scale (CFS). Twelve clinicians determined whether the were additionally asked to complete the Chalder Fatigue Scale (CFS). Twelve clinicians determined whether the 151 outpatients exhibited LJ or not. The internal consistency and construct validity for the LJQ were estimated using 151 outpatients exhibited LJ or not. The internal consistency and construct validity for the LJQ were estimated using data from the outpatient subjects. The CFS data were used to examine the concurrent validity of the LJQ. Total LJQ data from the outpatient subjects. The CFS data were used to examine the concurrent validity of the LJQ. Total LJQ scores and the clinicians' diagnoses of the outpatients were used to perform receiver operating characteristics (ROC) scores and the clinicians' diagnoses of the outpatients were used to perform receiver operating characteristics (ROC) curve analyses and to defi ne an optimum cut-off score for the LJQ. curve analyses and to defi ne an optimum cut-off score for the LJQ. Results: The 19-item LJQ had satisfactory internal : The 19-item LJQ had satisfactory internal consistency (α=0.828) and concurrent validity, with signifi cant correlations between the LJQ and the CFS subscales. consistency (α=0.828) and concurrent validity, with signifi cant correlations between the LJQ and the CFS subscales. In the test of construct validity using principal component analysis, a total of six factors were extracted, and the overall In the test of construct validity using principal component analysis, a total of six factors were extracted, and the overall variance explained by all factors was 59.5%. In ROC curve analyses, the sensitivity, specifi city, and area under the variance explained by all factors was 59.5%. In ROC curve analyses, the sensitivity, specifi city, and area under the curve were 76.0%, 59.2%, and 0.709, respectively. The optimum cut-off score was defi ned as six points. curve were 76.0%, 59.2%, and 0.709, respectively. The optimum cut-off score was defi ned as six points. Conclusions: Our results suggest that the LJQ is a reliable and valid instrument for evaluating LJ. Our results suggest that the LJQ is a reliable and valid instrument for evaluating LJ. KEYWORDS Chinese medicine, chronic fatigue syndrome, Chinese medicine-pattern Chinese medicine, chronic fatigue syndrome, Chinese medicine-pattern

Development and Validation of A

Self-report measures of adult attachment are typically scored in ways (e.g., averaging or summing items) that can lead to erroneous inferences about important theoretical issues, such as the degree of continuity in attachment security and the differential stability of insecure attachment patterns. To determine whether existing attachment scales suffer from scaling problems, the authors conducted an item response theory (IRT) analysis of 4 commonly used self-report inventories: Experiences in Close Relationships scales (K. A. Brennan, C. L. Clark, & P. R. Shaver, 1998), Adult Attachment Scales (N. L. Collins & S. J. Read, 1990), Relationship Styles Questionnaire (D. W. Griffin & K. Bartholomew, 1994) and J. Simpson's (1990) attachment scales. Data from 1,085 individuals were analyzed using F. Samejima's (1969) graded response model. The authors' findings indicate that commonly used attachment scales can be improved in a number of important ways. Accordingly, the authors show how IRT techniques can be used to develop new attachment scales with desirable psychometric properties.

An item response theory analysis of self-report measures of adult attachment.

A committee of international knee experts created the International Knee Documentation Committee Subjective Knee Form, which is a knee-specific, rather than a disease-specific, measure of symptoms, function, and sports activity. The purpose of this study was to evaluate the reliability and validity of the new International Knee Documentation Committee Subjective Knee Form. To provide evidence for reliability and validity, we administered the final version of the form, along with the Short Form-36, to 533 patients with a variety of knee problems. Analyses were performed to determine reliability, validity, and differential item function related to age, sex, and diagnosis. Factor analysis revealed a single dominant component, making it reasonable to combine all questions into a single score. Internal consistency and test-retest reliability were 0.92 and 0.95, respectively. Based on test-retest reliability, the value for a true change in the score was 9.0 points. The International Knee Documentation Committee Subjective Knee Form score was related to concurrent measures of physical function (r = 0.47 to 0.66) but not to emotional function (r = 0.16 to 0.26). Analysis of differential item function indicated that the questions functioned similarly for men versus women, young versus old, and for those with different diagnoses. In conclusion, the International Knee Documentation Committee Subjective Knee Form is a reliable and valid knee-specific measure of symptoms, function, and sports activity that is appropriate for patients with a wide variety of knee problems. Use of this instrument will permit comparisons of outcome across groups with different knee problems.

Development and Validation of the International Knee Documentation Committee Subjective Knee Form

Scale development and validation are critical to much of the work in the health, social, and behavioral sciences. However, the constellation of techniques required for scale development and evaluation can be onerous, jargon-filled, unfamiliar, and resource-intensive. Further, it is often not a part of graduate training. Therefore, our goal was to concisely review the process of scale development in as straightforward a manner as possible, both to facilitate the development of new, valid, and reliable scales, and to help improve existing ones. To do this, we have created a primer for best practices for scale development in measuring complex phenomena. This is not a systematic review, but rather the amalgamation of technical literature and lessons learned from our experiences spent creating or adapting a number of scales over the past several decades. We identified three phases that span nine steps. In the first phase, items are generated and the validity of their content is assessed. In the second phase, the scale is constructed. Steps in scale construction include pre-testing the questions, administering the survey, reducing the number of items, and understanding how many factors the scale captures. In the third phase, scale evaluation, the number of dimensions is tested, reliability is tested, and validity is assessed. We have also added examples of best practices to each step. In sum, this primer will equip both scientists and practitioners to understand the ontology and methodology of scale development and validation, thereby facilitating the advancement of our understanding of a range of health, social, and behavioral outcomes.

/pdf/best-practices-for-developing-and-validating-scales-for-1pgx889exg.pdf

Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer.

Background: There is no universally accepted instrument that can be used to evaluate changes in self-reported physical function for individuals with leg, ankle, and foot musculoskeletal disorders. The objective of this study was to develop an instrument to meet this need: the Foot and Ankle Ability Measure (FAAM). Additionally, this study was designed to provide validity evidence for interpretation of FAAM scores. Methods: Final item reduction was completed using item response theory with 1027 subjects. Validity evidence was provided by 164 subjects that were expected to change and 79 subjects that were expected to remain stable. These subjects were given the FAAM and SF-36 to complete on two occasions 4 weeks apart. Results: The final version of the FAAM consists of the 21-item activities of daily living (ADL) and 8-item Sports subscales, which together produced information across the spectrum ability. Validity evidence was provided for test content, internal structure, score stability, and responsivenes...

/pdf/evidence-of-validity-for-the-foot-and-ankle-ability-measure-2fcdv3o0si.pdf

Evidence of Validity for the Foot and Ankle Ability Measure (FAAM)

An NCME Instructional Module on Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development.

Item response models are finding increasing use in achievement and aptitude test development. Item response theory (IRT) test development involves the selection of test items based on a consideration of their item information functions. But a problem arises because item information functions are determined by their item parameter estimates, which contain error. When the "best" items are selected on the basis of their statistical characteristics, there is a tendency to capitalize on chance due to errors in the item parameter estimates. The resulting test, therefore, falls short of the test that was desired or expected. The purposes of this article are (a) to highlight the problem of item parameter estimation errors in the test development process, (b) to demonstrate the seriousness of the problem with several simulated data sets, and (c) to offer a conservative solution for addressing the problem in IRT-based test development. Over the last 20 years, many test developers have used item response theory (IRT) models and methods rather than classical measurement models in their test development and related technical work (Hambleton, 1989; Hambleton & Swaminathan, 1985; Lord, 1980). Item response theory, particularly as reflected in the one-, two-, and three-parameter logistic models for dichotomously scored items, is receiving increasing attention from test developers in test design and test item selection, in addressing item bias, in computeradaptive testing, and in the equating and reporting of test scores. Many major test publishers, state departments of education, and large school districts currently use IRT models in some capacity in their testing work. One problem that arises when applying IRT models in test development involves capitalizing on chance due to positive errors in some item parameter estimates. As a result, tests may often fall short, statistically, of what is expected, and standard errors associated with ability estimates may be correspondingly underestimated (if the inflated item parameter estimates are used), which will lead to overconfidence in the ability estimates. The same problem arises in item selection in computerized adaptive testing. Perhaps it should be The research described in this article was funded by the Graduate Management Admission Council. The GMAC encourages researchers to formulate and freely express their own opinions, and the opinions expressed here are not necessarily those of the GMAC. This article was presented at the meeting of APA, Boston, 1990. The authors benefited considerably from the suggestions of two anonymous reviewers and the editor.

Influence of Item Parameter Estimation Errors in Test Development.

For test developers working within an item response theory framework, the concepts of item and test information are important and useful. Unfortunately, errors in item parameter estimates have a negative impact on the accuracy of item and test information functions. The estimation errors may be random; but because items with the higher levels of discriminating power are more likely to be selected for a test, and these items are most apt to contain positive errors in their item parameter estimates, item information functions and corresponding test information functions tend to be inflated in relation to their true values. The purpose of this article was to investigate the impact of this "capitalization on chance" in item selection on the accuracy of test information functions associated with the three-parameter logistic model. Two variables seemed especially important in determining the size of the impact: (a) examinee sample size used in calibrating test items and (b) the ratio of item bank size to test l...

/pdf/item-parameter-estimation-errors-and-their-influence-on-test-3crrw5ut06.pdf

Item Parameter Estimation Errors and Their Influence on Test Information Functions

This study was undertaken to establish the existence of any gender-specific differences in the perceived antecedents of academic stress. The Academic Pressure Scale for Adolescents was administered to 112 girls and 160 boys attending high school. Significant gender-based differences were obtained on eight of the 35 questions of the scale. In each case girls reported greater stress than boys. This study provides strong evidence that girls and boys of high school age differentially experience the antecedents of academic stress and that adolescent girls experience greater academic stress than boys.

/pdf/gender-specific-differences-in-the-perceived-antecedents-of-8y2umdlzax.pdf

Gender-Specific Differences in the Perceived Antecedents of Academic Stress:

The purpose of this study was to improve both statistical and judgmental methods for detecting potentially biased items in a test in an attempt to examine the agreement between the results obtained with these methods. If greater agreement between methods can be achieved, test items can be more effectively screened using judgmental methods prior to field testing or actual test administrations. Steps were taken to address several methodological shortcomings of current empirical and judgmental methods. The test data came from samples of 2,000 Native American and 2,000 Anglo-American students who took a 150-item Statewide Proficiency Test. Fifteen Native American educators provided item bias reviews. The results suggest that a somewhat higher level of agreement between methods was obtained than has been observed in other studies. The use of cross-validation in empirically identifying potentially biased items was one reason for the higher level of agreement. However, the judgmental method implemented in this study appeared to have several shortcomings. Practical implications of the findings are presented. LR231 U.S. DEPARTMENT Of EDUCATION Moe of Educahonal Research end Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) This document has been reproduced as received front the person or organization originating it. 0 Minor changes have been made lo Improve reproduction OtPahlY Fonts ot view or opinions stated in this document do not necessarily represent official OERI postilion or policy "PERMISSION TO REPRODUCE THIS MATERIAL HAS BEEN GRANTED BY iee(kg AMBLEk) TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC)." 2 BEST COPY AVAILABLE Comparison of Empirical and Judgmental Methods for Detecting DifferenziAl Item Functionine2 Ronald K. Hambleton and Russell W. Jones3 University of Massachusetts at Amherst Paper-and-pencil tests are widely used as tools for selection, promotion, certification and licensure decisions throughout education, business, the armed services, and industry. As test use for important decisions has increased, the issue of item bias has achieved considerable significance. Test developers must now demonstrate that their tests are free of item bias. To this end, various judgmental and empirical methods for detecting potentially biased items have been proposed (see, for example, Berk, 1982; Hills, 1989; Scheuneman & Bleist(in, 1989). These "DIF" studies, as they are commonly called, are designed to detect differential item functioning (DIF) between reference and focal groups. Typically, judgmental and empirical methods for detecting differentially functioning items have shown little agreement (Plake, 1980; Engelhard, Hansche, & Rutledge, 1990). A partial explanation for this low agreement is that the judgmental review forms are sometimes focused on cultural and sexual stereotyping in items .rather than on factors which may lead to differential performance between subgroups of interest (Scheuneman, 1982). As a result, many undesirable item, are identified in the item bias review process, such as those which may show members of minority groups doing 1Laboratory of Psychometric and Evaluative Research Report No. 231. Amherst, MA: University of Massachusetts at Amhest, School of Education. 2Paper presented at the meeting of NCME, San Francisco, 1992. 3The authors wish to thank the 15 Native American educators who completed the item review task, and John Martois for his assistance in locating reviewers.

/pdf/comparison-of-empirical-and-judgmental-procedures-for-1kvlmfzcht.pdf

Russell W. Jones

Papers

An NCME Instructional Module on Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development.

Influence of Item Parameter Estimation Errors in Test Development.

Item Parameter Estimation Errors and Their Influence on Test Information Functions

Gender-Specific Differences in the Perceived Antecedents of Academic Stress:

Comparison of Empirical and Judgmental Procedures for Detecting Differential Item Functioning.