Journal ArticleDOI
Answering the Call for a Standard Reliability Measure for Coding Data
TLDR
This work proposes Krippendorff's alpha as the standard reliability measure, general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data.Abstract:
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.read more
Citations
More filters
Journal ArticleDOI
Nurse-physician collaboration: a meta-analytical investigation of survey scores.
TL;DR: It appeared that interprofessional education interventions were able to reduce the difference in IPC between nurses and physicians.
Journal ArticleDOI
Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation
TL;DR: This paper assessed evidence for the evaluation inference, which includes the assumption that Criterion provides students with accurate feedback, and used this inference to support the use of Criterion feedback for students to make decisions about revisions.
Proceedings ArticleDOI
Statistical affect detection in collaborative chat
Michael Brooks,Katie Kuksenok,Megan K. Torkildson,Daniel Perry,John J. Robinson,Taylor Jackson Scott,Ona Anicello,Ariana Zukowski,Paul W. R. Harris,Cecilia Aragon +9 more
TL;DR: A pipeline of natural language processing and machine learning techniques that can be used to build automated classifiers of affect in chat logs are presented and can successfully identify many commonly occurring types of affect.
Journal ArticleDOI
Using Twitter as a means of coping with emotions and uncontrollable crises
TL;DR: This article analyzed 818 tweets obtained immediately after a repeat crisis and analyzed each tweet according to the elements of perceived controllability and predictability, as well as the emotions and coping strategies used on behalf of the organization's external stakeholders.
Journal ArticleDOI
"How We Know What We Know": A Systematic Comparison of Research Methods Employed in Higher Education Journals, 1996−2000 v. 2006−2010
TL;DR: This article examined the content of three major higher education journals from 1996 to 2000 and found that the field of knowledge production in higher education is increasingly dominated by quantitative methods and which is increasingly using more advanced statistical techniques.
References
More filters
Journal ArticleDOI
Coefficient alpha and the internal structure of tests.
TL;DR: In this paper, a general formula (α) of which a special case is the Kuder-Richardson coefficient of equivalence is shown to be the mean of all split-half coefficients resulting from different splittings of a test, therefore an estimate of the correlation between two random samples of items from a universe of items like those in the test.
Book
An introduction to the bootstrap
Bradley Efron,Robert Tibshirani +1 more
TL;DR: This article presents bootstrap methods for estimation, using simple arguments, with Minitab macros for implementing these methods, as well as some examples of how these methods could be used for estimation purposes.
Book
Nonparametric statistics for the behavioral sciences
TL;DR: This is the revision of the classic text in the field, adding two new chapters and thoroughly updating all others as discussed by the authors, and the original structure is retained, and the book continues to serve as a combined text/reference.
Journal ArticleDOI
A Coefficient of agreement for nominal Scales
TL;DR: In this article, the authors present a procedure for having two or more judges independently categorize a sample of units and determine the degree, significance, and significance of the units. But they do not discuss the extent to which these judgments are reproducible, i.e., reliable.
Book
Content analysis: an introduction to its methodology
TL;DR: History Conceptual Foundations Uses and Kinds of Inference The Logic of Content Analysis Designs Unitizing Sampling Recording Data Languages Constructs for Inference Analytical Techniques The Use of Computers Reliability Validity A Practical Guide