Journal ArticleDOI
Answering the Call for a Standard Reliability Measure for Coding Data
TLDR
This work proposes Krippendorff's alpha as the standard reliability measure, general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data.Abstract:
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.read more
Citations
More filters
Journal ArticleDOI
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
TL;DR: This paper provides an overview of methodological issues related to the assessment of IRR with a focus on study design, selection of appropriate statistics, and the computation, interpretation, and reporting of some commonly-used IRR statistics.
Journal ArticleDOI
Inter-coder agreement for computational linguistics
TL;DR: It is argued that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder.
Posted Content
The Value of Crowdsourcing: Can Users Really Compete with Professionals in Generating New Product Ideas?
Marion Poetz,Martin Schreier +1 more
TL;DR: It is suggested that, at least under certain conditions, crowdsourcing might constitute a promising method to gather user ideas which can complement those of a firm’s professionals at the idea generation stage in NPD.
Journal ArticleDOI
Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines
Cliodhna O'Connor,Helene Joffe +1 more
TL;DR: In this paper, the intercoder reliability of a coding frame is evaluated as a good practice in qualitative analysis, and the ICR is a somewhat controversial topic in the qualitative research community.
Journal ArticleDOI
The Value of Crowdsourcing: Can Users Really Compete with Professionals in Generating New Product Ideas?
Marion Poetz,Martin Schreier +1 more
TL;DR: In this paper, a real-world comparison of ideas actually generated by a firm's professionals with those generated by users in the course of an idea generation contest is presented, which suggests that, at least under certain conditions, crowdsourcing might constitute a promising method to gather user ideas that can complement those of a firm' professionals at the idea generation stage in NPD.
References
More filters
Journal ArticleDOI
Coefficient alpha and the internal structure of tests.
TL;DR: In this paper, a general formula (α) of which a special case is the Kuder-Richardson coefficient of equivalence is shown to be the mean of all split-half coefficients resulting from different splittings of a test, therefore an estimate of the correlation between two random samples of items from a universe of items like those in the test.
Book
An introduction to the bootstrap
Bradley Efron,Robert Tibshirani +1 more
TL;DR: This article presents bootstrap methods for estimation, using simple arguments, with Minitab macros for implementing these methods, as well as some examples of how these methods could be used for estimation purposes.
Book
Nonparametric statistics for the behavioral sciences
TL;DR: This is the revision of the classic text in the field, adding two new chapters and thoroughly updating all others as discussed by the authors, and the original structure is retained, and the book continues to serve as a combined text/reference.
Journal ArticleDOI
A Coefficient of agreement for nominal Scales
TL;DR: In this article, the authors present a procedure for having two or more judges independently categorize a sample of units and determine the degree, significance, and significance of the units. But they do not discuss the extent to which these judgments are reproducible, i.e., reliable.
Book
Content analysis: an introduction to its methodology
TL;DR: History Conceptual Foundations Uses and Kinds of Inference The Logic of Content Analysis Designs Unitizing Sampling Recording Data Languages Constructs for Inference Analytical Techniques The Use of Computers Reliability Validity A Practical Guide