Inter-Coder Agreement for Computational Linguistics (bibtex)
by Artstein, Ron and Poesio, Massimo
Abstract:
This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in Computational Linguistics, may be more appropriate for many corpus annotation tasks – but that their use makes the interpretation of the value of the coefficient even harder.
Reference:
Inter-Coder Agreement for Computational Linguistics (Artstein, Ron and Poesio, Massimo), In Computational Linguistics, volume 34, 2008.
Bibtex Entry:
@article{artstein_inter-coder_2008,
	title = {Inter-{Coder} {Agreement} for {Computational} {Linguistics}},
	volume = {34},
	url = {http://ict.usc.edu/pubs/Inter-Coder%20Agreement%20for%20Computational%20Linguistics.pdf},
	abstract = {This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in Computational Linguistics, may be more appropriate for many corpus annotation tasks – but that their use makes the interpretation of the value of the coefficient even harder.},
	number = {4},
	journal = {Computational Linguistics},
	author = {Artstein, Ron and Poesio, Massimo},
	month = dec,
	year = {2008},
	pages = {555--596}
}
Powered by bibtexbrowser