Inter-Coder Agreement for Computational Linguistics (bibtex)

by Artstein, Ron and Poesio, Massimo

Abstract:

This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in Computational Linguistics, may be more appropriate for many corpus annotation tasks – but that their use makes the interpretation of the value of the coefficient even harder.

Reference:

Inter-Coder Agreement for Computational Linguistics (Artstein, Ron and Poesio, Massimo), In Computational Linguistics, volume 34, 2008.

Bibtex Entry:

@article{artstein_inter-coder_2008, title = {Inter-{Coder} {Agreement} for {Computational} {Linguistics}}, volume = {34}, url = {http://ict.usc.edu/pubs/Inter-Coder%20Agreement%20for%20Computational%20Linguistics.pdf}, abstract = {This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in Computational Linguistics, may be more appropriate for many corpus annotation tasks – but that their use makes the interpretation of the value of the coefficient even harder.}, number = {4}, journal = {Computational Linguistics}, author = {Artstein, Ron and Poesio, Massimo}, month = dec, year = {2008}, pages = {555--596} }

Powered by bibtexbrowser