Ron Artstein: “Evaluation of semantic corpus annotation”

August 1, 2007 | USC ICT

Speaker: Ron Artstein

Semantically annotated corpora are intended to capture semantic relations among elements in a text, but it is not known how uniform human intuitions are about such semantic relations. I present a novel methodology for evaluating the process of semantic corpus annotation, which is applicable in domains where we expect naive participants to have valid judgments, such as deciding whether two expressions refer to the same object. The annotation process is tested in experiments where many naive annotators mark the same text, based on a compact set of guidelines and minimal training.

The limited guidelines allow annotators to express their own judgments rather than rely on mechanical rule application, and the large number of participants yields a variety of responses which capture the range of possible judgments better than traditional comparisons of two or three highly trained annotators. The result is a somewhat noisy dataset which is evaluated both qualitatively, by looking at instances of disagreement, and quantitatively, using formal reliability statistics. The evaluation identifies difficult areas in the annotation scheme and guidelines, and also specific linguistic constructions that are not well defined with respect to the annotation. The results of such experiments were used to improve the annotation scheme, guidelines, and task definition of the Arrau corpus of anaphoric relations created at the University of Essex.