Eliza Margaretha, David DeVault: ” An Approach to the Automated Evaluation of Pipeline Architectures in Natural Language Dialogue Systems”

June 17, 2011 | Portland, OR

Speaker: Eliza Margaretha, David DeVault
Host: The 12th Annual SIGdial Meeting on Discourse and Dialogue

We present an approach to performing automated evaluations of pipeline architectures in natural language dialogue systems. Our approach addresses some of the difficulties that arise in such automated evaluations, including the lack of consensus among human annotators about the correct outputs within the processing pipeline, the availability of multiple acceptable system responses to some user utterances, and the complex relationship between system responses and internal processing results. Our approach includes the development of a corpus of richly annotated target dialogues, simulations of the pipeline processing that could occur in these dialogues, and an analysis of how system responses vary based on internal processing results within the pipeline. We illustrate our approach, and the kinds of insights it can provide into system performance, in two implemented virtual human dialogue systems.