An Assessment Framework for DialPort (bibtex)
by Kyusong Lee, Tiancheng Zhao, Stefan Ultes, Lina Rojas-Barahona, Eli Pincus, David Traum, Maxine Eskenazi
Abstract:
Collecting a large amount of real human-computer interaction data in various domains is a cornerstone in the development of better data-driven spoken dialog systems. The DialPort project is creating a portal to collect a constant stream of real user conversational data on a variety of topics. In order to keep real users attracted to DialPort, it is crucial to develop a robust evaluation framework to monitor and maintain high performance. Different from earlier spoken dialog systems, Dial-Port has a heterogeneous set of spoken dialog systems gathered under one outwardlooking agent. In order to access this new structure, we have identified some unique challenges that DialPort will encounter so that it can appeal to real users and have created a novel evaluation scheme that quantitatively assesses their performance in these situations. We look at assessment from the point of view of the system developer as well as that of the end user.
Reference:
An Assessment Framework for DialPort (Kyusong Lee, Tiancheng Zhao, Stefan Ultes, Lina Rojas-Barahona, Eli Pincus, David Traum, Maxine Eskenazi), In Proceedings of the International Workshop on Spoken Dialogue Systems (IWSDS), IWSDS, 2017.
Bibtex Entry:
@inproceedings{lee_assessment_2017,
	address = {Farmington, PA},
	title = {An {Assessment} {Framework} for {DialPort}},
	url = {https://www.uni-ulm.de/fileadmin/website_uni_ulm/iui.iwsds2017/papers/IWSDS2017_paper_1.pdf},
	abstract = {Collecting a large amount of real human-computer interaction data in various domains is a cornerstone in the development of better data-driven spoken dialog systems. The DialPort project is creating a portal to collect a constant stream of real user conversational data on a variety of topics. In order to keep real users attracted to DialPort, it is crucial to develop a robust evaluation framework to monitor and maintain high performance. Different from earlier spoken dialog systems, Dial-Port has a heterogeneous set of spoken dialog systems gathered under one outwardlooking agent. In order to access this new structure, we have identified some unique challenges that DialPort will encounter so that it can appeal to real users and have created a novel evaluation scheme that quantitatively assesses their performance in these situations. We look at assessment from the point of view of the system developer as well as that of the end user.},
	booktitle = {Proceedings of the {International} {Workshop} on {Spoken} {Dialogue} {Systems} ({IWSDS})},
	publisher = {IWSDS},
	author = {Lee, Kyusong and Zhao, Tiancheng and Ultes, Stefan and Rojas-Barahona, Lina and Pincus, Eli and Traum, David and Eskenazi, Maxine},
	month = jun,
	year = {2017},
	keywords = {Virtual Humans}
}
Powered by bibtexbrowser