An Assessment Framework for DialPort (bibtex)
by Lee, Kyusong, Zhao, Tiancheng, Ultes, Stefan, Rojas-Barahona, Lina, Pincus, Eli, Traum, David and Eskenazi, Maxine
Abstract:
Collecting a large amount of real human-computer interaction data in various domains is a cornerstone in the development of better data-driven spoken dialog systems. The DialPort project is creating a portal to collect a constant stream of real user conversational data on a variety of topics. In order to keep real users attracted to DialPort, it is crucial to develop a robust evaluation framework to monitor and maintain high performance. Different from earlier spoken dialog systems, DialPort has a heterogeneous set of spoken dialog systems gathered under one outward-looking agent. In order to access this new structure, we have identified some unique challenges that DialPort will encounter so that it can appeal to real users and have created a novel evaluation scheme that quantitatively assesses their performance in these situations. We look at assessment from the point of view of the system developer as well as that of the end user.
Reference:
An Assessment Framework for DialPort (Lee, Kyusong, Zhao, Tiancheng, Ultes, Stefan, Rojas-Barahona, Lina, Pincus, Eli, Traum, David and Eskenazi, Maxine), Chapter in Advanced Social Interaction with Agents, Springer International Publishing, volume 510, 2019.
Bibtex Entry:
@incollection{lee_assessment_2019,
	address = {Cham},
	title = {An {Assessment} {Framework} for {DialPort}},
	volume = {510},
	isbn = {978-3-319-92107-5 978-3-319-92108-2},
	url = {http://link.springer.com/10.1007/978-3-319-92108-2_10},
	abstract = {Collecting a large amount of real human-computer interaction data in various domains is a cornerstone in the development of better data-driven spoken dialog systems. The DialPort project is creating a portal to collect a constant stream of real user conversational data on a variety of topics. In order to keep real users attracted to DialPort, it is crucial to develop a robust evaluation framework to monitor and maintain high performance. Different from earlier spoken dialog systems, DialPort has a heterogeneous set of spoken dialog systems gathered under one outward-looking agent. In order to access this new structure, we have identified some unique challenges that DialPort will encounter so that it can appeal to real users and have created a novel evaluation scheme that quantitatively assesses their performance in these situations. We look at assessment from the point of view of the system developer as well as that of the end user.},
	language = {en},
	urldate = {2019-10-28},
	booktitle = {Advanced {Social} {Interaction} with {Agents}},
	publisher = {Springer International Publishing},
	author = {Lee, Kyusong and Zhao, Tiancheng and Ultes, Stefan and Rojas-Barahona, Lina and Pincus, Eli and Traum, David and Eskenazi, Maxine},
	month = jun,
	year = {2019},
	doi = {10.1007/978-3-319-92108-2_10},
	keywords = {Virtual Humans},
	pages = {79--85}
}
Powered by bibtexbrowser