Evaluating Subjective Feedback for Internet of Things Dialogues (bibtex)
by Carla Gordon, Kallirroi Georgila, Hyungtak Choi, Jill Boberg, David Traum
Abstract:
This paper discusses the process of determining which subjective features are seen as ideal in a dialogue system, and linking these features to objectively quantifiable behaviors. A corpus of simulated system-user dialogues in the Internet of Things domain was manually annotated with a set of system communicative and action responses, and crowd-sourced ratings and qualitative feedback of these dialogues were collected. This corpus of subjective feedback was analyzed, revealing that raters described top ranked dialogues as Intelligent, Natural, Pleasant, and as having Personality. Additionally, certain communicative and action responses were statistically more likely to be present in dialogues described as having these features. There was also found to be a lack of agreement among raters as to whether a direct communication style, or a conversational one was preferred, suggesting that future research and development should consider creating models for different communication styles.
Reference:
Evaluating Subjective Feedback for Internet of Things Dialogues (Carla Gordon, Kallirroi Georgila, Hyungtak Choi, Jill Boberg, David Traum), In Proceedings of the 22nd Workshop on the Semantics and Pragmatics of Dialogue, 2018.
Bibtex Entry:
@inproceedings{gordon_evaluating_2018,
	address = {Aix-en-Provence, France},
	title = {Evaluating {Subjective} {Feedback} for {Internet} of {Things} {Dialogues}},
	url = {https://amubox.univ-amu.fr/s/6YcAg3TpLpfzGEn#pdfviewer},
	abstract = {This paper discusses the process of determining which subjective features are seen as ideal in a dialogue system, and linking these features to objectively quantifiable behaviors. A corpus of simulated system-user dialogues in the Internet of Things domain was manually annotated with a set of system communicative and action responses, and crowd-sourced ratings and qualitative feedback of these dialogues were collected. This corpus of subjective feedback was analyzed, revealing that raters described top ranked dialogues as Intelligent, Natural, Pleasant, and as having Personality. Additionally, certain communicative and action responses were statistically more likely to be present in dialogues described as having these features. There was also found to be a lack of agreement among raters as to whether a direct communication style, or a conversational one was preferred, suggesting that future research and development should consider creating models for different communication styles.},
	booktitle = {Proceedings of the 22nd {Workshop} on the {Semantics} and {Pragmatics} of {Dialogue}},
	author = {Gordon, Carla and Georgila, Kallirroi and Choi, Hyungtak and Boberg, Jill and Traum, David},
	month = nov,
	year = {2018},
	keywords = {UARC, Virtual Humans},
	pages = {64--72}
}
Powered by bibtexbrowser