Evaluating Story Generation Systems Using Automated Linguistic Analyses (bibtex)
by Melissa Roemmele, Andrew S. Gordon, Reid Swanson
Abstract:
Story generation is a well-recognized task in computational creativity research, but one that can be di⬚cult to evaluate empirically. It is often ine⬚cient and costly to rely solely on human feedback for judging the quality of generated stories. We address this by examining the use of linguistic analyses for automated evaluation, using metrics from existing work on predicting writing quality. We apply these metrics speci⬚cally to story continuation, where a model is given the beginning of a story and generates the next sentence, which is useful for systems that interactively support authors' creativity in writing. We compare sentences generated by different existing models to human-authored ones according to the analyses. The results show some meaningful dfferences between the models, suggesting that this evaluation approach may be advantageous for future research.
Reference:
Evaluating Story Generation Systems Using Automated Linguistic Analyses (Melissa Roemmele, Andrew S. Gordon, Reid Swanson), In Proceedings of the SIGKDD-2017 Workshop on Machine Learning for Creativity, ACM, 2017.
Bibtex Entry:
@inproceedings{roemmele_evaluating_2017,
	address = {Halifax, Nova Scotia, Canada},
	title = {Evaluating {Story} {Generation} {Systems} {Using} {Automated} {Linguistic} {Analyses}},
	url = {http://people.ict.usc.edu/~roemmele/publications/fiction_generation.pdf},
	abstract = {Story generation is a well-recognized task in computational creativity research, but one that can be di⬚cult to evaluate empirically. It is often ine⬚cient and costly to rely solely on human feedback for judging the quality of generated stories. We address this by examining the use of linguistic analyses for automated evaluation, using metrics from existing work on predicting writing quality. We apply these metrics speci⬚cally to story continuation, where a model is given the beginning of a story and generates the next sentence, which is useful for systems that interactively support authors' creativity in writing. We compare sentences generated by different existing models to human-authored ones according to the analyses. The results show some meaningful dfferences between the models, suggesting that this evaluation approach may be advantageous for future research.},
	booktitle = {Proceedings of the {SIGKDD}-2017 {Workshop} on {Machine} {Learning} for {Creativity}},
	publisher = {ACM},
	author = {Roemmele, Melissa and Gordon, Andrew S. and Swanson, Reid},
	month = aug,
	year = {2017},
	keywords = {Narrative, UARC}
}
Powered by bibtexbrowser