Non-Photorealistic Computer Graphics Library

Speech-Driven Cartoon Animation with Emotions

Author(s): Yan Li, Feng Yu, Ying-Qing Xu, Eric Chang, Heung-Yeung Shum.
Proceedings: ACM Multimedia, pp. 365--371, Ottawa, Canada, Sep-Oct, 2001.
[BibTeX]

Abstract:
In this paper, we present a cartoon face animation system for multimedia HCI applications. We animate face cartoons not only from input speech, but also based on emotions derived from speech signal. Using a corpus of over 700 utterances from different speakers, we have trained SVMs (support vector machines) to recognize four categories of emotions: neutral, happiness, anger and sadness. Given each input speech phrase, we identify its emotion content as a mixture of all four emotions, rather than classifying it into a single emotion. Then, facial expressions are generated from the recovered emotion for each phrase, by morphing different cartoon templates that correspond to various emotions. To ensure smooth transitions in the animation, we apply lowpass filtering to the recovered (and possibly jumpy) emotion sequence. Moreover, lip-syncing is applied to produce the lip movement from speech, by recovering a statistical audiovisual mapping. Experimental results demonstrate that cartoon animation sequences generated by our system are of good and convincing quality.