Facial expression of emotions by a virtual narrator for children

Facial expression of emotions by a virtual narrator for children

Nesrine Fourati Adeline Richard Nicolas Sabouret Jean-Claude Martin Emilie Chanoni Céline Clavel 

LIMSI, CNRS, Univ. Paris-Sud, Université Paris-Saclay, Bat 508, rue von Neuman, 91405 Orsay Cedex, France

PSY-NCA, Université de Rouen, 76000 Rouen, France

Corresponding Author Email: 
fourati@limsi.f; nicolas.sabouret@limsi.f; jean-claude.martin@limsi.f; celine.clavel@limsi.f; adeline.richard@etu.univ-rouen.fr; emilie.chanoni@etu.univ-rouen.fr
31 October 2017
| Citation

This paper focuses on the development of an expressive virtual storyteller for children and raises the challenges encountered to endow a virtual agent with storytelling capabilities. We focus on facial expressions of emotions and we present our computational model based on an appraisal theory of emotion. We also discuss different viewpoints a storyteller can adopt to evaluate the expressive content of a story event. We describe a preliminary study that we conducted with 23 children. We discuss the impact of facial expressions of appraisals on children’s memorization of story events, their perception of characters’ appraisals, their subjective perception of the virtual storyteller and more generally how emotion combines with joint attention.


expressive virtual storyteller, facial expression, appraisal

1. Introduction
2. Méthodologie
3. Étude exploratoire
4. Conclusion et discussion

Cette recherche a bénéficié du soutien du programme de l’Agence Nationale de la Recherche (ANR), projet NARECA (ANR-13-CORD-0015).

Les auteurs tiennent à remercier les enfants et leurs parents pour avoir accepté de participer à cette étude ainsi que le directeur du centre de loisirs et les animateurs pour leur accueil et leur contribution au bon déroulement de l’étude. Enfin, les auteurs souhaitent remercier Matthieu Courgeon pour l’aide apportée dans le développement du conteur virtuel.


Adam C., Cavedon L. (2013).Once upon a time... a companion robot that can tell stories. Poster paper. In Intelligent Virtual Agents, Thirteenth International Conference, IVA, Edinburgh, UK.

Atkinson RK. (2002). Optimizing learning from examples using animated pedagogical agents. Journal of Educational Psychology, vol. 94, no 2, 416-427. doi:10.1037/0022-0663.94.2.416

Bartneck C., Croft E., Kulic D., Zoghbi S. (2009). Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. International Journal of Social Robotics, vol. 1, no 1,71-81. doi: 10.1007/s12369-008-0001-3

Bickmore T., Trinh H., Hoppmann M., Asadi R. (2016).Virtual Agents in the Classroom: Experience Fielding a Co-Presenter Agent in University Courses, In Sixteenth International Conference on Intelligent Virtual Agents, IVA, Los Angeles, California.

Bleackley P., Hyniewska S., Niewiadomski R., Pelachaud, C. Price M., (2010) Emotional Interaction Storyteller System, KEER International Conference on KANSEI Engineering and Emotion Research, Paris, 2-4 March 2010, pp. 1716-1726.

Bosse T., Pontier M., Siddiqui G. F., Treur J., (2007). Incorporating emotion regulation into virtual stories. In Intelligent Virtual Agents, 7th International Conference, IVA 2007, Paris, France, September 17-19, 2007, Proceedings, p. 339–347, 2007.

Bruner J. (1990). Car la culture donne forme à l’esprit. De la révolution cognitive à la révolution culturelle, Paris, ESHEL.

Bruner J. (1996). L’éducation, entrée dans la culture. Paris, Retz

Bruner J. (2000). Culture et modes de pensée. Paris : Retz

Bruner J. (1983). Le développement de l’enfant : Savoir faire savoir dire. Paris, PUF.

Buisine S., Wang Y., Grynszpan O., (2009). Empirical investigation of the temporal relations between speech and facial expressions of emotion. Journal on Multimodal User Interfaces, vol. 3, n° 4, p. 263–270.

Callaway C. B., Lester J. C. (2002). Narrative prose generation. Artificial Intelligence, vol. 139, n° 2, p. 213-252.

Carrera-Levillain P., Fernandez-Dols J. M., (1994). Neutral faces in context: Their emotional meaning and their function. Journal of Nonverbal Behavior, vol. 18, n°4, p. 281–299.

Cavazza M., Charles F., Mead S. J. (2002). Character-based interactive storytelling. In IEEE Intelligent Systems, vol. 17.

Chella A., Barone R.E., Pilato G., Sorbello R. (2008). An Emotional Storyteller Robot. AAAI Spring Symposium: Emotion, Personality, and Social Behavior 2008, 17-22

Courgeon M., Clavel C., Martin JC. (2014). Modeling Facial Signs of Appraisal During Interaction: Impact on Users’ Perception and Behavior. AAMAS vol. 59, no 1, 765-772

Courgeon M., Martin J.C., Jacquemin C. (2008). Marc: a multimodal affective and reactive character. In: Proceedings of the 1st Workshop on AFFective Interaction in Natural Environments

Crahay M. (1999). Psychologie de l’éducation. Paris, Presses universitaires de France.

De Jong M.T., Bus A.G. (2004). The efficacy of electronic books in fostering kindergarten children’s emergent story understanding. Reading Research Quarterly, vol. 39, p. 378-393.

Doty D.E., Popplewell S.R., Byers G.O. (2001). Interactive CD-ROM storybooks and young readers’ reading comprehension. Journal of Research on Computing in Education, vol. 33, p. 374-384.

Ekman P, Friesen W. (1975). Unmasking the Face: A guide to recognizing emotions from facial clues. Prentice-Hall, Inc.

Fridin M. (2014). Storytelling by a kindergarten social assistive robot: A tool for constructive learning in preschool education. Computers & Education, vol. 70, p. 53-64

Frijda N. (1993). The place of appraisal in emotion. Cognition and Emotion, vol. 7, no 3-4, 357-387;

Guo Y.R, Hoe-Lian Goh D., Luyt B., Sin SC., Ang R.P. (2015). The effectiveness and acceptance of an affective information literacy tutorial, Computers & Education, vol. 87, p. 368-384.

Kory J., Breazeal C. (2014). Storytelling with Robots: Learning Companions for Preschool Children’s Language Development. In P. A. Vargas & R. Aylett (Eds.), Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE: Washington.

Lester JC., Converse SA., Kahler SE., Todd Barlow S., Stone BA., Bhogal RS. (1997). The persona effect: affective impact of animated pedagogical agents, Proceedings of the ACM SIGCHI Conference on Human factors in computing systems, p.359-366, March 22-27, 1997, Atlanta, Georgia, USA.

Marsella S.C., Gratch J., Petta P. (2010). Computational Models of Emotion. In: Scherer, K.R., Banziger, T., Roesch (eds.) A blueprint for affective computing: A sourcebook and manual, p. 21–46. Oxford University Press, Oxford

Marsella S.C., Gratch J. (2009). EMA: A process model of appraisal dynamics. Cognitive Systems Research, vol. 10, no 1, p. 70-90.

Martin JC., Deviller L., Raouzaiou A., Caridakis G. (2011). Coordinating the generation of signs in multiple modalities in an affective agent. Emotion-Oriented Systems: The Humaine Handbook,Cognitive Technologies, p. 349-367.

Miletitch, R., Sabouret N., Ochs M. (2012). Susciter l’émotion dans la narration automatique, In Technique et Science de l’Informatique (TSI), vol. 31, n° 4.

Moody A.K., Justice L.M., Cabell S.Q. (2010). Electronic versus traditional storybooks: relative influence on preschool children’s engagement and communication. Journal of Early Childhood Literacy, vol. 10, n° 3, p. 294-313.

Mutlu B., Forlizzi J., Hodgins, J. (2006). A storytelling robot: Modeling and evaluation of human-like gaze behavior. In G. Sandini & A. Billard (Eds.), Proceedings of the 6th IEEE-RAS International Conference on Humanoid Robots, p. 518-523.

Noorhidawati A., Ghazal Ghalebandi S., Siti Hajar R. (2015). How do young children engage with mobile apps? Cognitive, psychomotor, and affective perspective, Computers & Education archive, vol. 87, n°C, p. 385-395.

Oatley K. (2004). Scripts, Transformations, and Suggestiveness of Emotions in Shakespeare and Chekhov. Review of General Psychology, vol. 8, n°4, p. 323-340.

Oberlander J., Robertson J. (2002). Ghostwriter: Educational Drama and Presence in a Virtual Environment. Journal Computer-Mediated Communication, vol. 8.

Ochs M., Niewiadomski R., Pelachaud C. (2010). How a virtual agent should smile? Morphological and dynamic chacteristics of virtual agent’s smiles. In Intelligent Virtual Agents, 10th International Conference, IVA, p. 427-440, Philadelphia, Pennsylvania, USA.

Raimundo G., Cabral J., Melo C., Oliveira LC., Paiva A. (2007). Telling Stories with a Synthetic Character: Understanding Inter-modalities Relations, COST workshop, Verbal and Nonverbal Communication Behaviours Lecture Notes in Computer Science, vol. 4775/2007.

Ricci C.M., Beal C.R. (2002). The effect of interactive media on children’s story memory. Journal of Educational Psychology, vol. 94, n° 1, p. 138-144.

Riedl M. O. (2004). Narrative generation: balancing plot and character. North Carolina State University.

Rimé B. (1997). Cognition et émotion in J.-P. Leyens & J.-L. Beauvois (Eds.), La psychologie sociale 3 : L’ère de la cognition Grenoble, p. 107-125, Presses Universitaires de Grenoble.

Scherer K. (2001). Appraisal considered as a process of multilevel sequential checking. Dans K. Scherer, A. Schorr, & T. Johnstone, Appraisal processes in emotion (pp. 92-121). New York, Oksford University Press.

Serban O., Pauchet A., Bersoult A., Chanoni E., (2013). Tell Me a Story : A Comparative Study of Child Interaction with Virtual Characters and Adults. R. Aylett et al. (Eds.), 13th international conference on Intelligent Virtual Agents (IVA), LNAI 8108, pp. 459-460, Springer, Edinburgh, UK, 2013.

Silva A., Raimundo G., Paiva A. (2003). Tell me that Bit Again... Bringing Interactivity to a Virtual Storyteller, Proceedings of the 2nd International Conference on Virtual Storytelling - ICVS 2003.

Silva A., Vala M., Paiva A. (2001). Papous: The Virtual Storyteller In Intelligent Virtual Agents, International Conference IVA 2001, Springer.

Swartjes I. M. T., Theune M. (2008). The virtual storyteller: Story generation by simulation.

Szafir D., Mutlu B. (2012). Pay attention!: designing adaptive agents that monitor and improve user engagement, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, May 05-10, 2012, Austin, Texas, USA.

Theune M., Linssen J., Alofs T. (2013). Acting, Playing, or Talking about the Story : An Annotation Scheme for Communication during Interactive Digital Storytelling. 6th International Conference on Interactive Digital Storytelling, p. 132-143.

Vygotski L. (1997). Pensée et Langage. Paris, La Dispute.