Ethical judgment in the decision process of a BDI agent

Ethical judgment in the decision process of a BDI agent

Nicolas Cointe Grégory Bonnet Olivier Boissier 

Institut Henri Fayol, Laboratoire Hubert Curien, UMR CNRS 5516, Mines Saint-Étienne, Saint-Étienne, 42023 France

Équipe Modèle Agent Décision GREYC, département Intelligence Artificielle et Algorithmique CNRS UMR 6072 F-14032, Normandie Université, Caen, France

Corresponding Author Email:;;
31 August 2017
| Citation



The increasing use of multi-agent technologies in various areas raises the necessity of designing agents that are able to take decisions based on ethical principles. More and more works propose such approaches. However, those systems consider mainly an agent-centered perspective, letting aside the fact that agents are in interaction with other artificial agents or human beings that can use other ethical concepts. In this article, we address this problem and propose a model of ethical judgment an agent can use in order to judge the ethical dimension of both its own behavior and the other agents’ behaviors. This model is based on a rationalist and explicit approach that distinguishes theory of good and theory of right. A proof-of-concept implemented in the multi-agent oriented programming platform JaCaMo and based on a simple scenario is given to illustrate those functionalities.


agent (architecture), multi-agent ethics

1. Introduction
2. Éthique et agents autonomes
3. Modèle de jugement éthique
4. Usage du modèle de jugement éthique
5. Preuve de concept
6. Conclusion

Les auteurs remercient l’Agence Nationale de la Recherche (ANR) pour sa contribution financière sous la référence ANR-13-CORD-0006.


Aldridge I. (2009). High-frequency trading: a practical guide to algorithmic strategies and trading systems (vol. 459). John Wiley and Sons.

Alexander L., Moore M. (2015). Deontological Ethics. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy, Spring 2015 éd..

Anderson M., Anderson S. (2014). Toward ensuring ethical behavior from autonomous systems: a case-supported principle-based paradigm. Industrial Robot, vol. 42, no 4, p. 324-331.

Arkin R. (2009). Governing lethal behavior in autonomous robots. CRC Press.

Arkoudas K., Bringsjord S., Bello P. (2005). Toward ethical robots via mechanized deontic logic. In AAAI fall symposium on machine ethics, p. 17–23.

Battaglino C., Damiano R., Lesmo L. (2013). Emotional range in value-sensitive deliberation.

In 12th international conference on autonomous agents and multi-agent systems, p. 769–776.

Berreby F., Bourgne G., Ganascia J.-G. (2015). Modelling moral reasoning and ethical responsibility with logic programming. In 20th international conference on logic for programming, artificial intelligence, and reasoning, p. 532-548.

Berreby F., Bourgne G., Ganascia J.-G. (2017). A declarative modular framework for representing and applying ethical principles. In Proceedings of the 16th conference on autonomous agents and multiagent systems, p. 96–104.

Boella G., Pigozzi G., Torre L. van der. (2009). Normative systems in computer science - Ten guidelines for normative multiagent systems. In Normative multi-agent systems.

Boissier O., Bordini R. H., Hübner J. F., Ricci A., Santi A. (2013). Multi-agent oriented programming with JaCaMo. Science of Computer Programming, vol. 78, no 6, p. 747–761.

Bono S., Bresin G., Pezzolato F., Ramelli S., Benseddik F. (2013). Green, social and ethical funds in Europe. Rapport technique. Vigeo.

Bratman M. (1987). Intention, plans, and practical reason.

Coelho H., Rocha Costa A. da. (2009, October). On the intelligence of moral agency. Encontro Português de Inteligência Artificial, p. 12–15.

Coelho H., Trigo P., Rocha Costa A. da. (2010). On the operationality of moral-sense decision making. In 2nd brazilian workshop on social simulation, p. 15–20.

Cointe N., Bonnet G., Boissier O. (2016a). Ethical judgment of agents’ behaviors in multi-agent systems. In 15th international conference on autonomous agents & multiagent systems, p. 1106-1114.

Cointe N., Bonnet G., Boissier O. (2016b). Jugement éthique dans les systèmes multi-agents. In Journées francophones sur les systèmes multi-agents.

Cointe N., Bonnet G., Boissier O. (2017). Coopération fondée sur l’éthique entre agents autonomes. In Journées francophones sur les systèmes multi-agents.

Damasio A. (2008). Descartes’ error: Emotion, reason and the human brain. Random House.

Dennis L., Fisher M., Winfield A. (2015). Towards verifiably ethical robot behaviour. In 1st international workshop on AI and ethics.

Ethical judgment. (2015, August). Free Online Psychology Dictionary.

Foot P. (1967). The problem of abortion and the doctrine of the double effect. Oxford Review, p. 5–15.

Friedman B. (1996). Value-sensitive design. Interactions, vol. 3, no 6, p. 16-23.

Friedman B., Kahn P., Borning A., Huldtgren A. (2013). Value sensitive design and information systems. In Early engagement and new technologies: Opening up the laboratory, p. 55-95. Springer Netherlands.

Ganascia J.-G. (2007a). Ethical system formalization using non-monotonic logics. In 29th annual conference of the cognitive science society, p. 1013–1018.

Ganascia J.-G. (2007b). Modelling ethical rules of lying with Answer Set Programming. Ethics and information technology, vol. 9, no 1, p. 39–47.

Gert B. (2015). The definition of morality. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy, Fall éd..

Greene J., Haidt J. (2002). How (and where) does moral judgment work? Trends in cognitive sciences, vol. 6, no 12, p. 517–523.

Hursthouse R. (2013). Virtue ethics. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy, Fall éd..

Johnson R. (2014). Kant’s moral philosophy. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy, Summer éd..

Kant E. (1967). Sur un prétendu droit de mentir par humanité (1797). Théorie et pratique/Droit de mentir, p. 67–73.

Kim K.-J., Lipson H. (2009). Towards a theory of mind in simulated robots. In 11th annual conference companion on genetic and evolutionary computation conference, p. 2071–2076.

Koopman G., Székely I. (2009). Impact of the current economic and financial crisis on potential output. European Economy, Occasional Paper, no 49.

MaoW., Gratch J. (2013). Modeling social causality and responsibility judgment in multi-agent interactions. In 23rd international joint conference on artificial intelligence, p. 3166–3170.

McConnell T. (2014). Moral dilemmas. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy, Fall éd..

McIntyre A. (2014). Doctrine of double effect. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy, Winter éd..

Professional Journalists S. of. (2014, September). Code of ethics.

Ricoeur P. (1995). Oneself as another. University of Chicago Press.

Rocha-Costa A. (2016). Moral systems of agent societies: Some elements for their analysis and design. In 1st workshop on ethics in the design of intelligent agents, p. 32-37.

Rokeach M. (1973). The nature of human values. New York Free Press.

Saptawijaya A., L. Moniz Pereira L. M. (2014). Towards modeling morality computationally with logic programming. In Practical aspects of declarative languages, p. 104–119.

Scheff T. J. (2003). Shame in self and society. Symbolic interaction, vol. 26, no 2, p. 239–262.

Schroeder M. (2016). Value theory. In E. N. Zalta (Ed.), The stanford encyclopedia of philosophy, Fall 2016 éd.. Metaphysics Research Lab, Stanford University.

Schwartz S. (2012). An overview of Schwartz theory of basic values. Online Readings of Psychology and Culture, vol. 2, no 1.

Schwartz S. H. (2006). Basic human values: Theory, measurement, and applications. Revue française de sociologie, vol. 47, no 4, p. 249–288.

Schwartz S. H., Bilsky W. (1990). Toward a theory of the universal content and structure of values: Extensions and cross cultural replications. Journal of Personality and Social Psychology, vol. 58, p. 878-891.

Stoeber J., Yang H. (2016). Moral perfectionism and moral values, virtues, and judgments: Further investigations. Personality and Individual Differences, vol. 88, p. 6–11.

Timmons M. (2012). Moral theory: an introduction. Rowman & Littlefield Publishers.

van Marrewijk M., Werre M. (2003). Multiple levels of corporate sustainability. Journal of Business Ethics, vol. 4, no 2-3, p. 107-119.

Walter S. (2015). Consequentialism. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy, Winter éd..

Wiegel V., Berg J. van den. (2009). Combining moral theory, modal logic and MAS to create well-behaving artificial agents. International Journal of Social Robotics, vol. 1, no 3, p. 233–242.

Wiener Y. (1988). Forms of value systems: A focus on organisational effectiveness and cultural change and maintenance. Academy of Management Review, vol. 13, no 4, p. 534-545.