Classification and Automatic Annotation Extension of Images Using a Bayesian Network. Classification et Extension Automatique D’Annotations D’Images en Utilisant un Réseau Bayésien

Classification and Automatic Annotation Extension of Images Using a Bayesian Network

Classification et Extension Automatique D’Annotations D’Images en Utilisant un Réseau Bayésien

Sabine Barrat Salvatore Tabbone 

LORIA-UMR7503, Université Nancy 2, BP 239, 54506 Vandœuvre-les-Nancy Cedex

15 December 2009
31 December 2009
| Citation



The rapid growth of Internet and multimedia information has shown a need in the development of multimedia information retrieval techniques,especially in image retrieval.We can distinguish two main trends.The first one,called “text-based image retrieval”,consists in applying text-retrieval techniques from fully annotated images.The textdescribes high-level concepts but this technique presents some drawbacks:it requires a tedious work of annotation. Moreover,annotations could be ambiguous because two users can use different keywords to describe a same image. Consequently some approaches have proposed to useWordnet in order to reduce these potential ambiguities.The second approach,called “content-based image retrieval”is a younger field.These methods rely on visual features (color, texture or shape) computed automatically,and retrieve images using a similarity measure.However,the obtained performances are not really acceptable,except in the case of well-focused corpus.In order to improve the recognition, a solution consists in combining visual and semantic information.In many vision problems,instead of having fully annotated training data,it is easier to obtain just a subset of data with annotations,because it is less restrictive for the user.This paper deals with modeling,classifying,and annotating weakly annotated images.More precisely,we propose a scheme for image classification optimization,using a joint visual-text clustering approach and automatically extending image annotations.The proposed approach is derived from the probabilistic graphical model theory and dedicated for both tasks of weakly-annotated image classification and annotation.We consider an image as weakly annotated if the number of keywords defined for it is less than the maximum defined in the ground truth.Thanks to their ability to manage missing values,a probabilistic graphical model has been proposed to represent weakly annotated images.We propose a probabilistic graphical model based on a Gaussian-Mixtures and Multinomial mixture.The visual features are estimated by the Gaussian mixtures and the keywords by a Multinomial distribution.Therefore,the proposed model does not require that all images be annotated:when an image is weakly annotated,the missing keywords are considered as missing values.Besides,our model can automatically extend existing annotations to weakly-annotated images,without user intervention.The uncertainty around the association between a set of keywords and an image is tackled by a joint probability distribution (defined from Gaussian-Mixtures and Multinomial mixture) over the dictionary of keywords and the visual features extracted from our collection of images.Moreover,in order to solve the dimensionality problem due to the large dimensions of visual features,we have adapted a variable selection method.Results of visual-textual classification,reported on a database of images collected from the Web,partially and manually annotated,show an improvement of about 32.3% in terms of recognition rate against only visual information classification.Besides the automatic annotation extension with our model for images with missing keywords outperforms the visual-textual classification of about 6.8%.Finally the proposed method is experimentally competitive with the state-of-art classifiers.


Nous proposons,dans cet article,d'améliorer la classification d'images,en utilisant une approche de classification visuo-textuelle (à base de caractéristiques visuelles et textuelles),et en étendant automatiquement les annotations existantes aux images non annotées. L'approche proposée est dérivée de la théorie des modèles graphiques probabilistes et dédiée aux deux tâches de classification et d'annotation d'images partiellement annotées. Nous considérons une image comme partiellement annotée si elle ne possède pas le nombre maximal de mots-clés disponibles par image dans la vérité-terrain. Grâce à leur capacité à fonctionner en présence de données manquantes,un modèle graphique probabiliste a été proposé pour représenter les images partiellement annotées. Ce modèle est basé sur un mélange de lois multinomiales et de mélanges de Gaussiennes. La distribution des caractéristiques visuelles est estimée par des mélanges de Gaussiennes et celle des mots-clés par une loi multinomiale. Par conséquent,le modèle proposé ne requiert pas que toutes les images soient annotées :lorsqu'une image est partiellement annotées, les mots-clés manquants sont considérés comme des valeurs manquantes. De plus,notre modèle peut automatiquement étendre des annotations existantes à des images partiellement annotées,sans l'intervention de l'utilisateur. L'incertitude autour de l'association entre un ensemble de mots-clés et une image est capturée par une distribution de probabilité jointe (définie par un mélange de lois multinomiales et de mélanges de Gaussiennes) sur le dictionnaire de mots-clés et les caractéristiques visuelles extraites de notre collection d'images. De plus,de façon à résoudre le problème de dimensionnalité dû à la grande dimension des caractéristiques visuelles,nous avons adapté une méthode de sélection de variables. Les résultats de la classification visuo-textuelle,obtenus sur une base d'images collectées sur Internet,partiellement et manuellement annotée,montrent une amélioration de 32.3 % en terme de taux de reconnaissance,par rapport à la classification basée sur l'information visuelle uniquement. Par ailleurs,l'extension automatique d'annotations,avec notre modèle,sur des images avec mots-clés manquants,améliore encore la classification visuo-textuelle de 6.8 %. Enfin,la méthode proposée s'est montrée compétitive avec des classificateurs de l'état de l'art. 


Probabilistic graphical models,Bayesian networks,variable selection,image classification,image annotation

Mots clés 

Modèles graphiques probabilistes,réseaux Bayésiens,sélection de variables,classification,annotation automatique

2.Représentation et Classification d’Images
3.Un Modèle de Mélange de Lois Multinomiales et de Densités à Mélange de Gaussiennes
4.Description des Images
5.Réduction de Dimensionnalité
6.Résultats Expérimentaux
7.Conclusion et Perspectives

[1] O. TERRADES, S. TABBONE, and E. VALVENY, «A review of shape descriptors for document analysis», International Conference on Document Analysis and Recognition, vol. 1, pp. 227-231, 2007.

[2] L. WENDLING, J. RENDEK, and P. MATSAKIS, «Selection of suitable set of decision rules using choquet integral», in SSPR/SPR, 2008, pp. 947-955. 

[3] V. GUNES, M. MÉNARD, P. LOONIS, and S. PETIT-RENAUD, «Combination, cooperation and selection of classifiers:A state of the art », IJPRAI, vol. 17, no. 8, pp. 1303-1324, 2003. 

[4] O. R. TERRADES, E. VALVENY, and S. TABBONE, «Optimal classifier fusion in a non-bayesian probabilistic framework», Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 31, no. 9, pp. 1630-1644, 2009. [Online]. Available:

[5] A. W. M. SMEULDERS, M. WORRING, S. SANTINI, A. GUPTA, and R. JAIN, «Content-based image retrieval at the end of the early years», IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 12, pp. 1349-1380, December 2000. [Online]. Available: 

[6] K. BARNARD, P. DUYGULU, D. FORSYTH, N. DE FREITAS, D. M. BLEI, and M. I. JORDAN, «Matching words and pictures», Journal of Machine Learning Research, vol. 3, no. 6, pp. 1107-1135, 2003.

[7] W. I. GROSKY and R. ZHAO, «Negotiating the semantic gap: From feature maps to semantic landscapes», in SOFSEM ‘01, 2001, pp. 33-52. 

[8] A. BENITEZ and C. SHIH-FU, «Perceptual knowledge construction from annotated image collections», in ICME ‘02, vol. 1, pp. 189-192, 2002. 

[9] Y. GAO, J. FAN, X. XUE, and R. JAIN, «Automatic image annotation by incorporating feature hierarchy and boosting to scale up svm classifiers», in ACM MULTIMEDIA ‘06, 2006, pp. 901-910.

[10] C. YANG, M. DONG, and J. HUA, «Region-based image annotation using asymmetrical support vector machine-based multipleinstance learning», in CVPR ‘06, 2006, pp. 2057-2063. 

[11] D. M. BLEI and M. I. JORDAN, «Modeling annotated data», in SIGIR ‘03, 2003, pp. 127-134. 

[12] S. FENG, R. MANMATHA, and V. LAVRENKO, «Multiple bernoulli relevance models for image and video annotation», in CVPR’04, vol. 2, pp. 1002-1009, 2004. 

[13] C. WANG, F. JING, L. ZHANG, and H.-J. ZHANG, «Image annotation refinement using random walk with restarts,? in ACM MULTIMEDIA’06, 2006, pp. 647-650. 

[14] X. RUI, M. LI, Z. LI, W.-Y. MA, and N. YU, «Bipartite graph reinforcement model for web image annotation», in MULTIMEDIA’07, 2007, pp. 585-594. 

[15] R. ZHANG, Z. M. ZHANG, M. LI, W.-Y. MA, and H.-J. ZHANG, «A probabilistic semantic model for image annotation and multimodal image retrieval», in ICCV’05, vol. 1, 2005, pp. 846-851. 

[16] R. JIN, J. Y. CHAI, and L. SI, «Effective automatic image annotation via a coherent language model and active learning,» in MULTIMEDIA’04, 2004, pp. 892-899. 

[17] M. L. KHERFI, D. BRAHMI, and D. ZIOU, «Combining visual features with semantics for a more effective image retrieval», in ICPR’04, vol. 2, 2004, pp. 961-964. 

[18] C. BURGES, «A tutorial on support vector machines for pattern recognition»,Data Mining and Knowledge Discovery,vol. 2,no.2, pp. 121167, 1998. 

[19] L. BREIMAN, «Random forests», in Machine Learning, 2001, pp. 5-32. 

[20] A. P. DEMPSTER, N. M. LAIRD, and D. B. RUBIN, «Maximum likelihood from incomplete data via the em algorithm»,Journal of the Royal Statistical Society. Series B (Methodological), vol. 39, no. 1, pp. 1-38, 1977. 

[21] M. I. JORDAN, Ed., Learning in graphical models. Cambridge, MA, USA: MIT Press, 1999. 

[22] J. H. KIM and J. PEARL, «A computational model for combined causal and diagnostic reasoning in inference systems», in IJCAI-83, 1983, pp. 190-193.

[23] C. ROBERT, A decision-Theoretic Motivation. Springer-Verlag, 1997. 

[24] S. TABBONE and L. WENDLING, «Technical symbols recognition using the two-dimensional radon transform», in ICPR’02, vol. 2, aug2002, pp. 200-203. 

[25] M. J. SWAIN and D. H. BALLARD, «Color indexing», Int. J. Comput. Vision, vol. 7, no. 1, pp. 11-32, 1991. 

[26] T. DENŒUX and M.-H. MASSON, «Dimensionality reduction and visualization of interval and fuzzy data: a survey», in Proceedings of the 56th session of the International Statistical Institute (ISI `S07), Lisboa, Portugal,August 2007. 

[27] M. PICCARDI, H. GUNES, and A. F. OTOOM, «Maximum-likelihood dimensionality reduction in gaussian mixture models with an application to object classification», in ICPR’08, 2008, pp. 1-4. 

[28] R. O. DUDA, P. E. HART, and D. G. STORK, Pattern Classification, Second Edition. Wiley-Interscience, 2001. 

[29] P. A. DEVIJVER and J. KITTLER, Pattern recognition: A statistical approach. Prentice Hall, 1982. 

[30] P. PUDIL, J. NOVOVI˘COV´ A, and J. KITTLER, «Floating search methods in feature selection», Pattern Recognition Letters, vol. 15, no. 11, pp. 1119-1125, 1994. 

[31] R. TIBSHIRANI, «Regression shrinkage and selection via the lasso», Journal of the Royal Statistical Society. Series B (Methodological), vol. 58, no. 1, pp. 267?288, 1996. 

[32] B. EFRON, T. HASTIE, I. JOHNSTONE, and R. TIBSHIRANI, «Least angle regression», Annals of Statitics, vol. 32, pp. 407-499, 2004. 

[33] C.-C. CHANG and C.-J. LIN, LIBSVM: a library for support vector machines, 2001. [Online]. Available: cjlin/libsvm [34] J. KELLER, M. GRAY, and J. GIVENS, «A fuzzy k-nearest neighbor algorithm», IEEE Transactions on Systems, Man, and Cybernetics, vol. 15, no. 4, pp. 580-585, 1985.