New Paper

Journal Articles

Affective Image Content Analysis: Two Decades Review and New Perspectives

Sicheng Zhao, Xingxu Yao, Jufeng Yang, Guoli Jia, Guiguang Ding, Tat-Seng Chua, Bjorn W. Schuller, Kurt Keutzer IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021[accepted].

PDF

中文翻译

BibTeX

Abstract

@article{zhao2021affective,
  author={Zhao, Sicheng and Yao, Xingxu and Yang, Jufeng and Jia, Guoli and Ding, Guiguang and Chua, Tat-Seng and Schuller, Bjoern W and Keutzer, Kurt},
  title={Affective Image Content Analysis: Two Decades Review and New Perspectives},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2021},
  publisher={IEEE}
}

    Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges -- the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.

Multimodal Event-Aware Network for Sentiment Analysis in Tourism

Lijuan Wang, Wenya Guo, Xingxu Yao, Yuxiang Zhang, Jufeng Yang
IEEE MultiMedia( (MM), 2021 [accepted].

PDF

BibTeX

Abstract

@article{wang2021multimodal,
  author={Wang, Lijuan and Guo, Wenya and Yao, Xingxu and Zhang, Yuxiang and Yang, Jufeng},
  title={Multimodal Event-aware Network for Sentiment Analysis in Tourism},
  year={2021},
  journal={IEEE MultiMedia},
  publisher={IEEE}
}

    Considering the application of a sentiment analysis in decision-making and personalized advertising, we adopt it in tourism. Specifically, we perform a sentiment analysis on the posted Weibos about the passengers' experience in civil aviation travel. Different travel events could influence passengers' sentiment, e.g., flight delay may cause negative sentiment. Inspired by this observation, we propose a novel multimodal event-aware network to analyze sentiment from Weibos that contain multiple modalities, i.e., text and images. We first extract features from each modality and, then, model the cross-modal associations to obtain more discriminative representations, based on which we simultaneously perceive the event and sentiment in a multitask framework. Extensive experiments demonstrate that the proposed method outperforms the existing state-of-the-art approaches.

LD-MAN: Layout-Driven Multimodal Attention Network for Online News Sentiment Recognition

Wenya Guo, Ying Zhang, Xiangrui Cai, Lei Meng, Jufeng Yang, Xiaojie Yuan IEEE Transactions on Multimedia (TMM), 2021[accepted].

PDF

BibTeX

Abstract

@article{guo2020ld,
  author={Guo, Wenya and Zhang, Ying and Cai, Xiangrui and Meng, Lei and Yang, Jufeng and Yuan, Xiaojie},
  title={LD-MAN: Layout-driven multimodal attention network for online news sentiment recognition},
  journal={IEEE Transactions on Multimedia},
  year={2021},
  volume={23},
  pages={1785--1798},
  publisher={IEEE}
}

    The prevailing use of both images and text to express opinions on the web leads to the need for multimodal sentiment recognition. Some commonly used social media data containing short text and few images, such as tweets and product reviews, have been well studied. However, it is still challenging to predict the readers' sentiment after reading online news articles, since news articles often have more complicated structures, e.g., longer text and more images. To address this problem, we propose a layout-driven multimodal attention network (LD-MAN) to recognize news sentiment in an end-to-end manner. Rather than modeling text and images individually, LD-MAN uses the layout of online news to align images with the corresponding text. Specifically, it exploits a set of distance-based coefficients to model the image locations and measure the contextual relationship between images and text. LD-MAN then learns the affective representations of the articles from the aligned text and images using a multimodal attention mechanism. Considering the lack of relevant datasets in this field, we collect two multimodal online news datasets, containing a total of 14,566 articles with 56,260 images and 251,202 words. Experimental results demonstrate that the proposed method performs favorably compared with state-of-the-art approaches. We will release all the codes, models and datasets to the community.

Emotion Recognition From Multiple Modalities: Fundamentals and methodologies

Sicheng Zhao, Guoli Jia, Jufeng Yang, Guiguang Ding, Kurt Keutzer
IEEE Signal Processing Magazine (SPM), 2021[accepted].

PDF

BibTeX

Abstract

@article{zhao2021emotion,
  author={Zhao, Sicheng and Jia, Guoli and Yang, Jufeng and Ding, Guiguang and Keutzer, Kurt},
  title={Emotion Recognition From Multiple Modalities: Fundamentals and methodologies},
  journal={IEEE Signal Processing Magazine},
  year={2021},
  volume={38},
  pages={59--73},
  publisher={IEEE}
}

    Humans are emotional creatures. Multiple modalities are often involved when we express emotions, whether we do so explicitly (such as through facial expression and speech) or implicitly (e.g., via text or images). Enabling machines to have emotional intelligence, i.e., recognizing, interpreting, processing, and simulating emotions, is becoming increasingly important. In this tutorial, we discuss several key aspects of multimodal emotion recognition (MER).

APSE: Attention-aware polarity-sensitive embedding for emotion-based image retrieval

Xingxu Yao, Sicheng Zhao, Yu-Kun Lai, Dongyu She, Jie Liang, Jufeng Yang
IEEE Transcation on Multimedia (TMM), 2020[accepted].

PDF

BibTeX

Project

Abstract

@ARTICLE{9281019,
  author={X. {Yao} and S. {Zhao} and Y. -K. {Lai} and D. {She} and J. {Liang} and J. {Yang}},
  journal={IEEE Transactions on Multimedia},
  title={APSE: Attention-aware Polarity-Sensitive Embedding for Emotion-based Image Retrieval},
  year={2020},
  publisher={IEEE}
}

    With the popularity of social media, an increasing number of people are accustomed to expressing their feelings and emotions online using images and videos. An emotion-based image retrieval (EBIR) system is useful for obtaining visual contents with desired emotions from a massive repository. Existing EBIR methods mainly focus on modeling the global characteristics of visual content without considering the crucial role of informative regions of interest in conveying emotions. Further, they ignore the hierarchical relationships between coarse polarities and fine categories of emotions. In this paper, we design an attention-aware polarity-sensitive embedding (APSE) network to address these issues. First, we develop a hierarchical attention mechanism to automatically discover and model the informative regions of interest. Specifically, both polarity-and emotion-specific attended representations are aggregated for discriminative feature embedding. Second, we propose a generated emotion-pair (GEP) loss to simultaneously consider the inter-and intra-polarity relationships of the emotion labels. Moreover, we adaptively generate negative examples of different hard levels in the feature space guided by the attention module to further improve the performance of feature embedding. Extensive experiments on four popular benchmark datasets demonstrate that the proposed APSE method outperforms the state-of-the-art EBIR approaches by a large margin.

Adaptive Deep Metric Learning for Affective Image Retrieval and Classification

Xingxu Yao, Dongyu She, Haiwei Zhang, Jufeng Yang, Ming-Ming Cheng and Liang Wang
IEEE Transcation on Multimedia (TMM), 2020[accepted].

PDF

BibTeX

Abstract

@article{2020Adaptive,
  title={Adaptive Deep Metric Learning for Affective Image Retrieval and Classification},
  author={ Yao, Xingxu and She, Dongyu and Zhang, Haiwei and Yang, Jufeng and Wang, Liang },
  journal={IEEE Transactions on Multimedia},
  year={2020},
  volume={PP},
  number={99},
  pages={1-1},
  publisher={IEEE}
}

    An image is worth a thousand words. Many researchers have conducted extensive studies to understand visual emotions since an increasing number of users express emotions via images and videos online. However, most existing methods based on convolutional neural networks aim to retrieve and classify affective images in a discrete label space while ignoring both the hierarchical and complex nature of emotions. On the one hand, different from concrete and isolated object concepts (e.g., cat and dog), a hierarchical relationship exists among emotions. On the other hand, most widely used deep methods depend on the representation from fully connected layers, which lacks the essential texture information for recognizing emotions. In this work, we address the above problems via adaptive deep metric learning. Specifically, we design an adaptive sentiment similarity loss, which is able to embed affective images considering the emotion polarity and adaptively adjust the margin between different image pairs. To effectively distinguish affective images, we further propose the sentiment vector that captures the texture information extracted from multiple convolutional layers. Finally, we develop a unified multi-task deep framework to simultaneously optimize both retrieval and classification goals. Extensive and thorough evaluations on four benchmark datasets demonstrate that the proposed framework performs favorably against the state-of-the-art methods.
Index Terms—Visual sentiment analysis, affective image retrieval, deep metric learning, convolutional neural network.

WSCNet: Weakly Supervised Coupled Networks for Visual Sentiment Classification and Detection

Dongyu She, Jufeng Yang, Ming-Ming Cheng, Yu-Kun Lai, Paul L. Rosin and Liang Wang
IEEE Transcation on Multimedia (TMM), 2019[accepted].

PDF

BibTeX

Project

Abstract

@article{She_2019_TMM,
  author = {She, Dongyu and Yang, Jufeng and Cheng, Ming-Ming and Lai, Yu-Kun and Rosin, Paul L. and Wang, Liang},
  title = {WSCNet: Weakly Supervised Coupled Networks for Visual Sentiment Classification and Detection},
  journal={IEEE Transactions on Multimedia},
  year = {2019}
  volume={22},
  number={5},
  pages={1358--1371},
  publisher={IEEE}
}

    Automatic assessment of sentiment from visual content has gained considerable attention with the increasing tendency of expressing opinions online. In this paper, we solve the problem of visual sentiment analysis, which is challenging due to the high-level abstraction in the recognition process. Existing methods based on convolutional neural networks learn sentiment representations from the holistic image, despite the fact that different image regions can have different influence on the evoked sentiment. In this paper, we introduce a weakly supervised coupled convolutional network (WSCNet). Our method is dedicated to automatically selecting relevant soft proposals from weak annotations (\eg global image labels), thereby significantly reducing the annotation burden, and encompasses the following contributions. First, WSCNet detects a sentiment-specific soft map by training a fully convolutional network with the cross spatial pooling strategy in the detection branch. Second, both the holistic and localized information are utilized by coupling the sentiment map with deep features for robust representation in the classification branch. We integrate the sentiment detection and classification branches into a unified deep framework, and optimize the network in an end-to-end way. Through this joint learning strategy, weakly supervised sentiment classification and detection benefit each other. Extensive experiments demonstrate that the proposed WSCNet outperforms the state-of-the-art results on seven benchmark datasets.

Learning Discriminative Sentiment Representation from Strongly- and Weakly-Supervised CNNs

Dongyu She, Ming Sun, Jufeng Yang
ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) (TOMM), 2019[accepted].

PDF

BibTeX

Dataset

Abstract

@article{she2019learning,
  author={She, Dongyu and Sun, Ming and Yang, Jufeng},
  title={Learning discriminative sentiment representation from strongly-and weakly supervised cnns},
  journal={ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)},
  year = {2019}
  volume={15},
  number={3s},
  pages={1--19},
  publisher={ACM New York, NY, USA}
}

    Visual sentiment analysis is attracting increasing attention with the rapidly growing amount of images uploaded to social network. Learning rich visual representations often requires training deep convolutional neural networks on massive manually labeled data, which is expensive or scarce especially for a subjective task like visual sentiment analysis. Meanwhile, a large quantity of social images is quite available yet noisy by querying social network using the sentiment categories as keywords, where a various type of images related to the specific sentiment can be easily collected. In this paper, we propose a multiple kernel network (MKN) for visual sentiment recognition, which learns representation from strongly- and weakly- supervised CNNs. Specifically, the weakly-supervised deep model is trained using the large-scale data from social images, while the stronglysupervised deep model is fine-tuned on the affecitve datasets with manual annotation. We employ the multiple kernel scheme on the multiple layers of CNNs, which can automatically select the discriminative representation by learning a linear combination from a set of pre-defined kernels. In addition, we introduce a large-scale dataset collected from popular comics of various countries, e.g., America, Japan, China and France, which consists of 11,821 images with various artistic styles. Experimental results show that MKN achieves consistent improvements over the state-of-the-art methods on the public affective datasets as well as the newly established Comics dataset. The Comics dataset can be found on http://cv.nankai.edu.cn/projects/Comic.

Visual Sentiment Prediction Based on Automatic Discovery of Affective Regions

Jufeng Yang, Dongyu She, Ming Sun, Ming-Ming Cheng, Paul L. Rosin, and Liang Wang
IEEE Transcation on Multimedia (TMM), 2018[accepted].

PDF

BibTeX

Project

Abstract

@ARTICLE{8283764,
  author={J. {Yang} and D. {She} and M. {Sun} and M. {Cheng} and P. L. {Rosin} and L. {Wang}},
  journal={IEEE Transactions on Multimedia},
  title={Visual Sentiment Prediction Based on Automatic Discovery of Affective Regions},
  year={2018},
  volume={20},
  number={9},
  pages={2513-2525},
  doi={10.1109/TMM.2018.2803520}
}

    Automatic assessment of sentiment from visual content has gained considerable attention with the increasing tendency of expressing opinions via images and videos online. This paper investigates the problem of visual sentiment analysis, which involves a high-level abstraction in the recognition process. While most of the current methods focus on improving holistic representations, we aim to utilize the local information, which is inspired by the observation that both the whole image and local regions convey significant sentiment information. We propose a framework to leverage affective regions, where we first use an off-the-shelf objectness tool to generate the candidates, and employ a candidate selection method to remove redundant and noisy proposals. Then, a convolutional neural network (CNN) is connected with each candidate to compute the sentiment scores, and the affective regions are automatically discovered, taking the objectness score as well as the sentiment score into consideration. Finally, the CNN outputs from local regions are aggregated with the whole images to produce the final predictions. Our framework only requires image-level labels, thereby significantly reducing the annotation burden otherwise required for training. This is especially important for sentiment analysis since sentiment can be abstract, and labeling affective regions is too subjective and labor-consuming. Extensive experiments show that the proposed algorithm outperforms the state-of-the-art approaches on eight popular benchmark datasets.

Conference Proceedings

Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Space

Sicheng Zhao, Yaxian Li, Xingxu Yao, Weizhi Nie, Pengfei Xu, Jufeng Yang, Kurt Keutzer
ACM International Conference on Multimedia( (ACM MM), 2020[accepted].

PDF

BibTeX

Abstract

@inproceedings{zhao2020emotion,
  author={Zhao, Sicheng and Li, Yaxian and Yao, Xingxu and Nie, Weizhi and Xu, Pengfei and Yang, Jufeng and Keutzer, Kurt},
  title={Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Space},
  booktitle={Proceedings of the 28th ACM International Conference on Multimedia},
  year={2020}
}

    Both images and music can convey rich semantics and are widely used to induce specific emotions. Matching images and music with similar emotions might help to make emotion perceptions more vivid and stronger. Existing emotion-based image and music matching methods either employ limited categorical emotion states which cannot well reflect the complexity and subtlety of emotions, or train the matching model using an impractical multi-stage pipeline. In this paper, we study end-to-end matching between image and music based on emotions in the continuous valence-arousal (VA) space. First, we construct a large-scale dataset, termed Image-Music-Emotion-Matching-Net (IMEMNet), with over 140K image-music pairs. Second, we propose cross-modal deep continuous metric learning (CDCML) to learn a shared latent embedding space which preserves the cross-modal similarity relationship in the continuous matching space. Finally, we refine the embedding space by further preserving the single-modal emotion relationship in the VA spaces of both images and music. The metric learning in the embedding space and task regression in the label space are jointly optimized for both cross-modal matching and single-modal VA prediction. The extensive experiments conducted on IMEMNet demonstrate the superiority of CDCML for emotion-based image and music matching as compared to the state-of-the-art approaches.

An End-to-End Visual-Audio Attention Networt=k for Emotion Recognition in User-Generated Videos

Sicheng Zhao, Yunsheng Ma, Yang Gu, Jufeng Yang, Tengfei Xing, Pengfei Xu, Runbo Hu, Hua Chai, Kurt Keutzer
AAAI Conference on Artificial Intelligence( (AAAI), 2020[accepted].

PDF

BibTeX

Project

Abstract

@inproceedings{zhao2020end,
  author={Zhao, Sicheng and Ma, Yunsheng and Gu, Yang and Yang, Jufeng and Xing, Tengfei and Xu, Pengfei and Hu, Runbo and Chai, Hua and Keutzer, Kurt},
  title={An End-to-End visual-audio attention network for emotion recognition in user-generated videos},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2020}
}

    Emotion recognition in user-generated videos plays an important role in human-centered computing. Existing methods mainly employ traditional two-stage shallow pipeline, i.e. extracting visual and/or audio features and training classifiers. In this paper, we propose to recognize video emotions in an end-to-end manner based on convolutional neural networks (CNNs). Specifically, we develop a deep Visual-Audio Attention Network (VAANet), a novel architecture hat integrates spatial, channel-wise, and temporal attentions into a visual 3D CNN and temporal attentions into an audio 2D CNN. Further, we design a special classification loss, i.e. polarity-consistent cross-entropy loss, based on the polarity-emotion hierarchy constraint to guide the attention generation. Extensive experiments conducted on the challenging VideoEmotion-8 and Ekman-6 datasets demonstrate that the proposed VAANet outperforms the state-of-the-art approaches for video emotion recognition. Our source code is released at: https://github.com/maysonma/VAANet.

Attention-Aware Polarity Sensitive Embedding for Affective Image Retrieval

Xingxu Yao, Dongyu She, Sicheng Zhao, Jie Liang, Yu-Kun Lai, Jufeng Yang
IEEE International Conference on Computer Vision( (ICCV), 2019[accepted].

PDF

BibTeX

Project

Abstract

@InProceedings{Yao_2019_ICCV,
  author = {Yao, Xingxu and She, Dongyu and Zhao, Sicheng and Liang, Jie and Lai, Yu-Kun and Yang, Jufeng},
  title = {Attention-Aware Polarity Sensitive Embedding for Affective Image Retrieval},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year = {2019}
}

    Images play a crucial role for people to express their opinions online due to the increasing popularity of social networks. While an affective image retrieval system is useful for obtaining visual contents with desired emotions from a massive repository, the abstract and subjective characteristics make the task challenging. To address the problem, this paper introduces an Attention-aware Polarity Sensitive Embedding (APSE) network to learn affective representations in an end-to-end manner. First, to automatically discover and model the informative regions of interest, we develop a hierarchical attention mechanism, in which both polarity- and emotion-specific attended representations are aggregated for discriminative feature embedding. Second, we present a weighted emotion-pair loss to take the interand intra-polarity relationships of the emotional labels into consideration. Guided by attention module, we weight the sample pairs adaptively which further improves the performance of feature embedding. Extensive experiments on four popular benchmark datasets show that the proposed method performs favorably against the state-of-the-art approaches.

Zero-Shot Emotion Recognition via Affective Structural Embedding

Chi Zhan, Dongyu She, Sicheng Zhao, Ming-Ming Cheng, Jufeng Yang
IEEE International Conference on Computer Vision( (ICCV), 2019[accepted].

PDF

BibTeX

Abstract

@inproceedings{2020Zero,
  title={Zero-Shot Emotion Recognition via Affective Structural Embedding},
  author={Zhan, Chi and She, Dongyu and Zhao, Sicheng and Cheng, Ming Ming and Yang, Jufeng },
  booktitle={2019 IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2020},
}

    Image emotion recognition attracts much attention in recent years due to its wide applications. It aims to understand the emotional response of humans, where candidate emotion categories are generally defined by specific psychological theories. However, with the development of psychological theories, emotion categories become increasingly diverse, fine-grained, and difficult to collect samples. In this paper, we investigate zero-shot learning (ZSL) problem in the emotion recognition task, which aims to recognize the new unseen emotions. Specifically, we propose an affective structural embedding framework, utilizing mid-level semantic representation, i.e., adjective-noun pairs (ANP) features, to construct an intermediate embedding space. By doing this, the learned intermediate space can bridge the affective gap between low-level visual features and highlevel semantics. In addition, we introduce an adversarial constraint to combine the visual and affective embeddings so as to retain the discriminative capacity of visual features and the affective structural information of semantic features during training process. Our method is evaluated on five widely-used affective datasets and the experimental results show the proposed algorithm outperforms the state-of-theart approaches. Our source code and trained models will be released.

Weakly Supervised Coupled Networks for Visual Sentiment Analysis

Jufeng Yang, Dongyu She, Yu-Kun Lai, Paul L. Rosin, Ming-Hsuan Yang
Internaltional Conference on Computer Vision and Pattern Recogintion (CVPR), Salt Lake City, Utah, USA, June. 2018.

PDF

BibTeX

Abstract

@InProceedings{Yang_2018_CVPR,
  author = {Yang, Jufeng and She, Dongyu and Lai, Yu-Kun and Rosin, Paul L. and Yang, Ming-Hsuan},
  title = {Weakly Supervised Coupled Networks for Visual Sentiment Analysis},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2018}
}

    Automatic assessment of sentiment from visual content has gained considerable attention with the increasing tendency of expressing opinions on-line. In this paper, we solve the problem of visual sentiment analysis using the high-level abstraction in the recognition process. Existing methods based on convolutional neural networks learn sentiment representations from the holistic image appearance. However, different image regions can have a different influence on the intended expression. This paper presents a weakly supervised coupled convolutional network with two branches to leverage the localized information. The first branch detects a sentiment specific soft map by training a fully convolutional network with the cross spatial pooling strategy, which only requires image-level labels, thereby significantly reducing the annotation burden. The second branch utilizes both the holistic and localized information by coupling the sentiment map with deep features for robust classification. We integrate the sentiment detection and classification branches into a unified deep framework and optimize the network in an end-to-end manner. Extensive experiments on six benchmark datasets demonstrate that the proposed method performs favorably against the state-ofthe-art methods for visual sentiment analysis.

Retrieving and Classifying Affective Images via Deep Metric Learning

Jufeng Yang, Dongyu She, Yu-Kun Lai, Ming-Hsuan Yang
AAAI Conference on Artificial Intelligence (AAAI), New Orleans, Louisiana, USA, Feb. 2018.

PDF

BibTeX

Abstract

Slides.

@InProceedings{Yang_2018_AAAI,   author = {Yang, Jufeng and She, Dongyu and Lai, Yu-Kun and Yang, Ming-Hsuan},
  title = {Retrieving and Classifying Affective Images via Deep Metric Learning},
  booktitle = {AAAI Conference on Artificial Intelligence (AAAI)},
  year = {2018}
}

    Affective image understanding has been extensively studied in the last decade since more and more users express emotion via visual contents. While current algorithms based on convolutional neural networks aim to distinguish emotional categories in a discrete label space, the task is inherently ambiguous. This is mainly because emotional labels with the same polarity (i.e., positive or negative) are highly related, which is different from concrete object concepts such as cat, dog and bird. To the best of our knowledge, few methods focus on leveraging such characteristic of emotions for affective image understanding. In this work, we address the problem of understanding affective images via deep metric learning and propose a multi-task deep framework to optimize both retrieval and classification goals. We propose the sentiment constraints adapted from the triplet constraints, which are able to explore the hierarchical relation of emotion labels. We further exploit the sentiment vector as an effective representation to distinguish affective images utilizing the texture representation derived from convolutional layers. Extensive evaluations on four widely-used affective datasets, i.e., Flickr and Instagram, IAPSa, Art Photo, and Abstract Paintings, demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods on both affective image retrieval and classification tasks.

Understanding Image Impressiveness Inspired by Instantaneous Human Perceptual Cues

Jufeng Yang, Yan Sun, Jie Liang, Yong-Liang Yang, Ming-Ming Cheng
AAAI Conference on Artificial Intelligence (AAAI), New Orleans, Louisiana, USA, Feb. 2018.

PDF

BibTeX

Abstract

@inproceedings{yang2018understanding,
  author={Yang, Jufeng and Sun, Yan and Liang, Jie and Yang, Yong-Liang and Cheng, Ming-Ming},
  title={Understanding image impressiveness inspired by instantaneous human perceptual cues},
  booktitle = {AAAI Conference on Artificial Intelligence (AAAI)},
  year = {2018}
}

    With the explosion of visual information nowadays, millions of digital images are available to the users. How to efficiently explore a large set of images and retrieve useful information thus becomes extremely important. Unfortunately only some of the images can impress the user at first glance. Others that make little sense in human perception are often discarded, while still costing valuable time and space. Therefore, it is significant to identify these two kinds of images for relieving the load of online repositories and accelerating information retrieval process. However, most of the existing image properties, e.g., memorability and popularity, are based on repeated human interactions, which limit the research and application of evaluating image quality in terms of instantaneous impression. In this paper, we propose a novel image property, called impressiveness, that measures how images impress people with a short-term contact. This is based on an impression-driven model inspired by a number of important human perceptual cues. To achieve this, we first collect three datasets in various domains, which are labeled according to the instantaneous sensation of the annotators. Then we investigate the impressiveness property via six established human perceptual cues as well as the corresponding features from pixel to semantic levels. Sequentially, we verify the consistency of the impressiveness which can be quantitatively measured by multiple visual representations, and evaluate their latent relationships. Finally, we apply the proposed impressiveness property to rank the images for an efficient image recommendation system.

Text Emotion Distribution Learning via Multi-Task Convolutional Neural Network

Yuxiang Zhang, Jiamei Fu, Dongyu She, Ying Zhang, Senzhang Wang, and Jufeng Yang
International Joint Conference on Artificial Intelligence (IJCAI), 2018[accepted].

PDF

BibTeX

Abstract

@inproceedings{Zhang2018Text,
  author={Zhang, Yuxiang and Fu, Jiamei and She, Dongyu and Ying, Zhang and Yang, Jufeng},
  title={Text Emotion Distribution Learning via Multi-Task Convolutional Neural Network},
  booktitle={Twenty-Seventh International Joint Conference on Artificial Intelligence IJCAI-18},
  year={2018},
}

    Emotion analysis of on-line user generated textual content is important for natural language processing and social media analytics tasks. Most of previous emotion analysis approaches focus on identifying users’ emotional states from text by classifying emotions into one of the finite categories, e.g., joy, surprise, anger and fear. However, there exists ambiguity characteristic for the emotion analysis, since a single sentence can evoke multiple emotions with different intensities. To address this problem, we introduce emotion distribution learning and propose a multi-task convolutional neural network for text emotion analysis. The end-to-end framework optimizes the distribution prediction and classification tasks simultaneously, which is able to learn robust representations for the distribution dataset with annotations of different voters. While most work adopt the majority voting scheme for the ground truth labeling, we also propose a lexiconbased strategy to generate distributions from a single label, which provides prior information for the emotion classification. Experiments conducted on five public text datasets (i.e., SemEval, Fairy Tales, ISEAR, TEC, CBET) demonstrate that our proposed method performs favorably against the stateof-the-art approaches.

Deep Coordinated Textual and Visual Network for Sentiment-Oriented Cross-Modal Retrieval

Jiamei Fu, Dongyu She, Xingxu Yao, Yuxiang Zhang, and Jufeng Yang
Pacific Rim International Conferences on Artificial Intelligence (PRICAI), 2018[accepted].

PDF

BibTeX

Abstract

@inproceedings{zhang2018text,
  author={Fu, Jiamei, She, Dongyu, Yao, Xingxu, Zhang, Yuxiang, Yang, Jufeng}
  title="Deep Coordinated Textual and Visual Network for Sentiment-Oriented Cross-Modal Retrieval",
  booktitle="PRICAI 2018: Trends in Artificial Intelligence",
  year="2018",
}

    Cross-modal retrieval has attracted more and more attention recently, which enables people to retrieve desired information efficiently from a large amount of multimedia data. Most methods on cross-modal retrieval only focus on aligning the objects in image and text, while sentiment alignment is also essential for facilitating various applications, e.g., entertainment, advertisement, etc. This paper studies the problem of retrieving visual sentiment concepts with a goal to extract sentiment-oriented information from social multimedia content, i.e., sentiment oriented cross-media retrieval. Such problem is inherently challenging due to the subjective and ambiguity characteristics of the adjectives like “sad” and “awesome”. Thus, we focus on modeling visual sentiment concepts with adjective-noun pairs, e.g., “sad dog” and “awesome flower”, where associating adjectives with concrete objects makes the concepts more tractable. This paper proposes a deep coordinated textural and visual network with two branches to learn a joint semantic embedding space for both images and texts. The visual branch is based on a convolutional neural network (CNN) pre-trained on a large dataset, which is optimized with the classification loss. The textual branch is added on the fully-connected layer providing supervision of the textual semantic space. In order to learn the coordinated representation for different modalities, the multi-task loss function is optimized during the end-to-end training process. We have conducted extensive experiments on a subset of the large-scale VSO dataset. The results show that the proposed model is able to retrieval sentiment-oriented data, which performs favorably against the state-of-the-art methods.

Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network

Jufeng Yang, Dongyu She, Ming Sun
International Joint Conference on Artificial Intelligence (IJCAI), 2017[accepted].

PDF

BibTeX

Project

Abstract

@inproceedings{yang2017joint,
  author={Yang, Jufeng and She, Dongyu and Sun, Ming},
  title={Joint Image Emotion Classification and Distribution Learning via Deep Convolutional Neural Network},
  booktitle={IJCAI},
  year={2017}
}

    Visual sentiment analysis is attracting more and more attention with the increasing tendency to express emotions through visual contents. Recent algorithms in Convolutional Neural Networks (CNNs) considerably advance the emotion classification, which aims to distinguish differences among emotional categories and assigns a single dominant label to each image. However, the task is inherently ambiguous since an image usually evokes multiple emotions and its annotation varies from person to person. In this work, we address the problem via label distribution learning and develop a multi-task deep framework by jointly optimizing classification and distribution prediction. While the proposed method prefers to the distribution datasets with annotations of different voters, the majority voting scheme is widely adopted as the ground truth in this area, and few dataset has provided multiple affective labels. Hence, we further exploit two weak forms of prior knowledge, which are expressed as similarity information between labels, to generate emotional distribution for each category. The experiments conducted on both distribution datasets, i.e . Emotion6, Flickr LDL, Twitter LDL, and the largest single label dataset, i.e . Flickr and Instagram, demonstrate the proposed method outperforms the state-of-the-art approaches.

Learning Visual Sentiment Distributions via Augmented Conditional Probability Neural Network

Jufeng Yang, Ming Sun, Xiaoxiao Sun
Internaltional Conference on Computer Vision and Pattern Recogintion (AAAI), San Francisco, California, USA, Feb. 2017.

PDF

BibTeX

Abstract

@article{Yang_Sun_Sun_2017,
  author={Yang, Jufeng and Sun, Ming and Sun, Xiaoxiao},
  title={Learning Visual Sentiment Distributions via Augmented Conditional Probability Neural Network},
  booktitle={Thirty-first AAAI conference on artificial intelligence},
  year={2017}
}

    Visual sentiment analysis is raising more and more attention with the increasing tendency to express emotions through images. While most existing works assign a single dominant emotion to each image, we address the sentiment ambiguity by label distribution learning (LDL), which is motivated by the fact that image usually evokes multiple emotions. Two new algorithms are developed based on conditional probability neural network (CPNN). First, we proposed BCPNN which encodes image label into a binary representation to replace the signless integers used in CPNN, and employ it as a part of input for the neural network. Then, we train our ACPNN model by adding noises to ground truth label and augmenting affective distributions. Since current datasets are mostly annotated for single-label learning, we build two new datasets, one of which is relabeled on the popular Flickr dataset and the other is collected from Twitter. These datasets contain 20,745 images with multiple affective labels, which are over ten times larger than the existing ones. Experimental results show that the proposed methods outperform the state-of-the-art works on our large-scale datasets and other publicly available benchmarks.

Discovering Affctive Regions In Deep Convolutional Neural Networks For Visual Sentiment Prediction

Ming Sun, Jufeng Yang, Kai Wang, Hui Shen
International Conference on Multimedia and Expositions (ICME), 2016[accepted].

PDF

BibTeX

Abstract

@INPROCEEDINGS{7552961,
  author={M. {Sun} and J. {Yang} and K. {Wang} and H. {Shen}},
  title={Discovering affective regions in deep convolutional neural networks for visual sentiment prediction},
  booktitle={2016 IEEE International Conference on Multimedia and Expo (ICME)},
  year={2016}
}

    In this paper, we address the problem of automatically recognizing emotions in still images. While most of current work focus on improving whole-image representations using CNNs, we argue that discovering affective regions and supplementing local features will boost the performance, which is inspired by the observation that both global distributions and salient objects carry massive sentiments. We propose an algorithm to discover affective regions via deep framework, in which we use an off-the-shelf tool to generate N object proposals from a query image and rank these proposals with their objectness scores. Then, each proposal's sentiment score is computed using a pre-trained and fine-tuned CNN model. We combine both scores and select top K regions from the N candidates. These K regions are regarded as the most affective ones of the input image. Finally, we extract deep features from the whole-image and the selected regions, respectively, and sentiment label is predicted. The experiments show that our method is able to detect the affective local regions and achieve state-of-the-art performances on several popular datasets.

Datasets

• Download Abstract and Artphoto dataset click here. [cite]
  @inproceedings{machajdik2010affective,
    author = {Machajdik, Jana and Hanbury, Allan},
    title = {Affective Image Classification Using Features Inspired by Psychology and Art Theory},
    year = {2010},
    booktitle = {Proceedings of the 18th ACM International Conference on Multimedia},
    doi = {10.1145/1873951.1873965}
  }
  
• Download ArtEmis dataset click here. [cite]
  @InProceedings{Achlioptas_2021_CVPR,
    author = {Achlioptas, Panos and Ovsjanikov, Maks and Haydarov, Kilichbek and Elhoseiny, Mohamed and Guibas, Leonidas J.},
    title = {ArtEmis: Affective Language for Visual Art},
    year = {2021},
    month = {June},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  }
  
• Download B-T4SA dataset click here. [cite]
  @InProceedings{Vadicamo_2017_ICCV,
    author = {Vadicamo, Lucia and Carrara, Fabio and Cimino, Andrea and Cresci, Stefano and Dell'Orletta, Felice and Falchi, Fabrizio and Tesconi, Maurizio},
    title = {Cross-Media Learning for Image Sentiment Analysis in the Wild},
    year = {2017},
    month = {Oct},
    booktitle = {Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops}
  }
  
• Download Comics dataset click here. [cite]
  @article{she2019learning,
    author = {She, Dongyu and Sun, Ming and Yang, Jufeng},
    title = {Learning Discriminative Sentiment Representation from Strongly- and Weakly Supervised CNNs},
    year = {2019},
    issue_date = {January 2020},
    publisher = {Association for Computing Machinery},
    volume = {15},
    number = {3s},
    issn = {1551-6857},
  }
  
• Download EMOd dataset click here. [cite]
  @InProceedings{Fan_2018_CVPR,
    author = {Fan, Shaojing and Shen, Zhiqi and Jiang, Ming and Koenig, Bryan L. and Xu, Juan and Kankanhalli, Mohan S. and Zhao, Qi},
    title = {Emotional Attention: A Study of Image Sentiment and Visual Attention},
    year = {2018},
    month = {June},
    booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}
  }
  
• Download EMOTIC dataset click here. [cite]
  @InProceedings{Kosti_2017_CVPR,
    author = {Kosti, Ronak and Alvarez, Jose M. and Recasens, Adria and Lapedriza, Agata},
    title = {Emotion Recognition in Context},
    year = {2017},
    month = {July},
    booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}
  }
  
• Download Emotion6 dataset click here. [cite]
  @InProceedings{Peng_2015_CVPR,
    author = {Peng, Kuan-Chuan and Chen, Tsuhan and Sadovnik, Amir and Gallagher, Andrew C.},
    title = {A Mixed Bag of Emotions: Model, Predict, and Transfer Emotion Distributions},
    year = {2015},
    month = {June},
    booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}
  }
  
• Download EmotionROI dataset click here. [cite]
  @inproceedings{peng2016emotions,
    author={Peng, Kuan-Chuan and Sadovnik, Amir and Gallagher, Andrew and Chen, Tsuhan},
    title={Where do emotions come from? Predicting the Emotion Stimuli Map},
    year={2016},
    booktitle={2016 IEEE International Conference on Image Processing (ICIP)}
  }
  
• Download Emotion-6 dataset click here. [cite]
  @InProceedings{Panda_2018_ECCV,
    author = {Panda, Rameswar and Zhang, Jianming and Li, Haoxiang and Lee, Joon-Young and Lu, Xin and Roy-Chowdhury, Amit K.},
    title = {Cross-Media Learning for Image Sentiment Analysis in the Wild},
    year = {2018},
    month = {September},
    booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)}
  }
  
• Download Event dataset click here. [cite]
  @inproceedings{ahsan2017towards,
    author={Ahsan, Unaiza and De Choudhury, Munmun and Essa, Irfan},
    title={Towards using visual attributes to infer image sentiment of social events},
    year = {2017},
    month = {Oct},
    booktitle={2017 International Joint Conference on Neural Networks (IJCNN)},
    doi={10.1109/IJCNN.2017.7966013}
  }
  
• Download FI dataset click here. [cite]
  @article{You_Luo_Jin_Yang_2016,
    author={You, Quanzeng and Luo, Jiebo and Jin, Hailin and Yang, Jianchao},
    title={Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark},
    year = {2016},
    month = {Feb.},
    journal={Proceedings of the AAAI Conference on Artificial Intelligence}
  }
  
• Download flickr and instagram dataset click here. [cite]
  @inproceedings{katsurai2016image,
    author={Katsurai, Marie and Satoh, Shin'ichi},
    title={Image sentiment analysis using latent correlations among visual, textual, and sentiment views},
    year = {2016},
    booktitle={2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
    doi={10.1109/ICASSP.2016.7472195}
  }
  
• Download FlickrLDL and TwitterLDL dataset click here. [cite]
  @inproceedings{yang2017learning,
    author={Yang, Jufeng and Sun, Ming and Sun, Xiaoxiao},
    title={Learning visual sentiment distributions via augmented conditional probability neural network},
    year = {2017},
    booktitle={Thirty-first AAAI conference on artificial intelligence}
  }
  
• Download GAPED dataset click here. [cite]
  @article{dan2011geneva,
    author={Dan-Glauser, Elise S and Scherer, Klaus R},
    title={The Geneva affective picture database (GAPED): a new 730-picture database focusing on valence and normative significance},
    year = {2011},
    journal={Behavior research methods},
    publisher={Springer},
    volume={43},
    number={2},
    pages={468--477}
  }
  
• Download IAPSa dataset click here. [cite]
  @article{mikels2005emotional,
    author={Mikels, Joseph A and Fredrickson, Barbara L and Larkin, Gregory R and Lindberg, Casey M and Maglio, Sam J and Reuter-Lorenz, Patricia A},
    title={Emotional category data on images from the International Affective Picture System},
    year = {2005},
    journal={Behavior research methods},
    publisher={Springer},
    volume={37},
    number={4},
    pages={626--630}
  }
  
• Download SE30K8 dataset click here. [cite]
  @inproceedings{wei2020learning,
    author={Wei, Zijun and Zhang, Jianming and Lin, Zhe and Lee, Joon-Young and Balasubramanian, Niranjan and Hoai, Minh and Samaras, Dimitris},
    title={Learning visual emotion representations from web data},
    year={2020},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}
  }
  
• Download StockEmotion dataset click here. [cite]
  @inproceedings{wei2020learning,
    author={Wei, Zijun and Zhang, Jianming and Lin, Zhe and Lee, Joon-Young and Balasubramanian, Niranjan and Hoai, Minh and Samaras, Dimitris},
    title={Learning visual emotion representations from web data},
    year={2020},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}
  }
  
• Download Twitter1269 dataset click here. [cite]
  @inproceedings{you2015robust,
    author={You, Quanzeng and Luo, Jiebo and Jin, Hailin and Yang, Jianchao},
    title = {Cross-Media Learning for Image Sentiment Analysis in the Wild},
    year = {2015},
    booktitle={Twenty-ninth AAAI conference on artificial intelligence}
  }
  
• Download Twitter603 dataset click here. [cite]
  @inproceedings{borth2013large,
    author={Borth, Damian and Ji, Rongrong and Chen, Tao and Breuel, Thomas and Chang, Shih-Fu},
    title={Large-scale visual sentiment ontology and detectors using adjective noun pairs},
    year = {2013},
    booktitle={Proceedings of the 21st ACM international conference on Multimedia}
  }
  
• Download UnBiasedEmo dataset click here. [cite]
  @InProceedings{Panda_2018_ECCV,
    author = {Panda, Rameswar and Zhang, Jianming and Li, Haoxiang and Lee, Joon-Young and Lu, Xin and Roy-Chowdhury, Amit K.},
    title = {Cross-Media Learning for Image Sentiment Analysis in the Wild},
    year = {2018},
    month = {September},
    booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)}
  }
  
• Download VSO dataset click here. [cite]
  @inproceedings{borth2013large,
    author={Borth, Damian and Ji, Rongrong and Chen, Tao and Breuel, Thomas and Chang, Shih-Fu},
    title={Large-scale visual sentiment ontology and detectors using adjective noun pairs},
    year = {2013},
    booktitle={Proceedings of the 21st ACM international conference on Multimedia}
  }
  
• Download WEBEmo dataset click here. [cite]
  @InProceedings{Panda_2018_ECCV,
    author = {Panda, Rameswar and Zhang, Jianming and Li, Haoxiang and Lee, Joon-Young and Lu, Xin and Roy-Chowdhury, Amit K.},
    title = {Cross-Media Learning for Image Sentiment Analysis in the Wild},
    year = {2018},
    month = {September},
    booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)}
  }
  

All Demos

Image Sentiment

Demo for Image Sentiment Recognization.

Try it out

Facial Expression

Demo for Facial Emotion Recognization.

Try it out