Emotion Model is the basis of facial expression recognition system. The constructed emotional model should not only match facial expressions with emotions, but also reflect the location relationship between different emotions. In this way, it is easy to understand the current emotion of an individual through the analysis of the acquired facial expression information. This paper constructs an improved three-dimensional model for emotion based on fuzzy theory, which corresponds to the facial features to emotions based on the basic emotions proposed by Ekman. What’s more, the three-dimensional model for motion is able to divide every emotion into three different groups which can show the positional relationship visually and quantitatively and at the same time determine the degree of emotion based on fuzzy theory.
The Emotion Model uses mathematical ideas to formalize changes in emotions, carries out quantitative analysis of emotions, and then reflects the changing trends of people’s psychology and emotions. At present, the existing emotion models have one-dimensional, two-dimensional, three-dimensional, four-dimensional representations, including OCC models (emotion model constructed by Ortony, Clore and Colins), Russel’s PAD model (Pleasure-Displeasure, Arousal-Nonarousal, Dominance-Submissiveness) emotional model, Plutchik’s cone-shaped model etc. Based on these classic emotional models, a glittering array of scholars have analyzed, improved and studied emotional models from various aspects. M. Hima Bindu, Priya Gupta, and USTiwary proposed 22 basic emotions in the corresponding OCC model in the 3D PAD model. Finally, the expression recognition was classified by changes in the face, mouth, etc., from which we can find out the relation between them [
By referring to the models mentioned above, this paper will mainly be focusing on the application background of the model, the influence of personal personality and external environment on emotions, and how to make emotions more detailed, aiming to make the model more in line with the actual situation as well as emotion and be more specific. However, the models proposed in the above literature are not intuitive and can not be quantitatively expressed. This paper proposes the three-dimension emotion model based on fuzzy theory can express the position relation of emotion intuitively and quantificationally. At the same time, the fuzzy theory is used to realize the division of each emotion in different degree.
Human beings express their emotions through facial expressions mainly by using special movement changes in various parts of the human face. Therefore, the premise of obtaining emotional information is that the emotional types need to correspond to facial expressions. By studying the different expressions of the faces of people from different countries, Ekman defines six types of emotions corresponding to the associated facial expressions: anger, disgust, fear, happiness, sadness, and surprise [
Emotion has multiple dimensions, which can be deemed as a feature of emotion. Different emotions of people are in different positions in the dimensional space
| Emotion Types | Mouth Expression | Eye Expression | Eyebrow Expression |
|---|---|---|---|
| Happiness | The lips are raised and pulled back; The mouth may be enlarged; May expose teeth; | The lower eyelid is pushed upward by the facial muscles; There may be wrinkles under the lower eyelids; | The eyebrows are slightly bent down; |
| Fear | Opened mouth; Pulling the lip back; The lip angle may be elongated; | The upper eyelid is lifted and the lower eyelid is tightened; | The eyebrows are raised and wrinkled together; Wrinkles appear in the middle of the forehead; |
| Surprise | The lower jaw falls; the mouth is open; Lips and teeth apart; The mouth is not stretched and not nervous; | Eyes are wide; Upper eyelid elevation; Lower eyelid drop; | The eyebrows are lifted and bent; Stretching the skin under the eyebrows; Wrinkles may span the forehead; |
| Sadness | Pull down the corner of the mouth; The corner of the mouth may tremble; | The upper eyelid of the intraocular corner is raised; | Skin muscle folded between the two eyebrows; Stacked together; Part of the eyebrows are raised; |
| Disgust | The upper lip is slightly raised; Combine the lips and push up on the upper lip; The corner of the mouth is pulled down and the lips are slightly convex; | There are non-longitudinal folds under the skin of both eyes; Pushing the lower eye muscles under the eyes; | Lowered eyebrows and lowered upper eyelids; |
| Anger | When the mouth is closed, the lips are straightened to the left and right; when the mouth is opened, the lip angle is pulled down; | The lower eyelids are tightened, raised or not lifted; the upper eyelids are tightened; the eyes are wide and may bulge; | The eyebrows are wrinkled together and depressed; Vertical wrinkles appear between the eyebrows; |
and are gradual. What’s more, the relation of different emotions can be demonstrated based on their position and distance and the application of dimension theory to emotion model establishment conforms more to the emotional changes and relationships of human beings in actual situation.
On the basis of a great deal of different dimensional models constructed by different scholars, as shown in
As one’s emotions change, the corresponding emotional position in the three-dimensional state-space emotion model will move to another position. Similar to the above process, the process of emotional change can be referred to as the Markov state-transition matrix at various locations in this confined region [
There are m kinds of emotion types, and each type of emotion can be divided into n levels, and the emotion space contains n m state. Let l = n m the Markov probability transfer matrix of the l dimension will be the formula (1).
P = [ p 1 , 1 p 1 , 2 ⋯ p 1 , j p 2 , 1 ⋮ p i , 1 p 2 , 2 ⋮ p i , 2 ⋯ ⋱ ⋯ p 2 , j ⋮ p i , j ] (1)
In formula (1), p i , j is probability, indicating the possibility of the i change to j emotion. p i , j will meet the following relationship:
∑ j = 1 l p i , j = 1 , ( i ∈ [ 1 , 2 , 3 , ⋯ , l ] ) (2)
The above process is a description of the probability that emotions change from one type to another. In this way, each emotion in the emotional model space can be divided into various degrees, and the possibility of emotional change is affected by the position and distance of coordinates corresponding to different emotions in three-dimensional space.
In order to evaluate and measure the expression of emotions in the emotional space, the mathematical tool of entropy can be used. For the sentiment model space containing l points and m dimensions expressed in formula (2), if the emotion is the i category at a certain time, the probability of the transition to other categories constitutes the probability vector of the emotional change:
e ˙ 1 = ( p i , 1 , p i , 2 , ⋯ , p i , i , ⋯ , p i , l ) (3)
Emotional entropy is defined as:
A i = − C ∑ i = 1 l p i log p i (4)
In formula (4), the emotional entropy of i state is A i so the probability that i emotional state changes to j emotional state is p i , i and C is constant.
The established emotional entropy shows and illustrates the possibility of emotional change. If the entropy value is large, it indicates that the emotional state transitions faster, which is reflected in the change of personality; if its value is too small, it indicates that the emotional state transition is slow. Based on the constructed sentiment model, the probability function of different types of emotions can be obtained, and then the corresponding entropy value can be calculated. The obtained entropy can be used to measure the different types of sentiment probability functions. In some respects, emotional entropy establishes a bond between macro and microcosmic emotional changes [
1) The possibility that an emotional category in an emotional space is transferred to another category (including itself) decreases as the distance between the category and other categories increases.
2) If there is no interference from the outside world, the emotion gradually approaches the vicinity of the origin (calm point), that is, the vicinity of the origin is a stable point, and the emotion gradually becomes stable.
3) If there is a contradiction between emotions, the probability of simultaneous performance is extremely small. For example, the two emotions of happiness and fear have a very low probability of simultaneous performance.
In actual daily life, lots of things, laws, and phenomena are described as uncertain, transitional, and non-absolute. For example, the temperature, the degree of beauty, the degree of youthfulness, etc., cannot be accurately quantified by traditional mathematics description. In order to solve these uncertainties, non-absoluteness, and transitional problems, fuzzy theory is adopted [
Set a domain as U, for any u ∈ U define a mapping to the closed interval [0, 1] as shown in equation (5):
u A ( u ) : U → [ 0 , 1 ] (5)
In formula (5), A is a fuzzy set on the domain U, also called fuzzy subset, as the membership function A of the mapping shown in formula (5), u A ( u ) is the membership degree A of the pair .
As can be seen from the above definition, the membership function u A ( u ) is a fuzzy measurement method, the size of u value indicates the extent of the degree of belonging, and its range is [0, 1]. If u A ( u ) tends to 1, it means that the degree of u belonging A is high, and if u A ( u ) is close to 0, it means that the degree of u belonging A is low. There are three ways to represent the membership function:
1) In general, it can be expressed as:
A = { ( u , u A ( u ) ) | u ∈ U } (6)
2) In the case that U is finite set, the fuzzy set is:
A = ∑ i u A ( u i ) u i (7)
In the formula (7), ∑ on the right side represents a symbol, and does not represent the addition of numbers. Or it can be expressed as a vector form of Equation (8), in which case the order of the elements in each set has been determined:
A = ( u A ( u 1 ) , u A ( u 2 ) , ⋯ , u A ( u n ) ) (8)
3) If the domain U is an infinite set, you can represent the fuzzy set A as:
A = ∫ U u A ( u ) u (9)
The premise of establishing a fuzzy set is to construct the membership degree. So far, the methods of constructing the membership function include a variety of methods. The fuzzy statistics method and fuzzy distribution are introduced below.
The fuzzy statistical method to obtain the mapping function from the domain U to the closed interval [0, 1] mainly includes four elements: the domain U; one element u 0 in the domain; a variable set A * of boundaries, A * is corresponding to a fuzzy concept a and fuzzy set A; condition s. In the idea of fuzzy statistical method, the appearance of ambiguity is caused by the uncertainty of the division s of corresponding concepts a in statistics.
In the process of realizing the fuzzy statistical method, it is necessary to judge whether u 0 belongs to by multiple tests A * , the number of u 0 ∈ A in the subtest n is calculated, and the membership frequency of u 0 can be obtained.
Frequencyof u 0 ∈ A = Numberof u 0 ∈ A n (10)
When the number of times increases gradually or even infinity, the frequency gradually remains unchanged, and the membership degree u 0 belongs to A is:
u A ( u 0 ) = lim n → ∞ Numberof u 0 ∈ A n (11)
The fuzzy distribution is a mapping function from the domain U to the closed interval [0, 1]. The method uses experimental statistics to obtain the approximate distribution of membership degrees, and compares with some functions to select similar function distributions. Finally, the distribution function related parameters are obtained on the basis of experiments. Commonly used fuzzy distributions include parabolic distribution, normal distribution, rectangular distribution, ridge distribution, Cauchy distribution, and trapezoidal distribution. The commonly used constructor domain U to the closed interval [0, 1] mapping function also includes expert binary comparison sorting method, illustration method, empirical method and so on. The construction methods of the above various mapping functions are based on experiments and experience. The construction of the membership function is realized by the above method, which paves the way for the application of the fuzzy theory.
According to the mapping function determined by the above process, the “If...Then...” rule can be defined. In the rule, the “If...” part is a precondition for the rule, and the “Then...” part is a condition based conclusion. The rules of “If...Then...” consist of three processes:
・ Determine the membership value of each proposition in the “If...” section.
・ If the “If...” part consists of multiple parts, use the fuzzy operator to calculate the total membership value of “If...”.
・ Determine the degree to which the conclusion is true according to the preconditions of the rule.
In the application of human-computer interaction systems, it is very important to obtain emotional information through recognizing facial expressions. It is very important to construct a suitable emotional model first. The model not only makes different facial expressions correspond to different emotions, but also reflects the positional relationship of different emotions. In the basic emotion model, Ekman’s six basic emotions correspond to facial expressions and emotions, but they do not indicate the relationship between different emotions. However, the three-dimensional state space emotion model takes anger, joy and fear as the axis, which can not only intuitively and quantitatively describe the type of emotion, and also analyze the probability of emotional changes. Thus combined with the expression recognition system applied in the human-computer interaction system, it has the following shortcomings:
・ The types of emotions considered in some models cannot be obtained only by expression classification recognition.
・ The model does not consider how to extract detailed main emotional information from facial expressions.
Combined with the application background of human-computer interaction system, in order to further improve the emotion model, this paper improves the original three-dimensional state space emotion model in the following aspects:
・ In order to match emotions with facial expressions, six types of emotions are selected as emotion categories. Ekman maps basic emotions to related basic facial expressions and defines six basic emotions including anger, fear, sadness, happiness, surprise, and disgust.
・ In order to visually express the position and relationship of different emotions, the three-dimensional state space method emotion model is selected as the basic structure model, considering three basic emotions, namely fear, anger, joy, and certain emotions at a certain time. These three basic emotions are added by different weights.
・ Projecting the six types of emotions in the selected Ekman into the state space emotion model to intuitively and quickly obtaining the positional relationship between the recognized emotion and the expected emotion from the three-dimensional map based on the expression recognition.
・ According to the actual situation, we can know that each type of emotion has a certain range of variation. Therefore, in order to distinguish the intensity of each type of emotion, combined with the fuzzy theory, each type of emotion is defined to have a certain spherical range, and different areas in the range are expressed. The degree of strong, medium, or weak emotions of a certain kind of emotion is defined by fuzzy rules. The emotional model obtained by combining Ekman’s basic emotion, state space method emotion model and fuzzy theory is shown in
Let D show the Euclidean distance from the measured emotion to the standard emotion [
D = ‖ X − Y ‖ = ∑ i = 1 n ( x i − y i ) 2 (12)
In the formula, X = ( x 1 , x 2 , ⋯ , x n ) T and Y = ( y 1 , y 2 , ⋯ , y n ) T show emotion
vectors of the standard emotion of n dimension and tested emotion of n dimension. Define the membership function:
u A ( X ) = { [ 0.00 , 0.35 ) ( 0 ≤ D ( X ) ≤ r 1 ) [ 0.35 , 0.70 ) ( r 1 ≤ D ( X ) ≤ r 2 ) [ 0.70 , 1.00 ) ( D ( X ) > r 2 ) (13)
According to the definition of the membership function and the actual situation, when the membership degree of each emotion is set [0, 0.35), the degree of emotion is strong. When the membership degree is [0.35, 0.7), the degree of emotion is medium, and the degree of membership is [0.7, 1.0), the level of emotion is weak. In conjunction with
・ If 0 ≤ | D | ≤ r 1 , then the degree of emotion is strong.
・ If r 1 < | D | ≤ r 2 , then the degree of emotion is medium.
・ If r 2 < | D | , then the degree of emotion is weak.
The fuzzy rules defined above are organized into the form shown in
Among them, each type of emotion is divided into different types, that is, each type of emotional degree is divided into six types of emotions r i 1 , r i 2 , i = { 1 , 2 , ⋯ , 6 } , which need to be obtained through experiments. Through the definition of the above fuzzy rules, the quantitative division of the emotional level is realized, that is, the fine classification of the expression is quantitatively realized on the basis of the rough classification of the expression, which is more in line with the actual situation. In the improved emotion model, the one-to-one correspondence between emotion and facial expression is realized, and the advantages of intuitively quantitative analysis of emotional positional relationship and analysis of emotional change probability in the three-dimensional state space emotion model are fully utilized, and the basis of the emotional category is obtained. The definition of fuzzy rules on the above can make the emotional information more detailed. While applying the model to the human-computer interaction system we can help the computer to obtain the user’s current detailed emotional state and improve the human-computer interaction capability.
Through the establishment of the above model, the basic process of using the
| Emotion Types | Strong | Medium | Weak |
|---|---|---|---|
| Happiness | 0 ≤ | D | ≤ r h a 1 | r h a 1 < | D | ≤ r h a 2 | r h a 2 < | D | |
| Fear | 0 ≤ | D | ≤ r f e 1 | r f e 1 < | D | ≤ r f e 2 | r f e 2 < | D | |
| Surprise | 0 ≤ | D | ≤ r s u 1 | r s u 1 < | D | ≤ r s u 2 | r s u 2 < | D | |
| Sadness | 0 ≤ | D | ≤ r s a 1 | r s a 1 < | D | ≤ r s a 2 | r s a 2 < | D | |
| Disgust | 0 ≤ | D | ≤ r d t 1 | r d t 1 < | D | ≤ r d t 2 | r d t 2 < | D | |
| Anger | 0 ≤ | D | ≤ r a n 1 | r a n 1 < | D | ≤ r a n 2 | r a n 2 < | D | |
model to perform facial expression recognition and obtain more detailed emotional information is also determined:
・ Obtaining a useful facial area by detecting a face.
・ Correspond different types of facial expression characteristics to emotion types, and perform rough classification recognition on facial expressions to obtain the main emotion types of facial expressions.
・ Understand the emotion division based on the fuzzy rules so as to get detailed information.
It can be seen from the above process that the model is not only used for the rough classification of emotions and facial expressions in the expression recognition process, but also the fuzzy rules in the model are used to implement the emotional classification of emotions.
This paper introduces a three-dimensional emotion model based on the improvement of fuzzy theory. Firstly, the two basic emotion models of Ekman’s six basic emotions and three-dimensional state space emotion model are introduced. Then the basic knowledge of fuzzy theory is introduced, including the definition and determination of membership function, the definition of fuzzy rules, etc. Finally, the merits as well as demerits of Ekman’s six basic emotion models and three-dimensional state space emotion models in the background of human-computer interaction system are analyzed. Moreover, an improved three dimensional model for emotion based on fuzzy theory is generated, which can map Ekman’s six emotions to three dimensional space-state model and realize a win-win situation. Furthermore, by defining the rules of fuzzy theory, we can make the emotion division more detailed and get more consistent with the actual situation. At the same time, the improved model can determine the basic process to get more detailed emotion information by using expression recognition.
This work was supported in part by a grant from the Characteristics innovation project of colleges and universities of Guangdong Province (Natural Science, No. 2016KTSCX182, 2016), a grant from the Youth Innovation Talent Project of colleges and universities of Guangdong Province (No. 2016KQNCX230, 2016).
The authors declare no conflicts of interest regarding the publication of this paper.
Zhu, Z.J., Li, J.S., Deng, X.G. and Hu, Y. (2018) An Improved Three-Dimensional Model for Emotion Based on Fuzzy Theory. Journal of Computer and Communications, 6, 101-111. https://doi.org/10.4236/jcc.2018.68008