As mentioned before, the appearance features are individually obtained from four independent facial regions. Thus, for each facial region we applied the proposed feature extraction method independently. Therefore, the first step of this process is facial region segmentation which is based on the distance between irises and its relation with the four facial regions [10] . Figure 1 illustrates an example of this automatic segmentation.

Our proposal is based on the sub-block analysis of eigenphases algorithm presented in [16] which concludes that for face recognition, the ideal size of a sub-block for facing illumination problems is the smallest possible. Since the minimum size of a sub-block is equal to 2 × 2 pixels, and due to the complex part of the Fourier transform being equal to zero in this particular case, we propose to use Fourier coefficients instead of the phase spectrum for the feature extraction process. For instance, consider as the FR-th facial region image of size, and for convenience I represents one of the four facial regions which is subdivided into sub-blocks of size. Then, the 2-D DFT of each sub-block of the facial region is given by

where, and represents the -th sub-block of the facial region I. Since the minimum sub-block size is, the imaginary component of complex Fourier coefficients is equal to zero so that

where and are the real and imaginary components of respectively. Thus, the final Fourier coefficient matrix is given by

where F is the Fourier coefficient matrix of the facial region, which has the same dimensions as I. Finally, in order to have correlated information with the set of training images and for dimensionality reduction we apply PCA. To this end, this confusion matrix should be converted into a column vector, so that

where is the column vector and. Moreover, Fourier coefficient vectors are independently projected into the discriminative facial region subspace obtained from the training set. These projections are considered as appearance feature vectors of the current facial region, given by

where is the facial region subspace and is the mean vector of all training images. It is worth noting that this PCA process is applied retaining 90% of the variance, so is calculated using the necessary eigenvectors for retaining that percent of the variance.

Geometric features are also focused in four independent regions of the face. However, they define the shape of specific facial regions instead of texture information. The shapes of eyes-eyebrows, nose, lips and the face outline were defined by manually picking up 163 feature points [21] . Figure 1 shows an example of these four shapes. It is worth noting that this paper intends to precisely analyze facial expression from two different racial groups which may imply subtle differences among specific facial regions. In this way, feature point extraction is crucial for the accuracy of feature extraction settling the analysis efficiency. Therefore, the feature points were extracted manually, even though we have also developed an automatic extraction method of a large number of feature points. The results of this automatic process tend to be affected by shooting conditions and it is still not easy to guarantee the sufficient accuracy for the analysis.

As well as the previous method, our proposal for geometric features is based on the Fourier transform. However, the application of DFT in this particular case is known as Fourier Descriptor (FD), which is a contour-based shape descriptor widely used for content-based image retrieval (CBIR) [22] . Moreover, FD has been used a few times for face recognition [23] , and as far as the authors know, it has not been applied to FER. For that reason, we proposed to use a variation of FD and PCA as a feature extraction method for these kinds of features. To this end, each facial region shape is considered as K-point coordinate pairs, K being the number of facial feature points of the shape, where K = 54, 42, 29 and 38 for eyes-eyebrows, nose, lips and the face outline, respectively. Thus, we assume that a specific shape of the FR-th facial region is represented as the sequence of coordinates:

where. From one of the four facial regions, complex numbers have to be generated from each coordinate pairs, as in

where represents the centroid of the shape, which is the average of the coordinate pairs so that

Subsequently, the DFT of is given by

for, where represents the Fourier coefficient vector which has to be projected into the current facial region subspace obtained by applying PCA to the training set. Therefore, geometric feature vectors are defined as

where is the current facial region subspace and is the mean vector of all training images. It is important to mention that unlike the appearance-based process given by Equation (5), here 99% of the variance is retained.

The combination of geometric and appearance features has been successfully applied for FER [7] [8] [9] . Some approaches perform the fusion at classification level but better results are obtained when the combination is done at the feature extraction phase [8] . From the literature we can see that feature extraction methods based on hybrid features are focused on appearance features and improve the final feature vector by adding facial landmarks. In this way, the characteristics of both types of features are different and the final feature vector could be directly affected by the individual problems of one kind of features. Therefore, the fusion process of our proposal is based on the application of PCA for correlating the information of both types of features. In addition, the proposed feature extraction method employs the same principle based on the Discreet Fourier Transform. Thus, the appearance features are extracted by using the 2D DFT and the geometric features by FD. It is worth noting that in order to efficiently correlate the features, PCA process has to be applied individually before the fusion.

The process of the complete fusion process is shown in Figure 2, where first we obtained individual feature vectors of appearance and geometric features, as defined in Equations ((5) and (10)), respectively. Thus, the process begins with the concatenation of both feature vectors, so that

Finally, the projection of H onto a new subspace of hybrid features of the current facial region represents the hybrid feature vector, which is given by

where is the current facial region subspace obtained by retaining the 99% of the variance, and is the mean hybrid vector of all training images. It is worth noting that only represents the feature vector of an individual facial region. Therefore, the final feature vector which is based on all possible combinations of the four facial regions is defined as

where represents one of the four hybrid feature vectors based on a specific facial region and Y the concatenation of them. Thus, it can be conformed up to four individual facial regions.

In order to design a FER system robust to the static structure of individual faces, some works assumed that facial expressions can be represented as a linear combination based on the difference between expressive facial images and neutral image of the same subject [23] [24] . It is well known that the structural characteristics and texture information which define a specific expression appear when the face images change from neutral to expressive, hence the difference image may represent those changes and it can reduce the dependency on the subject’s identity as well. Therefore, difference images may also reduce the physical differences among faces from different races and be focused only in the way to constitute the facial expressions. For that reason, we propose to make that difference between feature vectors instead of the raw images, so that

for, where Q is the total number of expressive images in the dataset, and represents the final feature vectors of expressive and neutral facial image, and Z the difference feature vector. It is important to mention that for this process, information of neutral images must be considered on each facial region subspace (W) defined in Equations ((5), (10) and (12)).

As a visual example of the effect of mentioned subtraction, Figure 3 shows two difference images of subjects from different races showing the same facial expression. As we can observe, the expressive images in the first column present physical changes related to gender and race. These noticeable differences between images of the same expression may affect the classification process. On the other hand, the difference images (third column) present clear similarities based on the facial actions produced by the expression rather than physical differences.

The visual representation of appearance features is based on the well-known algorithm of Eigenfaces. The ability of this method for reconstructing images from projected vectors of a previously defined Eigenspace has been successfully applied for analyzing facial expressions [25] . Therefore, using reconstructed images from feature vectors gives the opportunity to analyze the differences and similarities which may appear among the basic expressions of different cultures in detail. To this end, we have to obtain average projected vectors by each expression, which can be calculated from a cultural-specific or multicultural dataset. These average vectors are given by:

for which represents the number of basic expressions and P(r) the number of images per expression, so that. As mentioned before, in order to have a better analysis of facial expressions the number of frames per expression should be equal, then. Finally, reconstructed images are the reshaped matrix of reconstructed average projected vectors given by:

for, where W is the subspace where the Z projections were made, and µ is the mean feature vector of all training images. An example of one reconstructed image from an average happy expression is illustrated in Figure 4.

It is known that the raw data of geometric-based methods are easier to visualize than those of appearance-based. However, their projections made by PCA are more difficult to be visually analyzed. Therefore, in order to accurately analyze

the geometric features, we employed the DrawFace tool [26] , which has been successfully applied for representing facial expressions from geometric feature vectors using PCA [19] . DrawFace tool draws caricatures based on the Eigenfaces process, hence this tool requires to input individual eigenspaces of facial regions and the mean facial shape as initialization setup.

Figure 5 presents the general framework of DrawFace tool. Similar to the PCA process, the mean face of the complete dataset has to be subtracted from the input, but the eigenspaces are calculated individually for each facial region and the placement of them. Thus, the final caricature is drawn by integrating the projections of the input features into its respective eigenspace. In this way, these caricatures can be considered as a result of a projected feature vector into a subspace made by a set of specific facial shapes. For this analysis, we use the eigenspaces and average projected vectors as obtained by Equations ((10) and (15)). Examples of caricatures of an average happy expression and individual surprise face from ASN dataset can be seen in Figure 4 and Figure 5 respectively.

The experiment was applied to 40 students of the University of Electro-Com- munications in Tokyo Japan. Participants were divided into two groups: 20 East-Asians that include Japanese, Taiwanese and Chinese students (50% males); and 20 Western-Caucasians that include German, Swedish, American and Mexican students (50% males). Their ages ranged from 20 to 26 years old (mean 22). It is important to mention that the Western-Caucasians are currently exchange students and had newly arrived in an Asian country for the first time with a residence time no longer than 3 months on average.

A GUI-based script for collecting and presenting the survey was developed in MATLAB. After a brief explanation of the experiment by the software, the stimuli were automatically presented one by one. The instructions of the experiment were presented in Japanese and Chinese for East-Asian participants and in English for Westerns. Each stimulus appeared in the central visual field and remained visible just for 3 seconds, followed by a 6-way forced choice decision question related to the 6 basic expressions. The question presented was “What is the expression of the face?” and the participant has to choose one answer before clicking the button “Next” for the following stimulus. We randomized trials within each participant and all of them have to recognize the 240 stimuli of the dataset which have been presented by groups of 30 images with breaks of 30 seconds between them. Figure 6 presents two screenshots of the application software used for this study.

FER test, visual analysis and human study were evaluated using the same datasets. A total of 656 images were used for this paper. 480 expressive and 176 neutral facial images were selected from four standard datasets which were divided into two racial groups, representing three possible training sets: Western-Cau- casian dataset (WSN), East-Asian (ASN) dataset and the multicultural dataset (MUL).

WSN dataset is comprised of 330 facial images from 90 different subjects (40% males) of the extended Cohn-Kanade dataset (CK+) [27] , which in turn is com-

posed of 327 facial image sequences from 123 subjects performing the 6 basic emotions plus the neutral state. For this paper, the subset selected from CK+ only includes images of 90 Euro-American subjects.

ASN dataset contains 326 images selected from 86 subjects(47% males) of three different datasets: Japanese Female Facial Expression (JAFFE) dataset [28] , which comprises 213 images of 10 Japanese female models; Japanese and Caucasian Facial Expression of Emotion (JACFEE) dataset [29] , which contains 56 images from different individuals including 28 Japanese and 28 Caucasian subjects; and Taiwanese Facial Expression Image Database (TFEID) [30] , which includes 336 images from 40 Taiwanese models. In this way, ASN dataset is formed by 86 subjects from Japan and Taiwan.

Finally, MUL dataset is a combination of WSN and ASN. It is worth noting that the number of images per facial expression is equitable among each dataset, being 40 images per expression for ASN and WSN so that 240 expressive images correspond to each dataset. However, the neutral facial images vary from each dataset. Figure 7 shows an example of the six basic expressions displayed by subjects from WSN and ASN datasets.

All facial images were pre-processed in order to have the same inter-ocular distance and eye position as well as cropped with 280 × 280 pixels. For the regions of forehead, eyes-eyebrows, mouth and nose their sizes were normalized at 100 × 70, 200 × 80, 140 × 80 and 175 × 50 pixels respectively.

Feature vectors of the six basic expressions were classified by multi-class SVMs with RBF kernels [31] , and evaluated by leave one subject out cross-validation. Average recognition rates and confusion matrices are presented to show the accuracy of FER. The visual analysis is focused on the changes of facial images based on the FACS, for further information refer to [3] . Table 1 shows all results obtained by the proposed method and human study.

It is worth noting that the results of Table 1 were obtained by using hybrid features from facial regions obtained by combining appearance and geometric

features. Because the facial region of the forehead cannot be represented by geometric features, for convenience, this region was fused with the outline shape of the face. Other regions were fused with the same facial region of its counterpart kind of features.

From Table 1 we can observe a trend of the results which follows the order from top to bottom the accuracy obtained by the tests of in-group, out-multi- cultural, multicultural and out-group. Thus, the best result is reached by the in-group test of WSN (WSN vs WSN), followed by the out-multicultural test of the same dataset (MUL vs WSN) and the third place is for the multicultural test (MUL vs MUL). This trend is also followed by the human study, showing that the WSN dataset is easier for classifying the prototypic expressions.

The analysis per facial region combinations shows that the combination of E-N-M performs better in most of the training combinations. On the other hand, the best single region for FER is the mouth, followed by the nose. However, the eyes regions present interesting results, especially for the out-group and out-multicultural analysis, where this region seems to define the expressions of ASN better. In order to analyze the performance per expression of those results, Table 2 and Table 3 show the accuracy of the mouth region and E-N-M combination. In these tables, we can notice that the mentioned trend is not followed by all the expressions. For example, the out-multicultural test of the mouth shows better accuracy when anger is tested using ASN dataset rather than WSN. In summary, we can observe that the clearer difference between cultures is presented in the out-group test.

In order to have a robust results evaluation, an analysis of the differences between both racial groups based on its expressive faces must be done. Therefore, Figure 8 shows the visual representation of the average expressions of both datasets obtained from appearance and geometric features. The expressions represented by each row from left to right are: anger, disgust, fear, happiness, sadness and surprise. From this illustration, we can observe the same distinctive patterns of difference from specific expressions. For instance, disgust and fear expressions look different. Disgusted faces from WSN fulfill the necessary AUs to be classified as disgust (AU9, AU15, AU16). However, the same average face from ASN presents an extra AU22 and AU23 which are known to appear in anger expression. The average face of fear from WSN again covers the EMFACS for fear expression (AU1, AUN2, AU4, AU5, AU7, AU20, AU26). However, that of ASN lacksAU4 and AU20, given the impression of surprised in the eyes region and the mouth is not well defined.

Another way to analyze the distinctive properties of average projected vectors is simply by plotting them. In this way, we can visualize the behavior of each feature vector and the capability of discrimination of the six basic expressions based on them. Figure 9 shows the six average expression vectors of each dataset. It is easy to see that the six vectors of WSN present better distinctiveness

among themselves rather than those of ASN, which have problems especially for the expressions of fear, disgust and anger.

Thanks to the visual analysis we can better analyze the misrecognition problems which appear among the six basic expressions. Figure 10 and Figure 11 show the confusion matrices for WSN and ASN of Mouth and Eyes region, respectively. Note that these matrices present a gray scale color map for simplify-

ing the visualization, showing higher results in black and lower in white. The confusion matrices represent the true expression on the vertical axis and the decision made by the classifier is presented on the horizontalaxis so that each row of the matrix indicates the level of confusion of each expression with its counterparts.

From these confusion matrices, we can observe that the misclassification problems mainly reside in the expressions of fear and disgust for both facial regions. Some examples of the faces misrecognized even for the human study are presented in Figure 12, which shows the original expressive facial image, its feature vector extracted using E-M-N combination and the visual representation using appearance features (difference image).

Finally, Table 4 presents a comparison of our results with those found in the literature. A few studies have analyzed the cross-cultural recognition capabilities of their proposals. Nevertheless, none of them have covered the out-group multicultural analysis. Even though the recognition accuracy is vastly ranged, the same trend is found: the results of in-group tests are better than those of out-group and WSN datasets achieve higher accuracy than ASN.

It is important to mention that the results from Table 4 widely range from ours, mainly because of some specific reasons: the inconsistency on the number of samples per expression for training (some expressions used more than 200% of samples than others); the algorithms used for feature extraction (all of them only used appearance features for this task); and the null consideration of the static structure of individual faces for the analysis (some of the studies consider the neutral face as an extra class for the recognition task but not for the analysis itself). As an extra reference from the setup of mentioned works, Table 5 shows the datasets used for defining the datasets of WSN and ASN.