After dimensionality reduction of the original time series, if the distance between two points in the original space is less than the threshold, but the distance between the two points in the reduced feature space is greater than the threshold, the false dismissals will be caused.

Faloutsos et al. introduced lower bounding [18]. It is proved that in order to guarantee no false dismissals, the distance measure in the reduced feature space must satisfy the condition as described in Equation (4).

D f e a t u r e ( F ( O 1 ) , F ( O 2 ) ) ≤ D o b j e c t ( O 1 , O 2 ) (4)

where O 1 and O 2 are the two objects in original space, their distance is D o b j e c t (   ) ; F ( O 1 ) and F ( O 2 ) are the feature extracting function, the distance in reduced feature space.

To assess the distance quality, the Tightness of Lower Bound (TLB) [2] metric is used. TLB is defined as Equation (5).

TLB = lower bound distance true euclidean distance (5)

The value of TLB is between 0 and 1. The closer to 1 is this fraction, the better is the TLB, which shows that the distance after dimensionality reduction is closer to the original distance.

Based on this point, we propose an optimal feature vector selection algorithm. For a specific data set, the features which have larger TLB value are selected. Algorithm 1 describes the process of optimizing feature vector.

Firstly, N groups of samples are randomly selected from the training set and the test set. The candidate feature quantity is calculated, and the distance between words is calculated using the classic SAX. Divided by the Euclidean distance between the original samples to get the value of tlb. According to the definition of TLB, the closer to 1 is this fraction, the better is the TLB. Therefore, we keep the k features with the largest tlb mean value as the optimal features. In this paper, the value of N is one tenth of the length of the training set. If the length of the training set is less than 50, the value of n is half of the length of the training set to ensure the stability of the selected features. The k value is set to 4.

From the perspective of pattern recognition [25], it is more efficient to characterize time series using some key classes of methods, including autocorrelation, stationarity, entropy, and methods from the physics-based nonlinear time series analysis [26]. For the candidate feature list func, we refers to the feature extraction method in tsfresh [27] module. SAX_VFD proposed in this paper provides a total of 18 features in three categories: statistical features, entropy features and fluctuation features.

1) Statistical features

There are 9 statistical features, including maximum (max), minimum (min) and mean (μ), median (median), variance (σ²), skewness (SKEW), kurtosis (KURT), range (R) and interquartile range (IQR). X n = 〈 x 1 , x 2 , x 3 , ⋯ , x n 〉 is a time series of length n, the statistical features are calculated by Equations as follows:

μ = 1 n ∑ i = 1 n x i (6)

σ 2 = ∑ i = 1 n 1 n ( x i − μ ) 2 (7)

Algorithm 1. Feature vector selection (T, func, k, ω, α).

S K E W ( X ) = 1 n ∑ i = 1 n ( x i − μ ) 3 σ 3 (8)

K U R T ( X ) = 1 n ∑ i = 1 n ( x i − μ ) 4 σ 4 − 3 (9)

R = max ( X n ) − min ( X n ) (10)

I Q R = Q 3 − Q 1 (11)

2) Entropy features

Usually, the statistical characteristics alone cannot accurately distinguish time series very well. Entropy can be used to describe the uncertainty and complexity of time series. In this paper, four kinds of entropy features of time series are adopted, which are entropy, binned entropy, approximate entropy and sample entropy.

According to the concept of information entropy in information theory, the more orderly a system is, the lower its information entropy is. The more chaotic a system is, the higher its information entropy is. Information entropy is a measure of the degree of system ordering, so it can describe the randomness of time series to a certain extent. Information entropy is calculated by Equation (12).

e n t r o p y ( X ) = − ∑ i = 1 ∞ P { x = x i } ln ( P ( x = x i ) ) (12)

First bins the values of X_n into maxbin equidistant bins, then the binned entropy (bEn) is calculated by Equation (13).

b E n ( X ) = − ∑ K E = 0 min ( m a x b i n , l e n ( X ) ) P k ln ( P k ) ⋅ 1 ( P k > 0 ) (13)

where P_k is the percentage of samples in bin k, maxbin is the maximal number of bins, len(X) is the length of X.

If the bEn is large, it means that the values of this time series are evenly distributed in the range of min(X_n) and max(X_n). If the bEn is small, it means that the value of the time series is concentrated in a certain segment.

Approximate entropy (apEn) and sample entropy (sampEn) are created to measure the repeatability or predictability within a time series. Both features are extremely sensitive to their input parameters: m (the length of the compare window) and r (the similar tolerance border) and n (length of time series).

Approximate entropy is defined as the conditional probability to maintain its similarity when the similarity vector increases from m to m + 1 in dimension as described in Equation (14). The physical meaning is the probability of generating new patterns in a time series when the dimension changes. The more the probability to generate new patterns, the more complex the sequence becomes, and the larger the apEn. In theory, the apEn can reflect the degree and length of the numerical variation.

a p E n ( X ) = Φ m ( r ) − Φ m + 1 ( r ) (14)

The C i m ( r ) values measure the regularity within a tolerance r, or the frequency of patterns similar to a given pattern of the window length m. Φ m ( r ) is the average value of ln ( C i m ( r ) ) , where ln is the natural logarithm.

Φ m ( r ) = ∑ i = 1 N − m + 1 ln ( C i m ( r ) ) N − m + 1 (15)

The calculation process of sample entropy is similar to that of approximate entropy. The natural logarithm is not used to calculate the average similarity rate.

B m ( r ) = ∑ i = 1 N − m + 1 C i m ( r ) N − m + 1 (16)

But the natural logarithm is used to calculate the sample entropy.

s a m p E n ( X ) = − ln [ B m + 1 ( r ) B m ( r ) ] (17)

The smaller the sample entropy is, the stronger the autocorrelation is. In this paper, we utilize the recommended parameter values. The parameter m is set to 3, r is set to 0.2 times the standard deviation of the original sequence.

3) Fluctuation features

In addition, there are five features to characterize the fluctuation of time series. They are slope (slope), absolute energy (abs_energy), sum over first order differencing (absolute_sum_of_changes), average over first order differencing (mean_abs_changes) and the mean value of a central approximation of the second derivative (mean_second_derivative_central).

s l o p e = x p − x q p − q (18)

Here we use the slope of the tangent between the extreme values, where x_p, x_q, p, q represent the extremum and corresponding time axis respectively.

a b s e n e r g y = ∑ i = 1 n x i 2 (19)

Absolute energy value can describe the fluctuation of the squared values of time series data.

A non-stationary time series can be converted to a stationary time series through differencing. First order differencing series is the change between consecutive data points in the series.

a b s o l u t e _ s u m _ o f _ c h a n g e s = ∑ i = 1 n − 1 | x i + 1 − x i | (20)

m e a n _ a b s _ c h a n g e s = 1 n ∑ i = 1 , ⋯ , n − 1 | x i + 1 − x i | (21)

m e a n _ s e c o n d _ d e r i v a t i v e _ c e n t r a l = 1 n ∑ i = 1 , ⋯ , n − 1 1 2 ( x i + 2 − 2 ⋅ x i + 1 + x i ) (22)

Sum over first order differencing and average over first order differencing can describe the absolute fluctuation between consecutive data points in the series. The second derivative is defined by the limit definition of the derivative of the first derivative. In definition, it represents the rate of change of first derivative, that is, the rate of change of fluctuation between consecutive data points.

The technique proposed in this paper is based on the SAX. Therefore, afterk features are selected in the previous section, in each segment, k f feature values can be calculated and mapped to k symbol characters respectively. These k symbol characters constitute a feature string vector for each segment. V i = 〈 f i , 1 , f i , 2 , f i , 3 , ⋯ , f i , k 〉 is the feature string vector of the ith segment. And so on, the original time series T can be represented by in a ω feature string vectors T = 〈 V 1 , V 2 , V 3 , ⋯ , V ω 〉 . Each element in the vector is no longer a single character, but a feature string composed of k feature values.

Taking a time series of length 470 as an example, the length of time series is 470, when the value of ω is 5, α is 6 and k is 4, the optimized feature list is max, min, median and mean. Max values and min values are respectively shown in red circles and in green Triangle, while media values are shown in yellow diamond lines and mean values are shown in brown lines in Figure 2. The characters of different colors in the figure represent they are mapped by corresponding features. Using the technique of SAX_VFD, the original time series can be represented as <“ECED”, “ECDD”, “FCDD”, “FAFE”, “EAAA”>. The length of vector is ω × k = 20 . The feature string vector contains multiple features, so it can better distinguish the segments. It can be observed from Figure 2, the second and third segments’ min, median and mean values are in the same probability interval, but they can still be distinguished by the max values.

The extracted features are expanded from one to k, therefore, the distance measure need to be redefined. Section 2 introduces the distance measure in the

SAX. Based on this, we provide a new concept named the vector frequency difference.

The idea of vector frequency comes from the concept of term frequency in text mining. Term frequency (TF) is used in connection with information retrieval and shows how frequently an expression (term, word) occurs in a document [28]. Term frequency indicates the significance of a particular term within the overall document. The famous tf-idf algorithm believes that the smaller the term frequency, the greater its ability to distinguish different types of text [29].

The distance measure is proposed in this paper based on the assumption that the smaller the frequency of a certain feature vector in the feature vector set of the whole time series data set, the greater its ability to represent time series samples. Therefore, the larger the frequency difference between the two feature string vectors, the greater the difference in the ability to distinguish samples between them, so the corresponding distance between the two should be relatively increased. According to this, the distance measure is redefined.

Given two time series Q and C with the same length n, each segmented sequence can be represented by a feature string vector, which contains k transformed characters, after K features are selected, each segmented sequence can be represented by a feature string vector, and each feature string vector contains k transformed characters. In this case, a feature string vector in Q can be represented as V Q = 〈 q ^ 1 , q ^ 2 , q ^ 3 , ⋯ , q ^ k 〉 , and a feature string vector in C can be represented as V C = 〈 c ^ 1 , c ^ 2 , c ^ 3 , ⋯ , c ^ k 〉 . The distance vdis between two feature string vectors is defined as:

v d i s ( V Q , V C ) = ∑ j = 1 k ( d i s t ( q ^ j , c ^ j ) ) 2 (23)

Each time series is composed of ω feature string vectors, Q ^ and C ^ are the new representation of time series of Q and C. The distance between them is defined as follows:

D D I S T ( Q ^ , C ^ ) = n ω Δ v f ( V Q i , V C i ) × ( v d i s ( V Q i , V C i ) ) 2 (24)

where Δ v f ( V Q i , V C i ) us the frequency difference of two feature string vectors.

One of the most important features of the SAX is that it provides a lower bound distance measure. The lower bound is very useful for controlling the error rate and improving the calculation speed [2]. Because Δ v f ( V Q i , V C i ) < 1 , the new distance measure D D I S T ( Q ^ , C ^ ) proposed in this paper still satisfies the lower bound.

In this paper, we use 24 datasets from UCR repository [30]. These have been commonly adopted by TSC researchers. The basic information of the datasets is shown in Table 2. The types of datasets used are diverse and come from several fields, including sensor data, image contour information data, device data, simulated data, etc. All data sets are of labeled, univariate time series, without any preprocessing. We used the default train and test data splits.

The experiment mainly aims to confirm the proposed distance measure’s effectiveness. We do comparison experiments on the SAX_VFD with the Euclidean

distance and other preciously proposed symbolic techniques, which are the SAX and the ESAX. To the time series classification task, distance measurement will directly affect the classification results. Therefore, we construct 1 Nearest Neighbor (1 NN) classifier based on different distance measures, and compare the error rate to observe the advantages and disadvantages of different distance measures. The advantage of this design is that the underlying distance metric is critical to the performance of 1 NN classifier, hence, the accuracy of the 1 NN classifier directly reflects the effectiveness of a distance measure. Furthermore, 1 NN classifier is parameter free, allowing direct comparisons of different measures.

Three symbolic techniques in comparative experiments, which are the SAX, the ESAX and the SAX_VFD, are impacted by the choice of the segment number ω and the alphabet size α. We try the different combination of these two parameters to test their influence on the results. For ω, we choose the value from 2 to a quarter of the length of the time series. For α, we choose the value in the range of 3 and 10.

We compare the dimension reduction effect of different techniques by the dimensionality reduction ratios when we get the best classification results. The dimensionality reduction ratio is measured as Equation (25).

Dimensionality Reduction Ratio = Number of the reduced data points Number of the original data points (25)

The dimensionality reduction ratio of the SAX is ω n , The dimensionality reduction ratio of the ESAX is 3 ω n , The dimensionality reduction ratio of the SAX_VFD is 4 ω n .

The overall classification results are listed in Table 3, where entries with the lowest classification error rates are highlighted. In all 24 datasets, SAX_VFD has the lowest error in the most of the data sets (17/24), followed by the ESAX (6/24).

We use the sign test to test the significance of our proposed technique against other techniques. The sign test is a nonparametric test that can be used to test either a claim involving matched pairs of sample data.

The sign test results are displayed in Table 4, where n + , n − and n 0 denote on the numbers of data sets where the error rates of the SAX_VFD are lower, larger than and equal to those of another technique respectively. A p-value less than or equal to 0.05 indicates a significant improvement. The smaller a p-value, the more significant the improvement. The p-values demonstrate that the distance measure of the SAX_VFD achieves a significant improvement over the other techniques on classification accuracy.

To provide an illustration of the result in Table 3 more clearly, we use scatter plots for pair-wise comparisons as shown in Figure 3. In a scatter plot, the error rates of two measures under comparison are used as the x and y coordinates of a dot, where a dot represents a particular data set [31]. When a dot falls within a region, the corresponding technique in the region performs better than the other. In addition, the further a dot is from the diagonal line, the greater the margin of an accuracy improvement. The region with more dots indicates a better technique than the other.

In Figure 3, the blue dots indicate the classification error rates are lower in the upper triangle region, the red squares indicate the classification error rates are lower in the lower triangle region. In Figure 3(a), there is no obvious difference between the SAX and ED in these 24 data sets, and the number of points on both sides of the diagonal is basically the similar. In Figures 3(b)-(d), on the

whole, our method is obviously superior to the other three techniques, both in the number of points and the distance of these points from the diagonals.

In order to understand the influence of parameter selection on the performance, we run the experiments on data sets CBF and Small Kitchen Appliances using different parameter combinations. The results are shown in Figure 4. Specially, on CBF, we firstly compare the classification error rates with different ω while α is fixed at 10, and then with different α while ω is fixed at 8; on Small Kitchen Appliances, ω varies while α is fixed at 10, and then α varies while ω is fixed at 16. The experimental results are shown in Figure 5. The SAX_VFD obtain lower classification error rates when the parameter ω are small. For example, of CBF data set, the error rate of the SAX_VFD is 0.1667 while ω is 2, it was better than the SAX (0.4122) and the ESAX (0.5). The techniques are sensitive to the parameter α. Enlarge the value of α is easy to get better classification results. Generally, the SAX_VFD can achieve better accuracy with lower parameter α. These demonstrate that the proposed technique is more significant when the parameters are small.

Totally, 447 experiments are carried out using the SAX_VFD, 1788 optimized features are adopted in these experiments. The results of feature selection are shown in Figure 5. In this divergent bar chart, the green bar on the right

represents the feature that the number of times is greater than the average value of the overall data, and the red bar on the left represents the feature that the number of times used is less than the average value of the overall data. The most frequently selected features are still the statistical features, such as mean, median, minimum and maximum. In the entropy features, the number of selected times of binned entropy is the most, and the slope is dominant in the fluctuation features.

Since one major advantage of the SAX is its dimensionality reduction, we shall compare the dimensionality reduction of the proposed technique with the SAX and the ESAX. The dimensionality reduction ratios are calculated using the parameter ω when the three techniques achieve their lowest classification error rates on each data set. The results are shown in Figure 6. The SAX_VFD is competitive with the SAX on dimensionality reduction. For each segment of time series, the SAX_VFD extract four features, it is more features than the SAX

and the ESAX. However, the SAX_VFD can achieve the lowest classification error rate at a smaller number of segments ω. Therefore, as shown in Figure 6, in the aspect of dimensionality reduction, the SAX_VFD is competitive with the SAX, and has advantages compared with the ESAX. For example, on CBF data set, using the SAX, when the value of ω is 32, the lowest error rate is 0.1040; while using the SAX_VFD, when the value of ω is 8, the minimum error rate is 0.0756, so the dimension reduction rate of both is 0.25.