Due to various issues in underground mine video, such as uneven illumination, drastic light changes, and noise interference, we combined the HSV colour feature [16] with the scale invariant local ternary pattern (SILTP) texture feature [17] to describe the moving target to improve the robustness of the target template. Two feature types are described in this section.

Hue (H), saturation (S), and value (V) are the three components of The HSV colour space [16]. Compared with the three feature components of the red, green, and blue (RGB) colour space, the three feature components of the HSV colour space are uncorrelated and more consistent with the visual characteristics of the human eye. Therefore, the use of the HSV space as a colour feature model can mitigate the interference of lighting. The HSV colour histogram feature extraction method is described as follows [18]:

1) Colour space conversion

According to Equation (1), the image can convert from the RGB to the HSV:

{ V = 1 3 [ R + G + B ] S = 1 − 3 V min ( R , G , B ) H = { θ                                                                                 G ≤ B 2 π − θ                                         G < B (1)

and θ is the angle.

θ = cos − 1 [ 1 / 2 [ ( R − G ) + ( R − B ) ] ( R − G ) 2 + ( R − B ) ( G − B ) ] (2)

2) Conversion of the HSV colour space and histogram

The HSV colour space is uniformly quantized to a 256-dimension histogram using the uniformly spaced quantization method, in which the colour quantities of the H, S, and V were uniformly quantized to 16 parts, 4 parts, and 4 parts, respectively, i.e., the quantization levels of the three quantities are 16, 4, and 4, respectively.

3) The 256-dimension histogram is encoded using the Haar transform method, and the features of the encoded histogram are the desired HSV colour space-based features. As shown in Figure 1.

The SILTP feature is a modified local binary pattern (LBP) description operator with excellent robustness to illumination variation and noise; thus, it can effectively solve the light-derived histogram instability issue as it has a better adaptability to sudden lighting changes. The SILTP feature has excellent robustness to area-wide noise, especially shadows and local noise; thus, it is suitable for extracting features from underground mine video. The SILTP operator uses 00, 01, and 10 to represent the pixel texture, and increases it by one digit compared with the LBP operator. Assume that the position of the centre pixel of the image block is ( x c , y c ) and the grey value of the centre pixel is I_c. The encoding function is defined as

s τ ( I c , I k ) = { 01                           I k > ( 1 + τ ) I c 10                           I k > ( 1 + τ ) I c 00                                                 others (3)

S I L T P Q , R τ ( x c , y c ) = ⊕ k = 0 Q − 1 s τ ( I c , I k ) (4)

The grey value of the Q neighbourhood pixel with a threshold of τ and a

radius of R is I_k.

The binary values of the neighbourhoods of the centre pixel are connected in a counter clockwise direction by ⊕ to form a string

The original encoding process is shown in Figure 2(a); in the case of local noise in the environment, as shown in Figure 2(b), the noise within a certain range did not affect the encoding result. When the light changes, the light intensity is doubled, and the grey value of each pixel and the grey values of the centre point and the neighbourhood are doubled (Figure 2(c)), which causes no changes in the coding result. Therefore, the SILTP has excellent robustness to illumination change and local noise. The advantages of the SILTP operator are reflected in the following three aspects. First, for each neighbourhood, only one more comparison is needed relative to the LBP operator; thus, the computational efficiency is very high. Second, the SILTP operator is very robust to region-wide noise; especially when the detection region is darker or contains more noises, SILTP is more adaptive to these changes than LTP or LBP and generates better inspection results. Last, the scale invariance enables the SILTP operator to be

more robust to brightness change, and even in the case in which the light suddenly changes from dark to bright, the SILTP features do not change. The operator is also able to recognize when the background area is in a weak shadow, as the weak shadow retains the texture information of the dark background rather than the texture information represented by the scale factor of the local background area [19].

A particle filter is a statistical filter method based on Monte Carlo and recursive Bayesian estimation [20]. Recursive Bayesian estimation refers to the process in which the prior distribution is modified by adjusting the weights and positions of the particles based on the posterior probability distribution recursive from the existing prior probability distribution. The key idea of a particle filter is to extract a set of random weighted samples ( { x k − 1 i } , i = 1 , 2 , ⋯ , N ) from the prior probability distribution to approximately represent the posterior probability density function. Thus, the integral operation in the derivation process is converted to the summation operation of finite sample points. The approximation equation of the posterior density function is

p ( x k | z 1 : k ) ≈ ∑ i = 1 N s ω k i δ ( x k − x k i ) (5)

where δ ( x ) is the Dirac function; x k is the state value at time k; and z 1 : k is the observed values from time 1 to time k. Performing sampling from the posterior probability density function is often difficult, and generally, sampling is performed from an alternative significance density function q ( x k i | x k − 1 i , z k ) as the sampling by this approach can be readily performed. The update equation of particle weight in the recursive process is expressed as follows:

ω k i = ω k − 1 i p ( z k | x k i ) p ( x k i | x k − 1 i ) q ( x k i | x k − 1 i , z k ) (6)

After normalizing the weights, the status output is expressed as follows:

x k = ∑ i = 1 N ω k i x k i (7)

Derived from fruit fly foraging behaviour, the FOA is a global optimization-pursuing swarm intelligence algorithm [21]. Based on characteristic fruit fly foraging behaviours, the FOA consists of the following three steps [22]:

1) Population initialization: the number of fruit fly populations (N), the maximum number of iterations (m), and the initialized foraging locations of an individual fruit fly (X₀ , Y₀ ) are determined.

2) Foraging activities: starting from the initial position, the fly swarm uses the olfactory sense to search for food, and the search direction and distance of each individual fruit fly are random numbers with a set range. The fitness value of each individual fruit fly f_i (odour concentration judgement value) is calculated based on its current position (X_i , Y_i ), and the individual with the highest fitness value is determined. The swarm flies towards the position of this individual.

3) Update the swarm position: in each iteration, if the next optimal fitness value is greater than the prior optimal fitness value, the swarm updates its position of the optimal fitness value as a new foraging position (X₀ ,Y₀) towards which all individuals fly and then fly out to continue the foraging. This process is repeatedly iterated until the maximum number of iterations or the set accuracy is attained.

The resampling process of the traditional particle filter algorithm solves the particle degeneracy problem by duplicating large-weight particles and deleting small-weight particles. However, after multiple iterations, the particle depletion issue is observed. Specifically, the rationale of the fruit fly optimization particle filter algorithm is described as follows. In the standard particle filter process, after obtaining N particles by importance sampling, the fitness value of each particle is calculated based on the position of the particle and the fitness function. If the particle set is distributed in the vicinity of the true state, then the fitness value of each particle in the swarm is high; conversely, the particle set is not distributed in the vicinity of the true state. Optimization with the particle distribution by the FOA enables the particles to constantly fly towards the position with a high fitness value to ensure that the particles are constantly getting close to the true state, which improves the quality of the total samples of the particle swarm.

The FOA optimizes and improves the resampling process of particle filtering, in which the signal point particles are represented by the fruit fly swarm, and the more effective particles are obtained via the high optimization capability of the FOA. Thus, the particle depletion problem is solved.

By multiple iterations, the FOA updates the positions of fruit fly individuals, and the individual fruit fly with the highest odour concentration judgement value is deemed the optimal solution. The position update is implemented using the following equations:

X m + 1 i = X m i + 2 r × r a n d o m v a l u e − r (8)

Y m + 1 i = Y m i + 2 r × r a n d o m v a l u e − r (9)

The odour concentration value is controlled by the following equation:

M k i = exp { − 1 2 R k ( z n − z p ) } (10)

where m is the number of iterations, and i = 1 , 2 , ⋯ , N is fruit fly individuals; N is the total number of fruit fly individuals; r a n d o m v a l u e − r is a random number in the range of [ 0 , 1 ] ; R k is the measurement noise variance; z n is the updated measurement value, i.e., the actual distance from the particle to the target; and z p is the predicted measurement value, i.e., the predicted distance from the particle to the fruit fly individual.

The process of the FOA-optimized particle filter resampling is described as follows:

1) Initialization: let m in Equations (8) and (9) be m = 0 , i = 1 , 2 , ⋯ , N ;

2) Olfactory search: update paths according to Equations (8) and (9) to search the particle at the signal point with the largest weight;

3) Evaluation of individuals: calculate the concentration of the odour released by an individual fruit fly ( M k i ) using Equation (10);

4) Visual search: choose the individual fruit fly with a max { M k i } as the particle with a max { ω k i } ;

5) When the termination criteria are satisfied, the particle with the maximum weight is outputted. Otherwise, let m = m + 1 , go back to Step (2) [23].

Sparse representation is using a linear superposition of a base function dictionary to represent the natural images or signals [24]. For a given signal Y ∈ R n and a given base function dictionary D = { d 1 , d 2 , ⋯ , d n } ∈ R m × n , the sparse model of Y can be expressed as

min ‖ x ‖ 0             s .t           Y = D x (11)

where x is the sparse representation of signal Y in dictionary D and ‖ x ‖ 0 is the number of non-zero components in x. When the number of dimensions of dictionary D satisfies the condition of m > n , the linear equation Y = D x is an over determined condition, and a unique solution forx exists. When m < n , Y = D x is an underdetermined equation, and the solution of x is not unique. Certain constraints need to be added; thus, solving Equation (11) is converted to the optimization of Equation (12).

min ‖ x ‖ 0             s .t         ‖ Y − D x ‖ 2 2 ≤ ε (12)

Equation (12) is a sparse model with the condition that Y satisfies the approximation error, where ‖ x ‖ 0 represents the number of non-zero components in x, i.e., the sparsity of the matrix x, and ε represents the error. The main framework of the sparse representation-based target tracking algorithm is shown in Figure 3. The target template is manually chosen in the initial frame, and the target template dictionary is constructed. When tracking the current frame, particle sampling is performed on the current frame to obtain the candidate target. The sparse representation coefficient of each candidate target in the dictionary is calculated using the sparse representation method. The state of the target in the current frame is assessed based on the sparse representation coefficient, and thus, using the target in the current frame to update the dictionary.

However, directly solving the norm of l₀ in Equation (12) is an NP-hard problem, which is generally solved by the approximation method. The algorithms for sparse solution can be approximately categorized into three categories:

1) Greedy algorithm

The core idea of the greedy algorithm is that the maximum value of the inner product of the residual of the previous step and the optimal atom is chosen in each step. For the minimization of thel₀ norm, the solutions that use the greedy algorithm include the matching pursuit (MP) algorithm [25], which represents the first greedy algorithm, the orthogonal matching pursuit (OMP) algorithm [26] and its modified algorithms [27] [28]. The OMP algorithm uses orthogonal projection transformation to project the base vector into the subspace and then calculates the residual of the representation, which is iterated until the sparse coefficient of the original signal is obtained. Compared with the MP algorithm, the computational efficiency of the OMP algorithm has been significantly improved. The MP algorithm is adopted in this study to solve Equation (12).

2) Convex relaxation algorithm

The convex relaxation algorithm relaxes the minimization problem of the norml₁ into the minimization problem of the norml₁ with restricted isometry property (RIP) conditions. Chen [29] et al. proposed the basis pursuit (BP) algorithm, which employs the interior point method for linear programming to solve the minimization problem of the norml₁, which represents the optimization method by solving the approximated norm of the norml₁.

In this study, the OMP algorithm is used to solve the sparse coefficient of Equation (12).

During the tracking process, affected by various factors, such as posture change, brightness change, and occlusion, the appearance of the target may change. Thus, the target template set should be updated to adapt to changes in the appearance of the target. In this study, the following update strategy is adopted:

1) The target that is manually selected in the first frame to be unsubstituted, i.e., the first component of the target template set, is maintained;

2) Starting from the second frame, after determining the target y ^ , the probability of the similarity between the target template set and y ^ is evaluated. If the probability is lower than a given threshold, then the poorest target template is replaced with y ^ [30].