Multi-Information Flow CNN and Attribute-Aided Reranking for Person Reidentification

Abstract

This paper presents a multi-information flow convolutional neural network (MiF-CNN) model for person reidentification (re-id). It contains several specific multilayer convolutional structures, where the input and output of a convolutional layer are concatenated together on channel dimension. With this idea, layers of model can go deeper and feature maps can be reused by each subsequent layer. Inspired by an image caption, a person attribute recognition network is proposed based on long-short-term memory network and attention mechanism. By fusing identification results of MiF-CNN and attribute recognition, this paper introduces the attribute-aided reranking algorithm to improve the accuracy of person re-id further. Experiments on VIPeR, CUHK01, and Market1501 datasets verify the proposed MiF-CNN can be trained sufficiently with small-scale datasets and obtain outstanding accuracy of person re-id. Contrast experiments also confirm the availability of the attribute-assisted reranking algorithm.

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1. Introduction

Person reidentification (re-id) refers to matching and recognizing the identities of pedestrians captured by multicameras with nonoverlapping views, which is significant to improve the efficiency of the security system. Owing to the low resolution of cameras, it is hard to obtain discriminative face features, so the current person re-id methods are mainly based on visual features of pedestrians, such as color and texture [1]. In practice, changes in viewpoint, pose, and illumination among different camera views, as well as partial occlusions and background clutters, pose a great challenge to person re-id [2].

Two principal person re-id methods are feature representation and metric learning. Feature representation seeks to find features with stronger discrimination and better robustness to represent pedestrians. Many kinds of features have been utilized for this, in which appearance features are the simplest and the most popular ones. Color, texture, and shape are the features that can be extracted for human appearance [3]in feature representation, such as HSV color histogram, LBP texture, and Gabor features, and then used for reidentifying people with similarity among pedestrian features. Attribute features are also widely used in person re-id. Common attributes include gender, length of hair, and clothing. These attributes are highly intuitive and understandable descriptors which have proved to be successful in several tasks, such as face recognition and activity recognition [4]. Although attribute features are complicated in terms of extraction and expression, they contain rich semantic information and are more robust to illumination and viewpoint changes. Therefore, the combination of attribute features and low-level features can effectively improve the accuracy of person re-id [5]. The metric learning methods employ the machine learning algorithm to learn a good similarity metric, which makes the feature similarity of the same pedestrian greater than that of different pedestrians.

In recent years, deep learning has shown great success in a variety of tasks in image classification and frequency domain [6], where CNN is particularly outstanding. Compared with the traditional methods, CNN has stronger feature learning ability, and the learned features are more intrinsically representative to the original data, so it has better performance in extracting image features. Two types of CNN models are commonly employed in the community. The first type is the classification model as used in image classification and the second is the Siamese model using image pairs or triplets as input [7]. Most of the existing public datasets of person re-id only contain thousands of pedestrian image samples; a small number of training samples can easily lead to overfitting, which limits the performance of person re-id model. In addition, the deep neural networks for person re-id are similar in structure; that is, the feature maps extracted by convolutional layer are directly fed into the next convolutional layer [8–12]. Such structure usually ignores the correlation among features of each layer, thus reducing the mobility of feature information to some extent. In the process of back propagation, as the number of layers in neural network deepens, the gradient update information may attenuate in exponential form and cause vanishing gradient problem.

This work proposes to develop a modified deep neural network model for person re-id that could reduce overfitting caused by the lack of training samples. Moreover, this work aims to improve the identify accuracy of person re-id network with assistance of pedestrian attribute recognition. To this end, contribution of this paper is three-fold: first, this paper designs a multi-information flow convolutional neural network (MiF-CNN) to solve the person re-id problem. The network contains a series of multi-information flow convolution structures which connect the input and output of each convolutional layer together, realizes the reuse of features, and enhances the feature information flow and gradient back propagation of the entire network. Second, this paper designs a person attribute recognition network (PARN) based on long-short-term memory (LSTM) network and attention mechanism. The PARN decodes pedestrian visual features extracted by MiF-CNN into attribute features and outputs the attribute words of each person. Third, this paper presents an attribute-aided reranking algorithm which rematches attribute features among samples to aid more positive samples rank higher in rank list so as to improve the identify accuracy further.

The rest of this paper is organized as follows. Section 2 reviews the state of the art for person re-id. Section 3 introduces the details of MiF-CNN. Section 4 shows the principle of the PARN. The proposed attribute-aided reranking algorithm is detailed in Section 5. The experimental results and analysis are given in Section 6. Finally, conclusion and future works are discussed in Section 7.

2. Related Works

The early person re-id methods extract the manually designed features to represent pedestrians. Farenzena et al. divided pedestrian images into multiple areas and extracted three complementary kinds of features, weighted color histograms, maximally stable color regions, and recurrent high-structured patches. Then, match these features and measure the similarity between pedestrian pair [13]. Yang et al. proposed a novel salient color names based color descriptor (SCNCD), which was utilized to guarantee that a higher probability will be assigned to the color name near to the color. Based on SCNCD, color distributions in different color spaces were fused into feature representation for person [14]. Bazzani et al. proposed asymmetry-based HPE descriptor, which accumulated HSV histogram of multiple pedestrian images as a global appearance feature and detected patches portraying highly informative recurrent ingredient in local regions as local feature [15]. Wu et al. designed a novel gradient self-similarity (GSS) feature based on HOG to capture the patterns of pairwise similarities of local gradient patches. The combination of HOG and GSS achieved improvement in person re-id accuracy [16].

Apart from manually designed low-level features, attribute features that represent mid-level semantic information apply to person re-id as well. Compared with low-level descriptors, attributes are more robust to image translations [7]. Layne et al. labeled 15 binary attributes for the VIPeR dataset and trained SVM to detect attributes. They also learned a weighted L2-norm distance metric to fix each attribute and fused them with low-level visual features [17]. Wang et al. predicted complete attribute vector by exploiting both visual feature and marked attributes and obtained the overall ranking list by fusing the rank result from visual features and attribute vectors separately [18]. Chen et al. learned attribute of person by part-specific CNN and merged them with another identification CNN embedding in a triplet structure for person re-id task [19]. Wang et al. proposed a deep neural network that contains an auto-encoder model to learn hidden attributes of person from visual feature in an unsupervised manner, which alleviated the requirement of massive annotation [20].

Deep learning has become popular for solving person re-id problems in recent years. Ahmed et al. present a deep CNN architecture for person re-id. The architecture computed differences in feature values across the two views around a neighborhood of each feature location to add robustness to positional differences in corresponding features of the two input images [12]. Cheng et al. proposed a novel multichannel CNN. After the first layer of CNN, features were divided into four equal parts that aimed to learn features for the respective body part. The proposed CNN was trained with improved triplet loss function [21]. Lin et al. used ResNet [22] as the base network to learn low-level features and attributes jointly, and trained network with combining the person re-ID loss and attribute prediction loss [23]. Yan et al. proposed an attention block which learned par-level attention on different local regions, and integrated the proposed block into existing CNN structures for training with the identify loss [24]. Inspired by above works, this paper proposes a multi-information flow convolutional neural network to extract discriminative pedestrian features. In addition, this paper designs a person attribute recognition network based on LSTM and attention mechanism for improving person re-id results with assistance of attributes recognition.

3. Multi-Information Flow Convolutional Neural Network

The proposed MiF-CNN solves the person re-id problem with classification thought. The overall network structure is shown in Figure 1. The structure includes 2 shallow convolutional layers, 3 multi-information flow convolutional structures with novel connection pattern, fully connected layers, max pooling layers, and classification output layers. Low-level features of pedestrian images are extracted first by 2 shallow convolutional layers. After deeper multi-information flow convolutional structures, MiF-CNN extracted higher level features. The final discriminative pedestrian feature vectors are obtained after reducing dimensions by pooling layers and integrating by fully connected layers.

[figure omitted; refer to PDF]

3.1. Features Extraction

In MiF-CNN, all convolutional filters are 3 × 3 with stride 1. Batch normalization and ReLU activation function are applied after each convolutional layer. The operation process of convolutional layer can be formulated as$$\begin{array}{}\text{(1)}& \begin{array}{cc}\left\{\begin{array}{c}{z}_j^{\left(l\right)}=\sum _i{x}_i^{\left(l-1\right)}\otimes w_j^{\left(l\right)},\\ x_j^{\left(l\right)}=\sigma \left(z_j^l\right),\end{array}\right.& l>1\end{array},\end{array}$$where $x_i^{\left(l-1\right)}$ is the $i‐\text{th}$ feature map from the $\left(l-1\right)‐\text{th}$ convolutional layer, $z_j^{\left(l\right)}$ is the convolutional output of $x_i^{\left(l-1\right)}$, $x_j^{\left(l\right)}$ is the $j‐\text{th}$ feature map from the $l‐\text{th}$ convolutional layer, $w_j^{\left(l\right)}$ is the filter on the $j‐\text{th}$ feature map in the $l‐\text{th}$ convolutional layer, and $\otimes$ represents the convolutional operation. The process of convolutional layers extracting features is that neuron on the j-th feature map in the $l‐\text{th}$ convolutional layer sum each feature map after connecting and convolution by filter $w_j^{\left(l\right)}$, and map the extracted features on $j‐\text{th}$ feature map in the $l‐\text{th}$ convolutional layer. $\sigma \left(\cdot \right)$ is the ReLU activation function, which is formulated as $\sigma \left(x\right)=\max \left(0,x\right)$. Because of batch normalization, bias is ignored.

3.2. Multi-Information Flow Convolutional Structure

In this structure, both output and input of the current convolutional layer are concatenated together and fed into the next convolutional layer; i.e., the input of each layer is the connection combination of outputs from all previous layers. The detail of multi-information flow convolutional structures is shown in Figure 2.

[figure omitted; refer to PDF]

This connection pattern makes feature maps of each layers be reused by all subsequent layers in forward propagation process, which makes the whole CNN model learn more feature information of pedestrian images. It can be considered as a special “Data Augmentation” in feature maps so as to enhance the information mobility of the network. In back propagation process, gradient of input in each layer contains derivative of loss function with respect to input, which makes propagation of gradient more effective and network easier to be trained. In multi-information flow convolutional structure, the number of feature maps that each layer outputs is a constant value $\rho$, so the number of feature maps that $l‐\text{th}$ layers outputs is ${\rho }_0+\rho \left(l-1\right)$, where ${\rho }_0$ is the number of feature maps in the initial layer. Supposing the feature map of $l‐\text{th}$ channel in the initial layer is $x_i^{\left(0\right)}$, where $i\in \left(1,{\rho }_0\right)$, then the feature map of $j‐\text{th}$ channel that initial layer outputs can be expressed as$$\begin{array}{}\text{(2)}& z_j^{\left(1\right)}=\sum _i{x}_i^{\left(0\right)}\otimes w_j^{\left(1\right)},\end{array}$$where $w_j^{\left(1\right)}$ is the weight of the initial layer. The output after activation function is$$\begin{array}{}\text{(3)}& a_j^{\left(1\right)}=\sigma \left(z_j^{\left(1\right)}\right),\end{array}$$where $j\in \left(1,\rho \right)$. The feature map that the $l‐\text{th}$ layer outputs can be expressed as$$\begin{array}{}\text{(4)}& \begin{array}{cc}\left\{\begin{array}{c}{z}_q^{\left(l\right)}=\sum _p{x}_p^{\left(l-1\right)}\otimes w_q^{\left(l\right)},\\ a_q^{\left(l\right)}=\sigma \left(z_q^{\left(l\right)}\right),\end{array}\right.& l>0,\end{array}\end{array}$$where $p\in \left(1,\hspace{0.17em}\hspace{0.17em}{\rho }_0+\rho \left(l-1\right)\right)$, $q\in \left(1,\rho \right)$. The output of the $l‐\text{th}$ layer after concatenate operation is$$\begin{array}{}\text{(5)}& x_r^{\left(l\right)}=\left[x_p^{\left(l-1\right)},\hspace{0.17em}\hspace{0.17em}{a}_q^{\left(l\right)}\right],\hspace{1em}l>0,\end{array}$$where $r\in \left(1,\hspace{0.17em}\hspace{0.17em}{\rho }_0+\rho \cdot l\right)$, $\left[\cdot ,\cdot \right]$ represents the concatenate operation on channel dimension.

In the process of back propagation, supposing $\Delta x_r^{\left(l\right)}$ is the derivative of loss function with respect to $x_r^{\left(l\right)}$. Due to $x_r^{\left(l\right)}$ containing $x_p^{\left(l-1\right)}$ and $a_q^{\left(l\right)}$, it produces two parts of gradient as shown below:$$\begin{array}{}\text{(6)}& \left\{\begin{array}{c}\Delta a_q^{\left(l\right)}=\Delta x_r^{\left(l\right)}\cdot \frac{\partial x_r^{\left(l\right)}}{\partial a_q^{\left(l\right)}},\\ \Delta x_p^{\left(l-1\right)}=\Delta x_r^{\left(l\right)}\cdot \frac{\partial x_r^{\left(l\right)}}{\partial x_p^{\left(l-1\right)}},\end{array}\right.\end{array}$$where $\Delta a_q^{\left(l\right)}$ is the gradient of output from the $l‐\text{th}$ layer after activation function. $\Delta x_p^{\left(l-1\right)}$ is the gradient of output from the $\left(l-1\right)‐\text{th}$ layer. The gradient of weight in the $l‐\text{th}$ layer is$$\begin{array}{}\text{(7)}& \left\{\begin{array}{c}\Delta z_q^{\left(l\right)}=\Delta a_q^{\left(l\right)}\cdot \frac{\partial a_q^{\left(l\right)}}{\partial z_q^{\left(l\right)}}=\Delta x_r^{\left(l\right)}\cdot \frac{\partial x_r^{\left(l\right)}}{\partial a_q^{\left(l\right)}}\cdot {\sigma }^{\prime }\left(z_q^{\left(l\right)}\right),\\ \Delta w_q^{\left(l\right)}=\Delta z_q^{\left(l\right)}\cdot \frac{\partial z_q^{\left(l\right)}}{\partial w_q^{\left(l\right)}}=\Delta z_q^{\left(l\right)}\cdot x_p^{\left(l-1\right)},\end{array}\right.\end{array}$$where $\Delta z_q^{\left(l\right)}$ is the gradient of the convolution result in the $l‐\text{th}$ layer, ${\sigma }^{\prime }\left(z_q^{\left(l\right)}\right)$ is the derivative of the activation function with respect to $z_q^{\left(l\right)}$, and $\Delta w_q^{\left(l\right)}$ is the gradient of weights in the $l‐\text{th}$ layer. The network utilizes $\Delta w_q^{\left(l\right)}$ to update weights of each layer, which is formulated as$$\begin{array}{}\text{(8)}& {\left(w_q^{\left(l\right)}\right)}_{\text{new}}=w_q^{\left(l\right)}-\eta \cdot \Delta w_q^{\left(l\right)},\end{array}$$where $\eta$ is the learning rate. Gradient keeps back propagation to the $\left(l-1\right)‐\text{th}$ layer. The gradient of $x_p^{\left(l-1\right)}$ is$$\begin{array}{}\text{(9)}& \Delta x_p^{\left(l-1\right)}=\Delta z_q^{\left(l\right)}\cdot \frac{\partial z_q^{\left(l\right)}}{\partial x_p^{\left(l-1\right)}}=\Delta z_q^{\left(l\right)}\cdot w_q^{\left(l\right)}.\end{array}$$

As shown in equations (6) and (9), the loss function produces two flows of gradient with respect to outputs of each convolutional layer, which makes error information propagating more effective in network and restrains the vanishing gradient to a certain extent.

The proposed MiF-CNN includes three multi-information flow convolutional structures. Between every two multi-information flow convolutional structures, a middle pooling layer is applied to compares the redundancy features. Hyperparameter $\rho$ is set with a small value. When it comes to a new multi-information flow convolutional structure, $\rho$ is doubled. Such a design makes each convolution layer learn a small quantity of features and reduce the redundant features so as to optimize the efficiency of network. With deeper layers, the network can learn more high-level and complex pedestrian features and improve the final identification accuracy.

3.3. Loss Function

Current deep learning algorithms usually use cross-entropy loss as cost function, which is formulated as$$\begin{array}{}\text{(10)}& L_{\text{S}}=-\frac{1}{M}\sum _{i=1}^M{y}^{\left(i\right)}\log \frac{e^{{\theta }_{y\left(i\right)}^T{x}^{\left(i\right)}}}{{\sum }_{j=1}^k{e}^{{\theta }_j^T{x}^{\left(i\right)}}},\end{array}$$where $y^{\left(i\right)}$ is the ground truth of pedestrian categories in training set, $\theta$ is the parameter of the last fully connected layer, $x^{\left(i\right)}$ is the feature vector of training samples, k is the number of pedestrian categories in the training set, and M is the batch size.

However, in practice, when using cross-entropy loss merely, if the quality of extracted features is not good enough, it will lead to intraclass distance being greater than interclass distance. Aiming at this problem, Wen et al. proposed center loss in 2016 [25]. Combination of cross-entropy loss and center loss can enhance the discrimination and generalization ability of the network. Center loss is defined as follows:$$\begin{array}{}\text{(11)}& L_{\text{C}}=\frac{1}{2M}\sum _{i=1}^M{‖x_i-c_j‖}_2^2,\end{array}$$where $c_j$ is the center of the $j‐\text{th}$ pedestrian feature and $x_i$ is the feature vector of pedestrian. Center loss minimizes the distance between feature and its center in order to reduce the intraclass distance. Center $c_j$ is updated with equation (12):$$\begin{array}{}\text{(12)}& \left\{\begin{array}{c}\Delta c_j^t=\frac{{\sum }_{i=1}^M\delta \left(y_i=j\right)\cdot \left(c_j^t-x_i\right)}{1+{\sum }_{i=1}^M\delta \left(y_i=j\right)},\\ c_j^{t+1}=c_j^t-\beta \cdot \Delta c_j^t,\end{array}\right.\end{array}$$where $\beta$ is the update rate, $\delta \left(y_i=j\right)$ is 1 if prediction equals to ground truth, otherwise is 0. That is to say, center is updated only when network predicts correctly.

4. Person Attributes Recognition Network

The rank list of MiF-CNN is shown in Figure 3. In the incorrect identification results of rank 1 (pedestrian B and pedestrian C), there is a big difference in attribute features between the top-ranking negative samples and the query image, including gender, clothing, whether carrying handbag or not, and so on. Hence, this paper recognizes person attributes for improving accuracy of person re-id.

[figure omitted; refer to PDF]

Based on Encoder-Decoder idea, Xu et al. [26] proposed a neural network model that can learn and generate the content of images. The model utilized CNN as encoder to extract features of images which were then fed into a recurrent neural network (RNN) for decoding into language captions of images. Inspired by that, this paper presents a person attribute recognition network (PARN) with LSTM and attention mechanism. The proposed PARN takes pedestrian features that are extracted by MiF-CNN as input and outputs the attributes information of pedestrian images. The architecture of PARN is demonstrated in Figure 4.

[figure omitted; refer to PDF]

4.1. Input of PARN

In PARN, the input is the feature maps that are before the last fully connected layer in the MiF-CNN. The input feature is split into n feature vectors, each of which corresponds to a part of the image. Each feature vector is a D_X-dimensional vector which is represented as$$\begin{array}{}\text{(13)}& X=\left\{x_1,x_2,\mathrm{...},x_n\right\},\hspace{1em}{x}_i\in {ℝ}^{D_X}.\end{array}$$

Referring to the natural language processing method, PARN transforms words in person attribute labels into word embedding. As a part of the input of PARN, each word embedding is a D_Y-dimensional vector:$$\begin{array}{}\text{(14)}& y=\left\{y_1,y_2,\mathrm{...},y_m\right\},\hspace{1em}{y}_i\in {ℝ}^{D_Y},\end{array}$$where m is the number of attribute words in each pedestrian image and $y_i$ is the word embedding corresponding to each attribute word.

For attribute words, common one-hot encoding considers each word as an individual, which ignores the correlation among words. However, word embedding represents each word as a continuous dense vector, which makes those correlative words closer in space.

4.2. Attention Mechanism

Attention mechanism has been widely used in natural language processing and computer vision. By measuring the correlation between the output and different parts of the input, attention mechanism gives different weights to different parts of the input, enabling the network to use more important feature information for prediction and reduce the dimension of input data [27].

In practice, a certain attribute of pedestrian is only corresponding to a certain part of the image. For example, when recognizing whether a pedestrian is wearing a hat, people only pay attention to the area above the head of the pedestrian usually, instead of other areas irrelevant to the attribute. Therefore, before feeding the image features into the LSTM network, attention mechanism is introduced to calculate the correlation between different positions of image features and the hidden state of LSTM at the previous time. The schematic diagram of attention mechanism is shown in Figure 5.

[figure omitted; refer to PDF]

At time t, the fully connected layer and tanh function are used to integrate the information of the input feature vector and the hidden state of LSTM at the previous time, which is formulated as$$\begin{array}{}\text{(15)}& v_i=\tanh \left(W_{\text{att}X}{x}_i+W_{\text{att}h}{h}_{t-1}\right),\end{array}$$where $W_{\text{att}X}$ and $W_{\text{att}h}$ represent the fully connected layer weights of $x_i$ and $h_{t-1}$, respectively. The attention weights ${\alpha }_i$ is obtained by supplying softmax function on score vector $v_i$:$$\begin{array}{}\text{(16)}& {\alpha }_i=\text{softmax}\left(W_{\text{att}v}{v}_i\right),\end{array}$$where $W_{\text{att}v}$ represents the fully connected layer weights of $v_i$. The output $Z$ is the weighted sums of all ${\alpha }_i$:$$\begin{array}{}\text{(17)}& Z=\sum _{i=1}^n{\alpha }_i{x}_i.\end{array}$$

Feature $Z$ highlights the local features that are helpful to predict and suppress the other local features that make small contribution to prediction, which reduces the dimension of features to some extent and makes LSTM network focus on the part of input features that have greater correlation with prediction while recognizing person attributes so that improves the efficiency and prediction accuracy of the network.

4.3. LSTM

Normal RNN updates network parameters with back propagation, which easily suffers from the vanishing gradient when there is a long gap between relevant information and the current position to predict. LSTM network solves this problem well because of its own structure advantage. The architecture of LSTM unit is shown in Figure 6.

[figure omitted; refer to PDF]

Here, c is the cell state for storing and transferring information, f is the forget gate which decides what should be abandoned in the cell state, i is the input gate which decides what should be stored in the cell state, $g$ represents a vector of new candidate values that should be added to the cell state, o is the output gate which decides what parts of the cell state should be output to the next time, h is the hidden state of LSTM, x is the input of LSTM, and t denotes the current time.

In PARN, at any time t, the input of LSTM consists of two parts: word embedding $y_{t-1}$ at the previous time and image features $Z_t$ at the current time. The operation processing of LSTM in PARN is formulated as$$\begin{array}{}\text{(18)}& \left\{\begin{array}{c}f=\text{sigmoid}\left(W_f\left[h_{t-1},y_{t-1},Z_t\right]+b_f\right),\hfill \\ i=\text{sigmoid}\left(W_i\left[h_{t-1},y_{t-1},Z_t\right]+b_i\right),\hfill \\ g=\text{sigmoid}\left(W_g\left[h_{t-1},y_{t-1},Z_t\right]+b_g\right),\hfill \\ o=\text{sigmoid}\left(W_o\left[h_{t-1},y_{t-1},Z_t\right]+b_o\right),\hfill \\ c_t=f\odot c_{t-1}+i\odot g,\hfill \\ h_t=o\odot \tanh \left(c_t\right),\hfill \end{array}\right.\end{array}$$where W and b represent the weights and biases of each gate, respectively. Sigmoid function outputs a number between 0 and 1 to decide the quantity scale of values that go through these gates. Tanh function is the activation function of input. This design makes LSTM give different weights for information at different times so that it can choose what part of information to remember or forget in a long-term sequence.

The time step of LSTM in PARN is set as the number of attributes of each pedestrian image m. The output hidden state $h_t$ is calculated by a fully connected layer and softmax function at each time step to predict pedestrian attribute words.

It is worth noting that, at each time step, the input word embedding of LSTM is the one that LSTM learned at the previous time step, which takes advantage of the LSTM when solving the long-term sequence problem. Because of the correlation among attributes of pedestrian, like the pedestrian owning attribute “male” generally own attribute “short hair” and attribute “pants,” LSTM can utilize the previous attribute result to predict the current attribute more accurately.

4.4. Network Optimization

The loss function of PARN includes two parts, one is the cross-entropy between prediction and ground truth of network, which is expressed as$$\begin{array}{}\text{(19)}& L_{\text{sa}}=\sum _{i=1}^m{u}^{\prime }\log \left(\frac{e^{{\theta }_{ai}^{T}u}}{{\sum }_{j=1}^{n_w}{e}^{{\theta }_{aj}^{T}u}}\right),\end{array}$$where $u$ is the prediction value of the network, $u^{\prime }$ is the ground truth of pedestrian labels, m is the time step of LSTM, and $n_w$ is the total number of attribute words in each dataset. As shown in equation (19), the loss of a single image in single epoch is the cumulated loss value after m iterations. The other one is the loss function of attention mechanism as described in [40]:$$\begin{array}{}\text{(20)}& L_{\alpha }=\frac{1}{2n}{\sum _{i=1}^n\left(1-\sum _{i=1}^m{\alpha }_i^j\right)}^2,\end{array}$$where ${\alpha }_i^j$ represents the attention weights of the image feature $x_i$ and n is the number of parts in the image feature. The object function of PARN to optimize is$$\begin{array}{}\text{(21)}& J=-\frac{1}{M}\sum _{i=1}^M\left(L_{sa}^i+\lambda \cdot L_{\alpha }^i\right),\end{array}$$where M is the batch size and $\lambda$ is the rate of $L_{\alpha }$ in the object function.

5. Attribute-Aided Reranking Algorithm

Given a probe image $q_0$ and gallery image set $G=\left\{g_1,g_2,\mathrm{...},g_N\right\}$ including N pedestrian images, the initial similarity distance between $q_0$ and $g_i$ is computed with the Euclidean distance as$$\begin{array}{}\text{(22)}& d\left(q_0,g_i\right)=\sqrt{\sum _{i=1}^N{\left(x_{q_0}-x_{g_i}\right)}^2},\end{array}$$where $x_{q_0}$ and $x_{g_i}$ represent the features of $q_0$ and $g_i$, respectively, that extracted by MiF-CNN. The initial rank list $R_0=\left\{g_1^{0\prime },g_2^{0\prime },\mathrm{...},g_N^{0\prime }\right\}$ is obtained by sorting similarity distance $d\left(q_0,g_i\right)$ in ascending order, where $d\left(q_0,g_i^{0\prime }\right)<d\left(q_0,g_{i+1}^{0\prime }\right)$. If the top-1 gallery $g_1^{0\prime }$ is the positive sample, it means rank 1 is correct. If the positive sample is not the top-1 gallery but the top-5 gallery, it means rank 5 is correct.

The attribute-aided reranking algorithm reranks the rank list $R_0$ according to the attribute feature similarity between $q_0$ and $g_i$ so that more positive gallery gets higher ranking in $R_0$, which could improve the performance of person re-id. To be specific, when rank 5 is correct but rank 1 is not, the proposed algorithm distributes the initial feature score $s_{\text{f}}$ for top-5 gallery $g_1^{0\prime }\sim g_5^{0\prime }$ according to their ranking in $R_0$, the higher the ranking, the higher the $s_{\text{f}}$. Then, the proposed algorithm distributes attribute score $s_{\text{a}}$ for $g_1^{0\prime }\sim g_5^{0\prime }$ according to the number of their attributes that are the same as $q_0$, the more same attributes, the higher $s_{\text{a}}$. The total score of each gallery is $s=s_{\text{f}}\left(1-\gamma \right)+s_{\text{a}}\gamma$, where $\gamma$ is the score weight. The reranking rank list $R_0^{\ast }$ is obtained by reranking $g_1^{0\prime }\sim g_5^{0\prime }$ in descending order of their total score $s$. For M query images, the whole processing of the attribute-aided reranking algorithm is shown as Algorithm 1.

Algorithm 1: The attribute-aided reranking algorithm.

 

Input: Initial rank list R₀

 

Output: The reranking rank list $R^{\ast }$

(1)

for i = 1, 2, …, M do

(2)

if rank 1 correct

(3)

$R^{\ast }=R_0$

(4)

end if

(5)

elif rank 5 correct

(6)

Distributing initial feature score $s_{\text{f}}$ for top-5 gallery $g_1^{i\prime }\sim g_5^{i\prime }$ according to their ranking in $R_0$

(7)

Distributing attribute score $s_{\text{a}}$ for $g_1^{i\prime }\sim g_5^{i\prime }$ according to the number of their attributes that are the same as $q_i$

(8)

$s=s_{\text{f}}\left(1-\gamma \right)+s_{\text{a}}\gamma$

(9)

Reranking $g_1^{i\prime }\sim g_5^{i\prime }$ in descending order of their total score $s$ and obtain $R^{\ast }$

(10)

end if

(11)

elif rank 10 correct

(12)

Changing $g_1^{i\prime }\sim g_5^{i\prime }$ to $g_1^{i\prime }\sim g_{10}^{i\prime }$, repeat 6–10

(13)

end if

(14)

else

(15)

Changing $g_1^{i\prime }\sim g_5^{i\prime }$ to $g_1^{i\prime }\sim g_M^{i\prime }$, repeat 6–10

(16)

end if

(17)

end for

6. Experiments

This paper evaluates the attribute recognition accuracy of PARN on person re-id public datasets first and then compares our results with other person attribute recognition methods. Then, this paper evaluates the performance of MiF-CNN on three people re-id datasets and the availability of the proposed attribute-aided reranking algorithm for improving the accuracy of person re-id. The analysis of experiment results is also given after comparing our results with the state-of-the-art person re-id methods.

6.1. Datasets and Evaluation Protocol

This paper evaluates the proposed methods on three challenging person re-id public datasets including VIPeR [28], CUHK01 [29], and Market1501 [30].

 

VIPeR Dataset. It contains 1264 images of 632 persons. Because of the lack of image samples, low-resolution, a great change in pose, view, and illumination, it is one of the hardest difficult person re-id datasets. In experiment, the VIPeR dataset is randomly split into two parts: images of 316 persons for training and the images of remaining 316 persons for testing.

 

CUHK01 Dataset. It consists of 3884 images of 971 persons which are captured by two cameras with different views. 485 pedestrians are randomly chosen for training and other 486 pedestrians for testing.

 

Market1501 Dataset. It is one of the largest person re-id datasets that include 32268 images of 1501 persons. Each person has a number of images that are captured by six cameras with different views. The dataset is divided into two parts: 12936 images of 751 persons for training and 19732 images of 750 persons for testing.

 

Evaluation Protocol. For the single-shot datasets VIPeR and CUHK01, cumulative match characteristic (CMC) is used to record the ranks of correct identify [31]. For the multishot dataset Market1501, apart from CMC, mean Average Precision (mAP) is utilized to evaluate the performance of the proposed methods.

6.2. Experimental Results on Attribute Recognition

The attributes in PARN refer to pedestrian identity-level attributes. For the VIPeR dataset, attributes include “gender,” “length of hair,” “lower clothing,” “upper clothing,” “backpack or not,” and “carrying anything or not.” For the CUHK01 dataset, attributes include “gender,” “length of hair,” “backpack or not,” “handbag or not,” “color of upper,” and “color of lower.” For the Market1501 dataset, attributes contain “gender,” “length of hair,” “wearing hat or not,” “lower clothing,” “backpack or not,” “handbag or not,” “length of sleeve,” and “length of lower clothing.” Each person in each dataset owns an attribute word list as shown in Figure 7.

[figures omitted; refer to PDF]

This paper evaluates the attribute recognition accuracy of PARN on the Market1501 dataset and compares our results with outstanding person attribute recognition method APR [22]. The experiment results are reported in Table 1, where “L.slv” represents the “length of sleeve” and “L.low” represents the “length of lower clothing.”

Table 1

Recognition accuracy of different attributes in PARN and APR (%).

Method	Gender	Hair	Clothes	Hat	Backpack	Handbag	L.slv	L.low	Mean
APR	85.78	83.46	91.36	88.21	86.32	76.01	94.12	92.64	87.23
PARN	84.93	79.10	89.35	96.67	83.86	85.18	92.84	88.45	87.55

It can be observed from Table 1 that PARN obtains superior recognition accuracy of 8 attributes, especially “length of hair,” “handbag or not,” and mean accuracy are higher than APR, whereas accuracy of other attributes is closer with APR. Comparison with APR demonstrates the outstanding attribute recognition performance of PARN which plays an important role in improving the accuracy of person re-id.

6.3. Experimental Results on Person re-id

This paper evaluates the performance of the proposed methods on three datasets. The experimental results of the proposed methods and other state-of-the-art methods are presented in Table 2.

Table 2

Comparison of various methods with the proposed methods on three datasets (%).

Methods	VIPeR			CUHK01			Market1501
	Rank 1	Rank 5	Rank 10	Rank 1	Rank 5	Rank 10	Rank 1	Rank 5	Rank 10	mAP
DNS [32]	42.28	71.46	82.94	64.98	84.96	89.92	67.96	—	—	41.89
DLDA [33]	44.11	72.59	81.66	67.12	89.45	91.68	48.15	—	—	29.94
FT-JSTL + DGD [11]	38.6	—	—	66.60	—	—	—	—	—	—
PDC [34]	51.27	74.05	84.18	—	—	—	84.14	92.73	94.92	63.41
K-means-CNN [35]	46.50	69.30	80.70	53.50	82.50	91.20	—	—	—	—
Spindle [9]	53.80	74.10	83.20	—	—	—	76.90	91.50	94.60	—
PersonNet [36]	—	—	—	71.14	90.07	95.00	37.21	—	—	18.57
CSBT [37]	36.60	66.20	88.30	51.20	76.30	91.80	42.90	—	—	20.30
SDH-CNN [38]	—	—	—	—	—	—	58.12	68.50	80.82	48.20
M3TCP [21]	—	—	—	53.70	84.30	91.00	—	—	—	—
DM3 [39]	42.70	74.30	85.10	49.70	77.30	86.10	75.80	89.10	92.40	53.20
MiF	40.82	68.35	81.01	66.87	85.39	89.51	78.92	86.46	89.28	66.00
MiF + PARN	46.84	74.68	83.23	71.81	90.33	91.77	79.39	88.95	91.36	66.46

“MiF” represents the MiF-CNN method, and “MiF + PARN” represents the MiF-CNN with the attribute-aided reranking method.

As shown in Table 2, the proposed MiF-CNN model obtains a great performance among state-of-the-art methods, and on the contrary, comparison between MiF and MiF + PARN demonstrates that the proposed attribute-aided reranking algorithm is helpful to increase the person re-id accuracy. The improvement effect is especially obvious on VIPeR and CUHK01 datasets, where the identification accuracy of rank 1, rank 5, and rank 10 improves by 6.02%, 6.33%, and 2.22% and 4.94%, 4.94%, and 2.26%, respectively. The proposed MiF-CNN model with the attribute-aided reranking algorithm gets the best rank 5 accuracy on the VIPeR dataset, the best rank 1 and rank 5 accuracy on the CUHK01 dataset, and the best mAP on the Market1501 dataset among various methods.

6.4. Analysis of Experimental Results

Because of the small quantity of training samples in VIPeR and CUHK01 datasets, a deep CNN model is hard to be trained and easily suffers from overfitting. To handle this, the FT-JSTL + DGD method based on deep CNN learned deep features from multiple domains jointly by merging all the datasets together and fine-tuned the pretrained model on VIPeR and CUHK01 separately. The structure and hyperparameters of the deep CNN model in the M3TCP method need to be adjusted manually for adapting the scale of different datasets so as to overcome the training problem of the deep neural network.

In contrast, the proposed MiF-CNN has an immobile structure and can be trained on smaller datasets directly without any fine tuning. In spite of the deep structure, the model can converge quickly and well. MiF-CNN without the attribute-aided reranking algorithm outperforms FT-JSTL + DGD and M3TCP by 0.27% and 13.17%, respectively, at rank 1 accuracy on the CUHK01 dataset. It also outperforms FT-JSTL + DGD by 2.22% at rank 1 accuracy on the VIPeR dataset, which indicates that the proposed MiF-CNN model has an excellent ability of reducing overfitting and generalization and an outstanding performance in person re-id.

Figure 8 demonstrates the rank list of MiF and MiF + PARN. It can be seen that the MiF-CNN with the attribute-aided reranking method enables the lower ranked positive sample in rank list of MiF-CNN obtain a higher ranking so as to improve the accuracy of person re-id, which verifies the effectiveness of the attribute-aided reranking algorithm for improving performance of person re-id model.

[figure omitted; refer to PDF]

7. Conclusion

This paper studies and discusses the training problem of deep neural network in person re-id task, and the using of pedestrian attributes for further improving the accuracy of person re-id. The proposed MiF-CNN model realizes the reuse of feature maps and gradient information, which enhances the feature mobility of network and improves the efficiency of gradient propagation. The designed person attribute recognition network uses an attention mechanism to measure the correlation between input feature maps and the hidden state of LSTM at previous time for reducing the dimension of features. It also employs LSTM to decode image features into pedestrian attributes. On the basis of the attribute recognition results, the attribute-aided reranking algorithm is presented, which rematches attribute features among samples to aid more positive samples rank higher in rank list so as to improve the identify accuracy further.

The experimental results on three public person re-id datasets indicate the outstanding performance of MiF-CNN model in person re-id. The attribute-aided reranking algorithm makes a major contribution to improve the accuracy of person re-id. In the future, more improvement and optimization will be done on pedestrian attribute recognition network. Moreover, the caption of person images or videos can be obtained by the LSTM model with natural language process ideas which make person re-id methods more significant.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (no. 61773105), the Natural Science Foundation of Liaoning Province (no. 20170540675), and the Scientific Research Project of Liaoning Educational Department (no. LQGD2017023).

References

[1] L. Li, J. Guo, "Pedestrian re-identification based on reinforced deep feature fusion," Information Technology, vol. 42 no. 7, pp. 15-19, 2018. in Chinese

[2] J. Dai, Y. Zhang, H. Lu, H. Wang, "Cross-view semantic projection learning for person re-identification," Pattern Recognition, vol. 75, pp. 63-76, DOI: 10.1016/j.patcog.2017.04.022, 2018.

[3] T. T. T. Pham, T.-L. Le, H. Vu, "Fully-automated person re-identification in multi-camera surveillance system with a robust kernel descriptor and effective shadow removal method," Image and Vision Computing, vol. 59, pp. 44-62, DOI: 10.1016/j.imavis.2016.10.010, 2017.

[4] P. DaoDao, H. J. Lee, "Deep multi-task network for learning person identity and attributes," IEEE Access, vol. 6, pp. 60801-60811, 2018.

[5] J. Li, L. Zhuo, J. Zhang, F. Li, H. Zhang, "A survey of person re-identification," Acta Automatica Sinica, vol. 44 no. 9, pp. 1554-1568, 2018. in Chinese

[6] L. Wu, Y. Wang, J. Gao, "Deep adaptive feature embedding with local sample distributions for person re-identification," Pattern Recognition, vol. 73, pp. 275-288, DOI: 10.1016/j.patcog.2017.08.029, 2018.

[7] L. Li, Y. Yang, A. G. Hauptmann, "Person re-identification: past, present and future," 2016. http://arxiv.org/abs/1610.02984

[8] D. Wu, S. J. Zheng, C. A. Yuan, D. S. Huang, "A deep model with combined losses for person re-identification," Cognitive Systems Research, vol. 54, pp. 74-82, DOI: 10.1016/j.cogsys.2018.04.003, 2018.

[9] H. Zhao, M. Tian, S. Sun, "Spindle net: person re-identification with human body region guided feature decomposition and fusion," Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1077-1085, DOI: 10.1109/cvpr.2017.103, .

[10] W. Chen, X. Chen, J. Zhang, K. Huang, "Beyond triplet loss: a deep quadruplet network for person re-identification," vol. 2 no. 8,DOI: 10.1109/cvpr.2017.145, .

[11] T. Xiao, H. Li, W. Ouyang, "Learning deep feature representations with domain guided dropout for person re-identification," Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1249-1258, DOI: 10.1109/cvpr.2016.140, .

[12] E. Ahmed, M. Jones, T. K. Marks, "An improved deep learning architecture for person re-identification," Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3908-3916, DOI: 10.1109/cvpr.2015.7299016, .

[13] M. Farenzena, L. Bazzani, A. Perina, "Person re-identification by symmetry-driven accumulation of local features," Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2360-2367, .

[14] Y. Yang, J. Yang, J. Yan, "Salient color names for person re-identification," Proceedings of Computer Vision-ECCV 2014, pp. 536-551, DOI: 10.1007/978-3-319-10590-1_35, .

[15] L. Liao, M. Cristani, A. Perina, "Multiple-shot person re-identification by chromatic and epitomic analyses," Pattern Recognition Letters, vol. 33 no. 7, pp. 898-903, DOI: 10.1016/j.patrec.2011.11.016, 2012.

[16] S. Murino, R. Laganière, P. Payeur, "Improving pedestrian detection with selective gradient self-similarity feature," Pattern Recognition, vol. 48 no. 8, pp. 2364-2376, DOI: 10.1016/j.patcog.2015.01.005, 2015.

[17] R. Layne, T. M. Hospedales, S. Gong, "Person re-identification by attributes," vol. 2 no. 3, .

[18] Z. Wang, R. Hu, Y. Yu, C. Liang, W. Huang, "Multi-level fusion for person Re-identification with incomplete marks," Proceedings of the 23rd ACM international conference on Multimedia, pp. 1267-1270, .

[19] Y. Chen, S. Duffner, A. Stoian, "Deep and low-level feature based attribute learning for person re-identification," Image and Vision Computing, vol. 79, pp. 25-34, DOI: 10.1016/j.imavis.2018.09.001, 2018.

[20] Z. Dufour, X. Bai, M. Ye, S. Satoh, "Incremental deep hidden attribute learning," Proceedings of the 26th ACM international conference on Multimedia, pp. 72-80, DOI: 10.1145/3240508.3240510, .

[21] D. Cheng, Y. Gong, S. Zhou, "Person re-identification by multi-channel parts-based cnn with improved triplet loss function," Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1335-1344, DOI: 10.1109/cvpr.2016.149, .

[22] K. He, X. Zhang, S. Ren, "Deep residual learning for image recognition," Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, DOI: 10.1109/cvpr.2016.90, .

[23] Y. Lin, L. Zheng, Z. Zheng, "Improving person re-identification by attribute and identity learning," 2017. http://arxiv.org/abs/1703.07220

[24] Y. Yan, B. Ni, J. Liu, X. Yang, "Multi-level attention model for person re-identification," Pattern Recognition Letters, 2018. In press

[25] Y. Wen, K. Zhang, Z. Li, "A discriminative feature learning approach for deep face recognition," Proceedings of 14th European Conference, pp. 499-515, DOI: 10.1007/978-3-319-46478-7_31, .

[26] K. Xu, J. Ba, R. Kiros, "Show, attend and tell: neural image caption generation with visual attention," Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 2048-2057, .

[27] H. Choi, K. Cho, Y. Bengio, "Fine-grained attention mechanism for neural machine translation," Neurocomputing, vol. 284, pp. 171-176, DOI: 10.1016/j.neucom.2018.01.007, 2018.

[28] D. Gray, H. Tao, "Viewpoint invariant pedestrian recognition with an ensemble of localized features," Proceedings of European Conference on Computer Vision Lecture Notes in Computer Science, pp. 262-275, DOI: 10.1007/978-3-540-88682-2_21, .

[29] W. Li, R. Zhao, X. Wang, "Human reidentification with transferred metric learning," Proceedings of the 2008 Asian Conference on Computer Vision (ACCV), pp. 31-44, DOI: 10.1007/978-3-642-37331-2_3, .

[30] L. Zheng, L. Shen, L. Tian, "Scalable person re-identification: a benchmark," Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1116-1124, DOI: 10.1109/iccv.2015.133, .

[31] W. Li, R. Zhao, T. Xiao, "DeepReID: deep filter pairing neural network for person re-identification," Proceedings of the 2014 Computer Vision and Pattern Recognition (CVPR), pp. 152-159, DOI: 10.1109/cvpr.2014.27, .

[32] L. Zhang, T. Xiang, S. Gong, "Learning a discriminative null space for person re-identification," Proceedings of the IEEE Conference on Computer Cision and Pattern Recognition (CVPR), pp. 1239-1248, DOI: 10.1109/cvpr.2016.139, .

[33] L. Wu, C. Shen, A. van den Hengel, "Deep linear discriminant analysis on Fisher networks: a hybrid architecture for person re-identification," Pattern Recognition, vol. 65, pp. 238-250, DOI: 10.1016/j.patcog.2016.12.022, 2017.

[34] C. Su, J. Li, S. Zhang, "Pose-driven deep convolutional model for person re-identification," Proceedings of the 2016 International Conference on Computer Vision (ICCV), pp. 3980-3989, DOI: 10.1109/iccv.2017.427, .

[35] Z. Zhao, B. Zhao, F. Su, "Person re-identification via integrating patch-based metric learning and local salience learning," Pattern Recognition, vol. 75, pp. 90-98, DOI: 10.1016/j.patcog.2017.03.023, 2018.

[36] L. Wu, C. Shen, A. Hengel, "Personnet: person re-identification with deep convolutional neural networks," 2016. http://arxiv.org/abs/1601.07255

[37] J. Chen, Y. Wang, J. Qin, "Fast person re-identification via cross-camera semantic binary transformation," Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5330-5339, DOI: 10.1109/cvpr.2017.566, .

[38] L. Wu, Y. Wang, Z. Ge, "Structured deep hashing with convolutional neural networks for fast person re-identification," Computer Vision and Image Understanding, vol. 167 no. 10, pp. 63-73, DOI: 10.1016/j.cviu.2017.11.009, 2018.

[39] Z. Hu, R. Hu, C. Chen, Y. Yu, "Person reidentification via discrepancy matrix and matrix metric," IEEE Transactions on Cybernetics, vol. 48 no. 10, pp. 3006-3020, DOI: 10.1109/tcyb.2017.2755044, 2018.

[40] D. Jiang, K. Cho, Y. Bengio, "Neural machine translation by jointly learning to align and translate," , 2014. http://arxiv.org/abs/1409.0473