An fNIRS-based investigation of visual merchandising displays for fashion stores

Abstract

This paper investigates a brain-based approach for visual merchandising display (VMD) in fashion stores. In marketing, VMD has become a research topic of interest. However, VMD research using brain activation information is rare. We examine the hemodynamic responses (HRs) in the prefrontal cortex (PFC) using functional near-infrared spectroscopy (fNIRS) while positive/negative displays of four stores (menswear, womenswear, underwear, and sportswear) are shown to 20 subjects. As features for classifying the HRs, the mean, variance, peak, skewness, kurtosis, t-value, and slope of the signals for a 20-sec time window for the activated channels are analyzed. Linear discriminant analysis is used for classifying two-class (positive and negative displays) and four-class (four fashion stores) models. PFC brain activation maps based on t-values depicting the data from the 16 channels are provided. In the two-class classification, the underwear store had the highest average classification result of 67.04%, whereas the menswear store had the lowest value of 64.15%. Men’s classification accuracy for the underwear stores with positive and negative displays was 71.38%, whereas the highest classification accuracy obtained by women for womenswear stores was 73%. The average accuracy over the 20 subjects for positive displays was 50.68%, while that of negative displays was 51.07%. Therefore, these findings suggest that human brain activation is involved in the evaluation of the fashion store displays. It is concluded that fNIRS can be used as a brain-based tool in the evaluation of fashion stores in a daily life environment.

About the Authors:

Xiaolong Liu

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Project administration, Resources, Writing – original draft

Affiliations School of Mechanical Engineering, Pusan National University, Geumjeong-gu, Busan, Republic of Korea, School of Life Science and Technology, University of Electronic Science and Technology of China, West Hi-Tech Zone, Chengdu, Sichuan, P. R. China

Chang-Seok Kim

Roles Supervision, Validation, Visualization

Affiliation: Department of Cogno-Mechatronics Engineering, Pusan National University, Geumjeong-gu, Busan, Republic of Korea

Keum-Shik Hong

Roles Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing

* E-mail: [email protected]

Affiliations School of Mechanical Engineering, Pusan National University, Geumjeong-gu, Busan, Republic of Korea, Department of Cogno-Mechatronics Engineering, Pusan National University, Geumjeong-gu, Busan, Republic of Korea

ORCID logo http://orcid.org/0000-0002-8528-4457

Introduction

Customer behavior due to marketing is strongly influenced by the perception of the surrounding environment as shown by dandyism [1]. Visual merchandising display (VMD) is a marketing strategy to develop floor plans and displays to attract, engage, and stimulate the customer towards making a purchase. If done effectively, the marketers are able to maximize sales. It can also help marketers develop a unique marketing identity and brand, differentiating themselves in the market [2]. In this paper, by using functional near-infrared spectroscopy (fNIRS), we investigate the brain activation of potential customers when viewing various VMDs of four fashion stores.

From the early twenty-first century, VMD has become a science and one of the main tools for increasing sales. A good display can attract the consumer’s attention, which is essential in the decision-making process leading to a purchase. To capture a consumer’s mind, the needs of the consumer when viewing the VMD should be considered. Law et al. [3] investigated, from a psychological perspective, the affective responses of the consumers to visual stimuli in stores by considering the aesthetic and symbolic aspects of function-oriented products. The conformity between the preconceived fashion image and the image projected from a fashion store works as a mediator that affects the actual purchase decision. Shin et al. [4] investigated the benefits of eco-friendly VMDs on satisfaction as well as the effect of satisfaction on store displays. They demonstrated experimental evidence that psychological and social satisfaction play a fundamental role in the benefit of eco-friendly VMDs leading to positive store characteristics. Such results indicate that social information results in positive psychological and emotional satisfaction, which consequently leads to a positive view of the store for retail customers.

Currently, VMD studies of the influence of brain activity from a visual stimulus has become an important research issue. To investigate the activated brain region when viewing positive and negative stimuli, Zhang et al. [5] used functional magnetic resonance imaging (fMRI) and reported that the local connections in the brain increased while viewing affective pictures. In the positive emotional network, a positive affect score was correlated with the betweenness value of the right orbital frontal cortex (OFC) and the right putamen, while in the negative emotional network, a negative affect score was correlated with the betweenness value of the left OFC and the left amygdala. Local efficiencies in the left superior and inferior parietal lobes were correlated with the subsequent arousal ratings of the positive and negative pictures, respectively. Metereau et al. [6] found that the activities in the medial OFC and the ventromedial prefrontal cortex (PFC) were correlated positively in terms of the expected values of the cues in anticipation of rewards and punishments. This finding indicates that the medial OFC and the ventromedial PFC encode a generally unsigned anticipatory value signal, regardless of the valence of anticipation (positive/negative) and type (appetitive/aversive events). Bercik et al. [7] investigated the impact of color illumination (an essential factor in food shoppers’ perception) using EEG. Schmeichel et al. [8] found that exercising (vs. not exercising) self-control increases left frontal cortical activity during picture viewing, particularly among individuals with a relatively more active behavioral activation system than behavioral inhibition system, and particularly when viewing positive pictures. A similar but weaker pattern emerged when viewing negative pictures. Using fNIRS to measure the brain hemodynamic responses (HRs), the finding from Kreplin et al. [9] indicated that the right PFC was involved in positive evaluation of visual art that may be related to the judgment of pleasantness or attraction. This result suggests that emotional processing takes precedence over digressing viewing discipline during visual deliberation of art. In this case, the emotional nodule was brought to the beginning because of the participant's search for a personal association with the art images. Hence, participants relied on their own judgement and personal relationship with the art to make a subjective evaluation of the price.

In this work, fNIRS, an optical imaging method [10–17] is used to obtain brain signals. fNIRS offers several advantages over other modalities, such as being non-invasive, portable, no-noise, harmless, and inexpensive [18–22]. fNIRS can measure the light intensity changes in near-infrared lights of wavelength between 650 nm and 1,000 nm [23] and monitors the variation in regional cerebral blood flow. These changes are caused by concentration variations in oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) from neural activity. The principles and practical applications of fNIRS were described in detail previously [10].

In this paper, the HRs evoked by viewing positive and negative pictures of four different store categories (menswear store, womenswear store, underwear store, and sportswear store) are measured. The obtained results are portrayed in brain activation maps using the t-values of the measured HRs and the desired HRs [24]. Two-class and four-class linear discriminate analysis (LDA) [25] classifiers are used to distinguish the HRs from different picture stimuli, for which seven features (mean, peak, variance, skewness, kurtosis, t-value, and slope) are utilized.

Materials and methods

Ethics statement

The experiment was conducted upon the approval of the Pusan National University Institutional Review Board. Written consent was obtained from all subjects prior to starting. The experimental procedure was conducted in accordance with the ethical standards described in the latest Declaration of Helsinki.

Participants

Twenty healthy subjects (age 28 ± 3, all right handedness, 10 males and 10 females) participated in this experiment. All of them were healthy with no previous history of neurological disorders. All participants were informed about the procedure and operation of the fNIRS system prior to providing written consent. They were asked to avoid body motion and remain relaxed during the experiment. When viewing the stimuli, the subjects were asked to focus on each picture presented on the computer screen and indicate either like or dislike.

Visual stimuli

The visual stimuli were projected onto a monitor screen (Samsung LED Model: LS24A300) in front of the participants. The viewing distance to the screen was approximately 50 cm. The size of screen is 23 inch (16:9) and the resolution is 1,980 × 1,080 pixels. Four categories of fashion stores (menswear, womenswear, underwear, and sportswear) were shown on the screen to prompt negative and positive images. One trial consists of a 15 sec picture stimulus followed by a 30 sec rest for individual categories as shown in Fig 1. One session includes an initial 60 sec baseline correction period and a 450 sec task period (5 trials of positive and negative images in random order). The entire experiment for 4 categories takes a total of 2,040 sec.

[Figure omitted. See PDF.]

Fig 1. The experimental paradigm for ten trials of positive and negative store images: One trial includes a 15-sec stimulus followed by a 30-sec rest, and then a 60-sec baseline correction is performed followed by five trials of positive/negative images are randomly exhibited.

https://doi.org/10.1371/journal.pone.0208843.g001

Stimulation picture selection

The pictures used in this work were initially attained from the Internet (from a collection of standardized photographic materials) by typing “menswear store display,” “womenswear store display,” “underwear store display,” and “sportswear store display” into http://google.com [26, 27]. Only those pictures with good resolution were first screened. To narrow down the number of pictures to be used for experiment, an online survey was performed, in which two hundred sixteen people participated. In this survey, students and workers (all participants are adults) with different majors and positions were invited, yielding a total of 216 answers. In evaluating the quality of the pictures, a 10 points scale was used (10 means the most positive, and 1 means the most negative), see Fig 2 for instance. Lastly, 10 pictures (5 positive, 5 negative) were selected for each category, which resulted in a total of 40 pictures for 4 categories (menswear, womenswear, underwear, sportswear). It is noted that the scores of the selected 5 positive pictures ranged from 8 to 10 on average, and those of the 5 negative pictures ranged from 1 to 3. It is also remarked that four out of 10 underwear pictures were from men’s, and six out of 10 sportwear pictures were from women’s. Fig 3 shows typical examples of positive and negative images from the four categories.

[Figure omitted. See PDF.]

Fig 2. The example of the survey.

https://doi.org/10.1371/journal.pone.0208843.g002

[Figure omitted. See PDF.]

Fig 3. Examples of positive and negative displays: Four categories.

https://doi.org/10.1371/journal.pone.0208843.g003

Signal acquisition and processing

To measure brain activity, a multi-channel continuous fNIRS imaging system (DYNOT: Dynamic Near-infrared Optical Tomography; two wavelengths: 760 and 830 nm; NIRx Medical Technologies, NY) with a sampling rate of 1.81 Hz was used. A 16-channel probe configuration was placed on the prefrontal cortex [28–30] aligned to the locations of Fp1 and Fp2 in the International 10–20 System as shown in Fig 4.

[Figure omitted. See PDF.]

Fig 4. Optode arrangement in the prefrontal cortex: Squares and circles are the detectors and emitters, respectively, and Fp1 and Fp2 denote the reference points from the International 10–20 System.

https://doi.org/10.1371/journal.pone.0208843.g004

To investigate the fNIRS data, the open source software NIRS-SPM [31] was used together with our own MATLAB codes (Mathworks, USA). The obtained intensity signals of near-infrared lights were first converted to HbO and HbR concentration changes using the modified Beer-Lamberts law.(1)where ΔA denotes the variation in the optical density of the near-infrared light emitted at wavelength λj, αHbX(λ1) refers to the extinction coefficient of HbX in μMmm−1, d denotes the differential path length factor (DPF) that is unitless, and l indicates the emitter-detector distance (in mm). To remove the physiological noise related to respiration and cardiac signals, a 4th order Butterworth low-pass filter with a cutoff frequency of 0.15 Hz was used [32]. Additionally, to eliminate drift in the signal, de-trending was carried out inside the NIRS-SPM software [33].

Regions of interest

To visualize the brain activity from fNIRS data, t-value maps are the most intuitive method. In this paper, the t-values were computed by using the robustfit function available in MatlabTM, which is an iterative reweighted least squares estimation algorithm with a bisquare weighting function. Let be the measured data at the i-th channel for the j-th stimulus (i.e., 10 stimuli for one category), and let N be the data size per stimulus (in this study, N = 64 since 15 sec stimulus and 20 sec rest were adopted using a sampling frequency of 1.81 Hz). It is noted that out of the 30 sec rest period, only 20 sec data were used. Then, the linear regression model is defined as follows [24].(2)where Xr∈RN×1 is the expected hemodynamic response upon viewing the given stimuli, 1∈RN×1 is a column vector made of 1’s to correct the offset of the baseline, β is the unknown coefficient indicating the activity strength of the expected hemodynamic response, and ε∈RN×1 denotes white Gaussian noise. The following equations describe the steps to obtain the t-values.(3)(4)where indicates the estimate of and stats refers to the statistical data including t-value, p-value, standard error (SE) of the estimated coefficient.

The brain activity-related coefficient was related to the t-value to determine if its value was statistically greater than the t-critical (tcrt) value. The tcrt value depends on the degrees of freedom (i.e., 63, which is N—1), and it is 1.67 in this case. If the shape of the HbO response is closer to that of the expected HR (yr), the t-value is high. If the HbO response is different, the t-value is low (or negative). If a channel with the computed t-value is greater than tcrt, the channel is considered active. After obtaining all the t-maps from the 4 categories, the regions of interest (ROIs) [34, 35] for individual categories are investigated through the maps. The ROI denotes the area where the t-value(s) are higher than tcrt.

Fig 5 depicts the averaged brain activation maps over five trials for 20 subjects, in which MP, MN, WP, WN, UP, UN, SP, and SN stand for menswear positive, menswear negative, womenswear positive, womenswear negative, underwear positive, underwear negative, sportswear positive, and sportswear negative, respectively. Fig 6A compares the positive and negative maps averaged over all 20 subjects, Fig 6B contrasts those averaged over ten male subjects, and Fig 6C analyzes those averaged over 10 female subjects. As seen in Fig 6A, 6B and 6C, the ROIs when viewing positive/negative displays from the four categories are clearly different. For further analysis of feature selection and classification, only the active channels from each ROI were used because using the signals only from the associated ROI (rather than using the data from all channels) results in better classification accuracy [36, 37].

[Figure omitted. See PDF.]

Fig 5. Brain activation maps (averaged over 5 trials) from 20 subjects based on t-values: Positive and negative displays for the four categories (MP: Menswear positive, MN: Menswear negative, WP: Womenswear positive, WN: Womenswear negative, UP: Underwear positive, UN: Underwear negative, SP: Sportswear positive, SN: Sportswear negative).

https://doi.org/10.1371/journal.pone.0208843.g005

[Figure omitted. See PDF.]

Fig 6. The averaged maps (over 20 subjects) for 4 categories of positive and negative displays.

https://doi.org/10.1371/journal.pone.0208843.g006

Feature extraction and classification

In this study, the averaged HbO signals from the active channels were considered for classification [38, 39] (see Fig 7 and Fig 8). For features, the following seven statistical signal properties (mean, peak, variance, skewness, kurtosis, t-value, and slope) were considered [22, 40].

* Mean: the average signal value (for 20 sec from the start of the display),

* Peak: the maximum value of the selected window (for 20 sec),

* Variance: a measure of the signal spread (for 20 sec),

* Skewness: a measure of the asymmetry of the signal around its mean relative to a normal distribution,

* Kurtosis: a measure for the sharpness of the peak relative to a normal distribution,

* t-value: a measure of a test statistic for the difference between the dataset and the expected HR,

* Slope: the value that fits a regression line to the given data set.

[Figure omitted. See PDF.]

Fig 7. Comparison of the averaged HRs of positive and negative displays (the x-axis indicates the time series, in which 0–15 sec is the stimulus period and 16–35 sec is the rest period; the y-axis indicates the magnitude of HbO).

https://doi.org/10.1371/journal.pone.0208843.g007

[Figure omitted. See PDF.]

Fig 8. Comparison of the averaged HRs (over 5 trials and 20 subjects) of positive and negative displays for the 4 categories.

https://doi.org/10.1371/journal.pone.0208843.g008

To enhance the classification accuracy, a 20 sec window signal was used rather than using the entire 45 sec task-rest period. The previous study [41] used a 2–7 sec time window for a 10 sec stimulus for classification purposes of brain-computer interface (BCI) based on two features: mean and slope. However, in this study, we applied a different stimulus period (15s) with different features selection. Therefore, a 20 sec time window is selected for further classification. Additionally, the feature values have been rescaled between 0 and 1 using the following equation.(5)where z∈Rn, z′ is the rescaled value between 0 and 1, min z is the smallest value, and max z is the largest value. For efficient classifications, several classifiers have been discussed in the literature [42–48]. Here, we used an LDA classifier to verify the HbO signals.

Results

Fig 5 shows all brain activation maps from 20 participants for menswear, womenswear, underwear, and sportswear by projecting positive and negative pictures of store displays and whose data were obtained from the prefrontal cortex. There is some similarity across the subjects for a given category. Therefore, using the data from the ROI of each experiment, the averaged brain activation maps for the 20 subjects were obtained as shown in Fig 6A. In Figs 6B and 6C, the gender-based brain activation maps are drawn.

In the menswear experiment shown in Fig 6A, channels 1, 5, 8, 10, 11, 12, 13, 14, 15, and 16 were activated when viewing positive displays, and channels 12, 14, and 16 were activated when viewing negative displays. In the womenswear experiment, channels 3, 6, 8, and 14 were activated when viewing positive displays, and channels 1, 9, 10, 13, 14, and 15 were activated when viewing negative displays. In the underwear experiment, only channel 16 was activated when viewing both positive and negative displays. In the sportswear experiment, channels 2, 5, 10, 11, and 14 were activated when viewing positive displays, and channels 1, 2, 7, 12, and 16 were activated when viewing negative displays. Figs 6B and 5C present the male and female PFC activation maps. For male subjects, all channels with MP display stimulation were activated. Channel 8, 9, 10, 11, 12, 16 were activated by UP displays. In other stimulations, a few channels were activated (WP: channel 8, SP: channel 11, MN: channels 12 and 16, WN: channels 13 and 14, UN: channels 8 and 10, SN: channels 10 and 12). For female subjects, MP, WP, WN, and SP showed a large number of activated channels. Conversely, no channels were activated in UP, and only channel 16 was activated in UN. Based on averaging the activated channels, either among all subjects or by gender, through the intuitive observation a significant asymmetry of the t-maps is shown in the prefrontal cortex.

Fig 7 compares the average HbO signals over time and their standard deviations over 20 participants for the four categories. The following findings can be summarized as follows. i) For menswear, the peak HbO values from positive stimuli were higher than those of negative stimuli. ii) In contrast, the peak value of positive displays of underwear and sportswear were lower than that of negative displays. iii) However, peak values of positive/negative displays of womenswear were not distinguishable. Overall, out of four categories, the negative sportswear display was the most influential to the customer, showing the highest HbO response at its peak value. Fig 8 compares all the HbO responses of positive/negative displays from the four categories. We conclude that the order of hemodynamic activation levels for positive displays are sportswear, womenswear, menswear, and underwear.

Table 1 shows the two-class classification accuracies of all feature combinations (averaged over 20 subjects). In individual positive and negative classifications, the highest accuracies are marked in bold (blue): 64.15% (menswear), 64.95% (womenswear), 69.12% (underwear), and 66.27% (sportswear). Based on these best accuracies, Table 2 lists the subject-wise results of two-class classifications between positive and negative displays, in which good/bad looking underwear stores were classified most accurately, at 67.04%, and menswear stores were classified least accurately at 64.15%. Table 3 compares the four-class classification accuracies based on the average of 20 subjects with all feature combinations. In classifying four-store positive displays, the highest accuracy was 50.70%. In the case of negative displays, the highest accuracy was 51.10%. Table 4 shows the subject-wise four-class classification accuracies. The accuracy averaged across 20 subjects for the four positive displays was 50.68% (Sub.10 shows the highest accuracy of 76.67%). In the case of negative displays, the averaged accuracy was 51.07% (Sub.7 had 66.95%). Table 5 shows the gender-based two-class and four-class classification accuracies. For male subjects, the best classification category was underwear (71.38%), and the classification accuracy for four-class positive displays was 51.69%. The order of classification accuracies was underwear (71.38%), sportswear (68.33%), menswear (58.50%), and womenswear (p = 0.069 is considered to be not statistically significant). For female subjects, the womenswear display showed the best classification accuracy (73.00%). The order was womenswear (73.00%), menswear (69.81%), underwear (64.71%), and sportswear (61.70%). For four-store classification, negative displays had 52.34% classification accuracy, and positive displays had 49.67% accuracy. For the statistical test, the one-sample t test was applied to evaluate the accuracies based on its chance levels (Table 6).

[Figure omitted. See PDF.]

Table 1. Feature-wise two-class classification of positive and negative displays averaged over 20 subjects (%) (MP/N: Menswear pos./neg., WP/N: Womenswear pos./neg., UP/N: Underwear pos./neg., SP/N: Sportswear pos./neg.).

https://doi.org/10.1371/journal.pone.0208843.t001

[Figure omitted. See PDF.]

Table 2. Subject-wise two-class classification of positive and negative displays based on best feature combination (%).

https://doi.org/10.1371/journal.pone.0208843.t002

[Figure omitted. See PDF.]

Table 3. Feature-wise four-class classification of positive and negative displays averaged over 20 subjects (%).

https://doi.org/10.1371/journal.pone.0208843.t003

[Figure omitted. See PDF.]

Table 4. Subject-wise four-class classification of positive vs. negative displays based on best feature combination (%).

https://doi.org/10.1371/journal.pone.0208843.t004

[Figure omitted. See PDF.]

Table 5. Gender-specific classification accuracies (%).

https://doi.org/10.1371/journal.pone.0208843.t005

[Figure omitted. See PDF.]

Table 6. Statistical testing for comparison of classification accuracy: aOne sample t-test: * and ** indicate p < 0.05 and p < 0.01, respectively.

https://doi.org/10.1371/journal.pone.0208843.t006

Discussion

In the present study, the authors used fNIRS to monitor brain signals for visual merchandising displays because the fNIRS machine is portable, silent, noninvasiveness, and low cost and allows an experiment comparable to fMRI. The adopted fNIRS technique has high potential as a brain imaging modality. In examining the brain activation, the maps and classification results were discussed in relation to the types of stores and genders. In this study, positive and negative displays from four categories (menswear store, womenswear store, underwear store, and sportswear store) were investigated. Gender-based classification was typically distinguishable in the categories of womenswear and sportswear.

We found that the activated brain regions were asymmetrical across subjects (see Figs 5 and 6). This kind of asymmetry in functional responses in the prefrontal cortex was observed in other reports as well [5, 8, 49]. The results from Achterberg et al. [49] showed more activation in the right dorsal lateral PFC during negative social pictures. Our results are consistent with their report since the negative displays of three of four categories (menswear, womenswear, and sportswear) showed higher activation in the right PFC (see Fig 6A), which illustrates the asymmetry of brain responses. The activation channel number based on t-test analysis shows that the number of positive display channels is greater than that of negative displays in the menswear and sportswear cases. Some previous studies [50, 51] reported that activation in the PFC is positively correlated with positive and negative art picture stimulation. Vessel et al. [51] found greater activation in the PFC for esthetically pleasing images. The gender-based t-maps were different (Fig 6B and 6C). This conclusion is consistent with previous research [52]. There are more active channels in menswear and underwear displays in the case of male subjects. Very few active channels were found in the womenswear displays. However, in the case of female subjects, womenswear displays produced more activated channels. Underwear displays yielded the poorest activation (see Fig 6C). The active brain region was asymmetrical in gender-based maps as well.

We applied an LDA classifier to the two-class classification problem of positive and negative displays (in which 21 different feature combinations were examined) and to the four-class classification problem with four stores. In classifying positive and negative menswear stores (see Table 1), the best accuracy was obtained from the combination of variance and slope, which was 64.15%. In the case of womenswear, the best accuracy was achieved from the combination of mean and peak values. In the case of underwear, the combination of skewness and t-value resulted in the highest accuracy, and lastly, in the case of sportswear, the combination of variance and slope gave the highest accuracy. It is noted that no common feature combination covering all four categories was found. However, as seen in Table 2, all averaged accuracies were higher than chance level (50%). Some subjects (subs. 11 and 15) had very high accuracy, up to 83.33%. Additionally, some subjects’ data showed poor accuracy lower than the chance level, which are marked in parentheses.

In the case of four-class classification for the four different stores, the results are more meaningful because the accuracies are much higher than the chance level (25%). Furthermore, gender-based classification accuracies provide a direct indication of vulnerable categories: females are sensitive to womenswear displays, men are sensitive to underwear displays, and both males and females are less emotional with menswear and sportswear displays. Based upon classification accuracies and activation maps, it is concluded that gender-specific store displays should be pursued. Positive and negative displays for different products are conveyed differently to the customer by different emotions [52, 53]. Such brain responses are recognized differently, as seen in the maps. From this view, the overall intensity of a map may be related to the customer’s interest. Therefore, the brain response from the PFC related to emotion or pleasantness by viewing positive or negative store pictures can be scrutinized in the early stages of interior design. This study indicates that the examination of brain activation will help fashion marketing in the future.

Even though some conclusions like the above can be made, current research reveals that the relationship between VMD and brain signals is still vague. There are only a few papers reporting the effectiveness of brain-based approaches. This lack of research is not because VMDs do not influence consumers, but because the machines to measure brain signals do not provide enough data. Therefore, a new technique to investigate VMDs and brain activation should be developed for neuro-marketing applications.

There are some factors that influence classification accuracy. i) Signal strength variation is due to several factors, including skull scalp thicknesses and hair pigment. ii) In the present study, we measured the PFC as a brain region without hair, but a more emotion-related brain region needs to be investigated. iii) A subject’s concentration level is another important factor to consider. During the experiment, some subjects reported that they felt sleepy and fatigued during the experiment closer to the end. Therefore, minimizing such subject-wise variation is another issue to consider. iv) Additionally, the surrounding environments also should be considered, such as machine noise. v) In this work, the pictures used in the experiment were obtained from the Internet. However, pictures attained from a planned store will provide a direct evaluation of the planned store. Finally, vi) cognitive effects between fixed and moving displays would be interesting [54].

Conclusions

In this study, we examined the possibility of using fNIRS as a brain-based tool to investigate the consumer’s response for fashion store displays. The hemodynamic responses from the prefrontal cortices of twenty subjects were quite distinctive when viewing positive and negative store images. This finding implies that consumer responses to tentative plans of new stores can be evaluated in advance using fNIRS. Because new interior design costs a lot of money, the proposed brain-based approach can reduce total expenditure and construction time. In this study, even though hemodynamic responses to general (uncoordinated) store displays were examined, the hemodynamic responses of coordinated displays for a planned item would provide more design-specific evaluation for prospective consumers. Overall, consumer behavior analysis related to VMDs deserves to be evaluated using fNIRS. Future research should include a design-specific evaluation of VMDs focusing on store layouts, mannequins, light, points of purchase displays, colors, atmospherics, exterior displays, and others. As fashion changes quickly, to follow the directions of consumer behaviors, it is expected that we will see a rapid growth in interdisciplinary research and subsequent development of VMDs utilizing a brain-based approach in fashion marketing.

Supporting information

[Figure omitted. See PDF.]

S1 File. Dataset.

The dataset for the four stores with positive/negative display stimulation (averaged over 20 subjects). To understand our work, fNIRS task-related data were provided as supporting information.

https://doi.org/10.1371/journal.pone.0208843.s001

(XLSX)

S2 File. The survey.

https://doi.org/10.1371/journal.pone.0208843.s002

(PDF)

Acknowledgments

This work was supported by the China Scholarship Council (grant no. 201408260012) and the National Research Foundation of Korea under the auspices of the Ministry of Science and ICT, Republic of Korea (grant no. NRF-2017R1A4A1015627).

Citation: Liu X, Kim C-S, Hong K-S (2018) An fNIRS-based investigation of visual merchandising displays for fashion stores. PLoS ONE 13(12): e0208843. https://doi.org/10.1371/journal.pone.0208843

References

1. Baek E, Choo HJ, Lee SH. Using warmth as the visual design of a store: Intimacy, relational needs, and approach intentions. J Bus Res. 2018; 88:91–101.

2. Estes Z, Brotto L, Busacca B. The value of art in marketing: An emotion-based model of how artworks in ads improve product evaluations. J Bus Res. 2018; 85: 396–405.

3. Law D, Wong C, Yip J. How does visual merchandising affect consumer affective response? An intimate apparel experience. Eur J Market. 2012; 46(1/2):112–33.

4. Shin JK, Park MS, Moon MK. Do eco‐friendly VMD and store reputation increase satisfaction of retail customers? Psychol Mark. 2015; 32(12): 1148–57.

5. Zhang W, Li H, Pan X. Positive and negative affective processing exhibit dissociable functional hubs during the viewing of affective pictures. Hum Brain Mapp. 2015; 36(2): 415–26. pmid:25220389

6. Metereau E, Dreher JC. The medial orbitofrontal cortex encodes a general unsigned value signal during anticipation of both appetitive and aversive events. Cortex. 2015; 63: 42–54. pmid:25243988

7. Bercik J, Horska E, Wang RW, Chen YC. The impact of parameters of store illumination on food shopper response. Appetite. 2016; 106: 101–9. pmid:27083127

8. Schmeichel BJ, Crowell A, Harmon-Jones E. Exercising self-control increases relative left frontal cortical activation. Soc Cogn Affect Neurosci. 2015; 11(2): 282–8. pmid:26341900

9. Kreplin U, Fairclough SH. Activation of the rostromedial prefrontal cortex during the experience of positive emotion in the context of esthetic experience. An fNIRS study. Front Hum Neurosci. 2013; 7:879. pmid:24391572

10. Villringer A, Planck J, Hock C, Schleinkofer L, Dirnagl U. Near infrared spectroscopy (NIRS): a new tool to study hemodynamic changes during activation of brain function in human adults. Neurosci Lett. 1993; 154(1–2): 101–4. pmid:8361619

11. Hu XS, Hong K-S, Ge SS. fNIRS-based online deception decoding. J Neural Eng. 2012; 9(2): 026012. pmid:22337819

12. Zhang Y, Liu X, Yang C, Wang K, Sun J, Rolfe P. A new approach to separate haemodynamic signals for brain-computer interface using independent component analysis and least squares. J Spectrosc. 2013; 2013.

13. Hong K-S, Nguyen HD. State-space models of impulse hemodynamic responses over motor, somatosensory, and visual cortices. Biomed Opt Express. 2014; 5(6): 1778–98. pmid:24940540

14. Khan MJ, Hong MJ, Hong K-S. Decoding of four movement directions using hybrid NIRS-EEG brain-computer interface. Front Hum Neurosci. 2014; 8: 244. pmid:24808844

15. Schecklmann M, Giani A, Tupak S, Langguth B, Raab V, Polak T, et al. Functional near-infrared spectroscopy to probe state-and trait-like conditions in chronic tinnitus: a proof-of-principle study. Neural Plast. 2014; 2014.

16. Muthalib M, Re R, Zucchelli L, Perrey S, Contini D, Caffini M, et al. Effects of increasing neuromuscular electrical stimulation current intensity on cortical sensorimotor network activation: a time domain fNIRS study. PLoS One. 2015; 10(7): e0131951. pmid:26158464

17. Hoppes CW, Sparto PJ, Whitney SL, Furman JM, Huppert TJ. Functional near-infrared spectroscopy during optic flow with and without fixation. PLoS One. 2018; 13(3): e0193710. pmid:29513720

18. Boas DA, Elwell CE, Ferrari M, Taga G. Twenty years of functional near-infrared spectroscopy: introduction for the special issue. Neuroimage. 2014; 85, 1–5. pmid:24321364

19. Binzoni T, Spinelli L. Near-infrared photons: a non-invasive probe for studying bone blood flow regulation in humans. J Physiol Anthropol. 2015; 34(1): 28.

20. Chiarelli AM, Maclin EL, Fabiani M, Gratton G. A kurtosis-based wavelet algorithm for motion artifact correction of fNIRS data. Neuroimage. 2015; 112: 128–37. pmid:25747916

21. Guerrero-Mosquera C, Borragan G, Peigneux P. Automatic detection of noisy channels in fNIRS signal based on correlation analysis. J Neurosci Methods. 2016; 271: 128–38. pmid:27452485

22. Huppert TJ, Karim H, Lin CC, Alqahtani BA, Greenspan SL, Sparto PJ. Functional imaging of cognition in an old-old population: A case for portable functional near-infrared spectroscopy. PLoS One. 2017; 12(10): e0184918. pmid:29023452

23. Bhutta MR, Hong K-S, Kim BM, Hong MJ, Kim YH, Lee SH. Note: three wavelengths near-infrared spectroscopy system for compensating the light absorbance by water. Rev Sci Instrum. 2014; 85(2): 026111. pmid:24593411

24. Santosa H, Hong MJ, Hong K-S. Lateralization of music processing with noises in the auditory cortex: an fNIRS study. Front Behav Neurosci. 2014; 8: 418. pmid:25538583

25. Liu X, Hong K-S. Detection of primary RGB colors projected on a screen using fNIRS. J Innov Opt Health Sci. 2017; 10(03): 1750006.

26. Breazeale M, Ponder N. Get the picture? Visual servicescapes and self-image congruity. J Bus Res. 2013; 66(7): 839–46.

27. Petermans A, Kent A, Van Cleempoel K. Photo-elicitation: Using photographs to read retail interiors through consumers' eyes. J Bus Res. 2014; 67(11): 2243–9.

28. Naseer N, Hong MJ, Hong K-S. Online binary decision decoding using functional near-infrared spectroscopy for the development of brain–computer interface. Exp Brain Res. 2014; 232(2): 555–64. pmid:24258529

29. Issa M, Bisconti S, Kovelman I, Kileny P, Basura GJ. Human auditory and adjacent nonauditory cerebral cortices are hypermetabolic in tinnitus as measured by functional near-infrared spectroscopy (fNIRS). Neural Plast. 2016; 2016.

30. Wang P, Wang G, Niu X, Shang H, Li J. Effect of transcranial direct current stimulation of the medial prefrontal cortex on the gratitude of individuals with heterogeneous ability in an experimental labor market. Front Behav Neurosci. 2017; 11:217. pmid:29209179

31. Ye JC, Tak S, Jang KE, Jung J, Jang J. NIRS-SPM: statistical parametric mapping for near-infrared spectroscopy. Neuroimage. 2009; 44(2): 428–47. pmid:18848897

32. Bauernfeind G, Wriessnegger SC, Daly I, Muller-Putz GR. Separating heart and brain: on the reduction of physiological noise from multichannel functional near-infrared spectroscopy (fNIRS) signals. J Neural Eng. 2014; 11(5): 056010. pmid:25111822

33. Zafar A, Hong K-S. Detection and classification of three-class initial dips from prefrontal cortex. Biomed Opt Express. 2017; 8(1): 367–83. pmid:28101424

34. Hong K-S, Santosa H. Decoding four different sound-categories in the auditory cortex using functional near-infrared spectroscopy. Hear Res. 2016; 333: 157–66. pmid:26828741

35. Nguyen HD, Hong K-S, Shin YI. Bundled-optode method in functional near-infrared spectroscopy. PLoS One. 2016; 11(10): e0165146. pmid:27788178

36. Jeong S, Choi H, Go G, Lee C, Ko SY, Park JO, et al. Feasibility study on magnetically steerable guidewire device for percutaneous coronary intervention. Int. J. Control Autom. Syst. 2017; 15(1): 473–479.

37. Bui TTQ, Vu TT, Hong KS. Extraction of sparse features of color images in recognizing objects. Int. J. Control Autom. Syst. 2016; 14(2): 616–627.

38. Hiwa S, Hanawa K, Tamura R, Hachisuka K, Hiroyasu T. Analyzing brain functions by subject classification of functional near-infrared spectroscopy data using convolutional neural networks analysis. Comput Intell Neurosci. 2016; 2016: 3.

39. Hong K-S, Bhutta MR, Liu X, Shin YI. Classification of somatosensory cortex activities using fNIRS. Behav Brain Res. 2017; 333: 225–34. pmid:28668280

40. Tai K, Chau T. Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface. J NeuroEng Rehabil. 2009; 6(1): 39.

41. Hong K-S, Naseer N., Kim YH. Classification of prefrontal and motor cortex signals for three-class fNIRS-BCI. Neurosci Lett. 2015; 587: 87–92. pmid:25529197

42. Garrett D, Peterson DA, Anderson CW, Thaut MH. Comparison of linear, nonlinear, and feature selection methods for EEG signal classification. IEEE Trans Neural Syst Rehabil Eng. 2003; 11(2): 141–4. pmid:12899257

43. Liu H, Yu L. Toward integrating feature selection algorithms for classification and clustering. IEEE Trans Knowl Data Eng. 2005; 17(4): 491–502.

44. Pereira F, Mitchell T, Botvinick M. Machine learning classifiers and fMRI: a tutorial overview. Neuroimage. 2009; 45(1): S199–209.

45. Parisi GI, Weber C, Wermter S. Self-organizing neural integration of pose-motion features for human action recognition. Front Neurorobotics. 2015; 9: 3.

46. Adewuyi AA, Hargrove LJ, Kuiken TA. Evaluating EMG feature and classifier selection for application to partial-hand prosthesis control. Front Neurorobotics. 2016; 10: 15.

47. Jung S. Stability analysis of reference compensation technique for controlling robot manupulators by neural network. Int. J. Control Autom. Syst. 2017; 15(2): 952–958.

48. Kim HH, Park JK, Oh JH, Kang DJ. Multi-task convolutional neural network system for license plate recognition. Int. J. Control Autom. Syst. 2017; 15(6): 2942–2949.

49. Achterberg M, van Duijvenvoorde AC, Bakermans-Kranenburg MJ, Crone EA. Control your anger! The neural basis of aggression regulation in response to negative social feedback. Soc Cogn Affect Neurosci. 2016; 11(5): 712–20. pmid:26755768

50. Vessel EA, Starr GG, Rubin N. The brain on art: intense aesthetic experience activates the default mode network. Front Hum Neurosci. 2012; 6: 66. pmid:22529785

51. Hammerslag LR, Gulley JM. Sex differences in behavior and neural development and their role in adolescent vulnerability to substance use. Behavl Brain Res. 2016; 298: 15–26.

52. Tonoyan Y, Looney D, Mandic DP, Van Hulle MM. Discriminating multiple emotional states from EEG using a data-adaptive, multiscale information-theoretic approach. Int J Neural Syst. 2016; 26(02): 1650005.

53. Aydin S, Demirtaş S, Ateş K, Tunga MA. Emotion recognition with eigen features of frequency band activities embedded in induced brain oscillations mediated by affective pictures. Int J Neural Syst. 2016; 26(03): 1650013.

54. Lee HG, Hyung HJ, Lee DW. Egocentric teleoperation approach. Int. J. Control Autom. Syst. 2017; 15(6): 2744–2753.