EXPANSE: A novel narrative serious game for the behavioral assessment of cognitive abilities

Abstract

EFs are a set of processes that supports many cognitive domains as goal setting, monitoring, planning, and cognitive-behavioural flexible control. Currently, many standardized paper-and-pencil tests or scales are used to assess EFs. These tests are easy to administer, score, and interpret but present some limitations in terms of generalizability of behaviours in real life. More recently, Information and Communication Technology has provided a higher ecological validity in the EFs assessment. In order to increase the ecological validity, we have developed a serious game (SG), named EXPANSE, which aim was to compare the participants’ game performance (latency times, and correct answers) with the results obtained in the traditional tasks and scales. 354 healthy subjects participated to the study and the findings showed significant correlations among standard tasks and the serious game. The exploratory nature of the present study, on one hand, highlighted that SG could be an additional behavioral tool to assess EFs and, on the other, we need further investigations, including clinical populations, for better defining the game sensitivity toward EF components. Finally, the results show that serious games are a promising technology for the evaluation of real cognitive behavior along with traditional evaluation.

About the Authors:

Irene Alice Chicchi Giglioli

Contributed equally to this work with: Irene Alice Chicchi Giglioli, Carla de Juan Ripoll

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Validation, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation: Instituto de Investigación e Innovación en Bioingeniería (I3B), Universitat Politècnica de València, Camino de Vera s/n., València, Spain

ORCID logo http://orcid.org/0000-0003-2577-0039

Carla de Juan Ripoll

Contributed equally to this work with: Irene Alice Chicchi Giglioli, Carla de Juan Ripoll

Roles Data curation, Formal analysis, Investigation, Resources, Visualization, Writing – original draft, Writing – review & editing

Affiliation: Instituto de Investigación e Innovación en Bioingeniería (I3B), Universitat Politècnica de València, Camino de Vera s/n., València, Spain

Elena Parra

Roles Methodology, Project administration, Software, Writing – review & editing

¶‡ These authors also contributed equally to this work.

Affiliation: Instituto de Investigación e Innovación en Bioingeniería (I3B), Universitat Politècnica de València, Camino de Vera s/n., València, Spain

Mariano Alcañiz Raya

Roles Conceptualization, Funding acquisition, Supervision

¶‡ These authors also contributed equally to this work.

Affiliation: Instituto de Investigación e Innovación en Bioingeniería (I3B), Universitat Politècnica de València, Camino de Vera s/n., València, Spain

Introduction

Cognitive functions are critical human aspects involving the nature and heterogeneity of everyday experience. Cognitive functioning is a broad term referring to higher-level mental processes of processing information involving the ability to sustain attention, inhibiting responses, planning behaviors, maintaining goals and information in mind, and solving problems [1, 2]. These processes include a subset of functions of the brain, called executive functions (EF), such as attention, inhibitory control, cognitive flexibility, planning, and problem solving, governing goal-directed behaviors and adapting responses in accordance with specific situations [3,4]. EFs are essential abilities in every aspect of life and in the past two decades, EFs have become a major focus of research in psychology [5–8]. Indeed, EFs dysfunctions have been found in many mental disorders including addictions [5], depression [6], and attention deficit disorder [7, 8], producing inabilities in many everyday living activities, such as study, recreational activities, social relationships and work [9].

Currently, many standardized paper-and-pencil tests or scales are used to assess EFs, such as the Dot Probe Task, Stroop Test, Go/NoGo, the TMTA-B, the Tower of London and the Wisconsin Card Sorting Test (WCST) [10]. These tests are easy to administer, score, and interpret but present some limitations in terms of ecological validity, limiting the generalizability of behaviors in real life [11]. Ecological validity refers to the capacity of a test to predict individual’s real-world performance and the current EF measures are abstract, decontextualized and not able to capture the real dynamic and complex performance in daily life activities [12, 10]. Indeed, various studies showed that low scores on traditional measures, do not inevitably entail poor executive behaviors in real life and vice-versa [9, 13, 14]. In accordance with these findings, new instruments for evaluating the complexity of EFs in situations similar to daily ones, such as the Behavioral Assessment of Dysexecutive Syndrome (BADS), has been developed and tested [15]. The BADS consists of six subtests and a Disexecutive Questionnaire (DEX) that assess everyday cognitive, emotional, and behavioral changes together with a self-report and an independent rater questionnaire [15]. The BADS has been successfully tested in different clinical populations [16, 17, 18] showing good validity [19], the DEX showed some limitations in measuring performance during real-life tasks [20].

More recently, the use of Information and Communication Technology (ICT) has substantially increased, potentially providing a higher ecological validity in the EFs assessment. On one hand, various applications of virtual reality (VR) have been developing for the assessment of cognitive functions [21–23]. Currently, most of the studies are in experimental phase, attempting to verify the construct and/or ecological validity comparing healthy and clinical populations to support rehabilitation interventions [24–29]. For example, Lo Priore et al. (2003) [25] compared a 3D-VR and a 2D store in which patients vs. healthy subjects had to explore the environments and solve six sequences of tasks, ordered for complexity and created to stimulate executive functions, programming, categorical abstraction, short-term memory and attention. For the ecological validity, the authors used physiological (skin conductance response: SCR), neuropsychological, and questionnaire measures, showing a significantly higher SCR during tasks in the 3D-VR condition. Rand et al. (2005, 2009) [28, 29] developed a virtual mall in which participants (healthy and post-stroke) have been involved in a shopping task EFs-based. The results showed that the 3D task is sensitive to differentiate between the two groups and that it positively correlated with traditional measures, providing support to construct and ecological validity. However, VR approach shows some limitations. First, only few studies have established a significant construct and ecological validity of VR environment [30]. Indeed, if, theoretically, VR could enhance the ecological validity on cognitive assessment, predicting more realistic functional behaviors in real-life activities, another relevant issue to take into consideration in the conceptual creation of simulated daily activity could be represented by the heuristic bias of past experience. The success or failing of past experiences can have an influence on future behaviors [31]. More in detail, subjective perceptions of past behaviors influenced behaviors in future experiences. This phenomenon depends on cognitive activity, such as the memory of experiences, personality and motivational aims that influence the judgment of memory of past experience and the future. Second, VR can create cyber sickness, including discomfort, fatigue, headache, and nausea [32]. Third, at technological level, the development of VR is high-cost and implicates maintenance [33], as well as VR systems require specific settings, such as lighting and large space that limit their application in clinical or educational fields [34, 35].

Serious games (SG) constitute another promising approach that is showing to be positively related to a variety of cognitive abilities such as concentration, attention and working memory [36, 37]. SG can be defined as games developed for specific purposes that vary in accordance with the aims, the technology involved, and the interaction [38, 39]. Furthermore, SG allows creating multiple tasks that incorporate several executive functions to carry out behaviors as if the participant was in the real world [40, 41]. Serious games have been proposed for psychological interventions [42], due to three advantages: appealing, engagement, and effectiveness. As mentioned above, computer games are widespread and the amount of time expended playing game is increasing [43, 44], theoretically allowing reaching people that no receive treatments [44]. Second, games are fun and able to create engagement enhancing the motivation and reducing drop-out [45, 46]. Third, SG can provide safe and responsive environments in which users can test, shape, change, and learn new behaviors [47, 48].

So far the use of SGs has been well addressed for clinical treatments and poorly addressed for behavioral assessment [36, 37, 42]. The present study, propose the use of a SG, named EXPANSE, for EF assessment with high ecological validity. EXPANSE has been created starting from the cognitive constructs related to EFs and the traditional assessment tests. A narrative storytelling game, settled in a spaceship, which aim is to discover a new planet where to live because the Earth has become uninhabitable, has been created for leading participants in the play. EXPANSE is focused on the participant that is the protagonist in the interactive story that drives him/her in the EF’s situations and activities (for more details see the material and methods section). In addition, the participant can explore and navigate in the environment, manipulate and interact with objects. The narrative nature gives the context to the activities to be solved by the participant, and for solving them he/she needs to concentrate, evaluate and decide strategies.

Material and methods

Subjects

A total of 354 healthy subjects (MMSE > 24; 177 = women and 177 = men; Mean Age = 39.72; SD = 8.90) participated in this study [49]. Subjects were provided by a panel company, which operates with an incentive system and managed the survey and task responses of this research. Before participating in the study, each participant received written information about the study and was required to give written consent for inclusion in the study. The study obtained ethical approval by the Ethical Committee of the Polytechnic University of Valencia.

Questionnaires

Four surveys were administrated before participants completed the tasks: demographic questionnaire (e.g. gender, age, level of computer games expertise), Cognitive Flexibility Scale [50], Attentional Control Scale [51] and Barratt Impulsiveness Scale [52].

Tasks

The tasks were developed using Unity 5.5.1f1 software and completed on a personal computer. The c# programming language was applied using the Visual Studio tool. Participants completed a total of 14 tasks (6 standard tasks; 8 self-designed games) randomly presented. Computerized versions of these standard tasks were administered: Dot Probe Task neutral version published by Miller and Fillmore (2010) [53], the neutral pictures (20 in total) were selected from the International Affective Picture System (IAPS) [54]; Go/NoGo Task [55]; Stroop Test [56]; Trail Making Task, paper-and-pencil-based version published by Reitan (1955) [57]; Wisconsin Card Sorting Test [58] and Tower of London—Drexler (TOLDX) [59]

Eight games were designed, each one according to one of the standard tasks mentioned, and presented in one overarching game-scenario. Four of these tasks were aimed to measure attention (Dot Probe Task: AT1 –AT2; Go/NoGo task: AT3; Stroop Test: AT4), while three of the games were thought to measure cognitive flexibility (Trail Making Task: CF1; Wisconsin Card Sorting Test: CF2 –CF3), and one game was intended to assess planning (Tower of London–Drexler: PL1).

* AT1- “The takeoff: As a pilot, you will have to take off the spaceship”—A cross in the middle of the screen was followed by a pair of pictures that were presented together. After that, participants were ordered to press the E or I keys if the following X appeared on the left or on the right respectively.

* AT2: “Resources: You have reached a new planet! There are a lot of resources that you must organize”- This task followed the same dynamic than AT1, but in this case, we used different pictures according with another context.

* AT3: “The aliens: In space there are many elements that you must deal with. The most dangerous are the aliens that want to attack the ship”—Four kinds of objects could appear in front of the spacecraft that the participant was driving: a petrol tank, a meteorite, a spacecraft or an alien. These objects appeared in pairs, first the users saw the meteorite or the petrol tank (cue); and then this object was followed by the spacecraft or the alien (target). The subject was instructed to shoot the alien (go) and they had to do nothing if they saw the spacecraft (no-go) (Fig 1A and 1B).

[Figure omitted. See PDF.]

Fig 1.

a,b. Screenshot of the SG-AT3. Two objects appeared in pairs (a) and participants were instructed to shoot the alien (b).

https://doi.org/10.1371/journal.pone.0206925.g001

* AT4: “The strongbox: The oxygen valve has broken! You must fix it, but it´s behind a coded strongbox”—A strongbox with a screen, a group of rotatory letters and two switches was shown. A colored word was presented in the vault screen, and participants were asked to use the switches to rotate the letters until writing the color of the word in the screen, ignoring its meaning (Fig 2).

[Figure omitted. See PDF.]

Fig 2. Screenshot of the SG-AT4.

Participants were asked to use the switches to rotate the letters until writing the color of the word in the screen.

https://doi.org/10.1371/journal.pone.0206925.g002

* CF1: “Water and food (Part A): You have run out of water. To obtain it you will have to pump up the level of the water little by little. (Part B) This time you have run out of water and food. To obtain them you will have to pump and cultivate at the same time”—This game was inspired in Gonzalez and colleagues´ work (2013). As in the standard task, this was a two-part puzzle task. The first part of the task (A) consisted in a 10x10 squares puzzle, that subjects were asked to go through, making only horizontal or vertical movements by matching consecutive bottles from the empty one until the full one. The second part of the task (B) consisted in a 10 x 10 squares puzzle. In this case, participants were asked to match two kinds of objects: bottles witch different levels of water and plants in different growing levels. The matching criterion could be changed along the task (from water to plant or from plant to water).

* CF2: “The orchard: Your orchard is running empty. You need to make grow a series of plants”—Participants were shown four plant groups; they could see one plant representing each group. Each one of these plants had different characteristics: number of branches, fruit, and growing state. As in the Wisconsin Card Sorting Task, only one of these features was the rule by which a new plant should be matched with one of the four groups. The classification criterion changed during the task (Fig 3).

[Figure omitted. See PDF.]

Fig 3. Screenshot of the SG-CF2.

The card on the right should be matched with one of the four groups on the left following one of the classification criterions.

https://doi.org/10.1371/journal.pone.0206925.g003

* CF3: “Fuel: Your ship has run out of fuel. Inside the ship you will find some elements that can be used to start the turbine”—In this task, participants needed to activate a turbine, and they were given four elements to do it (water, wind, fire and magnetism), that could be mixed by two. In each trial, there was only one combination that could activate the turbine, but this criterion was changeable along the task (Fig 4).

[Figure omitted. See PDF.]

Fig 4. Screenshot of the SG-CF3.

There was only one combination that could activate the turbine, but this criterion was changeable along the task.

https://doi.org/10.1371/journal.pone.0206925.g004

* PL1: “Escape: You are locked up in the control room. You need to open the door if you want to escape”—The game started in the control room, where participants had stayed locked, such a way that they needed to open the door in order to complete the task. Different objects were randomly distributed along the room, and the subjects must combine and use them in order to accomplish the goal.

Data analysis

The analyses were performed using SPSS version 22.0 (Statistical Package for the Social Sciences for Windows, Chicago, IL) for PC. After identifying and removing outliers using Mahalanobis distance, we verified the assumption of normality applying Kolmogorov Smirnov. Internal consistency of the scales was assessed via Cronbach´s alpha.

Pearson correlations were calculated between standard tasks and self-report, and the SG version dependent variables (DVs). These correlations were consolidated by T-Student test for both standard tasks and self-report with DVs.

Results

The following analysis was carried out with the enduring 301 subjects (see Table 1). The assumption of normality was confirmed (Kolmogorov Smirnov p > .05) and the internal consistency of the self-report scales was measured (Cronbach´s alpha αattention = .839, αcognitive flexibility = .825, αimpulsiveness = .732; bootstrap 95%). Table 2 shows descriptive data for the Standard Task and Serious Games variables.

[Figure omitted. See PDF.]

Table 1. Demographic data of participants (n = 301).

https://doi.org/10.1371/journal.pone.0206925.t001

[Figure omitted. See PDF.]

Table 2. Mean (SD) and [Range] values for standard tasks and serious games variables.

https://doi.org/10.1371/journal.pone.0206925.t002

Standard tasks–Serious games correlations

Pearson correlations calculated for each Standard Task and its related SG showed statistically significant main relationships between variables (Fig 5).

[Figure omitted. See PDF.]

Fig 5. Results of Pearson correlations between associated Standard Tasks and Serious Games Variable.

*p < .05 **p < .01.

https://doi.org/10.1371/journal.pone.0206925.g005

T-test

Furthermore, for verifying whether the average values of the SG performance differ significantly from the reference neuropsychological standard tasks values we conducted the t-test analyses (Table 3).

[Figure omitted. See PDF.]

Table 3. Results of t-test between standard tasks and serious games performance.

https://doi.org/10.1371/journal.pone.0206925.t003

Questionnaires–Serious games

Only AT3 latency time (correct answers-go) had weak significant negative correlation (r = -.182**, p-value = .002) with Cognitive Flexibility Scale results that has been confirmed by t-test results. Low cognitive flexibility group registered higher latency times (correct answers-go) (mean = 0.34) in contrast with high cognitive flexibility group (mean = 0.30) (p-value = .000).

Discussion and conclusions

Cognitive functions, such as EFs, are important components of human behavior in daily living activities that can be related to important functional disabilities [60–64]. Due to the importance of cognitive functions, it is essential to investigate the distinctive structures of the EFs in real life behaviors. As previously described, one of the most crucial limitations is the poor ecological validity of the EF standardized tests [12, 1]. Indeed, studies on EFs have revealed that even if a subject can achieve good results in the traditional tests, he/she may encounter difficulties in real life activities [9, 13–14]

SG can provide a more ecological simulated environment in which the participant has an active role allowing evaluating real cognitive behaviors [64]. In SG tasks, such as EXPANSE, it is possible to simulate life-like activities, which can involve several and different goals to achieve and the cognitive flexibility and planning to elaborate different strategies to accomplish them and to inhibit inappropriate actions.

Starting from these premises, the main aim of this study was to investigate the plausibility and feasibility of a narrative SG, to integrate the traditional evaluation of EFs with a more ecologically valid assessment. The high Cronbach’s alphas for attention, cognitive flexibility, and impulsiveness suggest good internal consistency across trials and the positive correlations with the standard tests provide initial evidences for convergent validity.

Our results yielded four main findings. First, EXPANSE correlated with those standard tasks closely associated. More specifically as for the latency times, the SG tasks attentional-related strongly correlated with the standard attentional tasks, such the Dot Probe Task, as well as moderate correlations were found between SG tasks cognitive flexibility and planning-based and the standard tasks. Second, according to the accuracy of the responses, EXPANSE games moderately correlated with standard tasks. Third, comparing the SG with standard tasks performance, EXPANSE provided longer times and lower correct answers than standard tasks. This last result could depend on the different interaction required by the two technological systems. Fourth, we did not find strong correlations between SG and self-report and we found one low negative correlation between one SG behavioural measure and the cognitive flexibility scales showing that low cognitive flexibility scores registered higher latency times in contrast with high cognitive flexibility group scores. This could suggest discordance between EFs’ behavioural and self-report measures on revealing and discriminating different aspects of EFs.

Even though the present findings are relevant, they present some limitations. First, healthy subjects that limit the sensitivity of the results composed the sample. Second, considering the use of a SG, it would be important to also assess the individual’s perception of EXPANSE usability, as well as individual differences in cognitive functioning to the field of ICT. Indeed, several studies showed that individual variables, such as game skills or knowledge, personality, age and gender can affect user interactions and performance [65]. Further studies are required to examine plausibility, feasibility of EXPANSE game, mainly regarding its sensitivity, including clinical populations, such as patients that show altered performance in the executive domains and/or that present other cognitive symptoms (aphasia, apraxia, neglect, memory deficits) and/or motor problems (hemiparesis to upper and/or lower limbs)as well as its reliability and predictive validity according to the different criterions and the distinctive components of EFs. Nevertheless, the present study has yielded initial evidence on the potential use of a more ecological clinical rehabilitative tool to identify the (dys-)functional cognitive status in real-simulated contexts along with traditional evaluation.

Supporting information

[Figure omitted. See PDF.]

S1 Dataset. The data set contain the complete inventory and behavioural data.

https://doi.org/10.1371/journal.pone.0206925.s001

(XLSX)

Citation: Chicchi Giglioli IA, de Juan Ripoll C, Parra E, Alcañiz Raya M (2018) EXPANSE: A novel narrative serious game for the behavioral assessment of cognitive abilities. PLoS ONE 13(11): e0206925. https://doi.org/10.1371/journal.pone.0206925

References

1. Chan RC, Shum D, Toulopoulou T, Chen EY. Assessment of executive functions: Review of instruments and identification of critical issues. Archives of clinical neuropsychology. 2008; 23(2): 201–216. pmid:18096360

2. Lezak MD. The problem of assessing executive functions. International journal of Psychology. 1982; 17(1–4): 281–297.

3. Stuss DT, Alexander MP. Executive functions and the frontal lobes: a conceptual view. Psychological research. 2000; 63(3–4): 289–298. pmid:11004882

4. Diamond A. Executive functions. Annual review of psychology. 2013; 64: 135–168. pmid:23020641

5. Baler RD, Volkow ND. Drug addiction: the neurobiology of disrupted self-control. Trends in molecular medicine. 2006; 12(12): 559–566. pmid:17070107

6. Tavares J. V. T., Clark L., Cannon D. M., Erickson K., Drevets W. C., & Sahakian B. J. (2007). Distinct profiles of neurocognitive function in unmedicated unipolar depression and bipolar II depression. Biological psychiatry, 62(8), 917–924. pmid:17825802

7. Diamond A. Attention-deficit disorder (attention-deficit/hyperactivity disorder without hyperactivity): a neurobiologically and behaviorally distinct disorder from attention-deficit/hyperactivity disorder (with hyperactivity). Development and psychopathology. 2005; 17(3): 807–825. pmid:16262993

8. Lui M, Tannock R. Working memory and inattentive behaviour in a community sample of children. Behavioral and Brain Functions. 2007; 3(1): 12.

9. Chevignard M, Pillon B, Pradat-Diehl P, Taillefer C, Rousseau S, Le Bras C, Dubois B. An ecological approach to planning dysfunction: script execution. Cortex. 2000; 36(5): 649–669. pmid:11195912

10. Burgess PW, Alderman N, Volle E, Benoit RG,Gilbert SJ. Mesulam's frontal lobe mystery re-examined. Restorative neurology and neuroscience. 2009; 27(5): 493–506. pmid:19847072

11. Spooner DM, Pachana NA. Ecological validity in neuropsychological assessment: a case for greater consideration in research with neurologically intact populations. Archives of clinical neuropsychology. 2006; 21(4): 327–337. pmid:16769198

12. Chaytor N., Schmitter-Edgecombe M., & Burr R. (2006). Improving the ecological validity of executive functioning assessment. Archives of clinical neuropsychology. 2006; 21(3): 217–227. pmid:16554143

13. Barker LA, Andrade J, & Romanowski CAJ. Impaired implicit cognition with intact executive function after extensive bilateral prefrontal pathology: A case study. Neurocase. 2004; 10(3): 233–248. pmid:15788261

14. Manchester D, Priestley N, Jackson H. The assessment of executive functions: Coming out of the office. Brain injury. 2004; 18(11): 1067–1081. pmid:15545205

15. Wilson B. A., Alderman N., Burgess P. W., Emslie H., Evans J. J. (1996). Behavioural Assessment of the Dysexecutive Syndrome. London: Thames Valley Test Company.

16. Perfetti B., Varanese S., Mercuri P., Mancino E., Saggino A., & Onofrj M. (2010). Behavioural assessment of dysexecutive syndrome in Parkinson's disease without dementia: a comparison with other clinical executive tasks. Parkinsonism & related disorders, 16(1), 46–50.

17. Spitoni G. F., Aragonaa M., Bevacqua S., Cotugno A., & Antonucci G. (2018). An ecological approach to the behavioral assessment of executive functions in anorexia nervosa. Psychiatry research, 259, 283–288. pmid:29091830

18. Roy A., Allain P., Roulin J. L., Fournet N., & Le Gall D. (2015). Ecological approach of executive functions using the Behavioural Assessment of the Dysexecutive Syndrome for Children (BADS-C): Developmental and validity study. Journal of clinical and experimental neuropsychology, 37(9), 956–971. pmid:26332175

19. Wilson B. A., Evans J. J., Emslie H., Alderman N., & Burgess P. (1998). The development of an ecologically valid test for assessing patients with a dysexecutive syndrome. Neuropsychological rehabilitation, 8(3), 213–228.

20. Barker L. A., Morton N., Morrison T. G., & McGuire B. E. (2011). Inter-rater reliability of the Dysexecutive Questionnaire (DEX): comparative data from non-clinician respondents—all raters are not equal. Brain injury, 25(10), 997–1004. pmid:21749190

21. Martínez-Pernía D, González-Castán Ó, Huepe D. From ancient Greece to the cognitive revolution: a comprehensive view of physical rehabilitation sciences. Physiotherapy theory and practice. 2017; 33(2): 89–102. pmid:28071974

22. Parsons TD, Rizzo AA. Initial validation of a virtual environment for assessment of memory functioning: virtual reality cognitive performance assessment test. CyberPsychology & Behavior. 2008; 11(1): 17–25.

23. Knight RG, Titov N. Use of virtual reality tasks to assess prospective memory: applicability and evidence. Brain impairment. 2009; 10(1): 3–13.

24. McGeorge P, Phillips LH, Crawford JR, Garden SE, Sala SD, Milne AB, et al. Using virtual environments in the assessment of executive dysfunction. Presence: Teleoperators & Virtual Environments. 2001; 10(4): 375–383.

25. Lo Priore C, Castelnuovo G, Liccione D, Liccione D. Experience with V-STORE: considerations on presence in virtual environments for effective neuropsychological rehabilitation of executive functions. Cyberpsychology & behavior. 2003; 6(3): 281–287.

26. Marié RM, Klinger E, Chemin I, Josset M. Cognitive planning assessed by virtual reality. In VRIC 2003, Laval Virtual Conference. 2003: 119–125.

27. Klinger E, Chemin I, Hubner G, Marié RM. A virtual supermarket to assess cognitive planning. In CyberPsychology & Behavior. 2004; 7(3): 292–293.

28. Rand D, Katz N, Kizony R, Shahar M, Weiss PL. The virtual mall: A functional virtual environment for stroke rehabilitation. In Cyberpsychology & Behavior. 2005; 8(4): 369–370.

29. Rand D, Rukan SBA, Weiss PL, Katz N. Validation of the Virtual MET as an assessment tool for executive functions. Neuropsychological rehabilitation. 2009; 19(4): 583–602. pmid:19058093

30. Besnard J, Richard P, Banville F, Nolin P, Aubin G, Le Gall D, et al. Virtual reality and neuropsychological assessment: The reliability of a virtual kitchen to assess daily-life activities in victims of traumatic brain injury. Applied Neuropsychology: Adult. 2016; 23(3): 223–235.

31. Skinner B. F. (1953). Science and human behavior (No. 92904). Simon and Schuster.

32. Johnson DM. Introduction to and review of simulator sickness research. ARMY RESEARCH INST FIELD UNIT FORT RUCKER AL. 2005.

33. Parsey CM, Schmitter-Edgecombe M. Applications of technology in neuropsychological assessment. The Clinical Neuropsychologist. 2013; 27(8): 1328–1361. pmid:24041037

34. Rizzo AA, Schultheis M, Kerns KA, Mateer C. Analysis of assets for virtual reality applications in neuropsychology. Neuropsychological rehabilitation. 2004; 14(1–2): 207–239.

35. Werner P, Rabinowitz S, Klinger E, Korczyn AD, Josman N. Use of the virtual action planning supermarket for the diagnosis of mild cognitive impairment. Dementia and geriatric cognitive disorders. 2009; 27(4): 301–309. pmid:19252401

36. Andrews A. Serious games for psychological health education, In: Shumaker R (Eds), Virtual and Mixed Reality–Systems and Applications. Berlin, Heidelberg: Springer. 2011: pp. 3–10.

37. Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM. A systematic literature review of empirical evidence on computer games and serious games. Computers & Education. 2012; 59(2): 661–686.

38. Burke JW, McNeill MDJ, Charles DK, Morrow PJ, Crosbie JH,McDonough SM. Optimising engagement for stroke rehabilitation using serious games. The Visual Computer. 2009; 25(12): 1085.

39. Chatham RE. Games for training. Communications of the ACM. 2007; 50(7): 36–43.

40. Lamberts KF, Evans JJ, Spikman JM. A real-life, ecologically valid test of executive functioning: The executive secretarial task. Journal of Clinical and Experimental Neuropsychology. 2010; 32(1): 56–65. pmid:19408184

41. Poulin V, Korner‐Bitensky N, Dawson DR. Stroke‐specific executive function assessment: A literature review of performance‐based tools. Australian Occupational Therapy Journal. 2013; 60(1): 3–19. pmid:23414185

42. Fleming TM, Bavin L, Stasiak K, Hermansson-Webb E, Merry SN, Cheek C, et al. Serious games and gamification for mental health: current status and promising directions. Frontiers in psychiatry. 2017; 7: 215. pmid:28119636

43. Marchand A, Hennig-Thurau T. Value creation in the video game industry: Industry economics, consumer benefits, and research opportunities. Journal of Interactive Marketing. 2013; 27(3): 141–157.

44. Andrade LH, Alonso J, Mneimneh Z, Wells JE, Al-Hamzawi A, Borges G, et al. Barriers to mental health treatment: results from the WHO World Mental Health surveys. Psychological medicine. 2014; 44(6): 1303–1317. pmid:23931656

45. Batterham PJ, Neil AL, Bennett K, Griffiths KM, Christensen H. Predictors of adherence among community users of a cognitive behavior therapy website. Patient preference and adherence. 2008; 2: 97. pmid:19920949

46. Fleming TM, De Beurs D, Khazaal Y, Gaggioli A, Riva G, Botella C, et al. Maximizing the impact of e-therapy and serious gaming: time for a paradigm shift. Frontiers in psychiatry. 2016; 7: 65. pmid:27148094

47. Fleming TM, Cheek C, Merry SN, Thabrew H, Bridgman H, Stasiak K, et al. Serious games for the treatment or prevention of depression: a systematic review. 2014.

48. Cheek C, Fleming T, Lucassen MF, Bridgman H, Stasiak K, Shepherd M, Orpin P. Integrating health behavior theory and design elements in serious games. JMIR mental health. 2015; 2(2).

49. Folstein M. F., Folstein S. E., & McHugh P. R. (1975). “Mini-mental state”: a practical method for grading the cognitive state of patients for the clinician. Journal of psychiatric research, 12(3), 189–198. pmid:1202204

50. Martin MM, Rubin RB. A new measure of cognitive flexibility. Psychological reports. 1995; 76(2): 623–626.

51. Derryberry D, Reed MA. Anxiety-related attentional biases and their regulation by attentional control. Journal of abnormal psychology. 2002; 111(2): 225. pmid:12003445

52. Barratt ES. Anxiety and impulsiveness related to psychomotor efficiency. Perceptual and motor skills. 1959; 9(3): 191–198.

53. Miller MA, Fillmore MT The effect of image complexity on attentional bias towards alcohol‐related images in adult drinkers. Addiction. 2010; 105(5): 883–890. pmid:20148790

54. Lang PJ. International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical report. 2005.

55. Fillmore MT, Rush CR, Hays L. Acute effects of cocaine in two models of inhibitory control: implications of non‐linear dose effects. Addiction. 2006; 101(9): 1323–1332. pmid:16911732

56. Stroop JR. Studies of interference in serial verbal reactions. Journal of experimental psychology. 1935; 18(6): 643.

57. Reitan RM. Validity of the Trail Making Test as an indicator of organic brain damage. Perceptual and motor skills. 1958; 8(3): 271–276.

58. Grant DA, Berg E. A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. Journal of experimental psychology. 1948; 38(4): 404. pmid:18874598

59. Culberston WC, Zillmer EA. Tower of London-Drexel (TOLDX). Examiners’s manual. Research version. Canadá: Multi-Health Systems Inc. 1999.

60. Shallice TIM, Burgess PW. Deficits in strategy application following frontal lobe damage in man. Brain. 1991; 114(2): 727–741.

61. Royall DR, Lauterbach EC, Cummings JL, Reeve A, Rummans TA, Kaufer DI. Executive control function: a review of its promise and challenges for clinical research. A report from the Committee on Research of the American Neuropsychiatric Association. The Journal of neuropsychiatry and clinical neurosciences. 2002; 14(4): 377–405. pmid:12426407

62. Barker LA, Andrade J, Morton N, Romanowski CAJ, Bowles DP. Investigating the ‘latent’deficit hypothesis: Age at time of head injury, implicit and executive functions and behavioral insight. Neuropsychologia. 2010; 48(9): 2550–2563. pmid:20470806

63. Morton N, Barker L. The contribution of injury severity, executive and implicit functions to awareness of deficits after traumatic brain injury (TBI). Journal of the International Neuropsychological Society. 2010; 16(6): 1089–1098. pmid:20735889

64. Riva G. Virtual reality: an experiential tool for clinical psychology. British Journal of Guidance & Counselling. 2009; 37(3): 337–345.

65. Aykin NM, Aykin T. Individual differences in human-computer interaction. Computers & industrial engineering 1991; 20(3): 373–379.