Alternative Type of the Trail Making Test in Nonnative English-Speakers: The Trail Making Test-Black & White

Abstract

Objective

The Trail Making Test (TMT) has its limitations when applied to Eastern cultures due to its reliance on the alphabet. We looked for an alternative tool that is reliable and distinguishable like the TMT and devised the Trail Making Test Black & White (TMT-B&W) as a new variant. This study identifies the applicability of the TMT-B&W as a useful neuropsychological tool and determines whether the TMT-B&W could play an equivalent role as the TMT.

Methods

The TMT-B&W uses numbers encircled by black or white circles as stimuli, instead of using the alphabet. A total of 138 participants were including containing groups of 31 cognitively normal controls (NC), 55 mild cognitive impairment (MCI), and 52 people with Alzheimer’s disease (AD). Along with the TMT-B&W, the TMT and other neuropsychological tests were administered to all subjects.

Results

A considerably low dropout rate for TMT B&W demonstrates that all participants were more willingly engaged in the TMT B&W than the TMT. In particular, subjects with cognitive impairments or lower levels of education performed better on the TMT-B&W than the TMT. The difference in time-to-completion of the TMT-B&W was significant according to the level of cognitive impairment. The TMT-B&W revealed a high correlation with the TMT and frontal lobe function test.

Conclusion

The TMT-B&W is as reliable and effective as the TMT. It is worth developing a new variant of the TMT.

About the Authors:

Hyun Jung Kim

Affiliation: Clinical Neuroscience Institute, Seoul National University Bundang Hospital, Bundang-gu, Seoungnam-si, Kyeonggi-do, Korea

Min Jae Baek

Affiliation: Clinical Neuroscience Institute, Seoul National University Bundang Hospital, Bundang-gu, Seoungnam-si, Kyeonggi-do, Korea

SangYun Kim

* E-mail: [email protected]

Affiliations Clinical Neuroscience Institute, Seoul National University Bundang Hospital, Bundang-gu, Seoungnam-si, Kyeonggi-do, Korea, Department of Neurology, Seoul National University Bundang Hospital, Bundang-gu, Seoungnam-si, Kyeonggi-do, Korea

Introduction

The Trail Making Test (TMT) [1] is a widely used neuropsychological test for identifying mild cognitive impairment and mild dementia [2]–[4]. The TMT measures psychomotor speed, attention, sequencing, mental flexibility, and visual scanning [5]–[7].

The TMT is frequently administered in English-speaking countries with limited utilization in cross-cultural contexts because of the use of the English alphabet on TMT-B [8]. The TMT consists of two parts A and B (TMT-A, TMT-B) which assess the different cognitive processes. TMT-A requires an individual to connect randomly distributed numbers in an ascending order, while on the other hand TMT-B, consists of both numbers and letters, requiring participants to connect numbers and letters alternatively (See Figure 1–a and b). The TMT-A is primarily a test of visual attention skills. It includes perceptual tracking and simple sequencing tasks, whereas the TMT-B, with the additional tasks associated with alternating the sequence pattern, is a test which is used as an index of the frontal executive function [9]–[11]. Therefore, Part B is thought to be a more sensitive measure for cerebral dysfunction than Part A [12].

[Figure omitted. See PDF.]

Figure 1. Trail Making Test.

Part A consists of the encircled numbers from 1 to 25 randomly distributed on the test sheet. Part B constitutes of the encircled numbers from 1 to 13 and the encircled letters from A to L that are randomly distributed on the test sheet.

https://doi.org/10.1371/journal.pone.0089078.g001

The effects of age and level of education have been widely suggested to affect the completion time of the TMT. Most studies have shown that subjects who are old and/or with low education take more time to complete the TMT in total, most notably on the TMT-B. The obvious verbal or linguistic component in the alphabetic sequencing aspect of Part B places illiterate and nonnative English-speakers at a distinct disadvantage[13]–[17] which led to a poor performance bias on the TMT-B among Korean seniors, particularly those with a lower education level who could not read English letters. In a clinical domain, a number of studies reported Korean elderly; poorly educated elderly; and patients with mild cognitive impairment often fail to perform well on the TMT-B [18].

The Color Trails Test (CTT) was developed to overcome the shortcomings of the TMT, with the intention of minimizing the cultural bias and offering reliable cognitive measures for diverse populations [19]. However, the CTT was not considered to be similar to the TMT, due to differences in trace and a possible Stroop effect on CTT-B [20]. Moreover, its pink and yellow backgrounds prevent the CTT from being considered an accurate tool for those with color blindness or those with visual defects. In addition, preparing expensive colored paper for the test is burdensome.

For those reasons, we developed the Trail Making Test-Black & White (TMT-B&W). Unlike the CTT, part B of the TMT-B&W demonstrates two sets of numbers (from 1 to 25) in each color(black and white) which requires a subject to connect the numbers in ascending order alternating between the two color sets. We matched the trace of TMT-B&W with that of the TMT. The purpose of this study is: (1) to examine the applicability of the TMT-B&W (2) to evaluate if there is a potential advantage of TMT B&W when it is administered to Eastern populations.

Methods

Ethical Issue

This study was approved by the Seoul University Hospital Institutional Review Board of each participating site and written informed consent was obtained from all subjects before all procedures. Participants who declined to participate, or did not participate, were eligible for treatment and were not disadvantaged in any other way by not participating in this study.

The approval number of IRV is “B-1306/208–107”.

Study Population

This study included 138 outpatients with 31 cognitively normal controls (NC); 55 patients with mild cognitive impairment (MCI); and 52 patients with Alzheimer’s disease (AD), age ranged from 50 to 80. Those who could not read letters and were poorly educated were also included in the sample. Every participant had reportedly experienced some sort of memory impairment before they came to the Clinical Neuroscience Institute of Seoul National University Bundang Hospital. At the baseline visit, a clinician screened subjects and diagnose them into three groups.

We recruited some of the outpatients from a health care center of Seoul National University Bundang Hospital and caregivers who were serving patients undergoing treatment at the Clinical Neuroscience Institute of Seoul National University Bundang Hospital as subjects of the NC group. The criteria for the NC were as follows: 1) no cognitive complaints verified by an informant; 2) higher scores, or at most one standard deviation below, than the mean score of Mini-Mental State Examination (MMSE) for adjusted education and age [21]; 3) absence of significant impairment in any cognitive functions; 4) preserved activities of daily living (ADL) [22]–[23]; 4) no causes of diseases that would undermine cognitive functions [24]; 5) Geriatric Depression Scale (GDS) scored <17 on the 30 item scale in the past one week.

Criteria for diagnosing MCI patients [25], were as follows: 1) cognitive complaints verified by an informant; 2) objectively abnormal cognitive impairment in one or more cognitive functions; 3) preserved ADL; 4) normal visual and auditory functions; 5) no neurological or psychiatric diseases 6) failed to meet the diagnostic criteria of dementia based on the National Institute of Neurological and Communicative Disorders and Stroke and Alzheimer’s Disease and Related Disorders Association (NINCDS-ADRDA) [26].

Among AD patients, 52 were considered to have probable AD with mild dementia severity. All AD patients met the following criteria: 1) being diagnosed as probable AD according to the NINCDS-ADRDA; 2) having mild dementia severity with Clinical Dementia Rating Scale Sum of Box (CDR-SOB) scores between 2.5 and 4.0 [27]–[28]; 3) no impairment in vision and hearing; 4) no diseases related to neurological symptoms or psychological dysfunctions.

The demographic characteristics of the participants are described in Table 1. There was a significant difference of age and scores on MMSE among the groups of NC, MCI, and AD (F(2,135) = 23.31, F(2,135) = 51.86, p<0.05).

[Figure omitted. See PDF.]

Table 1. Demographic Data for Participant Groups.

https://doi.org/10.1371/journal.pone.0089078.t001

Procedure and Materials

After gathering all participants’ personal information through one-on-one interviews, Clinical Dementia Rating (CDR) [29] was also carried out. Each participant was then individually administered the TMT and TMT-B&W and then given a neuropsychological test to measure diverse cognitive functions. Given the fact that pre-exposure to cognitive tests may influence a test subject’ performance, we administered the TMT and TMT-B&W prior to the other tests. Half of the participants tested the TMT-B&W before the TMT.

Trail Making Test-Black & White

The TMT-B&W retains the same psychometric properties as the TMT, but relies on the use of encircled numbers with black and white backgrounds instead of an English alphabet letters. TMT-B&W consists of two subsets, TMT-B&W part A and TMT-B&W part B (TMT-B&W-A, TMT-B&W-B). The part A consists of 25 circled numbers (1–25), with even numbers in a black circle and odd numbers in a white circle (See Figure 2–a). The part B displays all numbers (2–25) twice -except 1, which is presented only one time in a white circle-, each corresponding number encompassed in both a black and white circle. The trace of TMT-B&W matches up with that of the TMT.

[Figure omitted. See PDF.]

Figure 2. Trail Making Test-Black & White.

The TMT-B&W Part A was similar to the TMT Part A, with an exception of all odd-numbers are within a white circle, and all even-numbers a black circle. The TMT-B&W Part B, each number is presented both in a white circle and a black circle, except the first encircled number is only written once.

https://doi.org/10.1371/journal.pone.0089078.g002

In TMT-B&W-A, the subject is instructed to draw a line to connect the circles in ascending order. TMT B&W-B consists of double the stimuli compared with the TMT B&W-A with two sets of the 25 numbers in each color (black and white). In TMT-B&W-B, the participant is required to connect the numbers in consecutive order alternating between the two color sets. A maximum time of five minutes was allowed. To make sure each subject fully understood how to do the task, a practice trial session was given to them before the test trial.

The time taken to complete the task was measured in seconds. Errors were also counted, but did not serve as an analysis factor.

Other Neuropsychological Assessments

A series of neuropsychological assessment tools were used to examine a wide variety of cognitive functions: attention, verbal and visual memory, visuospatial ability, frontal lobe functions, and language ability (See Table 2).

[Figure omitted. See PDF.]

Table 2. Neuropsychological assessments.

https://doi.org/10.1371/journal.pone.0089078.t002

Data Analysis

A one-way analysis of covariance (ANCOVA) test was performed to assess differences in time-to-completion on the TMT and TMT-B&W among the groups with the age entered as a covariate.

In order to determine whether the TMT-B&W can be applied to undereducated participants, the test results of participants with high levels (+6years) and low levels (0–5years) were separately calculated. A Correlation analysis was carried out to identify the construct validity of the TMT-B&W and the relations with the TMT-B&W and other neuropsychological tests including the TMT. All analyses were done with the PASW statistics, version 18.0. Statistical significance level was set to P<0.05.

Results

1. Overall Difference in the Time to Completion between the TMT and TMT-B&W

The results of time-to-completion in both the TMT Part A and B revealed that there were statistically significant differences among the groups (F(2,46) = 2.64, F (2,46) = 6.43, p<0.05) (See Table 3). The post hoc comparisons showed that performance of the NC and MCI group were significantly different from the AD group, while there was no significant gap between the NC and the MCI groups.

[Figure omitted. See PDF.]

Table 3. Summary Data for participant Groups on TMT.

https://doi.org/10.1371/journal.pone.0089078.t003

Among the groups, there were significant differences in regard to the time-to-completion on the TMT-B&W-B (F(2,85) = 15.83, p<0.05), however, not on Part A (See Table 4). The post hoc comparison suggested that the performance of the three groups were considerably different from one another.

[Figure omitted. See PDF.]

Table 4. Summary Data for participant Groups on TMT-B&W.

https://doi.org/10.1371/journal.pone.0089078.t004

2. Comparison of the Completion-ratio on the TMT-B&W and the TMT

Out of the total of 138 subjects, 88 were designated in the high education group(≥6 years). For the TMT, only 76% of the NC; about 60% of the MCI; and about the 37% with AD completed the tasks. Overall, the completion rate of the TMT-B&W was much higher than that of the TMT. Even participants with cognitive impairments complete the task. Among the 25 participants in the NC group, 24(96%) completed the task; of the total 33(MCI), 29(about 88%) subjects finished TMT-B&W amongst the 30 AD patients, as many as 18(60%) had successfully completed the TMT-B&W.

The observed phenomenon was even more drastic for participants with low education. Participants with low education (<6 years), including NC group, all failed to complete the TMT, while the TMT-B&W was completed by 4 out of 6 participants(about 67%) in the NC group. Among 22 MCI, 13(about 59%) subjects finished the task. As for 22 AD patients who had seen their cognition deteriorate, only one subject finished the TMT-B&W (about 5%). The completion rate of the TMT and the TMT-B&W are presented in Table 5.

[Figure omitted. See PDF.]

Table 5. Ratio of completing TMT and TMT-B&W.

https://doi.org/10.1371/journal.pone.0089078.t005

3. The Analysis of the TMT-B&W According to Education Group

When the participants were divided into two groups(lower and higher education), the results of participants with higher education suggested that only the time-to-completion in the TMT-B&W-B was significantly different among the groups(NC, MCI and AD) (F(2,67) = 44.14, p<0.05) (See Table 6). The post hoc comparison revealed all three diagnostic groups differed considerably in regards to the time-to-completion on TMT-B&W -B.

[Figure omitted. See PDF.]

Table 6. Summary Data on TMT-B&W for Participant Groups with over 6 years of education.

https://doi.org/10.1371/journal.pone.0089078.t006

4. Correlation of the TMT-B&W with the Other Tasks

To identify the validity of TMT-B&W, the concurrent validity was examined by comparing the correlation coefficients of the TMT-B&W with the value of the TMT, CDR (SOB), MMSE, and other neuropsychological tests respectively (See Table 7).

[Figure omitted. See PDF.]

Table 7. Correlation between TMT-B&W, TMT and cognitive measures.

https://doi.org/10.1371/journal.pone.0089078.t007

The results showed that the TMT-B&W significantly correlated with the TMT. Moreover, Part B of the TMT-B&W showed a high correlation with the Controlled Oral Word Association Test-Semantic word fluency (COWAT-S); Stroop test color reading (Stroop test-C). Part B of the TMT showed a correlation with those two tests, as well. Part B of the TMT also showed a correlation with the CDR SOB, MMSE, Boston Naming Test (BNT), Verbal Learning Test (VLT)-Immediate recall test, VLT-20-minute delayed recall test, VLT-Recognition test, Rey Complex Figure Test (RCFT) copy, RCFT-Immediate recall test, and the RCFT-20-minute delayed recall test.

Discussion

The TMT is one of the most useful neuropsychological tests. Many clinicians and neuropsychologists in non-native English speaking countries cannot effectively administer the TMT because of its limited utility in cross-cultural settings. In Korea, it has been hard to get test results from the TMT [30]. This study showed there were many patients who were not familiar with the English alphabet and thus did not even attempt the TMT. Even those who tried to conduct the TMT quit halfway through the test. Thus alternative forms of the TMT and other modifications are required to be introduced with an aim to improve diagnostic utility in the evaluation of neurobehavioral disorders.

This study aimed to identify a new type of TMT that maintains the diagnostic utilities of the TMT. First, we tried to identify if Korean participants willingly performed the TMT-B&W and then we were dedicated to verify if the TMT-B&W served as an adequate substitute to the original. If so is there a possibility that the TMT-B&W is applicable in a clinical setting.

As the results show, the completion rate of the TMT-B&W was much higher than that of the TMT. In other words, a significantly higher number of participants failed to complete the TMT. It could be explained in several ways: one is that a number of patients failed to finish the TMT; another is some of the subjects refused to start the task in the first place and others gave up halfway through the task. In particular this study revealed that 24% of participants with high education (NC) could not complete the task. The completion rate was 60% for MCI, and 36% for AD. Those numbers demonstrate well that the TMT has limited applicability in cross-cultural settings. The shortcoming stood out when the test was administered to patients with low education levels. Regardless of cognitive impairment levels, participants with low education (including NC) all failed to accomplish the TMT: some of them stubbornly resisted starting the test; others gave up from the beginning. Ironically the TMT, that was originally devised to measure the level of cognitive impairment, seemed to be inapplicable to cognitively impaired patients.

Interestingly the overall failure rate of the TMT-B&W was low. Even for those with low education and low a level of cognitive abilities, the failure rate of the TMT-B&W was much lower than that of the TMT. In participants with high education, among the 25 participants in the NC group, 1(4%) failed the task; of the total 33 MCI, 4(about 12%) subjects failed TMT-B&W amongst the 30 AD patients, as many as 12(40%) had uncompleted the TMT-B&W. The major reason for the failure in TMT-B&W was not because of participants’ unwillingness but because of time limitations. The number of participants who were unwilling to complete the test or gave up was 1 subject in MCI, 3 subjects in AD. One participant in NC, 3 in MCI and 9 in AD could not complete the task within the time limit. The TMT-B&W performance rate of those who finished up the task is evident in Table 8. If we were to also include the participants who over timed the task, the TMT-B&W performance rate would go up significantly. That means, if the task was given with a time extension or without a time limit, the performance rate of TMT-B&W would rise significantly. We would conclude that TMT-B&W is applicable for patients with low education and low level cognitive functions.

[Figure omitted. See PDF.]

Table 8. Ratio of performing the TMT-B&W.

https://doi.org/10.1371/journal.pone.0089078.t008

In a comparison to time consumed on the TMT and TMT-B&W among those with high education, the level of cognitive impairment contributed to an overall time-to-completion gap on both TMT -A and B. However, in the case of TMT-B&W, only Part B showed a significant influence on time-to-completion. However, there was no considerable time difference between the NC and the MCI on the TMT-B within the group comparison, and the significant difference existed only in the NC versus the AD groups and the MCI versus the AD groups. But the results showed that there were significant time-to-completion differences among the groups on the TMT-B&W-B. That indicates the TMT-B&W -B can be a more useful tool to determine cognitive impairments of patients. Therefore we may conclude that the TMT-B&W can play a more effective role in distinguishing the level of cognition for each patient as compared to the TMT, especially for non-English speakers.

But the gap in time-to-completion in TMT-B&W Part A and B among groups was not noticeably different for those with low education. That was because few of them managed to finish the task of the TMT-B&W within the time limit. Thus, as mentioned earlier, it is expected that when patients are given enough time to complete the task or allowed to perform the task without time limit, then more patients will be likely to complete the TMT-B&W. One other possible reason for no difference in lower educated participants is that the number of subjects with low education was not sufficient in this study. The small sample size in each group did not demonstrate significant differences in time-to-completion.

Like the TMT, TMT-B&W was found to be highly comparable with other neuropsychological tests, including frontal executive function tests. It is fair to say that the TMT-B&W serves a similar diagnostic function as the TMT does. In particular the results from Part B of TMT-B&W and the TMT-B showed a high correlation with COWAT-S and Stroop test-C demonstrating that TMT-B&W is highly likely viewed as a sensitive tool to estimate the frontal lobe functions. We may conclude that the TMT and TMT B&W are essentially equal in their diagnostic value.

Based on the results of this study, it is more desirable to use the TMT-B&W package to assess the level of cognitive impairment in non-native English speaking countries. The TMT-B&W serves as an adequate substitute to the TMT, and is even more useful in identifying differences in time completion between certain groups. Furthermore, it is aimed at assessing cognitive functionality of particularly those who don’t have knowledge of the English alphabet or undereducated. For further study, the extension of time limit and an increased number of subjects are recommended in order to identify a possibility of application of the TMT-B&W to poorly educated patients. We also recommend that the applicability of TMT-B&W be analyzed in other countries (i.e. non-native English speaking ones); and a comparison study of TMT B&W and the TMT in native English speaking countries carried out.

Acknowledgments

The author would like to thank Yuyong Cha from Korea Immigration Service, Ministry of Justice, Seoul university of Foreign Studies, and Cognitive Science, Sungkyunkwan University, for giving useful comments about the style, tone, and overall quality of the writing.

Author Contributions

Conceived and designed the experiments: HJK SYK. Performed the experiments: HJK MJB. Analyzed the data: HJK. Contributed reagents/materials/analysis tools: HJK. Wrote the paper: HJK.

Citation: Kim HJ, Baek MJ, Kim S (2014) Alternative Type of the Trail Making Test in Nonnative English-Speakers: The Trail Making Test-Black & White. PLoS ONE 9(2): e89078. https://doi.org/10.1371/journal.pone.0089078

References

1. Leitan LA, Wolfson DA (1994) Selective and critical review of neuropsychological deficits and the frontal lobes. Neuropsychol Rev 4: 161–198.

2. Lu L, Bigler ED (2002) Normative data on trail making test for neurologically normal, Chinneses-speaking adults. Appl Neuropsychol 9: 219–225.

3. Fernadez AL, Marcopulos BA (2008) A comparison of normative data for the considerations for interpretation. Scand J Psychol 49: 239–246.

4. Pena-Casanova J, Quinones-Ubeda S, Quintana-Aparicio M, Aguilar M, Badenes D, et al. (2009) Spanish Multicenter Normative Studies (NEURONORMA Project): norms for verbal span, visuospatial span, letter and number sequencing, trail making test, and symbol digit modalities test. Arch Clin Neurospychol 24: 321–341.

5. Oosterman JM, Vogels RL, van Harten B, Gouw AA, Poggesi A, et al. (2010) Assessing mental flexibility: neuroanatomical and neuropsychological correlates of the Trail Making Test in elderly people. Clin Neuropsychol 24: 203–219.

6. Moll J, de Oliveira-Souza R, Moll FT, Bramati IE, Andreiuolo PA (2002) The cerebral correlates of set-shifting: an fMRI study of the trail making test. Arq Neuropsiquiatr 60: 900–905.

7. Kortte KB, Horner MD, Windham WK (2002) The trail making test, part B: cognitive flexibility or ability to maintain set? Appl Neuropsychol 9: 106–109.

8. Barncord SW, Wanlass RL (2001) The symbol trail making test: test development and utility as a measure of cognitive impairment. Appl Neuropsychol 8: 99–103.

9. Arbuthnott K, Frank J (2000) Trail making test, part B as a measure of executive control: validation using a set-switching paradigm. Jô Clin Exp Neuropsychol 22: 518–528.

10. Boucugnani LL, Jones RW (1989) Behaviors analogous to frontal lobe dysfunction in children with attention deficit hyperactivity disorder. Arch Clin Neuropsychol 4: 161–173.

11. Shute GE, Huertas V (1990) Developmental variability in frontal lobe function. Dev Neuropsychol 6: 1–12.

12. Horton AM (1979) Some suggestions regarding the clinical interpretation of the trail making test. Clin Neuropsychol 1: 20–23.

13. Hashimoto R, Merguro K, Lee E, Kassai M, Ishii H, et al. (2006) Effect of age and education on the Trail Making Test and determination of normative data for Japanese elderly people: the Tajiri Project. Psychiatry Clin Neurosci 60: 422–428.

14. Waldmann BW, Dickson AL, Monohan MC, Kazelskis R (1992) The relationship between intellectual ability and adult performance in the Trail Making Test and the Digit Symbol Modalities Test. J Clin Psychol 48: 360–363.

15. Wiederholt WC, Cahn D, Butters NM, Salmon DP, Kritz-Silverstein D, et al. (1993) Effects of age, gender, and education on selected neuropsychological tests in an elderly community cohort. J Am Geriatr Soc 41: 639–647.

16. Gaudino EA, Geisler MW, Squires NK (1995) Construct validity in the Trail Making Test: What makes Part B harder? Jô Clin Exp Neuropsychol 17: 529–535.

17. Lezak MD (1995) Neuropsychological Assessment (3rd. ed). New York: Oxford.

18. Kim HJ, Beak MJ, Chang YH, Jang IM, et al. (2011) Comparison between the original version of Trail Making Test with two Korean-Trail Making Tests. J Korean Dement Assoc 10: 95–101.

19. D’Elia LF, Satz P, Uchiyama CL, White T (1996) Color Trails Test. Professional manual Psychological Assessment Resources, Odessa.

20. Spreen S (1998) A compendium of Neuropsychological Tests. Administration, Norms, and Commentary. New York: Oxford University press.

21. Kang SJ, Na DL, Hahn SH (1997) A validity study on the Korean version of Mini-Mental State Examination in Dementia patients. J Korean Neurol Assoc 24: 300–308.

22. Kang SJ, Choi SH, Lee BH, Kwon JC, Na DL, et al. (2002) The Reliability and Validity of the Korean Instrumental Activities of Daily Living (K-IADL). J Korean Neurol Assoc 20: 8–14.

23. Christensen KJ, Multhaup KS, Nordstrom S, Voss K (1991) A cognitive battery for dementia: Development and measurement characteristics. Psychol Asse 3: 168–174.

24. Marshall GA, Fairbanks LA, Tekin S, Vinters HV, Cummings JL (2006) Neuropathologic correlates of activities of daily living in Alzheimer’s disease. Alz Dis Assoc Disord 20: 56–59.

25. Peterson R (2004) Mild cognitive impairment as a diagnostic entity. J Int Med 256: 183–194.

26. McKhann G, Drachman D, Folstein M, Katzman R, Price D, et al. (1984) Clinical Diagnosis of Alzheimer’s disease: report of the NINCDS-ADRDA Work Group under the auspices of Department of Health and Human Services Task Force on Alzheimer’s Disease. Neurology 34: 939–944.

27. O’Bryant SE, Waring SC, Cullum CM, Hall J, Lacritz L, et al. (2008) Staging Dementia Using Clinical Dementia Rating Scale Sum of Boxes Scores: A Texas Alzheimer’s Research Consortium Study. Arch Neurol 65: 1091–1095.

28. O’Bryant SE, Lacritz LH, Hall J, Waring SC, Chan W, et al. (2010) Validation of the New Interpretive Guidelines for the Clinical Dementia Rating Scale Sum of Boxes Scores in the National Alzheimer’s Coordinating Center Database. Arch Neurol 67: 746–749.

29. Morris JC (1993) The clinical dementia rating (CDR): Current version and scoring rules. Neurology 43: 2412–2414.

30. Park JS, Kang YW, Yi HS, Kim YJ, Ma HI, et al. (2007) Usefulness of the Korean Trail Making Test for the Elderly (K-TMT-e) in Detecting the Frontal Lobe Dysfunction. J Korean Dement Assoc 6: 12–17.