Current Developments in Dementia Risk Prediction Modelling: An Updated Systematic Review

Abstract

Background

Accurate identification of individuals at high risk of dementia influences clinical care, inclusion criteria for clinical trials and development of preventative strategies. Numerous models have been developed for predicting dementia. To evaluate these models we undertook a systematic review in 2010 and updated this in 2014 due to the increase in research published in this area. Here we include a critique of the variables selected for inclusion and an assessment of model prognostic performance.

Methods

Our previous systematic review was updated with a search from January 2009 to March 2014 in electronic databases (MEDLINE, Embase, Scopus, Web of Science). Articles examining risk of dementia in non-demented individuals and including measures of sensitivity, specificity or the area under the curve (AUC) or c-statistic were included.

Findings

In total, 1,234 articles were identified from the search; 21 articles met inclusion criteria. New developments in dementia risk prediction include the testing of non-APOE genes, use of non-traditional dementia risk factors, incorporation of diet, physical function and ethnicity, and model development in specific subgroups of the population including individuals with diabetes and those with different educational levels. Four models have been externally validated. Three studies considered time or cost implications of computing the model.

Interpretation

There is no one model that is recommended for dementia risk prediction in population-based settings. Further, it is unlikely that one model will fit all. Consideration of the optimal features of new models should focus on methodology (setting/sample, model development and testing in a replication cohort) and the acceptability and cost of attaining the risk variables included in the prediction score. Further work is required to validate existing models or develop new ones in different populations as well as determine the ethical implications of dementia risk prediction, before applying the particular models in population or clinical settings.

About the Authors:

Eugene Y. H. Tang

* E-mail: [email protected]

Affiliation: Institute of Health and Society, Newcastle University Institute of Ageing, Newcastle University, Newcastle upon Tyne, NE2 4AX, United Kingdom

Stephanie L. Harrison

Affiliation: Institute of Health and Society, Newcastle University Institute of Ageing, Newcastle University, Newcastle upon Tyne, NE2 4AX, United Kingdom

Linda Errington

Affiliation: Medical School, Newcastle University, Newcastle upon Tyne, NE2 4HH, United Kingdom

Mark F. Gordon

Affiliation: Boehringer Ingelheim Pharmaceuticals, Inc., 900 Ridgebury Road, Ridgefield, Connecticut, 06877, United States of America

Pieter Jelle Visser

Affiliations Maastricht University, Department of Psychiatry & Neuropsychology, School for Mental Health and Neuroscience, Maastricht, The Netherlands, VU University Medical Centre, Department of Neurology, Alzheimer Centre, Neuroscience Campus, Amsterdam, The Netherlands

Gerald Novak

Affiliation: Janssen Pharmaceutical Research and Development, 1125 Trenton-Harbourton Road, Titusville, New Jersey, 08560, United States of America

Carole Dufouil

Affiliation: Inserm Research Centre (U897), Team Neuroepidemiology, F-33000, Bordeaux, France

Carol Brayne

Affiliation: Department of Public Health and Primary Care, Cambridge University, Cambridge, CB2 0SR, United Kingdom

Louise Robinson

Affiliation: Institute of Health and Society, Newcastle University Institute of Ageing, Newcastle University, Newcastle upon Tyne, NE2 4AX, United Kingdom

Lenore J. Launer

Affiliation: Laboratory of Epidemiology, Demography and Biometry, National Institute on Aging, National Institutes of Health (NIH), Bethesda, Maryland, United States of America

Blossom C. M. Stephan

Affiliation: Institute of Health and Society, Newcastle University Institute of Ageing, Newcastle University, Newcastle upon Tyne, NE2 4AX, United Kingdom

Introduction

Dementia is a complex disease often caused by a combination of genetic and environmental risk factors. Although many risk factors for the occurrence and progression of dementia have been identified, their utility for determining individual risk through dementia prediction models remains unclear.

Numerous models for predicting dementia, and more specifically Alzheimer’s Disease (AD), have been developed[1]. Such models could be used to refine inclusion criteria for clinical trials, focus treatment and intervention more effectively and help with health surveillance. A systematic review published in 2010 identified over 50 different dementia risk prediction models[1]. The models differed in the number and type of variables used for risk score calculation, follow-up time, disease outcome and model predictive accuracy. The review concluded that no model could be recommended for dementia risk prediction largely due to methodological weaknesses of the published studies. Model development had generally been based on small cohorts, restricted to Caucasians, and at the time of the review there had been a lack of objective and unbiased model evaluation, such as external validation.

Over the last five years, research into dementia risk prediction has greatly expanded and dementia prevention is a high policy priority in many countries. In order for clinicians, researchers and policy makers to keep up to date on relevant findings and make decisions about which model to apply to identify those high risk of future dementia, it is necessary to have an accurate knowledge of model development (including component variables and validation work), discriminative accuracy, and sensitivity and specificity of cut-off scores. In this review, based on the results of an updated literature search, we aim to evaluate the latest developments in dementia risk prediction modelling including a critique of the variables selected for model inclusion and an assessment of model prognostic performance.

Methods

Search Strategy

This review has been undertaken with adherence to the PRISMA statement[2]. MEDLINE, Embase, Scopus and ISI Web of Science were searched using combinations of the following terms and mapped to Medical Subject Headings (MeSH): “dementia”, “Alzheimer disease”, “Alzheimer and disease”, “predict”, “develop”, “incident”, “sensitivity”, “specificity”, “ROC” and “area under the curve”. The search included all literature published between the 1 January 2009 to 17 March 2014 (see Table 1 for the search strategy). Only articles published in English were considered. Additional articles were identified from the reference lists of eligible studies and relevant reviews. All articles published from January 2009 –November 2009 were also removed as this was covered in the original review.

[Figure omitted. See PDF.]

Table 1. Search Strategy.

https://doi.org/10.1371/journal.pone.0136181.t001

Selection of Studies

Selection of articles followed the same protocol as the 2010 review[1]. Two authors (ET and SH) independently searched publications using the following inclusion criteria: the sample was population-based and the article examined risk of dementia in non-demented individuals and included measurements of sensitivity, specificity or discrimination (i.e., area under the curve: AUC or c-statistic). Cross-sectional, case-control and clinical-based studies were excluded, as were studies that restricted the baseline sample by cognitive criteria other than dementia status; for example studies were excluded that focused only on subjects with Mild Cognitive Impairment (MCI). Articles where the outcome was a combined cognitive group, for example dementia cases combined with individuals with MCI, were also excluded. Titles and abstracts were searched first, followed by the full text of any identified articles. Where duplicate studies were identified, details of all novel risk models and their performance indices (AUC/c-statistic, sensitivity or specificity) were extracted from each publication and reported separately. One set of duplicate publications was found[3, 4]. However, the first paper[3] only presented the model and did not undertake testing. This paper was therefore excluded. Disagreements were solved by consensus between the two authors (ET and SH), or by a third author (BCMS) if the disagreement could not be resolved.

Data Extraction

Two authors (ET and SH) independently extracted information from each study including: sample, country, length of follow-up, baseline age, sex distribution, outcome tested (e.g., all-cause dementia vs. dementia subtypes), the components of each prediction model, discriminative accuracy (AUC or c-statistic), and where available, sensitivity and specificity estimates of cut-off scores and positive/negative likelihood ratio (LR+ or LR-, respectively). Two authors (ET, SH) independently assessed the quality of the included studies using an adapted version of the Newcastle-Ottawa Scale (NOS) for non-randomized studies, specifically cohort studies[5], as endorsed by the Cochrane collaboration [6]. The NOS uses a star rating system to assess selection, comparability and outcome criteria. Items describing a non-intervention cohort were excluded and therefore the total ranking was out of 6 (rather than 9).

Results

Included Studies

A total of 1,234 articles were identified from the electronic literature search (after removing duplicate publications) and 11 articles were identified from other sources. In total, 21 articles describing dementia risk prediction models were included in this review (Fig 1).

[Figure omitted. See PDF.]

Fig 1. PRISMA (2009) flow diagram of article selection.

https://doi.org/10.1371/journal.pone.0136181.g001

Table 2 summarises the study characteristics and predictive models. 13 new study populations[7–20] have been used to create risk models compared to the first review (all literature till 2009)[1]. There was however some overlap including use of data from the Cardiovascular Health Cognition Study[20, 21], Leipzig Longitudinal Study of the Aged (LEILA 75+)[22], Einstein Aging Study[23], Vienna TransDanube Aging[24], Canadian Study of Health and Aging[25] and the Kungsholmen Project[4]. Sample sizes ranged from 194[11] to 29,961[13]. Follow-up duration ranged from 1.4[22] to 20 years[7]. Follow-up rate ranged from 66%[12] to 86%[19], with 10 studies[4, 7, 11, 14, 15, 17, 20, 21, 23, 25] not reporting attrition or dropout. In ten studies[7, 10, 12–14, 16, 20–22, 26] the outcome tested was all-cause dementia, in nine studies[8, 9, 11, 15, 17–19, 23, 24] the outcome was AD and two studies had separate models for both AD and all-cause dementia [4, 25]. The number of predictors ranged from one[22] to 19[25].

[Figure omitted. See PDF.]

Table 2. Characteristics of included studies and reported dementia risk prediction models.

https://doi.org/10.1371/journal.pone.0136181.t002

Quality Assessment

Articles were assessed on selection, comparability and outcome (out of a maximum of six stars). In total, six[8, 13, 16, 18, 19, 26] articles scored six stars (maximum), 13 scored five stars[4, 7, 9–12, 14, 15, 17, 20, 22–24] and two[21, 25] scored four stars. This indicates that most articles were of high or moderate quality. Star ratings for each article are shown in Table 2.

Model Development

Most models have been derived using Logistic Regression[7, 9, 10, 21, 24–26] or Cox Proportional Hazards Regression analysis[8, 11, 13, 15–20, 22, 23], usually using stepwise selection to identify candidate predictors (e.g., based on a p-value; forwards or backwards), with one model using the Bayesian Information Criterion[19]). One model, the Australian National University Alzheimer’s Disease Risk Score (ANU-ADRI), was developed using an Evidence-Based Medicine Approach rather than through a data analytical approach[4] and another, the Brief Dementia Screening Indicator (BDSI) was developed using data synthesis based on the best dementia predictors identified in four different cohort studies[20]. When computed, simple risk scores have been derived from the model’s Beta-[4, 13, 16, 19, 20] or logit-coefficients[21]. This is similar to the methodology employed for risk model development in other fields of medicine such as cardiovascular disease[27, 28]. No models have yet been developed using systems biology or neural network approaches. Neural networks simulate the functions of neurons of human brains which can interact for processing data and learning from experiences [29]. Given the potential complexity of the risk factor variables used in dementia risk prediction, neural networks may also hold promise although this has yet to be evaluated.

Risk Models

Models can be broadly divided into the following categories: (1) demographic only models; (2) cognitive based models (incorporating cognitive test scores, with or without subjective memory/cognitive complaint indicators or demographic data); (3) health variables and health risk indices (incorporating self-reported or objectively measured health status); (4) genetic risk scores including APOE, PICALM (Phosphatidylinositol binding clathrin assembly protein), CLU (Clusterin) and other genes associated with AD (i.e., BIN1, CR1, ABCA7, MS4A6A, MS4A4E, CD2AP, EPHA1 and CD33), either alone or in combination with non-genetic variables; and, (5) multi-variable models typically incorporating demographic, health and lifestyle measures. Table 3 shows comparisons of the model components between this and the 2010 review and illustrates that there is large variability in model components and differences across the two reviews.

[Figure omitted. See PDF.]

Table 3. Component Variables Used (Either Alone or in Combination) in the Different Risk Prediction Models (Previous and Current Review).

https://doi.org/10.1371/journal.pone.0136181.t003

Differences are mainly in the addition of novel (non-traditional) dementia risk variables[25], information on diet[4], depression symptomology[4, 13, 24], ethnicity[14, 20, 23], and extension of genetic analysis to include non-APOE genes[17]. In addition, fewer cognitive tests are used and there is a smaller pool of candidate risk factors, likely due to more evidence being available.

Model Diagnostics

Performance of models has been assessed using measures of discriminative accuracy (e.g., AUC/c-statistic), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), internal calibration, LR+/LR-, the net reclassification index (NRI) and the integrated discrimination improvement (IDI) statistic. Discriminative accuracy was measured in all studies and ranged from low 0.49[4] to moderate 0.89[9]. Cut-off points with sensitivity and specificity estimates were only reported in five studies[7, 19, 22, 23, 26]. Where available, cut-off points were determined as follows: (1) maximisation of Youden’s index (Formula = Sensitivity + Specificity– 1)[22, 23, 26]; (2) computed by cross validation to correct for optimism as a result of validation on the learning data to correspond to a sensitivity value[7]; or, (3) defining the cut-off scores as those values with high specificity and increased PPV[19]. No model reported a cut-off score with both sensitivity and specificity over 80%. Three studies reported PPVs: (1) 9 to 41% (range across different length of follow-up interval and educational level)[7]; (2) 6.6 to 49.9% (range across different cut-off scores)[23]; and, (3) 14.7% with a cut-off that provided sensitivity of at least 80%[19]. Two studies reported NPVs: range across different cut-off scores: 86.0 to 99.0%[7] and 97.7%[19]. NPV should be higher than the proportion of subjects who did not have the outcome of dementia (i.e., the “stable” subjects), if the prediction is better than chance. Since the proportion of subjects not becoming demented was generally >85% (and always >70%)–as inferred from Table 2 –a high NPV is expected, because the number depends on the prevalence of the disease in the sample. The same logic can be applied to PPV. It should be noted that because the proportion of subjects becoming demented (or not) varies among the populations, variations in PPV and NPV do not necessarily reflect differences in performance of the models. Only one study[26] reported LR+/LR- and found that a neuropsychological prediction model provided a clinically important change in pre- to post-test probability of converting to dementia (all cause) over 5 and 10 years follow-up.

Model calibration was rarely reported, and where reported indicated good fit[13, 14, 20]. Reclassification indices including NRI and the IDI statistics that test the addition of variables to risk models, were used in two studies[14, 15]. The first study evaluated the influence of the APOE genotype on the accuracy of AD risk assessment using NRI and found a significant improvement (NRI 0.18, Z_NRI = 2.47, P = 0.01) compared to a non-APOE model; the IDI was estimated as 6.25 (Z_IDI = 3.75, P <0.001)[15]. The second used NRI and IDI to assess the improvements in performance of the Cardiovascular Risk Factors, Aging and Dementia (CAIDE) risk score by adding new risk factors[14]. Here, both the NRI and IDI showed no model improvements with the additional variables.

Models for Specific Subgroups of Individuals

One study developed a risk score, the Type-2 Diabetes Specific Dementia Risk Score (DSDRS), for predicting 10-year incident dementia, in a primary care setting, in a large cohort of individuals (N = 29,961) with type II diabetes[13]. The model incorporated age, education, microvascular disease, diabetic foot, cerebrovascular disease, cardiovascular disease, acute metabolic event and depression and was well calibrated and externally validated with moderate levels of predictive accuracy (AUC = 0.74 development cohort vs. AUC = 0.75 validation cohort). In another study, model development was undertaken in a sample stratified by education level (low: no elementary school diploma vs. high: secondary school or university) and follow-up time (3 vs. 10 years)[7]. This resulted in four different models that varied by education and length of follow-up as shown in Table 2.

Model Validation

Only four studies have undertaken validation[4, 13, 14, 20]. Differences between the AUCs in the development and validation cohorts for the different models tested are shown in Fig 2.

[Figure omitted. See PDF.]

Fig 2. Comparison of AUC indices in development vs. validation cohorts across different dementia risk prediction models.

Key: BDSI, Brief Dementia Screening Index; CAIDE, Cardiovascualr Risk Factors, Aging and Dementia; CHS, Cardiovascular Health Study; CVHS, Cardiovascular Health Cognition Study; DSDRS, Type-2 Diabetes Specific Dementia Risk Score; FHS, Framingham Heart Study; KP, Kungsholmen Project; KPNC, Kaiser Permanente Medical Care Program of Northern California; MAP, Rush Memory and Aging Project; PS-W, Pathways study cohort from Washington; Pts, Points; SALSA Sacramento Area Latino Study on Aging. References [1] Anstey KJ, Cherbuin N, Herath PM, Qiu C, Kuller LH, Lopez OL, et al. A Self-Report Risk Index to Predict Occurrence of Dementia in Three Independent Cohorts of Older Adults: The ANU-ADRI. PLoS One. 2014;9(1):e86141; [2] Exalto LG QC, Barnes D, Kivipelto M, Biessels GJ, Whitmer RA. Midlife risk score for the prediction of dementia four decades later. Alzheimers Dementia. 2013; [3] Exalto LG, Biessels GJ, Karter AJ, Huang ES, Katon WJ, Minkoff JR, et al. Risk score for prediction of 10 year dementia risk in individuals with type 2 diabetes: a cohort study. The Lancet Diabetes and Endocrinology. 2013; [4] Barnes DE, Beiser AS, Lee A, Langa KM, Koyama A, Preis SR, et al. Development and validation of a brief dementia screening indicator for primary care. Alzheimers Dementia. 2014:S1552-5260. Notes * No development dataset. Rather, model tested in different cohorts.

https://doi.org/10.1371/journal.pone.0136181.g002

CAIDE.

The CAIDE model was developed in sample of participants from Finland (N = 1,409, age range: 39 to 64 years) and uses risk factors in midlife to estimate an individuals’ risk of later life dementia (mean follow-up time = 20 years). The model incorporates age, education, sex, cholesterol level, BMI and systolic blood pressure (with and without APOE e4 status)[30]. Using data from the Kaiser Permanente (n = 9,480; age range: 40 to 55 years; mean follow up-time = 36.1 years) a similar AUC to the original study was reported (0.75 validation vs. 0.78 original cohort)[14]. Furthermore, when the Kaiser Permanente sample was stratified by ethnicity the CAIDE model was found to predict dementia well across different ethnicities including Asian (AUC = 0.81), Black (AUC = 0.75) and White (AUC = 0.74). These estimates are comparable to those reported in the original publication (model development dataset AUC = 0.78)[30]. This study also attempted to improve the discriminative accuracy of the CAIDE score with the addition of new variables such as central obesity, depressed mood, diabetes, head trauma, poor lung function and smoking but no significant improvement was shown[14].

When using the CAIDE risk score for predicting dementia in three older aged cohorts, the Rush Memory and Aging Project, the Kungsholmen Project and the Cardiovascular Health and Cognition Study, validation was found to be poor (AUC range all-cause dementia: 0.49 to 0.57; AUC range AD: 0.49 to 0.57)[4]. Interestingly, excluding BMI or BMI and cholesterol level together, modestly increased discriminative accuracy for all-cause dementia (AUC range: 0.55 to 0.60) and AD (AUC range 0.55 to 0.58)[4]. This result suggests that these variables may not be as important for predicting dementia in later vs. midlife. Indeed, in some studies of older aged cohorts higher BMI, cholesterol levels and blood pressure are found to be protective against dementia[31]. Therefore, poor transportability of the CAIDE model to these three cohorts may be due to the fact that the development dataset was a midlife cohort and the validation datasets were from older aged cohorts (Mean at baseline range: 72.3 to 81.5 years). The results could also be due to attrition rates as only 3% were lost to follow-up in the original study[30] compared to more than 20% of participants being lost in the three validation cohorts[4].

ANU-ADRI.

Using an evidence synthesis approach to model development, the ANU-ADRI model was developed to assess a persons’ risk for later life AD (i.e., over 60 years of age) based on exposure to 11 risk and four protective factors including: age, education, sex, BMI, diabetes, depression, cholesterol, traumatic brain injury, smoking, alcohol use, physical activity, pesticide exposure, social engagement, cognitive activity, and fish intake. Validation of the ANU-ADRI[4] score produced moderate levels of discrimination for dementia when between eight to 10 of the different risk/protective variables were mapped across three studies (AUC range: 0.64 to 0.74). The studies included the Rush Memory and Aging Study (N = 903), the Kungsholmen Project (N = 905) and the Cardiovascular Health Cognition Study (N = 2,496). The variables mapped included: demographic (age, gender, education), health (diabetes, traumatic brain injury, depressive symptoms), cognition (cognitive activity) and lifestyle factors (social network and engagement, smoking, alcohol, physical activity). When only common variables from all cohorts were used (n = 6 variables including: age, sex, education, diabetes, smoking, alcohol) the AUCs were: 0.69 (95% CI 0.65 to 0.73), 0.68 (0.63 to 0.70) and 0.73 (0.71 to 0.76) in the Rush Memory and Aging Study, the Kungsholmen Project and the Cardiovascular Health Cognition Study, respectively. The authors did not test whether the differences in AUCs when common variables were mapped across the different cohorts were statistically significant. These results are interesting and raise the question as to whether all or just some risk factors are needed to accurately predict future disease. It should be noted that one explanation for differences in AUC estimates could be variation in age. In particular the Kungsholmen Project, which included participants born before 1912 (baseline age > 75), was older than the other two samples (Rush Memory and Aging Study baseline age > 53, Cardiovascular Health Cognition Study > 65). The study also investigated the effect of gender on discriminative accuracy of the ANU-ADRI within each cohort and found only slight differences: the reported 95%CIs for males and females overlapped suggesting that any differences in discriminative accuracy were not significant.

BDSI.

The BDSI was developed with a three-step approach in four cohort studies including: the Cardiovascular Health Study, Framingham Heart Study, Health and Retirement Study, and the Sacramento Area Latino Study on Aging[20]. First, a list of potential predictive factors available in most or all cohorts was identified. Second, in each cohort, variables most predictive of dementia at six years were identified independently. Third, a subset of variables that were consistently found in all four cohorts was identified and used in the model including demographics (age, education), health (history of stroke, diabetes mellitus, BMI, depressive symptoms) and lifestyle (assistance needed with money or medications) factors. The c-statistic for predicting 6-year incident dementia varied between the 4 cohorts from 0.68 to 0.78. Sensitivity analyses, using data from the Health and Retirement Study and Cardiovascular Health Study, suggested that discrimination was good across different race/ethnic groups: Health and Retirement Study (c-statistic = 0.75 Whites, 0.70 Blacks, 0.71 Latinos) and the Cardiovascular Health Study (c-statistic = 0.70 Whites, 0.65 Blacks)[20].

Cost Considerations

One study considered the impact on discriminative accuracy of modifying the calculation of a resource intensive risk score to incorporate less expensive measures. The original resource intensive model, the Late Life Dementia Risk Index, included demographic (age), lifestyle (alcohol consumption), neuropsychological (Modified Mini Mental State Examination (MMSE) score and Digit Symbol Substitution Score), medical (history of coronary bypass surgery and BMI), physical functioning (time to put on and button a shirt in seconds), genetic (APOE), cerebral magnetic resonance imaging (MRI) (white matter disease and enlarged ventricles), and carotid artery ultrasound (internal carotid artery thickness >2.2mm) measures, and had good discrimination for the prediction of 6-year incident dementia (c-statistic = 0.81, 95%CI: 0.79 to 0.83)[32]. The revised model, the Brief Dementia Risk Index, incorporated age, neuropsychological testing (3 word delayed recall, interlocking pentagon copying, verbal instructions (paper taking and folding), four legged animal naming task (30 seconds)), self-reported attention difficulties (3 or more days per week in the last month), medical history (stroke, peripheral artery disease, or coronary artery bypass surgery and BMI) and alcohol consumption and had a significantly lower discriminative accuracy (c-statistic = 0.77, p<0.001), but was able to categorize subjects as having low, moderate, or high risk of dementia with similar accuracy compared to the more resource intensive score[21].

Discussion

The results from the review show that many new models for dementia risk prediction have been developed over the last five years. There have also been significant changes to the types of variables used when compared to the previous review. However few studies have addressed the issues of external validation and cost in using the risk model.

Strengths and Limitations

The strengths of this review are its systematic approach and inclusivity. There are some limitations. It is difficult to synthesise the literature on dementia risk prediction due to the large variability across studies in follow-up length (range: 1.5 years[9] to 17 years[15]), sample age (range: 40 to 99 years), outcomes tested (e.g., AD vs. all-cause dementia vs. dementia subtypes, quality of the diagnosis), source of population (volunteer vs. population representative) and the different variables incorporated into the prediction models. As such a meta-analysis was not possible. Furthermore, any meaningful conclusions for population screening are limited by the lack of cost-effective analysis and limited assessment of model transportability.

Clinical Implications

There is currently a clinical drive towards timelier diagnosis particularly in developed countries such as the UK with the introduction of primary care direct enhanced services looking to identify those at risk of developing dementia e.g. stroke, diabetes, cardiovascular disease[33]. A risk prediction tool, particularly in at risk populations such as diabetes[13] could further enhance existing services. However, if model development in the field of dementia continues at its current pace and if dementia risk prediction is found to be useful and cost-effective, then researchers and possibly clinicians will face difficult choices regarding which model to apply, particularly as study comparison is difficult.

Variables in the Prediction Models and Comparison to Results from the First Review.

Compared to the earlier review [1] elements common to the majority of risk scores include age, education, measures of cognition and health. However, new developments in dementia risk prediction include non-APOE genes and genetic risk scores[17, 18], testing of non-traditional dementia risk factors[25], incorporation of information on diet[4], physical function[4], physical activity/exercise and ethnicity[16] into risk modelling, and model development in specific subgroups of the population (e.g., individuals with diabetes[13] and those with low vs. high educational attainment[7]) and over different follow-up times. Furthermore, fewer cognitive tests have been used in prediction models, reflecting our increasing knowledge of risk and protective factors. Despite the dramatic increase in the number of models and novel risk scores, discriminative accuracy has not changed to a significant degree when compared to the previous review (range in the 2010 review: 0.49 to 0.91 vs. range in this review: 0.49 to 0.89). Aside from cognitive based models, generally, the best models are those that incorporate multiple risk factors across different variable categories (e.g., demographic, cognition, physical and health). Within the limits of our relatively limited knowledge of the genetic factors that influence dementia risk, where statistically tested, addition of novel (non-APOE) risk factors to prediction models do not appear to significantly increase discriminative accuracy[17]. In contrast, the APOE genotype, at least in some studies, appears to be informative. Future work into polygenic risk scores would help.

Further there has been a drive towards more accessible and potentially modifiable variables. Modifiable variables are important as they have the potential to be specifically targeted in primary or secondary prevention. There is now also evidence that around a third of AD cases worldwide may be due to modifiable risk factors [34]. With risk models likely to be used in the primary care setting, the availability of imaging variables may be difficult to obtain nor do they significantly improve discrimination in prediction of dementia beyond the more readily published multifactorial models [35].

Stratified Analyses

Results from stratified analysis suggest that unique dementia risk prediction models may need to be developed depending on follow-up length (e.g., BMI and hypertension may be more important in mid-life compared to later life models)[7, 23, 26, 36–38], an individual’s education level (found in one study and requires replication) [7], health status (e.g., diabetes)[13], APOE status[23] and the outcome tested (e.g., most studies focus on all-cause dementia and different models maybe needed depending on dementia subtype)[19]. Further research is required to develop risk models in samples stratified by other confounding factors known to influence the timing and presentation of dementia symptoms (such as mid vs. later life) as well as investigate the interactions between different risk variables (e.g., such as AOPE status and age).

Model Transportability

Before a risk prediction tool can be used in clinical practice or for research, transportability of the model outside the cohort from which it was developed needs to be assessed. Only four studies have externally validated dementia risk prediction models and the results were mixed[4, 13, 14, 20]. The DSDRS model was developed and validated with moderate but similar levels of discrimination (AUC 0.74 development vs. 0.75 validation)[13]. This is in comparison to the CAIDE score (originally developed for midlife), which was poorly validated in three separate (older) cohorts (AUC 0.77 development vs. AUC range 0.49–0.57 validation)[4]. However, it is important to note that in this validation of the CAIDE score rather different populations were used, that varied by age (mid vs. later life). Indeed, factors like higher BMI, blood pressure and cholesterol levels are associated with lower incidence of dementia in the oldest old [31]. In contrast, the CAIDE score was found to transport well when the test population better resembled the original population (i.e., was a mid-life cohort)[14]. Generally, external validation is difficult largely due to the lack of available cohorts with which to test models in terms of follow-up times, data collected, age groups and risk variable measurement.

Most studies have been developed in datasets from North America (N = 12 studies), with others developed in the UK (N = 1), Japan (N = 1), Austria (N = 1), the Netherlands (N = 2), Pan-Europe (N = 1), Germany (N = 3) and France (N = 1). Whether the different models are applicable across countries that vary by health and wealth is not known. Further, no models have been developed for predicting other dementia sub-types, such as vascular dementia or dementia with Lewy Bodies or for predicting different disease severity (e.g., mild, moderate and severe dementia). This may have implications for treatment options.

Issues Around Cost

The ability to assess the relative cost of calculating the different risk models and compare this against the model’s accuracy will significantly influence recommendations about possible protocols for screening for dementia risk. Further, the incorporation of readily accessible primary care related factors would be most useful for application within clinical and population based settings. Three studies [4, 20, 21] have considered time and/or financial implications in risk score calculation. However, in one study it was found that reducing the cost of risk score computation by removing the need for MRI, ECG and detailed neuropsychological measures and replacing them with less resource intensive variables (e.g., more detailed self-reported health history and simple cognitive test items) resulted in a significant decline in discriminative accuracy[21, 32]. Thus raising the issue of what is the best information and minimum data set needed for accurate dementia risk prediction. It is important to note that current models have not yet utilized cerebrospinal fluid or positron emission tomography data as recommended for classifying AD and its preclinical stages in new (clinical and research) criteria for AD and its preclinical/prodromal states[39, 40]. Although these factors can be used to assist dementia diagnosis, the feasibility and acceptance of incorporating them into risk prediction models and, particularly in population-based settings, would likely be low.

Conclusions

Before a risk assessment tool can be implemented we need to know its discriminative accuracy, predictive value, cost-effectiveness, transportability (e.g., to particular populations, ages and gender etc.), and the general availability of its variables (e.g., to enable cross-study comparison and result verification). We must also consider the design implications of the model: Are we interested in highly sensitive indicators of near term (i.e., 3-years) incidence of dementia? What operating characteristics are optimal for longer-term (5–10 years) predictive models? Are we contemplating a stratified approach, whereby low-cost screening identifies subjects with higher risk for more detailed and costly assessments?

It is not possible to state with certainty whether there exists one model that can be recommended for dementia risk prediction in population-based settings. This is largely due to the lack of risk score validation studies. Consideration of the optimal features of new models should largely focus on methodology (model development and testing) and the cost and acceptability of deriving the risk factors. Further work is required to validate existing models or develop new ones, as well as to assess their cost-effectiveness and ethical implications, before applying the particular models in population-based or clinical settings. While it is difficult to make a recommendation regarding which model, we nonetheless offer some recommendations of the optimal features for new models (see Table 4).

[Figure omitted. See PDF.]

Table 4. Optimum features of study design and variables selected for dementia risk prediction models.

https://doi.org/10.1371/journal.pone.0136181.t004

Supporting Information

[Figure omitted. See PDF.]

S1 PRISMA Checklist. PRISMA 2009 Checklist.

https://doi.org/10.1371/journal.pone.0136181.s001

(DOC)

Author Contributions

Conceived and designed the experiments: EYHT BCMS. Performed the experiments: EYHT SLH LE BCMS. Analyzed the data: EYHT SLH BCMS. Wrote the paper: EYHT SLH MFG PJV GN CD CB LR LJL BCMS.

Citation: Tang EYH, Harrison SL, Errington L, Gordon MF, Visser PJ, Novak G, et al. (2015) Current Developments in Dementia Risk Prediction Modelling: An Updated Systematic Review. PLoS ONE 10(9): e0136181. https://doi.org/10.1371/journal.pone.0136181

References

1. Stephan BCM, Kurth T, Matthews FE, Brayne C, Dufouil C. Dementia risk prediction in the population: are screening models accurate? Nat Rev Neurol. 2010;6(6):318–26. pmid:20498679

2. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of clinical epidemiology. 2009;62(10):e1–34. pmid:19631507.

3. Anstey KJ, Cherbuin N, Herath PM. Development of a new method for assessing global risk of Alzheimer's disease for use in population health approaches to prevention. Prevention science: the official journal of the Society for Prevention Research. 2013;14(4):411–21. Epub 2013/01/16. pmid:23319292; PubMed Central PMCID: PMCPmc3696462.

4. Anstey KJ, Cherbuin N, Herath PM, Qiu C, Kuller LH, Lopez OL, et al. A Self-Report Risk Index to Predict Occurrence of Dementia in Three Independent Cohorts of Older Adults: The ANU-ADRI. PLoS One. 2014;9(1):e86141. eCollection 2014. pmid:24465922

5. Wells GA, Shea B, O'Connell B, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa scale (NOS) for assessing the quality of nonrandomised studies in meta-analysis [cited 2014 19/03/2014]. Available: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.

6. Higgins JPT, Green S, Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions. Chichester: Wiley-Blackwell; 2009. xxi, 649 p. p.

7. Chary EAH, Pérès K, Orgogozo JM, Dartigues JF, Jacqmin-Gadda H. Short- versus long-term prediction of dementia among subjects with low and high educational levels. Alzheimers Dementia. 2013;9(5):562–71.

8. Okereke OI, Pantoja-Galicia N, Copeland M, Hyman BT, Wanggaard T, Albert MS, et al. The SIST-M Predictive Validity of a Brief Structured Clinical Dementia Rating Interview. Alzheimer Disease & Associated Disorders. 2012;26(3):225–31. WOS:000308186900005.

9. Wolfsgruber S, Jessen F, Wiese B, Stein J, Bickel H, Mösch E, et al. The CERAD Neuropsychological Assessment Battery Total Score Detects and Predicts Alzheimer Disease Dementia with High Diagnostic Accuracy. American Journal of Geriatric Psychiatry. 2013.

10. Madureira S, Verdelho A, Moleiro C, Ferro JM, Erkinjuntti T, Jokinen H, et al. Neuropsychological Predictors of Dementia in a Three-Year Follow-Up Period: Data from the LADIS Study. Dementia and Geriatric Cognitive Disorders. 2010;29(4):325–34. WOS:000278130700006. pmid:20389074

11. Grober E, Sanders AE, Hall C, Lipton RB. Free and cued selective reminding identifies very mild dementia in primary care. Alzheimer Disease and Associated Disorders. 2010;24(3):284–90.

12. Restaino M, Matthews FE, Minett T, Albanese E, Brayne C, Stephan BCM. Predicting risk of 2-year incident dementia using the camcog total and subscale scores. Age and Ageing. 2013;42(5):649–53.

13. Exalto LG, Biessels GJ, Karter AJ, Huang ES, Katon WJ, Minkoff JR, et al. Risk score for prediction of 10 year dementia risk in individuals with type 2 diabetes: a cohort study. The Lancet Diabetes and Endocrinology. 2013.

14. Exalto LG Q C, Barnes D, Kivipelto M, Biessels GJ, Whitmer RA. Midlife risk score for the prediction of dementia four decades later. Alzheimers Dementia. 2013. [Epub ahead of print].

15. Ohara T, N T, Kubo M, Hirakawa Y, Doi Y, Hata J, Iwaki T, Kanba S, Kiyohara Y. Apolipoprotein genotype for prediction of Alzheimer's disease in older Japanese: the Hisayama Study. J Am Geriatr Soc. 2011;59(6):1074–9. pmid:21649613

16. Reitz C T M, Schupf N, Manly JJ, Mayeux R, Luchsinger JA. A summary risk score for the prediction of Alzheimer disease in elderly persons. Archives of Neurology. 2010;67(7):835–41.

17. Seshadri SFA, Ikram MA, DeStefano AL, Gudnason V, Boada M, Bis JC, et al. Genome-wide analysis of genetic loci associated with Alzheimer disease. JAMA. 2010;303(18):1832–40. pmid:20460622

18. Verhaaren BFJ, Vernooij MW, Koudstaal PJ, Uitterlinden AG, van Duijn CM, Hofman A, et al. Alzheimer's Disease Genes and Cognition in the Nondemented General Population. Biological Psychiatry. 2013;73(5):429–34. WOS:000314634900008. pmid:22592056

19. Jessen FWB, Bickel H, Eiffländer-Gorfer S, Fuchs A, Kaduszkiewicz H, Köhler M, et al. Prediction of dementia in primary care patients. PLoS One. 2011;6(2):e16852. pmid:21364746

20. Barnes DE, Beiser AS, Lee A, Langa KM, Koyama A, Preis SR, et al. Development and validation of a brief dementia screening indicator for primary care. Alzheimers Dementia. 2014:S1552–5260.

21. Barnes DE, C K, Whitmer RA, Juller LH, Lopez OL, Yaffe K. Dementia Risk Indices: A Framework for Identifying Individuals with a High Dementia Risk. Alzheimers Dementia. 2010;6(2):138–41.

22. Ehreke L, Luppa M, König HH, Villringer A, Riedel-Heller SG. Does the clock drawing test predict dementia? Results of the Leipzig longitudinal study of the aged (LEILA 75+). Dementia and Geriatric Cognitive Disorders. 2011;31(2):89–97. pmid:21242690

23. Derby CA, Burns LC, Wang C, Katz MJ, Zimmerman ME, L'Italien G, et al. Screening for predementia AD: Time-dependent operating characteristics of episodic memory tests. Neurology. 2013;80(14):1307–14. pmid:23468542

24. Mossaheb N, Zehetmayer S, Jungwirth S, Weissgram S, Rainer M, Tragl KH, et al. Are specific symptoms of depression predictive of Alzheimer's dementia? Journal of Clinical Psychiatry. 2012;73(7):1009–15. pmid:22687442

25. Song X, Mitnitski A, Rockwood K. Nontraditional risk factors combine to predict Alzheimer disease and dementia. Neurology. 2011;77(3):227–34. pmid:21753161

26. Tierney MCMR, McDowell I. Prediction of all-cause dementia using neuropsychological tests within 10 and 5 years of diagnosis in a community-based sample. J Alzheimers Disease. 2010;22(4):1231–40.

27. Wilson PW, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97(18):1837–47. pmid:9603539

28. Siontis GC, Tzoulaki I, Siontis KC, Ioannidis JP. Comparisons of established risk prediction models for cardiovascular disease: systematic review. BMJ. 2012;344:e3318. pmid:22628003

29. Sheikhtaheri A, Sadoughi F, Hashemi Dehaghi Z. Developing and using expert systems and neural networks in medicine: a review on benefits and challenges. Journal of medical systems. 2014;38(9):110. Epub 2014/07/17. pmid:25027017.

30. Kivipelto M, Ngandu T, Laatikainen T, Winblad B, Soininen H, Tuomilehto J. Risk score for the prediction of dementia risk in 20 years among middle aged people: a longitudinal, population-based study. The Lancet Neurology. 2006;5(9):735–41. pmid:16914401

31. Martin-Ponce E, Santolaria F, Aleman-Valls MR, Gonzalez-Reimers E, Martinez-Riera A, Rodriguez-Gaspar M, et al. Factors involved in the paradox of reverse epidemiology. Clinical nutrition (Edinburgh, Scotland). 2010;29(4):501–6. Epub 2010/02/02. pmid:20116147.

32. Barnes DE, Covinsky KE, Whitmer RA, Kuller LH, Lopez OL, Yaffe K. Predicting risk of dementia in older adults The late-life dementia risk index. Neurology. 2009;73(3):173–9. WOS:000268147100004. pmid:19439724

33. Board NC. Facilitating Timely Diagnosis and Support for People with Dementia. In: Board NC, editor. 2013.

34. Norton S, Matthews FE, Barnes DE, Yaffe K, Brayne C. Potential for primary prevention of Alzheimer's disease: an analysis of population-based data. Lancet Neurol. 2014;13(8):788–94. Epub 2014/07/18. pmid:25030513.

35. Stephan BC, Tzourio C, Auriacombe S, Amieva H, Dufouil C, Alperovitch A, et al. Usefulness of data from magnetic resonance imaging to improve prediction of dementia: population based cohort study. Bmj. 2015;350:h2863. Epub 2015/06/24. pmid:26099688.

36. Jorm AF, Masaki KH, Petrovitch H, Ross GW, White LR. Cognitive deficits 3 to 6 years before dementia onset in a population sample: the Honolulu-Asia aging study. J Am Geriatr Soc. 2005;53(3):452–5. pmid:15743288

37. Tierney MC, Yao C, Kiss A, McDowell I. Neuropsychological tests accurately predict incident Alzheimer disease after 5 and 10 years. Neurology. 2005;64(11):1853–9. pmid:15955933

38. Mitnitski A, Skoog I, Song X, Waern M, Ostling S, Sundh V, et al. A vascular risk factor index in relation to mortality and incident dementia. Eur J Neurol. 2006;13(5):514–21. pmid:16722978

39. Jack CRJ, Albert MS, Knopman DS, McKhann GM, Sperling RA, Carrillo MC, et al. Introduction to the recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease. Alzheimers Dement. 2011;7(3):257–62. pmid:21514247

40. Dubois B, Feldman HH, Jacova C, Hampel H, Molinuevo JL, Blennow K, et al. Advancing research diagnostic criteria for Alzheimer's disease: the IWG-2 criteria. Lancet Neurol. 2014;13(6):614–29. pmid:24849862