Critical Appraisal of the Health Research Literature Prevalence or Incidence of a Health Problem

Headnote

Abstract

This article identifies and discusses criteria that can be used by health professionals to critically appraise research articles that estimate the prevalence or incidence of a disease or health problem. These guidelines will help determine the validity and usefulness of such community assessment studies. The criteria relate to the validity of the study methods (design, sampling frame, sample size, outcome measures, measurement and response rate), interpretation of the results and applicability of the findings. The research question "What is the prevalence of dementia in Canada?" is used as an example for this paper.

Key words: critical appraisal; dementia; prevalence

Introduction

The published health research literature often contains valuable information that provides the necessary scientific evidence to help direct health policy decisions. Critical appraisal of this literature is important to identify and separate useful from useless information. We found no formalized guidelines for the critical appraisal of research articles that estimate the prevalence and/or incidence of a health problem. The purpose of this paper is to identify and discuss criteria that can be used to assess the methodological strengths, results and relevance of articles on prevalence or incidence of a health problem. We use a health research question regarding the prevalence of dementia in Canada to help clarify these guidelines with examples.

Prevalence or Incidence

Prevalence refers to the proportion of a defined population at risk that has a defined health problem at a particular point in time (point prevalence) or during a period of time (period prevalence).1 It is usually measured by surveying a particular population containing people with and without the condition of interest,2 and prevalence rates are usually reported as percentages. For chronic disease conditions that do not resolve, individuals cannot become classified as prevalent cases again. This differs from individuals with acute conditions that resolve completely and can occur again (e.g. low back pain).

By comparison, incidence refers to the frequency or number of new occurrences of a health problem (clinical condition) in a population of susceptible individuals who were initially free ofthat condition before the time period being examined.1'3 In summary, incident cases are new cases while prevalent cases are the total number of existing cases, old and new, for the time period studied.

Dementia

Dementing disorders are common among the elderly, especially the very elderly.4 The onset is gradual, and determining the point that distinguishes disease from normality can be difficult.5 Dementia is characterized by a profound influence on cognitive functioning in addition to the behaviour and emotional state of the person affected.6 Screening instruments such as the Mini-Mental State Examination (MMSE) are used for screening cognitive decline. Diagnostic criteria such as the DSM M-R8 assist in diagnosing dementia. The terms used in the DSM IH-R criteria to indicate the severity of dementia are mild, moderate and severe.

Materials and Methods

Literature Search

The search strategy used for this paper included a computerized search, personal files and references from articles. We searched the MEDLEsIE CD-ROM from 1992 to 1996, using the MeSH headings "dementia," "Alzheimer's disease" and "prevalence." The search was limited to studies printed in English and to study subjects aged 65 and over. Nine articles were identified, originating from different countries that reported prevalence rates of dementia for people in the 65+ age group.4'6'9'10-15 According to these articles, prevalence rates for dementia varied from 2% in Taiwan to 4% in the United States, 5% from two studies in Spain, 6% in Japan, 8% in Canada and Italy, and 9% in Belgium and Holland.

Critical Appraisal Background

Several published review articles16-19 have identified criteria indicating the validity of research articles determining prevalence. Jorm et al.(1987)16 identified that the design of the study, characteristics of the sample and methods of assessment and diagnosis were important features. Rockwood and Stadnyk (1994)1 outlined specific criteria that included the study's purpose, design, sampling frame or method of subject recruitment, sample size, population demographics, location and reporting of the prevalence related to subgroups. In the review by Forbes and Barham (199 1),19 adequate response rates were emphasized along with the concern of trying to identify mild cases of dementia. Corrada et al. (1995)17 reported that they coded studies according to features related to whether studies included mild cases, institutionalized persons and appropriate diagnostic and screening measures.

We incorporated many of the above criteria in our development of critical appraisal criteria for prevalence or incidence studies.

Critical Appraisal Guidelines and Scoring

We developed three broad organizing questions to structure the critique of articles determining the prevalence or incidence of a health problem or clinical issue, as illustrated in Table 1.

For our specific critique example, two authors appraised the selected articles on the prevalence of dementia, using consensus, based on the guidelines outlined in Table 1 . They used a methodological scoring system20 previously used to rate the quality of studies on the prevalence of back pain (Table 2). To determine the relative weighting of each item for the scoring system, three authors were asked to recommend weights for each item and all agreed that each item should be weighted equally. Thus each item was assigned a score of 1 point, making 8 the maximum score possible.

Critical Appraisal of Studies

A. ARE THE STUDY METHODS VALID?

1. Study Design and Sampling Method: Are the study design and sampling method appropriate for the research question?

A survey (observational study) is the appropriate study design to determine the prevalence of a particular health problem. If the whole population of interest is not surveyed, then the best sampling technique is random (probability) sampling of persons from a defined subset of the population. Stratification (sampling purposely from subgroups) may be required to appropriately represent subgroups such as the very old. Stratified random sampling, with eligibility criteria, will ensure that the sample is representative of the population to whom the researchers wish to generalize the results.

For larger surveys, cluster sampling is sometimes used. In cluster sampling, groups of individuals (e.g. families or people living in defined geographical areas) are selected as the survey units.3 If the population is small, some studies survey the whole population and do not generalize the results to other populations.

A study to determine the incidence of a disease must have a prospective or longitudinal design, and should include persons known not to have the disease, who are then observed over a suitable time period.1'3

As an example, one of the studies reviewed in our critical appraisal of studies on the prevalence of dementia in Canada was the Canadian Study of Health and Aging (CSHA).15 This was a survey that covered five regions across Canada through a random sample of 10,263 persons in community and institutional settings to determine the prevalence of dementia and Alzheimer's disease. The CSHA used random sampling, stratified for age, sex, region and place of residence (community or institution).

2. Sampling Frame: Is the sampling frame appropriate?

The type of sampling frame (list for study recruitment) from which subjects are selected is important. Census data provide one of the few data sets from which one can draw a sample that is thought to have minimal bias since certain groups of persons are thought not to be excluded as they might be in an electoral list or telephone list. For example, electoral lists may underrepresent the elderly or people who are cognitively impaired. In relation to our critical appraisal of dementia prevalence studies, a sample of "convenience" could be very biased in that persons with dementia were hard to reach, thus reducing the prevalence of dementia in the sample. Studies of whole, narrowly defined communities are usually done as doorto-door surveys, but this limits the generalizability of the findings outside that community.

In the Canadian Study of Health and Aging,15 the following databases were used for sample selection: provincial health insurance plans, enumeration composite records, election records and municipal records. The study included both institutionalized persons and community dwellers. In the one province (Ontario) where election records were used, the sample may have been biased if the extreme elderly were missed.

Bachman et al. (1992) 10 used the Framingham cohort in the United States, which limits the generalizability of these results to a particular group of subjects. Some electoral or physician utilization lists might not represent all elderly persons (i.e. both those in the community and those in institutions, and healthy and unhealthy seniors), and thus might underrepresent persons with dementia.

3. Sample Size: Is the sample size adequate?

A large sample size produces narrow confidence limits, which is doubly important if the prevalence or incidence of a given condition is low. Small sample sizes produce large confidence intervals, making the findings less precise. It is critical to be as confident as possible that any changes in health care policy are based on results that did not occur by chance due to probability sampling inadequacy. In fact, the sample size required to estimate a proportion (prevalence of a disease) with a specified degree of precision (i.e. 95% confidence) can be calculated.21

Using a conservative sample size estimate of proportions for our review of dementia prevalence studies (assumptions based on CSHA study:15 prevalence = 8%, error rate <3%, 95% confidence level), the calculated sample size needed would be 3 14. ] In their study of dementia prevalence, Rockwood and Stadnyk (1994)18 indicated that the sample should be at least 300 subjects. Thus, a sample size of 300 was considered adequate for the purposes of our review. If prevalence rates were needed for subgroups, then the suggested sample size would be required for each subgroup.

The sample sizes used in the dementia prevalence studies we critiqued varied from 50013 to 10,263. 15 The largest sample, from the CSHA, produced smaller error rates and smaller confidence intervals, which is important when making a health care decision or policy.

4. Appropriate Measurement: Are objective, suitable and standard criteria used for measurement of the health outcome?

Often crude outcome measures are used in population health research due to the expense of complicated diagnostic tests. However, these measures may not be capable of including or excluding appropriate levels or stages of the health problem. It is important that published studies describe the measurements used well enough that the different outcome measures can be compared. If a worldwide standard measure of a particular health outcome exists, any studies not using it should indicate how their measure is related to the more common accepted measure.

The outcome measure must be reliable (reproducible) and valid with high sensitivity and specificity. Since health problems can be defined in many ways, the measurement of the problem must be the best possible one used by health care providers. If a disease is rare, there are often two phases to a prevalence study: subjects are first screened quickly for the condition using an inexpensive, broad screening test with good sensitivity and specificity, and then more complicated and detailed clinical assessments are made in the second phase. The screening test should not miss true positives - people who truly have the disease - and it should also have a low false negative rate, meaning it does not incorrectly label subjects with the disease as being disease-free.

For example, dementia is sometimes classified in research studies according to different systems from the United States (DSM), continental Europe (International Classification of Diseases) and the United Kingdom (CAMDEX: Cambridge Mental Disorders of the Ederly Examination).22 Research has indicated that these commonly used criteria can differ by a factor of 10 in the number of subjects classified as having dementia.22 In the CSHA,15 a variety of measures were used by independent assessors who were unaware of the initial screening test results. The community screening measure was the Modified Mini-Mental State Examination (3MS),7 which was given by trained interviewers in the subject's home. Subjects who screened positive (score <78) and a randomly selected group of subjects who screened negative were given clinical examinations by a nurse, a psychometrician (blind to 3MS testing), a neuropsychologist and a physician. These health professionals were trained and given guidance about how to assess for dementia. In addition, biological tests were carried out. Other assessments included the DSM III-R criteria8 and CAMDEX.23 In the American Framingham study,10 presence of dementia was determined through the MMSE,24 the CES-D25 (Center for Epidemiologic Studies Depression Scale) and general examinations by an independent neurologist.

Many health problems are not easily diagnosed or defined, and some, such as dementia, include stages where mild cases are not always easily distinguished.

5. Unbiased Measurement: Is the health outcome measured in an unbiased fashion?

Considerable judgment by assessors or interviewers is required to determine the presence of some health outcomes under scrutiny; thus, it is best that trained assessors are independent and not aware (i.e. blinded) of the subjects' clinical status or, sometimes, even the purpose of the study. It is important that the subjects under assessment include those thought to be negatives as well as positives.

If more than one rater is used, interobserver and/or intraobserver reliability of clinical assessments must be high and should be noted in the articles published. The interviewers or assessors must all be using the same criteria, including specifics related to each health problem, such as its duration. This is especially pertinent when diagnosing an illness such as Alzheimer's disease, since investigators must evaluate clinical signs and symptoms in the subjects in addition to caregivers' views of these. Sometimes, as for Alzheimer's disease, multiple measurements or assessments are conducted to rule out other health conditions. Thus the numerators (health problems) of the rates must all be defined or diagnosed in the same way.

6. Response Rate: Is the response rate adequate? Are the refusers described?

The greater the number of selected subjects who are not available for measurement, the less valid the estimate. A response rate in population surveys of two thirds to three quarters has been suggested to be generalizable to the population samples.26 Therefore, we chose a response rate of 70% as acceptable in our review. In the case of dementia, a significant proportion of those persons not responding to a survey might be suffering from dementia, which could lead to an underestimate of its prevalence.19

Since a large number of dropouts, refusals or "not founds" among the subjects selected may jeopardize a study's validity, the authors should describe the reasons for non-response and compare persons in the study with those not in the study as to their sociodemographic characteristics. If the reasons for non-response seem unrelated to the health outcome measured and the characteristics of those individuals not in the sample are comparable to those in the study, researchers may be able to justify a more modest response rate.

Response rates may be improved if the assessment or measurement is easily accessible, conveniently timed for the subjects, acceptable in length and suitable in content. Home visits may be more acceptable for many elderly persons.

In our review, prevalence rates of dementia differed as did study response rates. The CSHA15 accounted for all subjects, giving reasons for non-response. The compliance rate for the initial screening (phase 1) was 72%, and 73% of these respondents were compliant for clinical examination during the prevalence study. The CSHA authors considered these rates slightly low and thought that dementia prevalence might be underrepresented in the sample since 27% refused the clinical exam and their reasons for refusal might have included the presence of dementia.

To determine incidence ideally, all study subjects should be followed and measured to prevent bias. Usually patients are available for follow-up and if randomly selected subjects are not found or studied, one is never sure if there is a consistent bias known to influence incidence. If persons die during the period of the study, the cause of death must be ascertained. It is necessary to follow subjects over a clinically sensible period of time, depending on the illness under study and the age of the population. For dementia, if the follow-up period is too long, cases may be missed due to death, especially in the older subgroups.

B. WHAT IS THE INTERPRETATION OF THE RESULTS?

7. Results: Are the estimates of prevalence or incidence given with confidence intervals and in detail by subgroup, if appropriate?

The quantitative results from studies of prevalence or incidence are proportions or rates over a fixed period of time. The prevalence rates found in studies reviewed provide only estimates of the true prevalence of a problem in the larger population. Confidence intervals then indicate the level of confidence one can have in the estimates and their range. Since some subgroups are very small, usually 95% confidence intervals are given.

The CSHA authors15 provided confidence intervals and described prevalence rates in detail by age group, sex, setting (community or institution) and region of Canada. Their estimates of the prevalence of dementia ranged from 2.4% among persons aged 65-74 years, to 34.5% among those aged 85 and over.

C WHA T IS THE APPLICABILITY OF THE RESULTS?

8. Study Subjects: Are the study subjects and the setting described in detail and similar to those of interest to you?

Certain diseases are known to vary in prevalence or incidence across different geographic regions and population sectors. For example, persons over 85 years of age and those residing in institutions are expected to have higher prevalence rates of dementia. For some health problems, rates for women may differ from those for men.

Sociodemographic variables, such as educational status, may vary between countries. Therefore, the study sample needs to be described in enough detail that other researchers can determine if it is comparable to the population of interest to them.

In the CSHA article,15 study subjects are described in detail by age, sex and region of residence in Canada. Institutionalized subjects are also included in the sample.

If the study being appraised estimates the prevalence of a sign or symptom in an experimental group, such as a control group in a randomized controlled trial, the sociodemographic characteristics of the subjects must be reported in order to understand the applicability of the results. Similarly, providing a comparison of study participants with those who refused or were ineligible can help others determine for whom the study group is representative.

Conclusion

Table 3 summarizes the findings from our critical appraisal of studies on the prevalence of dementia according to the guidelines we developed. Prevalence rates from these nine studies varied from 2% to 9%, and the scores we assigned varied from 3 to 8, the maximum possible.

Some of the studies seemed inappropriate for determining the prevalence of dementia in Canada due to the lack of relevance of the subjects studied, while others had methodological weaknesses. For example, if the assessors did not have negatively screened subjects included in their sample, the results might be biased. In addition, some studies had low response rates, raters were not always blinded and the sampling frames were not always the best.

The study by the CHSA Working Group estimated the prevalence of dementia in Canada to be at least 8%. The sample was randomly selected, subjects in institutions were included and outcome measures were appropriate. The results included subgroup analysis, the sample size was large and the setting and subjects were applicable to our community situation. We found the weaknesses of the study to be a slightly low response rate and use of electoral lists in one province, suggesting an underreporting of cases of dementia. Our final rating of this study was a favourable score of 7.

Discussion

We developed and applied guidelines for the critical appraisal of published articles estimating the prevalence of dementia. These guidelines incorporated and organized many existing criteria indicating the validity of research articles on disease prevalence. We investigated the prevalence of dementia in Canada to help clarify the criteria.

The guidelines here can be used in the critical appraisal of published research concerning most health conditions to assess the "burden of illness,"27 prevalence or incidence. Once the true burden is known, based on methodologically sound research, health policy makers can use this information to aid in organizing and prioritizing community health care.

References

References

1. Jekel J, Elmore J, Katz D. Epidemiology, biostatistics and preventive medicine. Toronto: WB Saunders Co, 1996.

2. Fletcher R, Fletcher S, Wagner E. Clinical epidemiology, the essentials. 2nd ed. Baltimore: Williams and Wilkrns, 1998.

3. Streiner D, Norman G. PDQ epidemiology. 2nd ed. Toronto: Mosby, 1996.

4. Ott A, Breteler M, van Harskamp F, Claus J, van der Cammen T, Grobbee D, et al. Prevalence of Alzheimer's disease and vascular dementia: association with education. The Rotterdam Study. Br J Med 1995;310:970-3.

5. Hebert L, Scherr P, Bechert L, Albert M, Pilgrim D, Chown M, et al. Age-specific incidence of Alzheimer's disease in a community population. JAMA 1995;273(17):1354-9.

6. Roelands M, Wostyn P, Dom H, Baro F. The prevalence of dementia in Belgium: a population-based door to door survey in a rural community. Neuroepidemiology 1994;13:155-61.

7. Teng EL, Chui HC. The modified mini-mental state (3MS) examination. J Clin Psychiatry 1987;48:314-8.

8. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 3rd ed, rev. Washington: American Psychiatric Association, 1987.

9. Komashashi T, Ohmori K, Nakano T, Fijinuma H, Higashimoto T, et al. Epidemiological survey of dementia and depression among the aged living in the community in Japan. Japanese J Psychiatry Neurol 1994;48(3):517-26.

10. Bachman D, WoIfP, Linn R, Knoefel J, Cobb J, Belanger A, et al. Prevalence of dementia and probable senile dementia of the Alzheimer type in the Framingham Study. Neurology 1 992;42: 115-9.

11. Coria F, Gomez J, Minguez L, Rodriguez- Artalej o F, Claveria L. Prevalence of age-associated memory impairment and dementia in a rural community. J Neurol Neurosurg Psychiatry 1993;56:973-6.

12. Liu H, Lin K, Teng E, Wang S, Fuh J, Guo N, et al. Prevalence and subtypes of dementia in Taiwan: a community survey of 5297 individuals. J Am Geriatr Soc 1995;43:144-9.

13. Lobo A, Saz P, Marcos G, Dia J-L, De-la-Camara C. The prevalence of dementia and depression in the elderly community in a southern European population. Arch Gen Psychiatry 1995;52:497-506.

14. Prencipe M, Casini A, Ferretti C, Lattanzio M, Fiorelli M, Culasso F. Prevalence of dementia in an elderly rural population: effects of age, sex, and education. J Neurol Neurosurg Psychiatry 1 996;60:628-33.

15. The Canadian Study of Health and Aging Working Group. Canadian Study of Health and Aging: study methods and prevalence of dementia. Can Med Assoc J 1994;150(6):899-912.

16. Jorm A, Korten A, Henderson A. The prevalence of dementia: a quantitative integration of the literature. Acta Psychiatry Scand 1987;76:456-79.

17. Corrada M, Brookmeyer R, Kawas C. Sources of variability in prevalence rates of Alzheimer's disease. IntJ Epidemiol 1995;24(5):1000-5.

18. Rockwood K, Stadnyk K. The prevalence of dementia in the elderly: a review. Can J Psychiatry 1994;39(5):253-7.

19. Forbes W, Barham J. Concerning the prevalence of dementia. Can J Public Health 1991;82:185-7.

20. Loney P, Stratford P. Prevalence of low back pain in adults: a methodological review of the literature. Physical Therapy. In press 1999.

21. Katchigan S. Statistical analysis: an interdisciplinary introduction to univariate and multivariate methods. New York: Radius Press, 1986; 158-9.

22. Erkinjunti T, Ostbye T, Steenhuis R, Hachinski V. The effect of different diagnostic criteria on the prevalence of dementia. N Engl J Med 1997;337(23): 1667-74.

23. Roth M, Huppert FA, Tym E. CAMDEX: the Cambridge examination for mental disorders of the elderly. Cambridge (England): Cambridge University Press, 1988.

24. Folstein M, Felstone SE, McHugh PR. Mini-mental state: a practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 1975; 12: 189-98.

25. Radloff LS. The CES-D scale: a self-report depression scale for research in the general population. Applied Psychological Measurement 1 977; 1 :385-401.

26. Marshall V. Factors affecting response and completion rates in some Canadian studies. Can J Aging 1987;6:217-27.

27. Tugwell P, Bennett K, Sackett D, Haynes R The measurement iterative loop: a framework for the critical appraisal of needs, benefits and costs of health interventions. J Chronic Dis 1985;38(4):339-51.