Validation of a modified Berger HIV stigma scale for use among patients with hepatitis C virus (HCV) infection

Abstract

Background

Stigma around hepatitis C virus (HCV) infection is an important and understudied barrier to HCV prevention, treatment, and elimination. To date, no validated instrument exists to measure patients’ experiences of HCV stigma. This study aimed to revise the Berger (2001) HIV stigma scale and evaluate its psychometric properties among patients with HCV infection.

Methods

The Berger HIV stigma scale was revised to ask about HCV and administered to patients with HCV (n = 270) in Philadelphia, Pennsylvania. Scale reliability was evaluated as internal consistency by calculating Cronbach’s alpha. Exploratory factor analysis was performed to evaluate construct validity by comparing item clustering to the Berger HIV stigma scale subscales. Item response theory was employed to further evaluate individual items and to calibrate items for simulated computer adaptive testing sessions in order to identify potential shortened instruments.

Results

The revised HCV Stigma Scale was found to have good reliability (α = 0.957). After excluding items for low loadings, the exploratory factor analysis indicated good construct validity with 85% of items loading on pre-defined factors. Analyses strongly suggested the predominance of an underlying unidimensional factor solution, which yielded a 33-item scale after items were removed for low loading and differential item functioning. Adaptive simulations indicated that the scale could be substantially shortened without detectable information loss.

Conclusions

The 33-item HCV Stigma Scale showed sufficient reliability and construct validity. We also conducted computer adaptive testing simulations and identified shortened six- and three-item scale alternatives that performed comparably to the original 40-item scale.

About the Authors:

M. Elle Saine

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Validation, Writing – original draft

* E-mail: [email protected]

Affiliations Center for Clinical Epidemiology and Biostatistics, Center for Pharmacoepidemiology Research and Training, Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America, Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA, United States of America

ORCID logo http://orcid.org/0000-0002-1037-5610

Tyler M. Moore

Roles Formal analysis, Methodology, Supervision, Validation, Writing – review & editing

Affiliation: Department of Psychiatry, Brain Behavior Laboratory, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America

Julia E. Szymczak

Roles Conceptualization, Investigation, Methodology, Supervision, Writing – review & editing

Affiliation: Center for Clinical Epidemiology and Biostatistics, Center for Pharmacoepidemiology Research and Training, Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America

ORCID logo http://orcid.org/0000-0002-3230-8670

Laura P. Bamford

Roles Project administration, Resources, Supervision, Writing – review & editing

Affiliations Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America, Philadelphia FIGHT Community Health Centers, Philadelphia, PA, United States of America

Frances K. Barg

Roles Conceptualization, Methodology, Supervision, Writing – review & editing

Affiliations Center for Clinical Epidemiology and Biostatistics, Center for Pharmacoepidemiology Research and Training, Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America, Department of Family Medicine and Community Health, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, United States of America

Nandita Mitra

Roles Conceptualization, Formal analysis, Methodology, Supervision, Validation, Writing – review & editing

Affiliations Center for Clinical Epidemiology and Biostatistics, Center for Pharmacoepidemiology Research and Training, Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America, Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA, United States of America

Jason Schnittker

Roles Conceptualization, Methodology, Resources, Supervision, Writing – review & editing

Affiliations Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA, United States of America, Department of Sociology, University of Pennsylvania, Philadelphia, PA, United States of America

John H. Holmes

Roles Conceptualization, Funding acquisition, Investigation, Methodology, Supervision, Validation, Writing – review & editing

Affiliation: Center for Clinical Epidemiology and Biostatistics, Center for Pharmacoepidemiology Research and Training, Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America

Vincent Lo Re III

Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Writing – review & editing

Affiliations Center for Clinical Epidemiology and Biostatistics, Center for Pharmacoepidemiology Research and Training, Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States of America

Introduction

Over 4.5 million people in the United States (US) have chronic hepatitis C virus (HCV) infection [1]. From 2010 to 2016, the incidence of HCV infection in the US increased 3.5-fold, driven largely by the opioid epidemic [2–4]. HCV-associated mortality now exceeds death from 60 other combined notifiable infectious diseases, including human immunodeficiency virus (HIV) infection [2]. Despite national action plans to eliminate HCV infection as a major public health threat by 2030 [5–7], fewer than half of HCV-infected persons in the US have been diagnosed or are aware of their infection [8], and treatment uptake remains <15% [9–13], undermining public health efforts.

Stigma is defined as a “deeply discrediting attribute” that differs from a normal attribute anticipated by society; functionally, stigma reduces “a whole and usual person to a tainted, discounted one” in the minds of dominant social groups [14]. Stigmatization is a social phenomenon that can be an immense burden for patients by impacting self-esteem, quality of life [15–17], and personal identity [18], ultimately impacting access to effective and equitable health care. Stigma can be psychologically taxing to patients, hindering them from seeking treatment, and inducing depression and/or self-isolation [19–21]. Clinical care that recognizes and addresses patient experiences of stigma can improve patient-provider communication, disease management, and health-related quality of life [6, 22–27].

HCV-related stigma has been identified as an important, but understudied, barrier to HCV treatment and elimination [8, 28–33]. Qualitative studies among patients with chronic HCV describe experiences of stigma in healthcare, such as poor patient-provider communication, insensitivity, and refusal of treatment [15, 17, 34]. However, to date, no validated scales exist to measure stigma among patients with HCV. Research that develops validated stigma measures and elucidates the nature of stigma is essential to patient-centered responses to public health. Improving understanding of HCV stigma can lead to interventions to increase rates of HCV diagnosis, linkage into care, and treatment. To address this knowledge gap, we revised the Berger HIV Stigma Scale (Berger-HSS) [35] to evaluate stigma associated with HCV infection. The Berger-HSS is a widely used measure of HIV stigma, which has been externally validated across HIV-infected patient populations within and outside of the US, [36–47] as well as adapted for use among non-HIV patient populations [48, 49]. We hypothesized that the revised HCV Stigma Scale would have good validity and reliability among patients with HCV infection due to similarities with HIV in modes of transmission and risk factors.

Methods

Instruments

The Berger-HSS [35] is a 40-item self-administered questionnaire using a four-point Likert scale with responses ranging from 1 (strongly disagree) to 4 (strongly agree). Two items (8 and 21) are reverse-scored. Scores are summed, with higher scores indicating greater experiences of stigma. Elements of the scale were adapted from the Rosenberg Self-Esteem Index [50], the Center for Epidemiologic Studies-Depression Scale [51], and the Multicenter AIDS Cohort Study/Coping and Change Study [52] to asses four factors of stigma: 1) personalized stigma; 2) fear of disclosure; 3) negative self-image; and, 4) concern about public attitudes. The Berger-HSS was developed and validated among 318 patients with HIV infection in the Midwestern US. The scale demonstrated excellent internal consistency (Cronbach α, 0.90–0.93) and test-retest correlation (0.89–0.92), and is widely considered the benchmark instrument for assessing HIV stigma among people living with HIV/AIDS [53], primarily due to its stable external validity across diverse populations of patients [37–39, 54, 55]. Modified/shortened versions have also been validated across HIV-infected populations [42, 44–47, 56, 57]. For this study, we revised the Berger-HSS to ask about HCV-related stigma (HCV-SS), by replacing “HIV” with “Hepatitis C” in all instructions and questions. No other revisions to the Berger-HSS were made.

From July 2018 to May 2019, HCV-SS instruments were administered on laptop computers equipped with headphones, using audio computer-assisted self-interview software (ACASI). ACASI has been used extensively in healthcare studies to improve data quality and minimize social desirability bias around sensitive topics and risk behaviors, including stigma, knowledge, and disease experience [37, 43, 58–73]. Each response set includes the options “I don’t know the answer” and “I don’t want to answer.” A free-text response box following the HCV-SS allowed participants to share any additional information that they felt was important or not collected by other survey questions.

Study design and setting

We conducted a cross-sectional study among patients presenting for care at Philadelphia outpatient clinics specializing in the treatment of HCV infection. From 2013–2016, the rate of HCV infection in Philadelphia was among the highest in the US, with over 410 cases per 100,000 people [2]. Participants were recruited from five clinics across two Philadelphia health systems: 1) University of Pennsylvania Health System (UPHS); and, 2) Philadelphia Field Initiation Group for HIV Trials (FIGHT) Community Health Centers.

UPHS is a nationally-ranked academic health system providing outpatient specialty HIV and viral hepatitis care. Patients were recruited from two UPHS clinics: 1) Center for Viral Hepatitis at Penn Presbyterian Medical Center (PPMC); and, 2) Hospital of the University of Pennsylvania MacGregor Infectious Diseases Clinic (HUP). The demographic characteristics of patients within the UPHS service area are largely representative of the Philadelphia patient population [74].

Philadelphia FIGHT is a Federally-Qualified Health Center providing comprehensive health services to individuals with low income, people living with HIV/AIDS, and those at high risk of contracting HIV/AIDS.[75] Patients were recruited from three FIGHT clinics: 1) Jonathan Lax Treatment Center (LAX); 2) John Bell Health Center (JBHC); and, 3) Clinica Bienestar. LAX and JBHC provide primary medical care to adult patients ≥18 years of age. Clinica Bienestar is a comprehensive HIV primary care clinic in partnership with and housed within Prevention Point Philadelphia, Philadelphia’s only syringe-service program. Clinica Bienestar providers specialize in the care of Puerto Rican patients and patients with active or previous injection drug use. The clinic has been recognized as an Innovative Practice by the Health Resources and Service Administration [76]. The demographics of the Philadelphia FIGHT patient population are largely representative of the larger population of chronic HCV-infected patients in Philadelphia [3, 76].

This study was approved by the Institutional Review Boards of both the University of Pennsylvania and Philadelphia FIGHT.

Participants and recruitment

Patients were eligible for study inclusion if they were identified by their provider to be: 1) ≥18 years of age at enrollment; 2) positive for HCV antibody; and, 3) English-speaking. The HCV antibody test criterion was chosen to include patients who had ever been diagnosed with HCV infection, including patients who have chronic HCV, spontaneously cleared the virus, or cured with antiviral treatment. All eligible patients were invited to participate until the desired sample size was reached.

Data collection

During survey administration, we also collected self-reported sex, gender identity, race, ethnicity, age, HIV coinfection status, and stage of HCV-management.

Statistical analysis

Baseline demographic and clinical characteristics were collected as categorical variables and presented descriptively as counts and proportions. Descriptive statistics were calculated overall and by HIV coinfection status. Covariate differences by HIV coinfection status were assessed using chi-square and Fisher’s exact tests, as appropriate.

Because the item responses are on an ordinal scale, we calculated the polychoric correlation matrix, setting negative eigenvalues to zero to obtain the least-squares positive semidefinite approximation of the matrix. We inspected the correlation matrix to confirm that variables within a hypothesized factor were moderately correlated (≥0.3), and a Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was performed to ensure that the data were suitable for factor analysis. Consistent with Berger et al. [35], we performed exploratory factor analysis (EFA) to evaluate the latent structure of the items, allowing items to freely load on all factors [35]. We employed the iterated principal factor extraction method and inspected the pattern matrix for Heywood cases [77, 78].

We evaluated the number of factors using multiple approaches: 1) visual examination of the scree plot; 2) Kaiser Criterion of eigenvalues ≥ 1.0 [78]; 3) parallel analysis comparing the progressive eigenvalues of the dataset to eigenvalues from randomly generated data of the same dimensions in order to account for variance due to chance [77, 78]; 4) the Very Simple Structure Criterion, which compares the correlation matrix to a simplified model composed of the largest loadings for each variable [79]. While the Kaiser Criterion and parallel analysis extracted a four-factor solution, the Very Simple Structure Criterion suggested a single factor solution. Moreover, as shown in the scree plot (Fig 1), the first factor explains a substantial proportion of the variance, with a 1st/2nd eigenvalue ratio of 9.64, indicating that a single factor (unidimensional) solution may be appropriate. We therefore evaluated a four-factor solution and a one-factor solution.

[Figure omitted. See PDF.]

Fig 1. Parallel analysis scree plot of eigenvalues.

Scree plot of actual and simulated data for factor analysis (FA) and principal component (PC) analyses.

https://doi.org/10.1371/journal.pone.0228471.g001

Items were assigned to a factor if they loaded at ≥0.4 [78]. We assessed reliability by calculating the internal consistency, estimated by Cronbach α, for the scale and each subscale, with a sufficient Cronbach α ≥0.7 [80, 81]. We investigated the absence of floor and ceiling effects, defined as 15% or more of the respondents with the lowest or highest possible score, as an estimate of content validity.

The four-factor solution allows replication of subscale analyses by Berger to evaluate item clustering as a measure of construct validity [35]. Construct validity, defined as the extent to which items in a scale are consistent with theoretical hypotheses regarding the phenomenon being measured [80], was determined by ≥75% of items loading on the same factor as in the Berger-HSS [80]. Items in our analyses with two or more loadings at ≥0.4 were considered cross-loading items and were assigned to the highest loading factor. For cross-loading items in the Berger-HSS, if the HCV-SS item loaded on at least one of the predefined Berger factors, it was considered to support the construct validity of the scale. The explained common variance of the four-factor solution was calculated using minimum rank factor analysis.

In order to evaluate individual scale items, we chose to further analyze the single factor solution using item response theory (IRT). IRT assumes that the unidimensional latent trait under study exists on a continuum and allows investigation of how each item performs among participants at varying levels of the underlying trait [2]. A key benefit of IRT is that it allows evaluation of the amount of information provided by each individual scale items, thereby assessing the quality of each item [82]. Moreover, IRT allows evaluation of shortened scales using computer adaptive testing (CAT).

Items were calibrated using the IRT-graded response model [83], an extension of the traditional 2-parameter (2PL) model, to handle ordered polytomous categories, such as the Likert scale. Given an individual’s trait level (θ), the probability of a participant selecting item response k or higher for item i is dependent on two parameters: 1) discrimination (??i), can be thought of as a measure of an item’s precision in placing an individual on the trait spectrum; and, 2) difficulty (??i), is the probability that an individual at a specific point on the trait spectrum would endorse an item. The graded response model is defined by the following equation:Where for item I, ??i is the discrimination parameter and bik is the difficulty parameter for each response category k. The amount of information that an item provides increases as the item discrimination increases, and is maximized when the difficulty equals an individual’s trait level (when θ = b in the above equation) [2, 82–85].

Because the Berger-HSS was developed among patients with HIV infection, we conducted differential item functioning analyses using the lordif package [79] in R (R Core Team, 2016) in order to evaluate whether the probability of endorsement for any HCV-SS items differed between participants who were monoinfected with HCV compared to those coinfected with HIV/HCV. Differential item functioning analyses were conducted via ordinal logistic regression, with the graded response model used for trait estimation and the theta estimated as the conditioning variable. Items were flagged when differences in difficulty and discrimination parameters were detected between subgroups of the study population. Differential item functioning analyses allow the elimination of items that may provide misleading results, analogous to effect modification or confounding [86].

Using both a ratio of five participants for each measured item and the assumption of moderate conditions (communalities of 0.40–0.70 and ≥3 measured variables loading on each factor), a sample size of ≥200 participants was considered sufficient for exploratory factor analyses [78, 80]. For IRT analyses, Chang et al demonstrated that a sample size ≥250 participants can produce reasonable estimates for a 4 point Likert scale like ours, although ≥500 participants are required for accurate parameter estimates when using an IRT model with a five-point Likert scale [87–90]. There was less than 5% missingness on items; missing data were imputed using multiple imputation by chained equations. Data were analyzed using Stata 15.0 (Stata Corporation, College Station, TX) and the mirt [91], psych [79] and lordif [79] packages in R (R Core Team, 2016).

Shortening of HCV-SS

To optimize efficiency of the HCV-SS, we simulated computerized adaptive testing (CAT) to determine, 1) if a subset of items could be used in a shortened form, and if so, 2) which items an optimal subset would comprise. Briefly, CAT determines which item in the question bank will provide the most information about a participant’s trait level by roughly estimating the trait level based on the participant’s response to the administered item and then choosing a subsequent item that will provide the most additional information. The algorithm updates information on the participant as she/he responds to each question and stops when the participant’s standard error of measurement (SEM) reaches a predetermined lower bound. CAT minimizes test burden by omitting items that provide little information about a participant and retaining items that are the most informative [19, 82, 92].

Estimated item parameters from the one-factor graded response model were imputed into Firestar [93], a CAT simulation program. All simulations were performed using real data simulation. The first administered item was selected based on maximum information at the mean. The default settings were kept as maximum posterior weighted information for the item selection method and the interim theta estimator was expected a posteriori [82, 94, 95]. Two CAT simulations were run, first with the stopping rule set at a SEM≤0.3, and second with the stopping rule set at SEM≤0.4. SEM of 0.3 and 0.4 are equivalent to an alpha 0.91 and 0.84 respectively, using the classical test theory conversion of α = 1 –SEM2 [82]. The output of the simulation program indicates the average number of items administered across simulations and the distribution of item usage in order to identify shortened scales composed of the most useful items.

Results

Patient characteristics

We approached 288 patients with a history of a positive HCV antibody test, of whom 270 (96.43%) agreed to participate in the survey. 265 participants completed the survey, all of whom had sufficient data for analyses (<5% missing data). Participants were predominantly male (68.68%), 55 years of age or older (35.47%), and white (46.79%) or black/African American (39.25%). Most participants had been diagnosed with HCV infection more than one year prior to the survey (81.13%) and were either currently being treated (29.06%) or cured (41.51%) of chronic HCV infection. The total instrument took an average of 16 minutes to complete, with the stigma scale alone taking 10 minutes.

Slightly more than half of participants were coinfected with HIV (147, 55.47%). Compared to HCV-monoinfected participants, HCV/HIV-coinfected participants were older (P = 0.001), more frequently identified as black/African American (P<0.001), and had a higher rate of HCV treatment and cure (P<0.001; Table 1).

[Figure omitted. See PDF.]

Table 1. Baseline characteristics of participants, overall and by HIV infection status.

https://doi.org/10.1371/journal.pone.0228471.t001

Exploratory factor analyses

The data were found to be suitable for EFA with a KMO of 0.913. Floor and ceiling effects were absent.

For the four-factor solution, six items (5, 8, 11, 19, 21, 22) had loadings below the 0.40 cutoff used here and were removed from subsequent EFA analyses. Item clustering for the 34 (85%) items with loadings ≥0.40 matched the Berger-HSS four-factor EFA [35], providing evidence for the construct validity of the sub-scales among patients with HCV. Additionally, two items (26 and 37) cross-loaded on factors 1 and 2; both items had the highest loading on factor 2 and were therefore assigned to this factor (Table 2).

[Figure omitted. See PDF.]

Table 2. Exploratory factor analysis of four- and one-factor solutions and graded response model parameter estimates of one-factor solution.

Factor loadings presented include four-factor and one-factor solutions. Bolded loadings indicate agreement of factor loadings in four-factor solution with Berger Subscales*.

https://doi.org/10.1371/journal.pone.0228471.t002

The four-factor 34-item scale was determined to be reliable with a Cronbach α of 0.957. All four factors also had sufficient reliability with Cronbach α results ranging from 0.819–0.946. The inter-factor correlations (range, 0.371 to 0.599) indicate that, in addition to inter-item covariance being explained by individual factors, there exists a strong underlying factor (stigma) that explains a large portion of the covariance among all items (Table 3). The explained common variance of the four-factor solution was 59.0%.

[Figure omitted. See PDF.]

Table 3. Inter-factor correlations and reliability of four-factor solution*.

https://doi.org/10.1371/journal.pone.0228471.t003

Three items (1, 8, 21) had loadings <0.4 on the one-factor solution. Factor loadings are presented in Table 2. The one-factor model was determined to be reliable with a Cronbach alpha of 0.963 and explained 51.8% of the variance.

Item response theory analyses

The one-factor solution was further evaluated using item response theory. Mean discrimination for the total scale was 1.894 (range, 0.179 to 3.685). Difficulty parameters varied across the polytomous response categories, with the lowest threshold (b1) indicating the “easiest” (mean -1.489; range -10.527 to -0.109), difficulty progressing as level of stigma endorsement (b2) increased (mean 0.189; range -3.894 to -1.652), and the upper thresholds (b3) all positive (mean 2.146; range 0.842 to 5.785; Table 2). Five items (2, 8, 12, 22, 36) were flagged for differential item functioning between patients monoinfected with HCV vs coinfected with HCV/HIV. Items with low factor loadings and those flagged for differential item functioning were removed, resulting in a 33-item scale (S1 Appendix).

Shortened HCV-SS scales

Simulated CAT using item parameters estimated in the 33-item sample yielded a six-item scale (items 24, 28, 32, 33, 35, 39) when using a stopping criterion of SEM≤0.3 and a three-item scale (items 32, 33, 35) when using a stopping criterion of SEM≤0.4. Fig 2 shows the percent of total items used on the y-axis, which is calculated by taking the total number of times each item was administered, divided by the total number of items administered across all examinees; this adds up to 100% for all items. Specifically, item 33 provides enough information that it was always administered, regardless of the examinee’s level of stigma; similarly, items 28, 32, 35, and 39 were administered in most simulations.

[Figure omitted. See PDF.]

Fig 2.

Histogram of item usage for adaptive simulation with standard error of measurement ≤0.3 (a) and ≤0.4 (b).

https://doi.org/10.1371/journal.pone.0228471.g002

Both the six-item and three-item shortened scales demonstrated good internal consistency reliability (0.909 and 0.861, respectively). After correcting for item redundancy using Levy’s formula,[96] scores for the six-item and three-item shortened scales demonstrated good correlation (>80%) with scores from the 40-item scale (Table 4). Furthermore, the test characteristic curves for the short scales all had nearly identical shapes to the 40-item Berger HSS and 33-item full HCV-SS (Fig 3).

[Figure omitted. See PDF.]

Fig 3. Test characteristic curves for total scale and short forms.

Comparison of test characteristic curves (TCC) for full 40-item scale (a) with full 33-item HCV Stigma Scale (b), 6-item (c), and 3-item (f) scales.

https://doi.org/10.1371/journal.pone.0228471.g003

[Figure omitted. See PDF.]

Table 4. Item selection and correlation of 33-item scale and shortened 6-item and 3-item scales with Berger 40-Item Scale.

https://doi.org/10.1371/journal.pone.0228471.t004

Discussion

Stigma, along with other forms of social marginalization, can influence a patient’s decision to undergo testing, his/her adherence to treatment, and retention in health care globally [97]. Understanding how perceptions of stigma influence decision-making among HCV-infected patients is essential to understanding barriers to healthcare among this population and to planning interventions to address the public health threat of this disease. To our knowledge, this is the first study to validate a scale to measure perceptions of stigma among patients with HCV infection. Development of a validated questionnaire increases precision in understanding patients’ perspectives on HCV-related stigma and its impact on HCV diagnosis, retention in care, and antiviral treatment [18, 23].

In this study, we revised the Berger-HSS [35] to evaluate perceptions of HCV-related stigma among patients. The exploratory factor analysis supported the construct validity of the scale with 85% of items loading on one of four factors and item-clustering matching the Berger subscales for all items that sufficiently loaded on a factor. Furthermore, the Cronbach α results for the overall scale and subscales all exceeded 0.70, supporting internal consistency reliability.

Additionally, our analyses indicated that the common variance of the items is explained most by a single underlying factor (i.e., stigma). The large ratio of the 1st/2nd eigenvalues, strong inter-factor correlations, and very high alpha support the unidimensionality of the scale. These findings are consistent with the analyses by Berger et al [35], who found similar inter-factor correlations and the emergence of a single factor solution during scale development [35]. Additionally, we found that the single factor solution explained a comparable proportion of the variance, compared to the four-factor solution (51.8% vs 59.0%). The practical conclusion from these findings is that investigators should use caution in calculating HCV-SS subscale scores as the highly correlated domains measure similar aspects of patients’ experience; the global score of the 33-item one-factor scale is arguably more interpretable.

The unidimensionality of the scale permitted IRT analysis of individual items’ discriminative ability and utility across a wide range of trait levels (item difficulty). The IRT analyses demonstrated that most items had moderate to very high discrimination. Moreover, all but five items showed no measurement invariance between HCV-monoinfected participants and HCV/HIV-coinfected participants, supporting the revision of the Berger HIV-SS for use among patients with HCV infection. IRT-based CAT simulation allowed shortening of the scale while maintaining structural validity and reliability. We found that a few questions accounted for the majority of information provided by the HCV-SS, while several items provided no information across ability levels. The CAT simulation demonstrated low measurement error (SEM<0.3) after only 6 items and acceptable measurement error (SEM<0.4) after only 3 items. The use of polytomous items to measure a narrow construct allowed substantial shortening of the scale while maintaining acceptable test information. These findings support the use of these shorter assessments as practical alternatives to the overall scale, which can reduce survey burden while preserving validity.

This study had several strengths. To our knowledge, this is the first study to validate a scale to specifically measure perceptions of stigma among patients with HCV. By using ACASI software across several sites, including within a syringe-service program, we were able to include participants with low literacy and active drug use, increasing the generalizability of our study. Third, by employing IRT in addition to EFA, we were able to evaluate the psychometric properties of individual items and develop alternative shortened scales.

This study also had several potential limitations. First, while the CAT simulations demonstrate promise to substantially shorten the scale, we did not administer this shortened scale to study participants. To assess its reliability and validity more directly, we plan to administer the shortened scale in a separate sample in future research. Secondly, the sample size used here (N = 265) is relatively small for IRT analyses; however, the large number of items and item fit statistics indicate a stable model. Third, the results here indicate that this scale is measuring a narrow construct, and therefore one could argue that there are aspects of patients’ experiences of HCV stigma that are not being captured by the items. Qualitative research among patients with HCV may inform additional questions to better asses the breadth of patient experiences with HCV stigma. Fourth, since our study is the first to validate a measure of HCV-related stigma, assessment of concurrent validity was not possible due to the lack of availability of other validated HCV stigma instruments. Finally, since participants in our study were recruited from a mainly urban setting, the results might not be generalizable to individuals who reside in non-urban areas. Future analyses should examine the validity of the HCV-SS in those settings.

In conclusion, we found that the revised Berger-HSS has sufficient validity and reliability to measure HCV-related stigma. Validation of this measure has potential utility in evaluating interventions to reduce HCV-related stigma, as well as practical implications for future research on stigma as a barrier to healthcare.

Supporting information

[Figure omitted. See PDF.]

S1 Appendix. Hepatitis C Virus (HCV) Stigma Scale.

33-item Hepatitis C virus Stigma Scale (HCV-SS), administered on audio-computer assisted self-interview software. All items included response options “I don’t know the answer” and “I don’t want to answer.”

https://doi.org/10.1371/journal.pone.0228471.s001

(DOC)

Acknowledgments

The authors would like to thank the providers and staff at the participating clinical sites for their support of this study.

Citation: Saine ME, Moore TM, Szymczak JE, Bamford LP, Barg FK, Mitra N, et al. (2020) Validation of a modified Berger HIV stigma scale for use among patients with hepatitis C virus (HCV) infection. PLoS ONE 15(2): e0228471. https://doi.org/10.1371/journal.pone.0228471

References

1. Edlin BR, Eckhardt BJ, Shu MA, Holmberg SD, Swan T. Toward a more accurate estimate of the prevalence of hepatitis C in the United States. Hepatology (Baltimore, Md). 2015;62(5):1353–63. Epub 2015/07/15. pmid:26171595; PubMed Central PMCID: PMC4751870.

2. Raykov T. Introduction to psychometric theory. 2011. alma9977509724103681.

3. Philadelphia Department of Public Health. 2017 Division of Disease Control (DDC) Annual Report. Annual Report. Philadelphia,: 2017.

4. Moore TM. Iteration of Target Matrices in Exploratory Factor Analysis. In: UCLA, editor. ProQuest ID: Moore_ucla_0031D_11064. Merritt ID: ark:/13030/m5vx1wjr. Retrieved from https://escholarship.org/uc/item/2sk7w2hr2013.

5. National Academies of Sciences E, and Medicine. A National Strategy for the Elimination of Hepatitis B and C: Phase Two Report. Washington, DC: The National Academies Press, 2017.

6. Grebely J, Dore GJ, Morin S, Rockstroh JK, Klein MB. Elimination of HCV as a public health concern among people who inject drugs by 2030—What will it take to get there? J Int AIDS Soc. 2017;20(1):22146. Epub 2017/08/07. pmid:28782335; PubMed Central PMCID: PMC5577699.

7. World Health Organization. Combating Hepatitis B and C to Reach Elimination by 2030: Advocacy Brief. MAy 2016.

8. National Academies of Sciences E, and Medicine,. Eliminating the Public Health Problem of Hepatitis B and C in the United States. Washington, DC: The National Academies Press; 2016.

9. Gowda C, Lott S, Grigorian M, Carbonari DM, Saine ME, Trooskin S, et al. Absolute Insurer Denial of Direct-Acting Antiviral Therapy for Hepatitis C: A National Specialty Pharmacy Cohort Study. Open Forum Infectious Diseases. 2018;5(6). Epub 2018/06/01. pmid:29977955

10. Lo Re V, Gowda C, Urick PN, Halladay JT, Binkley A, Carbonari DM, et al. Disparities in Absolute Denial of Modern Hepatitis C Therapy by Type of Insurance. Clinical gastroenterology and hepatology: the official clinical practice journal of the American Gastroenterological Association. 2016;14(7):1035–43. PMC4912853. pmid:27062903

11. Roberson JL, Lagasca AM, Kan VL. Comparison of the Hepatitis C Continua of Care Between Hepatitis C Virus/HIV Coinfected and Hepatitis C Virus Mono-Infected Patients in Two Treatment Eras During 2008–2015. AIDS research and human retroviruses. 2018;34(2):148–55. Epub 2017/10/05. pmid:28974107.

12. Collins LF, Chan A, Zheng J, Chow S-C, Wilder JM, Muir AJ, et al. Direct-Acting Antivirals Improve Access to Care and Cure for Patients With HIV and Chronic HCV Infection. Open Forum Infectious Diseases. 2018;5(1):ofx264. PMC5753271. pmid:29308413

13. Jemal A, Fedewa SA. Recent Hepatitis C Virus Testing Patterns Among Baby Boomers. American journal of preventive medicine. 2017;53(1):e31–e3. Epub 2017/03/13. pmid:28284746.

14. Goffman E. Stigma: Notes on the Management of Spoiled Identitiy. New York: Simon & Schuster, Inc.; 1963.

15. Zickmund S, Ho EY, Masuda M, Ippolito L, LaBrecque DR. “They Treated Me Like a Leper”: Stigmatization and the Quality of Life of Patients with Hepatitis C. Journal of General Internal Medicine. 2003;18(10):835–44. PMC1494932. pmid:14521647

16. Marinho RT, Barreira DP. Hepatitis C, stigma and cure. World J Gastroenterol. 2013;19(40):6703–9. Epub 2013/11/05. pmid:24187444; PubMed Central PMCID: PMC3812468.

17. Dowsett LE, Coward S, Lorenzetti DL, MacKean G, Clement F. Living with Hepatitis C Virus: A Systematic Review and Narrative Synthesis of Qualitative Literature. Can J Gastroenterol Hepatol. 2017;2017:3268650. Epub 2017/05/23. pmid:28529936; PubMed Central PMCID: PMC5424189.

18. Armstrong AR, Herrmann SE, Chassany O, Lalanne C, Da Silva MH, Galano E, et al. The International development of PROQOL-HCV: An instrument to assess the health-related quality of life of patients treated for Hepatitis C virus. BMC Infectious Diseases. 2016;16(1):443. PMC4994204. pmid:27553866

19. Moore TM, Scott JC, Reise SP, Port AM, Jackson CT, Ruparel K, et al. Development of an abbreviated form of the Penn Line Orientation Test using large samples and computerized adaptive test simulation. Psychological assessment. 2015;27(3):955–64. Epub 2015/03/31. pmid:25822834; PubMed Central PMCID: PMC4549167.

20. Sengupta S, Banks B, Jonas D, Miles MS, Smith GC. HIV interventions to reduce HIV/AIDS stigma: a systematic review. AIDS Behav. 2011;15(6):1075–87. Epub 2010/11/23. pmid:21088989; PubMed Central PMCID: PMC3128169.

21. Nyblade L. Measuring HIV stigma: Existing knowledge and gaps. Psychology, Health & Medicine. 2006;11(3):335–45. pmid:17130069.

22. Younossi Z, Henry L. Systematic review: patient-reported outcomes in chronic hepatitis C—the impact of liver disease and new treatment regimens. Alimentary Pharmacology & Therapeutics. 2015;41(6):497–520. pmid:25616122

23. Fontana RJ, Kronfol Z. The patient's perspective in hepatitis C. Hepatology (Baltimore, Md). 2004;39(4):903–5. pmid:15057891

24. Butt G, Paterson BL, McGuinness LK. Living with the stigma of hepatitis C. Western journal of nursing research. 2008;30(2):204–21; discussion 22–33. Epub 2007/07/17. pmid:17630381.

25. Golden J, Conroy RM, Marie O’Dwyer A, Golden D, Hardouin J-B. Illness-related stigma, mood and adjustment to illness in persons with hepatitis C. Social Science & Medicine. 2006;63(12):3188–98. https://doi.org/10.1016/j.socscimed.2006.08.005.

26. Zacks S, Beavers K, Theodore D, Dougherty K, Batey B, Shumaker J, et al. Social stigmatization and hepatitis C virus infection. J Clin Gastroenterol. 2006;40(3):220–4. Epub 2006/04/25. pmid:16633123.

27. Owen G. An 'elephant in the room'? Stigma and hepatitis C transmission among HIV-positive 'serosorting' gay men. Cult Health Sex. 2008;10(6):601–10. Epub 2008/07/24. pmid:18649198.

28. Centers for Disease Control and Prevention. Surveillance for Viral Hepatitis–United States, 2015. 2015.

29. Tohme RA, Holmberg SD. Is sexual contact a major mode of hepatitis C virus transmission? Hepatology (Baltimore, Md). 2010;52(4):1497–505. Epub 2010/07/17. pmid:20635398.

30. Parker R, Aggleton P. HIV and AIDS-related stigma and discrimination: a conceptual framework and implications for action. Social Science & Medicine. 2003;57(1):13–24. https://doi.org/10.1016/S0277-9536(02)00304-0.

31. Yehia BR, Schranz AJ, Umscheid CA, Lo Re V 3rd. The treatment cascade for chronic hepatitis C virus infection in the United States: a systematic review and meta-analysis. PLoS One. 2014;9(7):e101554. Epub 2014/07/06. pmid:24988388; PubMed Central PMCID: PMC4079454.

32. New Hepatitis C Infections Nearly Tripled over Five Years. [Internet]. Atlanta, GA; 2017. Deadly virus concentrated among baby boomers and increasing rapidly among new generations of Americans [cited 21 June 2018]. Available from: https://www.cdc.gov/nchhstp/newsroom/2017/Hepatitis-Surveillance-Press-Release.html

33. Paterson BL, Backmund M, Hirsch G, Yim C. The depiction of stigmatization in research about hepatitis C. International Journal of Drug Policy. 2007;18(5):364–73. pmid:17854724

34. Madden A, Hopwood M, Neale J, Treloar C. Beyond interferon side effects: What residual barriers exist to DAA hepatitis C treatment for people who inject drugs? PLOS ONE. 2018;13(11):e0207226. pmid:30500863

35. Berger BE, Ferrans CE, Lashley FR. Measuring stigma in people with HIV: psychometric assessment of the HIV stigma scale. Res Nurs Health. 2001;24(6):518–29. Epub 2001/12/18. pmid:11746080.

36. Boyes ME, Mason SJ, Cluver LD. Validation of a brief stigma-by-association scale for use with HIV/AIDS-affected youth in South Africa. AIDS Care. 2013;25(2):215–22. Epub 2012/07/11. pmid:22774842.

37. Clum G, Chung SE, Ellen JM. Mediators of HIV-related stigma and risk behavior in HIV infected young women. AIDS care. 2009;21(11):1455–62. Epub 2009/12/22. pmid:20024724; PubMed Central PMCID: PMC2860279.

38. Dowshen N, Binns HJ, Garofalo R. Experiences of HIV-related stigma among young men who have sex with men. AIDS patient care and STDs. 2009;23(5):371–6. Epub 2009/03/27. pmid:19320600; PubMed Central PMCID: PMC2856436.

39. Fongkaew W, Viseskul N, Suksatit B, Settheekul S, Chontawan R, Grimes RM, et al. Verifying quantitative stigma and medication adherence scales using qualitative methods among Thai youth living with HIV/AIDS. J Int Assoc Provid AIDS Care. 2014;13(1):69–77. Epub 2013/01/17. pmid:23321804.

40. Lindberg MH, Wettergren L, Wiklander M, Svedhem-Johansson V, Eriksson LE. Psychometric evaluation of the HIV stigma scale in a Swedish context. PLoS One. 2014;9. pmid:25522127

41. Jeyaseelan L, Kumar S, Mohanraj R, Rebekah G, Rao D, Manhart LE. Assessing HIV/AIDS Stigma in South India: Validation and Abridgement of the Berger HIV Stigma Scale. AIDS Behav. 2013;17(1):434–43. PMC3404245. pmid:22246514

42. Reinius M, Wettergren L, Wiklander M, Svedhem V, Ekström AM, Eriksson LE. Development of a 12-item short version of the HIV stigma scale. Health and Quality of Life Outcomes. 2017;15(1):115. pmid:28558805

43. Mason S, Berger B, Ferrans CE, Sultzman V, Fendrich M. Developing a Measure of Stigma by Association With African American Adolescents Whose Mothers Have HIV. Research on Social Work Practice. 2009;20(1):65–73.

44. Murphy DA, Austin EL, Greenwell L. Correlates of HIV-related stigma among HIV-positive mothers and their uninfected adolescent children. Women Health. 2006;44(3):19–42. Epub 2007/01/27. pmid:17255064.

45. Rongkavilit C, Wright K, Chen X, Naar-King S, Chuenyam T, Phanuphak P. HIV stigma, disclosure and psychosocial distress among Thai youth living with HIV. Int J STD AIDS. 2010;21. pmid:20089999

46. Wright K, Naar-King S, Lam P, Templin T, Frey M. Stigma scale revised: reliability and validity of a brief measure of stigma for HIV+ youth. J Adolesc Health. 2007;40. pmid:17185215

47. Wiklander M, Rydstrom LL, Ygge BM, Naver L, Wettergren L, Eriksson LE. Psychometric properties of a short version of the HIV stigma scale, adapted for children with HIV infection. Health Qual Life Outcomes. 2013;11:195. Epub 2013/11/15. pmid:24225077; PubMed Central PMCID: PMC3842678.

48. Overholt L, Wohl DA, Fischer WA II, Westreich D, Tozay S, Reeves E, et al. Stigma and Ebola survivorship in Liberia: Results from a longitudinal cohort study. PLOS ONE. 2018;13(11):e0206595. pmid:30485311

49. Dadun , Peters RMH, Van Brakel WH, Lusli M, Damayanti R, Bunders JFG, et al. Cultural validation of a new instrument to measure leprosy-related stigma: the SARI Stigma Scale. Lepr Rev. 2017;88(1):23–42. Epub 2017/03/01. pmid:30188086.

50. Rosenberg M. Society and the Adolescent Self-Image. Princeton, NJ: Princeton University Press; 1965.

51. Radloff LS. The CES-D Scale:A Self-Report Depression Scale for Research in the General Population. Applied Psychological Measurement. 1977;1(3):385–401.

52. O'Brien K, Wortman CB, Kessler RC, Joseph JG. Social relationships of men at risk for AIDS. Social Science & Medicine. 1993;36(9):1161–7. https://doi.org/10.1016/0277-9536(93)90236-W.

53. Rao D, Pryor JB, Gaddist BW, Mayer R. Stigma, Secrecy, and Discrimination: Ethnic/Racial Differences in the Concerns of People Living with HIV/AIDS. AIDS and Behavior. 2008;12(2):265–71. pmid:17588146

54. Emlet C. Experiences of Stigma in Older Adults Living with HIV/AIDS: a mixed-methods analysis2007. 740–52 p.

55. Cluver L, Orkin M. Cumulative risk and AIDS-orphanhood: interactions of stigma, bullying and poverty on child mental health in South Africa. Social science & medicine (1982). 2009;69(8):1186–93. Epub 2009/08/29. pmid:19713022.

56. Cluver LD, Gardner F, Operario D. Effects of stigma on the mental health of adolescents orphaned by AIDS. J Adolesc Health. 2008;42(4):410–7. Epub 2008/03/19. pmid:18346667.

57. Fair C, Brackett B. “I Don't Want to Sit by You”: A Preliminary Study of Experiences and Consequences of Stigma and Discrimination from HIV-positive Mothers and Their Children. Journal of HIV/AIDS Prevention in Children & Youth. 2008;9(2):219–42.

58. Rothman KJ, Greenland S, Lash TL. Modern epidemiology. 3rd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams & Wilkins; 2008. x, 758 p. p.

59. Kim J, Dubowitz H, Hudson-Martin E, Lane W. Comparison of 3 data collection methods for gathering sensitive and less sensitive information. Ambulatory pediatrics: the official journal of the Ambulatory Pediatric Association. 2008;8(4):255–60. Epub 2008/07/23. pmid:18644548.

60. Wu Y, Newfield SA. Comparing data collected by computerized and written surveys for adolescence health research. The Journal of school health. 2007;77(1):23–8. Epub 2007/01/11. pmid:17212756.

61. Ghanem KG, Hutton HE, Zenilman JM, Zimba R, Erbelding EJ. Audio computer assisted self interview and face to face interview modes in assessing response bias among STD clinic patients. Sexually transmitted infections. 2005;81(5):421–5. Epub 2005/10/04. pmid:16199744; PubMed Central PMCID: PMC1745029.

62. Estes LJ, Lloyd LE, Teti M, Raja S, Bowleg L, Allgood KL, et al. Perceptions of audio computer-assisted self-interviewing (ACASI) among women in an HIV-positive prevention program. PLoS One. 2010;5(2):e9149. Epub 2010/02/18. pmid:20161771; PubMed Central PMCID: PMC2818842.

63. Metzger DS, Koblin B, Turner C, Navaline H, Valenti F, Holte S, et al. Randomized controlled trial of audio computer-assisted self-interviewing: utility and acceptability in longitudinal studies. HIVNET Vaccine Preparedness Study Protocol Team. American journal of epidemiology. 2000;152(2):99–106. Epub 2000/07/26. pmid:10909945.

64. Waruru AK, Nduati R, Tylleskär T. Audio computer-assisted self-interviewing (ACASI) may avert socially desirable responses about infant feeding in the context of HIV. BMC Medical Informatics and Decision Making. 2005;5:24–. PMC1190182. pmid:16076385

65. Vanable PA, Carey MP, Blair DC, Littlewood RA. Impact of HIV-Related Stigma on Health Behaviors and Psychological Adjustment Among HIV-Positive Men and Women. AIDS Behav. 2006;10(5):473–82. PMC2566551. pmid:16604295

66. Okumu E, Jolly DH, Alston LM, Eley NT, Laws M, MacQueen KM. Relationship between Human Immunodeficiency Virus (HIV) Knowledge, HIV-Related Stigma, and HIV Testing among Young Black Adults in a Southeastern City. Frontiers in Public Health. 2017;5(47). pmid:28349049

67. Kerr JC, Valois RF, DiClemente RJ, Carey MP, Stanton B, Romer D, et al. The effects of a mass media HIV-risk reduction strategy on HIV-related stigma and knowledge among African American adolescents. AIDS patient care and STDs. 2015;29(3):150–6. pmid:25738952

68. Swenson RR, Rizzo CJ, Brown LK, Vanable PA, Carey MP, Valois RF, et al. HIV knowledge and its contribution to sexual health behaviors of low-income African American adolescents. Journal of the National Medical Association. 2010;102(12):1173–82. Epub 2011/02/04. pmid:21287898; PubMed Central PMCID: PMC3095017.

69. Strauss SM, Astone-Twerell J, Munoz-Plaza CE, Des Jarlais DC, Gwadz M, Hagan H, et al. Drug treatment program patients' hepatitis C virus (HCV) education needs and their use of available HCV education services. BMC health services research. 2007;7(1):39. pmid:17346346

70. Rashrash ME, Maneno MK, Wutoh AK, Ettienne EB, Daftary MN. An evaluation of hepatitis C knowledge and correlations with health belief model constructs among African American “baby boomers”. Journal of Infection and Public Health. 2016;9(4):436–42. pmid:26706773

71. MacQueen KM, Chen M, Jolly D, Mueller MP, Okumu E, Eley NT, et al. HIV testing experience and risk behavior among sexually active Black young adults: a CBPR-based study using respondent-driven sampling in Durham, North Carolina. American journal of community psychology. 2015;55(0):433–43. PMC4439184. pmid:25893817

72. Garfein RS, Golub ET, Greenberg AE, Hagan H, Hanson DL, Hudson SM, et al. A peer-education intervention to reduce injection risk behaviors for HIV and hepatitis C virus infection in young injection drug users. Aids. 2007;21(14):1923–32. Epub 2007/08/28. pmid:17721100.

73. Group NMHSPTfAAC. Designing an Audio Computer-Assisted Self-Interview System in a Multisite Trial: A Brief Report. Journal of acquired immune deficiency syndromes (1999). 2008;49(Suppl 1):S52–S8. PMC2834963. pmid:18724191

74. University of Pennsylvania Health System (UPHS). 2016 Community Health Needs Assessment. Philadelpha, Pa; 2016.

75. Philadelphia FIGHT Community Health Centers. Annual Report 2017. Philadelphia, PA: Philadelphia FIGHT Community Health Centers, 2017.

76. Philadelphia Fight. Philadelphia FIGHT Community Health Centers Annual Report 2018. Annual Report. Philadelphia: Philadelphia FIGHT, 2018.

77. Fabrigar LR, Wegener DT. Exploratory Factor Analysis: OUP USA; 2012.

78. Habing B. Exploratory Factor Analysis. University of South Carolina; 2003.

79. Choi SW, Gibbons, L. E., Crane, P. K. lordif: Logistic Ordinal Regression Differential Item Functioning using IRT 0.3–3 ed. https://cran.r-project.org/web/packages/lordif/lordif.pdf2016. p. Analysis of Differential Item Functioning (DIF) for dichotomous and polytomous items using an iterative hybrid of ordinal logistic regression and item response theory (IRT).

80. Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42. Epub 2006/12/13. pmid:17161752.

81. Sikorskii A, Noble PC. Statistical Considerations in the Psychometric Validation of Outcome Measures. Clinical Orthopaedics and Related Research. 2013;471(11):3489–95. PMC3792291. pmid:23645337

82. Moore TM, Calkins ME, Reise SP, Gur RC, Gur RE. Development and public release of a computerized adaptive (CAT) version of the Schizotypal Personality Questionnaire. Psychiatry research. 2018;263:250–6. Epub 2018/04/08. pmid:29625786; PubMed Central PMCID: PMC5911247.

83. Samejima F. (1969). Estimation of Latent Ability Using a Response Pattern of Graded Scores (Psychometric Monograph No. 17). Richmond, VA: Psychometric Society. Retrieved from http://www.psychometrika.org/journal/online/MN17.pdf.

84. StataCorp. 2017. Stata 15 Base Reference Manual. College Station, TX: Stata Press.

85. Samejima F. Graded Response Models Handbook of Item Response Theory. Boca Raton, FL: CRC Press; 2016.

86. Crane P, Gibbons L, Jolley L, van Belle G. Differential item functioning analysis with ordinal logistic regression techniques: DIFdetect and difwithpar2006. S115–23 p.

87. Thorpe Geoffrey L. and Favia Andrej, "Data Analysis Using Item Response Theory Methodology: An Introduction to Selected Programs and Applications." (2012). Psychology Faculty Scholarship. 20. https://digitalcommons.library.umaine.edu/psy_facpub/20.

88. Reeve BB, Fayers P. Applying item response theory modeling for evaluating questionnaire item and scale properties. In: Fayers P, Hays RD, editors. Assessing quality of life in clinical trials: Methods of practice. 2nd ed. New York: Oxford University Press; 2005.

89. Medicare Payment Advisory Commission. Report to the Congress: New Approaches in Medicare. 2004. Accessed at: http://qio.ipro.org/wp-content/uploads/2012/12/june04_entire_report.pdf. Accessed on November 8, 2015.

90. Chang C-C, Lin C-Y, Gronholm PC, Wu T-H. Cross-Validation of Two Commonly Used Self-Stigma Measures, Taiwan Versions of the Internalized Stigma Mental Illness Scale and Self-Stigma Scale–Short, for People With Mental Illness. Assessment. 2016;25(6):777–92. pmid:27385391

91. Chalmers RP. mirt: A Multidimensional Item Response Theory Package for the R Environment. Journal of Statistical Software; 2012.

92. Roalf DR, Moore TM, Wolk DA, Arnold SE, Mechanic-Hamilton D, Rick J, et al. Defining and validating a short form Montreal Cognitive Assessment (s-MoCA) for use in neurodegenerative disease. Journal of neurology, neurosurgery, and psychiatry. 2016;87(12):1303–10. Epub 2016/04/14. pmid:27071646; PubMed Central PMCID: PMC5061594.

93. Choi SW, Gibbons LE, Crane PK. lordif: An R Package for Detecting Differential Item Functioning Using Iterative Hybrid Ordinal Logistic Regression/Item Response Theory and Monte Carlo Simulations. Journal of Statistical Software. 2011;39(8), 1–30. pmid:21572908

94. Bock RD, Mislevy RJ. Adaptive EAP estimation of ability in a microcomputer environment. Applied psychological measurement. 1982;6(4):431–44.

95. van der Linden WJ. Bayesian item selection criteria for adaptive testing. Psychometrika. 1998;63(2):201–16.

96. Levy P. The correction for spurious correlation in the evaluation of short-form tests. J Clin Psychol. 1967;23(1):84–6. Epub 1967/01/01. pmid:6031038.

97. Chambers LA, Rueda S, Baker DN, Wilson MG, Deutsch R, Raeifar E, et al. Stigma, HIV and health: a qualitative synthesis. BMC Public Health. 2015;15:848. Epub 2015/09/04. pmid:26334626; PubMed Central PMCID: PMC4557823.