Psychometric properties of the acceptance and action questionnaire (AAQ II) Malay version in cancer patients

Abstract

Psychological inflexibility has been found as one of the predictor to psychopathology in cancer patient. Cancer patients tend to experience psychological inflexibility as a reaction to cancer diagnosis, prognosis and treatment. Therefore, there is a need to identify psychological inflexibility due to its impact on quality of life among cancer patients.

Objectives

Acceptance and action questionnaire (AAQ II) is a scale used to assess psychological inflexibility. The aim of this study is to translate AAQ II into Malay language and evaluate the psychometric properties of AAQ II Malay version.

Methods

The AAQ II which has been translated into Malay language via back translation procedure was distributed to 101 cancer patients and 100 non-cancer patients. The evaluation of psychometric properties in this study included content validity index, internal consistency, parallel reliability, exploratory factor analysis, concurrent validity, sensitivity and specificity of AAQ II Malay version.

Results

AAQ II Malay version has established good content validity index, acceptable internal consistency with Cronbach’s alpha value of 0.91, excellent parallel reliability and adequate concurrent validity. Exploratory factor analysis (EFA) results demonstrated AAQ II Malay version is a unidimensional factor instrument. The result of sensitivity and specificity of AAQ II Malay version indicated cancer patients who scored more than 17.5 were having significant psychological inflexibility.

Conclusion

AAQ II Malay version is a reliable and valid instrument to measure psychological inflexibility among cancer patient in Malaysia.

About the Authors:

Nurul Izzah Shari

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation: Lifestyle Science Cluster, Advanced Medical and Dental Institute, Universiti Sains Malaysia, Kepala Batas, Malaysia

ORCID logo http://orcid.org/0000-0003-3343-2793

Nor Zuraida Zainal

Contributed equally to this work with: Nor Zuraida Zainal, Ng Chong Guan

Roles Conceptualization, Methodology, Supervision, Validation

Affiliation: Department of Psychological Medicine, Faculty of Medicine, University Malaya, Kuala Lumpur, Malaysia

Ng Chong Guan

Contributed equally to this work with: Nor Zuraida Zainal, Ng Chong Guan

Roles Conceptualization, Methodology, Supervision, Validation

Affiliation: Department of Psychological Medicine, Faculty of Medicine, University Malaya, Kuala Lumpur, Malaysia

Zuraida Ahmad Sabki

Roles Formal analysis, Methodology

¶‡ These authors also contributed equally to this work.

Affiliation: Department of Psychological Medicine, Faculty of Medicine, University Malaya, Kuala Lumpur, Malaysia

Nor Aziyan Yahaya

Roles Formal analysis, Methodology

¶‡ These authors also contributed equally to this work.

Affiliation: Department of Nursing Sciences, Faculty of Medicine, University Malaya, Kuala Lumpur, Malaysia

Introduction

Acceptance and action questionnaire (AAQ II) is an instrument to assess experiential avoidance and psychological inflexibility. Experiential avoidance can be defined as an attempt to avoid or neglect unpleasant thought, unpleasant feelings, bitter memories, uncomfortable physical sensations, and consequently lead to an action that is against one’s values and causing long-term harm [1]. Empirical evidence has found contribution of experiential avoidance on psychopathology in cancer patients [2,3]. Cancer patients commonly experienced experiential avoidance as a reaction to cancer diagnosis, prognosis and treatment [4]. Experiential avoidance is found to be one of the coping strategies among cancer patients and become a predictor to psychosocial problem such as depression, anxiety and distress [5,6]. The attempts to avoid the unpleasant thought, feeling and memories related to cancer elevate their discomfort and lead to the lack of involvement in their valued activities and life. As a result, this circumstance becomes the source of psychosocial and emotional issues such as distress, anxiety and depression [7]. Hence, it affects their quality of life [2,3], perceived health, psychosocial life [2] and career [8].

Meanwhile, psychological inflexibility refers to rigid psychological reaction against one's value in order to avoid distress, uncomfortable feeling and thought and tend to ignore the present moment [9]. Psychological inflexibility has been suggested as an important element in the etiology and the preservation of psychopathology [10,11]. Psychological inflexibility has positive correlation with symptoms burden that commonly experienced by cancer patient including pain, fatigue, sleep disturbance, cognitive concerns, anxiety, and depressive symptoms. The increase of psychological inflexibility among cancer patient will heighten their symptom burden and leads to the lack of engagement with valuable activities and relationship [12]. In contrast, psychological flexibility is the ability to fully experience present moment that includes one’s thoughts and feelings without struggling to control or change it, and the ability to either persist or change behavior in the given context that is consistent with one’s values and goals [10]. Individual who is psychologically flexible able to disengage with unworkable thought and actions, enjoy the present moment, accept the thing and circumstance that beyond control, and choose to live in held values and move closely to those values [13]. Psychological flexibility is important element in cancer survivorship particularly in prevent enhancement of psychological problems such as anxiety, depression and negative affect [14].

Previous studies demonstrated that AAQ II is the most widely used instrument to measure the effectiveness of Acceptance and Commitment Therapy (ACT) due to its high reliability, validity and sensitivity to identify changes in experiential avoidance [15–17]. ACT is a psychological intervention that specifically developed to manage experiential avoidance and encourage psychological flexibility.

Measurement of experiential avoidance

AAQ II is a revised version of original version of AAQ due to its limitations. The original version of AAQ consists of 16 items and 9 items (two versions). The usage of AAQ was proven useful to measure psychological inflexibility and to predict the quality of life with average effect size in the variety of psychopathology illnesses such as depression, anxiety, work performance, pain and etc. However, the original version of AAQ encountered two main weaknesses which included low to moderate internal consistency and unstable factor structure [9]. The original version has been revised and the stable factor with high internal consistency across variety of population namely AAQ II [9]. AAQ II has been acknowledged with its high internal consistency and reliability across different types of psychopathology illnesses and different target groups. Unlike AAQ, AAQ II consisted of 7 items and a stable unidimensional measure across different population and time. The unidimensional factor indicates that AAQ II is a specific measurement tool for experiential avoidance. The AAQ II also has been translated multiple languages namely Turkish [15], Chinese [16], French, Italian, Arabic and etc. There were few studies that examined the differences between clinical and nonclinical population [15,17,18]. There were inconsistent results of the scores’ differences between clinical and nonclinical sample. However most of the studies demonstrated significantly higher scores in clinical sample compared to non-clinical sample [15,18–20]. However there were few limitations of these studies that need to be considered such as these studies were using heterogeneous sample for clinical sample.

Overview of the present study

Even though AAQ II has been translated into multiple languages, but to the best of our knowledge, there is still no AAQ II Malay version that has been validated. Therefore, the overall aim of this study is to validate the AAQ II Malay version. The finding will provide evidence of the robustness of AAQ II and the generality of construct experiential avoidance in Asian population particularly in Malaysia which is a multi-racial and multicultural country. The first aim of this study is to examine the content validity index (CVI) of AAQ II Malay version. Content validity index (CVI) [21] is calculated to determine the content validity of the AAQ II Malay version. Content validity measured the comprehensiveness and representativeness of the content domain of an instrument [22]. Content validity is important because it is concerning the degree to which the content of the construct is adequately represented by the items in the instrument [23]. In the present study, content validity was conducted to obtain the agreement among subject matter experts for each item (I-CVI) in the instrument and for the whole instrument (S-CVI).

Then, the second aim is to explore internal consistency, to examine the parallel form reliability and to explore the potential factors in AAQ II Malay version. The parallel form reliability was measured by assessing the correlation between AAQ II English version and AAQ II Malay version. Factor analysis was conducted via exploratory factor analysis (EFA) using sample of cancer patients. The third aim is to explore the concurrent validity of AAQ II Malay version. The concurrent validity was measured by assessing the relationship of AAQ II Malay version with other instruments; The Hospital Anxiety and Depression Scale for Anxiety (HADS-A), The Hospital Anxiety and Depression Scale for Depression (HADS-D) and White Bear Suppression Inventory (WBSI). The fourth aim of this study is to examine the sensitivity and specificity of AAQ II Malay version in cancer patients by comparing it with AAQ II in non-cancer patients.

Methodology

Research design

In measuring content validity index, descriptive research design with purposive sampling method was used to select subject matter experts to evaluate content validity. In measuring reliability, validity, sensitivity and specificity, descriptive research design with random sampling method were used to select respondents.

Sample

101 cancer patients were recruited from the oncology department in University Malaya Medical centre (UMMC) and National cancer institute (NCI). The inclusive criteria for this validation study are cancer patient in outpatient department (newly diagnosed and recurrent), above 18 years old, bilingual (Malay and English), and no prior psychiatric history. Meanwhile, 100 non cancer respondents were randomly selected at public places like in community hall and hospital.

Procedure

The study was conducted starting from October 2016 to December 2017 at oncology clinic UMMC and a daycare at NCI. The ethical approval was obtained from Medical Ethic Committee (MEC) of the University Malaya Medical Centre and National Medical Research Register (NMRR).

Respondents who met inclusion criteria were approached and explained on regards to research procedure. Once the written informed consent was obtained, a self-administered questionnaire was distributed.

Back translation procedure.

AAQ II was translated into Malay Language via back translation. By translating to respondents’ native languages, it will facilitate respondents in understanding the questionnaire. The English version of AAQ-II was translated into Malay version by 2 independent translators (first and second translators). First and second translators are experts in languages (Malay and English). Then, the third translator who is expert in language and psychology evaluated the two versions of Malay translation and chose the most accurate interpretation of the real meaning of the translation in the target language. Then, another two independent translators (fourth and fifth translators) translated back from Malay version (determined by third translator) to English version. The fourth and fifth translators have no prior knowledge of the original version. The translated versions were compared with the original English version to evaluate the equivalence of meaning between the original content and translated version.

Expert review.

Once translated, AAQ II Malay version was submitted to six subject matter experts in psychology and oncology; four clinical psychologists, one psychiatrist (psycho-oncology) and one oncology nurse. The experts evaluated the items in the AAQ II based on clarity and relevance on 4 points Likert scale. The experts reviewed the questionnaire and commented the questionnaire based on the relevancy of the item to represent the measured construct, accuracy and clarity of the translation. They identified the item that did not clearly express the accurate meaning or did not fit with Malay idiomatic expressions and gave suggestions for improvement. In term of clarity, the items were rated based on 1) not clear, 2) item need some revision, 3) clear but need minor revision, or 4) very clear. In term of relevancy, the items were rated based on how adequately the items matched the main construct of AAQ II Malay version by using the following four-point scale: (1) not relevant, (2) item need some revision (3) relevant but need minor revision, or (4) very relevant.

Questionnaires

Socio-demographic.

Respondents were required to provide their socio-demographic information such as gender, age, employment status, educational background, diagnosis and stage of cancer.

Malay version of Acceptance and action II.

7 items AAQ II Malay version as outlined in the final Malay version was used. The items as shown in S1 Appendix were rated on a 7 point Likert type scale from 1 (never true) to 7 (always true). Higher score on AAQ II indicates a greater level of experiential avoidance.

Acceptance and action II (English version).

AAQ II (S2 Appendix) was originally from AAQ developed by Hayes et al, [24] and has been revised by Bond et al. [9]. AAQ II is a unidimensional scale with 7 items and is rated based on 7 point Likert scale. AAQ II has good internal consistency (α = 0.88) [9] and good test retest reliability over 3 and 12 months at 0.81 and 0.79 respectively. The translated version of AAQ II demonstrated acceptable reliability in Chinese version with α = 0.86 [16], Turkish version with α = 0.84[15] and Greek version with α = 0.92[17].

White Bear Suppression Inventory (WBSI).

White Bear Suppression Inventory is developed by Wegner & Zanakos [25] in 1994. WBSI measures the tendency to suppress the undesirable thought including obsessive thinking and negative affect related with depression and anxiety. WBSI consists of 15-items and rated based on 5-point Likert scale. Higher scores on the WBSI scale indicate a higher tendency to suppress thoughts. The WBSI scores demonstrated a good internal consistency with α > 0.87 [25].

Hospital anxiety depression scale malay version (HADS-M).

The Hospital Anxiety and Depression Scale is developed by Zigmond and Snaith [26] in 1983. HADS measures mood disorder particularly depression and anxiety. HADS consists of 14 items; 7 items for anxiety scale (HADS-A) and 7 items for depression scale (HADS-D). Each item is evaluated on a 4 point (0–3) Likert scale and a higher score indicates higher symptoms of anxiety and depression. HADS Malay version has a good internal consistency with α = 0.82 [27].

Statistical analysis.

The proportion is used to analyse content validity index of AAQ II particularly to calculate content validity index item level (I-CVI), content validity index scale-level (S-CVI) and modified Kappa statistic. Item level CVI (I-CVI) for relevancy and clarity of each item were calculated by dividing the number of experts who evaluate the item as 3 or 4 with the number of content experts. If the score of I-CVI is higher than 0.79, the item is considered as excellent and appropriate, score between 0.7 and 0.7 indicates that the item needs a revision and score below than 0.7 indicates that the item should be eliminated [28,29]. In order to calculate the S-CVI, the sum of I-CVIs was divided by the total number of items. The score of S-CVI which is 0.9 and higher is considered as appropriate and excellent [23,30]. Modified kappa statistic (k*) is defined as an index of agreement among the subject matter experts that the item is relevant and clear [30]. In term of calculation for modified Kappa statistic, the probability of chance agreement (pc) will be calculated for each item by the following formula:In this formula, N = number of experts and A = number of experts who agree that the item is relevant. After that, Kappa will be computed by entering the values of probability of chance agreement (Pc) and CVI of each item (I-CVI) by following formula:The interpretation of Kappa value above 0.74 is considered as excellent, between 0.60 and 0.74 is considered as good, and the value between 0.40 and 0.59 is considered as fair [31].

Statistical Package for Social Sciences version 22.0 (SPSS) was used to analyse reliability, validity, sensitivity and specificity. Data pertaining socio-demographic were computed using descriptive statistic. In the evaluation of internal consistency of AAQ II Malay version, Cronbach’s alpha and homogeneity of the scales were assessed. The intra-class correlation coefficient (ICC) was computed to examine the parallel form reliability between AAQ II and AAQ II Malay version. Principles axis factoring was performed to extract the dimensionality of AAQ II Malay version. In establishing concurrent validity, the Spearman’s test was performed to examine the correlation between AAQ II Malay version with HADS-A, HADS-D, and WBSI. Analysis of covariance was performed to examine the differences between cancer patients and non-cancer respondents with control of gender, age and educational level. The rates of sensitivity and specificity were analyzed via area under the curve (AUC) and the Youden Index. Cut-off point of AAQ II Malay version for cancer patients was determined based on the highest score of Youden Index (J). The flow of study was as shown by (Fig 1).

[Figure omitted. See PDF.]

Fig 1. Flow of Psychometric properties study.

https://doi.org/10.1371/journal.pone.0212788.g001

Result

Table 1 shows demographic information across respondents with cancer and without cancer. 101 respondents with cancer (14.9% male and 85.1% female) and 100 respondents without cancer (31% male and 69% female) were recruited. Mean age for cancer respondents is 48.32 and mean age for non-cancer respondents is 43.44. Only age has significant mean difference with total scores of AAQ II Malay version.

[Figure omitted. See PDF.]

Table 1. Demographic information.

https://doi.org/10.1371/journal.pone.0212788.t001

Content validity

Content validity index (CVI) was calculated via agreement of six subject matter experts for each item (I-CVI) in the instrument and for the whole instrument (S-CVI). Content validity index item level (I-CVI) scores for all items in respect of relevance, five items had perfect scores and two items were 83% relevance for experiential avoidance construct. In term of clarity, six items scored perfect score and only one item scored 0.83. After determining the CVI-I, modified kappa statistic were calculated, 5 items had perfect score (1) for k*index for relevance and two items scored 0.82 for relevance. Meanwhile, six items had perfect score for k*index and one item scored 0.82 for clarity. Then, the score for entire items in the instrument was calculated after obtaining the I-CVI for each item (Table 2). The content validity index scale level (S-CVI) for relevance and clarity were 0.95 and 0.97 respectively.

[Figure omitted. See PDF.]

Table 2. Content validation of AAQ II Malay version.

https://doi.org/10.1371/journal.pone.0212788.t002

Reliability

The AAQ II Malay version provided good internal consistency which Cronbach’s alpha value was 0.91 for cancer respondents and 0.90 for non-cancer respondents. The mean values were around 3.00 indicating that overall there was a moderate level of psychological inflexibility. The highest correlation for each item with at least one other item in the construct is between 0.3 and 0.9. Hence, the items correlate adequately in the construct. The lowest corrected item-total correlation (CITC) value was 0.605> 0.3. Elimination of any item in the AAQ II Malay version scale would not increase the Cronbach’s alpha value. Intra-class correlation coefficient ICC between AAQ II (English) and AAQ II Malay version (ICC = 0.938, p<0.00) indicated excellent parallel reliability.

Dimensionality of the AAQ II Malay version

Values of the Kaiser-Meyer-Olkin which were more than 0.5 (0.874) indicated the sampling were adequate and a significant value (p < .001) of Barlett’s Test of Sphericity also suggested that relationship exist between at least some of the subscales and the data were suitable for factor analysis. A single factor was extracted using principle axis factoring and that explained 59.78% of the variation in the 7 items (eigen value > 1.00). The minimum factor loading was 0.640>0.5.

Concurrent validity

The AAQ II Malay version was significantly correlated with the HADS-D (p<0.01), HADS-A (p<0.01) and WBSI (p<0.01) scores (Table 3). Hence, concurrent validity of AAQ II Malay version was established.

[Figure omitted. See PDF.]

Table 3. Spearman's rho correlations (r) among the AAQ II Malay version, HADS-A and HADS-D.

https://doi.org/10.1371/journal.pone.0212788.t003

There was a significant difference in mean in the AAQ II Malay version scores [F(1,199) = 51.34, p = 0.000] between the cancer and non-cancer respondents when the age was adjusted. Cancer patients (M = 20.63, SD = 8.03) reported significantly higher AAQ II Malay version scores than non-cancer respondents (M = 14.17, SD = 4.90).

Specificity, positive predictive value, and negative predictive value of AAQ II Malay version

Based on ROC curve analysis (Table 4), the area under the curve was 0.747 (95% CI = 0.68–0.82). Based on the Youden Index, the optimal cut off score to distinguish respondents with cancer and without cancer was >17.5 with a sensitivity of 63% and specificity of 78% (Table 5).

[Figure omitted. See PDF.]

Table 4. Area under the curve.

https://doi.org/10.1371/journal.pone.0212788.t004

[Figure omitted. See PDF.]

Table 5. Coordinates of the receiver operating characteristic (ROC) curve.

https://doi.org/10.1371/journal.pone.0212788.t005

Discussion

The purposes of this study were to translate, adapt and validate the AAQ II Malay version. The first objective was to evaluate the content validity index of AAQ II Malay version. Content validity implied the robustness of the interpretation of the items in the instrument to reflect and represent the construct. In this context, the results of this current study were determined by CVI and modified Kappa coefficient. The findings reveal that all items scored I-CVI equal to 0.83 and above. According to Polit & Beck [29], the I-CVI scores that fall above 0.79 are considered as excellent and appropriate. The S-CVI for both relevance and clarity that exceed 0.9 indicate as appropriate and excellent. The results from modified kappa statistic demonstrated the index of agreement between subject matter experts in AAQ II items that were relevant and cleared, were 82% and reached excellent level. Based on the results, all items in the instrument AAQ II were validated.

The findings in this study revealed that AAQ II Malay version has good internal consistency with Cronbach’s alpha value of 0.91 (cancer respondents) and Cronbach’s alpha value of 0.90 (non-cancer respondents). This result is comparable with the original AAQ II with α = 0.84 [9], AAQ II Turkish version with α = 0.84 [15], AAQ II Chinese version with α = 0.82 [16], AAQ II Portuguese version with α = 0.89 [20] and AAQ II Greek version with α = 0.92 [17]. The high internal consistency of AAQ II Malay version among cancer and non-cancer respondents indicated that the items in this scale measured the same construct namely experiential avoidance. In addition, the result of parallel for reliability also demonstrated excellent reliability between Malay version and original version of AAQ II. This result supports that AAQ II Malay version is acutely assessed and the construct of experiential avoidance is in line with the original version.

Exploratory factor analysis (EFA) was conducted to determine the dimensionality of AAQ II Malay version. The results of this study demonstrated that there is a single factor that accounted the large proportion of variance (59.78%) in AAQ II Malay version. The result of unidimensional factor is similar with the study conducted by Bond et al., [9], Karekla & Michaelides [17] and Zhang et al., [16].

In term of examining the concurrent validity, the correlation between AAQ II Malay version and HADS-A, HADS-D and WBSI were performed. The results demonstrated the moderate positive correlations between experiential avoidance and anxiety, depression and suppression thoughts related to anxiety and depression. These correlations supported the evidence that experiential avoidance as significant element in the etiology and the preservation of psychopathology problems such as depression and anxiety.

In this study, the mean total score of AAQ II Malay version was reported as significantly higher than non-cancer respondents. However the mean of clinical sample in this study was lower than clinical sample in the study conducted by Bond et al., [9] (M = 28.34) and other studies conducted in Portuguese population (M = 24.72) [18], Turkish population (M = 26.17) [15]. The novelty finding of this study is that the results provide information on the sensitivity and the specificity of AAQ II Malay version among cancer patients. To the best our knowledge, there is still no study that examines the cut off score of AAQ II for cancer patients. Though AAQ II is not designed as a diagnostic tool, but the cut off score can be used as a reference to identify the scores of AAQ II Malay version that is associated with the significant indicator of experiential avoidance among cancer patients. The result indicated that the score of AAQ II Malay version was more than 17.5, the cut off point for cancer patients that experienced significant experiential avoidance. The cut off score of AAQ II Malay version for cancer patients was lower than the cut off score of AAQ II for sample that was seeking treatment for substance misuse which fall between 24 to 28 [9]. In spite of different type of clinical sample, the lower mean total score and cut off score may be related with cultural differences between Malaysian and others countries in term of how Malaysian expressed their feeling and thought especially the negative thought and feelings. Malaysians, regardless the races, tend to be very careful when exhibiting their negative emotions especially when they are emotionally depressed [32]. They tend to remain silent as well as underrated their emotion and feeling due to culturally acquired rules of interaction [32].

There are few limitations in this study. First, this is a cross sectional study, therefore it is impossible to identify the causal factors and effect of experiential avoidance in cancer patients. Second, the cancer patients in this study were recruited from oncology clinic and day-care using convenience sampling, thus generalization of the findings is one of the possible limitations. Lastly, small sample size could explain the moderate factor loading (0.64) in the item six in the instrument found in this study. Larger sample sizes in future possibly will yield higher factor loading for this item.

Conclusion

In spite of these limitations, the AAQ II Malay version demonstrated acceptable and promising psychometric properties of AAQ II Malay version. The results in term of dimensionality, reliability and validity were consistent with previous studies [9,15,16]. In addition, the current study also provided the empirical evidence on sensitivity and specificity of AAQ II Malay version in cancer patients that is devoted to Malaysian population.

Supporting information

[Figure omitted. See PDF.]

S1 Appendix. Soal selidik penerimaan dan tindakan II versi Bahasa Melayu (AAQ II).

https://doi.org/10.1371/journal.pone.0212788.s001

(PDF)

S2 Appendix. Acceptance and action questionnaire II (AAQ-II).

https://doi.org/10.1371/journal.pone.0212788.s002

(PDF)

Citation: Shari NI, Zainal NZ, Guan NC, Ahmad Sabki Z, Yahaya NA (2019) Psychometric properties of the acceptance and action questionnaire (AAQ II) Malay version in cancer patients. PLoS ONE 14(2): e0212788. https://doi.org/10.1371/journal.pone.0212788

References

1. Hayes S, Strosahl K, Wilson K (1999). Acceptance and Commitment Therapy: An experiential approach to behavior change New York: Guilford Press.

2. Hulbert-Williams NJ, Storey L, Wilson KG (2015) Psychological interventions for patients with cancer: psychological flexibility and the potential utility of Acceptance and Commitment Therapy. European Journal of Cancer Care 24: 15–27. pmid:25100576

3. Aguirre-Camacho A, Pelletier G, Gonzalez-Marquez A, Blanco-Donoso LM, Garcia-Borreguero P, et al. (2017) The relevance of experiential avoidance in breast cancer distress: insights from a psychological group intervention. Psychooncology 26: 469–475. pmid:27228257

4. Donovan E, Caughlin J (2011) Breast cancer patients’ topic avoidance and psychological distress: The mediating role of coping. Journal of Health Psychology 16: 596–606. pmid:21346008

5. Salber K, Nezu A, Nezu C (2015) Experiential Avoidance and Distress in Breast Cancer Patients. [Ph.D thesis]: Drexel University.

6. Karekla M, Forsyth J, Kelly M (2004) Emotional avoidance and panicogenic responding to a biological challenge procedure. Behavior Therapy 35: 725–746.

7. Gonzalez-Fernandez S, Fernandez-Rodriguez C, Mota-Alonso MJ, Garcia-Teijido P, Pedrosa I, et al. (2017) Emotional state and psychological flexibility in breast cancer survivors. Eur J Oncol Nurs 30: 75–83. pmid:29031317

8. Bond FW, Bunce D (2003) The role of acceptance and job control in mental health, job satisfaction, and work performance. J Appl Psychol 88: 1057–1067. pmid:14640816

9. Bond FW, Hayes SC, Baer RA, Carpenter KM, Guenole N, et al. (2011) Preliminary psychometric properties of the Acceptance and Action Questionnaire-II: a revised measure of psychological inflexibility and experiential avoidance. Behav Ther 42: 676–688. pmid:22035996

10. Hayes SC, Luoma JB, Bond FW, Masuda A, Lillis J (2006) Acceptance and commitment therapy: model, processes and outcomes. Behav Res Ther 44: 1–25. pmid:16300724

11. Lilly MM, Allen CE (2015) Psychological Inflexibility and Psychopathology in 9-1-1 Telecommunicators. J Trauma Stress 28: 262–266. pmid:25964163

12. Mosher CE, Tometich DB, Hirsh A, Rand KL, Johns SA, et al. (2017) Symptom experiences in metastatic breast cancer patients: relationships to activity engagement, value-based living, and psychological inflexibility. Psychooncology 26: 1944–1951. pmid:27648927

13. Hayes SC, Strosahl KD, Wilson KG (2012) Acceptance and Commitment Therapy: the Process and Practice of Mindful Change. New York: Guilford Publications.

14. Berrocal Montiel C, Rivas Moya T, Venditti F, Bernini O (2016) On the contribution of psychological flexibility to predict adjustment to breast cancer. Psicothema 28: 266–271. pmid:27448259

15. Yavuz F, Ulusoy S, Iskin M, Esen FB, Burhan HS, et al. (2017) Turkish Version of Acceptance and Action Questionnaire-II (AAQ-II): A reliability and Validity Analysis in Clinical and Non-Clinical Samples. Klinik Psikofarmakoloji Bülteni-Bulletin of Clinical Psychopharmacology 26: 397–408.

16. Zhang C, Chung P, Si G, Liu JD (2014) Psychometric Properties of the Acceptance and Action Questionnaire-11 for Chinese College Students and Elite Chinese Athletes. Measurement and Evaluation in Counseling and Development 47: 256–270.

17. Karekla M, Michaelides MP (2017) Validation and invariance testing of the Greek adaptation of the Acceptance and Action Questionnaire -II across clinical vs. nonclinical samples and sexes. Journal of Contextual Behavioral Science 6: 119–124.

18. Costa J, Maroco J, Pinto-Gouveia J, Galhardo A (2014) Validation of the Psychometric Properties of Acceptance and Action Questionnaire-II in Clinical and Nonclinical Groups of Portuguese Population. International Journal of Psychology and Psychological Therapy 14: 353–364.

19. Ruiz FJ, Suárez-Falcón JC, Cárdenas-Sierra S, Durán Y, Guerrero K, et al. (2016) Psychometric Properties of the Acceptance and Action Questionnaire–II in Colombia. The Psychological Record 66: 429–437.

20. Pinto-Gouveia J, Gregório S, Dinis A, Xavier A (2012) Experiential Avoidance in Clinical and Non-Clinical Samples: AAQ-II Portuguese Version. International Journal of Psychology & Psychological Therapy 12: 139–156.

21. Lynn M (1986) Determination and Quantification Of Content Validity. Nursing Research 35: 382–385. pmid:3640358

22. Yaghmale F (2003) Content validity and its estimation. Journal of Medical Education 3: 25–27.

23. Waltz CF, Strickland O, Lenz ER (2005) Measurement in nursing and health research New York: Springer Publishing Company.

24. Hayes S, Strosahl K, Wilson K, Bissett R, Pistorello J, et al. (2004) Measuring experiential avoidance: A preliminary test of a working model. The Psychological Record 54: 553–578.

25. Wegner D, Zanakos S (1994) Chronic thought suppression. Journal of Personality 62: 615–640.

26. Zigmond AS, Snaith RP (1983) The hospital anxiety and depression scale. Acta Psychiatrica Scandinavica 67: 361–370. pmid:6880820

27. Lua PL, Wong SY (2012) The Reliability of the Malay Versions of Hospital Anxiety Depression Scale (HADS) and Mcgill Quality of Life Questionnaire (MQOL) among a Group of Patients with Cancer in Malaysia. Malaysian Journal of Psychiatry 21: 25–37.

28. Abdollahpour I, Nedjat S, Noroozian M, Majdzadeh R (2011) Performing Content Validation Process in Development of Questionnaires. Iran J Epidemiol 6: 66–74.

29. Polit DF, Beck CT (2006) The content validity index: Are you sure you know what's being reported? critique and recommendations. Research in Nursing & Health 29: 489–497.

30. Polit DF, Beck CT, Owen SV (2007) Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health 30: 459–467. pmid:17654487

31. Cicchetti DV, Sparrow SA (1981) Developing criteria for establishing inter-rater reliability of specific items: applications to assessment of adaptive behavior. Am J Ment Defic 86: 127–137. pmid:7315877

32. Kuang CH, Wong NL, Maya KH (2015) The Perceived Value of Silence and Spoken Words in Malaysian Interactions. The Journal of the South East Asia Research centre for Communications and Humanities 7: 53–70.