Construction and validation of an Educational Content Validation Instrument in Health

Abstract

[...]it is imperative that tools to validate educational material contents in the health areas can evaluate any themes, featuring proper reliability and validity. According to these principles, educational materials should provide interactivity, be attractive and written in an appropriate language to the target audience, promote relevant and contextualized activities, allow the exchange of experiences, and present quality information®. DISCUSSION Developing and validating the ECVIH contributes to the clinical and scientific praxis because this instrument is an innovative tool to be used to validate educational content available in materials such as videos, albums, booklets, games, websites, and software. [...]considering that it does not specify information about the theme, target audience, and application circumstances, the instrument can support educational activities in health. [...]the final configuration of the instrument has eighteen items, divided into three domains, with response options ranging from 0 to 2.

Headnote

Submission: 09-21-2017 Approval: 01-27-2018

ABSTRACT

Objective: to construct and validate the Educational Content Validation Instrument in Health. Method: methodological study that includes the establishment of the conceptual structure; definition of objectives and population; construction of items and response scale; selection and arrangement of items; instrument structuring; opinion of experts; pre-test and content validation. Results: an instrument with 15 items was constructed and, after expert evaluation, eighteen items were obtained, divided into three domains: objectives (four), structure/presentation (nine), and relevance (two). Six items were modified since they presented a percentage of agreement below 0.8. Items of the total instrument presented good internal consistency (0.877) regarding domains. Conclusion: an Educational Content Validation Instrument in Health was elaborated and validated, presenting good reliability, and may contribute to the practice of researchers and health professionals in the development of educational content.

Descriptors: Methodological Research in Nursing; Health Education; Data Collection; Validation Studies; Nursing.

RESUMO

Objetivo: construir e validar o Instrumento de Validaçâo de Conteúdo Educativo em Saúde. Método: estudo metodológico contemplando estabelecimento da estrutura conceitual; defi niçâo dos objetivos e populaçâo; construçâo dos itens e escala de resposta; seleçâo e organizaçâo dos itens; estruturaçâo do instrumento; opiniâo de especialistas; pré-teste e validaçâo de conteúdo. Resultados: construiu-se instrumento com quinze itens e após avaliaçâo dos especialistas obtiveram-se dezoito itens, divididos em tres dominios: objetivos (quatro), estrutura/apresentaçâo (nove) e relevancia (dois). Seis itens foram modificados, pois apresentaram percentual de concordancia inferior a 0,8. Itens do instrumento total apresentaram boa consistencia interna (0,877) quanto aos seus dominios. Conclusâo: foi elaborado e validado Instrumento de Validaçâo de Conteúdo Educativo em Saúde, que apresentou boa confiabilidade, podendo contribuir para a prática de pesquisadores e profissionais das áreas de saúde na elaboraçâo de conteúdos educativos.

Descritores: Pesquisa Metodológica em Enfermagem; Educaçâo em Saúde; Coleta de Dados; Estudos de Validaçâo; Enfermagem.

RESUMEN

Objetivo: construir y validar el Instrumento de Validación de Contenido Educativo en Salud. Método: estudio metodológico contemplando el establecimiento de la estructura conceptual; definición de los objetivos y población; construcción de los items y escala de respuesta; selección y organización de los items; estructura del instrumento; opinión de expertos; pre-test y validación de contenido. Resultados: se construyó un instrumento con quince items y después de la evaluación de expertos, se obtuvieron dieciocho items, divididos en tres áreas: objetivos (cuatro), estructura/presentación (nueve) y relevancia (dos). Seis items fueron modificados, pues presentaron un porcentaje de concordancia inferior a un 0,8. Los items del instrumento total presentaron una buena consistencia interna (0,877) en cuanto a sus dominios. Conclusión: fue elaborado y validado el Instrumento de Validación de Contenido Educativo en Salud, que presentó buena confiabilidad, contribuyendo a la práctica de investigadores y profesionales de las áreas de salud en la elaboración de contenidos educativos.

Descriptores: Investigación Metodológica en Enfermería; Educación en Salud; Recolección de Datos; Estudios de Validación; Enfermería.

INTRODUCTION

Materials with educational content are facilitators of the teaching-learning process, enabling the transfer of knowledge, through the individual's participation and involvement, and the exchange of experiences conducive to the improvement of skills(1). Such materials have been widely used for health education, which is a vehicle of knowledge socialization for contributing to the improvement of living conditions and health of the population(2).

Educational materials should be properly prepared and evaluated before being used by the target population. One of the essential steps for developing an effective educational material is the content validation, a process that evaluates its representativeness in properly addressing the universe with which it intends to deal and aims to measure or attend to the absence of unnecessary elements(3).

Currently, measuring instruments have been used to aid the content validation, constituting tools that measure indicators and attribute numeric values to abstract concepts, which can be observable and measurable. Thus, they contribute to enhancing the health praxis(4).

To develop tools that validate educational contents, an essential step is defining the construct to be investigated(5). Among the aspects evaluated by such instruments, issues involving promotion, prevention, and recovery of health are frequent, such as quality of life, vitality and limitations, adherence to treatment, and emotional and psychosocial factors(6).

However, there are recurring problems concerning the validation of instruments related to educational contents in health. It should be noted that inappropriate validation forms, without rigorous methodological criteria, the absence of instruments validated by capable professionals, as well as the fragmentation of educational instruments for health in topics or areas of interest, contribute to this process not always be effective, hindering the improvement and dissemination of knowledge about the phenomenon under study(7). This may result in the provision of educational materials with mistaken technical and didacticpedagogical aspects. Thus, it is imperative that tools to validate educational material contents in the health areas can evaluate any themes, featuring proper reliability and validity. This will avoid unnecessary adaptation of instruments that validate educational contents for new research contexts, in addition to optimizing the time of researchers in this stage of data collection.

In the scientific literature, one can observe the absence of a universal validation instrument for contents with educational purposes. Given this context, the construction and validation of a reliable instrument through a scientific basis becomes necessary to validate several contents in health.

OBJECTIVE

To construct and validate the Educational Content Validation Instrument in Health (ECVIH), to be used by researchers and health professionals in the orientation and development of educational content.

METHOD

Ethical aspects

The study development complied with national and international ethical standards of research with human beings. This research was approved by the Research Ethics Committee of the Federal University of Ceará (UFC). Experts who participated in this study were clarified as to the purposes of the research and the nature of data collection. Those who agreed in participate signed the Informed Consent Form (ICF).

Study design, location, and period

This is a methodological study that includes the establishment of the conceptual structure; definition of objectives and population; construction of items and response scale; selection and arrangement of items; instrument structuring; opinion of experts; pre-test, and content validation(4).

For the construction of the instrument, a narrative review(8) in the databases PubMed, Scopus, and Cinahl, and a theoretical framework based on the principles of the instructional design of Filatro(9) was used.

The instrument construction and validation steps were carried out from March to December 2016. The sixth and seventh step, referring to the experts' opinion and to pre-testing, were performed in person in the Laboratory of Communication and Health (Labcom_Saúde) of UFC. The last step, content validation, occurred virtually.

Population or sample, inclusion and exclusion criteria

A total of 27 experts participated in the study. Were selected, by trial, the experts who met the following inclusion criteria: Ph.D. and/ or Master's Degree in Health areas, professional experience (clinical, teaching, or research), and an article published in indexed journals in the study area of interest. Experts who have not responded to the invitation to participated in the study were excluded.

Study protocol

The process of instrument construction, on the first step, deals with the theoretical foundation of what will be studied for defining construct operations and dimensionality. Such step was based on the narrative review(8) and in the principles of instructional design, of Filatro(9).

The narrative review occurred in the databases PubMed, Scopus, and Cinahl, using the controlled keywords "Validation Studies" and "Teaching Materials", and the non-controlled "Measuring Instruments". The following crossings were performed: "Validation Studies" and "Teaching Materials"; "Validation Studies" and "Measuring Instruments"; "Teaching Materials" and "Measuring Instruments". This step was based on the guiding question: which evaluation criteria were used to validate educational contents?

As inclusion criteria, we defined: research available in full and with free access in the selected databases, in English, Portuguese, or Spanish languages, which used instruments to evaluate the content in educational materials and were published from 2006 to 2016. Editorials, letters to the editor, reflective studies, experience reports, annals of scientific events (abstracts), and duplicate publications were excluded. Thus, 1,600 articles were found, and fifty were included in the research. However, after reading the full texts, 36 articles were excluded, totaling fourteen analyzed articles.

The examined literature showed that items of instruments used to evaluate educational material in health focus three groups of content-related factors: objectives, structure and presentation, and relevance. Items belonging to the group "objects" were related to purpose, goals, or target of the use of educational materials. The group "structure and presentation" contained information concerning the general organization, structure, strategy, consistency, and sufficiency of the presentations. In "relevance", the items assessed the significance degree of the educational content and its ability to cause impact, motivation and/or interest.

For the theoretical support of the evaluated instruments, it was also used a theoretical reference based on the principles of instructional design, which is defined as the systematic actions of planning and construction of teaching methods and/ or materials that can be used for classroom teaching, blended education, or distance learning. According to these principles, educational materials should provide interactivity, be attractive and written in an appropriate language to the target audience, promote relevant and contextualized activities, allow the exchange of experiences, and present quality information®. In this study, the items of the ECVIH sought to cover such instructions, which are regarded as essential for the development of materials for educational purposes.

Then, goals and population were defined. The ECVIH aims to provide a scientific basis for the content validation of educational materials in health. It is destined to higher-level health professionals who wish to construct and validate educational contents to any target audience.

Instrument construction refers to the preparation or selection of its content according to the construct operational definitions. The conceptual structure step allowed the construction of nineteen items, with response options by a Likert-type scale, being 0 = disagree, 1 = partially agree, and 2 = totally agree. Following, items were selected and organized to evaluate the relevance of the drafting, adopting the behavioral criterion, in addition to objectivity, simplicity, clarity, accuracy, validity, relevance, and interpretability(10). After that, eighteen items remained.

Subsequently, the instrument was structured. This step aimed at consolidating the previous ones, namely, organizing the content to obtain a logical presentation order and thus establish the general format. Therefore, the ECVIH was structure by subdividing it into objectives, structure/presentation, and relevance. The last three steps were held with experts in the area of interest in measuring instrument, educational material, and health education.

Five experts in health, education, and statistics participated in the step of opinions and validation of the constructed items, to expand the diversity in the studied area of knowledge. All of them were Ph.Ds., had a scientific production and more than 10 years of experience in the area of interest of this study. The nominal group technique was adopted, being interactive, face-to-face, simultaneous, and held in group consensus(11).

Eleven experts participated in the instrument pre-testing, all of them having from three to five years of experience in the development of methodological studies in the area of health. Data collection happened only once, in and individual encounter, and came because of the discussion on instrument items and associated appendices.

The validation process counted with eleven experts - nine nurses and two pedagogues. The mean age was 38.9 ± 10,7 years. As for professional qualification, nine were Ph.Ds. and two had Master's degree. We emphasize that tens experts were professors in Brazilian public universities (UFC, UECE, UNILAB, UNICAMP, and UFPI) and all had more than five years of scientific production in the study area. This process was made possible by electronic mail, via invitation letter and request for assessing the ECVIH regarding clarity, pertinence, relevance, and dimension in the writing of each item.

Analysis of results and statistics

Data were analyzed by the intraclass correlation coefficient (ICC), being regarded as great reliability (ICC > 0.9), good reliability (0.7 < ICC < 0.8), and poor reliability for values below 0.6(12). The significance level adopted was 95%. An agreement percentage of 80% was considered as a criterion for deciding on the pertinence of the instrument items, or its modification.

RESULTS

The initially constructed instrument comprised eighteen items and, after the five-expert consensus, fifteen remained (Chart 1). It should be noted that, after pre-testing, there was no modification in the items devised. Eleven experts have then performed the content validation process for the instrument items, following the criteria: clarity, practical pertinence, theoretical relevance, and theoretical dimension (Table 1). Items that did not reach an 80% agreement percentage were reformulated.

In the reliability assessment, the ECVIH presented a satisfactory total internal consistency (ICC > 0.8) and a significant p-value (p < 0.05), similarly to the domains "structure/presentation" and "relevance". However, in the domain "objectives", a weak internal consistency was identified (ICC = 0.432) and p-value > 0.05, i.e. indicating the need for adjustments to increase domain reliability (Table 2).

The final configuration of the ECVIH has 18 items, divided into three areas: objectives, structure/presentation, and relevance. The total score of the instrument is calculated through the sum of all domains (Chart 2).

DISCUSSION

Developing and validating the ECVIH contributes to the clinical and scientific praxis because this instrument is an innovative tool to be used to validate educational content available in materials such as videos, albums, booklets, games, websites, and software. Moreover, considering that it does not specify information about the theme, target audience, and application circumstances, the instrument can support educational activities in health.

The use of the nominal group technique was advantageous for the ECVIH construction since the consensus on the maintenance, modification, or exclusion of items much contributed for the construction of a more reliable instrument due to the evaluators' expertise(11).

The prior use of the instrument in a small sample, characterizing a pre-test, allowed us to reach a preliminary version with more harmonic methodological choices.

For the ECVIH validation process, the experts evaluated three domains, namely: objectives, structure/presentation, and relevance. In the first domain, the modifications consisted in the inclusion of the items "provides reflection on the subject" and "encourages behavior changes", as well as the offset of the item "stimulates the learning" to another domain. Although the five items belonging to this domain presented an agreement percentage of more than 80% among the experts in all criteria assessed, the researchers complied with the modifications suggested by the experts to improve the ECVIH items.

Educational materials evaluated by content instruments in the health area should aim at facilitating the work of health professionals in orienting and educating patients and family(13). Furthermore, they must aim at promoting healthy, ensuring the effectiveness of care for the target population. Therefore, the domain "objective" of an instrument is required for the understanding on the content to be studied.

In the domain "structure/presentation," the items complied with the criteria of practical pertinence and theoretical relevance and dimension, with an agreement of more than 80% among experts, requiring adjustments in five items to make them more understandable since they featured low agreement on the criterion language clarity. The main modifications were in relation to the items writing, such as replacing "proper language" for "language appropriated to the target audience" and "interactive language" for "Interactive language, enabling active involvement in the educational process". We also decided to standardize nomenclatures, such as "thematic content" and "theme".

Paying attention to the evaluation of educational materials in health is essential, through specific content instruments, especially regarding a consistent, cohesive, organized, sufficient structure, and an appropriate language. Regarding cohesion and coherence, the content must remain focused on the proposed theme, and in its topics, paragraphs, or portions there should be a logical sequence of ideas, starting from general and impersonal issues until more delicate or specific subjects. Sufficiency is one of the aspects closely related to material validity since sufficient educational contents explain relevant topics of the proposed theme in a comprehensive way, in addition to preventing doubts of the reader(9-14).

Among the necessary aspects for the language of a good measuring instrument, objectivity and clarity are highlighted. Long and detailed sentences may lead to dispersion and make the reading tiresome, and an extensive instrument can compromise the willingness in replying it. Ambiguity caused by the use of vague terms is another potential problem related to the language, which may result in inadequate and inaccurate responses(15-16). To avoid such problems, one of the aspects this study has prioritized was the use of a clear and direct language. The ECVIH items are objective, comprising solely the words required for understanding what is being measured. The size of the instrument also contributes to its fast and simple filling since it consists of only eighteen items. We also emphasize that each item measures only one attribute. Therefore, the conciseness is considered the strong point of this instrument.

Regarding "relevance", the experts suggested significant changes in the items of this domain, such as the exchange of some items with the dimension "objective", in addition to the inclusion of the items "contributes to knowledge in the area" and "arouses interest in the topic".

For the content evaluated by health areas instruments to be current and relevant, it is essential that the researcher is inserted in the social context of the target population, knowing its needs and circumstances. Theme and topics covered in educational contents should meet the needs of the participant, promoting empowerment on the addressed subject. A previous study on education through a simple educational brochure on low-carbohydrate and low-sugar diet shows the increased knowledge on the theme after using the resource, in addition to emphasizing the importance of educational materials with this purpose(17).

In the reliability assessment, the total ECVIH presented a good level of internal consistency, as well as the domains "structure/ presentation" and "relevance", and a weak level only for the domain "objective". It is important to mention that, although the instrument present this weakness, the recommended modifications enabled the improvement of reliability in the domains "objective", considering the experience of the experts in devising and validating the measuring instrument and in educational contents in the area of health.

Thus, the ECVIH points to several relevant aspects to educational content, which are essential to fulfill the purposes of this kind of material and provide education as efficiently as possible. It is therefore considered that the instrument was able to meet the purposes for which it was developed, being able to orientate the construction of educational contents in health, in different formats, since it was overall considered appropriate by the experts.

Study limitations

As a limitation, we emphasize that the instrument here constructed and validated aims solely to evaluate the written content of education materials, being excluded pictures, images, and drawings. It is understood that these items will be evaluated in the subsequent validation process of construct appearance.

Contributions to the field of nursing, health, or public policies

This study will contribute to the practice of researchers and health professionals in developing contents for educational materials that go beyond health and traditional education.

It is believed that the elaboration of a research instrument based on general standards may be the first step to enhance educational intervention in health, evaluating multi-professional phenomena in different health contexts.

CONCLUSION

This study constructed and validated the ECVIH, which presented a good reliability (ICC > 0.8) and value p < 0.05. The participation of specialists in the development and validation of this instrument was essential to avoid inaccurate results or biased measures that could lead to wrong conclusions. It should be noted that the evaluation of instruments in the field of health must be multi-professional, aiming to increase phenomenon objectivity.

Thus, the final configuration of the instrument has eighteen items, divided into three domains, with response options ranging from 0 to 2. We considered the ECVIH a valid and reliable alternative to evaluate educational contents in health.

It is expected that, in subsequent studies, the clinical validation will be performed, to evaluate the efficiency of the instrument here constructed and validated as a facilitator in obtaining data on educational materials in the health area.

FUNDING

The Ceará Foundation of Support to the Scientific and Technological Development allowed financial support to the research.

Sidebar

How to cite this article:

Leite SS, Áfio ACE, Carvalho LV, Silva JM, Almeida PC, Pagliuca LMF. Construction and validation of an Educational Content Validation Instrument in Health. Rev Bras Enferm [Internet]. 2018;71(Suppl 4):1635-41. [Thematic Issue: Education and teaching in Nursing] DOI: http://dx.doi.org/10.1590/0034-7167-2017-0648

CORRESPONDING AUTHOR Sarah de Sá Leite E-mail: [email protected]

References

REFERENCES

1. Barros EJL, Santos SSC, Gomes GC, Erdmann AL. Gerontotecnologia educativa voltada ao idoso estomizado a luz da complexidade. Rev Gaucha Enferm [Internet]. 2012 [cited 2016 Nov 28];33(2):95-101. Available from: http://www.scielo.br/pdf/rgenf/v33n2/14.pdf

2. Salci MA, Maceno P, Rozza SG, Silva DMGV, Boehs AE, Heidemann ITSB. Health education and its theoretical perspectives: a few reflections. Texto Contexto Enferm[Internet]. 2013[cited 2017 Jan 12];22(1):224-30. Available from: http://www.scielo.br/pdf/ tce/v22n1/27.pdf

3. Tibúrcio MP, Melo GSM, Balduíno LSC, Freitas CCS, Costa IKF, Torres GV. Content validation of an instrument to assess the knowledge about the measurement of blood pressure. Rev Pesqui Cuid Fundam[Internet]. 2015[cited 2017 Jan 15];7(2):2475-85. Available from: http://www.seer.unirio.br/index.php/cuidadofundamental/article/view/3585/pdf_1578

4. Coluci MZO, Alexandre NMC, Milani D. Construçao de instrumentos de medida na área da saúde. Cienc Saúde Colet[Internet]. 2015[cited 2016 Nov 28];20(3):925-36. Available from: http://www.scielo.br/pdf/csc/v20n3/1413-8123-csc-20-03-00925.pdf

5. Segabinazi JD, Zortea M, Zanon C, Bandeira DR, Giacomoni CH, Hutz CS. Escala de afetos positivos e negativos para adolescentes: adaptaçao, normatizaçao e evidencias de validade. Aval Psicol[Internet]. 2012[cited 2017 Jan 31];11(1):1-12. Available from: http:// pepsic.bvsalud.org/pdf/avp/v11n1/v11n1a02.pdf

6. Rattray J, Jones MC. Essential elements of questionnaire design and development. J Clin Nurs [Internet]. 2007[cited 2017 Jan 25];16(2):234-43. Available from: http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2702.2006.01573.x/epdf

7. Cano SJ, Hobart JC. The problem with health measurement. Patient Prefer Adher [Internet]. 2011 [cited 2017 Jan 31];5:279-90. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3140310/pdf/ppa-5-279.pdf

8. Elias CSR, Silva LA, Martins MTSL, Ramos NAP, Souza MGG, Hipólito RL. When is the end? a narrative review on the termination of the school term for mentally disable students. Rev Eletr Saúde Mental Álcool Drog [Internet]. 2012[cited 2017 Jan 31];8(1):1-6. Available from: http://www.revistas.usp.br/smad/article/view/49594/53719

9. Filatro A. Design instrucional contextualizado: educaçao e tecnología. 3a ed. Sao Paulo: Senac; 2011.

10. Cardoso CS, Bandeira M, Ribeiro ALP, Oliveira GL, Caiaffa WT. Escalas de satisfaçao com o atendimento as doenças cardiovasculares: CARDIOSATIS usuario e equipe. Cienc Saúde Colet[Internet]. 2011 [cited 2016 Oct 12];16(Supl-1):1401-07. Available from: http:// www.scielo.br/pdf/csc/v16s1/a75v16s1.pdf

11. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm [Internet]. 2016 [cited 2017 Jan 31];38(3):655-62. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4909789/pdf/11096_2016_Article_257.pdf

12. Pestana MH, Gageiro JN. Análise de dados para Ciencias Sociais: a complementaridade do SPSS. 4a ed. Lisboa: Ediçöes Sílabo; 2005.

13. Áfio ACE, Balbino AC, Alves MDS, Carvalho LV, Santos MCL, Oliveira NR. Analysis of the concept of nursing educational technology applied to the patient. Rev Rene [Internet]. 2014 [cited 2016 Nov 28];15(1):158-65. Available from: http://www.redalyc.org/ pdf/3240/324030684020_2.pdf

14. Khurana S, Rao BK, Lewis LES, Bhat R, Purkayastha J, Kamath A, et al. Development and Validation of Educational Leaflet for Caregivers of Preterm Infants. J Clin Diagn Res[Internet]. 2016[cited 2017 Jan 31];10(7):1-4. Available from: https://www.ncbi.nlm. nih.gov/pmc/articles/PMC5020196/pdf/jcdr-10-YC01.pdf

15. Keszei AP, Novak M, Streiner DL. Introduction to health measurement scales. J Psychosom Res [Internet]. 2010[cited 2017 Jan 31];68(4):319-23. Available from: http://www.jpsychores.com/article/S0022-3999(10)00011-5/pdf

16. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res [Internet]. 2010[cited 2016 Nov 28];19:539-49. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2852520/ pdf/11136_2010_Article_9606.pdf

17. An YM, Jun DW, Lee SM. Development and application of low-carbohydrates and low-simple sugar nutrition education materials for non-alcoholic fatty liver disease patients. Clin Nutr Res[Internet]. 2015[cited 2016 Nov 29];4(4):250-58. Available from: https:// www.ncbi.nlm.nih.gov/pmc/articles/PMC4641987/pdf/cnr-4-250.pdf