Impact Factor: an appropriate criterion for the Qualis journals classification in the Pharmacy area?

Abstract

[...]the number of journals is variable and, consequently, the total number of citations. [...]the Journal Impact Factor (JIF) allows comparisons among journals only within a given subject category, as explained by the very databases. According to its creator, this indicator does not allow the comparison among journals of different categories, since these can have great variability of size and citation behaviors. JIF Percentil Web of Science / Thomson Reuters It calculates the JIF percentiles in each subject category, considering the ranking of the journal in the respective category. Since a journal may appear in more than one category, it will display a JIF percentile value for each category. TABLE II Bibliometric data of some periodicals classified in the Qualis 2015 in the area of Pharmacy JOURNAL QUALIS JIF (JCR) Cite Score SJR SNIP Average JIF percentile Average CiteScores percentile JIF percentile (in the category) Category Web of Science Cite Scores percentile (in the category) Category Scopus INDUSTRIAL CROPS AND PRODUCTS A2 3,449 3,70 1,064 1,707 91,3 94 89,286 Agricultural Engineering; 94 Agronomy and Crop Science 93,373 Agronomy SOCIAL SCIENCE & MEDICINE B1 2,814 3,48 1,894 1,835 86,3 94 90,523 Public, environmental & occupational health SSCI ; 97 Health (social science) 88,462 Social sciences, biomedical SSCI 91 History and Philosophy of Science 80,058 Public, environmental & occupational health SCIE JOURNAL OF ETHNOPHARMACOLOGY A2 3,055 3,54 1,156 1,551 80,5 83,5 73,729 Chemistry, Medicinal; 86 Drug Discovery; 84,091 Integrative & Complementary Medicine; 81 Pharmacology 70,307 Pharmacology & Pharmacy; 83,014 Plant Sciences INTERNATIONAL JOURNAL OF NANOMEDICINE A1 4,32 4,88 1,351 1,322 76,7 90,4 69,277 Nanoscience & Nanotechnology; 85 Bioengineering; 84,118 Pharmacology & Pharmacy 86 Biomaterials; 94 Biophysics; 95 Drug Discovery; 92 Organic Chemistry CYTOMETRY PART A A2 3,181 2,32 1,63 0,917 59,7 68 69,481 Biochemical research methods; 82 Pathology and Forensic Medicine 50 Cell biology; 81 Histology 41 Cell Biology ANALYTICAL METHODS B1 1,915 1,98 0,623 0,626 55,7 74,67 47,333 Chemistry, Analytical; 60 Analytical Chemistry; 65,2 Food Science & Technology; 77 Chemical Engineering (all); 54,651 Spectroscopy 87 Engineering (all) According to the document ‘Considerations on the Qualis Journals Classification 2016’, the Pharmacy area used as a classification criterion the Impact Factor indicator from database (1) Web of Science/Thomson Reuters for strata A1 to A2, and a combination of this and the SJR (SCImago Journal Rank) indicator from database (2) Scopus/SCImago/Elsevier for strata B1 to B4.

Annually, during the evaluation process of the Graduate Programs in Brazil, the discussion about the Qualis journals classification resurfaces. In late 2016, the CAPES’s (Brazilian Federal Agency for Support and Evaluation of Postgraduate Education) pharmacy webpage¹ published three important documents: (i) the 2017 Document of the Pharmacy Area, (ii) the Considerations on the Qualis Journals Classification 2016, and (iii) the article by Rita de Cassia Barradas ^{Barata “Ten things you should know about the Qualis” (Barata, 2016}). These documents explain in detail how the criteria for the journals classification in the scope of Postgraduate Programs evaluation are defined, including the purposes, applications and limitations of the system.

The system is based on strata pre-established by the CAPES Board of Directors, where A1 <A2, A1 + A2 ≤ 25% and A1 + A2 + B1 ≤ 50% of the total number of journals in which academics published their articles in the time period under review. The criteria for defining and distributing the scientific journals in each stratum are established by respective areas of evaluation, which may result in different classifications for the same journal in each of these areas. In view of that, this is an opportune time to discuss the criteria used by the Pharmacy area towards a classification that more well-balancedly comprises the various fields of knowledge related to Postgraduate Programs in the Pharmacy area.

Concerning the two bibliographic databases used by the area - (1) Web of Science® (former Web of Knowledge)/Thomson Reuters and (2) Scopus/SCImago/Elsevier -, it is crucial to understand some aspects of their structuring and purposes, and the bibliometric indicators used to analyze the journals.

Web of Science® is a comprehensive research platform whose rights belong to Thomson Reuters Publishing, with publications in the areas of science, social sciences, arts and humanities. The database allows evaluating and comparing journals with citation data drawn from approximately 12,000 academic and technical journals and conference proceedings from more than 3,300 publishers in over 60 countries. It is connected to the InCites tool and the Journal Citation Reports (JCR) module so it provides citation metrics and indicators of contents linked to Web of Science.

Scopus is a database whose rights belong to Elsevier Publishing and contains articles, abstracts, conference proceedings, books, among others, with about 67 million records. In 2016, the database counted 22,794 peer-reviewed journals, of which 3,643 are full open access. In the health area, it offers full coverage of the Medline database.

For the journals to be indexed in those databases, they are analyzed according to criteria pre-established by their respective publishers. The initiative is usually from the publisher of the journal and, once approved, maintenance expenses are to be paid. For this reason, the first limitation of such systems concerns the indexed journals databases themselves, and consequently the market dispute between two of the world’s largest scientific publishers (^{Oosthuizen, Fenton, 2014}). The two databases in question offer bibliometric indicators to measure the prestige of the indexed journals. These indicators are essentially based on citation indices, usually from recent years, referred to as “impact factor”.

However, the pattern of article citation is highly variable in the various fields of knowledge. Examples that illustrate this point are the journals in the field of mathematics, which generally accept about ten references and show a tendency of citations of old, usually classic, articles on the subject. Another example is the field of biological sciences, whose journals accept a greater number of references (about 40) and the cited articles tend to be recent ones (^{Fonseca, 2015}; ^{Andrés, 2011}; ^{Pendlebury, 2009}). In this sense, the two databases under analysis classify the journals into categories, considering the scope of the journal and its characteristics. For instance, in 2015 the median JCR impact factor for the Mathematics category was 0.614, while for the Biochemistry & Molecular Biology category it was 2,670. In addition, the number of journals is variable and, consequently, the total number of citations.

Therefore, the Journal Impact Factor (JIF) allows comparisons among journals only within a given subject category, as explained by the very databases. The two databases in question provide other metrics for the comparison among different categories, which are displayed in Table I.

Table I Bibliometric indicators available from the databases Web of Science/Thomson Reuters and Scopus/SCImago/Elsevier

Indicator	Database	Description
JIF Journal Impact Factor	Web of Science / Thomson Reuters	It measures the citation index of the journal, that is, the number of citations received by articles published in the journal in the two years prior to the evaluation, divided by the number of articles published in the period, including self-citations. It considers original articles, reviews and conference proceedings. According to its creator, this indicator does not allow the comparison among journals of different categories, since these can have great variability of size and citation behaviors.
JIF Percentil	Web of Science / Thomson Reuters	It calculates the JIF percentiles in each subject category, considering the ranking of the journal in the respective category. Since a journal may appear in more than one category, it will display a JIF percentile value for each category.
JIF Quartile	Web of Science / Thomson Reuters	As a comparative measure, based on JIF percentiles in each subject category, journals are classified into quartiles (Q1, Q2, Q3 and Q4) in the respective categories. As a journal may appear in more than one category, it can have more than one “Q”.
Average JIF percentil	Web of Science / Thomson Reuters	With the percentiles of each subject category in which the journal is classified, an average is calculated. It allows, according to the database, comparing journals of different areas. (http://www.istl.org/09-spring/refereed1.html)
CiteScore	Scopus/SCImago/ Elsevier	It considers the number of citations received in the three years prior to the evaluation, divided by the number of documents published in the same period. It considers original articles, reviews, conference proceedings, editorials, errata, letters, notes, and short surveys.
CiteScore rank	Scopus/SCImago/ Elsevier	It calculates the CiteScore percentiles in each category of the Scopus database, considering the ranking of the journal in the respective category. A journal will have a percentile value for each category in which it is ranked.
SJR SCImago Journal Rank	Scopus/SCImago/ Elsevier	It measures the weighted citations received by the journal. The indicator is calculated using an iterative algorithm that distributes prestige values among journals until a stable solution is reached. The weighting considers the category and the prestige of the citation.
SNIP Source Normalized Impact per Paper	Centre for Science and Technology Studies (CWTS), University of Leiden/ Scopus/ Elsevier	It measures the citation impact of scientific journals. It considers the categories in which the journals are classified, smoothing out differences such as the number of citations per article, the limit of references accepted and the speed of the publication process. It allows comparing among journals of different categories within the Scopus database.

TABLE II Bibliometric data of some periodicals classified in the Qualis 2015 in the area of Pharmacy

JOURNAL	QUALIS	JIF (JCR)	Cite Score	SJR	SNIP	Average JIF percentile	Average CiteScores percentile	JIF percentile (in the category)	Category Web of Science	Cite Scores percentile (in the category)	Category Scopus
INDUSTRIAL CROPS AND PRODUCTS	A2	3,449	3,70	1,064	1,707	91,3	94	89,286	Agricultural Engineering;	94	Agronomy and Crop Science
								93,373	Agronomy
SOCIAL SCIENCE & MEDICINE	B1	2,814	3,48	1,894	1,835	86,3	94	90,523	Public, environmental & occupational health SSCI ;	97	Health (social science)
								88,462	Social sciences, biomedical SSCI	91	History and Philosophy of Science
								80,058	Public, environmental & occupational health SCIE
JOURNAL OF ETHNOPHARMACOLOGY	A2	3,055	3,54	1,156	1,551	80,5	83,5	73,729	Chemistry, Medicinal;	86	Drug Discovery;
								84,091	Integrative & Complementary Medicine;	81	Pharmacology
								70,307	Pharmacology & Pharmacy;
								83,014	Plant Sciences
INTERNATIONAL JOURNAL OF NANOMEDICINE	A1	4,32	4,88	1,351	1,322	76,7	90,4	69,277	Nanoscience & Nanotechnology;	85	Bioengineering;
								84,118	Pharmacology & Pharmacy	86	Biomaterials;
										94	Biophysics;
										95	Drug Discovery;
										92	Organic Chemistry
CYTOMETRY PART A	A2	3,181	2,32	1,63	0,917	59,7	68	69,481	Biochemical research methods;	82	Pathology and Forensic Medicine
								50	Cell biology;	81	Histology
										41	Cell Biology
ANALYTICAL METHODS	B1	1,915	1,98	0,623	0,626	55,7	74,67	47,333	Chemistry, Analytical;	60	Analytical Chemistry;
								65,2	Food Science & Technology;	77	Chemical Engineering (all);
								54,651	Spectroscopy	87	Engineering (all)

According to the document ‘Considerations on the Qualis Journals Classification 2016’, the Pharmacy area used as a classification criterion the Impact Factor indicator from database (1) Web of Science/Thomson Reuters for strata A1 to A2, and a combination of this and the SJR (SCImago Journal Rank) indicator from database (2) Scopus/SCImago/Elsevier for strata B1 to B4. In these cases, the indicator with the highest value was chosen to classify each journal in the mentioned strata. The indexed journals in the databases PubMed/Medline, Scielo and LILACS - instead of ISI/Web of Knowledge/Thomson Reuters and Scopus/SCImago/Elsevier - determined the classification B5.

The ‘2017 Document of the Pharmacy Area’ emphasizes the multidisciplinary and interdisciplinary nature of the Pharmacy area, which is evidenced by the diversity of the professors’ doctoral careers, the engagement of professors in other programs of different CAPES evaluation areas, and the profile of publications in the area.

Regarding the publications in the area, the document shows that the 25 scientific journals most used by professors from Pharmacy programs represent less than 2.0% of the Qualis of the area and more than 15.0% of the articles published in the period under consideration. These journals are classified in a wide variety of categories in both the Web of Science and Scopus databases. These data express the need for indicators that take such a diversity into account. Table I shows some examples of journals, in the 2015 Qualis classification, the categories to which the journals belong in the two databases, and the respective indicators.

Initially, it can be seen that the values attributed to the JIF and CiteScore indicators show a greater proximity between themselves than between those to JIF and SJR. This was expected since the SJR indicator measures the weighted citations, considering the categories.

However, the percentiles of the journals’ ranking in the respective categories indicate a variation, which is a consequence of the variation in the citation profile of the journals, as per the characteristics of the fields of knowledge. Therefore, it would be more appropriate to use the journals’ ranking in the categories to compare journals of different categories. The JCR database, particularly, provides each journal with the Average JIF % indicator; in the case of the Scopus database, the corresponding value can be calculated.

Although the JIF and CiteScore indicators, and respective percentiles, are based on similar assumptions, the values are different for the same journal, since they use their own systems, which depend on the journal’s registration in the database. The two databases under analysis are key for the Pharmacy area, especially in strata A1 and A2, as most journals are registered in the two of them. With respect to the percentiles, among the 25 most cited journals used in the area, the majority of them has a higher average percentile value in the categories of the Scopus database than in that of JCR.

Finally, it is worth noting that these indicators have been internationally discussed as to their applicability for evaluating the quality of publications (^{Fonseca, 2015}). The use of the designation ‘Impact Factor’ for the citation index calculation in a recent period leads to the misinterpretation that this represents the impact of the journal. The citation per se does not necessarily mean that the article has quality. The classic example is the publication on cold fusion, which received between the years 1988-1992 approximately 700 citations, but largely negative ones (http://www.scielo.br/pdf/qn/v22n3/1101.pdf). Moreover, these indicators measure only recent citations and do not consider the classic references, which are generally of greater academic impact.

Considering that both the (1) Web of Science/Thomson Reuters and (2) Scopus/SCImago/Elsevier databases provide bibliometric indicators that allow comparing the impact/quality of the journals, it seems to be outdated to classify the journals based on the impact factor using absolute counts instead of relative measures (^{Pendlebury, 2009}).

Another point to consider about the system is the induced preference for publishing in high-ranked journals, which can be classified in a certain stratum - regardless of the bibliometric indicators used - due to their relevance to the area. The Pharmacy area has four induced journals, two of them are classified as B2, one B3, and the other B5.

Considering that the journals classification criteria in the different areas of the Qualis system evaluation are defined by each corresponding area, it is worth noting that there is space and an urgent need to deeply discuss the fact that, ultimately, the evaluation should underline aspects such as: the current system’s real contribution to the quality assessment of the publications; possible biases; and the perspectives towards proposing a more balanced and consistent system embracing the multi- and interdisciplinary nature of the Pharmacy area.

1https://www.capes.gov.br/component/content/article/44-avaliacao/4671-farmacia

References

Andrés A. Evaluating research using impact and Hirsch factors. Europhysics News. 2011:42(2):29-31.

Barata A. Dez coisas que você deveria saber sobre o Qualis. Rev Bras Pós-grad. 2016;13(30):13-40.

Fonseca, J. Para além do fator de impacto: O artigo científico e a disseminação de conhecimento em Medicina. Rev Port Imunoalergol. 2015;23(2):67-9.

Oosthuizen JC, Fenton JE. Alternatives to the impact fator. Surgeon. 2014;12(4):239- 243.

Pendlebury DA. The use and misuse of journal metrics and other citation indicators. Arch Immunol Ther Exp. 2009;57(1):1-11.