Elsevier's Journal Metrics for the Identification of a Mainstream Journals Core: A Case Study on Mexico

Abstract

Over the past four decades there have been considerable arguments against the arbitrary use of Garfield's Impact Factor in research evaluation. Over the past decade, the emergence of databases with considerable breadth of coverage, such as Scopus and Google Scholar, together with the general demand for greater accuracy in journal assessments, has motivated the emergence of alternative journal indicators. However, the study of Latin American journals using these quantitative measures is still scarce. The purpose of this paper is to highlight the use of advanced bibliometric indicators to analyze the performance of scientific journals, and to identify a mainstream journals core in national and regional contexts. Thomson Reuters' Journal Citation Report and Elsevier's Journal Metrics were used as information sources. Four bibliometric indicators were selected to identify clusters of high-visibility journals: Impact Factor, Eigenfactor Score, Source Normalized Impact per Paper, and SCImago Journal Rank. A Self-Organizing Map (SOM) based artificial intelligence method to identify a mainstream journals core was also used. Mexican serials were selected for the case study. The coverage of Mexican journals in Web of Science and Scopus during the period 2004-2013 was studied. The performance of Mexican scientific journals was also evaluated. Mexican journal rankings based on Scopus-based metrics versus Web of Science-based metrics were compared. The most visible Mexican mainstream serials core is identified. Librarians, researchers and research managers can use alternative indicators and Self-Organizing Maps to characterize journals rankings.

Headnote

ABSTRACT

Background. Over the past four decades there have been considerable arguments against the arbitrary use of Garfield's Impact Factor in research evaluation. Over the past decade, the emergence of databases with considerable breadth of coverage, such as Scopus and Google Scholar, together with the general demand for greater accuracy in journal assessments, has motivated the emergence of alternative journal indicators. However, the study of Latin American journals using these quantitative measures is still scarce.

Objectives. The purpose of this paper is to highlight the use of advanced bibliometric indicators to analyze the performance of scientific journals, and to identify a mainstream journals core in national and regional contexts.

Methods. Thomson Reuters' Journal Citation Report and Elsevier's Journal Metrics were used as information sources. Four bibliometric indicators were selected to identify clusters of high-visibility journals: Impact Factor, Eigenfactor Score, Source Normalized Impact per Paper, and SCImago Journal Rank. A Self-Organizing Map (SOM) based artificial intelligence method to identify a mainstream journals core was also used. Mexican serials were selected for the case study.

Results. The coverage of Mexican journals in Web of Science and Scopus during the period 2004-2013 was studied. The performance of Mexican scientific journals was also evaluated. Mexican journal rankings based on Scopus-based metrics versus Web of Science-based metrics were compared.

Contributions. The most visible Mexican mainstream serials core is identified. Librarians, researchers and research managers can use alternative indicators and Self-Organizing Maps to characterize journals rankings.

INTRODUCTION

Garfield's Journal Impact Factor (JIF) has been the bibliometric indicator most commonly used by librarians, researchers and research managers. Although JIF has faced criticism, and reports of malpractices have surfaced, it is still considered by some authors the most relevant indicator to evaluate the influence of scientific journals (Zitt, 2012). However, arguments against the arbitrary use of this indicator in research evaluation are well-known, and many authors have disclosed unethical manipulations by journal editors and common misuses by individuals who lack competence in the field of quantitative studies of science (Archambault & Lariviere, 2009; Pendlebury & Adams, 2012; Smith, 2012; Weingart, 2005). Hence, bibliometricians have expressed a need for greater rigor and accuracy in journal assessments, as well as for more inclusive and viable alternatives.

The relevance of JIF is directly related to the essential role of the citation indexes created also by Eugene Garfield since 1963, currently covered by Thomson Reuters' Web of Science (WoS). These sources were considered the mainstream of scientific bibliometric analysis for more than four decades. Over the last ten years, the emergence of new citation indexes and wide-ranging scientific databases as Google Scholar or Scopus has in turn brought about the emergence of new journal indicators, long sought after by the academic community (Brown, 2011; Fragkiadaki & Evangelidis, 2014). Scopus, the database of peerreviewed literature developed by Elsevier, has become one of the main data sources for new journal indicators, which have been developed and tested with the aim of complementing and overcoming the limitations of the impact factor highlighted by the scientific literature (Leydesdorff, 2009; Torres-Salinas & Jimenez-Contreras, 2010). Journal Metrics, a new Web service launched by Elsevier in 2014, provides free accessible indicators to measure the citation impact of the journals indexed by Scopus. The impact metrics provided are based on methodologies developed by the Centre for Science and Technology Studies (CTWS) of Leiden University (The Netherlands) and the SCImago Research Group (Spain).

Among the journal metrics provided (free of charge) at this website, there were two that were considered as viable alternatives to JIF: the Source Normalized Impact per Paper (SNIP) and the SCImago Journal Rank (SJR). Thomson Reuters also took the initiative to include advanced journal indicators in its Journal Citation Report as the Eigenfactor Score (EFS) (Jacso, 2010). However, judging by Scopus' wide coverage and quality, SNIP and SJR have positioned themselves ahead of JIF and EFS as real contenders to measure the influence and prestige of scientific journals (Falagas, Kouranos, Arencibia-Jorge & Karageorgopoulos, 2008; Leydesdorff & Opthof, 2010; Moed, 2011; Schoepfel & Prost, 2009; Torres-Salinas & Jimenez-Contreras, 2010). Thus, in the race to obtain advanced bibliometric indicators as support tools for peer review, both alternatives are gaining an important degree of acceptance.

In spite of their relevance, studies about the use of these new indicators to analyze the behavior of Latin American scientific journals are still scarce. Most of the bibliometric studies of Latin American journals use Thomson Reuters' citation indexes as data sources (CollazoReyes, 2014; Collazo-Reyes, Luna-Morales, Russell, & Perez-Angon, 2008; Gomez, Sancho, Moreno & Fernandez, 1999; Macias-Chapula, 2010; Torricella-Morales, Van Hooydonk & Araujo-Ruiz, 2000).

The search for a strategy to enhance the visibility of Latin American journals and to realize their inclusion in the mainstream core, has been an objective of regional scientific policies since the beginning of the 1990s (Gomez, Sancho, Moreno & Fernandez, 1999; Meneghini, Mugnaini & Packer, 2006; Vélez-Cuartas, Lucio-Arias & Leydesdorff, 2016; Vessuri, 1995). In fact, the alliance between Scopus and SciELO in 2007 and the integration of SciELO Citation Index into the larger WoS platform in 2014, were decisive steps in the race to avoid the "underestimation" of research from Latin America and the Caribbean (Vélez-Cuartas, Lucio-Arias, & Leydesdorff, 2016). But few studies have used Scopus to either analyze regional or national outputs (Arencibia-Jorge & de Moya-Anegon, 2010; Cabrini Gracio, Tannuri de Oliveira, Gurgel, Isabel Escalona & Pulgarin Guerrero, 2013; Chinchilla-Rodríguez, Arencibia-Jorge, de Moya-Anegón & Corera-Álvarez, 2015; Wainer, Xavier & Bezerra, 2009; Zacca-Gonzalez, Chinchilla-Rodriguez, Vargas-Quesada & de Moya-Anegon, 2014; Zacca-González, Chinchilla-Rodríguez, Vargas-Quesada & de MoyaAnegón, 2015), or to observe the behavior of Latin American coverage (Collazo-Reyes, 2014). A similar situation is present in the Mexican context. The Mexican scientific output in different research fields has been reflected by the mirror of Thomson Reuters' databases (Macias-Chapula, 2013; Rojas-Sola & Jorda-Albinana, 2011; Rojas-Sola & San-AntonioGomez, 2010; Sierra-Flores, Guzman, Raga & Perez, 2009), and to our knowledge there is only one paper analyzing the behavior of a Mexican scientific journal in Scopus (Lifshitz, Halabe & Ramiro, 2014).

The purpose of this short paper is to highlight the use of advanced and novel bibliometric journal indicators to analyze, not only in the WoS but also in Scopus, the performance of Mexican scientific journals, and to identify, in terms of these bibliometric measures, the most visible Mexican mainstream journals core. The specific aims are:

1. to analyze the evolution of the coverage of Mexican journals in the Web of Science and Scopus databases over a ten-year period from 2004 to 2013;

2. to obtain and compare various Mexican journal rankings based on Scopus-based journal metrics versus WoS-based journal metrics;

3. to carry out an artificial intelligence multi-factorial classification according to four journal indicators (JIF as well as three new indicators), and to provide a visual display of the results through knowledge maps.

MATERIALS AND METHOD

Two data sources for the analysis of Mexican journals were used:

a) Journal Metrics, a new bibliometric tool developed by Elsevier,

b) Journal Citation Report, one of the classic tools of Thomson Reuters.

Four journal indicators for the characterization of Mexican journals were used (see Table 1). Impact Factor (JIF) and Eigenfactor Score (EFS) were provided by Journal Citation Report. Source Normalized Impact per Paper (SNIP) and SCImago Journal Rank (SJR) were provided by Elsevier's Journal Metrics.

* JIF: Journal Impact Factor measures the influence or impact of a scientific journal, based on citations received by papers published by this journal. The measure for a journal is calculated as JIFi = Ci/A, where Ci = Citations received in the year i by articles published by the journal in the two previous years, and A = Total of articles published by the journal in the two previous years (Garfield, 2006; Garfield & Sher, 1963).

* EFS: Eigenfactor Score is an indicator of the total influence of a journal. It uses a FUCNN-FREC Journal-Citation graph, with a five-year citation window and exclusion of journal self-citations. It imitates the original PageRank algorithm by calculating the journal influence vector, which in turn is used to calculate the Eigenfactor Score as the percentage of citations received by the journal in question from all other journals included in the graph (Bergstrom, West & Wiseman, 2008; Fragkiadaki & Evangelidis, 2014; West, Bergstrom & Bergstrom, 2010).

* SNIP: Source Normalized Impact per Paper is an indicator of the scientific impact of scientific journals that uses a source normalized approach to correct for differences in citation practices between scientific fields. It is defined as the ratio of a journal's raw impact per paper (RIP) and a journal's relative database citation potential. The measure is calculated as SNIP=RIP/(R/M), where R=citation potential and M=median database citation potential (Moed, 2010; Waltman, van Eck, van Leeuwen & Visser, 2013).

* SJR: SCImago Journal Rank indicator is a size-dependent metric that calculates the prestige per paper published in a specific journal. It is calculated by dividing the Prestige SCImago Journal Rank (PSJR) value of a journal by the number of papers published (Art) and multiplying the result by a constant value c that makes the outcome more easily readable. The measure for a journal i is calculated as SJRSi = c x (PSJR/Arti) (GonzalezPereira, Guerrero-Bote, & Moya-Anegon 2010; Guerrero-Bote & Moya-Anegon, 2012).

EFS, SNIP and SJR approaches are considered more robust than JIF, taking into account how they evaluate incoming citations. The JIF approach counts all citations received without considering the significance of those citations. However, for a given number of citations, citations from highly cited journals will result in higher values of EFS and SJR. On the other hand, the SNIP approach focuses on the citation context of a subject field. The impact of a single citation in this case depends on the total number of citations in the subject field; that is, the value of a single citation is higher in areas where citations are less likely, and vice versa. Table 1 shows the characteristics of each of these indicators.

A total of 34 Mexican journals covered by Web of Science and/or Scopus were selected for the study and top 10 rankings based on these indicators were obtained, to identify and characterize the most visible core of Mexican journals in terms of these indicators.

Finally, self-organizing artificial neural networks trained with the Self-Organizing Map (SOM) algorithm (Kohonen, 2013; Kohonen & Somervuo, 2002) were used to construct a two-dimensional display, in which we exhibit a mapping of the journal distribution according to each one of the four selected bibliometric indicators, and highlight the journals that constitute the most visible Mexican mainstream journals core.

The design of a neural network model based on Self-Organizing Maps consists of a neural network with two dimensional SOM hexagonal grids. Each hexagon is an artificial neuron which occupies a place where data points can be assigned. This allocation is the result of the training algorithm of the neural network, which is an iterative process in which the network projects similar patterns in enclosed spaces (hexagons). The application of the neural network's non-linear SOM algorithm allows the neural network to project data on the grid. This projection is useful for analysis and visualization through mapping the structure of data groups that share the same multi-dimensional space in a two-dimensional mapping. The use of this technique in the current paper offers an easy way to analyze common or atypical behaviors of the four studied indicators.

The methodology for the visualization of Self-Organizing Maps (SOM) was implemented in the software LabSOM, developed by the Laboratory of Nonlinear Dynamics at the Faculty of Sciences of the National Autonomous University of Mexico, and the Company Tecnologías Inteligentes y Modelación de Sistemas (TIMS).

RESULTS AND DISCUSSION

First, we analyzed the evolution of the coverage of Mexican journals by WoS and Scopus. Figure 1 shows that the Mexican output in these two main citation indexes has grown significantly during the past decade, in spite of the fact that these databases are not designed to take in account the emergent scientific agendas of Latin American countries (AguadoLópez, Garduño-Oropeza, Rogel-Salazar & Zúñiga-Roca, 2012).

In 2004, when Scopus was launched, there were only 11 Mexican journals listed in the Journal Citation Report, but by 2011 this number had increased to 37. This "mainstream" representation of the second most productive country from Latin America, in a context characterized by institutional endogamy and language barriers, was very limited compared to the coverage of other regional journal collections as SciELO (www.scielo.org.mx), RedALyC (www.redalyc.org) or LATINDEX (www.latindex.unam.mx). In fact, these 37 journals represented only 4.3% of Mexican journals included in the LATINDEX catalog. 21.3% of Mexican serials were covered by RedALyC, and 29.8% of Mexican scholarly serials were covered by SciELO.

In contrast, the Scopus coverage was more comprehensive, with a total of 52 serials indexed in 2004, reaching a total of 91 in 2013. These represent 73.4% of Mexican SciELO coverage, which is evidence of the breadth of Scopus' biomedical coverage. However, in RedALyC and LATINDEX, the coverage grew to 52.3 and 10.9% respectively, revealing a considerable gap in Social Sciences, Arts and Humanities coverage. Clearly, 10.9% coverage in high visibility journals still is not a representative number for Mexico.

In addition, a change in Web of Science's policy coverage was observed during the period. The total number of Mexican mainstream journals increased threefold, and coverage in Scopus increased from 20% to 40%. This significant increase had been clearly observed since 2009, when the Scopus development was recognized by the entire scientific community and even the OCDE started to use Scopus-based statistics in scientometric reports.

The battery of journal indicators that we selected to characterize the most visible Mexican journals, include two indicators from Thomson Reuters' Journal Citation Report and the other two from Elsevier's Journal Metrics. As was to be expected, Table 2 shows that there is an important degree of correlation among these indicators. The pairs (JIF, SJR) and (EFS, SJR) exhibit the highest correlation values, while (SNIP, EFS) and (SNIP, SJR) exhibit lower ones. The independence of the SNIP indicator with regard to the other three indicators produces a remarkable difference in the top 10 journal rankings for the year 2013, which is evident in Table 3.

In this table we displayed the four top 10 journal rankings after taking into consideration each of the four journal indicators: JIF, EFS, SJR and SNIP. Since there is an important degree of correlation among these indicators, four Mexican journals are included in the top 10 rankings of the four indicators (Annals of Hepatology, Salud Pública de Mexico, Revista Mexicana de Ciencias Geológicas, and Atmósfera), and one of them (Annals of Hepatology) is the leader according to JIF, EFS and SJR. These three indicators allow the inclusion of another two journals (Revista Mexicana de Astronomía y Astrofísica, and Ciencias Marinas) in the top 10 rankings. However, it is in the SNIP top 10 that the highlights emerge:

1. There are two journals (Advances in Applied Clifford Algebras and Revista Mexicana de Sociología) that are the most visible in Scopus according to SNIP, which are not even indexed by the Web of Science.

2. The SNIP top 10 ranking includes three journals from the Social Sciences (Revista Mexicana de Sociología, Revista Latinoamericana de Investigación en Matemática Educativa and Economía Mexicana, Nueva Época), which are not in the top 10 ranking according to the other journal indicators: JIF, EFS and SJR.

There is no doubt that the high correlation observed is evidence of the complementary role of the new indicators proposed by the Elsevier's Journal Metrics. However, the differences in coverage between Scopus and Web of Sciences support the position of SJR and SNIP as real contenders of JIF in scientometric environments (Falagas, Kouranos, ArencibiaJorge & Karageorgopoulos, 2008; Leydesdorff, 2009), and the thematic normalization involved in SNIP calculation allows the authors of this paper to consider this indicator as the strongest alternative to JIF in research evaluation exercises.

The advantages of the Scopus-based indicators can also be inferred from bibliometric maps based on self-organizing artificial neural networks (Figure 2). The visibility areas (red color) observed in the maps based on SNIP and SJR are more extensive. There is less distance between the most visible journals and the rest, which is an obvious improvement on the Scopus-based bibliometric measures for journal evaluation. Other Latin American countries or research fields can be explored in this way to assess the proposed method.

CONCLUSION

Thomson Reuters and Elsevier are currently two of the main providers of wide spectrum scientific publication information. The evolution of their databases, WoS and Scopus over the past decade corresponds with the conspicuous increase in the number of Mexican journals indexed in these databases in 2009 and 2010. However, our analysis shows that the coverage of Mexican journals by these international high visibility databases is still very limited.

The bibliometric tools of Thomson Reuters and Elsevier, Journal of Citation Reports and Journal Metrics, provide useful resources for quantitative journal assessment. In particular, the new alternatives to the classical JIF that we have analyzed in this paper have enriched the bibliometric toolbox for journal ranking. In order to be used for journal characterization, all of these bibliometric indicators (Journal Impact Factor, Eigenfactor Score, Source Normalized Indicator per Paper, and SCImago Journal Rank) are valuable, and each of the journal rankings that they provide could offer benefits for the decision-making process in which they could be involved.

Although the study carried out in a sample of the most visible Mexican journals shows an important degree of correlation among these four bibliometric indicators, our analysis confirms that there are still differences which provide complementary views that could be useful for journal assessment. SOM techniques allow for better characterization of each measure. In particular, by taking into account the diversity in citation practices between scientific fields, the SNIP indicator of Elsevier's Journal Metrics provides a ranking for the studied sample that is radically different to those provided by the other three bibliometric indicators. Taking this indicator into consideration, Elsevier's Journal Metrics allows us to obtain a relevant multi-dimensional high-visibility Mexican journals core which is more comprehensive, not only for Social Science journals but also for technical ones.

ACKNOWLEDGEMENTS

This work was made possible through the research agenda of TIMS Company Tecnologías Inteligentes y Modelación de Sistemas. The authors want to thank all members of the Laboratory of Nonlinear Dynamics at the Faculty of Sciences of the National Autonomous University of Mexico.

References

REFERENCES

Aguado-López, E., Garduño-Oropeza, G.A., Rogel-Salazar, R., & Zúñiga-Roca, M F. (2012). The need and viability of a mediation index in Latin American scientific production and publication. Aslib Proceedings, 64(1), 8-31.

Archambault, E., & Lariviere, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635-649. Retrieved from doi: 10.1007/s11192-007-2036-x

Arencibia-Jorge, R., & de Moya-Anegon, F. (2010). Challenges in the study of Cuban scientific output. Scientometrics, 83(3), 723-737. Retrieved from doi: 10.1007/s 11192009-0150-7

Bergstrom, C.T., West, J. D., & Wiseman, M.A. (2008). The eigenfactor (TM) metrics. Journal of Neuroscience, 25(45), 11433-11434. Retrieved from doi: 10.1523/jneurosci.0003-08.2008

Brown, T. (2011). Journal quality metrics: Options to consider other than impact factors. American Journal of Occupational Therapy, 65(3), 346-350. Retrieved from doi: 10.5014/ajot.2011.001396

Cabrini Gracio, M.C., Tannuri de Oliveira, E.F., Gurgel, J. d. A., Isabel Escalona, M., & Pulgarin Guerrero, A. (2013). Dentistry scientometric analysis: a comparative study between Brazil and other most productive countries in the area. Scientometrics, 95(2), 753-769. Retrieved from doi: 10.1007/s11192-012-0847-x

Collazo-Reyes, F. (2014). Growth of the number of indexed j ournals of Latin America and the Caribbean: the effect on the impact of each country. Scientometrics, 98(1), 197-209. doi: 10.1007/s 11192-013-1036-2

Collazo-Reyes, F., Luna-Morales, M E., Russell, J.M., & Perez-Angon, M.A. (2008). Publication and citation patterns of Latin American & Caribbean journals in the SCI and SSCI from 1995 to 2004. Scientometrics, 75(1), 145-161. Retrieved from doi: 10.1007/s 11192-007-1841 -6

Chinchilla-Rodríguez, Z., Arencibia-Jorge, R., de Moya-Anegón, F., & Corera-Álvarez, E. (2015). Some patterns of Cuban scientific publication in Scopus: the current situation and challenges. Scientometrics, 103(3), 779-794. Retrieved from doi: 10.1007/s 11192015-1568-8

Collazo-Reyes, F. (2014). Growth of the number of indexed j ournals of Latin America and the Caribbean: The effect on the impact of each country. Scientometrics, 98(1), 197209.

Collazo-Reyes, F., Luna-Morales, M., Russell, J., & Pérez-Angón, M. A. (2008). Publication and citation patterns of Latin American & Caribbean journals in the SCI and SSCI from 1995 to 2004. Scientometrics, 75(1), 145-161.

Falagas, M.E., Kouranos, V.D., Arencibia-Jorge, R., & Karageorgopoulos, D.E. (2008). Comparison of SCImago journal rank indicator with journal impact factor. Faseb Journal, 22(8), 2623-2628. Retrieved from doi: 10.1096/fj.08-107938

Fragkiadaki, E., & Evangelidis, G. (2014). Review of the indirect citations paradigm: theory and practice of the assessment of papers, authors and journals. Scientometrics, 99(2), 261-288. Retrieved from doi: 10.1007/s11192-013-1175-5

Garfield, E. (2006). The history and meaning of the journal impact factor. Jama-Journal of the American Medical Association, 295(1), 90-93. Retrieved from doi: 10.1001/jama.295.1.90

Garfield, E., & Sher, I.H. (1963). New factors in evaluation of scientific literature through citation indexing. American Documentation, 14(3), 195-201.

Gomez, I., Sancho, R., Moreno, L., & Fernandez, M.T. (1999). Influence of Latin American journals coverage by international databases. Scientometrics, 46(3), 443-456. Retrieved from doi: 10.1007/bf02459603

Gonzalez-Pereira, B., Guerrero-Bote, V.P., & Moya-Anegon, F. (2010). A new approach to the metric of journals' scientific prestige: The SJR indicator. Journal ofInformetrics, 4(3), 379-391. Retrieved from doi: 10.1016/jjoi.2010.03.002

Guerrero-Bote, V. P., & Moya-Anegon, F. (2012). A further step forward in measuring journals' scientific prestige: The SJR2 indicator. Journal of Informetrics, 6(4), 674-688. Retrieved from doi: 10.1016/j .joi.2012.07.001

Jacso, P. (2010). Eigenfactor and article influence scores in the journal citation reports. Online information review, 34(2), 339-348. Retrieved from doi: 10.1108/14684521011037034

Kohonen, T. (2013). Essentials of the self-organizing map. Neural Networks, 37, 52-65. Retrieved from doi: 10.1016/j.neunet.2012.09.018

Kohonen, T., & Somervuo, P. (2002). How to make large self-organizing maps for nonvectorial data. Neural Networks, 15(8-9), 945-952.

Leydesdorff, L. (2009). How are new citation-based journal indicators adding to the bibliometric toolbox? Journal of the American Society for Information Science and Technology, 60(7), 1327-1336. Retrieved from doi: 10.1002/asi.21024

Leydesdorff, L., & Opthof, T. (2010). Scopus's source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61(11), 2365-2369. Retrieved from doi: 10.1002/asi.21371

Lifshitz, A., Halabe, J., & Ramiro, M.H. (2014). Medical Gazette of Mexico in the bibliometrics. Gaceta Medica De Mexico, 150(2), 121-121.

Macias-Chapula, C.A. (2010). Influence of local and regional publications in the production of public health research papers in Latin America. Scientometrics, 84(3), 703-716. Retrieved from doi: 10.1007/s11192-009-0153-4

Macias-Chapula, C.A. (2013). Comparative analysis of health public policy research results among Mexico, Chile and Argentina. Scientometrics, 95(2), 615-628. Retrieved from doi: 10.1007/s11192-012-0855-x

Meneghini, R., Mugnaini, R., & Packer, A.L. (2006). International versus national oriented Brazilian scientific journals. a scientometric analysis based on SciELO and JCR-ISI databases. Scientometrics, 69(3), 529-538. Retrieved from doi: 10.1007/s11192-0060168-z

Moed, H.F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265-277. Retrieved from doi: 10.1016/jjoi.2010.01.002

Moed, H.F. (2011). SNIP and beyond. In E. Noyons, P. Ngulube & J. Leta (Eds.), Proceedings ofISSI2011: The 13th Conference of the International Society for Scientometrics and Informetrics, Vols 1 and 2 (pp. 584-594).

Pendlebury, D.A., & Adams, J. (2012). Comments on a critique of the Thomson Reuters journal impact factor. Scientometrics, 92(2), 395-401. Retrieved from doi: 10.1007/s11192-012-0689-6

Rojas-Sola, J.I., & Jorda-Albinana, B. (2011). Bibliometric analysis of Mexican scientific production in hydraulic engineering based on journals in the science citation indexexpanded database (1997-2008). Tecnología Y Ciencias Del Agua, 2(4), 195-213.

Rojas-Sola, J.I., & San-Antonio-Gomez, C. (2010). Bibliometric analysis of Mexican scientific publications in the category engineering, chemical from the Web of Science data base (1997-2008). Revista Mexicana De Ingenieria Quimica, 9(3), 231-240.

Schoepfel, J., & Prost, H. (2009). Comparison of SCImago journal rank indicator (SJR) with JCR journal impact factor (IF) for French journals. Psychologie Francaise, 54(4), 287305. Retrieved from doi: 10.1016/j.psfr.2009.07.002

Sierra-Flores, M.M., Guzman, M.V., Raga, A.C., & Perez, I. (2009). The productivity of Mexican astronomers in the field of outflows from young stars. Scientometrics, 81(3), 765-777. Retrieved from doi: 10.1007/s11192-008-2264-8

Smith, D.R. (2012). Impact factors, scientometrics and the history of citation-based research. Scientometrics, 92(2), 419-427. Retrieved from doi: 10.1007/s11192-012-0685-x

Torres-Salinas, D., & Jimenez-Contreras, E. (2010). Introduction and comparative study of the new scientific journals citation indicators in journal citation reports and scopus. Profesional De La Informacion, 19(2), 201-207. Retrieved from doi: 10.3145/epi.2010.mar.12

Torricella-Morales, R.G., Van Hooydonk, G., & Araujo-Ruiz, J.A. (2000). Citation analysis of cuban research. Part 1. A case study: the Cuban journal of agricultural science. Scientometrics, 47(2), 413-426. Retrieved from doi: 10.1023/a: 1005655530277

Vélez-Cuartas, G., Lucio-Arias, D., & Leydesdorff, L. (2016). Regional and global science: publications from Latin America and the Caribbean in the scielo citation index and the Web of science. El Profesional de la Información, 25(1), 35-48.

Vessuri, H. (1995). Recent strategies for adding value to scientific journals in Latin-America Scientometrics, 34(1), 139-161. Retrieved from doi: 10.1007/bf02019178

Wainer, J., Xavier, E.C., & Bezerra, F. (2009). Scientific production in computer science: a comparative study of Brazil and other countries. Scientometrics, 81(2), 535-547. Retrieved from doi: 10.1007/s11192-008-2156-y

Waltman, L., van Eck, N.J., van Leeuwen, T.N., & Visser, M.S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7(2), 272-285. Retrieved from doi: 10.1016/jjoi.2012.11.011

Weingart, P. (2005). Impact of bibliometrics upon the science system: Inadvertent consequences? Scientometrics, 62(1), 117-131. Retrieved from doi: 10.1007/s 11192005-0007-7

West, J.D., Bergstrom, T.C., & Bergstrom, C.T. (2010). The Eigenfactor Metrics (TM): a network approach to assessing scholarly journals. College & Research Libraries, 71(3), 236-244.

Zacca-Gonzalez, G., Chinchilla-Rodriguez, Z., Vargas-Quesada, B., & de Moya-Anegon, F. (2014). Bibliometric analysis of regional Latin America's scientific output in Public health through SCImago journal & country rank. Bmc Public Health, 14. Retrieved from doi: 10.1186/1471-2458-14-632

Zacca-González, G., Chinchilla-Rodríguez, Z., Vargas-Quesada, B., & de Moya-Anegón, F. (2015). Patterns of communication and impact of the Cuban scientific output in public health. Revista Cubana de Salud Publica, 41(2), 200-216.

Zitt, M. (2012). The journal impact factor: angel, devil, or scapegoat? A comment on J.K. Vanclay's article 2011. Scientometrics, 92(2), 485-503. Retrieved from doi: 10.1007/s11192-012-0697-6