Journal Impact Factor: Do the Numerator and Denominator Need Correction?

Abstract

To correct the incongruence of document types between the numerator and denominator in the traditional impact factor (IF), we make a corresponding adjustment to its formula and present five corrective IFs: IF_Total/Total, IF_Total/AREL, IF_AR/AR, IF_AREL/AR, and IF_AREL/AREL. Based on a survey of researchers in the fields of ophthalmology and mathematics, we obtained the real impact ranking of sample journals in the minds of peer experts. The correlations between various IFs and questionnaire score were analyzed to verify their journal evaluation effects. The results show that it is scientific and reasonable to use five corrective IFs for journal evaluation for both ophthalmology and mathematics. For ophthalmology, the journal evaluation effects of the five corrective IFs are superior than those of traditional IF: the corrective effect of IF_AR/AR is the best, IF_AREL/AR is better than IF_Total/Total, followed by IF_Total/AREL, and IF_AREL/AREL. For mathematics, the journal evaluation effect of traditional IF is superior than those of the five corrective IFs: the corrective effect of IF_Total/Total is best, IF_AREL/AR is better than IF_Total/AREL and IF_AREL/AREL, and the corrective effect of IF_AR/AR is the worst. In conclusion, not all disciplinary journal IF need correction. The results in the current paper show that to correct the IF of ophthalmologic journals may be valuable, but it seems to be meaningless for mathematic journals.

About the Authors:

Xue-Li Liu

* E-mail: [email protected]

Affiliations Henan Research Center for Science Journals, Xinxiang Medical University, Xinxiang, Henan Province, China, Periodicals Publishing House, Xinxiang Medical University, Xinxiang, Henan Province, China

Shuang-Shuang Gai

Affiliations Periodicals Publishing House, Xinxiang Medical University, Xinxiang, Henan Province, China, Management Institute, Xinxiang Medical University, Xinxiang, Henan Province, China

Jing Zhou

Affiliations Periodicals Publishing House, Xinxiang Medical University, Xinxiang, Henan Province, China, Management Institute, Xinxiang Medical University, Xinxiang, Henan Province, China

Introduction

One of the most famous researchers in bibliometrics, Garfield [1] first proposed the term of impact factor (IF) in his paper published in Science in 1955. At first, IF meant citations to articles, not the current IF used for evaluating journals [2]. Journal IF is the natural result of the establishment of the Science Citation Index (SCI) database. In the 1960s, supported by the US National Institutes of Health, the Genetics Citation Index was established successfully, which led to the direct generation of the SCI database. Garfield and Sher then proposed IF as an indicator for evaluating the academic impact of journals in 1963, and IF was applied to assist the selection of source journals in the SCI database [3–5]. In 1972, Garfield [6] formally established the concept and formula of journal IF. However, IF really became a journal evaluation indicator in 1975, when the Journal Citation Reports (JCR) was established, and ever since has been one of the most authoritative scientometric indicators for assessing the international status and academic impact of journals [7].

To be exact, IF is defined as the number of citations within a given year of items published by a journal in the preceding two years divided by the number of citable items published by the journal in these two years [8]. Obviously, the numerator of IF is the number of citations of all types of items, whereas the denominator just includes citable items. Thus, it is undeniable that the design formula of IF is not perfect, and this fact has attracted much attention and aroused much controversy in academia.

Thomson Reuters divides source items into citable items and non-citable items. Citable items include only original research articles and reviews. Meeting abstracts, editorials, letters, news items, corrections, book reviews, biographical items, reprints, and so on are all rated as non-citable items [9–11]. In fact, non-citable items are not uncitable, some that are published by many journals, especially several prestigious international journals, are cited frequently and do contribute to a journal’s citations [12–13]. When these non-citable items are cited, their citations are counted in the numerator of IF, but are excluded in the denominator. Therefore, citations to non-citable items are usually known as a “free lunch” [14–17].

Many scholars have noticed this defect of the IF and questioned its fairness and reasonability for journal evaluation. Bensman [18] indicated that distinguishing citable and non-citable items was a fatal flaw that has existed since IF was first presented. Some scholars like Campanario [19–21] believe IF is a biased indicator and have pointed out that non-citable items have an important influence on it, and as a consequence the IF of a considerable number of journals including some of the most prestigious ones might be inflated by 30%–40%. Some researchers have pointed out that the incongruence of source items between the numerator and denominator seriously distort the true IF value of journals, and cause its manipulation and abuse [22–23]. More incredibly, one has pointed out that the number of the citable items in the denominator could be negotiated with Thomson Reuter, which might result in an IF variation of more than 300% [24–25].

Some scholars have presented several suggestions and solutions based on the inconsistency of source items in IF, and these fall into three main approaches, as follows:

1. Releasing new journal evaluation indicators. The Scopus database launched the SJR and SNIP indicators in 2007 and 2010. Similar to IF, these two new indicators are average citations to journals. However, they break some of the limitations of IF, such as prolonging the citation time window to three years and considering the citation peaks of different subjects. Among them, one of the most obvious advantages is their consistency in the source items between numerator and denominator, which counts reviews and articles, including proceedings papers [26–27]. Based on the principle and algorithm of weights based on order relation and experts’ suggestions, Xu and Fang [28] constructed a weighted IF that changed the order relation of different document types, including non-citable items.

2. Computing the IF per type of document. Moed et al. [29] found that the IF of The Lancet was reduced by almost 40% when just counting citations of the citable items, and Nature’ IF would be reduced by some 30% by including letters. Hence, they concluded that IF depended strongly on the set of documents used and suggested having a separate calculation for the IF of each type of document.

3. Redefining the types of source items in the numerator and denominator of the IF. Wu [30] asserted that the denominator should count the quantity of all source items now that the non-citable items were citable. He proposed that the definition of IF would be better readjusted. Some scholars [31–32] have argued that the type of source items between numerator and denominator should be consistent and suggested counting citations to articles and reviews in the numerator only. Some have suggested redefining citable items such as articles, notes, letters, and reviews and just count citations to these four source items [33]. Lozano et al. [34] corrected the inconsistency of document types between the numerator and denominator when analyzing the correlation between IF and citations in the digital age.

Even so, the current studies on the inconsistency between the numerator and denominator of the IF mainly focus on academic criticisms, proposals for modifications, and selecting sample journals to compute the corrected IF and compare it with traditional IF. These studies mainly remain at theory stage, and there is no empirical research on the journal evaluation effect of corrective IFs.

In the current study, based on the quantitative and cited characteristics of non-citable items, we have selected ophthalmologic journals and mathematical journals as research objects to correct the design formula of traditional IF. Considering the fact that scholars in a country might be not familiar with some journals from other countries, we have investigated only American researchers in the fields of ophthalmology and mathematics to obtain the real impact ranking of American ophthalmologic and mathematical journals in the minds of peer experts as the “gold standard” of journal impact. The correlations between the corrective IFs and questionnaire score were analyzed to verify the corrective effect of IF in different subjects to help us determine if there is a need to correct the numerator and denominator of traditional IF. These results have important theoretical significance and practical worth to the whole research community.

Research Methods

Determining research objects

There are significant differences in citation behavior between different subjects. For example, mathematics usually has a later citation peak and is known as a slow moving discipline, but papers in some disciplines in medicine often reach their citation peaks quickly [35–37]. Considering that research classification and discipline boundary in ophthalmology are relatively definite and clear, we selected ophthalmologic and mathematical journals as our research objects. As mentioned before, scholars in a country might be not familiar with some journals from other countries, considering America had a large number of journals included in the JCR database in 2014, we meant to investigate American researchers in the fields of ophthalmology and mathematics to obtain the real impact ranking of American ophthalmologic and mathematical journals in the minds of peer experts. There were 30 ophthalmologic journals and 82 mathematical journals published in America. Considering a large sample size may influence the questionnaire results, we just selected comprehensive mathematical journals for our study, 27 journals.

Obtaining the quantity and citations of different types of document

We retrieved all types of document published by each ophthalmologic and mathematical journal from 2012 to 2013, obtained the quantity per type of document through the “Refine” function in the SCI database, and obtained their citations in 2014 using the “Create Citation Report” function.

Corrective methods of traditional IF

Based on the quantitative and citation characteristics of different type of documents in the SCI database, we propose five corrective definitions for IF as follows:(1)(2)(3)(4)(5)

In formulae (1) and (2), C_t-1 and C_t-2 are citations given in year t to all source items published in years (t–1) and (t–2), respectively, N_t-1 and N_t-2 are the numbers of all source items published in years (t–1) and (t–2), respectively, N_AREL(t-1) and N_AREL(t-2) are the numbers of items including articles, reviews, editorials, and letters published in years (t–1) and (t–2), respectively. In formula (3), C_AR(t-1) and C_AR(t-2) are citations in year t given to articles and reviews published in years (t–1) and (t–2), respectively, N_AR(t-1) and N_AR(t-2) are the numbers of articles and reviews published in years (t–1) and (t–2), respectively. In formulae (4) and (5), C_AREL(t-1) and C_AREL(t-2) are citations in year t given to items including articles, reviews, editorials, and letters published in years (t–1) and (t–2), respectively, and the variables N_AR(t-1), N_AR(t-2), N_AREL(t-1), and N_AREL(t-2) are defined similarly to those in formulae (2) and (3).

Questionnaire survey

First, we selected American researchers as corresponding authors who have published ophthalmologic papers in the last 10 years or mathematical papers in the last five years in the SCI database as our respondents, and we obtained their email addresses by the field of ‘E-mail Addresses’ provided by WoS database. We obtained 9145 email addresses of ophthalmologic researchers and 7810 email addresses of mathematical researchers. We then designed a questionnaire survey about journal academic impact through the AskForm platform. Finally, we sent emails to those researchers, explained our research object, and requested them to answer the questionnaire via the provided web link. We sent successfully 7077 e-mails to American ophthalmologic researchers and 5136 e-mails to American mathematical researchers. In the questionnaire, researchers gave a score from 1.0–10.0 (the score is accurate to 1 decimal place) to each journal according to its academic impact. A score of 10 is the highest and 1 is the lowest. We calculated the total score of each journal as its real impact standard. We received 124 responses for the ophthalmologic journals and 123 responses for the mathematical journals. We excluded the questionnaires that scored less than three journals, gave the same score to all journals, or gave a score to each journal according to the sequence in which they were presented. Finally, we obtained 112 valid questionnaires for the ophthalmologic journals and 117 valid questionnaires for mathematical journals. The survey was conducted from August 4, 2015 to September 15, 2015.

Statistical analysis method

SPSS 22.0 was used to analyze the relationships between the corrective IFs and questionnaire score using the Spearman rank correlation test.

Results and Discussion

Ophthalmologic Journal Results

Quantity and citations per type of document published by 30 ophthalmologic journals.

There are significant differences in the quantity of and number of citations to each type of document for different journals. The quantity and number of citations of each type of document published by 30 ophthalmologic journals from 2012 to 2013 are shown in Table 1.

[Figure omitted. See PDF.]

Table 1. Quantity and citations of each type of document published by 30 ophthalmologic journals.

https://doi.org/10.1371/journal.pone.0151414.t001

Overall, these ophthalmologic journals mainly published articles, followed by letters, editorials, and reviews. Articles received the highest number of citations, followed by reviews, letters, and editorials. Although the total number of letters is far higher than that of the editorials, the number of citations to both are nearly identical. Therefore, for ophthalmologic journals, editorials are more often cited than letters. As non-citable items, some editorials and letters can be cited, but biographical items, corrections, and news items are hardly ever cited. Surv Ophthalmol and Curr Opin Ophthalmol published many reviews and few editorials and letters, Surv Ophthalmol only published one article, and Curr Opin Ophthalmol did not publish any articles.

Ophthalmologic journal questionnaire scores and corrective IFs.

We analyzed 112 valid questionnaires and obtained total expert score for the ophthalmologic journals. In addition, we calculated the five corrective IF values for each journal and retrieved its traditional IF through the JCR database. The results can be seen in Table 2.

[Figure omitted. See PDF.]

Table 2. Questionnaire score and various IFs of 30 ophthalmologic journals.

https://doi.org/10.1371/journal.pone.0151414.t002

Correlations between the questionnaire scores and various IFs of 30 ophthalmologic journals.

The evaluation of an expert in the field is considered to be the most important criterion for verifying the validity of citation indicators, and the true impact of different journals in researchers’ true estimation can be reflected by the expert scores in the questionnaire surveys [38, 39]. Hence, we can verify the effect of journal evaluation for each corrective IF by analyzing the correlations between them and the corresponding expert score.

First, we illustrate whether it is reasonable to use corrective IFs for journal valuation by looking at the correlations between questionnaire score and the five corrective IFs defined above. The scatter diagrams of these relationships are presented in Fig 1. In addition, the K-S normal distribution test was conducted using the variables, and the results show that IF_AREL/AR is not subject to a normal distribution. Thus, we analyzed the relationships between the questionnaire score and various IFs using the Spearman rank correlation test. The results are shown in Table 3.

[Figure omitted. See PDF.]

Fig 1. Correlations between the questionnaire score and various IFs of 30 ophthalmologic journals.

https://doi.org/10.1371/journal.pone.0151414.g001

[Figure omitted. See PDF.]

Table 3. Correlations between the questionnaire score and various IFs of 30 phthalmologic journals.

https://doi.org/10.1371/journal.pone.0151414.t003

Table 3 shows that the five corrective IFs and traditional IF are significantly correlated with the questionnaire score for the 30 ophthalmologic journals. The correlation coefficients between traditional IF and the five corrective IFs are above 0.9. The traditional IF is most relevant to IF_AREL/AR and IF_AR/AR, for which the correlation coefficients are 0.967 and 0.966, respectively. There are high correlations between the five corrective IFs. The correlation coefficients between IF_Total/Total and both IF_Total/AREL and IF_AREL/AREL reach 0.999. IF_AR/AR is the most relevant to IF_AREL/AR, and IF_Total/AREL is most relevant to IF_AREL/AREL. The questionnaire score is more relevant to the five corrective IFs than to traditional IF. IF_AR/AR is the most relevant to the questionnaire score, with a correlation coefficient of 0.715, followed by IF_AREL/AR, for which the correlation coefficient is 0.713. The correlation coefficient between IF_Total/AREL and the questionnaire score is the same as that between IF_AREL/AREL and the questionnaire score, and the correlation coefficients for both are 0.692.

Mathematical Journal Results

Quantity and citations for each type of document published by 27 mathematical journals.

To determine the distribution of non-citable items in mathematics, we analyzed the quantity and number of citations for each type of document published by 27 mathematical journals. The results are shown in Table 4.

[Figure omitted. See PDF.]

Table 4. Quantity and number of citations for each type of document published by 27 mathematical journals.

https://doi.org/10.1371/journal.pone.0151414.t004

Table 4 shows that these mathematical journals mainly published articles, and only Commun Pur Appl Math published one review from 2012 to 2013. There were 153 editorials and 123 letters published by the mathematical journals, but only 78 other non-citable items. Compared with the number of articles, the number of non-citable items is very small. The citations to articles are far higher in number than those to non-citable items. The review published by Commun Pur Appl Math was not cited in 2014. The citations to the 153 editorials in 2014 numbered only 13, the 123 letters were just cited once in 2014, the citations to other non-citable items numbered only three.

Questionnaire score and various IFs of 27 mathematical journals.

We analyzed 117 valid questionnaires to obtain the total expert score of 27 mathematical journals. In addition, we calculated the five corrective IF values for each journal and retrieved its traditional IF through the JCR database. The results are shown in Table 5.

[Figure omitted. See PDF.]

Table 5. Questionnaire score and various IFs of 27 mathematical journals.

https://doi.org/10.1371/journal.pone.0151414.t005

Correlations between questionnaire score and various IFs of 27 mathematical journals.

Fig 2 shows the scatter diagrams of the relationships between the questionnaire score and various IFs of the mathematical journals. In addition, the K-S normal distribution test results show that the five corrective IFs and traditional IF are not subject to a normal distribution. Thus, we analyzed the relationships between the questionnaire score and corrective IFs using the Spearman rank correlation test. The results are shown in Table 6.

[Figure omitted. See PDF.]

Fig 2. Correlations between the questionnaire score and various IFs of 27 mathematical journals.

https://doi.org/10.1371/journal.pone.0151414.g002

[Figure omitted. See PDF.]

Table 6. Correlations between questionnaire score and various IFs of 27 mathematical journals.

https://doi.org/10.1371/journal.pone.0151414.t006

Table 6 shows that the five corrective IFs and traditional IF are significantly correlated with questionnaire score for 27 mathematical journals. The correlation coefficients between the traditional IF and corrective IFs vary between 0.8 and 0.9. The traditional IF is most relevant to IF_Total/AREL and IF_AREL/AREL, for which the correlation coefficients are both 0.865. Similar to ophthalmology, there are high correlations between the five corrective IFs. IF_Total/Total is most relevant to IF_Total/AREL and IF_AREL/AREL, for which the correlation coefficients are both 0.991. IF_AR/AR is the most relevant to IF_AREL/AR, and IF_AREL/AREL is most relevant to IF_AR/AR. In contrast to ophthalmology, traditional IF is the most relevant to the questionnaire score, with a correlation coefficient of 0.545, followed by IF_Total/Total, for which the correlation coefficient is 0.510. The correlation coefficient between IF_AR/AR and the questionnaire score is the lowest.

Above all, whether ophthalmologic journals or mathematical journals, corrective IFs are highly correlated with traditional IF, and there are high correlations between any pair of corrective IFs. The correlations between corrective IFs and the questionnaire score of the ophthalmologic journals are higher than those of the mathematical journals. For ophthalmologic journals, the effect of journal evaluation shows that the corrective IFs are superior to traditional IF; however, there is little value to using the corrective IFs for mathematical journals. The reason for this could be that ophthalmologic journals publish more non-citable items that attract more citations, while mathematical journals publish fewer non-citable items and these non-citable items have fewer citations. Therefore, the corrective effect of traditional IF might be related to the quantitative and citation characteristics of non-citable items published by journals in different disciplines.

Conclusions

Five Corrective IFs and traditional IF are highly correlated

For both ophthalmologic and mathematical journals, the five corrective IFs are significantly positively and highly correlated with traditional IF. Although both ophthalmologic journals and mathematical journals publish a certain number of non-citable items, their quantities and number of citations are very low compared with those of articles. The numerator and/or denominator in the traditional IF are correspondingly improved. Thus, although there are some differences in the quantity of source items used for the corrective IFs and traditional IF, these differences are not significant. For both ophthalmology and mathematics, the journal ranks based on the corrective IFs and traditional IF are largely consistent, especially for the mathematical journals.

In general, the correlations between the five corrective IFs and traditional IF for mathematical journals are lower than those for ophthalmologic journals. For ophthalmologic journals, the source items in the denominator for IF_AR/AR and IF_AREL/AR, which correct the numerator of traditional IF, are the same as traditional IF. Among the non-citable items published by ophthalmologic journals, letters and editorials attract more citations; other non-citable items are rarely cited. Nevertheless, the number of citations to letters and editorials are much lower than those to articles and reviews. Hence, IF_AR/AR and IF_AREL/AR are highly correlated with traditional IF. The research results for mathematical journals are totally different from those of ophthalmologic journals. For mathematical journals, IF_Total/AREL and IF_AREL/AREL are the most correlated with traditional IF. IF_Total/AREL is constructed by re-defining the denominator in traditional IF, and IF_AREL/AREL is constructed by simultaneously correcting both numerator and denominator in the traditional IF. Therefore, because of the difference in subject nature, the correlations between corrective IFs and the traditional IF of journals are clearly different. The correlations between corrective IFs and traditional IF in other disciplines have yet to be studied.

Five corrective IFs are highly correlated

The statistical analysis shows that there are high correlations between the five corrective IFs for both ophthalmologic and mathematical journals. For ophthalmologic journals, the only difference between IF_Total/Total and IF_Total/AREL is the denominator. Because document types published by ophthalmologic journals are relatively concentrated into articles, reviews, letters, and editorials, IF_Total/Total is highly correlated with IF_Total/AREL. Articles, reviews, letters, and editorials have more citations than other non-citable items, so IF_AREL/AREL is highly correlated with IF_Total/Total. In addition, citations to letters and editorials are far fewer than to articles and reviews, thus IF_AREL/AR is highly correlated with IF_AR/AR. The research results for mathematical journals are the same as those for ophthalmologic journals.

It is scientific and reasonable to use five corrective IFs for journal evaluation

The scatter plots show that there are significant linear correlations between the five corrective IFs and questionnaire score for both ophthalmologic and mathematical journals. Further statistical analysis shows that for both ophthalmologic journals and mathematical journals, the five corrective IFs are significantly correlated with the questionnaire score. For ophthalmologic journals, the correlation coefficients between the various corrective IFs and questionnaire score vary from 0.6 to 0.8. For mathematical journals, the correlation coefficients vary from 0.4 to 0.6. Therefore, for both ophthalmologic and mathematical journals, it is scientific and feasible to use all the corrective IFs for journal evaluation.

Corrective effect of journal IF in different subjects is obviously different

The results of the study show that the correlations between five corrective IFs and questionnaire score for ophthalmologic journals are obviously higher than those for mathematical journals. For ophthalmologic journals, five corrective IFs are more relevant to the questionnaire score than to traditional IF. Therefore, the journal evaluation using the five corrective IFs is better than an evaluation using traditional IF. IF_AR/AR is the most relevant to the questionnaire score, followed by IF_AREL/AR. Thus, we conclude that the corrective effect of IF_AR/AR is best for ophthalmologic journals, IF_AREL/AR is better than IF_Total/Total, and the effects on journal evaluation from using IF_Total/AREL and IF_AREL/AREL are the worst. For mathematical journals, journal evaluation using the traditional IF is superior than that using corrective IFs. Among the five corrective IFs, the effect on journal evaluation of IF_Total/Total is the best, IF_AREL/AR is better than IF_Total/AREL and IF_AREL/AREL, and the effect in journal evaluation of IF_AR/AR is the worst.

In conclusion, although the assumptions behind correcting traditional IF are reasonable in theory, our empirical study finds that not all subjects need to correct their journal IF. In our study, it is valuable to correct the traditional IF of ophthalmologic journals; however, it may not make much sense to correct the traditional IF of mathematical journals. Overall, the quantity and citations of non-citable items are lower than those of citable items. Hence, the corrective effect of IF might be not obvious for most disciplines, but it might be valuable to correct the traditional IF of journals publishing more editorials and letters.

Supporting Information

[Figure omitted. See PDF.]

S1 Dataset. Data of questionnaire survey from American ophthalmologic researchers giving each American ophthalmology journal a score.

https://doi.org/10.1371/journal.pone.0151414.s001

(XLS)

S2 Dataset. Data of questionnaire survey from American mathematical researchers giving each American mathematical journal a score.

https://doi.org/10.1371/journal.pone.0151414.s002

(XLS)

Author Contributions

Conceived and designed the experiments: XLL. Performed the experiments: SSG JZ. Analyzed the data: SSG XLL. Contributed reagents/materials/analysis tools: SSG. Wrote the paper: SSG.

Citation: Liu X-L, Gai S-S, Zhou J (2016) Journal Impact Factor: Do the Numerator and Denominator Need Correction? PLoS ONE 11(3): e0151414. https://doi.org/10.1371/journal.pone.0151414

References

1. Garfield E. Citation Indexes for Science: a new dimension in documentation through association of ideas. Science. 1990; 122:108–111.

2. Wu XF, Fu Q, Rousseau R. On indexing in the Web of Science and predicting journal impact factor. Journal of Zhejiang University SCIENCE B. 2008; 9:582–590. pmid:18600790

3. Garfield E. The history and meaning of the journal impact factor. JAMA-Journal of the American Medical Association. 2006; 295:90–93.

4. Wan H, Tan ZY, Lu JJ, Zhu XL. Summary of the Evolution of Citation Analysis Research: 2001–2014. Library and Information Service. 2015; 59:120–136.

5. Garfield E, Sher IH. New factors in the evaluation of scientific literature through citation indexing. American Documentation. 1963; 14(3):195–201.

6. Garfield E. Citation analysis as a tool in journal evaluation. Science & Justice. 1972; 178:471–479.

7. Campanario JM. Large increases and decreases in journal impact factors in only one year: The effect of journal self-citations. Journal of the American Society for Information Science and Technology. 2011; 62:230–235.

8. Fersht A. The most influential journals: Impact Factor and Eigenfactor. Proceedings of the National Academy of Sciences of the United States of America. 2009; 106: 6883–6884. pmid:19380731

9. Liu XL. Structural characteristics of impact factors of the ten top international journals. Acta Editologica. 2014; 26:296–300.

10. Gai SS, Liu XL, Zhang SL. Methodology of calculation and structural analysis of journal impact factor based on Web of Science database: a case study of Nature. Chinese Journal of Scientific and Technical Periodicals. 2014; 25:980–984.

11. Journal Citation Reports. Journal Source Data[EB/OL]. (2015-10-28) [2012-05-22]. http://admin-apps.webofknowledge.com/JCR/help/h_sourcedata.htm#sourcedata.

12. Moed HF, Vanleeuwen TN. Improving the accuracy of Institute for Scientific Informations journal impact factors. Journal of the Association for Information Science and Technology. 1995; 46:461–467.

13. Liu XL, Gai SS, Zhang SL, Zhou J. Citation characteristics of non-citable documents and their contribution to journal impact factor. Acta Editologica. 2015; 27:495–499.

14. Opthof T. Impact factor 2013 of the Netherlands Heart Journal Surpasses 2.0. Netherlands Heart Journal. 2014; 22: 167–138.

15. Van Leeuwen T. Discussing some basic critique on journal impact factors: revision of earlier comments. Scientometrics. 2012; 92: 443–455. pmid:22844166

16. Bornmann L, Neuhaus C, Daniel HD. The effect of a two-stage publication process on the journal impact factor: a case study on the interactive open access journal Atmospheric Chemistry and Physics. Scientometrics. 2011; 86:93–97.

17. McKerahan TL, Carmichael SW. What is the impact factor, anyway?. Clinical Anatomy. 2012; 25:283–283. pmid:22034097

18. Bensman SJ. Distributional differences of the impact factor in the sciences versus the social sciences: an analysis of the probabilistic structure of the 2005 journal citation reports. Journal of the American Society for Information Science and Technology. 2008; 59: 1366–1382.

19. Campanario JM, Gonzalez L. Journal self-citations that contribute to the impact factor: Documents labeled "editorial material" in journals covered by the Science Citation Index. Scientometrics. 2006; 69: 365–386.

20. Gonzalez L, Campanario JM. Structure of the impact factor of journals included in the Social Sciences Citation Index: Citations from documents labeled "editorial material". Journal of the American Society for Information Science and Technology. 2007; 58: 252–262.

21. Jacso P. Five-year impact factor data in the Journal Citation Reports. Online Information Review. 2009; 33: 603–614.

22. Jones AW. The distribution of forensic journals, reflections on authorship practices, peer-review and role of the impact factor. Forensic Science International. 2007; 165:115–128. pmid:16784827

23. Fleck C. The impact factor fetishism. Arch Eur Sociol. 2013; 54: 327–356.

24. Jackson A. The impact factor game: the rising impact factor of the British Journal of Radiology—a success story? British Journal of Radiology. 2010; 83: 93–98. pmid:20139258

25. Reynolds JC, Menegazzi JJ, Yealy DM. Emergency medicine journal impact factor and change compared to other medical and surgical specialties. Academic Emergency Medicine. 2012; 19: 1248–1254. pmid:23167855

26. Wang P, Liu XL, Liu RY, Zheng CM. Review on SNIP and Revised SNIP2 Based on Scopus Database. Chinese Journal of Scientific and Technical Periodicals. 2013; 24(5):838–842.

27. Gonzalez-Pereira B, Guerrero-Bote VP, Moya-Anegon F. A new approach to the metric of journals' scientific prestige: The SJR indicator. Journal of Informetrics. 2010; 4(3): 379–391.

28. Xu HY, Fang S. Study on impact factor based on ordering relations converted to weight factor according to the types of journal literature. Journal of Intelligence. 2012; 31:50–53.

29. Tort ABL, Targino ZH, Amaral OB. Rising publication delays inflate journal impact factors. PLOS ONE. 2012; 7: 1–6.

30. Wu YS. The definition of impact factor has better be adjusted. [EB/OL]. (2014-06-25) [2015-09-25]. Available: http://blog.sciencenet.cn/blog-1557-806325.html.

31. Herna´n MA. Epidemiologists (of all people) should question journal impact factors (with discussion). Epidemiology. 2008; 19: 366–368.

32. Pendlebury DA, Adams J. Comments on a critique of the Thomson Reuters journal impact factor. Scientometrics. 2012; 92: 395–401.

33. Moed HF, vanLeeuwen TN, Reedijk J. A critical analysis of the journal impact factors of Angewandte Chemie and the Journal of the American Chemical Society—Inaccuracies in published impact factors based on overall citations only. Scientometrics. 1996; 37(1): 105–116.

34. Lozano GA, Lariviere V, Gingras Y. The weakening relationship between the impact factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology. 2012; 63: 2140–2145.

35. Dorta-Gonzalez P, Dorta-Gonzalez MI, Santos-Penate DR, Suarez-Vega R. Journal topic citation potential and between-field comparisons: the topic normalized impact factor. Journal of Informetrics. 2014; 8: 406–418.

36. Brzezinski M. Empirical modeling of the impact factor distribution. Journal of Informetrics. 2014; 8: 362–368.

37. Van Nierop E. The introduction of the 5-year impact factor: does it benefit statistics journals?. Statistica Neerlandica. 2010; 64: 71–76.

38. Harnad S. Validating Research Performance Metrics against Peer Rankings. Ethics in Science and Environmental Politic. 2008; 11: 103–107.

39. Weale AR, Bailey M, Lear P. The Level of Non-Citation of Articles within a Journal as a Measure of Quality: a Comparison to the Impact Factor. BMC Medical Research Methodology. 2004; 4: 1–8.