Neurol India Home 

Year : 2018  |  Volume : 66  |  Issue : 1  |  Page : 96--104

Alternative bibliometrics from the web of knowledge surpasses the impact factor in a 2-year ahead annual citation calculation: Linear mixed-design models' analysis of neuroscience journals

Araceli Diaz-Ruiz1, Ulises Orbe-Arteaga2, Camilo Rios1, Ernesto Roldan-Valadez3,  
1 Departament of Neurochemistry, Mexican National Institute of Neurology and Neurosurgery, Mexico City, Mexico
2 Magnetic Resonance Unit, Medica Sur Clinic and Foundation, Mexico City, Mexico
3 Directorate of Research, Hospital General de Mexico “Dr. Eduardo Liceaga”, Mexico City, Mexico

Correspondence Address:
Dr. Ernesto Roldan-Valadez
Directorate of Research, Hospital General de Mexico “Dr. Eduardo Liceaga”, Dr. Balmis 148 street, Col. Doctores, Del. Cuauhtemoc, 06726. Mexico City


Context: The decision about which journal to choose for the publication of research deserves further investigation. Aims: In this study, we evaluate the predictive ability of seven bibliometrics in the Web of Knowledge to calculate total cites over a 7-year period in neuroscience journals. Settings and Design: Coincidental bibliometrics appearing during 2007, 2008, 2009, 2010, and 2011, along with their corresponding cites in 2009, 2010, 2011, 2012, and 2013, were recorded from the journal citation reports (JCR) Science Edition. This was a retrospective study. Materials and Methods: This was a bibliographic research using data from the Web of Knowledge in the neuroscience category. Statistical Analysis Used: A linear-mixed effects design using random slopes and intercepts was performed on 275 journals in the neuroscience category. Results: We found that Eigenfactor score, cited half-life, immediacy index, and number of articles are significant predictors of 2-year-ahead total cites (P ≤ 0.010 for all variables). The impact factor, 5-year impact factor, and article influence score were not significant predictors; the global effect size was significant (R2= 0.999; P < 0.001) with a total variance of 99.9%. Conclusions: An integrative model using a set of several metrics could represent a new standard to assess the influence and importance of scientific journals, and may simultaneously help researchers to rank journals in their decision-making during the manuscript submission phase.

How to cite this article:
Diaz-Ruiz A, Orbe-Arteaga U, Rios C, Roldan-Valadez E. Alternative bibliometrics from the web of knowledge surpasses the impact factor in a 2-year ahead annual citation calculation: Linear mixed-design models' analysis of neuroscience journals.Neurol India 2018;66:96-104

How to cite this URL:
Diaz-Ruiz A, Orbe-Arteaga U, Rios C, Roldan-Valadez E. Alternative bibliometrics from the web of knowledge surpasses the impact factor in a 2-year ahead annual citation calculation: Linear mixed-design models' analysis of neuroscience journals. Neurol India [serial online] 2018 [cited 2022 Jan 19 ];66:96-104
Available from:

Full Text

In the process of writing a scientific article, submission to a journal is considered the last primary stage in publishing a manuscript.[1] In that process, it is very common that researchers look for journals with the highest impact factor (IF) instead of journals with the best audience for their research.[2] The IF is a part of a set of bibliometric tools used by the Web of Knowledge (WOK) managed by Thomson Reuters.[3] WOK encompasses six additional metrics: 5-year impact factor, immediacy index, number of articles (published), cited half-life, Eigenfactor™ score (ES), and article influence score. The IF has become an increasingly used metric for consideration of tenure and promotion, as well as for budget and resource planning within universities, research institutions, and colleges.[4],[5] However, this behavior seems counterintuitive to the cumulated opinions, year after year (including published views in high IF journals as Nature), stating that the IF is not a perfect metric and has its limitations.[2],[6],[7] In 1997, the IF was already considered still far from being a quality indicator and not recommended for evaluating research.[8] IF is not associated with factors such as quality of peer review process and quality of content of the journal. It is a measure that reflects the relative importance of a journal within its field and quantifies the frequency with which an “average article” in a journal has been cited in a particular period.[9] In 2014, the American Society for Cell Biology, along with journal editors, publishers, and other stakeholders, issued a pledge to move away from an over-reliance on journal IF and to seek new ways of assessing research output.[10] A similar opinion appeared in 2017 in an editorial of BMC Medicine.[11]

Some publications consider that alternative bibliometrics in the WOK like the ES can assess the real dissemination of an article (i.e., its use as well as the category of journals which include it in their reference lists).[12] Thus, the ES would better capture the prestige of a journal than the IF.[13] A recent study using linear-mixed models found that the ES and cited half-life are the two most significant metrics able to predict citation rates in a 2-year-ahead citation calculation in gastroenterology and hepatology journals.[14]

To prove if the findings mentioned above would apply to other fields, we purposed to evaluate the predictive ability of bibliometric tools included in the WOK in the Neurosciences category to calculate the annual total cites during a 7-year period. Our findings might help authors to answers the questions: which journals within my field should I select for the submission of my manuscripts?; and, which bibliometrics should I use to rank my preferred journals for submission?

 Materials and Methods

Study design

This was a retrospective study that did not require approval by the Institutional Review Board. We selected bibliometrics from the WOK [3] used to evaluate journals' performance in the Journal Citation Index (JCI). JCI includes medical journals that are categorized in the Science Citation Index (SCI) and the SCI Expanded (SCIE).[15] WOK website allowed us to record bibliometrics values over a 7-year period. Bibliometrics were defined as follows:

The IF for a given indexed journal is calculated by dividing the number of times the articles published in a journal is cited during the two previous years by the number of articles released by that journal in the same interval [16]The 5-year impact factor (5-yIF) is the IF of a given publication in a particular year but calculated over a 5-year periodThe cited half-life (CHL) is a measure of the rate of decline of the citation curve. That is, the number of years that the number of current citations takes to decrease to 50% of its initial value. It measures how long articles in a journal continue to be cited after publication [17]The immediacy index (ImIn) of journal depicts how often, on an average, authors cite very recent articles from a particular journal; and hence, how rapidly ordinary papers from that journal are adopted into the literature [17]Eigenfactor™ score (ES) is considered to be an indicator of the global influence or repercussion of manuscripts published online in Journal Citation Reports® (JCR), the annual list of indexed journals part of the Web of Knowledge SM. Its calculation is based on the number of times articles from the journal published in the past 5 years have been cited in the JCR annually, but it also considers which journals have contributed to these citations so that highly cited journals will influence the network more than lesser-cited journals. References from one article in a journal to another article from the same journal are removed, so that journal self-citation does not bias Eigenfactor Scores [18],[19]Article Influence™ Score (AIS) determines the average impact of a journal's articles over the first 5 years after publication. It is obtained from ES based on its same iterative algorithm, but taking into account the number of items in the journal [12]Article number (AN) is the number of articles published in the journal during the selected year.

Selection of journals and measured periods

We recorded the bibliometric values of 275 journals in the Neurosciences category of the JCR Science Edition [Table 1]. We chose this group because journals in this field published several of our articles. For the analysis, only journals that coincidentally appeared between the 2007 and 2013JCR Science Editions were included.{Table 1}

We recorded the bibliometric values offered by the WOK including five sets of bibliometrics for each journal and for each selected year that matched to its total citations 2-year ahead:

Set 1: 2007 Bibliometrics vs. 2009 Total CitesSet 2: 2008 Bibliometrics vs. 2010 Total CitesSet 3: 2009 Bibliometrics vs. 2011 Total CitesSet 4: 2010 Bibliometrics vs. 2012 Total CitesSet 5: 2011 Bibliometrics vs. 2013 Total Cites.

Sample size calculation

We followed the recommendation of Tabachnik and Fidell [20] for multivariate normality: we included seven measurements (per 275 journals), as even with different n and only a few dependent variables (DVs), a sample size of about 20 in the smallest cell ensure robustness in the analysis.

Statistical analysis

Design of the mixed-effects model

Our predictive model of 2-years ahead Total Cites combined the overall effect of bibliometrics while taking into account within-journal cites in five repeated measures. The use of random slopes and intercepts allowed us to test the hypothesis that alternative bibliometrics of WOK surpasses the IF as predictors of a total-cite calculation. Our mixed-model comprised a three-level hierarchy – level 1, selected bibliometrics (continuous variables) consecutively measured during five years; level 2, the year of citation; and level 3, the journal ID. [Figure 1] shows the three-level hierarchical model; it is evident how levels 1 and 2 are nested within each journal.{Figure 1}

Justification for multilevel modeling

Experts consider that a multilevel modeling is needed when a high intraclass correlation coefficient (ICC) value points to variations in the mean selected biomarker levels across time (year-over-year data). ICC was used to estimate the selected biomarker variance that occurs both across journals and time of measurement.[21]

Selection of dependent and independent variables

The 2-years ahead Total Cites was the dependent variable (DV). Independent variables (IV) included: seven continuous variables: impact factor, 5-year impact factor, immediacy index, number of articles, cited half-life, Eigenfactor score, article influence score; and one categorical variable: year of measurement (time-set of citations).

Mixed-model effects analysis

We used the maximum likelihood (ML) estimation. ML is an appropriate approach for studying individual changes as well as fixed and random effects of the entire model. ML created a hierarchical model with repeated measures nested at six consecutive years within journals.[22] To specify the within-individual error covariance structure that best fits the data, we evaluated the four most common covariance matrixes types (unstructured, scaled identity, compound symmetry, diagonal).[23] The fitness of the model was assessed with the 2-log likelihood, Akaike's Information Criterion (AIC), and Bayesian Information Criterion (BIC).

Our model considered the fixed effects of the selected IV and a random effect for the repeated measures to characterize the characteristic variation caused by individual differences.[24] A scatter plot depicted the fitting of the predicted versus the observed values of Total Cites for the whole model labeled by subgroups (year of cites). A linear regression (LR) analysis calculated the R2 and P values.[25]

Measure of the effect size

The effect size was defined as the proportion of the variance in the DV explained by the IV by using the R proposed by Cohen;[26] where 0.1 to 0.29 = small effect, 0.30 to 0.49 = moderate effect, and ≥0.5 = large effect. We computed a pseudo-R2 as a measure of global effect size statistic (squared correlation between the observed and predicted scores).[21]

Evolution of IF and ES over time

We plotted the evolution of the IF and ES (separately) in the neurosciences category by sorting the journals in five groups based on their values of IF [(level 1, 0 to 6.0); (level 2, 6.1. to 12.0); (level 3, 12.1 to 18); (level 4, 18.1 to 24); (level 5 >24)]; and ES [(level 1., 0 to. 090); (level 2.,091 to. 180); (level 3.,181 to. 270); (level 4.,271 to. 360); (level 5>.361)] from 2007 to 2013 and performed a split-plot factorial analysis of variance (ANOVA).[27]

Displacement of the ranking place

We drew a graph line to show how the top 25 journals initially ranked by the IF were re-ordered based on their ES values. All analyses were carried out using the IBM's SPSS software (version, IBM Corporation, Armonk, NY, USA). Statistical significance was indicated by P< 0.05 (two-tailed).


Justification for multilevel modeling

We found significant ICC values in all selected bibliometrics: impact factor (ICC = 0.993, P = <0.001); 5-year impact factor (ICC = 0.989, P = <0.001); immediacy index (ICC = 0.970, P = <0.001); number of articles (ICC = 0.992, P< 0.001); cited half-life (ICC = 0.985, P = <0.001); Eigenfactor score (ICC = 0.998, P = <0.001), and article influencescore (ICC = 0.998, P = <0.001).

Overall fit for models

The diagonal matrix type represented the best model by depicting the smallest values in the information criteria table. [Table 2] shows the assessment of the overall fit of the multivariate models. [Figure 2] shows graph lines identifying each selected journal in the final model; the existence of random slopes and intercepts is evident.{Figure 2}{Table 2}

Significant predictors of total cites and beta coefficients of the regression model

The by-subject random slopes and intercepts for the effect of repeated measures (time) showed a substantial effect (Wald Z = 6.331, P<.001) for four IVs: immediacy index, number of articles, cited half-life, and Eigenfactor score. The impact factor, 5-year impact factor, and article influence score were not significant. [Table 3] shows the main effects for the selected model.{Table 3}

The more significant coefficient corresponded to the ES followed by the CHL; both variables depicted a positive, meaningful direction in the outcome. [Table 4] shows the unstandardized beta coefficients for each variable and its 95% confidence intervals.{Table 4}

A linear equation representing our assembled model was structured as:

Total cites (number) = −1063.40 + 51.95 (impact factor) + 65.37 (5-year impact factor) + 257.59 (immediacy index) + 3.22 (number of articles) + 899.57 (cited half-life) + 59690.66 (Eigenfactor score) + 191.37 (article influence score)

Global effect size of the model

The global effect size showed an R2 value = 0.999, P< 0.001; this value corresponded to a large effect size. [Figure 3] represents the regression line between the observed and predicted values for total cites.{Figure 3}

Evolution of IF and ES over time

There was no significant interaction between the IF or ES of the journals when grouped by a five-level rank. The IF showed an increasing trend from 2008 to 2011 with a plateau from 2011 to 2013 (for journals with an IF > 24). Conversely, the ES depicted a decreasing trend from 2008 to 2013 (for journals with an ES > 0.361); [Figure 4]a and b shows the IF and ES patterns.{Figure 4}

Comparison of journals' ranking using IF, ES, and CHL

Only 13 of the 25 journals previously ranked in the top 25 by the IF stayed in this selected group after they were re-sorted using decreasing values of ES, with striking differences in their new positions. For example, J Neurosci from place 24th climbed to 1st and Mol Psychiatr dropped from the 5th to the 24th spot. An additional resorting using the CHL showed that only 2 of the first 25 journals stayed in the top 25 places. [Figure 5] depicts the displacements in rankings based on decreased values of IF, ES, and CHL.{Figure 5}


Active investigators will remain apprised not only of the latest discoveries in their field but also will improve their understanding of which audience might be more interested in a particular research topic, ideally before a manuscript submission. Recently, journals from diverse fields have been publishing reports about the apparent limitations of the IF.[6],[28],[29],[30],[31],[32] It has been accepted that neither the IF nor the total number of citations is, per se, the metric of the overall influence of a journal. All journals have a diverse set of citations, and even the best publications contain some papers that are never cited.[33] That is, citations are not equally distributed, with fewer than 20% of the articles accounting for more than 50% of the total number of citations.[34],[35] Despite these facts, the misuse of the journal IF for judging the value of science persists because it confers significant benefits to individual scientists and journals.[36],[37] In this scenario, becoming familiar with bibliometric indices is a substantial concern for individual researchers because they may influence their teaching and research activities. Nowadays, measuring the impact of publication is mandatory not only for the individual but also for universities and research institutes; thus, journals and institutions are sufficiently correlated with the prestige and ranking outcome.[38]

We considered in this study the “2-years ahead total cites” as our DV. Some readers might ask why not use that metric to gauge the reputation of a journal; we did not use this because, by the time (chosen year) a researcher makes a bibliometric analysis of his target journals, the number of years ahead total cites is unknown. Our findings that four meaningful metrics (CHL, ImIn, ES, and AN) depict significant predictive ability for a total-cites calculation 2-year ahead match with the results of a similar study evaluating bibliometrics in the Gastroenterology and Hepatology category.[14]

Although our results might seem obvious, we were compelled to assess the predictive ability of the same bibliometrics in Neuroscience journals considering the striking facts about how IF are different between various specialties. For example, a journal in the oncology field might have an IF up to 30 times as high as the corresponding figure in the forensic medicine category.[4]

Some explanations for our findings acknowledge that, although IF is a per-article measure, it has not been replaced by any other means of rating the quality of journals.[39] People most of the times use it, although several editors are gradually removing the IF indicator from their journals' websites.[11] On the other hand, ES is a per-journal measure that scales with journal size, as is “total cites;” then, the superiority of the ES evaluating the quality of journals has been formerly reported.[13] The nonsignifi cance of the IF and 5-yIF as predictive bibliometrics in our study could be justified as they measure citations “per article,” are poor indicators of total cites given that scholarly journals vary in size over multiple orders of magnitude; and the volume of journals normalizes IF, 5-yIF, and AIS. The AN (number of articles) at least scales with the journal size but does not account for quality; hence, the ES is a winner in an unbalanced competition.

An additional feature of the ES that helps to explain its predictive ability is that ES ranks journals similar to how Google ranks websites: the ES algorithms use the structure of the entire network and not just citation information to evaluate the importance of each journal. ES provides a 5-year analysis of articles published and citations rather than a 2-year evaluation, and it does not include self-citations in its studies.[40] Furthermore, the ES formula has no denominator; journals that publish copious articles have a higher ES than those that publish sparse articles if the average quality of the published reports is coincident among these journals.[41]

Some advantages of using a mixed-model design are worth mentioning. In contrast to conventional multivariate analysis techniques that only focus on group differences in patterns of change, mixed-models allow the study of both intra- and interindividual variations (e.g., slopes and intercepts).[23] In addition, multilevel linear models dismiss the assumption of homogeneity of regression slopes, cast aside the assumption of independence (between cases), and expect the presence of missing data.[24]

Several limitations of this study need to be addressed – a detailed explanation of each of the presented bibliometric indices is beyond the scope of this article. We acknowledge that, besides numerical markers, authors of articles also use the perceived reputation of a journal to decide the target journal for their articles. This depends on the publisher, affiliation to a reputed society, editor, and editorial board members; we did not include these variables as their measurement is not standardized and they are not included in the WOK database. We observed in our data bibliometrics with high correlations, in those situations, the usual solution would be the deletion of the redundant variable. However, we decided to retain all variables in the model considering that, in a linear mixed-effects model, the assumption of independence (between cases) is cast aside and correlation among variables is expected.[24] Our predictive analysis was limited to time-sets in a 2-year comparison period (five repeated measures of bibliometrics from 2007 until 2013), which might seem a bit meager. Readers should be aware that the first publications by Bergstrom et al., about ES methodology appeared in 2008. We did not include additional confounding factors such as longer time frames, number of articles published in each issue, and circulation of each journal because the WOK does not consider them. The JCR includes approximately 171 categories in the sciences and about 54 in the social sciences; then, the publication of future studies validating our model in other specialties would be desirable.

In conclusion, it is in the researchers' hands to stop the misuse of IF alone to evaluate journals; we found evidence that alternative bibliometrics from the WOK provide a better assessment of the influence and importance of scientific journals in a particular discipline. An integrative model using a set of metrics could represent a useful tool for researchers in their decision-making during the manuscript submission phase; it may even supplement new “standards” of the quality and validity of the research.

Financial support and sponsorship


Conflicts of interest

There are no conflicts of interest.


1Provenzale JM. Ten principles to improve the likelihood of publication of a scientific manuscript. AJR Am J Roentgenol 2007;188:1179-82.
2Rawat S. How is impact factor impacting our research? Biomed J 2014;37:415-6.
3Thomson_Reuters. Web of Knowledge 2014. Available from: [Last accessed on 2017 Dec 01].
4Baethge C. Impact factor - a useful tool, but not for all purposes. Deutsches Arzteblatt Int 2012;109:267-9.
5Butler JS, Sebastian AS, Kaye ID, Wagner SC, Morrissey PB, Schroeder GD, et al. Understanding traditional research impact metrics. Clin Spine Surg 2017;30:164-6.
6Elliott DB. The impact factor: A useful indicator of journal quality or fatally flawed? Ophthalmic Physiol Optics 2014;34:4-7.
7Callaway E. Beat it, impact factor! Publishing elite turns against controversial metric. Nature 2016;535:210-1.
8Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314:498-502.
9Sharma M, Sarin A, Gupta P, Sachdeva S, Desai AV. Journal impact factor: Its use, significance and limitations. World J Nucl Med 2014;13:146.
10Ending the tyranny of the impact factor. Nature Cell Biol 2014;16:1.
11Diamandis EP. The journal impact factor is under attack-use the CAPCI factor instead. BMC Med 2017;15:9.
12Ascaso FJ. Impact factor, eigenfactor and article influence. Arch Soc Esp Oftalmol 2011;86:1-2.
13Sillet A, Katsahian S, Range H, Czernichow S, Bouchard P. The Eigenfactor score in highly specific medical fields: The dental model. J Dent Res 2012;91:329-33.
14Roldan-Valadez E, Rios C. Alternative bibliometrics from impact factor improved the esteem of a journal in a 2-year-ahead annual-citation calculation: Multivariate analysis of gastroenterology and hepatology journals. Eur J Gastroenterol Hepatol 2015;27:115-22.
15Garfield E. How can impact factors be improved? BMJ 1996;313:411-3.
16Kumar V, Upadhyay S, Medhi B. Impact of the impact factor in biomedical research: Its use and misuse. Singapore Med J 2009;50:752-5.
17Mathur VP, Sharma A. Impact factor and other standardized measures of journal citation: A perspective. Indian J Dent Res 2009;20:81-5.
18Bergstrom CT, West JD. Assessing citations with the Eigenfactor metrics. Neurology 2008;71:1850-1.
19Bergstrom CT, West JD, Wiseman MA. The Eigenfactor metrics. J Neurosci 2008;28:11433-4.
20Tabachnick BG, Fidell LS. Multivariate Normality. In: Tabachnick BG, Fidell LS, editors. Using Multivariate Statistics. 6th ed; 2013. p. 253.
21Peugh JL. A practical guide to multilevel modeling. J Sch Psychol 2010;48:85-112.
22Miyazaki Y, Raudenbush SW. Tests for linkage of multiple cohorts in an accelerated longitudinal design. Psychol Methods 2000;5:44-63.
23Shek DT, Ma CM. Longitudinal data analyses using linear mixed models in SPSS: Concepts, procedures and illustrations. Scientific World Journal 2011;11:42-76.
24Field A. Multilevel Linear Models. In: Field A, editor. Discovering Statistics using SPSS. 4th ed. London: SAGE Publications Inc.; 2013. p. 734-76.
25Pallant J. Multiple regression. SPSS Survival Manual. Crows Nest, NSW, Australia: Allen & Unwin; 2011. p. 148-67.
26Cohen JW. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates; 1988.
27Roldan-Valadez E, Rios-Piedra E, Favila R, Alcauter S, Rios C. Diffusion tensor imaging-derived measures of fractional anisotropy across the pyramidal tract are influenced by the cerebral hemisphere but not by gender in young healthy volunteers: A split-plot factorial analysis of variance. Chinese Med J 2012;125:2180-7.
28Zupanc GK. Impact beyond the impact factor. J Comp Physiol A Neuroethol Sens Neural Behav Physiol 2014;200:113-6.
29Citrome L. How we rate: Is impact factor the most important measure? Int J Clin Pract 2013;67:819-20.
30Kodumuri P, Ollivere B, Holley J, Moran CG. The impact factor of a journal is a poor measure of the clinical relevance of its papers. Bone Joint J 2014;96-B: 414-9.
31Wilms G. The impact factor. Neuroradiology 2013;55:803-6.
32Yang ZG, Zhang CT. A proposal for a novel impact factor as an alternative to the JCR impact factor. Scientific Rep 2013;3:3410.
33Fersht A. The most influential journals: Impact Factor and Eigenfactor. Proc Natl Acad Sci U S A 2009;106:6883-4.
34Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA 2002;287:2847-50.
35Weale AR, Bailey M, Lear PA. The level of non-citation of articles within a journal as a measure of quality: A comparison to the impact factor. BMC Med Res Methodol 2004;4:14.
36Casadevall A, Fang FC. Causes for the persistence of impact factor mania. mBio 2014;5:e00064-14.
37Reider B. Brace for Impact. Am J Sports Med 2017;45:2213-6.
38Gasparyan AY. Thoughts on impact factors and editing of medical journals. Inflamm Allergy Drug Targets 2010;9:2-5.
39Falagas ME, Zouglakis GM, Papastamataki PA. Trends in the impact factor of scientific journals. Mayo Clin Proc 2006;81:1401-2.
40Roberts WC. Piercing the impact factor and promoting the Eigenfactor. Am J Cardiol 2011;108:896-8.
41Rizkallah J, Sin DD. Integrative approach to quality assessment of medical journals using impact factor, eigenfactor, and article influence scores. PloS One 2010;5:e10204.