Home 
Alternative bibliometrics from the web of knowledge surpasses the impact factor in a 2year ahead annual citation calculation: Linear mixeddesign models' analysis of neuroscience journals Correspondence Address: Context: The decision about which journal to choose for the publication of research deserves further investigation. Aims: In this study, we evaluate the predictive ability of seven bibliometrics in the Web of Knowledge to calculate total cites over a 7year period in neuroscience journals. Settings and Design: Coincidental bibliometrics appearing during 2007, 2008, 2009, 2010, and 2011, along with their corresponding cites in 2009, 2010, 2011, 2012, and 2013, were recorded from the journal citation reports (JCR) Science Edition. This was a retrospective study. Materials and Methods: This was a bibliographic research using data from the Web of Knowledge in the neuroscience category. Statistical Analysis Used: A linearmixed effects design using random slopes and intercepts was performed on 275 journals in the neuroscience category. Results: We found that Eigenfactor score, cited halflife, immediacy index, and number of articles are significant predictors of 2yearahead total cites (P ≤ 0.010 for all variables). The impact factor, 5year impact factor, and article influence score were not significant predictors; the global effect size was significant (R^{2}= 0.999; P < 0.001) with a total variance of 99.9%. Conclusions: An integrative model using a set of several metrics could represent a new standard to assess the influence and importance of scientific journals, and may simultaneously help researchers to rank journals in their decisionmaking during the manuscript submission phase.
Materials and Methods Study design This was a retrospective study that did not require approval by the Institutional Review Board. We selected bibliometrics from the WOK [3] used to evaluate journals' performance in the Journal Citation Index (JCI). JCI includes medical journals that are categorized in the Science Citation Index (SCI) and the SCI Expanded (SCIE).[15] WOK website allowed us to record bibliometrics values over a 7year period. Bibliometrics were defined as follows: The IF for a given indexed journal is calculated by dividing the number of times the articles published in a journal is cited during the two previous years by the number of articles released by that journal in the same interval [16]The 5year impact factor (5yIF) is the IF of a given publication in a particular year but calculated over a 5year periodThe cited halflife (CHL) is a measure of the rate of decline of the citation curve. That is, the number of years that the number of current citations takes to decrease to 50% of its initial value. It measures how long articles in a journal continue to be cited after publication [17]The immediacy index (ImIn) of journal depicts how often, on an average, authors cite very recent articles from a particular journal; and hence, how rapidly ordinary papers from that journal are adopted into the literature [17]Eigenfactor™ score (ES) is considered to be an indicator of the global influence or repercussion of manuscripts published online in Journal Citation Reports® (JCR), the annual list of indexed journals part of the Web of Knowledge SM. Its calculation is based on the number of times articles from the journal published in the past 5 years have been cited in the JCR annually, but it also considers which journals have contributed to these citations so that highly cited journals will influence the network more than lessercited journals. References from one article in a journal to another article from the same journal are removed, so that journal selfcitation does not bias Eigenfactor Scores [18],[19]Article Influence™ Score (AIS) determines the average impact of a journal's articles over the first 5 years after publication. It is obtained from ES based on its same iterative algorithm, but taking into account the number of items in the journal [12]Article number (AN) is the number of articles published in the journal during the selected year. Selection of journals and measured periods We recorded the bibliometric values of 275 journals in the Neurosciences category of the JCR Science Edition [Table 1]. We chose this group because journals in this field published several of our articles. For the analysis, only journals that coincidentally appeared between the 2007 and 2013JCR Science Editions were included.{Table 1} We recorded the bibliometric values offered by the WOK including five sets of bibliometrics for each journal and for each selected year that matched to its total citations 2year ahead: Set 1: 2007 Bibliometrics vs. 2009 Total CitesSet 2: 2008 Bibliometrics vs. 2010 Total CitesSet 3: 2009 Bibliometrics vs. 2011 Total CitesSet 4: 2010 Bibliometrics vs. 2012 Total CitesSet 5: 2011 Bibliometrics vs. 2013 Total Cites. Sample size calculation We followed the recommendation of Tabachnik and Fidell [20] for multivariate normality: we included seven measurements (per 275 journals), as even with different n and only a few dependent variables (DVs), a sample size of about 20 in the smallest cell ensure robustness in the analysis. Statistical analysis Design of the mixedeffects model Our predictive model of 2years ahead Total Cites combined the overall effect of bibliometrics while taking into account withinjournal cites in five repeated measures. The use of random slopes and intercepts allowed us to test the hypothesis that alternative bibliometrics of WOK surpasses the IF as predictors of a totalcite calculation. Our mixedmodel comprised a threelevel hierarchy – level 1, selected bibliometrics (continuous variables) consecutively measured during five years; level 2, the year of citation; and level 3, the journal ID. [Figure 1] shows the threelevel hierarchical model; it is evident how levels 1 and 2 are nested within each journal.{Figure 1} Justification for multilevel modeling Experts consider that a multilevel modeling is needed when a high intraclass correlation coefficient (ICC) value points to variations in the mean selected biomarker levels across time (yearoveryear data). ICC was used to estimate the selected biomarker variance that occurs both across journals and time of measurement.[21] Selection of dependent and independent variables The 2years ahead Total Cites was the dependent variable (DV). Independent variables (IV) included: seven continuous variables: impact factor, 5year impact factor, immediacy index, number of articles, cited halflife, Eigenfactor score, article influence score; and one categorical variable: year of measurement (timeset of citations). Mixedmodel effects analysis We used the maximum likelihood (ML) estimation. ML is an appropriate approach for studying individual changes as well as fixed and random effects of the entire model. ML created a hierarchical model with repeated measures nested at six consecutive years within journals.[22] To specify the withinindividual error covariance structure that best fits the data, we evaluated the four most common covariance matrixes types (unstructured, scaled identity, compound symmetry, diagonal).[23] The fitness of the model was assessed with the 2log likelihood, Akaike's Information Criterion (AIC), and Bayesian Information Criterion (BIC). Our model considered the fixed effects of the selected IV and a random effect for the repeated measures to characterize the characteristic variation caused by individual differences.[24] A scatter plot depicted the fitting of the predicted versus the observed values of Total Cites for the whole model labeled by subgroups (year of cites). A linear regression (LR) analysis calculated the R2 and P values.[25] Measure of the effect size The effect size was defined as the proportion of the variance in the DV explained by the IV by using the R proposed by Cohen;[26] where 0.1 to 0.29 = small effect, 0.30 to 0.49 = moderate effect, and ≥0.5 = large effect. We computed a pseudoR2 as a measure of global effect size statistic (squared correlation between the observed and predicted scores).[21] Evolution of IF and ES over time We plotted the evolution of the IF and ES (separately) in the neurosciences category by sorting the journals in five groups based on their values of IF [(level 1, 0 to 6.0); (level 2, 6.1. to 12.0); (level 3, 12.1 to 18); (level 4, 18.1 to 24); (level 5 >24)]; and ES [(level 1., 0 to. 090); (level 2.,091 to. 180); (level 3.,181 to. 270); (level 4.,271 to. 360); (level 5>.361)] from 2007 to 2013 and performed a splitplot factorial analysis of variance (ANOVA).[27] Displacement of the ranking place We drew a graph line to show how the top 25 journals initially ranked by the IF were reordered based on their ES values. All analyses were carried out using the IBM's SPSS software (version 24.0.0.3, IBM Corporation, Armonk, NY, USA). Statistical significance was indicated by P< 0.05 (twotailed). Results Justification for multilevel modeling We found significant ICC values in all selected bibliometrics: impact factor (ICC = 0.993, P = <0.001); 5year impact factor (ICC = 0.989, P = <0.001); immediacy index (ICC = 0.970, P = <0.001); number of articles (ICC = 0.992, P< 0.001); cited halflife (ICC = 0.985, P = <0.001); Eigenfactor score (ICC = 0.998, P = <0.001), and article influencescore (ICC = 0.998, P = <0.001). Overall fit for models The diagonal matrix type represented the best model by depicting the smallest values in the information criteria table. [Table 2] shows the assessment of the overall fit of the multivariate models. [Figure 2] shows graph lines identifying each selected journal in the final model; the existence of random slopes and intercepts is evident.{Figure 2}{Table 2} Significant predictors of total cites and beta coefficients of the regression model The bysubject random slopes and intercepts for the effect of repeated measures (time) showed a substantial effect (Wald Z = 6.331, P<.001) for four IVs: immediacy index, number of articles, cited halflife, and Eigenfactor score. The impact factor, 5year impact factor, and article influence score were not significant. [Table 3] shows the main effects for the selected model.{Table 3} The more significant coefficient corresponded to the ES followed by the CHL; both variables depicted a positive, meaningful direction in the outcome. [Table 4] shows the unstandardized beta coefficients for each variable and its 95% confidence intervals.{Table 4} A linear equation representing our assembled model was structured as: Total cites (number) = −1063.40 + 51.95 (impact factor) + 65.37 (5year impact factor) + 257.59 (immediacy index) + 3.22 (number of articles) + 899.57 (cited halflife) + 59690.66 (Eigenfactor score) + 191.37 (article influence score) Global effect size of the model The global effect size showed an R2 value = 0.999, P< 0.001; this value corresponded to a large effect size. [Figure 3] represents the regression line between the observed and predicted values for total cites.{Figure 3} Evolution of IF and ES over time There was no significant interaction between the IF or ES of the journals when grouped by a fivelevel rank. The IF showed an increasing trend from 2008 to 2011 with a plateau from 2011 to 2013 (for journals with an IF > 24). Conversely, the ES depicted a decreasing trend from 2008 to 2013 (for journals with an ES > 0.361); [Figure 4]a and b shows the IF and ES patterns.{Figure 4} Comparison of journals' ranking using IF, ES, and CHL Only 13 of the 25 journals previously ranked in the top 25 by the IF stayed in this selected group after they were resorted using decreasing values of ES, with striking differences in their new positions. For example, J Neurosci from place 24th climbed to 1st and Mol Psychiatr dropped from the 5th to the 24th spot. An additional resorting using the CHL showed that only 2 of the first 25 journals stayed in the top 25 places. [Figure 5] depicts the displacements in rankings based on decreased values of IF, ES, and CHL.{Figure 5} Discussion Active investigators will remain apprised not only of the latest discoveries in their field but also will improve their understanding of which audience might be more interested in a particular research topic, ideally before a manuscript submission. Recently, journals from diverse fields have been publishing reports about the apparent limitations of the IF.[6],[28],[29],[30],[31],[32] It has been accepted that neither the IF nor the total number of citations is, per se, the metric of the overall influence of a journal. All journals have a diverse set of citations, and even the best publications contain some papers that are never cited.[33] That is, citations are not equally distributed, with fewer than 20% of the articles accounting for more than 50% of the total number of citations.[34],[35] Despite these facts, the misuse of the journal IF for judging the value of science persists because it confers significant benefits to individual scientists and journals.[36],[37] In this scenario, becoming familiar with bibliometric indices is a substantial concern for individual researchers because they may influence their teaching and research activities. Nowadays, measuring the impact of publication is mandatory not only for the individual but also for universities and research institutes; thus, journals and institutions are sufficiently correlated with the prestige and ranking outcome.[38] We considered in this study the “2years ahead total cites” as our DV. Some readers might ask why not use that metric to gauge the reputation of a journal; we did not use this because, by the time (chosen year) a researcher makes a bibliometric analysis of his target journals, the number of years ahead total cites is unknown. Our findings that four meaningful metrics (CHL, ImIn, ES, and AN) depict significant predictive ability for a totalcites calculation 2year ahead match with the results of a similar study evaluating bibliometrics in the Gastroenterology and Hepatology category.[14] Although our results might seem obvious, we were compelled to assess the predictive ability of the same bibliometrics in Neuroscience journals considering the striking facts about how IF are different between various specialties. For example, a journal in the oncology field might have an IF up to 30 times as high as the corresponding figure in the forensic medicine category.[4] Some explanations for our findings acknowledge that, although IF is a perarticle measure, it has not been replaced by any other means of rating the quality of journals.[39] People most of the times use it, although several editors are gradually removing the IF indicator from their journals' websites.[11] On the other hand, ES is a perjournal measure that scales with journal size, as is “total cites;” then, the superiority of the ES evaluating the quality of journals has been formerly reported.[13] The nonsignifi cance of the IF and 5yIF as predictive bibliometrics in our study could be justified as they measure citations “per article,” are poor indicators of total cites given that scholarly journals vary in size over multiple orders of magnitude; and the volume of journals normalizes IF, 5yIF, and AIS. The AN (number of articles) at least scales with the journal size but does not account for quality; hence, the ES is a winner in an unbalanced competition. An additional feature of the ES that helps to explain its predictive ability is that ES ranks journals similar to how Google ranks websites: the ES algorithms use the structure of the entire network and not just citation information to evaluate the importance of each journal. ES provides a 5year analysis of articles published and citations rather than a 2year evaluation, and it does not include selfcitations in its studies.[40] Furthermore, the ES formula has no denominator; journals that publish copious articles have a higher ES than those that publish sparse articles if the average quality of the published reports is coincident among these journals.[41] Some advantages of using a mixedmodel design are worth mentioning. In contrast to conventional multivariate analysis techniques that only focus on group differences in patterns of change, mixedmodels allow the study of both intra and interindividual variations (e.g., slopes and intercepts).[23] In addition, multilevel linear models dismiss the assumption of homogeneity of regression slopes, cast aside the assumption of independence (between cases), and expect the presence of missing data.[24] Several limitations of this study need to be addressed – a detailed explanation of each of the presented bibliometric indices is beyond the scope of this article. We acknowledge that, besides numerical markers, authors of articles also use the perceived reputation of a journal to decide the target journal for their articles. This depends on the publisher, affiliation to a reputed society, editor, and editorial board members; we did not include these variables as their measurement is not standardized and they are not included in the WOK database. We observed in our data bibliometrics with high correlations, in those situations, the usual solution would be the deletion of the redundant variable. However, we decided to retain all variables in the model considering that, in a linear mixedeffects model, the assumption of independence (between cases) is cast aside and correlation among variables is expected.[24] Our predictive analysis was limited to timesets in a 2year comparison period (five repeated measures of bibliometrics from 2007 until 2013), which might seem a bit meager. Readers should be aware that the first publications by Bergstrom et al., about ES methodology appeared in 2008. We did not include additional confounding factors such as longer time frames, number of articles published in each issue, and circulation of each journal because the WOK does not consider them. The JCR includes approximately 171 categories in the sciences and about 54 in the social sciences; then, the publication of future studies validating our model in other specialties would be desirable. In conclusion, it is in the researchers' hands to stop the misuse of IF alone to evaluate journals; we found evidence that alternative bibliometrics from the WOK provide a better assessment of the influence and importance of scientific journals in a particular discipline. An integrative model using a set of metrics could represent a useful tool for researchers in their decisionmaking during the manuscript submission phase; it may even supplement new “standards” of the quality and validity of the research. Financial support and sponsorship Nil. Conflicts of interest There are no conflicts of interest. References


