The paper focuses on the robustness of rankings of academic journal quality and research impact of 10 leading econometrics journals taken from the Thomson Reuters ISI Web of Science (ISI) Category of Economics, using citations data from ISI and the highly accessible Research Papers in Economics (RePEc) database that is widely used in economics, finance and related disciplines. The journals are ranked using quantifiable static and dynamic Research Assessment Measures (RAMs), with 15 RAMs from ISI and five RAMs from RePEc. The similarities and differences in various RAMs, which are based on alternative weighted and unweighted transformations of citations, are highlighted to show which RAMs are able to provide informational value relative to others. The RAMs include the impact factor, mean citations and non-citations, journal policy, number of high quality papers, and journal influence and article influence. The paper highlights robust rankings based on the harmonic mean of the ranks of 20 RAMs, which in some cases are closely related. It is shown that emphasizing the most widely-used RAM, the two-year impact factor of a journal, can lead to a distorted evaluation of journal quality, impact and influence relative to the harmonic mean of the ranks. Some suggestions regarding the use of the most informative RAMs are also given.
References
[1]
ISI Web of Science. Journal Citation Reports; Essential Science Indicators; Thomson Reuters: New York, NY, USA, 2011.
[2]
Seglen, P.O. Why the impact factor of journals should not be used for evaluating research. BMJ: Br. Med. J. 1997, 314, 498–502, doi:10.1136/bmj.314.7079.497.
[3]
Chang, C.-L.; McAleer, M.; Oxley, L. What makes a great journal great in economics? The singer not the song. J. Econ. Surv. 2011, 25, 326–361, doi:10.1111/j.1467-6419.2010.00648.x.
[4]
Chang, C.-L.; McAleer, M.; Oxley, L. What makes a great journal great in the sciences? Which came first, the chicken or the egg? Scientometrics 2011, 87, 17–40, doi:10.1007/s11192-010-0335-0.
[5]
Chang, C.-L.; McAleer, M.; Oxley, L. Great expectatrics: Great papers, great journals, great econometrics. Econ. Rev. 2011, 30, 583–619.
[6]
Chang, C.-L.; McAleer, M.; Oxley, L. How are journal impact, prestige and article influence related? An application to neuroscience. J. Appl. Stat. 2011, 38, 2563–2573, doi:10.1080/02664763.2011.559212.
[7]
Chang, C.-L.; Maasoumi, E.; McAleer, M. Robust ranking of journal quality: An application to economics. Econ. Rev. 2013. (submitted). Available online: http://ideas.repec.org/s/emo/wp2003.html. (accessed on 4 October 2013).
[8]
Chang, C.-L.; McAleer, M. Ranking journal quality by harmonic mean of ranks: An application to ISI Statistics & Probability. Stat. Neerland. 2013, 67, 27–53, doi:10.1111/j.1467-9574.2012.00529.x.
[9]
Chang, C.-L.; McAleer, M. What do experts know about forecasting journal quality? A comparison with ISI research impact in finance. Ann. Financ. Econ. 2013, 8, 1–30.
[10]
Zimmermann, C. Academic rankings with RePEc. Econometrics, 2013. (submitted). Available online: as Working Paper 2012-023A, Federal Reserve Bank of St. Louis: St. Louis, MO, USA, 2012: ..
[11]
Eigenfactor. Available online: www.eigenfactor.org (accessed on 28 September 2013).
[12]
Bergstrom, C. Eigenfactor: Measuring the value and prestige of scholarly journals. C&RL News 2007, 68, 314–316.
[13]
Bergstrom, C.T.; West, J.D. Assessing citations with the Eigenfactor？ metrics. Neurology 2008, 71, 1850–1851, doi:10.1212/01.wnl.0000338904.37585.66.
[14]
Bergstrom, C.T.; West, J.D.; Wiseman, M.A. The Eigenfactor？ metrics. J. Neurosci. 2008, 28, 11433–11434, doi:10.1523/JNEUROSCI.0003-08.2008.
[15]
Hirsch, J.E. An index to quantify an individual’s scientific research output. Proc. Natl. Acad. Sci. USA 2005, 102, 16569–15572, doi:10.1073/pnas.0507655102.
[16]
IDEAS:Rankings. Available online: http://ideas.repec.org/top/ (accessed on 4 October 2013).
[17]
Chang, C.-L.; McAleer, M.; Oxley, L. Coercive journal self-citations, impact factor, journal influence and article influence. Math. Comput. Simul. 2013, 93, 190–197, doi:10.1016/j.matcom.2013.04.006.
