Measuring Academic Impact

December 10, 2013

The academic adage “publish or perish” certainly keeps professors busy trying to communicate their research, but how do they measure the real impact of their work? CEU Visiting Professor of Network Science Albert-Laszlo Barabasi sought to quantify the influence of academic papers and subsequent progression of knowledge in a field by creating a predictive tool to determine just how successful an academic paper will be.

In October, Barabasi and his co-authors published their paper “Quantifying Long-Term Scientific Impact” in the journal Science.

“The basic question is: How do we quantify the impact of our work and perceived impact?” Barabasi said. “This is on everyone's mind. Academics want to know: Is this a great paper? Traditionally the way we measure impact is through word of mouth. The assumption is if you publish in a better journal, it's a better paper.”

Interestingly, although some journals enjoy more prestige than others, Barabasi and his colleagues found that the journal in which a paper is published has no relevance to the overall impact of a paper. The model that Barabasi and his colleagues created takes into account the number of times a paper is cited in other academic works, “...allowing us to collapse the citation histories of papers from different journals and disciplines into a single curve, indicating that all papers tend to follow the same universal temporal pattern,” the authors wrote.


CEU Visiting Professor of Network Science Laszlo Barabasi discusses the tool he and his colleagues developed to determine the impact of academic papers. Image credit: CEU/Daniel Vegel

“No matter where you publish, the paper's impact follows a very precise mathematical law,” Barabasi said. “You can fit this to the citations you've gotten so far and this allows you to predict the total impact of the paper. This is the total number of citations that this paper will get in its 'lifetime.' We've tested this back about 100 years with a number of physics papers and it works very accurately.”

Three parameters characterize a paper in the model: first, in what year the paper gets the most citations; second, the actual decay rate; and, third, what Barabasi and his team call the paper's “fitness,” or how fast it will “jump up” in citations. The first two parameters really matter if you want to determine when the paper's peak will come. The typical path of a paper finds the most citations popping up in the first two to two-and-a-half years and then a slow decay. However, it is only the “fitness” that matters in the long-run (for the whole “life” of the paper).

Barabasi and his colleagues have even used the tool to retrace the success of their own papers with accurate results. What initially piqued Barabasi's interest in citation curves was a network science paper that he published in 1999 that did not seem particularly well received within the first few years of its “life” but, with time, was cited more and more. It reached its peak only a decade after its publication, which is, Barabasi said, a rarity.

Far from being a popularity measurement, the predictive model could have potential policy implications, particularly in the academic world. Knowing the possible impact of a paper could, for instance, help compare younger professors with older professors who might have published more, hence had more of a track record in the field.

For more information on the “Science” paper, see http://iononfaccioniente.wordpress.com/2013/10/04/paper-quantifying-long-term-scientific-impact/?amp;co=f000000009816s-1158206718.

For more information about CEU's Center for Network Science, see https://cns.ceu.hu/.