There are two recent posts on citation practices that merit comparison.
The first is Citations for sale by Megan Messerly, which reads in part:
The U.S. News and World Report rankings have long been regarded as the Bible of university reputation metrics.
But when the outlet released its first global rankings in October, many were surprised. UC Berkeley, which typically hovers in the twenties in the national pecking order, shot to third in the international arena. The university also placed highly in several subjects, including first place in math.
Even more surprising, though, was that a little-known university in Saudi Arabia, King Abdulaziz University, or KAU, ranked seventh in the world in mathematics — despite the fact that it didn’t have a doctorate program in math until two years ago.
“I thought this was really bizarre,” said UC Berkeley math professor Lior Pachter. “I had never heard of this university and never heard of it in the context of mathematics.”
As he usually does when rankings are released, Pachter received a round of self-congratulatory emails from fellow faculty members. He, too, was pleased that his math department had ranked first. But he was also surprised that his school had edged out other universities with reputable math departments, such as MIT, which did not even make the top 10.
For the sake of ranking
It was enough to inspire Pachter to conduct his own review of the newly minted rankings. His inquiry revealed that KAU had aggressively recruited professors from a list of top scientists with the most frequently referenced papers, often referred to as highly cited researchers.
“The more I’ve learned, the more shocked and disgusted I’ve been,” Pachter said.
Citations are an indicator of academic clout, but they are also a crucial metric used in compiling several university rankings. There may be many reasons for hiring highly cited researchers, but rankings are one clear result of KAU’s investment. The worry, some researchers have said, is that citations and, ultimately, rankings may be KAU’s primary aim. KAU did not respond to repeated requests for comment via phone and email for this article.
On Halloween, Pachter published his findings about KAU’s so-called “highly-cited researcher program” in a post on his blog. It elicited many responses from his colleagues in the comment section, some of whom had experience working with KAU.
Pachter refers to earlier work of his own that makes claims about ranking universities highly suspect so one wonders why the bother?
I first saw this in a tweet by Lior Pachter.
In any event, you should also consider: Best Papers vs. Top Cited Papers in Computer Science (since 1996)
From the post:
The score in the bracket after each conference represents its average MAP score. MAP (Mean Average Precision) is a measure to evaluate the ranking performance. The MAP score of a conference in a year is calculated by viewing best papers of the conference in the corresponding year as the ground truth and the top cited papers as the ranking results.
Check the number out (the hyperlinks take you to the section in question):
AAAI (0.16) | ACL (0.13) | ACM MM (0.17) | ACSAC (0.27) | ALT (0.07) | APSEC (0.33) | ASIACRYPT (0.16) | CHI (0.2) | CIKM (0.19) | COMPSAC (0.6) | CONCUR (0.09) | CVPR (0.25) | CoNEXT (0.16) | DAC (0.07) | DASFAA (0.27) | DATE (0.11) | ECAI (0.0) | ECCV (0.42) | ECOOP (0.22) | EMNLP (0.14) | ESA (0.4) | EUROCRYPT (0.07) | FAST (0.18) | FOCS (0.07) | FPGA (0.59) | FSE (0.4) | HPCA (0.31) | HPDC (0.59) | ICALP (0.2) | ICCAD (0.13) | ICCV (0.07) | ICDE (0.48) | ICDM (0.13) | ICDT (0.25) | ICIP (0.0) | ICME (0.43) | ICML (0.12) | ICRA (0.16) | ICSE (0.24) | IJCAI (0.11) | INFOCOM (0.18) | IPSN (0.69) | ISMAR (0.57) | ISSTA (0.33) | KDD (0.33) | LICS (0.26) | LISA (0.07) | MOBICOM (0.09) | MobiHoc (0.02) | MobiSys (0.06) | NIPS (0.0) | NSDI (0.13) | OSDI (0.24) | PACT (0.37) | PLDI (0.3) | PODS (0.13) | RTAS (0.03) | RTSS (0.29) | S&P (0.09) | SC (0.14) | SCAM (0.5) | SDM (0.18) | SEKE (0.09) | SIGCOMM (0.1) | SIGIR (0.14) | SIGMETRICS (0.14) | SIGMOD (0.08) | SODA (0.12) | SOSP (0.41) | SOUPS (0.24) | SPAA (0.14) | STOC (0.21) | SenSys (0.4) | UIST (0.32) | USENIX ATC (0.1) | USENIX Security (0.18) | VLDB (0.18) | WSDM (0.2) | WWW (0.09) |
Universities and their professors conferred validity on the capricious ratings of U.S. News and World Report. Pachter’s own research has shown the ratings to be nearly fictional for comparison purposes. Yet at the same time, Pachter decrys what he sees as gaming of the rating system.
Crying “foul” in a game of capricious ratings, a game favors one’s own university, seems quite odd. Social practices at KAU may differ from universities in the United States but being ethnocentric about university education isn’t a good sign for university education in general.