Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
×
Education Technology IT

Google Scholar is Manipulatable (arxiv.org) 16

Abstract of a paper [PDF] the on pre-print server Arxiv: Citations are widely considered in scientists' evaluation. As such, scientists may be incentivized to inflate their citation counts. While previous literature has examined self-citations and citation cartels, it remains unclear whether scientists can purchase citations. Here, we compile a dataset of about 1.6 million profiles on Google Scholar to examine instances of citation fraud on the platform. We survey faculty at highly-ranked universities, and confirm that Google Scholar is widely used when evaluating scientists. Intrigued by a citationboosting service that we unravelled during our investigation, we contacted the service while undercover as a fictional author, and managed to purchase 50 citations. These findings provide conclusive evidence that citations can be bought in bulk, and highlight the need to look beyond citation counts.
This discussion has been archived. No new comments can be posted.

Google Scholar is Manipulatable

Comments Filter:
  • This is an example of Goodhart's Law [wikipedia.org]:

    "When a measure becomes a target, it ceases to be a good measure"

    There is an article in this week's Economist about how the peer review system and promotions based on publishing and accumulating citations are failing. Companies that might benefit from research mostly see research from universities as worthless, yet also do much less of their own research because of perverse incentives. This leads to less innovation and lower productivity growth.

    Universities are failing [economist.com]

    • by ebcdic ( 39948 )

      "Of course, there is a place for speculative research unconstrained by economic concerns, but perhaps we have gone too far in that direction". But that's the very opposite of what's happening. It's getting harder and harder to get funding for research unless you can demonstrate the likelihood of short-term commercialization.

      • It's getting harder and harder to get funding for research unless you can demonstrate the likelihood of short-term commercialization.

        Do you have a citation for that? That's the opposite of what the Economist says.

        Or, more accurately, the Economist says that the research isn't actually useful, but not necessarily that it wasn't expected to be useful.

        • by ebcdic ( 39948 )

          No, I don't have a citation, I have 40 years of experience. If you want a citation you'll need to demonstrate the short-term commercial benefit of performing such a study.

    • Companies that might benefit from research mostly see research from universities as worthless, yet also do much less of their own research because of perverse incentives.

      For-profit companies do their own research because they want proprietary innovations. If companies just relied on public research, then they become manufacturers of commodities, which is not a great position for those companies.

      Most published work is either incremental or not all that relevant. That's true for both academia and corporate research. However, a small percentage is useful, and an even smaller percentage is seminal. Missing out on the useful and especial seminal work is a bad thing.

      Most good

  • Apart from doing important, relevant, useful research, there's other ways for academics to increase their citation counts. In the social sciences at least, one is to make provocative claims that will more than likely draw the ire & indignant outrage of other academics who, in turn, will publish papers with rebuttals & counter-arguments, all citing the offending paper.

    Another is to take one research project but publish several papers on it, each reporting a different feature or aspect, rather than
    • Who knows of any more?

      Publishing essentially the same thing in multiple places.

      Getting your name added to your students' papers even though you did little to no actual work on those papers.

      Working on research with a team and then break your findings into multiple parts, then all cite each other.

      Contributing something to the CERN, then getting your name added to thousands of papers that you know nothing about.

      Letting a pharmaceutical company write papers for you.

      Create your own journal and self-publish anything you couldn't

      • Yes, & I'm sure there are others. I guess they could play a kind of whack-a-mole game where they to regulate, ban, etc., new ways of skewing & outright defrauding the stats, rather than dealing with the cause of the problem; mindless metrics & overly competitive, race-to-the-bottom style academic employment. Knowledge is only valuable if its of high enough quality. I very much doubt that papers published purely to fill metrics quotas are going to be of good enough quality to be valuable. So what
  • The notion that a paper has merit because it has a lot of cites is intrinsically faulty. It's easy to posit a paper that is so advanced that no one cites it all all, since it obviates all existing research. This is more of the "closed shop" mentality working in academia, it's the same with the peer review process, which has shown time and again to be easily manipulated. This is just another and obviously flawed example of how easy it is to show "measurables" to non-technical people.

  • by Roger W Moore ( 538166 ) on Monday February 12, 2024 @05:57PM (#64235304) Journal
    Google Scholar is incredibly inaccurate because there is no way to control the algorithm and tell it to exclude papers that have nothing to do with you. I tried for a while to clear out all the papers it added to my profile that had nothing to do with me but I work in a field with large publication rates from big collaborations and there was just no way to keep up with the deluge. As a result, I would not trust Google Scholar's citation record for anything important - if you want accurate information you need to use something like ORCID.
    • by pjt33 ( 739471 )

      I'm not sure the field you work in is relevant. It attributes at least one of my papers to someone with the same name in a different field, and it won't let me correct the data because I don't have an email address which matches its whitelist.

  • The system is not broken, except for the incompetent. No one relies solely on citation counts to hire or to give awards. It is a first-level weed-out mechanism. After that first step, the specific papers are examined, and the quality and impact of those papers will be considered along with other things.

    In a way, the citation count is like a GPA. No one hires a student based solely on the basis of a 4.0 GPA. The 4.0 can lead to an initial interview, but it doesn't replace the interview.

  • Even worse for academic publications is that on average claimed results in peer reviewed journals are slightly less than 50% reproducible for many fields. See for instance https://royalsocietypublishing... [royalsocie...ishing.org] ... and the principals in these fields are well aware of this problem.
  • Since this paper is a preprint published on arXiv, and seems that it hasn't been peer reviewed. How do we know that it isn't a LLM creation in its entirety? How do we know that in general of any paper published in the LLM era? Attentive peer review, that's our only defense. In this age of paper mills and "minimum publishable units", pay-to-pay publishers, all bets are off. The publishing game seems to be irreparably damaged. It is clear that the sites that compile these metrics can do much more to im

Nothing succeeds like excess. -- Oscar Wilde

Working...