The dark side of post-publication peer review

On February 5, 2015

sockpuppets780

Peer review is a cornerstone of science: before work is published in a journal it is sent to an independent expert, who quickly and anonymously assesses its quality; if the work is up to scratch, it gets published. This seemingly sensible system is now, unfortunately, outdated – and easy to manipulate.

There are now so many journals that the same work can just be submitted again and again until it finally gets accepted. All of the work of past reviewers is never seen by the community. Even the reviews at the journal of publication are never posted. This made sense when a few publishers printed real physical journals and physically shipped them around the world, but in the modern age of online publishing all of these important critiques could easily be put online alongside the papers.

Some new journals are trying to implement a much more sensible and open system, for example F1000 Research and ScienceOpen. For more traditional publishers who like to keep the old system, websites have been set up where people can review the papers after publication. However there is a bit of a disagreement on anonymity: with some believing that reviewers deserve credit, such as Publons or PubMed Commons, while others, such as PubPeer, believe anonymity to be important.

Some more forward-thinking journals such as PLOS ONE have set up an online comment system. In theory, this is perfect: now scientists can openly and publicly discuss science. What could be better than the free flow of ideas? Online post-publication peer review definitely has had its successes – arsenic-based life, nanochopsticks, and STAP being the most famous. But it also seems to have its dark side.

My paper criticising the evidence for striped nanoparticles seems to be the gift that keeps on giving when it comes to exploring issues in scientific publishing. Previously we made quite a stir when traditional journals refused to grant us copyright to reuse figures as part of our criticism. (Not to mention the fact they were also completely uninterested in publishing the criticism itself.) For post-publication peer-review it fails to disappoint.

Starting off on a positive note: A pre-print of our paper was first published on the arXiv, and it was extensively discussed at PubPeer. One of the reviewers there was Brian Pauw, an expert with important insight into some of the newest data we discussed. He even blogged on it, and eventually became an author on the final paper, which was improved by his expertise.

That same PubPeer thread was not entirely a shining example of a victory for post-publication peer review. PubPeer have a very sensible policy of numbering registered users so that a conversation can be followed without removing anonymity. Registered users also have the option to forgo anonymity. To register, you must be first or last author on a published paper – the system helps to prevent sockpuppetry and spam. PubPeer, however, also allow unregistered users to comment, and these unregistered comments have no identifying numbers. Some unregistered comments were helpful, and some were also signed with a pseudonym to help people follow the conversation. Unfortunately the vast majority appeared to be a Gish Gallop, an endless number of inconsistent and repeated arguments to give the appearance of a true debate, from someone we began to refer to as UnReg. This is where complete anonymity fails us. Anyone who wants to disrupt an online debate can simply flood it, and then hide under the veil of anonymity. Other, signed unregistered comments from ‘ProfSTM’ and ‘Bionanochair’ agreed with UnReg, but due to the lack of any registration these could easily be the same person. Sockpuppetry is a huge problem in online debates.

Anonymous comments are useful for someone to raise an issue, which can then be answered, but lengthy debates with one-way anonymity are dangerous. If you put your name to a comment, you are probably willing to stand by it, and willing for people to know this is your stance. Anonymity allows one to comment first, and run away from it later. This is unfortunate, as there have been a number of very helpful anonymous comments.

Our paper also received post-publication peer review on the PLOS ONE website, where all commenters are asked to identify themselves with their first and last names and their geographic location. We first were contacted by someone claiming to be a Gustav Dhror. We engaged with him, but after a quick google we realised that it is a fake name. Not only is there no academic record for Dhror, there is no record of anyone of that name ever having existed. Once this was mentioned, ‘Gustav’ went quiet.

Soon afterwards, we were contacted by someone claiming to be Wei Chen, with a homepage of the Suzhou Institute of Nano-tech and Nano-bionics (SINANO). Checking the SINANO website, Chen is real. Finally, a real person to engage with! But the tone of the discussion felt oddly familiar, which prompted co-author Raphaël Levy (who has also blogged on this) to phone Chen just to confirm it was really him. The real Chen had no idea about the comments, and copied me into an email where he confirmed he had “never posted anything at PLOS ONE”.

Commandeering the identity of another researcher for the purpose of sockpuppetry is truly shocking. Furthermore, to accuse other people of dishonesty while using a stolen identity is beyond belief. We have no idea who is behind this, but I would wager that the comments of UnReg, ProfSTM, Bionanochair, Gustav Dhror and Wei Chen were all written by the same person.

Sockpuppetry and fake identities are what we expect from flame wars in YouTube comments, not in scientific discussions. I also highly doubt this is a random internet prankster. They are able to quote text from inside paywalled journal articles, which implies they are at an institution with access to these papers and have taken the time to read the literature.

Post-publication peer review has the potential to speed up and enhance scientific discussion. But, whether or not comments are anonymous, there need to be checks to verify the identity of commenters – otherwise it quickly degrades into farce.

Image: Sockpuppets are fun for children but a nuisance for scientists. Credit: Shutterstock/Africa Studio

About Julian Stirling

Julian Stirling is a postdoctoral guest researcher at NIST, specialising in atomic force microscopy and scanning tunnelling microscopy. He is interested in instrumentation and accurate measurements of nano-scale forces. He also has a keen interest in outreach.

24 Responses to The dark side of post-publication peer review

  1. This is a very relevant point being discussed. I have just felt exactly the sensation that many are abusing the PubPeer commenting system likewise, both as unregistered peers and as authors. I have seen several threads in which registered comments by anonymous peers raising issues in papers were met by registered replies claiming their comments were “pointless”, even “witch-hunt”. After some time, unregistered comments claimed by the authors started answering the questions directly in a polite manner, which is awkward since the same authors had used their IDs in registered posts elsewhere in the system. Then several unregistered comments start showing praising the authors, their answer, their work, etc. Making matters even more fishy, comments asking why the authors did not use their registered IDs in commenting get reported and deleted by moderation.
    Such cases left me with an uncomfortable sensation that not only authors had manipulated the thread, but also somehow PubPeer moderation. If true, this means peer review in fact stands uncorrected… Not sure if anyone else feels likewise?

  2. Also interesting was that the commenter on Pubpeer attributed a quote from a 2014 paper to a 2006 paper. If deliberate, it implies that the commenter is trying to retcon previous work. If accidental, it implies he is intimately familiar with the text of all the work (but just confuses which paper it was said in).

    Either way, not you regular troll!

  3. There is a need for sites that allow discussion of problems in publications. Every opportunity to discussion is an opportunity to sabotage discussion unless a sufficient defence is maintained. A possible defence is moderation. If done properly, it greatly reduces abuse of a dicussion forum and keeps the quality of discussion on an acceptable level. But such moderation is much hard work. In addition, if the intent is to have a scientific discussion, the moderators must have a good understanding of science and scientific discussion. That is easiest to arrange if the scope of the forum is narrow, but then we need a large number of such forums. Of course one web site may have many such forums with the same software and technical support but separate moderators and moderation policies. For example,
    http://www.sciphysicsforums.com/spfbb1/
    has, in spite of a different purpose, a good structure.

    The problem with PubPeer is that it tries to cover everything from physics to social sciences without sufficient resources for proper moderation. The maintainers understand that no moderation means failure and good moderation means hard work. They apparenly don’t understand or are just starting to find out that sloppy moderation means hard work and failure.

    Sloppy moderation means as much work as good moderation, for postings that moderators would not permit were not attempted in large scale.

  4. “But, whether or not comments are anonymous, there need to be checks to verify the identity of commenters – otherwise it quickly degrades into farce.”

    Never going to happen, purely for economic reasons. The old model of peer review relies on academic slave labor, i.e. Profs telling grad students to spend their “free time” as unpaid peer reviewers. This current iteration of peer review is based on reducing time and cost further by getting more volunteers and making it easier to access. But you cannot do background checks for free. Either journals pay for that, or they put the background-check burden back on the universities.

    I have a mild dislike of articles with this tone. They keep implying that the traditional peer review system is not full of sock puppets, and that you get good review from indentured grad students, and that the newer version of peer review would be better if we just had people do 10 times as much work.

    I’m probably just jaded from working in the academic publishing industry.

    • Hi Joe

      I have to say I disagree with the idea that for economic reasons identities cannot be validated. A first step is to simply only allow one registration per email address (with confirmation email) and make sure that is an institutional email address. This instantly stops people using fake email addresses, email addresses they don’t have access to, or making lots of accounts at say gmail. Further checks could be done, for example, by requiring you to register with an email address which is corresponding on a paper with a DOI. This sort of thing can be pretty easily scripted and is completely free. PubPeer already does something similar to get a Peer number, they also allow unregistered comments which is where our issues arose.

      I am also sorry that my tone annoyed you. I do, however, fail to understand your argument that the traditional peer-review is full of sock puppets. Sock-puppetry normally refers to people online making fake identities to praise themselves, or agree with themselves in an argument. Despite the many flaws of traditional peer review, the author of the paper can’t anonymously review it themselves, nor could a reviewer submit a second review under a different identity to agree with their first comment.

      I also never implied that you always get good reviews from traditional peer review. I feel you are putting words into my mouth. I have had horribly lazy reviews from traditional peer review. If traditional peer review always worked then repeatedly resubmitting to other journals would never work for a bad paper.

      Again, I don’t understand your point about 10 times the work. Post-publication peer review is entirely optional, but a great way for people who find issues with papers, or have questions to raise them publicly, if they want to. Normally this will be because they already found the issue when doing a literature search for research they are doing. Where does the extra work come in? The issue I am concerned with in this post is that the registration on many of these sites is based on the assumption that people will not try to make fake accounts and this has been abused in the case of my paper and has wasted a huge amount of my time and the time of my co-authors. This is why I think PPPR sites need to do more to verify identities. As I said at the top of this comment, doing this is actually pretty simple to program.

  5. “A first step is to simply only allow one registration per email address (with confirmation email) and make sure that is an institutional email address.”

    But this checking mechanism means people must own an institutional email address to be allowed to comment on articles?

    So you would restrict online comments to working scientists only?

    • I would argue that generally people with the expertise to review papers are most likely working scientists, or retired scientists who will probably still have email access.

      It would be best to allow anyone to comment, but it becomes a lesser of two evils argument. Sock-puppetry and fake identities cause a real problem, also cranks like creationists or anti-vaxxers could disrupt discussions on papers they don’t like. We can remove/reduce this at the risk of blocking a few informed comments from those without institutional email addresses. Such comments could still be aired on personal websites or social media.

      • The commenting systems could somehow indicate the level of confirmation a certain pseudonym has obtained: “valid email account”, “verified university affiliation”, “real name known to PubPeer”, etc. Here’s an example for domain verification. Also, software exists, that is built-from the ground up for civilised discussion. I think there are even ways to verify commenter credentials/claims while maintaining anonymity between the PPPR plattform and the commenter.

        In summary, all the options should remain available, but more quality control is necessary, for which a range of technology exists. Deploying it should be a decentralised effort by both OA publishers and research institutions.

        • Pub-Peer have tried something like this. To get a peer number you need to have verified your identity in full, and be the first or last author on a paper. They maintain complete anonymity, but this avoids sock puppetry. They also allow people with no registration to comment. Not a single person negatively commenting on our work had a peer number. The comments instead filled up with the unregistered submissions. This leaves us with a dilemma, we don’t want to ignore misrepresentations of our work, but we also don’t want to get sucked into a long argument with sock puppets.

  6. It wouldn’t just restrict it to working scientists- many scientists in Asia and Africa use gmail and yahoo addresses for perfectly valid reasons.

  7. The elephant in the room here is that the paper authored by Stirling (the author of this post) is full of inconsistencies and suspect claims.

    Rather than seriously considering the many objective criticisms that have been posted on the PubPeer and PLOS ONE pages of Stirling’s paper (links below), Stirling, Moriarty and Levy turned the debate into one where they (and post publication peer review!) are under attack.

    Moriarty and Levy have praised the idea of post publication peer review and of anonymity, to allow the young and powerless a voice in a pyramidal academic system. Yet when they are faced with serious criticisms to their work, they ignore these ideals, ridicule serious critique and put the emphasis on WHO are behind the criticisms rather than on the criticisms themselves. That’s hypocrisy as good as it gets.

    https://pubpeer.com/publications/B02C5ED24DB280ABD0FCC59B872D04

    https://pubpeer.com/publications/25402426

    http://www.plosone.org/article/Comments/info:doi/10.1371/journal.pone.0108482

    • JD,

      Thanks for including the links to the PubPeer threads. Interested physicsfocus readers can — if they haven’t yet lost the will to live after reading about this stripy nanoparticle saga for so long — scroll down through our extensive responses and judge for themselves as to whether we have put the emphasis on “who” is making the comments rather than the substance of what they’ve said. (I’ll note in passing that it’s just a little bit difficult to focus on who made the comments when not one proponent of Stellacci et al’s work has yet been willing to reveal their identity!).

      Here’s a link to the referees’ comments on our paper in return. Note that they don’t share your opinion of our paper as being full of inconsistencies.

      https://raphazlab.wordpress.com/2014/09/14/stripes-open-access-and-copyright-as-a-form-of-censorship/

      All the best,

      Philip

      • Philip,

        The emphasis on “who” is self-evident from this post of your colleague Stirling.

        On passing, you should also have said that no proponent of your criticisms has yet been willing to reveal their identity. On the one side we have you, Levy, Pauw and co-authors (one senior STM expert), and on the other Stellacci, de Feyter, Renner and co-authors (three senior STM experts). On the one hand, you have three anonymous reviewers of your paper (opinionated but technically bland), and on the other probably tens of anonymous reviewers and papers. On the one hand we have lots of unsatisfactorily answered criticisms on your paper, on the other hand we have your paper criticizing a body of work.

        Post-publication peer review has opened a large number of holes in your paper. Whether the holes are bigger or larger in number than those that you claim to have opened on Stellacci’s work is up to the experts to gauge. Your attitude and that of Levy has not help the debate; actually, it has backfired, as most of the community supports Stellacci.

        • “On passing, you should also have said that no proponent of your criticisms has yet been willing to reveal their identity.”

          This is wrong. Brian Pauw was initially a contributor in the PubPeer discussion for the preprint of our paper. He registered but did not initially reveal his identity. (We knew him as Peer 4). He subsequently revealed his identity and became a co-author of the paper.

          – “opinionated but technically bland”

          That does a great disservice to those referees who spent a considerable amount of time reading our paper and returning a comprehensive review. I’ll leave it up to the readers of physicsfocus to make up their own minds as to whether they agree with you.

          I would assume that you would not expect that Prof. Stellacci’s review of our paper – which he returned, unanonymously, as part of the PLOS ONE review process — would be “opinionated but technically bland”? And yet we didn’t have to make a single change to the arguments in our paper as a result of Prof. Stellacci’s review.

          –”Your attitude and that of Levy has not help the debate; actually, it has backfired, –as most of the community supports Stellacci.”

          Anyone can make this type of comment under the cloak of anonymity. This amounts to no more than a spineless ad hominem attack.

          I am absolutely fine with the scientific content of our work being criticised by anonymous critics. It’s when critics cross the line to ad hominem slurs that I question their integrity. My name is at the top of this comment, and every other comment I have made on Stellacci et al’s work. The same is true for Julian Stirling. And Raphael Levy.

          Either have the integrity to reveal your identity or stick to criticising the scientific content of our work.

        • P.S. I should also be noted that a significant number of the proponents of our criticisms of Stellacci et al’s work registered with PubPeer and were allocated a peer number (i.e. Peer 1, Peer 2, Peer 3, Peer 4…etc…).

          Not one of the proponents of Stellacci et al’s work has registered at PubPeer to be allocated a peer number. Each ‘criticism’ of our PLOS ONE paper has come from an unregistered submission (with all the associated issues re. identity theft and sockpuppetry to which Julian refers above).

  8. Bev:
    <>

    That is definitely true. Big institutions usually have official email addresses, and permanent faculty/staff in “lesser” institutions may have an official email address, but postdocs, phd students, etc. may very well not.

    It is possible, however, to work around this issue through the registration process, but it takes time and a moderator to do it. One solution might be to restrict one verified anonymous pseudonym per user and zero tolerance for any abuse. Maybe the reviewer can opt to be anonymous in some reviews and not in others. There could be some small literary value in posting an anonymous comment with an apt pseudonym for a particular instance, but maybe this is the price that we pay for those who would prefer to hide behind the wall of anonymity.

    I still think the ideal situation is that people simply cannot comment anonymously; after all, if one’s comments are really something worthy of print, those comments are also worthy of having a name on it. Corruption on the part of authors and reviewers was a problem with the traditional system, and it appears that corruption will probably remain with us in the new system as well — only the forms and strategies are changed.

    • ..and your comment has now been removed by PLOS ONE because, yet again, your identity couldn’t be verified and your profile did not adhere to PLOS ONE commenting guidelines. This is now the third time you’ve lied about this.

      • No lie. I am independent consultant Wei Chen.

        Here my scientific comments about your paper on PLOS website, for everyone to see:

        This paper has many flaws. In general Stirling et al grab evidence from few figures of body of literature on striped nanoparticles and make general statements that contradict other published works.

        Stirling et al claim “we show that the scanning probe data published to date provide no evidence for stripe formation and instead can be explained by a combination of instrumental artifacts, data selection, and observer bias” (page-2). However, when analyzing images for scanning artifacts they only analyze 1 image (Figure-1) out of many tens that have been published. It is granted that raw data may not be available for all images but surely it is for many.

        Then in Figure-2 Stirling et al compare 1 image of striped nanoparticles with simulated STM feedback results. No one doubts that one can simulate feedback ringing and generate stripes. Stellacci and colleagues have shown this in their papers, and demonstrated that one can distinguish between images that arise from feedback ringing and images that do not by looking at how spatial features change with tip speed. But Stirling et al ignored this.

        The conclusion of Stirling et al from Figures-2 and 3 is that “improper feedback loop settings produce stripes whose spacing depends on the loop gain”. This is no news to any STM scientist. But this does not mean that stripes cannot arise from physically reasonable arguments rather than imaging artifacts. Stirling et al. “forget” to criticize evidence in the body of literature of striped nanoparticles that shows how features from imaging artifacts can be distinguished from real features (spacing differences, changes with tip speeds,etc)

        Then Stirling et al attempt to show with Figure-4 that feature spacing of the stripes “falls within the broad background noise measured for the whole image”. This has been heavily criticized already here (http://www.plosone.org/annotation/listThread.action?root=84721) and in pubpeer (https://pubpeer.com/publications/B02C5ED24DB280ABD0FCC59B872D04) and dismissed by Stirling and Moriarty with secondary arguments such as poor quality images and interpolation issues. The problem with Figure-4 is that Stirling et al assumed that features in the images analyzed in Figure-4 all come from feedback ringing without saying so in the paper !
        If one assumes that there is feedback ringing in the whole image, their conclusion regarding broad background noise is not surprising but also irrelevant to the discussed images.

        In Figure-5 Stirling et al show that arithmetic addition of a sets of published trace and retrace images of same area showing stripes leads to particles without stripes. This has also been criticized before in pubpeer — for summation procedure to work, resolution of the images has to be good enough, which is not the case in Figure-5. If not, is like summing peaks and valleys that fall out of sync. Not surprisingly, one gets a mostly flat result. But Stirling et al omit this point and claim that summation method shows that there were no stripes in first place. They also omit to say that Stellacci and colleagues have demonstrated that the summation of trace and retrace images with better resolution shows clear stripes (Moglianetti et al, doi:10.1039/c3sc52595c).

        In the next Figure, Figure-6, Stirling et al. show that trace and retrace images sometimes do not match, which is also no news to STM researchers. But Stirling et al do not say that Stellacci and colleagues have published examples of trace and retrace images that show the same features on same place independent of scanning angle (also in Moglianetti et al).

        With Figure-7 and Figure-10, Stirling et al argue that PSD analyses show that features with random order can lead to a PSD with a broad shoulder. This is also well known, e.g. PSD of a gel or glass structure has a broad shoulder. But Stirling et al do not say that broad PSD shoulders for homoligand (usually non striped) and mixed ligand striped nanoparticles correspond to different spacings, and that these can be distinguished. This is explained well in Ong et al. (doi 10.1021/nn402414b).

        In Figure 8 it is shown that tip artifacts can generate “Janus ” nanoparticles. Agreed, but this does not mean that stripe features with the spacing shown by Stellacci and colleagues can be generated by tip artifacts, or that all the images with striped nanoparticles arise from tip artifacts. If one finds a broken glass on floor, it is possible to conclude that someone might have hit the glass with a hammer, but there are of course more plausible conclusions such as the glass falling from table.

        Stirling et al criticize the work in Moglianetti et al in Figure-9 by writing that “there is strong observer bias in the identification of stripes”. However, stripe spacings in Moglianetti are consistent with the spacings in all other works of Stellacci where stripes can be seen by identified by eye without doubt and where observer bias is not possible (example: Ong et al doi:10.1039/c4cc04114c).

        There are other flaws in this paper that have been discussed elsewhere. The arguments of Stirling et al pick a few images of striped nanoparticles published in the literature and apply one of their battery of criticisms in a selective and biased way; if it is not observer bias, it’s feedback ringing or other tip artifacts, or lack of features in a summation or similar PSD broad peaks from random features. The papers of Stellacci et al have analyzed all this points, but Stirling et al have ignored this and applied arguments inconsistenly. Stirling et al also ignored that spacings from PSDs agree for tens of images from different labs in different years. They also “forgot” that observer bias is negligible in the most recent images (notably 10.1039/c4cc04114c) and that feedback ringing has proved to be negligible in many published papers (Moglianetti et al, Ong et al, Jackson et al…).

  9. It is quite odd to see critics of anonymous PPPR. For one simple reason: most traditional peer review is built upon the concept of anonymity, either single-blind or double-blind. In these cases, authors do not know the identity of “peers”, and vice versa. Yet, the decision to accept or reject papers is based on the unknown identity, even if the editor knows the identity. Recent cases have shown how editors have been duped by the use of false e-mail addresses and identities on online submission systems. But the bottom line is, anonymity and unknown identities form the core of traditional peer review, so those who cry foul about such a process in PPPR are either living in the clouds, or fail to actually see the parallels. PPPR is essential as it allows for gaps and errors not caught in traditional peer review to be detected. The anti-PPPR crowd represents either those who have been challenged publicly, or those editors whose papers in their journals have been challenged. I see no others crying foul, which indicates that those crying foul represent, to some extent, a biased pool. I agree that sock-puppetry and false identities are no-go methods for PPPR, but all errors, minor or major, have the right to be pointed out, and those in powerful positions have no right in interfering with the democratic process of making a comment, or pointing out possible or actual errors. “Recent” events at PubPeer reveal that comment manipulation* may be the downfall of PubPeer, because it reveals that either a) outside forces are influencing the decisions being made, or b) the moderators are clueless about how to handle increasingly borderline cases. Unless PubPeer gets its act together quickly, and employs a peer-board system to moderate even anonymous comments, it’s going to become a failed system, and may hurt PPPR more than it benefits it.
    * https://pubpeer.com/topics/1/1A0BDA96C3E7BD263364577E50E08F#fb25366 (see comments on February 18 and 19)

  10. The comparison to traditional peer review is entirely misplaced. With traditional peer review there is an editorial board and an editor (or editors) moderate the comments. A reviewer is fully aware that their name and affiliation are both known to the editorial board. Any type of slur or slander would not be tolerated by any editorial board worth its salt. Moreover, a good editor can assess reviewers and authors’ comments and reach an independent decision.

    In the case Julian discusses in his post above, the issue is not with anonymous commenting. When this is based on the science, and conforms to certain ethical standards (such as not stealing someone’s identity!), then there isn’t a problem. The points of Julian’s blog post are that (a) identity theft is shockingly underhand behaviour and simply should not be tolerated, and (b) sockpuppetry is similarly inherently dishonest and has no place in scientific debate. This type of behaviour does not happen — or, at least, there are checks and balances in place which make it much more unlikely to happen — in traditional peer review.

    I am a huge fan of PPPR. I am not a huge fan of identity theft, sockpuppetry, and despicably underhand behaviour. These two statements are not incompatible.