When to trust (and not to trust) peer reviewed science

  • Written by Merlin Crossley, Deputy Vice-Chancellor Academic and Professor of Molecular Biology, UNSW
When to trust (and not to trust) peer reviewed science

The article is part of our occasional long read series Zoom Out, where authors explore key ideas in science and technology in the broader context of society.

The words “published in a peer reviewed journal” are sometimes considered as the gold standard in science. But any professional scientist will tell you that the fact an article has undergone peer review is a long way from an ironclad guarantee of quality.

To know what science you should really trust you need to weigh the subtle indicators that scientists consider.

Read more: Why I disagree with Nobel Laureates when it comes to career advice for scientists

Journal reputation

The standing of the journal in which a paper is published is the first thing.

For every scientific field, broad journals (like Nature, Science and Proceedings of the National Academy of Science) and many more specialist journals (like the Journal of Biological Chemistry) are available. But it is important to recognise that hierarchies exist.

Some journals are considered more prestigious, or frankly, better than others. The “impact factor” (which reflects how many citations papers in the journal attract) is one simple, if controversial measure, of the importance of a journal.

In practice every researcher carries a mental list of the top relevant journals in her or his head. When choosing where to publish, each scientist makes their own judgement on how interesting and how reliable their new results are.

If authors aim too high with their target journal, then the editor will probably reject the paper at once on the basis of “interest” (before even considering scientific quality).

If an author aims too low, then they could be selling themselves short – this could represent a missed opportunity for a trophy paper in a top journal that everyone would recognise as significant (if only because of where it was published).

Read more: Not just available, but also useful: we must keep pushing to improve open access to research

Researchers sometimes talk their paper up in a cover letter to the editor, and aim for a journal one rank above where they expect the manuscript will eventually end up. If their paper is accepted they are happy. If not, they resubmit to a lower ranked, or in the standard euphemism, a “more specialised journal”. This wastes time and effort, but is the reality of life in science.

Neither editors nor authors like to get things wrong. They are weighing up the pressure to break a story with a big headline against the fear of making a mistake. A mistake in this context means publishing a result that becomes quickly embroiled in controversy.

To safeguard against that, three or four peer reviewers (experienced experts in the field) are appointed by the editor to help.

The peer review process

At the time of submitting a paper, the authors may suggest reviewers they believe are appropriately qualified. But the editor will make the final choice, based on their understanding of the field and also on how well and how quickly reviewers respond to the task.

The identity of peer reviewers is usually kept secret so that they can comment freely (but sometimes this means they are quite harsh). The peer reviewers will repeat the job of the editor, and advise on whether the paper is of sufficient interest for the journal. Importantly, they will also evaluate the robustness of the science and whether the conclusions are supported by the evidence.

This is the critical “peer review” step. In practice, though, the level of scrutiny remains connected to the standing of the journal. If the work is being considered for a top journal, the scrutiny will be intense. The top journals seldom accept papers unless they consider them to be not only interesting but also water tight and bullet proof – that is they believe the result is something that will stand the test of time.

If, on the other hand, the work is going into a little-read journal with a low impact factor, then sometimes reviewers will be more forgiving. They will still expect scientific rigour but are likely to accept some data as inconclusive, provided the researchers point out the limitations of their work.

Knowing this is how the process goes, whenever a researcher reads a paper they make a mental note of where the work was published.

Read more: What was missing in Australia's $1.9 billion infrastructure announcement

Journal impact factor

Most journals are reliable. But at the bottom of the list in terms of impact lie two types of journals:

  1. respectable journals that publish peer reviewed results that are solid but of limited interest – since they may represent dead ends or very specialist local topics

  2. so-called “predatory” journals, which are more sinister – in these journals the peer review process is either superficial or non-existent, and editors essentially charge authors for the privilege of publishing.

Professional scientists will distinguish between the two partly based on the publishing house, and even the name of the journal.

The Public Library of Science (PLOS) is a reputable publisher, and offers PLOS ONE for solid science – even if it may only appeal to a limited audience.

Read more: Universities spend millions on accessing results of publicly funded research

Springer Nature has launched a similar journal called Scientific Reports. Other good quality journals with lower impact factors include journals of specialist academic societies in countries with smaller populations – they will never reach a large audience but the work may be rock solid.

Predatory journals on the other hand are often broad in scale, published by online publishers managing many titles, and sometimes have the word “international” in the title. They are seeking to harvest large numbers of papers to maximise profits. So names like “The International Journal of Science” should be treated with caution, whereas the “Journal of the Australian Bee Society” may well be reliable (note, I invented these names just to illustrate the point).

The value of a journal vs a single paper

Impact factors have become controversial because they have been overused as a proxy for the quality of single papers. However, strictly applied they reflect only the interest a journal attracts, and may depend on a few “jackpot” papers that “go viral” in terms of accumulating citations.

Additionally, while papers in higher impact journals may have undergone more scrutiny, there is more pressure on the editors and on the authors of these top journals. This means shortcuts may be taken more often, the last, crucial control experiment may never be done, and the journals end up being less reliable than their reputations imply. This disconnect sometimes generates sniping about how certain journals aren’t as good as they claim to be – which actually keeps everyone on their toes.

While all the controversies surrounding impact factors are real, every researcher knows and thinks about them or other journal ranking systems (SNP – Source Normalised Impact per Paper, SJR – Scientific Journal Rankings, and others) when they are choosing which journal to publish in, which papers to read, and which papers to trust.

Read more: Science isn't broken, but we can do better: here's how

Nothing is perfect

Even if everything is done properly, peer review is not infallible. If authors fake their data very cleverly, for example, then it may be difficult to detect.

Deliberately faking data is, however, relatively rare. Not because scientists are saints but because it is foolish to fake data. If the results are important, others will quickly try to reproduce and build upon them. If a fake result is published in a top journal it is almost certain to be discovered. This does happen from time to time, and it is always a scandal.

Errors and sloppiness are much more common. This may be related to the increasing urgency, pressure to publish and prevalence of large teams where no one may understand all the science. Again, however, only inconsequential mistakes will survive – most important errors will quickly be picked up.

Read more: Not just available, but also useful: we must keep pushing to improve open access to research

Can you trust the edifice that is modern science?

Usually, one can get a feel for how likely it is that a piece of peer reviewed science is solid. This comes through relying on the combination of the pride and the reputation of the authors, and of the journal editors, and of the peer reviewers.

So I do trust the combination of peer review system and the inherent fact that science is built on previous foundations. If those are shaky, the cracks will appear quickly and things will be set straight.

I am also heartened by new opportunities for even better and faster systems that are arising as a result of advances in information technology. These include models for post-publication (rather than pre-publication) peer review. Perhaps this creates a way to formalise discussions that would otherwise happen on Twitter, and that can raise doubts about the validity of published results.

Read more: Bored reading science? Let's change how scientists write

The journal eLife is turning peer review on its head. It’s offering to publish everything it deems to be of sufficient interest, and then letting authors choose to answer or not answer points that are raised in peer review after acceptance of the manuscript. Authors can even choose to refrain from going ahead if they think the peer reivewers’ points expose the work as flawed.

ELife also has a system where reviewers get together and provide a single moderated review, to which their names are appended and which is published. This prevents the problem of anonymity enabling overly harsh treatment.

All in all, we should feel confident that important science is solid (and peripheral science unvalidated) due to peer review, transparency, scrutiny and reproduction of results in science publication. Nevertheless in some fields where reproduction is rare or impossible – long term studies depending on complex statistical data – it is likely that scientific debate will continue.

But even in these fields, the endless scrutiny by other researchers, together with the proudly guarded reputations of authors and journals, means that even if it will never be perfect, the scientific method remains more reliable than all the others.

Authors: Merlin Crossley, Deputy Vice-Chancellor Academic and Professor of Molecular Biology, UNSW

Read more http://theconversation.com/when-to-trust-and-not-to-trust-peer-reviewed-science-99365

Australia Today

10 Unmissable Attractions for first time visitors to Melbourne

News Company

Melbourne, the most fantastic city in Australia! If this is your first ever travel experience in Melbourne, well it is guaranteed that you are never going to forget it. The metropolitan hub has a lot to offer to its visors, both within the city and the outskirts. There are so many attractions tha...

Read more

Grass trees aren't a grass (and they're not trees)

John Patykowski, Plant ecologist, Deakin University

Xanthorrhoea have no real trunk – just tightly packed leaves. CC BY-SASign up to the Beating Around the Bush newsletter here, and suggest a plant we should cover at batb@theconversation.edu.au.Grass trees (genus Xanthorrhoea) look like they were imagined by Dr Seuss. An unmistakable tuft of wi...

Read more

Peter Grimes is a thrilling and moving staging of the great English opera

Trevor Jones, Lecturer in Musical Theatre, Griffith University

Stuart Skelton in the title role of Peter Grimes.Brisbane FestivalReview: Peter Grimes, Brisbane FestivalPeter Grimes, one of the centrepiece events of this year’s Brisbane Festival, is a remarkable collaboration between the festival, Opera Queensland, Philip Bacon AM, the Queensland Performin...

Read more

Michelle Grattan on strawberries, Sudmalis, schools, and the au pair affair

Michelle Grattan, Professorial Fellow, University of Canberra

Michelle Grattan speaks with University of Canberra Vice-Chancellor Deep Saini about the week in politics. They discuss rushed legislation on strawberry contamination, the Liberals continued women problem with Ann Sudmalis’ announcement she won’t recontest her seat, the government’...

Read more

Super. If Labor really wanted to help women in retirement, it would do something else

Brendan Coates, Fellow, Grattan Institute

Labor's plan to pay super to women on paid parental leave would barely boost their retirement incomes.ShutterstockWhen it comes to the gender gap in retirement incomes, symbolism appears to matter more than actually achieving something.Labor’s plan to add super contributions to government-fund...

Read more

Is it time for Australia to be more open about research involving animals?

Tyler Paytas, Research Fellow in Philosophy, Australian Catholic University

Most people never have the chance to see how animals live in laboratories.from www.shutterstock.com The use of animals in scientific research is a complex ethical issue, and these studies typically take place behind closed doors.But since 2012, more than 120 of Britain’s universities, research...

Read more

Holidays & Travel

Things to Do and See in Ballina, Australia

The intricate coastline of New South Wales is renowned for its sumptuous, decadent beaches, hidden resorts and secret camping spots. A long and slow ride down the winding highway that...

Guide to Travel Necessities for 2018

Sure, it sounds like quite an adventure to hit an open road with nothing but a one-way ticket and an open mind, but the harsh reality is – you won’t...

Top Sightseeing Spots in New Zealand

New Zealand is one of the most fascinating locations for anyone to visit in the world. After all, it’s a place with a variety of unique landscapes, but it isn’t...

Must-Pack Items for a Travel Blogging Boss

As a travel blogger, you get to see and explore so many wonderful destinations and meet various interesting people, but before you embark on a new adventure you need to...

Luxury Camping Adventures Around the World

Perhaps in direct response to the dominance of technology in the modern world, many travelers are feeling a strong draw towards nature. While some are able to fully embrace their...

Dredging works underway Plantation Island

Tourists and workers travelling to popular tourist destination Plantation Island, Fiji, are set to benefit from safer access when marine channel dredging is completed at the end of June 2018...

Top 5 Tips for Traveling With Dogs

Travelling with your dog can be thrilling, but sometimes it can turn into a real nuisance. Unless you know how to properly restrain the pooch, whether it’ll be able to...

Top Island Destinations for 2018

When we picture island destinations in our minds, we often imagine a tropical beach of white sand, a sun lounger laid out in the shade of a line of palm...

Why Car Rental Excess Insurance is a Smart Choice

Some car rental agencies include basic damage cover in the standard rental fee, with the option to add on liability reduction, or excess insurance. The addition of excess insurance normally...

Fashion & Beauty

Groom To Do List

Sorry Aussie Men, you no longer just pay for the alcohol, flowers and marriage license. Times have certainly changed. Wedding Expert shares what Australian Grooms are now expected to do before...

Simple Secrets To A More Confident Smile

We all know just how important first impressions are. After a handshake, what’s one of the first things people will notice about you? It’s your smile. A friendly smile is...

Improve Your Body Confidence and Feel Great

No matter how good you feel about yourself, sometimes you just can’t help but see a few little flaws that seem gigantic to you and make you feel self-conscious. Whether...

First-Timer Tips For Your First Lip Filler Appointment

Getting injectable lip fillers can be a quick and effective way to plump up your smile with minimal drama. They’re a cost-effective and safe way to boost the natural shape...

The Most Common Skin Conditions and How to Deal with Them

We often underestimate the importance of our skin, but as the largest organ in our body, and our primary layer of defense, it is constantly affected by many environmental factors...

Beach Babe Basics: Everything You Need to Look Stylish at the Beach

Summer is right around the corner, and this year, we all should make the best of it. Right now, it is not just about looking stylish on the streets, it...

HEALTH

Virectin Reviews and Results (LATEST) - Honest Review

Virectin is a potent, natural male enhancement supplement, recommended to all those who are interested in improving their sexual performance. According to the manufacturers, it can provide a positive effect...

How to Choose the Best Lasik Surgeon?? It’s as easy as it can get!

Whether you have had a vision problem since childhood, or you have developed one recently, eye surgeries are the answer to all your problems. It can be anything, ranging from...

Laser Liposuction: The Myths and Facts

Laser liposuction is a revolutionary new way to get rid of unwanted fat without the downtime experienced with traditional liposuction. This procedure offers a number of amazing benefits and is...

How To Show Your Elderly Relatives Love

At every stage of our lives, there’s one thing we truly need above all else: love. As our elderly parents, grandparents, or other relatives we care about grow older, it...

9 Ways to Recognize the Early Signs of Dementia

Dementia is a condition that is, unfortunately, very present among our elderly population. What makes this fact even worse is that, in spite of our best intentions and will to...

Co-Occurring Disorders and Overcoming Addiction

Co-occurring disorders, also known as dual diagnosis, refer to the combination of substance abuse and the presence of a mental disorder.  Many individuals who enter treatment for substance abuse are...

LIFE STYLE

Should You Repair Or Replace Your Air Conditioning Unit?

Whenever the summer arrives to the hilt and you feel that your air conditioning isn’t up to the mark, there is always one question that pops up into your mind...

6 Rug Trends That Will Boost Style and Comfort in Your Home

If you want to break up that boring and colorless look of your home, one of the easiest and quickest ways is to throw in a rug! Check out these...

Put your problems to rest with Antenna Genie

When the idea of television was first being tested, Darryl Zanuck of 20th Century Fox famously remarked that “no one would want to stare at a plywood box every night”...

Top 5 Design Trends in Commercial Interior

Commercial buildings include offices, restaurants, hotels, etc. and their design can play a crucial role in all the important aspects of a business such as brand reinforcement, customer engagement and...

Add a little colour to your move - and life!

How a little ‘colour’ and planning can take your Move, and make it a more enjoyable one Colour plays a huge role in our lives, it can take us up, or...

6 Simple Rules for Redesigning Your Kitchen

How do you feel about your kitchen? Well, if you feel like your home needs a bit of lifting up, one of the best and most effective places to start...

FOOD & DINING

Morning greens: the breakfast skincare regime

It’s safe to assume that most Aussies fail to eat enough greens. Although this was momentarily resolved with the rise and popularity of the green juice movement, the lack of...

Best Hidden Gem Hong Kong Restaurants

You finally find yourself enjoying the endless beauties of China and this fascinating city. In a place with so much variety, all your senses will be wide awake, and your...

JOHNNY DI FRANCESCO NAMED OFFICIAL AMBASSADOR OF EAT STREET MELBOURNE

Johnny Di Francesco, world champion pizzaiolo and owner of Melbourne’s Gradi Group of restaurants, is excited to announce he has been named official ambassador for Eat Street Melbourne 2018. As Ambassador...

The best Mother's Day gift EVER!

Whether it’s to celebrate a special occasion, or just wind down at the end of a long day, there’s nothing much better than pouring yourself a glass (or two…) of...

Sing for Your Supper at Chargrill Charlie’s

To celebrate the final curtain call of the Barden Bellas, Chargrill Charlie’s is giving you the chance to Sing for your Supper! Show off your best acapella vocals to win...

MAKE IT WINE TIME, ANY TIME

Whether it’s to celebrate a special occasion, or just wind down at the end of a long day, there’s nothing much better than pouring yourself a glass (or two…) of...