Google AI


Modern Australian

A new ‘AI scientist’ can write science papers without any human input. Here’s why that’s a problem

  • Written by: Karin Verspoor, Dean, School of Computing Technologies, RMIT University, RMIT University
A new ‘AI scientist’ can write science papers without any human input. Here’s why that’s a problem

Scientific discovery is one of the most sophisticated human activities. First, scientists must understand the existing knowledge and identify a significant gap. Next, they must formulate a research question and design and conduct an experiment in pursuit of an answer. Then, they must analyse and interpret the results of the experiment, which may raise yet another research question.

Can a process this complex be automated? Last week, Sakana AI Labs announced the creation of an “AI scientist” – an artificial intelligence system they claim can make scientific discoveries in the area of machine learning in a fully automated way.

Using generative large language models (LLMs) like those behind ChatGPT and other AI chatbots, the system can brainstorm, select a promising idea, code new algorithms, plot results, and write a paper summarising the experiment and its findings, complete with references. Sakana claims the AI tool can undertake the complete lifecycle of a scientific experiment at a cost of just US$15 per paper – less than the cost of a scientist’s lunch.

These are some big claims. Do they stack up? And even if they do, would an army of AI scientists churning out research papers with inhuman speed really be good news for science?

How a computer can ‘do science’

A lot of science is done in the open, and almost all scientific knowledge has been written down somewhere (or we wouldn’t have a way to “know” it). Millions of scientific papers are freely available online in repositories such as arXiv and PubMed.

LLMs trained with this data capture the language of science and its patterns. It is therefore perhaps not at all surprising that a generative LLM can produce something that looks like a good scientific paper – it has ingested many examples that it can copy.

What is less clear is whether an AI system can produce an interesting scientific paper. Crucially, good science requires novelty.

But is it interesting?

Scientists don’t want to be told about things that are already known. Rather, they want to learn new things, especially new things that are significantly different from what is already known. This requires judgement about the scope and value of a contribution.

The Sakana system tries to address interestingness in two ways. First, it “scores” new paper ideas for similarity to existing research (indexed in the Semantic Scholar repository). Anything too similar is discarded.

Second, Sakana’s system introduces a “peer review” step – using another LLM to judge the quality and novelty of the generated paper. Here again, there are plenty of examples of peer review online on sites such as openreview.net that can guide how to critique a paper. LLMs have ingested these, too.

AI may be a poor judge of AI output

Feedback is mixed on Sakana AI’s output. Some have described it as producing “endless scientific slop”.

Even the system’s own review of its outputs judges the papers weak at best. This is likely to improve as the technology evolves, but the question of whether automated scientific papers are valuable remains.

The ability of LLMs to judge the quality of research is also an open question. My own work (soon to be published in Research Synthesis Methods) shows LLMs are not great at judging the risk of bias in medical research studies, though this too may improve over time.

Sakana’s system automates discoveries in computational research, which is much easier than in other types of science that require physical experiments. Sakana’s experiments are done with code, which is also structured text that LLMs can be trained to generate.

AI tools to support scientists, not replace them

AI researchers have been developing systems to support science for decades. Given the huge volumes of published research, even finding publications relevant to a specific scientific question can be challenging.

Specialised search tools make use of AI to help scientists find and synthesise existing work. These include the above-mentioned Semantic Scholar, but also newer systems such as Elicit, Research Rabbit, scite and Consensus.

Text mining tools such as PubTator dig deeper into papers to identify key points of focus, such as specific genetic mutations and diseases, and their established relationships. This is especially useful for curating and organising scientific information.

Machine learning has also been used to support the synthesis and analysis of medical evidence, in tools such as Robot Reviewer. Summaries that compare and contrast claims in papers from Scholarcy help to perform literature reviews.

All these tools aim to help scientists do their jobs more effectively, not to replace them.

AI research may exacerbate existing problems

While Sakana AI states it doesn’t see the role of human scientists diminishing, the company’s vision of “a fully AI-driven scientific ecosystem” would have major implications for science.

One concern is that, if AI-generated papers flood the scientific literature, future AI systems may be trained on AI output and undergo model collapse. This means they may become increasingly ineffectual at innovating.

However, the implications for science go well beyond impacts on AI science systems themselves.

There are already bad actors in science, including “paper mills” churning out fake papers. This problem will only get worse when a scientific paper can be produced with US$15 and a vague initial prompt.

The need to check for errors in a mountain of automatically generated research could rapidly overwhelm the capacity of actual scientists. The peer review system is arguably already broken, and dumping more research of questionable quality into the system won’t fix it.

Science is fundamentally based on trust. Scientists emphasise the integrity of the scientific process so we can be confident our understanding of the world (and now, the world’s machines) is valid and improving.

A scientific ecosystem where AI systems are key players raises fundamental questions about the meaning and value of this process, and what level of trust we should have in AI scientists. Is this the kind of scientific ecosystem we want?

Authors: Karin Verspoor, Dean, School of Computing Technologies, RMIT University, RMIT University

Read more https://theconversation.com/a-new-ai-scientist-can-write-science-papers-without-any-human-input-heres-why-thats-a-problem-237029

The Fire Safety Checklist Every Australian Short-Stay Host Should Run

Picture an Airbnb guest arriving late, tired, in a place they’ve never been. They don’t know where the exits are, which door leads to the garage, or h...

Better Business Photos Without Misleading Customers: An Australian Guide

A timber dining set looks inviting against a sunny deck, with cushions, lanterns and enough chairs for the whole family. But imagine the retailer sell...

Understanding a dental implant Melbourne Treatment Plan

A dental implant Melbourne patients may consider is one option for replacing a missing tooth, but suitability depends on oral health, bone support a...

How 3PL Logistics Australia Supports Faster and More Efficient Supply Chains

Why Has 3PL Logistics Become Important for Growing Businesses?  Gone are the days when supply chains were simpler and in minimal quantity. Now, the...

Common Skin Concerns and Treatment Options Available in Melbourne

Skin concerns can develop gradually or become noticeable following changes in age, lifestyle, sun exposure or hormonal factors. Pigmentation, rednes...

Why a Code-Compliant Home Can Still Feel Uncomfortable

When people build or renovate a home, they usually expect that meeting current building requirements will also produce a comfortable place to live. ...

Breach of Contract in Business: What Are Your Options?

Contracts are essential to business relationships. They establish expectations, allocate responsibilities and provide a framework for delivering goods...

How Pest Control Companies Can Rank for the Problems Customers Search First

Introduction SEO for Pest Control needs to start with the customer’s problem. Most people do not search for pest services in a neat or technical wa...

What Are the Benefits of Renovating an Older Home in Sydney?

Sydney has plenty of character homes, from Federation cottages to mid-century brick homes. Renovating rather than rebuilding can preserve what makes t...

Pool and Deck Design: How to Plan the Perfect Outdoor Living Space for Your Sydney Home

For many Australians, the backyard is where life happens. Summer barbecues, weekend swims and long evenings outdoors are all part of the lifestyle, ...

Is Solar Pool Heating Worth It? What Sydney Homeowners Should Know

There's nothing quite like a backyard pool on a hot Sydney day. But once autumn rolls in, many pools sit unused for months because the water is simp...

Planning a Luxury House Move: A Week-by-Week Timeline for Prestige Sydney Homes

Selling or buying a prestige home is a major milestone. Whether it's a waterfront residence in Birchgrove, a grand Federation home in Haberfield or ...

Downsizing or Upgrading Your Caravan? Here's How to Sell It Without the Hassle

Selling a caravan can feel like a major task, especially when you are unsure about its value, paperwork, or how to find a buyer. Whether you are dow...

The Best Overseas Adventure Holidays for Australians Who Love the Outdoors

Australia offers no shortage of incredible outdoor experiences, but sometimes the best way to satisfy your sense of adventure is to head overseas. A...

Cape Town Wine Shuttle: Winelands Tasting & Tours

Embark on an unforgettable journey through the picturesque Cape Winelands, where world-class wines and breathtaking scenery await. Our Cape Town Win...

Metal Fabrication: Choosing Metal Fabrication Melbourne Services for Custom Projects

What Modern Metal Fabrication Involves From individual components to complete structures, metal fabrication brings together processes such as desig...

Why Giant Rats Tail Grass Keeps Coming Back After Spraying

Giant Rats Tail Grass (GRT) is one of the most frustrating pasture weeds for farmers and lifestyle property owners. You spray an infested area, see th...

When Custom Cardboard Boxes Make Sense for Your Business

Custom cardboard boxes can be useful when a standard carton does not fit a product, packing method or presentation requirement particularly well. A ...