In ai-land, technologies move from remarkable to old hat at the speed of light. Only 18 months ago the release of Chatgpt, Openai’s chatbot, launched an ai frenzy. Today its powers have become commonplace. Several firms (such as Anthropic, Google and Meta) have since unveiled versions of their own models (Claude, Gemini and Llama), improving upon Chatgpt in a variety of ways.
That hunger for the new has only accelerated. In March Anthropic launched Claude 3, which bested the previous top models from Openai and Google on various leaderboards. On April 9th Openai took back the crown (on some measures) by making tweaks to its model. In the coming weeks Meta is expected to release Llama 3. Openai is sure to make a splash at some point thereafter when it releases gpt-5, which may have capabilities beyond any current large language model (llm). If the rumours are to be believed, the next generation of models will be even more remarkable—able to perform multi-step tasks, for instance, rather than merely responding to prompts, or analysing complex questions carefully instead of blurting out the first algorithmically available answer.
For those who believe that this is the usual tech hype, consider this: investors are deadly serious about backing the next generation of models. gpt-5 and other next-gen models are expected to cost billions of dollars to train. Openai is also reportedly partnering with Microsoft, a tech giant, to build a new $100bn data centre. Based on the numbers alone, it seems as though the future will hold limitless exponential growth. This chimes with a view shared by many ai researchers called the “scaling hypothesis”, namely that the architecture of current llms is on the path to unlocking phenomenal progress. All that is needed to exceed human abilities, according to the hypothesis, is more data and more powerful computer chips. Look closer at the technical frontier, however, and some daunting hurdles become evident.
Beauty’s not enough
Data may well present the most immediate bottleneck. Epoch ai, a research outfit, estimates the well of high-quality textual data on the public internet will run dry by 2026. This has left researchers scrambling for ideas. Some labs are turning to the private web, buying data from brokers and news websites. Others are turning to the internet’s vast quantities of audio and visual data, which could be used to train ever-bigger models for decades. Video can be particularly useful in teaching ai models about the physics of the world around them. If a model can observe a ball flying through the air, it might more easily work out the mathematical equation that describes the projectile’s motion. Leading models like gpt-4 and Gemini are now “multimodal”, capable of dealing with various types of data.
When data can no longer be found, it can be made. Companies like Scale ai and Surge ai have built large networks of people to generate and annotate data, including phd researchers solving problems in maths or biology. One executive at a leading ai startup estimates this is costing ai labs hundreds of millions of dollars per year. A cheaper approach involves generating “synthetic data” in which one llm makes billions of pages of text to train a second model. Though that method can run into trouble: models trained like this can lose past knowledge and generate uncreative responses. A more fruitful way to train ai models on synthetic data is to have them learn through collaboration or competition. Researchers call this “self-play”. In 2017 Google DeepMind, the search giant’s ai lab, developed a model called AlphaGo that, after training against itself, beat the human world champion in the game of Go. Google and other firms now use similar techniques on their llms.
Extending ideas like self-play to new domains is hot topic of research. But most real-world problems—from running a business to being a good doctor—are more complex than a game, without clear-cut winning moves. This is why, for such complex domains, data to train models is still needed from people who can differentiate between good and bad quality responses. This in turn slows things down.
More silicon, but make it fashion
Better hardware is another route to more powerful models. Graphics-processing units (gpus), originally designed for video-gaming, have become the go-to chip for most ai programmers thanks to their ability to run intensive calculations in parallel. One way to unlock new capabilities may lie in using chips designed specifically for ai models. Cerebras, a chipmaker based in Silicon Valley, released a product in March containing 50 times as many transistors as the largest gpu. Model-building is usually hampered by data needing to be continuously loaded on and off the gpus as the model is trained. Cerebras’s giant chip, by contrast, has memory built in.
New models that can take advantage of these advances will be more reliable and better at handling tricky requests from users. One way this may happen is through larger “context windows”, the amount of text, image or video that a user can feed into a model when making requests. Enlarging context windows to allow users to upload additional relevant information also seems to be an effective way of curbing hallucination, the tendency of ai models to confidently answer questions with made-up information.
But while some model-makers race for more resources, others see signs that the scaling hypothesis is running into trouble. Physical constraints—insufficient memory, say, or rising energy costs—place practical limitations on bigger model designs. More worrying, it is not clear that expanding context windows will be enough for continued progress. Yann LeCun, a star ai boffin now at Meta, is one of many who believe the limitations in the current ai models cannot be fixed with more of the same.
Some scientists are therefore turning to a long-standing source of inspiration in the field of ai—the human brain. The average adult can reason and plan far better than the best llms, despite using less power and much less data. “ai needs better learning algorithms, and we know they’re possible because your brain has them,” says Pedro Domingos, a computer scientist at the University of Washington.
One problem, he says, is the algorithm by which llms learn, called backpropagation. All llms are neural networks arranged in layers, which receive inputs and transform them to predict outputs. When the llm is in its learning phase, it compares its predictions against the version of reality available in its training data. If these diverge, the algorithm makes small tweaks to each layer of the network to improve future predictions. That makes it computationally intensive and incremental.
The neural networks in today’s llms are also inefficiently structured. Since 2017 most ai models have used a type of neural-network architecture known as a transformer (the “t” in gpt), which allowed them to establish relationships between bits of data that are far apart within a data set. Previous approaches struggled to make such long-range connections. If a transformer-based model were asked to write the lyrics to a song, for example, it could, in its coda, riff on lines from many verses earlier, whereas a more primitive model would have forgotten all about the start by the time it had got to the end of the song. Transformers can also be run on many processors at once, significantly reducing the time it takes to train them.
Albert Gu, a computer scientist at Carnegie Mellon University, nevertheless thinks the transformers’ time may soon be up. Scaling up their context windows is highly computationally inefficient: as the input doubles, the amount of computation required to process it quadruples. Alongside Tri Dao of Princeton University, Dr Gu has come up with an alternative architecture called Mamba. If, by analogy, a transformer reads all of a book’s pages at once, Mamba reads them sequentially, updating its worldview as it progresses. This is not only more efficient, but also more closely approximates the way human comprehension works.
llms also need help getting better at reasoning and planning. Andrej Karpathy, a researcher formerly at Openai, explained in a recent talk that current llms are only capable of “system 1” thinking. In humans, this is the automatic mode of thought involved in snap decisions. In contrast, “system 2” thinking is slower, more conscious and involves iteration. For ai systems, that may require algorithms capable of something called search—an ability to outline and examine many different courses of action before selecting the best one. This would be similar in spirit to how game-playing ai models can choose the best moves after exploring several options.
Advanced planning via search is the focus of much current effort. Meta’s Dr LeCun, for example, is trying to program the ability to reason and make predictions directly into an ai system. In 2022 he proposed a framework called “Joint Embedding Predictive Architecture” (jepa), which is trained to predict larger chunks of text or images in a single step than current generative-ai models. That lets it focus on global features of a data set. When analysing animal images, for example, a jepa-based model may more quickly focus on size, shape and colour rather than individual patches of fur. The hope is that by abstracting things out jepa learns more efficiently than generative models, which get distracted by irrelevant details.
Experiments with approaches like Mamba or jepa remain the exception. Until data and computing power become insurmountable hurdles, transformer-based models will stay in favour. But as engineers push them into ever more complex applications, human expertise will remain essential in the labelling of data. This could mean slower progress than before. For a new generation of ai models to stun the world as Chatgpt did in 2022, fundamental breakthroughs may be needed.
