AI in 2024: A year of plateau and new directions

For years, the AI industry has followed the so-called "scaling laws" which suggest that increasing the size of models (in terms of computing power, data, and parameters) will lead to more powerful AI systems.

Let’s make it simpler. Imagine you're baking a cake. You know that the bigger the cake, the more ingredients and oven time you'll need. AI scaling laws are similar, but instead of a cake, we're talking about AI models.

These laws tell us that bigger AI models, trained on more data, generally perform better. It's like saying that a larger cake, baked longer, will taste better.

OpenAI co-founder Ilya Sutskever, who now runs his own AI lab, Safe Superintelligence (SSI), claims that the ChatGPT-maker’s recent tests trying to scale up its models suggest that those efforts have plateaued.

"The 2010s were the age of scaling, now we're back in the age of wonder and discovery once again," Sutskever, who quit OpenAI in May last year, told Reuters in an interview. "Everyone is looking for the next thing."

OpenAI’s Orion model, for instance, reportedly faced issues in performing coding tasks due to insufficient training data in the domain. While improvements have been made through a post-training process, Orion is still not ready for public release, with OpenAI reportedly set to delay its rollout until early next year.

According to a report in The Information, although OpenAI has completed only 20 percent of Orion’s training, it is already on par with GPT-4 in intelligence, task fulfillment, and question-answering abilities. While Orion outperforms previous models, the quality improvement is less dramatic than the leap from GPT-3 to GPT-4.

Similarly, Anthropic’s Claude 3.5 Opus has not significantly outperformed earlier models, despite being larger and more resource-intensive.

The data dilemma

A key issue with training AI models is the availability of high-quality, human-made data. While AI companies have relied on scraping publicly available data from the internet, this approach is starting to hit limits. To train models that can handle complex, specialised tasks, companies need access to high-quality datasets, which are harder and more expensive to acquire.

In response, companies like OpenAI have started to form partnerships with publishers and other organisations to get more targeted and diverse datasets. This approach is time-consuming and costly.

Moreover, there's the growing problem of synthetic data. While computer-generated content (like text or images) is useful, it doesn’t always have the richness or unpredictability of human-created material.

The cost

With the rising cost of AI research and development, the stakes are getting higher.

To develop cutting-edge models like GPT-4 and Google’s Gemini, companies have to spend tens or even hundreds of millions of dollars. And as the models grow larger, so too do the costs.

The cost of training a cutting-edge AI model today is estimated to be around $100 million, with that figure expected to balloon into the billions in the future.

Tanuj Bhojwani, head of People + AI argues that the real challenge is not about scaling these models. "There's no company which wins a billion dollars for having the biggest models. Companies win when their models are being used in real-life applications and they're getting paid, and they continue to have money and free cash flow to invest, and keep pushing the technology boundary because customers will always be demanding of the best."

Bhojwani believes that once companies realise that scaling alone isn't a sustainable strategy, they can leverage their existing resources to concentrate on practical use cases.

"So it's just a question of when do you give up on that scaling ambition which is FOMO-driven. If OpenAI is investing $10 billion, then Anthropic wants to invest that much, and somebody else wants to invest similar amounts," Bhojwani told Moneycontrol.

2025 and beyond

While traditional scaling laws focus on pre-training larger models, companies like OpenAI are adopting new ways to scale models, by focusing on training time and inference time.

Training time refers to the amount of time it takes to train a model on a dataset. Inference time refers to the amount of time it takes for a model to process a query and generate a response.

OpenAI has also been shifting its focus away from simply building larger models to creating new use cases for existing models, such as developing AI “agents”.

"We will have better and better models, but I think the thing that will feel like the next giant breakthrough will be agents," Altman said in a Reddit post.

Looking ahead, the AI industry may not see the same explosive breakthroughs as it did in 2023 and 2024. Google CEO Sundar Pichai has suggested that the low-hanging fruit in AI research is now gone, and future progress will require more profound breakthroughs.

While the dream of AGI may still be distant, the journey forward promises to be full of innovation—just in ways that may be less flashy, but more meaningful.

As 2024 comes to a close, one thing is clear: the race for AI dominance is far from over.