OpenAI models may be starting to plateau
Word from inside OpenAI is that the company’s next-gen “Orion” model may not yield the performance leap that was expected.
A new report from The Information reveals that the company may be shifting strategies as its up-and-to-the-right AI performance curve starts to flatten out.
OpenAI and the other big AI companies like Meta, Google, and Anthropic have generally been following the same pattern to achieve more and more capable AI models: gather as much training data as possible (words, images, videos) and feed it into increasingly huge supercomputers powered by expensive GPUs, such as Nvidia’s popular H100 chip. This demand for larger training clusters has helped push Nvidia to become the most valuable company in the US.
But fresh, unseen training data is becoming harder and harder to find. This is one of the drivers of OpenAI brokering so many media-licensing deals. Companies are now starting to train models on “synthetic data” generated by current LLMs, like OpenAI’s GPT-4o. But researchers have been warning that this can lead to “model collapse” when a model starts eating its own data.
OpenAI may be leaning more on the new “reasoning” capabilities like the ones found in its latest OpenAI o1 model to help eke out performance gains. But this requires more computing, however, and more energy.
OpenAI and the other big AI companies like Meta, Google, and Anthropic have generally been following the same pattern to achieve more and more capable AI models: gather as much training data as possible (words, images, videos) and feed it into increasingly huge supercomputers powered by expensive GPUs, such as Nvidia’s popular H100 chip. This demand for larger training clusters has helped push Nvidia to become the most valuable company in the US.
But fresh, unseen training data is becoming harder and harder to find. This is one of the drivers of OpenAI brokering so many media-licensing deals. Companies are now starting to train models on “synthetic data” generated by current LLMs, like OpenAI’s GPT-4o. But researchers have been warning that this can lead to “model collapse” when a model starts eating its own data.
OpenAI may be leaning more on the new “reasoning” capabilities like the ones found in its latest OpenAI o1 model to help eke out performance gains. But this requires more computing, however, and more energy.
