Is this the future of AI? A new decentralized model is unveiled

Intellect-1
(Image credit: INTELLECT-1)

The AI company PrimeIntellect recently started training a new 10 billion parameter model, a task it hopes to complete using the help of users around the world.

On its blog, PrimeIntellect said its new model, INTELLECT-1, will be the product of the first decentralized training run of a model of this scale. However, this still falls significantly short of even four-year-old models like OpenAI’s GPT-3 which featured over 175 billion parameters.

This project started out with research on the open-source implementation and scaling of globally distributed AI model training. The method worked for a model 1 billion parameters large and now the next step is to scale it up by a factor of ten.

Size isn't everything though. Newer models like Microsoft's Phi and Meta's Llama are proving you can achieve GPT-3 and even GPT-4 level performance with a fraction of the parameters through efficiency improvements.

This brings us one step closer towards open source AGI

PrimeIntellect

The company’s goal is to find a way to make decentralized training a reality to ensure that the next generation of AI, artificial general intelligence (AGI), is open-source, transparent, and accessible. This reduces the risk of only a few large companies having access to this advanced technology.

For now, users can only contribute to the project through the company’s own platform. You can do this by renting GPUs that PrimeIntellect selected, specifically NVIDIA’s H100 Tensor Core GPU, which cost around $20 per hour to run. But in the future, you should be able to contribute to the model’s training with your own hardware.

The training is made possible through separate clusters of devices that process data to train the AI model. However, new features allow the different clusters to communicate less frequently with each other to synchronize their progress, thus freeing up bandwidth requirements. The training framework can also handle nodes joining or leaving without leading to system crashes.

Nodes that join training that has already started would need to be brought up to speed with the latest state of the model before being able to contribute. Delays with this catching-up process have been solved by having new nodes request checkpoints from their peers.

What happens next for INTELLECT-1

INTELLECT-1 is based on the Llama-3 architecture and is being trained on four different datasets. It’s mainly training on a Hugging Face dataset called FineWeb-Edu which contains content from educational web pages.

In the future, PrimeIntellect wants to train even larger models and create ways for anyone to create their own similar AI model training project to which other users can also contribute their processing power.

More from Tom's Guide

Category
Arrow
Arrow
Back to MacBook Air
Brand
Arrow
Processor
Arrow
RAM
Arrow
Storage Size
Arrow
Screen Size
Arrow
Colour
Arrow
Storage Type
Arrow
Condition
Arrow
Price
Arrow
Any Price
Showing 10 of 63 deals
Filters
Arrow
Load more deals
Christoph Schwaiger

Christoph Schwaiger is a journalist who mainly covers technology, science, and current affairs. His stories have appeared in Tom's Guide, New Scientist, Live Science, and other established publications. Always up for joining a good discussion, Christoph enjoys speaking at events or to other journalists and has appeared on LBC and Times Radio among other outlets. He believes in giving back to the community and has served on different consultative councils. He was also a National President for Junior Chamber International (JCI), a global organization founded in the USA. You can follow him on Twitter @cschwaigermt.