There are so many new pathways and possibilities and these large companies, who keep their research under lock and key, present the perception of oversimplified major training methods. But if you look at the amount of global research that surfaces daily, you know that the possibilities are endless, even without indefinite scaling.
Ilia's predictions on pre-training’s limits are game-changing. Shifting to synthetic data and reasoning models could redefine AI’s future and unlock a new era of intelligent systems. Exciting times ahead!
I really expected that it was perhaps a 10x higher parameter count, say,10-100 trillion, which would achieve the next level of intelligence regardless of a potential 'peak data.' Sounds like Ilya and others are indicating that this hasn't panned out. Bummer, if so.
What!? Peak data? Exceptional human data maybe? Private curated data will be the answer moving forward, probably, as the internet is now poisoned with AI slop. The internet will remain valuable for capturing snapshots of culture. Anyways - Isn't the end game to be training models on, like, reality itself? Every modality that can be measured, human perceptible or not? :shrug:
Missing Data: Perhaps the data of a plumber.... the visual cues. Telsa FSD, has the data/visual cues of a gazzillion hours of actual driving... My point is, physical tasks, do they require real world data or can they be trained soley within the new robot training software from Google and or Nvidia? Robots will not be completely agentic until they know all knowledge. So yes, some knowledge has not really been recorded. Has the visual cues of a professional basketball player been recorded (or set up in software).. yet? - just rambling
It would seem that hominid evolution certainly found data acquisition coming from world modeling. There is so much more data in an actual blade of grass.
There are so many new pathways and possibilities and these large companies, who keep their research under lock and key, present the perception of oversimplified major training methods. But if you look at the amount of global research that surfaces daily, you know that the possibilities are endless, even without indefinite scaling.
Ilia’s talk at NeurIPS signals a new AI era beyond pre-training. Scaling up was just the start! Time for agents and synthetic data to shine .
Ilia's predictions on pre-training’s limits are game-changing. Shifting to synthetic data and reasoning models could redefine AI’s future and unlock a new era of intelligent systems. Exciting times ahead!
Ilia's view on hitting peak data signals a shift in AI research. Exploring new methods like inference time compute could unlock exciting potentials!
Of course the way of improving the models is changing, but the performances are improving faster all the time
I really expected that it was perhaps a 10x higher parameter count, say,10-100 trillion, which would achieve the next level of intelligence regardless of a potential 'peak data.' Sounds like Ilya and others are indicating that this hasn't panned out. Bummer, if so.
Verses AI !!- active inference based on biological processes
In other words: c"opyright holders are on to us so we have to find a new way to train our models."
How come nobody's talking about Realis Worlds yet? Take a look at it please. I can't keep up with this technology. Is AI embodiment really here?
What!? Peak data? Exceptional human data maybe? Private curated data will be the answer moving forward, probably, as the internet is now poisoned with AI slop. The internet will remain valuable for capturing snapshots of culture. Anyways - Isn't the end game to be training models on, like, reality itself? Every modality that can be measured, human perceptible or not? :shrug:
Missing Data:
Perhaps the data of a plumber.... the visual cues.
Telsa FSD, has the data/visual cues of a gazzillion hours of actual driving...
My point is, physical tasks, do they require real world data or can they be trained soley within the new robot training software from Google and or Nvidia?
Robots will not be completely agentic until they know all knowledge. So yes, some knowledge has not really been recorded.
Has the visual cues of a professional basketball player been recorded (or set up in software).. yet?
- just rambling
It would seem that hominid evolution certainly found data acquisition coming from world modeling. There is so much more data in an actual blade of grass.
Chatbot 03
meh, all the badly trained data will wind up getting hand curated to make it better semantically.
Synthetic data.