Learning from almost no data - Dr Ilia Sucholutsky (Princeton)

Invited talk at the 5th International Convention on the Mathematics of Neuroscience and Artificial Intelligence, Rome, 2024 (neuromonster.org).
Recorded and hosted with generous funding from the Kavli Foundation, Gatsby Foundation, Templeton Foundation, Harvard University, European Research Council, Artificial Intelligence Journal, and Google DeepMind.
© Thinking About Thinking, Inc, a 501(c)3 Nonprofit registered in New Jersey, USA.
thinkingaboutt...

[@amortalbeing]

thanks this was very interesting

[@AryaKode]

He said Timber not timbre 😢, I jest, great presentation

[@jks234]

Just looked it up for anyone curious.
Timber is pronounced as you would expect. (Tim Burr) It means wood material.
Timbre is pronounced "tamburr". And it means quality of voice.
I learned from your comment.
I also learned quite late that melee is may-lay. Not mee-lee.

[@gabrielescheler2522]

Insanity squared

[@nonamenoname1942]

1:22 you know it absolutely wouldn't hurt to remind the public that the table on a right is a russian chemist Dmitri Mendeleev creation.

[@amortalbeing]

what table?

[@nonamenoname1942]

@@amortalbeing Periodic table of the elements

[@Adhil_parammel]

He named after ilya

[@dennisalbert6115]

Incorporate constructor theory

[@GerardSans]

While his thesis has some merits it fails when comparing human beings with AI. The most important error is that we don’t share a ground truth or from a semiotic perspective a referent. Whatever the representation is flawed if it’s pointing to something different or is not grounded in the same thing. AI is grounded on its training data not reality. Eg: science fiction, social media, Internet sources are all biased, subjective and not necessarily grounded in reality.

[@azi_and_razi]

But what is the source of those biases? Humankind. So are we really grounded in reality or maybe just in our flawed representation of reality? Maybe it's better for AI to be grounded on training data produced by us rather than in true reality? Are you sure AI will be aligned with us when grounded in real world?

[@jks234]

Sounds like you're explaining how the AI and humans do the same thing.
When I speak English with you and you can understand me, where is the ground truth of English?
Something quite akin to training data.
There is no reality of English. English has and will shift and change. The words we use are simply the words we hear others use and we adjust with them.
(No cap, foreal.)
We use training data to inform inference.
"Ground truth" is a sorting exercise. In a way, it can be thought of as "meta-training data".
Where we have evaluated the information out there and chosen the ones we deem acceptable.
We cannot get much better than that. No human automatically learns "ground truth".
We use other accepted measurements to define ground truth. And then, using these tools we deem objective, we collectively agree upon ground truth.
But I wasn't the one that learned that. I once again, learned it using the tribal collective of training data.

[@azi_and_razi]

@@jks234 So do you assume language is biased but somehow pople learn grand truth and in our brains the ground truth resides? People are not objective beings. We are all subjective and the best thing we can do is intersubjectivity. Even with science.

[@davidebic]

I don't know if you ever heard of the Platonic Representation Hypothesis. A little while back a paper came out showing how with model size increase, distances of same concepts in LLMs and Image Generation Networks converged to some quantity. Aka even though LLMs and Diffusion Models work on completely different data types (texts and images), they converge to the same inner representation of the world as they get better. Still an hypothesis, but there was some good statistical evidence that at least for now this is happening.