The intuition for why we current speech models can't produce good unconditional samples (see wavenet) is simply mindblowing. Phonemes occupy a small number of bits as compared with the overall signal (~10/s as compared with 16 k/s)!
In 12:07, are cognitive states low dimensional if that is the case are they sparse? If they are both sparse and low dimensional it contradicts with what he said in his MSS talk in 2012, where he states high dimensional and sparse is better than low dimensional
He's allowed to change his opinion and improve his theories over time. That's (interestingly enough) the kind of stuff that general intelligence allows
In 1:09:33, there was a question on gradual change in hypothesis space from very few samples - theory revision. I feel like neural nets may be quite ill-suited for fast change of learnt knowledge as the weights take a long time to change by backpropagation. What I believe is necessary, will be to imbue some form of learnable external memory bank on which we draw our knowledge from (in addition to neural nets), so we can just change that knowledge bank and learn new concepts instantly.
In Lexes Podcast just the mention of the example of speech as becoming unrecognicable gibberish (due to the amount of data) but when you seperate the gibberish to get a baisc feel for intonation or sound and speach as vocilisation of cerain tones that humans think of as speach you get a functional way to work it out would have totally suffieced to get the thing
You can turn artificial neural networks inside-out by using fixed dot products (weighted sums) and adjustable (parametric) activation functions. The fixed dot products can be computed very quickly using fast transforms like the FFT. Also the number of overall parameters required is vastly reduced. The dot products of the transform act as statistical summary measures. Ensuring good behavour. See Fast Transform (fixed filter bank) neural networks. The variance equation for linear combinations of random variables is very useful for understanding dot products in neural networks especially in conjunction with cosine angle. Also ReLU is a switch. The electricty in your house is a sine wave. Turn on a switch and the output is f(x)=x. Again the same sine wave as the input. Off(x)=0. A ReLU neural network then is a switched composition of dot products. If the switch states are known then there is a linear mapping between the input vector and the output vector which you can check out with various metrics.
Sampling rate * bit depth is a big overestimate of the amount of information in speech audio signals - look at the compression ratios that audio codecs can achieve
Sounds right to me. But why do they assume that the traditional neural net and deep learning are the best or only possible fundamental structures and processes for a system with these capabilities of disentangled abstractions working together with granular representations?
Good point. I'd like to hear something in that sense. I fell the current popularization of NN and DL has lead a lot of people to not consider any other alternatives and then losing other ways to solve problems and useful insights.
For example, when I heard the term disentangling I cannot stop thinking on this as a fancy (and potentially more sophisticated) way to refer to blind source separation.
Yes, but the benefits of dimensionality reduction far outweigh that. You don't need to consider every pixel of a picture to reason about the objects contained within that picture and their features.
No, not if the information content is low dimensional to begin with. Consider a circle of radius r is rendered at location (x, y) on a bit map. The information in pixel space is high dimensional - the number of pixels in the bit map. But the same circle can be transformed into a 3-dimensional parameter space representation - (x, y , r) with no loss of information. The same circle in pixel space can always be regenerated using the parameter space representation.
46:00 re: attention as gating the conscious and unconscious thoughts - can you imagine a machine which can widen and narrow its aperture of attention to accomplish different tasks?
@@tomm7273 If you mean Deep Learning I'd say: yeah the direction seems ok. But the way computers work, and any representation they use is, quantitive and they are absolutely precise with those numbers. While humans think in qualitative terms, like "this rough concept is very similar to that one". Indeed they can't quantify things precisely, but that is what makes humans more capable to deal with all this ambiguous complexity.
@@MartinLichtblau Why do you think fuzziness has to contradict precise numbers? I'm not arguing it doesn't, I'm just asking because we can fuzzify and work with vagueness to eliminate as much of the search space as possible.
@@ahilanpalarajah3159 It's complicated, but in it's basic sense it doesn't. I just couldn't find simple terms to tell them apart. Perhpas better say Accurate vs. Approximate or rigid vs. flexible...
The intuition for why we current speech models can't produce good unconditional samples (see wavenet) is simply mindblowing. Phonemes occupy a small number of bits as compared with the overall signal (~10/s as compared with 16 k/s)!
I liked that point very much.
However, it is also quite obvious: we don't analyse the signal with our brain, we analyse the sentenses and the meaning.
Interpolating in abstract space, exactly what Stable Diffusion is doing. This idea is really impactful.
Wonderful video. You can't help but admire his approach for what is AI, and the way he manages to convey these concepts. Brilliant!
In 12:07, are cognitive states low dimensional if that is the case are they sparse? If they are both sparse and low dimensional it contradicts with what he said in his MSS talk in 2012, where he states high dimensional and sparse is better than low dimensional
He's allowed to change his opinion and improve his theories over time. That's (interestingly enough) the kind of stuff that general intelligence allows
In 1:09:33, there was a question on gradual change in hypothesis space from very few samples - theory revision. I feel like neural nets may be quite ill-suited for fast change of learnt knowledge as the weights take a long time to change by backpropagation. What I believe is necessary, will be to imbue some form of learnable external memory bank on which we draw our knowledge from (in addition to neural nets), so we can just change that knowledge bank and learn new concepts instantly.
Who is the gentleman at 1:09:35 asking a question, and bringing up gradual learning?
Its Patrice Simard from Microsoft Research
In Lexes Podcast just the mention of the example of speech as becoming unrecognicable gibberish (due to the amount of data) but when you seperate the gibberish to get a baisc feel for intonation or sound and speach as vocilisation of cerain tones that humans think of as speach you get a functional way to work it out would have totally suffieced to get the thing
Anyone has a link to the slides? And come on camera people, it's not a beauty pageant, it's ok if you show slides instead of the speaker's face :)
@@ProfessionalTycoons 404 not found!!! Can you share a revised link? Thanks in advance!!!
medium.com/@SeoJaeDuk/archived-post-from-deep-learning-of-disentangled-representations-to-higher-level-cognition-b848fdc0de2c
Thanks for the interesting talk! Please post the slides as well!
51:00 I like the idea of a two-level system but disagree with the mutual information criterion.
Interesting. Could you say a bit more on this?
You can turn artificial neural networks inside-out by using fixed dot products (weighted sums) and adjustable (parametric) activation functions. The fixed dot products can be computed very quickly using fast transforms like the FFT. Also the number of overall parameters required is vastly reduced. The dot products of the transform act as statistical summary measures. Ensuring good behavour. See Fast Transform (fixed filter bank) neural networks.
The variance equation for linear combinations of random variables is very useful for understanding dot products in neural networks especially in conjunction with cosine angle.
Also ReLU is a switch. The electricty in your house is a sine wave. Turn on a switch and the output is f(x)=x. Again the same sine wave as the input. Off(x)=0. A ReLU neural network then is a switched composition of dot products. If the switch states are known then there is a linear mapping between the input vector and the output vector which you can check out with various metrics.
Sampling rate * bit depth is a big overestimate of the amount of information in speech audio signals - look at the compression ratios that audio codecs can achieve
Deepening of learning into a higher cognitive level:
Very good.
What and where are the works, who is working on this approach?
Sounds right to me. But why do they assume that the traditional neural net and deep learning are the best or only possible fundamental structures and processes for a system with these capabilities of disentangled abstractions working together with granular representations?
Good point. I'd like to hear something in that sense. I fell the current popularization of NN and DL has lead a lot of people to not consider any other alternatives and then losing other ways to solve problems and useful insights.
For example, when I heard the term disentangling I cannot stop thinking on this as a fancy (and potentially more sophisticated) way to refer to blind source separation.
Doesn't translation into an abstract space necessitate a loss of information?
Yes, but the benefits of dimensionality reduction far outweigh that. You don't need to consider every pixel of a picture to reason about the objects contained within that picture and their features.
There is also information stored in the decoder.
No, not if the information content is low dimensional to begin with. Consider a circle of radius r is rendered at location (x, y) on a bit map. The information in pixel space is high dimensional - the number of pixels in the bit map. But the same circle can be transformed into a 3-dimensional parameter space representation - (x, y , r) with no loss of information. The same circle in pixel space can always be regenerated using the parameter space representation.
@@nauy Thanks. I've been learning about matrix transformations and PCA lately. It took me a few years to get here.
The camera work negatively affects a wonderful lecture
46:00 re: attention as gating the conscious and unconscious thoughts - can you imagine a machine which can widen and narrow its aperture of attention to accomplish different tasks?
amazing talk.
Humans use fuzzy approaches, while computers use precise numbers. Which one can work in this complex world?
Computers can use fuzzy approaches as well. Almost all modern machine learning techniques are fuzzy.
@@tomm7273 If you mean Deep Learning I'd say: yeah the direction seems ok. But the way computers work, and any representation they use is, quantitive and they are absolutely precise with those numbers. While humans think in qualitative terms, like "this rough concept is very similar to that one". Indeed they can't quantify things precisely, but that is what makes humans more capable to deal with all this ambiguous complexity.
@@MartinLichtblau Why do you think fuzziness has to contradict precise numbers? I'm not arguing it doesn't, I'm just asking because we can fuzzify and work with vagueness to eliminate as much of the search space as possible.
@@ahilanpalarajah3159 It's complicated, but in it's basic sense it doesn't. I just couldn't find simple terms to tell them apart. Perhpas better say Accurate vs. Approximate or rigid vs. flexible...
Humans have emotion , humans care .... robots will never care
Someone should write a detailed blog explaining stuff in this
I want to talk with the guy talking about barycentres and wasserstein distance!
around 1:06:00
What's the neurips optimal transport tutorial mentioned?
Wow the subtitles are terrible :( ... GAN -> gown, k means -> keys lol
He is genius
Someone should write a detailed blog explaining stuff in this