Tap to unmute

How are memories stored in neural networks? | The Hopfield Network

  • เผยแพร่เมื่อ 23 พ.ค. 2024
  • Can we measure memories in networks of neurons in bytes? Or should we think of our memory differently?
    Submission to the Summer of Math Exposition 2022 (#SoME2). More information: summerofmathexposition.substa...
    Time stamps:
    0:00 - Where is your memory?
    1:41 - Computer memory in a nutshell
    2:58 - Modeling neural networks
    4:42 - Memories in dynamical systems
    9:54 - Learning
    13:36 - Memory capacity and conclusion
    Animations largely made using the manim community edition:
    Original Paper on Hopfield Networks:
    Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8), 2554-2558.
    Neuron image by Santiago Ramón y Cajal, The pyramidal neuron of the cerebral cortex, 1904 Ink and pencil on paper, 8 5/8 x 6 7/8 in. Credit: Cajal Institute (CSIC), Madrid
    Music: Aakash Gandhi - "Dreamland"

ความคิดเห็น • 1K

  • @aegontargaryen597
    @aegontargaryen597 8 หลายเดือนก่อน +425

    As a neural network, I confirm that this is how we store memories

    •  7 หลายเดือนก่อน


    • @ugestacoolie5998
      @ugestacoolie5998 6 หลายเดือนก่อน +6


    • @claudiamanta1943
      @claudiamanta1943 4 หลายเดือนก่อน

      All the screws of those who are OBSESSED to think that people are nothing but cybernetic machines are rather loose…

    • @aegontargaryen597
      @aegontargaryen597 4 หลายเดือนก่อน

      @@claudiamanta1943 I haven't seen anyone thinking that people are cybernetic machines lol

    • @merozemory
      @merozemory 4 หลายเดือนก่อน


  • @murilospineli2241
    @murilospineli2241 ปีที่แล้ว +1106

    is this a first video? it's surprisingly good

    • @jasoneel76
      @jasoneel76 ปีที่แล้ว +34

      My god I need to subscribe immediately

    • @jaumeivars
      @jaumeivars ปีที่แล้ว +10

      Oh my god i didn't figures it out until you commented!

    • @ryanpmcguire
      @ryanpmcguire ปีที่แล้ว +31

      When something is made by someone who cares, you get a good result. Caring is the first ingredient in the recipe for success.

    • @fadonoso
      @fadonoso ปีที่แล้ว +2

      I thought the same!!! This guy is really good. Thanks!!

    • @NickSklias
      @NickSklias ปีที่แล้ว +23

      The first video uploaded to a channel does not mean the person/people editing/producing the video had no prior experience. That being said, it's a fantastic first upload and this channel is off to a tremendous start!

  • @luke2642
    @luke2642 ปีที่แล้ว +317

    You nailed the 3blue1brown feeling whilst being ligther on the maths. I wish you the best of luck with future videos! 🙂

    • @ianthehunter3532
      @ianthehunter3532 ปีที่แล้ว +10

      He likely uses Manim, the software made and used by 3b1b.

    • @poultrypants
      @poultrypants หลายเดือนก่อน +1

      Are you guys also watching the 3B1B videos about GPTs?

    • @luke2642
      @luke2642 23 วันที่ผ่านมา

      Aww, you never did a follow up!

  • @piggy9539
    @piggy9539 ปีที่แล้ว +784

    This guy explained such a complex topic so well, great job.

    • @alexanderhandjiev2845
      @alexanderhandjiev2845 ปีที่แล้ว +3

      It's going viral soooo ye

    • @mihailmilev9909
      @mihailmilev9909 ปีที่แล้ว

      @Yatish38 yeah and the subs for one video is crazy but deserved too

    • @RoseOnFire
      @RoseOnFire ปีที่แล้ว

      @Yatish38 Really good, yes, deserves more, also yes

    • @TheRainHarvester
      @TheRainHarvester ปีที่แล้ว +1

      Yeah i had trouble with the super/subscripts in all the math. So i just made a video easily explaining multilayer back prop with only 1 subscript. It's so easy now!

    • @pizzainc.1465
      @pizzainc.1465 3 หลายเดือนก่อน

      If humans know how it works, it’s a simple subject that is explained badly. If they don’t, it’s too complicated anyway. For example, we know nothing about how the brain works because it is simple too complicated. However, we know plenty about gravity, other areas of anatomy, computers, and more.

  • @Ana_Saudah
    @Ana_Saudah 4 หลายเดือนก่อน +9

    Dude made one video and then stops uploading.
    You have a talent.

  • @keithmiller3770
    @keithmiller3770 ปีที่แล้ว +116

    This is remarkable. How the heck does this channel only have 1 video?!

    • @rabiaedaylmaz1198
      @rabiaedaylmaz1198 ปีที่แล้ว +8

      Still waiting :( ...

    • @alexb3617
      @alexb3617 10 หลายเดือนก่อน +2

      yeah i knew this video would be good. i noticed it few months back when i was beginning to study neural networks, just stored it, knew wasn't ready yet. now finely watched, as expected, it turned out good. nothing like clickbat stuff that usually out there on youtube

    • @Mikee512
      @Mikee512 หลายเดือนก่อน

      Everyone must start somewhere :)

    • @dunar1005
      @dunar1005 16 วันที่ผ่านมา +1

      Because it was a submission for a convention, and not his goal to continue this

  • @Sazazezer
    @Sazazezer ปีที่แล้ว +306

    I've been idly musing about the brain's memory in terms of binary and bytes for years and always thought it was such a limited way of looking at the subject but didn't know any better. I asked around in the past about alternative ways of looking at the subject but never came away with a decent answer. This video has been such an eye opener and i think i'm going to have to take a deeper dive into the subject. Thanks!

    • @Daniel-ih4zh
      @Daniel-ih4zh ปีที่แล้ว +4

      You should haken's book on synergetics

    • @andrewhancock2451
      @andrewhancock2451 ปีที่แล้ว +2

      You mean, like, deep into neural nets? ;P

    • @SpringySpring04
      @SpringySpring04 ปีที่แล้ว

      I heard in an scifi anime (which is surprisingly more accurately scientific than it may seem despite being scifi) called Steins;Gate that said the brain can hold a little over 4 terabytes of memories, but that was just a claim, I'm not sure if that's true.

    • @pyropulseIXXI
      @pyropulseIXXI ปีที่แล้ว +3

      Memory doesn't just exist in the brain; there is evidence it is dispersed throughout the entire body. People have had large chunks of their brain removed and their memory was unaffected. People born with basically no brain, just a small layer around the skull and the 'cavity' where the brain should be was filled with cerebral-spinal fluid. And yet their IQ and memory was well above average.
      I know it is hard for purely materialistic to understand, but consciousness is more fundamental than physical matter, with the brain merely being an interface for the physical world
      A true ghost in the machine; consciousness does not arise from the machine being complex; it is controlling the biological machine

    • @goldnutter412
      @goldnutter412 ปีที่แล้ว

      @@pyropulseIXXI Exactly, I posted a heap of times about this already, this video really inspired.. brain caching, permanent information, rinse and repeat with new context.
      Optimization is often overlooked. It will be obvious sooner or later, there is no other way to efficiently battle entropy. We get sense data and need a nervous system and brain in general, but uncertainty is the ultimate optimization. Probabilistic by design.. part of why the speed of light is so high. We see the very very distant past, and are disconnected from it.. we get data and interpret it and wonder how the heck physical matter organized itself a certain way and why the math doesn't work
      0 to 1 is a logical impossibility.. the Big Bang is a misinterpretation, but a plausible back story for the universe and adds to the immersion. We process things bound by our brain capacity (see Einstein's freak brain!) and easily fall into materialist traps of our own design. Logically the universe and biology is simple, but obfuscated by so many LAYERS of background processes ! we call it subconscious.. yeah it's busy !

  • @portal128
    @portal128 23 วันที่ผ่านมา +7

    bro made a channel dedicated to neural networks, posted one video, gained 24k subs, decided this was enough and left

  • @gianlucazinarof8617
    @gianlucazinarof8617 ปีที่แล้ว +17

    You cannot fathom my disappointment when I looked at your channel and discovered that this is your only video. What do you mean there aren't any others? I need more! Seriously, interesting topic, crystal clear presentation, keep up the great work!

    • @Mangputt
      @Mangputt ปีที่แล้ว

      Do you know, what software able to produce and animate these animation? adobe after effects or any software else?

    • @layerwiselectures
      @layerwiselectures  ปีที่แล้ว +1

      It is a custom toolbox written in Python, called Manim. Here it is the community edition, but Manim was originally devised and open-sourced by 3blue1brown. There is a link in the description if you want to learn more.

    • @Mangputt
      @Mangputt ปีที่แล้ว

      @@layerwiselectures thankyou... I still waiting for your next videos. Keep it going Bro 🔥

  • @mmarrotte101
    @mmarrotte101 ปีที่แล้ว +62

    The amount of effort required to not only curate such complex information in an intuitive way, but also to animate the concepts in the way you've done, is clearly tremendous - thank you for taking the time to produce this! I'm inspired by your work, really looking forward to more!

  • @sebastianmuller4649
    @sebastianmuller4649 ปีที่แล้ว +2

    "Contrast this with what we believe about the brain..." I like how careful you are with how you make your statements.

  • @HansLemurson
    @HansLemurson ปีที่แล้ว +117

    Different "seed" states will call up different memories. I wonder if this could be analogous to how certain ideas link together, and maybe some of the underlying information of related ideas is actually a correlated section of their memory patterns.

    • @MrNatsuDragneel
      @MrNatsuDragneel ปีที่แล้ว

      This is wrong, our brain can excite and depress certain neurons with neurotransmitters, hormones, data from sensors of the environment and our own body etc...

    • @hufficag
      @hufficag ปีที่แล้ว +52

      This explains why I have an idea at my desk, get up to walk across the room and forget what I was thinking about. Get back to sitting at my desk seed state, and remember it, get up and forget it again.

    • @mihailmilev9909
      @mihailmilev9909 ปีที่แล้ว +4

      @@hufficag omg we figured it our also lmfao

    • @floriantesto4925
      @floriantesto4925 ปีที่แล้ว +1

      @@hufficag just take a look on en.wikipedia.org/wiki/Encoding_specificity_principle

    • @woolfel
      @woolfel ปีที่แล้ว +18

      the video is meant to explain how modern artificial neural networks work. biological brains don't work that way. Take for example a deer. Within an hour of being born, a deer can walk and within a few hours it can run. Clearly the baby deer didn't learn how to run in the womb. The evidence suggests some kinds of memory are genetic, while other skills are learned. Even if there was seed states, keep in mind memories are stored redundantly in biological brains. the obvious example is stroke patients that temporarily loose skills, but then recover them. Assuming there something like a seed, it would be seeds or parts of a seed stored in multiple places. then there's the fact that brains are analog, not digital. In classic computer science, associative memory is two elements linked with an address that points to both elements. In neural networks like GPT3, the association is captured by many weights. One could consider those weights as a seed, but the analogy breaks down.
      A common problem with neurons in a neural network is that a single neuron may participate in multiple patterns. If you change 1 weight, it can cause the network to perform better on one type of task and worse on a different task. The problem is called polysemantic neurons. There are people working on fixing polysemantic neurons, but there's no general purpose fix today.

  • @NajibElMokhtari
    @NajibElMokhtari ปีที่แล้ว +10

    This is amazing! I finished the video and immediately went looking for the next one where you'd explain the more accurate models. Then realized this is the first and only video on this channel.
    Can't wait for the upcoming stuff. Congrats for the launch!

  • @frenk3928
    @frenk3928 ปีที่แล้ว +18

    I studied Hopfield networks last year in the framework of spin glasses and all assumptions were dropped seemingly out of nowhere in a big confusing mess (as usually happens in lectures). This video really tied up all together and now makes so much sense beyond memorising proofs. Excellent video and brilliant demonstration.

  • @sciencelab4225
    @sciencelab4225 ปีที่แล้ว +26

    This is one of the most important pieces of media I have yet consumed to better understand this world. It is incredible to realize how one's brain works in principle. Thank you so much!

  • @macchiato_1881
    @macchiato_1881 ปีที่แล้ว +20

    It's really amazing how something as simple as a neuron has highly emergent properties that can do very complex tasks.

  • @Wonders_of_Reality
    @Wonders_of_Reality ปีที่แล้ว +13

    Looking forward to watching the second part! And many thanks for choosing dark text on a light background. It looks like a calm creamy ocean of fascination!

  • @martingisser273
    @martingisser273 ปีที่แล้ว +6

    Brilliant exposition for both mathematicians and nonmathematicians. It took a few decades to get me (mathematician) interested in neural networks. Now I'm hooked. Musing about connections to Markov processes and PDE...

  • @question_mark
    @question_mark 20 วันที่ผ่านมา

    guys, just wanted to remind you that this video is easily in the top 1% videos ever produced by a human

  • @konradd8545
    @konradd8545 ปีที่แล้ว +2

    Please continue making videos! This was not only super interesting but also very well explained and animated. I hope to see more videos on this channel soon!

  • @TR-kn3sn
    @TR-kn3sn ปีที่แล้ว +3

    This is absolutely incredible. Just immense, wow. I was initially skeptical of neural networks because they lacked the "logic" side of the problem (as I think a lot of people were at first) but I knew they were unreasonably effective, so I assumed they had their place. But now, I think they have more than a place, they are officially critical to our understanding of the world. In short, they were always cool, but now we've proven it. Thank you for making this video! It is so well made by the way!

  • @YasheshBharti
    @YasheshBharti ปีที่แล้ว +3

    Such a unique topic, so beautifully explained! Thank you so much for this piece of art!

  • @alexharvey9721
    @alexharvey9721 ปีที่แล้ว

    Super interesting and fantastically presented! I wish there were more videos like this! Love that you're not afraid of diving into deep concepts and details. Instant sub.

  • @histeward
    @histeward ปีที่แล้ว

    It’s rare to find a concise video about a learning topic. This is awesome! Hope your channel grows huge!!🎉

  • @BrandonWong55
    @BrandonWong55 ปีที่แล้ว +4

    Wow, this video is really eye-opening. As someone who has been interested in both memory sports and math, it really intrigues me how these false memories arise. Whenever I make a mistake memorizing a deck of cards, I'm always trying to figure out what when wrong, and it's cool to see a mathematical model also make similar mistakes in your video.

  • @redaabakhti768
    @redaabakhti768 ปีที่แล้ว +25

    Perfect video. I didn't know you could look at neural networks from this neat mathematical vantage point. I thought about eigenvectors and how the eigenvector with the largest eigenvalue would influence the network sort of as a dominant weight that increase exponentially , this would need a slightly different f the function you operate on the resulting weighted sum this would work for storing one memory maybe with increase possible states of single neuron and the right activtion function you will be ablse to store many patterns. Amazing video once again.

  • @brandontylerburt
    @brandontylerburt ปีที่แล้ว +1

    An excellent explanation. I'm starting to realize what a profound influence this type of neural network model has had in recent decades on all kinds of disciplines ranging from machine learning to the development on new theories on PTSD and how traumatic experiences alter brain function. Thank you for providing such a succinct and illuminating video on a complex and vitally important subject.

  • @eldertom
    @eldertom ปีที่แล้ว +1

    This is… ridiculously good. Thank you very much for making this, and really looking forward to the next one!

  • @br.3250
    @br.3250 ปีที่แล้ว +4

    This is just an outstanding explanation and it was a very interesting subject.
    And it's even more amazing that even though I know very little about this subject I could understand almost everything
    Keep up with the great work :)

  • @cmilkau
    @cmilkau ปีที่แล้ว +49

    I guess you have to make a distinction between short term memory (stored on activity patterns) and long term memory (this video? stored in the network weights). The latter is much simpler and in theory there could be a single neuron for a single memory, although in practice there are more efficient encodings.

  • @exinangai2216
    @exinangai2216 ปีที่แล้ว

    The best illustrated explanation on Memory modelling of inter-connected neuron (switch), bit and quirks of Neural Networks , formation and significance of multiple "stable states", and how they are presented as row & col vectors, update function as Matrices. Your video is both clear and excellent visual intro to the topics. I am really looking forward to your next video. Helping more ppl to visualize how transformers, recurrent net, back propagation, main idea of ML papers, etc are being modeled.

  • @pafloxyq
    @pafloxyq ปีที่แล้ว

    Hopfield nets were the first thing that got me interested into studying neural networks some years back, but I hadn't seen such an eloquent presentation on it since now. Will wait for the subsequent videos of this series.

  • @tf3confirmedbuthv54
    @tf3confirmedbuthv54 ปีที่แล้ว +40

    Love it! Hopefully this video will get big, this question is wonderful and really makes you think outside the box, and I think you explained it perfeclty.

  • @dererzherzog
    @dererzherzog ปีที่แล้ว +11

    Even a reductionist model of neural network (disregarding the dynamic spatio-temporal interplay of a plethora of neurotransmitters, neuromodulators, receptors, secondary messengers and downward pathways) leaves one awe-struck by hinting toward the tremendous scale of capabilities in a biological neural system.
    Thank you for this wonderful presentation and thanks to the YT algorithm for suggesting it! May your channel and its viewership grow exponentially!

    • @LuaanTi
      @LuaanTi ปีที่แล้ว +7

      And don't forget, the _in_ capabilities. Even from this extremely simplified model, you can see why it's hard to write memories, and keep them stable against change (which of course happens even when you're just _reading_ the memory). The computer on your desk can easily store terabytes of information without a glitch just by "remembering" the information one by one, while the computer in your skull needs to keep reinforcing what you want to remember, and keeps overwriting related memories as others are read or changed.
      Mind, this is most likely a critical component of learning and intelligence in the first place - your memories aren't just static data stored elsewhere, they're ultimately a part of the computational network itself. Every experience has _some_ effect on your decision-making, and memories interact with each other (and with other inputs from all over the brain and the outside world).

    • @goldnutter412
      @goldnutter412 ปีที่แล้ว

      @@LuaanTi apparently the brain throws out TONS of data all the time.. anyway won't repeat the other posts.. maybe read those first above.. but it's a good filter/cache for information processing.. short, long and permanent information. Only what is really significant to us, lasts.. and we come back to evolve past our problem with entropy. The unknown is enticing.. fear of the unknown is like getting stuck in loops on tangents.. or outright stuck and unable to deal with whatever patterns we're experiencing.
      The optimal configuration for experience is of course permanent state, temporary brain and universe. Next time, the patterns will have a completely different basis, while still understandable ! almost every random set of DNA and environment will be the same pattern. Choose, which gives result (quantum mechanics delivering data on demand), JIT type of interpretation with the brain cache, more choices, more life experience, on and on we go.
      Because we are self aware, we are aware of entropy.. time advances, we collect more data. More entropy unless we evolve away from "fear" or.. I guess you could call it over management of the data structure. That's the nature of information ?
      "Let it be" playing in the background..

  • @rafaeladivapinho2399
    @rafaeladivapinho2399 ปีที่แล้ว

    You are by far the best teacher for soft soft . It's very complicated at first - overwhelming, actually - but, you make it doable for

  • @JoelGreenyer
    @JoelGreenyer ปีที่แล้ว

    Amazing video. Very insightful. I cannot imagine how much work went into the animations. Very well done!

  • @nulled7888
    @nulled7888 ปีที่แล้ว +3

    Super interesting video, got me really thinking about the correlations to real neurons.. very well made as well :)

  • @venamaybe
    @venamaybe ปีที่แล้ว +3

    This was super interesting and awesome!! I'm excited for the next video :)

    • @LoongBerries
      @LoongBerries 5 หลายเดือนก่อน +1

      looking forward to the next video too

  • @Gnevnyj
    @Gnevnyj ปีที่แล้ว

    Instant subscription. This video was extremely clear and well-made. Looking forward to your next lectures!

  • @LiquidMasti
    @LiquidMasti ปีที่แล้ว

    can't believe it's your first video. it's very well explained and the background sound makes it even more compiling.

  • @brandonzhang5808
    @brandonzhang5808 ปีที่แล้ว +7

    What I'm getting out of this is that there is still a scalar memory parameter in the number of neurons or size of the weight matrix, and storing patterns in it is similar to file compression, allowing the same network to represent multiple patterns, but with a compression ratio that is actually disadvantageous, requiring O(n^2) weights to store O(n) patterns. Something of a Rube-Goldberg for memory systems.

    • @dominikwitczak1655
      @dominikwitczak1655 ปีที่แล้ว +1

      Eactly, this does not prove that our memory can't be measured in Gb of information.

  • @anon_y_mousse
    @anon_y_mousse ปีที่แล้ว +3

    I don't know why you haven't made another video in 9 months, but I'm hoping you continue this series. Maybe go into the math a little bit more, but keep it simple so even a simpleton like me can understand it.

  • @hcky6054
    @hcky6054 ปีที่แล้ว

    I really like how you explain neural networks with necessary elements only which helps me focus on the concept itself!
    Nice work!

  • @jaumefabregaroda
    @jaumefabregaroda ปีที่แล้ว

    Mindblowingly good! You left me in awe, please keep uploading videos!!

  • @schnitzelsemmel
    @schnitzelsemmel ปีที่แล้ว +3

    Yes yes yes this is exactly the video i didnt even know i needed. incredibly good explanation and animation

  • @somedudeok1451
    @somedudeok1451 ปีที่แล้ว +4

    Love the presentation. I think that the mistakes you were frustrated and fascinated with are maybe a chance to generate something new from something stored. It reminds me of our ability to be creative by morphing our memories over time. It's like when you have memory A and memory be and over time they fade and become memory AB which then turn into memory C. And sometimes memory C might actually be something useful. It also reminds me of generative AI. Of course, this is all just layman talk and I have no idea if this has anything to do with how human memory actually works.

  • @fabiospinelli4179
    @fabiospinelli4179 ปีที่แล้ว

    Wow this video is Gold, I'm impressed by the fact that this is your first!
    I will look at your career with great interest

  • @arvid_music
    @arvid_music ปีที่แล้ว

    Thank you for this very high quality and interesting video! The visuals and examples really help in making the topic easier to understand.
    God bless you and keep up the great work!

  • @ofrigal9557
    @ofrigal9557 ปีที่แล้ว +3

    Amazing video! Great job, i would love you to explain autoencoders

  • @alzario
    @alzario ปีที่แล้ว +4

    Awesome video, comparatively digestible explanations and really good visualisation of what you're describing.

  • @bakdiabderrahmane8009
    @bakdiabderrahmane8009 ปีที่แล้ว

    the IDEA explained in this video is something a was looking for for a looong time, thank you so much you have no idea how helpful you have been.

  • @Neptoid
    @Neptoid ปีที่แล้ว +1

    This had amazing animations and insights I have been wondering about for a long time, and although I didn't understand much of the math I still felt grounded by insight after insight to the end

  • @theClaytron
    @theClaytron ปีที่แล้ว +3

    If you change the procedure to use a synchronous update, and make the activation function the identity, then you can analyze the dynamics of the weights matrix W. That is, v_{n+1} = W*v_{n}. In this case, a memory which you converge to is simply a stable eigenvector of the matrix W. When you "train" the network with multiple memories, you are performing multiple rank-one updates to the matrix. This also explains why the maximum number of memories you can store in W varies linearly with the size of W, since a matrix of size NxN cannot have a rank greater than N. In other words, if you tried to train W on N+1 memories, you are guaranteed that at least one of those memories is a linear combination of the other memories. Not only that, if you add a memory with a rank-one update to W, that does not necessarily mean that memory will be an eigenvector of W. The memories can combine such that an eigenvector emerges which is not equal to any of the memories.
    This approach also gives another perspective on why you can converge to the "flipped" version of the memory. An eigenvector defines a 1-dimensional space which is simply a scalar times a vector, where the scalar can be positive or negative.
    Also, the properties of the W that emerges from Hebbian Learning are very interesting. That is, W is symmetric and has a zero-diagonal. It is a property of symmetric matrices that their eigenvalues are all real. So we would not expect the network to oscillate around some solution. Also, the trace of a symmetric matrix is equal to the sum of the eigenvalues, so that means if we have negative eigenvalues, we must also have positive eigenvalues.
    Again, this is just some conclusions for the case of synchronous update and identity activation function. It may not be so relevant to the case you discuss.

    • @layerwiselectures
      @layerwiselectures  ปีที่แล้ว +3

      That's perfect, thank you. The original paper does not use this reasoning to arrive at the linear memory capacity, but the similarity of the conclusions is striking. Thank you for this addition!

    • @theClaytron
      @theClaytron ปีที่แล้ว

      @@layerwiselectures Thank you for your excellent video! I am looking forward to the sequel.

    • @isaacaraya3848
      @isaacaraya3848 ปีที่แล้ว

      @@layerwiselectures Have you heard of adaptive resonance theory? Could the stable states in the network be harmonics? Like if each pattern is stored not just spatially but at a specific frequency of oscillation then you could maximize the number of memories a single neuron could contribute to. Kind of like the tv's that play multiple images on the same screen at once

    • @isaacaraya3848
      @isaacaraya3848 ปีที่แล้ว

      Great insight to complement video. Could you expand on why symmetric matrices can't oscillate around a solution? Might save me some googling

  • @dominicbaumer6038
    @dominicbaumer6038 ปีที่แล้ว +6

    Thanks for this great lecture! My idea for a next video topic is that you can talk about differences between artificial perceptrons and biological neurons

  • @mo_i_nas
    @mo_i_nas 4 หลายเดือนก่อน +1

    in a neural network of 3 words... THIS WAS AMAZING

  • @hengchunsong301
    @hengchunsong301 ปีที่แล้ว +1

    Very rarely TH-cam has video with this much deep insight. Treasure.

  • @drdca8263
    @drdca8263 ปีที่แล้ว +4

    I don’t know that this really shows that the memory capacity can’t be appropriately described in terms of “how many bits”.
    That there aren’t addresses for physical locations for different chunks of memory, sure.
    But, information theory isn’t fundamentally inapplicable to the brain’s ability to remember things?
    Like, if there is some random distribution of possible observations (e.g. a sequences of symbols, or other combinations of e.g. color, shape, texture, whatever) at one time, and then at a later time the person says their guess about all the information needed to describe which of the observations were randomly selected,
    then, ((the entropy of the distribution of observations) - (the conditional entropy of the distribution of observations given the person’s recollection of the observation)) should give a lower bound of how much information about the observation the person remembers on average. This can be expressed in bits.
    Of course, this isn’t describing the person’s entire memory capacity! Just how much they remember from these observations in such tests.
    This could, I think, be used to measure an estimate of a lower bound on how much information (measured in bits) a person can rapidly memorize for a short amount of time.
    Presumably how much info one could memorize would depend on the way the information is expressed. For example, I would expect a random sequence of words would be easier to memorize than a random sequence of literal bits with approximately the same entropy. And that’s part of why I say it would only be an (approximation of) a lower bound. (Approximation because there’d be measurement error due to the random variation.)
    We can also probably put an extremely weak upper bound on total capacity based on the amount of information theoretically needed to describe the physical state of the brain. If nothing else, there is at least the Bekenstein bound based on just the size of the brain along with the fact that it isn’t part of a black hole. This is so weak of an upper bound that it is barely worth mentioning.
    I don’t know how to get a better estimate of “the entire memory capacity of the brain” than these though.
    Well, of course, you can look at how much data people have memorized over longer periods of time.
    Oh, maybe some estimates could be made based on people with eidetic memory.
    Like, if we query facts about randomly selected dates and check how accurate what is recalled is, then, we could extrapolate that and assume it would be equally accurate for all the many dates in the same range which we didn’t ask about, then we could estimate from that how much they remember overall from that range of time.
    This could probably give an estimate which reveals a rather large amount of information stored.
    And, presumably, people with an eidetic memory don’t have just, an inherently larger memory capacity, so much as, just more stuff gets committed to memory? (I could be wrong about that.) So I think this would still give a good estimated bound on how much memory capacity a brain in general has.

    • @layerwiselectures
      @layerwiselectures  ปีที่แล้ว +2

      Wow, thanks for the comment. There's a lot to think about. My favorite part is the black hole information bound. No you're right, information theory isn't fundamentally inapplicable here and I didn't mean to imply it was.
      Personally I'm not clear on whether the "bits" in a hard disk are information theoretic bits. Are they? Maybe in the sense that, if I didn't know the state of the hard disk, and then I looked to find a certain pattern there, I would have gained that many bits of information? But then on the other hand I wouldn't expect every pattern to be equally likely to appear on a hard disk or in RAM.
      What I'm getting at is that, yes, there must be a way to compare the information in both memory systems (computer and neural) in bits, but no, it isn't usually being done when we talk about computer memory.
      My goal was to show that casually speaking of bits of memory has a tendency to obscure real differences between the two memory systems. If in the process I appeared to discount information theory ... oh my, not my intention.
      Thanks for the comment again!

    • @drdca8263
      @drdca8263 ปีที่แล้ว +1

      @@layerwiselectures I think I phrased my comment a bit too critically given how much I liked the video. I quite liked the video, and subscribed (before your reply).
      I think computer bits can be related to bits in information theory by, if you sample from a distribution many times, the optimal way to store such a sequence on disk, would have its length be on average, the number of samples times the entropy of the distribution. And, I think if the number of samples is large, then the variance in how much storage is needed overall should go down, eventually to the point where you can regard it as nearly always the same. ...uh, assuming the variance of the number of bits each sample contributes, is finite (which it will be if the number of possible outcomes is finite).
      So that’s at least *a* connection. It perhaps isn’t quite as close a correspondence as one might hope? Well, the amount of information in an information theory sense that can be stored by a hard drive, is bounded by the storage capacity, and at least for some distributions of a random variable, this bound is exact. But for other random variables where the outcomes aren’t just 2^n equally likely outcomes, then I guess there can be some overhead compared to the ideal, in storing a finite number of samples. But in the limit of many samples, I think the two notions of “amount of information” become the essentially the same? In some sense anyway.

    • @sullivan3503
      @sullivan3503 ปีที่แล้ว

      @@layerwiselectures You're correct in saying that information theoretic bits don't always correspond exactly to hard drive bits. In order to calculate the amount of information stored, you need to also understand the encoding. For instance, ASCII text uses 7-bits of information per character to store text, but these 7 bits of information are often stored at byte boundaries, resulting in each character taking up 8 bits of space on the disk. So the leading bit is wasted. Compression can also reduce the amount of bits needed to store a certain piece of text drastically. So, as long as your data is not maximally compressed (which is hard to tell), then your hard drive bits is not the best measure of the amount of information stored. However, in practice, we typically use file size as a good proxy for information.

  • @Kaepsele337
    @Kaepsele337 ปีที่แล้ว +3

    That's really neat. I wonder what happens if you add "eyes" to the network, i.e. some subset of neurons whose activation is driven by an external signal instead of the update function.

    • @layerwiselectures
      @layerwiselectures  ปีที่แล้ว +2

      Good point. The original paper actually mentions an input current to the neurons. This would then be combined with the summed input of all other neurons, so it would modify the update function rather than replacing it. I left this out for simplicity. However, it's actually an important detail once you start thinking about it. It sort of biases the network to a certain state and would explain where the starting state of the network comes from, if it's not random. The story would then go like this: the "eyes" of the network (love the metaphor btw) would cue the network with a partial state and the network would then autocomplete to its memory state. Kinda-sort-of-almost-like seeing a visual cue of some sort and then being reminded of a scene from your past - if we're being a little bit fanciful. Thanks for the comment.

  • @matthiasdenhamer3925
    @matthiasdenhamer3925 ปีที่แล้ว

    Absolutely mindbogglingly well done. Can't wait to see more videos from you!

  • @pumkin7222
    @pumkin7222 ปีที่แล้ว

    great first video! keep it going I'll be here to support you!

  • @user-of9sr8bm9i
    @user-of9sr8bm9i ปีที่แล้ว +2


  • @d0tz_
    @d0tz_ ปีที่แล้ว +9

    This is the best SoME video I’ve seen so far. Such an intuitive explanation for a niche yet mindblowing idea.
    Maybe it’s my lack of intuition for high level math or my affinity to computer science, but most of the other videos had me closing out of it mid way, while this one was interesting in its entirety.

  • @Ryush806
    @Ryush806 ปีที่แล้ว +1

    I just learned your video was for 3Blue1Brown's contest by watching his video about the contest. Makes sense why you only have 1 video. I originally watched it weeks ago and was dumbfounded by it being your debut. PPLLLEEEAASSSEE make more! You have a knack for explaining a very complex topic in terms us commoners can understand!

  • @TenderBug
    @TenderBug ปีที่แล้ว

    I find this explanation is fascinating. Shines light on neural networks from an uncommon direction. Great Video 👍

  • @okopyl
    @okopyl ปีที่แล้ว +10

    Too complicated for somebody with 0 knowledge in math

    • @BrandonMcCurry999
      @BrandonMcCurry999 26 วันที่ผ่านมา +2

      Do you have the ability to learn?
      I do, and I'm slow, especially on things with many steps that must be done in an exact way or the whole "thing" will be incorrect

    • @BrandonMcCurry999
      @BrandonMcCurry999 26 วันที่ผ่านมา +2

      Or were you not talking about you

    • @BrandonMcCurry999
      @BrandonMcCurry999 26 วันที่ผ่านมา +1

      Difficult doesn't mean it can't be done, many things have taken alot of work for me specifically
      Some things come naturally
      I tend to go with what comes naturally, Plato my strengths

    • @dorionporath4363
      @dorionporath4363 20 วันที่ผ่านมา +1

      Elementary school

  • @LamNguyen-zc8xs
    @LamNguyen-zc8xs ปีที่แล้ว

    This is brilliant. I've never seen any video about Hopfield network as interesting as this video. All they show are just math. Man you should do more video on brain and neuron networks like this :D

  • @fabioteixeira868
    @fabioteixeira868 ปีที่แล้ว +1

    Wow! Awesome first video! Subbed and eager for more!

  • @hardronix
    @hardronix ปีที่แล้ว

    Thanks for opening my eyes. I already knew about recurrent networks and stuff, but now I think about them with an existential crisis. Thanks for such an awesome explanation!

  • @Tigrou7777
    @Tigrou7777 ปีที่แล้ว

    By looking at this video, you just updated the neural network in your brain to be able to remember stuff about neural networks. Brilliant.

  • @awritrojitbanerjee1055
    @awritrojitbanerjee1055 ปีที่แล้ว +1

    you knocked it out of the park dude! need more of the good stuff!

  • @sinaomidvar2807
    @sinaomidvar2807 3 หลายเดือนก่อน +1

    The content you provide is greatly appreciated. We are looking forward to your new contents. ❤

  • @minh9625
    @minh9625 7 วันที่ผ่านมา +1

    Guy, it's 1 year already. Cmon comeback bro

  • @donewithwork
    @donewithwork ปีที่แล้ว

    Nice video!! Really like how you broke down complicated terms into more manageable discussions.

  • @meteorinc4259
    @meteorinc4259 ปีที่แล้ว

    What a great quality video! Please keep releasing more like this!

  • @ariondorin4480
    @ariondorin4480 ปีที่แล้ว

    This is awsome ! Man, you really understand things deep and what is more, you really explain the essential!

  • @loganpowell
    @loganpowell 2 หลายเดือนก่อน +1

    Dude, this is the best video on the topic I've found (and I've looked). When can we expect a follow-up? Would love to watch it.

  • @htfx11
    @htfx11 ปีที่แล้ว

    This video REALLY helped me make sense of Hopfield Networks. Outstanding work

  • @satadrudas3675
    @satadrudas3675 2 หลายเดือนก่อน

    This was a fantastic explanation. Waiting for more from you!!

  • @AdobadoFantastico
    @AdobadoFantastico ปีที่แล้ว

    Awesome video, this clarified some things I'd wondered about and taught me some things I hadn't thought about yet.

  • @artysanmobile
    @artysanmobile 4 หลายเดือนก่อน

    The thumbnail image makes a very important point on the subject of human thought. We don’t look for the room, shelf, book, page. We build a complete path to every experience we have ever had since birth. If we contextualize it by re-use, or by interconnecting it with another path, we improve its clarity, its importance, and remember that address. The classic explanation for having no idea what you ate this morning. Breakfast has not yet been made important.

  • @clementej03
    @clementej03 ปีที่แล้ว

    Amazing video, I tried my self to model a neuron network on my own but barely even knew where to start. Fascinating subject and an excellent video.

  • @generativeresearch
    @generativeresearch ปีที่แล้ว

    This video is really well made for a first video. Keep up the good work man

  • @spin6720
    @spin6720 ปีที่แล้ว +2

    I don't usually comment on youtube but.. I don't know what to say. This is by far one of the most enlightening videos I've watched ever. I just subscribed and I'm looking for your next video

  • @Prince-xp9vk
    @Prince-xp9vk ปีที่แล้ว +2

    had an idea to write about this, "is the brain really a RAM or ROM?", but seeing you do justice to it, I can safely procrastinate for another day.
    it's kinda sad too that you only have one video. please do more.

  • @manofculture69
    @manofculture69 ปีที่แล้ว

    Astonished that this video is the first video by you. Kudos 👌

  • @Personal-M.I.S.
    @Personal-M.I.S. 4 หลายเดือนก่อน +1

    Well done! I'm sure I'm not the only one who would appreciate it if you did more =)

  • @FelipeCampelo0
    @FelipeCampelo0 4 หลายเดือนก่อน +1

    When a video catches your curiosity in the first two minutes that’s because it’s going to be a piece of art !

  • @YZhou-mq1bw
    @YZhou-mq1bw 6 วันที่ผ่านมา

    You definitely should keep making the vids if you are still interested, this is insanely amazing! You must be a big name there!

  • @Mangputt
    @Mangputt ปีที่แล้ว

    I can't imagine, how great you put effort to this Informative Video. The way you explain, animate and Compositing evidence, awesome..
    (Sorry for my bad English)

  • @user-pd4bb8tx2n
    @user-pd4bb8tx2n 6 หลายเดือนก่อน +1

    What a fantastic explanation. I'm looking towards more content from you!

  • @fr3zer677
    @fr3zer677 ปีที่แล้ว

    What a fantastic video! I'm eager to see what comes next!

  • @TheMan84466
    @TheMan84466 ปีที่แล้ว

    just stumble into this video on my feed absolute awesome please make more videos like this !!!!

  • @persifalellian5690
    @persifalellian5690 ปีที่แล้ว

    This is awesome, please continue your work. Good luck!🎉

  • @waitingforgodot355
    @waitingforgodot355 4 หลายเดือนก่อน

    That was phenomenal. Fingers crossed for the "follow-up video"!

  • @alexanderk5835
    @alexanderk5835 ปีที่แล้ว

    Nice way of presenting and storytelling, got me motivated to do some experiments.
    Thank you for the video.

  • @wolfpants
    @wolfpants ปีที่แล้ว

    Hey, you are really good. Very brain-friendly combo of writing, graphics, and voice (at least for me). Eagerly awaiting another.

  • @asanzramaxi
    @asanzramaxi ปีที่แล้ว +1

    didnt realize the views and subscribers until casually saw comments. keep it up, such a good video!

  • @acyned8079
    @acyned8079 ปีที่แล้ว +1

    The conclusion really was what I was here for. I didn't get most of it but I got the point of it!

  • @hvdveer
    @hvdveer ปีที่แล้ว +2

    Amazing, how the brain stores memories is one of the biggest enigmas in neuroscience and you've just provided an answer using it's digital equivalent. This is worthy of a scientific publication. Brilliant stuff.

    • @layerwiselectures
      @layerwiselectures  ปีที่แล้ว +5

      I should clarify: the video is based on an existing scientific publication (Hopfield, 1982) as mentioned in the video. See the link in the description.
      It's brilliant, but not *my* brilliance, is what I should make clear.

  • @AK-vx4dy
    @AK-vx4dy 4 หลายเดือนก่อน

    Excelent explanation in quite concise form. Great job !