i was so excited after reading this paper, thanks for having him on the podcast! great talk! I’m currently working on an fleshing out an architecture idea specifically designed to take advantage of the insights from this paper. goal is to start coding & training soon 🤞
Sumeet, is the any insight into the size of embedding to which dimension gets most reward of entropy gradient?[ I am assuming embedding dimensions somehow contribute in entropy of a point in embedding]
Sumeet , in your argument of sampling the next token, makes a sort of random choice in the neighbourhood , which in a black-box view gives illusion of learning .
Illusion of learning comes from the fact that a context sequence (prompt) induces the Transformer to walk on a (predetermined) path that appears like intelligent behaviour to us humans. Randomness is in the choosing of one of the predetermined paths. But if all the low-entropy paths appeared like intelligent behaviour to humans, then no matter what path was randomly picked, it would still appear as if the model was saying something intelligent. If I understand you correctly, you're saying that because the model can produce a variety of paths due to random sampling, it appears intelligent to us, as opposed to a parrot regurgutating from memory. Yes, that would be accurate under this viewpoint.
i was so excited after reading this paper, thanks for having him on the podcast! great talk! I’m currently working on an fleshing out an architecture idea specifically designed to take advantage of the insights from this paper. goal is to start coding & training soon 🤞
Sumeet, is the any insight into the size of embedding to which dimension gets most reward of entropy gradient?[ I am assuming embedding dimensions somehow contribute in entropy of a point in embedding]
Sumeet , in your argument of sampling the next token, makes a sort of random choice in the neighbourhood , which in a black-box view gives illusion of learning .
Illusion of learning comes from the fact that a context sequence (prompt) induces the Transformer to walk on a (predetermined) path that appears like intelligent behaviour to us humans. Randomness is in the choosing of one of the predetermined paths. But if all the low-entropy paths appeared like intelligent behaviour to humans, then no matter what path was randomly picked, it would still appear as if the model was saying something intelligent. If I understand you correctly, you're saying that because the model can produce a variety of paths due to random sampling, it appears intelligent to us, as opposed to a parrot regurgutating from memory. Yes, that would be accurate under this viewpoint.