Amazing content as always. You have mentioned that Bert provides Word embeddings. However, it also provides Sentence embeddings : CLS token, and it has been proved that it works very well on text classification tasks.
IMHO the next step WILL be to detach the "database" function of the FeedForward layer of GPT architecture, but that mean being able to retrieve and distinguish syntactic vs ontological aspects of data. Syntactic knowledge is what give the reasoning power and ontology is the facts. Given that the power of the system is to "tag" words with hidden classes (or type), the difficulty is that both syntax and ontology are expressed the same way, with "bag of words", noun -> class/type of word that refer to objects (syntactic), capital -> class/type of place with a specific meaning (ontology).
Amazing content as always. You have mentioned that Bert provides Word embeddings. However, it also provides Sentence embeddings : CLS token, and it has been proved that it works very well on text classification tasks.
I teach AI. And you do a great job at explaining these concepts well. Kudos.
Thanks a ton for the compliments
Great stuff as usual. Add that to the history section of your upcoming "Survival manual from our AI overlord" compendium.
Hahaha! Yesss! Thanks so much for commenting :)
IMHO the next step WILL be to detach the "database" function of the FeedForward layer of GPT architecture, but that mean being able to retrieve and distinguish syntactic vs ontological aspects of data. Syntactic knowledge is what give the reasoning power and ontology is the facts. Given that the power of the system is to "tag" words with hidden classes (or type), the difficulty is that both syntax and ontology are expressed the same way, with "bag of words", noun -> class/type of word that refer to objects (syntactic), capital -> class/type of place with a specific meaning (ontology).
Your make this concept s easy to understand.
Thank you for the complements!
Bravo !!!
Bravo !!!