how does the gaussian initialization of the key and query values factor into the model? wouldnt it be enough to initialize the learnable parameters? in the second iteration of the processing, the key and query values would be computed based on the parameters and therefore overwritten anyways.
Amazing videos on Transformers, really enjoyed them
Thank you. You are very welcome.
Thank you for the great explanation. Greets from Austria
I am happy you liked it. Grüße zurück!
how does the gaussian initialization of the key and query values factor into the model? wouldnt it be enough to initialize the learnable parameters? in the second iteration of the processing, the key and query values would be computed based on the parameters and therefore overwritten anyways.