TLDR - from llama 2 to llama3 they switched from sentencepiece to tiktoken - vocab size 32k -> 128k - ~15% fewer tokens for english, ~50% fewer for "some other languages"
could someone from the meta LLaMa 3 team please explain how to train my very own tiktoken tokenizer like you guys did for llama 3. there is no opensource steps to recreate this
Classic example of a provably smart guy not being able to express his thoughts... 5 minutes of pain is all I managed to force myself to watch. A shame.
TLDR
- from llama 2 to llama3 they switched from sentencepiece to tiktoken
- vocab size 32k -> 128k
- ~15% fewer tokens for english, ~50% fewer for "some other languages"
adding to the list
compression ration played a vital role in reducing the token length
Aston please explain the attention mechanism , Actually I am stuck in the chapter "Attention and transformer" of your book d2l
My instinct is tokenization is underestimate in importance. Usually the hard and boring fundamentals are where the magic happens. Across all fields.
Informative
Is the Llama 3 paper out yet? He mentions it @ 24:02
Yes! You can read the Llama 3 research paper here: ai.meta.com/research/publications/the-llama-3-herd-of-models/
could someone from the meta LLaMa 3 team please explain how to train my very own tiktoken tokenizer like you guys did for llama 3. there is no opensource steps to recreate this
You develop Tamil language for Tamil users
so this guy is payied to use open sourced tiktoken
ok.
x
Classic example of a provably smart guy not being able to express his thoughts... 5 minutes of pain is all I managed to force myself to watch. A shame.
i don't think this format works unless the intent is to discuss at a high level.