Speculative Decoding: When Two LLMs are Faster than One

Speculative decoding (or speculative sampling) is a new technique where a smaller LLM (the draft model) generates the easier tokens which are then verified by a larger one (the target model). This make the generation faster computation without sacrificing accuracy.
0:00 - Introduction
1:00 - Main Ideas
2:27 - Algorithm
4:48 - Rejection Sampling
7:52 - Why sample (q(x) - p(x))+
10:55 - Visualization and Results
Deepmind Paper: arxiv.org/abs/2302.01318
Google Paper: arxiv.org/abs/2211.17192

[@igorfilippov1221]

Very clear explanation, thank you!

[@kaenovama]

Really well done, my brain bulb went light up when you show the table! Thank you, keep it up!

[@vukrosic]

Thank you for explaining it!

[@kevintai8656]

Great work! Thank you

[@chaidaro]

I haven’t read the paper yet but my understanding is that we sample from q(x) - p(x) because we want the most surprising token that the draft model does not anticipate. It should maximize the entropy but then it should have log on the equation, anyway, I gotta read the paper to understand the math.

[@magalodontestnoneworld9043]

Very good video!

[@paull923]

great explanation
thank you

[@dorianlin491]

Really helpful video!

[@einsteinsapples2909]

Thank you for the video, when i first heard this idea in February i was wondering how it made sense because i was picturing a large K, now seeing that the recommended K is about 3 I understand how most of the output will be the same.

[@user-yf7nq3mg3s]

Great video 👍

[@decycle2912]

really informative! One thing that I don't understand is how does the LLM knows the previous probability distributions in a single pass? I thought decoder llm's only outputs the new token's probability distribution

[@EfficientNLP]

This is due to the way the transformer architecture is set up: during decoding it takes as input all previous tokens and computes the hidden states for all previous tokens at each layer. Since we have the final layer hidden states, it is possible to obtain the probability distributions for all previous tokens.

[@henkjekel4081]

@@EfficientNLP Now I get it. Did you also explain this somewhere in the video? Maybe link to it in the describtion because this is also where I didnt understand it.

[@chaidaro]

We can compute the probability for each token in a batch fashion. The trick is to use mask attention.

[@420_gunna]

Love your video, thanks!
If I had to give one request/critique, it'd be that I wish there were some slides in here similar to Samuel Albanie's videos that are quite information-dense recaps that could be lifted out of the presentations and put into our notes (or into a powerpoint for a paper club, or something).

[@EfficientNLP]

Interesting idea, though my videos often contain animations, drawings, screencasts, etc., and are not directly a recording of PowerPoint slides. Feel free to take screenshots of my videos for any educational purposes though!

[@waynelau3256]

Thanks for this! I've been enjoying your videos! Do you think you do a review / explanation on flash-decoding by tri dao? I have been reading the pytorch blog but I don't really understand it

[@EfficientNLP]

Thanks for the suggestion, I will add it to my list of future topics!

[@Basant5911]

made very simple, but one more variable is choosing right draft model. Suppose if one chooses that is too too away from larger one's distribution then its also a problem.

[@EfficientNLP]

If the draft model is far from the target model's distribution, then speculative decoding will be less effective because it will have a higher rejection rate, thus reducing the speedup. However, the algorithm guarantees that the output sequence will be identical; therefore, even if the draft model is of poor quality, the text generation quality will not be affected.

[@mingzhou2213]

thank you for the explanations and the visuals. Does speculative decoding work with beam search? I understand that for LLM we generally just do greedy decoding in one pass, but for translation models like whisper, the performance increase significantly if we use beam search. I see even from hugging face official post discussing how speculative decoding improve whisper large inference speed by 2x, but to be honest, for non english audio data, with greedy decoding whisper is barely usable...

[@EfficientNLP]

Interesting idea, but I don't know of any attempts to combine them. Speculative decoding relies heavily on random sampling, whereas beam search is deterministic, so probably they are incompatible. For speeding up whisper inference, you might try using smaller or quantized models on faster engines like CTranslate2 or faster-whisper.

[@laulinky334]

Thanks for sharing, I am wondering how target model check the generated tokens of draft model and produce probability distribution q of x for each token?

[@EfficientNLP]

This is due to the parallel nature of transformers - when given a sequence of tokens, it can generate the logits for all of them in parallel, unlike generation which must be done autoregressively.

[@kylewilliams9214]

Google and DeepMind doing the Spiderman meme 😅

[@ariellubonja7856]

Thanks! But doesn't the google paper define Mq as the draft model i.e. flips the definitions?

[@EfficientNLP]

You are right; the Google paper uses a different notation from the DeepMind paper, in this video I'm using the DeepMind notation.

[@domenvake3077]

this is great! is there any chance you could demonstrate something like this in code?

[@EfficientNLP]

Good question - it appears that neither of the deepmind or google papers have an official source code implementation, but there are several implementations of this idea on GitHub, but I have not looked at them.

[@feixyzliu5432]

why does target model running with K new tokens spend almost the same computation than with just 1 new token? I know K new tokens can be computed in parallel at one single forward pass, but self-attension with K new tokens indeed need more works than 1 token (suppose KV-cache is used), isn't it?

[@EfficientNLP]

It's true that the computation time in the two scenarios might not be exactly the same due to KV cache and other implementation details; however, for simplicity, we can assume one forward pass through a model as taking one unit of time. Both decoding one token and checking the probabilities of multiple tokens require one forward pass.

[@gnorts_mr_alien]

For this to work, the two models need to have identical tokenizations right? Is there any way around it?

[@EfficientNLP]

That's right - the two models need to use the same vocabulary so that we can compare their logits meaningfully.

[@gnorts_mr_alien]

@@EfficientNLP thank you for the quick response. that makes sense!

[@murtazanazir9997]

Not necessarily. We can retokenize the predicted text by draft model. That can be slow though.

[@christospapadopoulos7894]

Nothing really new about this, it seems that big tech companies really do have it easier when publishing research

[@EfficientNLP]

That’s the way it tends to go! One small step at a time