I wanted to know what mel spectrograms are but then I watched the first video of the series and now I'm learning a far better-stepped approach to Audio Signal Processing and I'm loving it!
Just would thought i would let you know that i am about to finish my thesis on Dolphin vocalization feature extraction and distinction using ML classifiers and just found your videos. Very easy to understand and visualize concepts that took me more time that a wished to understand. Keep up the good work
Thank your for your great work ! At 23:00 there is something I don't understand : you divide by two the frame size, because of the nyquist rule. You obtain 501. But the 501 frequency bins are for...the interval (0 , sampling rate/2). So, in the end, you divide by 4 ! The sampling rate is divided by 2, THEN the framesize is also divided by 2. Could you explain this,? Thank you.
Wow, what a great explanation, this has really answered a lot of questions I had about how FFT would work on longer samples. Grazie mille, è stato un piacere guardare il video..
Thanks a lot for the wonderful explanation. I very first started to learn abt MFCC later, am interested in watching all the videos related to the audio signal processing series.
Am I correct in saying that multiplying the signal for a rectangular windowing function makes no sense, since you are only taking into consideration the samples of each frame and not the rest? It makes sense if the function is not a rectangle, and you have a curve at the edges.
Thanks for the fantastic video. One quick question. What are the advantage and disadvantage for setting frame_size > window_size ? What is the use case for this parameter choice ?
When you multiply your signal with the window function in the time domain you are convolving the frequency response of the window function with your signal in the frequency domain. The frequency response of most window functions is some form of a sinc function. Sinc functions are long and ringy, so the result of the convolution is to smear out the frequency response of your output. This reduces the accuracy of the output of the STFT. There are other spectral decomposition algorithms that produce more accurate results. The STFT is popular because of it's ease of computation, not because of it's accuracy.
Yes this is the dream scenario emerging! - So I am a bit slow but I think I could use e Spectrograms as the input feature layer directly into a convolutional networks or LSTM networks.
Hey man! Your fourier stuff is great! I've been playing with audio a bit and found that tensorflow has an awesome method for getting real-valued spectrograms using a method called MDCT (modified discrete cosine transform). Have you ever considered making a video about that? I'm curious to know how that's different from STFT
The "temporal" information isn't contained in the DFT phase?, because the DFT is a linear transform, there is not any lose of information, applying the inverse DFT we can get back our signal with it's temporal distribution
a Wonderful video, we need to know more about speech processing and especially the mirror process of inverting spectrograms and STFT, could you please suggest me where i can find explanation about the Griffin & Lim algorithm.
hello, please if you can provide us the process of the inverse short time fourier transform for overlapped frames, in order to recover the original time domain signal.
@@ValerioVelardoTheSoundofAI Thank you for your quick response. Actually, I'm a master's student in audio signal processing and your videos helped me a lot to do my master's project. Thanks an ocean :)
@@ValerioVelardoTheSoundofAI I hope you can help me with this doubt. Is the Hann Window applied after STFT to each frequency bin or before the STFT to each sample within the frame? I got a bit confused since in the STFT formula you have used w(n) and on the Hann window formula w(k). Could you please clarify this? Thanks once again for your amazing videos.
Dont you know why when using librosa stft function in python, the resulting number of temporal bins does not equal to math.ceil of (num_of_samples - frame_size) / hop_length + 1, but rather to math.ceil of (num_of_samples) / hop_length? I am processing some audio files and calculating stft for 66150 samples of signal as input to the function, using 2048 window and 512 samples as hop size... So in theory, I should get (66150 - 2048) / 512 + 1 = 126,199... ~127 temporal bins... But rather than that, the temporal output shape of stft does have 130 elements.. How is it calculating the last few windows, for which the function should not actually have enough samples available, as they are out of provided signal range?
I encountered the same thing using Librosa. ``` number_of_samples = 661500 FRAME_SIZE = 2048 HOP_LENGTH = 512 debussy_spec = librosa.stft(debussy, n_fft=FRAME_SIZE, hop_length=HOP_SIZE) debussy_spec.shape ``` This gives: (1025, 1292) ~ (#frequency bins, #frames) 1. [According to the given formula] #frames = ((number_of_samples - frame_size) / hop_length) + 1 = ((661500 - 2048) / 512) + 1 = 1288.9921875 = 1289 2. [In reality] #frames = number_of_samples / hop_length = 661500 / 512 = 1291.9921875 = 1292 I feel like, 2 is more intuitive too. We have X number of samples. In each iteration of calculation we move over 512 samples to the right. How many times do we get to do that? Samples/512. Anyone knows where I am going wrong?
Hi @Valerio, this is great stuff. Do you have any recommendations for a book to soak all this in ? These would need intense focus and going multiple back and forth on the videos. Thanks.
Thank you for the video! If you set 'frame size' bigger than the 'window size', does that increase frequency resolution while keeping the time resolution the same (but at more computational cost)?
Thanks for a great series of videos! However, I have a question regarding the resolution trade-off between time and frequency. The time part makes sense, but I do not understand why the frequency resolution depends on the frame size. Obviously something I am missing here, but in my head the frequency range we can represent does not rely on the number of samples, but the sampling rate, ref. the Nyquist sampling theorem. Lets say we have a signal with a sampling rate of 10 samples/sec and we choose a frame size corresponding to 1 second of signal, i.e. 10 samples. Then we can represent frequencies up to max. 5 Hz. If we double the frame size to 2 seconds of signal we now have 20 samples instead of 10. However, the sample rate is still fixed at 10 samples/sec and hence we can still only represent frequencies up to 5Hz...? Would appreciate if you (or anyone else) could explain this.
I have the same doubt! I think @22:00 he explained that the frequency bins gives the information about the frequencies present in the (0,Sr/2) range equidistant from each other. Bigger the frame size more will be the freq-bins -> more detailed Information about freq when frame size becomes the whole wave -> it will become dft and we will have N samples in the range (0,Sr/2).
Generally what is the reason for introducing 2 parameters such that one of them reduces the need for the other? I am talking about window size and frame size, in case they are not equal. We could achieve any uniform split just by varying the window size, couldn't we? Just by watching the video, me personally sees no reason for having frame_size != window_size. Of course, I am missing something out, but what?
For me, the 2 parameters are conceptually different. Frame size is a param is STFT and window size is a param in window function. Pragmatically, it is pointless to assign frame_size != window_size because those samples in between the gap are going to be 0 padding anyways.
Hey, this is an amazing video. Thanks. I have a question though. Through spectrogram, we know the magnitude and phase of the signal at a given time. So is it possible to reconstruct the signal from that domain?
@@ValerioVelardoTheSoundofAI But spectrogram is the abs of STFT right? So will we need STFT for the reconstruction or just the spectrogram plot will be enough?
@@ValerioVelardoTheSoundofAI but if we take spectogram of STFT, then certain frequencies won't show in the spectogram plot till such time our window doesn't process that, right? So we kinda get some information about the phase of the signal.
I cant believe you have only 9k views with this level and clarity of teaching!
Thank you Pritam!
Fantastic video. Much more helpful than any of the other STFT videos on youtube. Thanks a lot!
Thank you!
I wanted to know what mel spectrograms are but then I watched the first video of the series and now I'm learning a far better-stepped approach to Audio Signal Processing and I'm loving it!
Same!
Us
Just would thought i would let you know that i am about to finish my thesis on Dolphin vocalization feature extraction and distinction using ML classifiers and just found your videos. Very easy to understand and visualize concepts that took me more time that a wished to understand. Keep up the good work
Amazing video. Made complex concepts look so simple
Thank you so much, most detailed and clear explanation!
Give this guy a noble prize
Thank you so much for this its not only the content and the educational approach is also your style that keeps the interest high
The concept of the time/frequency trade off in the STFT is greatly introduced!
Thank you so much for all these knowledge sharing. These are one of the best video series I have watched in you-tube!
To avoid confusion it will be better to choose another variable (for example L for sfft) instead of N (for fft)
Fantastic video! It's engaging all the way through and a wonderfully clear explanation of everything.
You were born to explain, thank you
Great job on simplifying such a complex topic. Thanks!
Ver comprehensive. Your body language is also awesome! Thanks so much Prof.
@20:00 How come we don't use a similar definition of frequency bins for DFT (where the frame size is the size of the whole signal ofcourse)?
This tutorial is really helpful! Please keep making contents for us.
Though I have said that for a couple of times, but still thank you for all this passionate series!
Could you please make a video on MFCC?
That is planned over the next few weeks!
@@ValerioVelardoTheSoundofAI waiting for it. And thank you for the whole series!
Thank your for your great work !
At 23:00 there is something I don't understand : you divide by two the frame size, because of the nyquist rule. You obtain 501. But the 501 frequency bins are for...the interval (0 , sampling rate/2).
So, in the end, you divide by 4 ! The sampling rate is divided by 2, THEN the framesize is also divided by 2.
Could you explain this,? Thank you.
I wish there was Sound Of AI Discord community! Thanks for these videos a lot!
There's a TSOAI Slack community though ;)
Cool! This video really helped me for my signal processing subject
I love the bit; "the k-th frequency at the end temporal uh! bin or n-th frame." 12:18
Wow! I love this series!
Thank you!
Wow, what a great explanation, this has really answered a lot of questions I had about how FFT would work on longer samples. Grazie mille, è stato un piacere guardare il video..
Thank you for your kind words Michael!
Thanks a lot for the wonderful explanation. I very first started to learn abt MFCC later, am interested in watching all the videos related to the audio signal processing series.
Am I correct in saying that multiplying the signal for a rectangular windowing function makes no sense, since you are only taking into consideration the samples of each frame and not the rest? It makes sense if the function is not a rectangle, and you have a curve at the edges.
Amazing! Love your work and engagement, thank you!
Thank you!
3 years late, but I love the video
Thanks for the fantastic video. One quick question. What are the advantage and disadvantage for setting frame_size > window_size ? What is the use case for this parameter choice ?
lovin your videos man keep up the good work :)
When you multiply your signal with the window function in the time domain you are convolving the frequency response of the window function with your signal in the frequency domain. The frequency response of most window functions is some form of a sinc function. Sinc functions are long and ringy, so the result of the convolution is to smear out the frequency response of your output. This reduces the accuracy of the output of the STFT. There are other spectral decomposition algorithms that produce more accurate results. The STFT is popular because of it's ease of computation, not because of it's accuracy.
Thank you for your great video. It really helped me understand STFT. I subscribed your channel and will eagerly waiting for your videos.
beautifully explained. Thank you
Very informative and well-presented. Thank you!
Awesome explanation, thanks !!!
you used a phrase called 'pure term' at 17:03, what do you mean by that?
He said "pure tone"
Yes this is the dream scenario emerging! - So I am a bit slow but I think I could use e Spectrograms as the input feature layer directly into a convolutional networks or LSTM networks.
Thank you for the video!
Hi Valerio, may I notice that the calculation for #frames at 22:22 should return 39 and not 19?
#frames = (10000 - 1000)/500 + 1 = 39
His calculation is correct: (10000 - 1000)/500 + 1 = 19 .🙂
@@ganmohim4273 Ah, yes. Don't know what I was thinking. Apologies
Great job!
Hey man! Your fourier stuff is great! I've been playing with audio a bit and found that tensorflow has an awesome method for getting real-valued spectrograms using a method called MDCT (modified discrete cosine transform). Have you ever considered making a video about that? I'm curious to know how that's different from STFT
Awesome explanation !
Great work. Thanks you so much
Is the frequency bin parameter in STFT (in the example is 501 bins) means for each frame in the signal?
The "temporal" information isn't contained in the DFT phase?, because the DFT is a linear transform, there is not any lose of information, applying the inverse DFT we can get back our signal with it's temporal distribution
Thank you for sharing!!!
a Wonderful video, we need to know more about speech processing and especially the mirror process of inverting spectrograms and STFT, could you please suggest me where i can find explanation about the Griffin & Lim algorithm.
hello, please if you can provide us the process of the inverse short time fourier transform for overlapped frames, in order to recover the original time domain signal.
Very cool lecture !!
The video was great, thank you. Do you have any instructional videos explaining zero padding?
I've used zero-padding here and there in some videos, but never dedicated a full video to it only.
@@ValerioVelardoTheSoundofAI Thank you for your quick response. Actually, I'm a master's student in audio signal processing and your videos helped me a lot to do my master's project. Thanks an ocean :)
Great Video
You should write your own book, I'd definitely buy it
This is great, thanks a lot!
Just brilliant once again
Thanks Dario!
@@ValerioVelardoTheSoundofAI I hope you can help me with this doubt. Is the Hann Window applied after STFT to each frequency bin or before the STFT to each sample within the frame? I got a bit confused since in the STFT formula you have used w(n) and on the Hann window formula w(k). Could you please clarify this? Thanks once again for your amazing videos.
Dont you know why when using librosa stft function in python, the resulting number of temporal bins does not equal to math.ceil of (num_of_samples - frame_size) / hop_length + 1, but rather to math.ceil of (num_of_samples) / hop_length? I am processing some audio files and calculating stft for 66150 samples of signal as input to the function, using 2048 window and 512 samples as hop size... So in theory, I should get (66150 - 2048) / 512 + 1 = 126,199... ~127 temporal bins... But rather than that, the temporal output shape of stft does have 130 elements.. How is it calculating the last few windows, for which the function should not actually have enough samples available, as they are out of provided signal range?
I encountered the same thing
I encountered the same thing using Librosa.
```
number_of_samples = 661500
FRAME_SIZE = 2048
HOP_LENGTH = 512
debussy_spec = librosa.stft(debussy, n_fft=FRAME_SIZE, hop_length=HOP_SIZE)
debussy_spec.shape
```
This gives: (1025, 1292) ~ (#frequency bins, #frames)
1. [According to the given formula] #frames = ((number_of_samples - frame_size) / hop_length) + 1 = ((661500 - 2048) / 512) + 1 = 1288.9921875 = 1289
2. [In reality] #frames = number_of_samples / hop_length = 661500 / 512 = 1291.9921875 = 1292
I feel like, 2 is more intuitive too. We have X number of samples. In each iteration of calculation we move over 512 samples to the right. How many times do we get to do that? Samples/512. Anyone knows where I am going wrong?
Hop size and frame size are like stride and kernel size in CNNs.
Hi @Valerio, this is great stuff. Do you have any recommendations for a book to soak all this in ? These would need intense focus and going multiple back and forth on the videos. Thanks.
Wonderful video!!
Thank you for the video! If you set 'frame size' bigger than the 'window size', does that increase frequency resolution while keeping the time resolution the same (but at more computational cost)?
You are awesome! I love you!
That was a greate video sir. I have got just one question :what happens if we use a very swall hop length for instance 2 or 4?
You would have a lot of redundancy in the data and a greater memory footprint.
beautiful!
Thanks for a great series of videos!
However, I have a question regarding the resolution trade-off between time and frequency.
The time part makes sense, but I do not understand why the frequency resolution depends on the frame size.
Obviously something I am missing here, but in my head the frequency range we can represent does not rely on the number of samples, but the sampling rate, ref. the Nyquist sampling theorem.
Lets say we have a signal with a sampling rate of 10 samples/sec and we choose a frame size corresponding to 1 second of signal, i.e. 10 samples. Then we can represent frequencies up to max. 5 Hz. If we double the frame size to 2 seconds of signal we now have 20 samples instead of 10. However, the sample rate is still fixed at 10 samples/sec and hence we can still only represent frequencies up to 5Hz...?
Would appreciate if you (or anyone else) could explain this.
I have the same doubt!
I think @22:00 he explained that the frequency bins gives the information about the frequencies present in the (0,Sr/2) range equidistant from each other.
Bigger the frame size more will be the freq-bins -> more detailed Information about freq
when frame size becomes the whole wave -> it will become dft and we will have N samples in the range (0,Sr/2).
Thanks Master
Please call me Valerio :)
Thanks!
I still don't get the reason for the +1 fft point, though.
For the 0th bin
you are a legend!!!!!!!!
wonderful!!
what is the k and K in Hann window formula??
sample number and number of samples in window
Generally what is the reason for introducing 2 parameters such that one of them reduces the need for the other? I am talking about window size and frame size, in case they are not equal. We could achieve any uniform split just by varying the window size, couldn't we? Just by watching the video, me personally sees no reason for having frame_size != window_size. Of course, I am missing something out, but what?
For me, the 2 parameters are conceptually different. Frame size is a param is STFT and window size is a param in window function. Pragmatically, it is pointless to assign frame_size != window_size because those samples in between the gap are going to be 0 padding anyways.
@@ericchuhaochan2066 Agree with you now.
Hey, this is an amazing video. Thanks. I have a question though. Through spectrogram, we know the magnitude and phase of the signal at a given time. So is it possible to reconstruct the signal from that domain?
Yes, for reconstruction from a complex spectrogram, you would use the inverse short-time Fourier transform.
@@ValerioVelardoTheSoundofAI But spectrogram is the abs of STFT right? So will we need STFT for the reconstruction or just the spectrogram plot will be enough?
@@sarvagyagupta1744 no, the abs of STFT is the magnitude spectrogram, which loses its imaginary part. STFT is complex.
@@ValerioVelardoTheSoundofAI but if we take spectogram of STFT, then certain frequencies won't show in the spectogram plot till such time our window doesn't process that, right? So we kinda get some information about the phase of the signal.
thks valerio
amazing
Thanks is not enough!
Thank you for this beautiful video
Fantastic explanation!
Great Video, Thank you
great Video, thank you again