Dear creators, I really really love your work. Your videos save my student life at very close death situation. Wish I known your channel few months earlier. Keep up your great work! Honestly thank you!!!
Thanks! Happy to hear it. I'm actually working on a new batch of videos, with much-improved audio and visual recording quality. Check back to my channel in January!
You've really knocked it out of the park with these videos - all of the information I'm looking for, none of the fluff 👌 (Will be recommending to my colleagues)
Hi. That's difficult to compute exactly. Higher order means longer filter kernel, which gives a better spectral response. But longer filter kernels eventually become unstable and also increase computation time. One thing you can do is set the order to be some multiple of one cycle of the lower frequency. For example, if the lower frequency bound is 10 Hz, then you can have a filter order of 10*100 = 1000 (thus, 10 times 100 ms, assuming a 1 kHz sampling rate). Regardless, you should always inspect the time- and frequency-domain filter kernel representations. I have a course on signal processing where I talk about designing filters more. You can find it linked from sincxpress.com
Let's say your analyzing time series data, and you convert the data from the time domain to the frequency domain, with intentions of looking for spikes in the frequency domain because you determined that those spikes will reveal relevant features about your data. However, you don't necessarily know at what specific frequencies to expect those spikes at. Would you recommend using the Morlet wave form, or this narrow band method?
Good question, and it's related to signal stationarity. If you expect the main spectral features of the signal to remain roughly consistent over time, then the best and easiest analysis is an FFT and extract power (in MATLAB that would be abs(fft(signal)) ). That will decompose the signal over many frequencies, so you don't need to know the frequencies a priori. Time-frequency analysis methods like wavelet convolution or filter-Hilbert are advantageous when the spectral features change over time (thus, when the signal is nonstationary). It's useful here to have some ideas of the frequencies of interest, mainly to focus the analyses on those frequencies.
Mike, thanks for getting back to me. I've spent close to 15 hours over the past 4 days watching and re-watching your youtube play list on these topics. From your experience, could you recommend any signal processing/Matlab approaches to attenuating noisy frequency domain data? So for instance, after converting my time domain signal to the frequency domain signal, the frequency spikes that i am looking for aren't readily noticeable because they do not "spike" up higher than the rest of the noise spikes in the data. For certain cases, where I do know what frequency range to expect the spike, I can zoom in super close to the signal at that particular frequency, and then see a frequency spike. However its impractical to zoom in this close and scan the entire length of the signal for these spikes. Also, does the fact that my time domain signal is non-stationary hamper the ease with which I can notice these frequency spikes? (Would the spikes be more apparent if the time series data was stationary)? Intuitively, I feel like that is the real issue at hand here. I have read a few papers where the authors have sampled their non-stationary time domain signals in a stationary like fashion to help get around this issue. Thanks in advance.
Simple visual interpretation of the results of an FFT is valid only for stationary data. I talk about that more in a video on nonstationarities (must be in the 16th hour of the lectures ;) ). If you know the frequency at which the effects should occur, then it might be best to apply a narrowband filter, like a wavelet or an FIR filter. If you want to see the frequency-domain representation of a nonstationary signal, then you could apply a time-frequency analysis and then sum the coefficients over time.
Mike, would it be at all possible for me to share my data with you and get your feedback (help) in trying to perform these exact same signal processing techniques on my data. I'm trying to publish a paper by the end of this summer. Any help that you might be able to provide would be a huge help. I'll even throw in an acknowledgement and reference to your work if I can get it published. Thanks a lot for all your help so far.
Fantastic series!. One thing you mentioned about the filtfilt for FIR filters and morlet convolution: both involve convolution to generate band pass filtered signals, except that the morlet is even around 0 and thus by construction has zero phase delay ( a non causal filter). I guess one could use the filtfilt function when using an odd morlet that is not symmetric around 0 to get the correct response?
Hi Nikhilesh. I'm not sure what an "odd Morlet" would be -- the definition is a sine wave tapered by a Gaussian, which is by definition symmetric. But in principle, yes, you could use a wavelet as the filter kernel in filtfilt. However, I'm not sure what the advantage would be. The great thing about MATLAB (or python...) is that you can try all of these things and see what the results are. I think you'll find that the results are nearly identical, with some differences at the edges of the time series. Mike
Hey Mike, in this lecture you mentioned that its not good to create FIR band-pass filters with sharp edges because they can create artifacts in the time domain reconstructions. However, what if I am only interested in doing frequency analysis, and don't plan on converting or analyzing the band-pass signal back in the time domain. Are sharp edges ok then, will they have any adverse effects in the frequency domain?
Hi Brit. Nice to hear from you again ;) Indeed, edges in the frequency domain are only potentially negative for time-domain filtering. If you will stay in the frequency domain, you don't have to worry about it.
Yeah, sorry about that :P These videos were my first attempt at doing home-recordings. A bit embarrassing ;) You can check out the "new ANTS" series for waaaaaayyyy improved videos.
lol, these videos are the first recordings I made in 2015. The content is fine but the quality is crap. I often think about deleting them, but people tell me that they still benefit from the content. So yeah. Anyway, feel free to check out my newer (and much higher quality) videos!
That could work. It's important for the signal to be narrowband, otherwise the output of the Hilbert transform isn't easily interpretable. So it should be a tight low-pass filter, I'd say. You'd also want to remove the DC (0 Hz component).
I'm sorry, just had to throw out my frustration somewhere. I have mixed signal of heartbeat, breathing and other stuff in a file and I have to filter out heartbeat, can you help me what the right approach would be? The signal is periodic, but not with mathematical precision, how to model it or should I try to model it at all? how to build the software filter?
Heartbeats are tricky, because they do not have a sinusoidal shape, and because the heartbeat features change slightly from beat to beat. That means that they don't have a clear representation in the spectral domain that is easily filtered out. If you have a multichannel acquisition, you might consider using a spatial filter instead of a temporal filter. ICA might be a good solution. ICA of multichannel EEG or MEG data will usually isolate a heartbeat component, and you could then project out that component. If the removal needs to be done via temporal filter, you might consider a template-removal approach. This would involve building a template heartbeat based on averaging many QRS peaks. Then run a cross-correlation of that template with the data, and regress the template out of any peaks in the cross-correlation. There will probably still be some residual heartbeat artifact, but I guess this approach would eliminate most of it. Mike
Thanks for the tips. Is using numerical Fourier' transform an option to look? I imagine somehow numerically transform the signal into frequency domain, eliminating unwanted bands in frequency domain and transforming result back to time domain?
Dear creators, I really really love your work. Your videos save my student life at very close death situation. Wish I known your channel few months earlier.
Keep up your great work!
Honestly thank you!!!
Thanks! Happy to hear it. I'm actually working on a new batch of videos, with much-improved audio and visual recording quality. Check back to my channel in January!
Mike X Cohen Absolutely will.
Congratulations!!!
You've really knocked it out of the park with these videos - all of the information I'm looking for, none of the fluff 👌 (Will be recommending to my colleagues)
Thanks!
Love the lectures Mike! I have just started working with neuroscience and your book and lectures have proved amazingly helpful.
Amazing. Thanks for the perfect clarity explaining such complicated concepts.
You're welcome!
Such a great series of lectures. Thank you.
Hello. Would you please explain how to determine the best filter order? Thanks!
Hi. That's difficult to compute exactly. Higher order means longer filter kernel, which gives a better spectral response. But longer filter kernels eventually become unstable and also increase computation time. One thing you can do is set the order to be some multiple of one cycle of the lower frequency. For example, if the lower frequency bound is 10 Hz, then you can have a filter order of 10*100 = 1000 (thus, 10 times 100 ms, assuming a 1 kHz sampling rate).
Regardless, you should always inspect the time- and frequency-domain filter kernel representations.
I have a course on signal processing where I talk about designing filters more. You can find it linked from sincxpress.com
Thanks Mike.
Let's say your analyzing time series data, and you convert the data from the time domain to the frequency domain, with intentions of looking for spikes in the frequency domain because you determined that those spikes will reveal relevant features about your data. However, you don't necessarily know at what specific frequencies to expect those spikes at. Would you recommend using the Morlet wave form, or this narrow band method?
Good question, and it's related to signal stationarity. If you expect the main spectral features of the signal to remain roughly consistent over time, then the best and easiest analysis is an FFT and extract power (in MATLAB that would be abs(fft(signal)) ). That will decompose the signal over many frequencies, so you don't need to know the frequencies a priori.
Time-frequency analysis methods like wavelet convolution or filter-Hilbert are advantageous when the spectral features change over time (thus, when the signal is nonstationary). It's useful here to have some ideas of the frequencies of interest, mainly to focus the analyses on those frequencies.
Mike, thanks for getting back to me. I've spent close to 15 hours over the past 4 days watching and re-watching your youtube play list on these topics. From your experience, could you recommend any signal processing/Matlab approaches to attenuating noisy frequency domain data? So for instance, after converting my time domain signal to the frequency domain signal, the frequency spikes that i am looking for aren't readily noticeable because they do not "spike" up higher than the rest of the noise spikes in the data. For certain cases, where I do know what frequency range to expect the spike, I can zoom in super close to the signal at that particular frequency, and then see a frequency spike. However its impractical to zoom in this close and scan the entire length of the signal for these spikes.
Also, does the fact that my time domain signal is non-stationary hamper the ease with which I can notice these frequency spikes? (Would the spikes be more apparent if the time series data was stationary)? Intuitively, I feel like that is the real issue at hand here. I have read a few papers where the authors have sampled their non-stationary time domain signals in a stationary like fashion to help get around this issue.
Thanks in advance.
Simple visual interpretation of the results of an FFT is valid only for stationary data. I talk about that more in a video on nonstationarities (must be in the 16th hour of the lectures ;) ). If you know the frequency at which the effects should occur, then it might be best to apply a narrowband filter, like a wavelet or an FIR filter. If you want to see the frequency-domain representation of a nonstationary signal, then you could apply a time-frequency analysis and then sum the coefficients over time.
Mike, would it be at all possible for me to share my data with you and get your feedback (help) in trying to perform these exact same signal processing techniques on my data. I'm trying to publish a paper by the end of this summer. Any help that you might be able to provide would be a huge help. I'll even throw in an acknowledgement and reference to your work if I can get it published. Thanks a lot for all your help so far.
Sure, happy to help. Send me an email, ideally with some screenshots or perhaps some of your code.
Dear Mike, do you know any methods in extracting the relevant IMFs having useful signals from the decomposed signal with the help of EMD/EEMD/CEEMD.
Fantastic series!. One thing you mentioned about the filtfilt for FIR filters and morlet convolution: both involve convolution to generate band pass filtered signals, except that the morlet is even around 0 and thus by construction has zero phase delay ( a non causal filter). I guess one could use the filtfilt function when using an odd morlet that is not symmetric around 0 to get the correct response?
Hi Nikhilesh. I'm not sure what an "odd Morlet" would be -- the definition is a sine wave tapered by a Gaussian, which is by definition symmetric. But in principle, yes, you could use a wavelet as the filter kernel in filtfilt. However, I'm not sure what the advantage would be.
The great thing about MATLAB (or python...) is that you can try all of these things and see what the results are. I think you'll find that the results are nearly identical, with some differences at the edges of the time series.
Mike
Hey Mike, in this lecture you mentioned that its not good to create FIR band-pass filters with sharp edges because they can create artifacts in the time domain reconstructions. However, what if I am only interested in doing frequency analysis, and don't plan on converting or analyzing the band-pass signal back in the time domain. Are sharp edges ok then, will they have any adverse effects in the frequency domain?
Hi Brit. Nice to hear from you again ;)
Indeed, edges in the frequency domain are only potentially negative for time-domain filtering. If you will stay in the frequency domain, you don't have to worry about it.
Is someone using a centrifuge in the background? :-)
i think its his washing machine :D
Yeah, sorry about that :P These videos were my first attempt at doing home-recordings. A bit embarrassing ;) You can check out the "new ANTS" series for waaaaaayyyy improved videos.
I thought he was on a plane. It's very distracting, but the content is great so I'm suffering through.
You could've used some band pass filtering for that fan noise/tea kettle of yours...
lol, these videos are the first recordings I made in 2015. The content is fine but the quality is crap. I often think about deleting them, but people tell me that they still benefit from the content. So yeah. Anyway, feel free to check out my newer (and much higher quality) videos!
is there any specific reason to use bandpass filter or is it okay to filter the signal using a low pass signal before taking the hilbert transform?
That could work. It's important for the signal to be narrowband, otherwise the output of the Hilbert transform isn't easily interpretable. So it should be a tight low-pass filter, I'd say. You'd also want to remove the DC (0 Hz component).
thanks for your helpful video ...
bro is there a bomb about to go off in the background at 14:44? (btw this lecture is FIRE 🔥🔥🔥)
lol, yeah these were my first ever recordings in 2015. The audio is pretty crappy. Sorry about that :(
thank you for the wonderful video :')
Happy to help ;)
best tutorial ever!
Aww, thanks!
I want to understand the principle, but you are showing me some mathlab shit
Thank you for the observation, Jaanus. Indeed, this video focuses more on explanation and implementation in MATLAB, than on theory and proofs.
Mike
I'm sorry, just had to throw out my frustration somewhere. I have mixed signal of heartbeat, breathing and other stuff in a file and I have to filter out heartbeat, can you help me what the right approach would be? The signal is periodic, but not with mathematical precision, how to model it or should I try to model it at all? how to build the software filter?
Heartbeats are tricky, because they do not have a sinusoidal shape, and because the heartbeat features change slightly from beat to beat. That means that they don't have a clear representation in the spectral domain that is easily filtered out. If you have a multichannel acquisition, you might consider using a spatial filter instead of a temporal filter. ICA might be a good solution. ICA of multichannel EEG or MEG data will usually isolate a heartbeat component, and you could then project out that component.
If the removal needs to be done via temporal filter, you might consider a template-removal approach. This would involve building a template heartbeat based on averaging many QRS peaks. Then run a cross-correlation of that template with the data, and regress the template out of any peaks in the cross-correlation. There will probably still be some residual heartbeat artifact, but I guess this approach would eliminate most of it.
Mike
Thanks for the tips. Is using numerical Fourier' transform an option to look? I imagine somehow numerically transform the signal into frequency domain, eliminating unwanted bands in frequency domain and transforming result back to time domain?