Real time style transfer and latent manipulation with breaks: neural audio with RAVE and nn~ in PD

In this video, I'm feeding a breakbeat sample to the encoder unit of RAVE, play around with additional signal sources and manipulation techniques in latent space and output through the decoder unit of the same model.
The RAVE model used in this video has been trained on a selection of my tracks. Realtime processing is done via the nn~ object in Pure Data (PD).
RAVE is "A variational autoencoder for fast and high-quality neural audio synthesis” created by Antoine Caillon and Philippe Esling of Artificial Creative Intelligence and Data Science (ACIDS) at IRCAM, Paris.
RAVE on GitHub: github.com/aci...
nn~ on GitHub: github.com/aci...
To train RAVE models on Colab or Kaggle, you can use these Jupyter notebooks i've set up: github.com/dev...

[@keiwee95]

anywhere to start learning how to do this? craziest thing ive ever seen..

[@martsm_n]

Thanks! Not half as crazy as it might look. If you want to start with Pure Data, I can recommend @QCGInteractiveMusic amongst others. The RAVE part is not thoroughly covered here on TH-cam. You might want to dig your way through the repo on GitHub or have a look at these notebooks for starters: github.com/devstermarts/notebooks

[@tsdoihasdoihasdoih2493]

what's in the patcher at the very end of the chain? :o

[@martsm_n]

You mean the box on top of the master out abstraction? It's a basic stereo delay unit with a few extras. I'm using short delay buffers here to glitch things up a bit. An earlier version is in my components repo (still need to change a few bits on this one): github.com/devstermarts/PD-components/blob/main/various/glitzher.pd

[@tsdoihasdoihasdoih2493]

@@martsm_n nice! biased by my own workflow I assumed it's some abstraction on top of nn~ with stereo delay feedback going back into the encoder :p

[@martsm_n]

Well that again sparks a bunch of new ideas, thanks! :)
In my patch, right below the decoder I'm using the metro object to generate bangs in the same BPM as the break. These bangs are used to trigger the delay lines as well as a noise seed (top end of the patch) that goes directly into the decoder latents again.