It's a good practice to have most of the code in python files - benefiting from the IDE of your choice, and separate reporting to notebooks focused on presentation. The mix of code (functionality) vs reporting (visualisation) depends on the data science project.
This video and workflow is great, however you're also right about that. It depends what you're doing. I will be looking forward to trying to adapt to this workflow in the future. Thanks, the both of ya
This makes so much sense! I am only just beginning my python journey and am loving your videos. Finding information like this so early on for me seems invaluable to build better habits now with regards to best practices and efficient workflows
I used to use jupyter notebooks for everything back in college as well (even stats/math homework). Switched to using neovim and the ipython shell in the terminal, once you learn neovim using the terminal becomes so much faster than jupyter in vscode. Also you can use kitty terminal to display plots in the terminal and termpandas to have scrollable dataframes like in vscode jupyter, all in the terminal. (i3 window manager/tmux also make the workflow so much faster and enjoyable.)
@@KamKamKamKam yes kitty terminal allows for displaying images directly in the terminal its pretty cool (save to png and then use os.system("kitty icat fig.png"), python package termpandas (I created it) allows scrollable data frames in the terminal, and ipython shell allows for the whole REPL experience in python. All inside the terminal. The only thing I have not been able to get to work have been interactive html plots in the terminal like plotly express. However, the best data science experience for this in my opinion is split screen neovim and having a streamlit dashboard that automatically updates on save on the other side of the screen. This allows for moving super fast in neovim on the left side of the screen (also using i3 window manager to move around) and a browser with a streamlit app so I can have fully interactive dataframes, plots, filters, and basically any web technology in the streamlit app. Let me know if you are more interested in my setup, I could probably write a medium blog or something explaining my setup, and listing the list of tools/packages I use.
While I am about to get into Marimo, I also found your tips about interactive Python along side code very interesting and something I've been missing out on big time.
You can also use #%% comments to mark sections of your code as "cells" that would be executed as a whole with the keybinding. I'm not sure if this "run cell" does not conflict with "run line/selection" on some platforms, but you can fix that in the config.
I couldn't agree with you more, Dave. Thanks for your informative video on how to use my favorite IDE VS Code with Jupyter Notebooks. This combination seems to be the best duet for data scientific research.
Definitely, I prefer your workflow over Jupyter Notebook. I am in the phase of testing Cursor, which is similar to VS Code but has some more AI features.
Great tip, might migrate completely to this workflow. Jupyter Notebooks are great, but as you say, very clunky when approaching as a software engineer.
In matlab you are able to define a code block with a ‘##’ that once you are inside it you can shift-enter and it will run. You can also have a variable explorer window. I wonder if there is such functionality in VSCode?
Hi Dave - it is very very difficult to follow you when you keep moving your mouse all over the place at speed. Trying to figure out how to this works, but man, just move your mouse to where you are explaining and leave it there!
Python Smart Execute. I have no direct experience using it, but it apparently runs code blocks in a console window using a Jupyter cell behind the scenes. Requires the Jupyter plugin to be installed.
I completely moved to google colab to save RAM and space being overused by Jupyter, only while complex coding do i use vscode else, even docker images are my option
Isn't there an .ipy file type or something. There like Python built-in interactive file type, i think. And you can click one segment at a time. I think it was the basis of .ipynb
I'm curious if this style of coding creates any lingering limitations or style / organization choices in your "pure" python files due to the holdover of executing "cells" of code during the development cycle. Stated differently, do you find that any production-ready refactoring is necessary after the functionality is ironed out using Jupyter-esque interactive coding?
You can start Jupyter Server in your local machine and connect Colab to a local environment, awesome ! But doesn’t solve the Jupyter notebook challenges, but it’s a much better IDE than VS Code with some many "Microsoft UX” that forces you to adapt in a non efficient way ,plugins and environment challenges…. Modularizing the code and using GIT helps a lot in my experience .
Mate, thanks a lot for this. I never liked Jupiter notebooks and I’m glad I’m not the only one haha. Either way, how would you send your structured codes in different python files to cloud computing if you wanted to use that
Jupyter Notebooks are only bad if its the only thing you use. For exploration and prototyping its king. Also VS Code sucks still big time when it comes to NBs. So many things not showing properly or kernels not restarting properly, not releasing memory when training torch models. I use neovim for coding and jupyter lab for exploration. They serve two very different stages of my projects
There’s something I’ve been struggling to understand. Why do so many indie devs, especially those not using windows, continue to use vs code? Unless you’re living under a rock, and even a dev living under a rock would know this, it’s pretty much a given that whatever you type is being shipped off via telemetry, and that then opens up the risk of people with way more resources at their disposal, to simply copy, iterate quickly on whatever you’re building, and bring it to market before you’ve even had a chance to test. What gives??
It's a good practice to have most of the code in python files - benefiting from the IDE of your choice, and separate reporting to notebooks focused on presentation. The mix of code (functionality) vs reporting (visualisation) depends on the data science project.
This video and workflow is great, however you're also right about that. It depends what you're doing.
I will be looking forward to trying to adapt to this workflow in the future.
Thanks, the both of ya
This makes so much sense! I am only just beginning my python journey and am loving your videos. Finding information like this so early on for me seems invaluable to build better habits now with regards to best practices and efficient workflows
i started with R, and i'm having a hard time to get used to notebook workflow so this basically helped a lot~! thanks~!
I used to use jupyter notebooks for everything back in college as well (even stats/math homework). Switched to using neovim and the ipython shell in the terminal, once you learn neovim using the terminal becomes so much faster than jupyter in vscode. Also you can use kitty terminal to display plots in the terminal and termpandas to have scrollable dataframes like in vscode jupyter, all in the terminal. (i3 window manager/tmux also make the workflow so much faster and enjoyable.)
Can it display images directly (using matplotlib or something)?
What plugins are needed?
I'm also trying to move my jupyter work flow into neovim.
@@KamKamKamKam yes kitty terminal allows for displaying images directly in the terminal its pretty cool (save to png and then use os.system("kitty icat fig.png"), python package termpandas (I created it) allows scrollable data frames in the terminal, and ipython shell allows for the whole REPL experience in python. All inside the terminal. The only thing I have not been able to get to work have been interactive html plots in the terminal like plotly express. However, the best data science experience for this in my opinion is split screen neovim and having a streamlit dashboard that automatically updates on save on the other side of the screen. This allows for moving super fast in neovim on the left side of the screen (also using i3 window manager to move around) and a browser with a streamlit app so I can have fully interactive dataframes, plots, filters, and basically any web technology in the streamlit app. Let me know if you are more interested in my setup, I could probably write a medium blog or something explaining my setup, and listing the list of tools/packages I use.
While I am about to get into Marimo, I also found your tips about interactive Python along side code very interesting and something I've been missing out on big time.
I had same problems!😅 This will help me so much!🎉
This is very true. I love the idea of Jupiter notebooks, and I've used them for years, but they not composable.
Jupyter notebook can actually work with AI assistant like codeium same way you do in VSCode.
Codeium is buggy in Jupyter notebook
You can also use #%% comments to mark sections of your code as "cells" that would be executed as a whole with the keybinding.
I'm not sure if this "run cell" does not conflict with "run line/selection" on some platforms, but you can fix that in the config.
Thanks! For your info, I just dropped pycharm+jupyter, switch to vs now!
I couldn't agree with you more, Dave. Thanks for your informative video on how to use my favorite IDE VS Code with Jupyter Notebooks. This combination seems to be the best duet for data scientific research.
This methid is like using R. Thanks You! Now I can learn programming in Python sane as in R. Excellent trick ! 👏👏
really good. thanks so much for teaching this feature I didnt know before. I like this more.
Thanks!
Thank you! 🙏🏻
Thank you for sharing this is the type of thing a seasoned developer needs to hear.
Definitely, I prefer your workflow over Jupyter Notebook. I am in the phase of testing Cursor, which is similar to VS Code but has some more AI features.
How do you like it so far?
Great tip, might migrate completely to this workflow. Jupyter Notebooks are great, but as you say, very clunky when approaching as a software engineer.
It depends on where you are on the learning curve. I personally love Jupyter notebooks.
Thank you , I'm following your suggestions
this video is for the people who are already expert in data analysis. letting ya'll new analyst like me before you start watching the video.
In matlab you are able to define a code block with a ‘##’ that once you are inside it you can shift-enter and it will run. You can also have a variable explorer window. I wonder if there is such functionality in VSCode?
Hi Dave - it is very very difficult to follow you when you keep moving your mouse all over the place at speed. Trying to figure out how to this works, but man, just move your mouse to where you are explaining and leave it there!
Cool. Is there anything simillar in PyCharm ?
Python Smart Execute. I have no direct experience using it, but it apparently runs code blocks in a console window using a Jupyter cell behind the scenes. Requires the Jupyter plugin to be installed.
I completely moved to google colab to save RAM and space being overused by Jupyter, only while complex coding do i use vscode else, even docker images are my option
If you use VS Code then you need to use your existing GPU/CPU but with collab you can use TPU AND GPU free
Isn't there an .ipy file type or something. There like Python built-in interactive file type, i think. And you can click one segment at a time. I think it was the basis of .ipynb
3:37 cannot find it in there.
I'm curious if this style of coding creates any lingering limitations or style / organization choices in your "pure" python files due to the holdover of executing "cells" of code during the development cycle. Stated differently, do you find that any production-ready refactoring is necessary after the functionality is ironed out using Jupyter-esque interactive coding?
Hi Dave, what VsTheme(s) are you using ? looking very clean. Like always, appreciate the sharing of your knowledge.
i would recommend the google collab but only the runtime problem with free version
You can start Jupyter Server in your local machine and connect Colab to a local environment, awesome ! But doesn’t solve the Jupyter notebook challenges, but it’s a much better IDE than VS Code with some many "Microsoft UX” that forces you to adapt in a non efficient way ,plugins and environment challenges…. Modularizing the code and using GIT helps a lot in my experience .
I'm assuming you don't do any heavy pytorch stuff where you need a GPU? Or do you just run them on your mac?
I typically use Azure for that, but I don't often have to run very heavy models.
In Jupyter, why can I still not cut and paste text in a cell using the mouse?
One can use marimo instead of juoyter, or one can even make Code use a notebook. I think the presenter did the latter, but not marimo.
Is there anything same in nvim? I'm looking for a plugin like this
Mate, thanks a lot for this. I never liked Jupiter notebooks and I’m glad I’m not the only one haha. Either way, how would you send your structured codes in different python files to cloud computing if you wanted to use that
would this work with a deno kernel in a jupyter notebook too?
Thank you!
Jupyter Notebooks are only bad if its the only thing you use. For exploration and prototyping its king. Also VS Code sucks still big time when it comes to NBs. So many things not showing properly or kernels not restarting properly, not releasing memory when training torch models. I use neovim for coding and jupyter lab for exploration. They serve two very different stages of my projects
That is why there is Jupyter lab, all the issues u are talking about has been addressed in jupyter lab
This is amazing.
I don't get it 😅
There’s something I’ve been struggling to understand. Why do so many indie devs, especially those not using windows, continue to use vs code? Unless you’re living under a rock, and even a dev living under a rock would know this, it’s pretty much a given that whatever you type is being shipped off via telemetry, and that then opens up the risk of people with way more resources at their disposal, to simply copy, iterate quickly on whatever you’re building, and bring it to market before you’ve even had a chance to test. What gives??
what to use instead ?
Agreed. It's the only way I work.
i am so switching already
great tip
notebooks are amazing for DE.
Thanks for this fidio
@dave did you try aider with your workflow. I think that could work for us data people working with vs code supported by ai
nbdev is all you need
Rstudio FTW
we don't give a fuck.
Why are you here, Sir?
nah