It's a good practice to have most of the code in python files - benefiting from the IDE of your choice, and separate reporting to notebooks focused on presentation. The mix of code (functionality) vs reporting (visualisation) depends on the data science project.
I used to use jupyter notebooks for everything back in college as well (even stats/math homework). Switched to using neovim and the ipython shell in the terminal, once you learn neovim using the terminal becomes so much faster than jupyter in vscode. Also you can use kitty terminal to display plots in the terminal and termpandas to have scrollable dataframes like in vscode jupyter, all in the terminal. (i3 window manager/tmux also make the workflow so much faster and enjoyable.)
@@KamKamKamKam yes kitty terminal allows for displaying images directly in the terminal its pretty cool (save to png and then use os.system("kitty icat fig.png"), python package termpandas (I created it) allows scrollable data frames in the terminal, and ipython shell allows for the whole REPL experience in python. All inside the terminal. The only thing I have not been able to get to work have been interactive html plots in the terminal like plotly express. However, the best data science experience for this in my opinion is split screen neovim and having a streamlit dashboard that automatically updates on save on the other side of the screen. This allows for moving super fast in neovim on the left side of the screen (also using i3 window manager to move around) and a browser with a streamlit app so I can have fully interactive dataframes, plots, filters, and basically any web technology in the streamlit app. Let me know if you are more interested in my setup, I could probably write a medium blog or something explaining my setup, and listing the list of tools/packages I use.
This makes so much sense! I am only just beginning my python journey and am loving your videos. Finding information like this so early on for me seems invaluable to build better habits now with regards to best practices and efficient workflows
You can also use #%% comments to mark sections of your code as "cells" that would be executed as a whole with the keybinding. I'm not sure if this "run cell" does not conflict with "run line/selection" on some platforms, but you can fix that in the config.
In matlab you are able to define a code block with a ‘##’ that once you are inside it you can shift-enter and it will run. You can also have a variable explorer window. I wonder if there is such functionality in VSCode?
Python Smart Execute. I have no direct experience using it, but it apparently runs code blocks in a console window using a Jupyter cell behind the scenes. Requires the Jupyter plugin to be installed.
I completely moved to google colab to save RAM and space being overused by Jupyter, only while complex coding do i use vscode else, even docker images are my option
Isn't there an .ipy file type or something. There like Python built-in interactive file type, i think. And you can click one segment at a time. I think it was the basis of .ipynb
I couldn't agree with you more, Dave. Thanks for your informative video on how to use my favorite IDE VS Code with Jupyter Notebooks. This combination seems to be the best duet for data scientific research.
I'm curious if this style of coding creates any lingering limitations or style / organization choices in your "pure" python files due to the holdover of executing "cells" of code during the development cycle. Stated differently, do you find that any production-ready refactoring is necessary after the functionality is ironed out using Jupyter-esque interactive coding?
Definitely, I prefer your workflow over Jupyter Notebook. I am in the phase of testing Cursor, which is similar to VS Code but has some more AI features.
Great tip, might migrate completely to this workflow. Jupyter Notebooks are great, but as you say, very clunky when approaching as a software engineer.
Mate, thanks a lot for this. I never liked Jupiter notebooks and I’m glad I’m not the only one haha. Either way, how would you send your structured codes in different python files to cloud computing if you wanted to use that
You can start Jupyter Server in your local machine and connect Colab to a local environment, awesome ! But doesn’t solve the Jupyter notebook challenges, but it’s a much better IDE than VS Code with some many "Microsoft UX” that forces you to adapt in a non efficient way ,plugins and environment challenges…. Modularizing the code and using GIT helps a lot in my experience .
I am bounded to use jupyter notebooks because of low level processor (AMD Athlon Silver 3050U) which is present in low priced laptops, as I am searching for a job, I had less money hence I bought this laptop.
There’s something I’ve been struggling to understand. Why do so many indie devs, especially those not using windows, continue to use vs code? Unless you’re living under a rock, and even a dev living under a rock would know this, it’s pretty much a given that whatever you type is being shipped off via telemetry, and that then opens up the risk of people with way more resources at their disposal, to simply copy, iterate quickly on whatever you’re building, and bring it to market before you’ve even had a chance to test. What gives??
It's a good practice to have most of the code in python files - benefiting from the IDE of your choice, and separate reporting to notebooks focused on presentation. The mix of code (functionality) vs reporting (visualisation) depends on the data science project.
I used to use jupyter notebooks for everything back in college as well (even stats/math homework). Switched to using neovim and the ipython shell in the terminal, once you learn neovim using the terminal becomes so much faster than jupyter in vscode. Also you can use kitty terminal to display plots in the terminal and termpandas to have scrollable dataframes like in vscode jupyter, all in the terminal. (i3 window manager/tmux also make the workflow so much faster and enjoyable.)
Can it display images directly (using matplotlib or something)?
What plugins are needed?
I'm also trying to move my jupyter work flow into neovim.
@@KamKamKamKam yes kitty terminal allows for displaying images directly in the terminal its pretty cool (save to png and then use os.system("kitty icat fig.png"), python package termpandas (I created it) allows scrollable data frames in the terminal, and ipython shell allows for the whole REPL experience in python. All inside the terminal. The only thing I have not been able to get to work have been interactive html plots in the terminal like plotly express. However, the best data science experience for this in my opinion is split screen neovim and having a streamlit dashboard that automatically updates on save on the other side of the screen. This allows for moving super fast in neovim on the left side of the screen (also using i3 window manager to move around) and a browser with a streamlit app so I can have fully interactive dataframes, plots, filters, and basically any web technology in the streamlit app. Let me know if you are more interested in my setup, I could probably write a medium blog or something explaining my setup, and listing the list of tools/packages I use.
This makes so much sense! I am only just beginning my python journey and am loving your videos. Finding information like this so early on for me seems invaluable to build better habits now with regards to best practices and efficient workflows
Jupyter notebook can actually work with AI assistant like codeium same way you do in VSCode.
Codeium is buggy in Jupyter notebook
You can also use #%% comments to mark sections of your code as "cells" that would be executed as a whole with the keybinding.
I'm not sure if this "run cell" does not conflict with "run line/selection" on some platforms, but you can fix that in the config.
I had same problems!😅 This will help me so much!🎉
Thanks!
Thank you! 🙏🏻
This is very true. I love the idea of Jupiter notebooks, and I've used them for years, but they not composable.
In matlab you are able to define a code block with a ‘##’ that once you are inside it you can shift-enter and it will run. You can also have a variable explorer window. I wonder if there is such functionality in VSCode?
Cool. Is there anything simillar in PyCharm ?
Python Smart Execute. I have no direct experience using it, but it apparently runs code blocks in a console window using a Jupyter cell behind the scenes. Requires the Jupyter plugin to be installed.
Thank you for sharing this is the type of thing a seasoned developer needs to hear.
If you use VS Code then you need to use your existing GPU/CPU but with collab you can use TPU AND GPU free
It depends on where you are on the learning curve. I personally love Jupyter notebooks.
I completely moved to google colab to save RAM and space being overused by Jupyter, only while complex coding do i use vscode else, even docker images are my option
Isn't there an .ipy file type or something. There like Python built-in interactive file type, i think. And you can click one segment at a time. I think it was the basis of .ipynb
I couldn't agree with you more, Dave. Thanks for your informative video on how to use my favorite IDE VS Code with Jupyter Notebooks. This combination seems to be the best duet for data scientific research.
really good. thanks so much for teaching this feature I didnt know before. I like this more.
I'm curious if this style of coding creates any lingering limitations or style / organization choices in your "pure" python files due to the holdover of executing "cells" of code during the development cycle. Stated differently, do you find that any production-ready refactoring is necessary after the functionality is ironed out using Jupyter-esque interactive coding?
Thanks! For your info, I just dropped pycharm+jupyter, switch to vs now!
Definitely, I prefer your workflow over Jupyter Notebook. I am in the phase of testing Cursor, which is similar to VS Code but has some more AI features.
How do you like it so far?
I'm assuming you don't do any heavy pytorch stuff where you need a GPU? Or do you just run them on your mac?
I typically use Azure for that, but I don't often have to run very heavy models.
This methid is like using R. Thanks You! Now I can learn programming in Python sane as in R. Excellent trick ! 👏👏
Hi Dave, what VsTheme(s) are you using ? looking very clean. Like always, appreciate the sharing of your knowledge.
In Jupyter, why can I still not cut and paste text in a cell using the mouse?
Thank you , I'm following your suggestions
this video is for the people who are already expert in data analysis. letting ya'll new analyst like me before you start watching the video.
Great tip, might migrate completely to this workflow. Jupyter Notebooks are great, but as you say, very clunky when approaching as a software engineer.
would this work with a deno kernel in a jupyter notebook too?
Mate, thanks a lot for this. I never liked Jupiter notebooks and I’m glad I’m not the only one haha. Either way, how would you send your structured codes in different python files to cloud computing if you wanted to use that
i would recommend the google collab but only the runtime problem with free version
You can start Jupyter Server in your local machine and connect Colab to a local environment, awesome ! But doesn’t solve the Jupyter notebook challenges, but it’s a much better IDE than VS Code with some many "Microsoft UX” that forces you to adapt in a non efficient way ,plugins and environment challenges…. Modularizing the code and using GIT helps a lot in my experience .
One can use marimo instead of juoyter, or one can even make Code use a notebook. I think the presenter did the latter, but not marimo.
I am bounded to use jupyter notebooks because of low level processor (AMD Athlon Silver 3050U) which is present in low priced laptops, as I am searching for a job, I had less money hence I bought this laptop.
Agreed. It's the only way I work.
I don't get it 😅
There’s something I’ve been struggling to understand. Why do so many indie devs, especially those not using windows, continue to use vs code? Unless you’re living under a rock, and even a dev living under a rock would know this, it’s pretty much a given that whatever you type is being shipped off via telemetry, and that then opens up the risk of people with way more resources at their disposal, to simply copy, iterate quickly on whatever you’re building, and bring it to market before you’ve even had a chance to test. What gives??
what to use instead ?
Thank you!
i am so switching already
notebooks are amazing for DE.
This is amazing.
great tip
nbdev is all you need
Thanks for this fidio
Debuging with Jupyter Notebook? Hmm...
Rstudio FTW
nah
we don't give a fuck.
Why are you here, Sir?