Hi This is the best video on duckdb atm. How do you get resources on it? I heard you can connect to a postgres db using a Postgres scanner. How does that work?
Thanks as always for the great content Marc! Just as a sanity check for myself, OLAP Databases don't necessarily need to be columnar based correct? Recent big data databases use columnar based storage, but any database that processes analytical workloads is considered an OLAP database even if it is row based correct?
Thanks, great lesson! Could you help to fix an error, it occurs when I try to run the sales table that we`ve created. The error is "Conversion Error: Could not convert DATE (0019-04-19) to nanoseconds" Don`t understand why it`s trying to get nanoseconds with the date format. Thanks!
for example i don't think it supports the nice extended Syntax like EXCLUTE. but the main reason is speed. duckDB will execute way faster, pandas SQL is still pandas in the end. (correct me if i'm wrong)
i'm sorta stuck... your VSCode is running like a Jupiter notebook... how so... what did i miss... your screen shows a run cell run above debug above... where does that come from
I am getting this error Conversion Error: Could not convert DATE (0019-04-19) to nanoseconds and run conn.execute("from sales").df() when i create a sales table. Not sure if you have seen it?
I was able to solve it by dropping the non date values df = df.loc[df['Order Date'] != 'Order Date'] df['Order Date'] =pd.to_datetime(df['Order Date'],format='%m/%d/%y %H:%M') In the sql block i made this change to match df format strptime("Order Date", '%Y-%m-%d %H:%M:%S') Looks like the trycast is not working or some bug
Oh man, oooofff! I tried following your video here and your blog with the stock analysis data. I've run into too many errors unfortunately. That's the thing about these dependencies - there's too many moving parts where errors are persistent. I can sort through them, but it would take up too much time. We all got work to do :-). For data analysis work, my go to still at this point is using an AWS S3 bucket to load the raw data, use an AWS Glue to create a database and an AWS Glue Crawler to upload the data while creating a table within that database. I can also use AWS Glue Studio to convert the data from .cvs format to .parquet format. From there, I can use AWS Athena to query the data.
Here is important information. Jupyter notebook extension needs to be added to VSCode. How about using the StackOverflow principle and avoid the worthless thank you comments.
@@MarcLamberti In order to work with this tutorial, you provided the sales dataset in the materials. There is a link in the description. On this page, there is a link to the kaggle dataset that you used in the video. However, this one got removed. So it is not possible anymore to follow this tutorial :(
When I ran this Python/Duckdb it only shape returned (186862, 6): # with duckdb cur_time = time.time() df = conn.execute(""" SELECT * FROM '/kaggle/input/sales-product-data/*.csv' """).df() print(f"time: {(time.time() - cur_time)}") print(df.shape)
Would love a full Udemy Course on DuckDB & dbt from you!
I would buy it
great tutorial, thanks for inspiration and your interstings lessons
This is a great intro video, thank you.
Thank you ❤️
That's game changing, very nice video, Marc, Thank you.
Dude.....I like your style! Subscribed and hit the bell....
Thank you 🙏
I did smashed the like "booton", thanks brother, very helpful content!
Thank you 🙏
Brilliant tutorial, appreciate your work.
Awesome content! Thanks a lot!
Thank you! Great content as always!
The link in the description only provides one dataset. How do I get all the CSVs?
Please make more videos
do you have a video on your vscode config setup?
thx a lot, I learned a lot
Thanks Marc
Thank you!
Hi This is the best video on duckdb atm. How do you get resources on it? I heard you can connect to a postgres db using a Postgres scanner. How does that work?
Great, thanks!
You’re welcome ☺️
Thanks as always for the great content Marc! Just as a sanity check for myself, OLAP Databases don't necessarily need to be columnar based correct? Recent big data databases use columnar based storage, but any database that processes analytical workloads is considered an OLAP database even if it is row based correct?
Would love an dbt+ duckdb course from u on Udemy. Big fan of yours 🎉
Awesome ❤
Thank you 🙏
Thanks, great lesson! Could you help to fix an error, it occurs when I try to run the sales table that we`ve created. The error is "Conversion Error: Could not convert DATE (0019-04-19) to nanoseconds" Don`t understand why it`s trying to get nanoseconds with the date format. Thanks!
Why not use the pandas SQL functions?
for example i don't think it supports the nice extended Syntax like EXCLUTE. but the main reason is speed. duckDB will execute way faster, pandas SQL is still pandas in the end. (correct me if i'm wrong)
i'm sorta stuck... your VSCode is running like a Jupiter notebook... how so... what did i miss... your screen shows a run cell run above debug above... where does that come from
Thanks.
I am getting this error
Conversion Error: Could not convert DATE (0019-04-19) to nanoseconds and run conn.execute("from sales").df()
when i create a sales table. Not sure if you have seen it?
Let me double check that
I'm getting this error too!
I was able to solve it by dropping the non date values
df = df.loc[df['Order Date'] != 'Order Date']
df['Order Date'] =pd.to_datetime(df['Order Date'],format='%m/%d/%y %H:%M')
In the sql block i made this change to match df format strptime("Order Date", '%Y-%m-%d %H:%M:%S')
Looks like the trycast is not working or some bug
like always... way to few likes for such a great video!🎉
Oh man, oooofff! I tried following your video here and your blog with the stock analysis data. I've run into too many errors unfortunately. That's the thing about these dependencies - there's too many moving parts where errors are persistent. I can sort through them, but it would take up too much time. We all got work to do :-).
For data analysis work, my go to still at this point is using an AWS S3 bucket to load the raw data, use an AWS Glue to create a database and an AWS Glue Crawler to upload the data while creating a table within that database. I can also use AWS Glue Studio to convert the data from .cvs format to .parquet format. From there, I can use AWS Athena to query the data.
Here is important information. Jupyter notebook extension needs to be added to VSCode. How about using the StackOverflow principle and avoid the worthless thank you comments.
The files are not available anymore :(
What files?
@@MarcLamberti In order to work with this tutorial, you provided the sales dataset in the materials. There is a link in the description. On this page, there is a link to the kaggle dataset that you used in the video. However, this one got removed. So it is not possible anymore to follow this tutorial :(
oh oh, let me check if I can fix that
@@MarcLamberti Thank you! Let us know if it works again
Here we go: www.kaggle.com/datasets/kushagra1211/usa-sales-product-datasetcleaned
4:28 it is written redhshit :D Please don't mind. Thanks for the video.
🫣🥹🥹🥹🥹
Except that baziz, everything is perfectly fine.😀
When I ran this Python/Duckdb it only shape returned (186862, 6):
# with duckdb
cur_time = time.time()
df = conn.execute("""
SELECT *
FROM '/kaggle/input/sales-product-data/*.csv'
""").df()
print(f"time: {(time.time() - cur_time)}")
print(df.shape)