Chandeep, did you say you can do **Incremental Refresh** when files are saved in SharePoint?! 🤯 Can you point to where I can learn how to do that because that would be a game changer for me!! 😁 Thanks!!
Dear Chandeep! I suggest, You could also make additional videos on the power query optimization options as discussed in the video. Please give me your suggestion. I have a problem with the notebook I use at my work. It has an old i3 cpu, 8GB of ram and use Office 365. Dozens of (currently 80 files) csv files need to be processed. Their individual file sizes are approx. 800 MB and the number of rows sometimes exceeds 1 million per file before cleaning. The files are distributed in directories. Imagine that there are cities, stores in the cities, products in the stores, and finally sales data for each product in the stores per customers. Should I read the individual files and then combine them, or should I read the main library? Than filter the necessary data types? Should I store the queries in memory or load them into excel pages? What if there are more rows than 1 million? How should I handle large amounts of data? Maybe I should switch to the MS Access program? Or should I upload all csv files to SharePoint and read them from there with PoweQuery?
Hello Goodly, a question. Is it possible to generate a sheet automatically in Power Query? I have a table with a question number column and the rows are the answers. I want to separate the question numbers into sheets. Sheet="1" with all the answers to question 1, sheet="2" with all the answers to question 2... sheet =62 with all the answers to question 62.
why isn't Folder.Contents showing in the intellisense? ..... ok solved. the intellisense was not working well at all. I closed excel and opened it again and everything is normal again. Another Excel mystery
Some of these tips I picked up from different tutorials and some I was able to figure out myself. And indeed, they help improve refresh of the query. 😃
One small question. Even though csv is a better option than excel when it comes to raw data but their file sizes are quite higher than excel. In older Office, I have seen csv s to be quite smaller in sizes when I would convert them from Excel but it doesn't happen nowadays. How can we address this?
@@dharmmuI mean join :) you can combine a lot of files but what about join/merge. If you want to join tables there is only possibility to do it with max 2 tables using specific ID columns. I would like to have one "base table" and join multiple tables (saved in one folder) to them.
@@CiachoAcM the other tables are in different sheets or workbook or in one sheet itself you have many different tables. Also, tables are in proper excel table format or it's just a sets of data in excel.
Btw, I believe you can transform a subset of the data (like sample file in the transform step) with power query, then convert these power query applied steps into SQL to apply to the larger dataset.
Awesome as alwys🎉 please can you make a video on snowflake database data load to power bi and best practices for handling huge database in power query... Thanks
These tips are absolutely essential and apply way over importing data from multiple csv/excel files. That’s what any sql, bi, data analyst should know and apply🎉
Chandeep, did you say you can do **Incremental Refresh** when files are saved in SharePoint?! 🤯 Can you point to where I can learn how to do that because that would be a game changer for me!! 😁 Thanks!!
+1!
I think the incremental refresh is only possible with a Premium account.
Dear Chandeep!
I suggest, You could also make additional videos on the power query optimization options as discussed in the video.
Please give me your suggestion. I have a problem with the notebook I use at my work. It has an old i3 cpu, 8GB of ram and use Office 365. Dozens of (currently 80 files) csv files need to be processed. Their individual file sizes are approx. 800 MB and the number of rows sometimes exceeds 1 million per file before cleaning. The files are distributed in directories. Imagine that there are cities, stores in the cities, products in the stores, and finally sales data for each product in the stores per customers.
Should I read the individual files and then combine them, or should I read the main library? Than filter the necessary data types?
Should I store the queries in memory or load them into excel pages? What if there are more rows than 1 million?
How should I handle large amounts of data?
Maybe I should switch to the MS Access program?
Or should I upload all csv files to SharePoint and read them from there with PoweQuery?
Hello Goodly, a question. Is it possible to generate a sheet automatically in Power Query? I have a table with a question number column and the rows are the answers. I want to separate the question numbers into sheets. Sheet="1" with all the answers to question 1, sheet="2" with all the answers to question 2... sheet =62 with all the answers to question 62.
why isn't Folder.Contents showing in the intellisense? ..... ok solved. the intellisense was not working well at all. I closed excel and opened it again and everything is normal again. Another Excel mystery
For Folder Contents portion, could you create a tutorial on how to do it with Sharepoint Folder Path? Thank you for these tips! Very helpful!
.xls file not uploaded in query why?
Hi Goodly, do you have a way to combine xls files?
Some of these tips I picked up from different tutorials and some I was able to figure out myself. And indeed, they help improve refresh of the query. 😃
One small question. Even though csv is a better option than excel when it comes to raw data but their file sizes are quite higher than excel. In older Office, I have seen csv s to be quite smaller in sizes when I would convert them from Excel but it doesn't happen nowadays. How can we address this?
I have the same observation with CSV files.
Can we perform upsert in power query
How to merge multiple files, not combine? Do you know The solutions to merge more than 2 files/tables?
Sorry, but what you meant by merge but not combine?
@@dharmmuI mean join :) you can combine a lot of files but what about join/merge. If you want to join tables there is only possibility to do it with max 2 tables using specific ID columns. I would like to have one "base table" and join multiple tables (saved in one folder) to them.
@@CiachoAcM the other tables are in different sheets or workbook or in one sheet itself you have many different tables. Also, tables are in proper excel table format or it's just a sets of data in excel.
I have a question. Do you have a video on how to get data from folder without combining the files, but creating separate queries for all of them?
Create a reference query pointing yo the folder where you have your files and load individually by referencing and filtering
@@justapasserby69420 I’m looking for a more automatic way to
Hi,
How to Save All the custom functions from one pbix file as a file and import those custom functions into other pbix files?
Thanks Chandeep! Awesome as always
It’s simpler to control formatting with xlsx files than csv - especially with leading 0’s
Btw, I believe you can transform a subset of the data (like sample file in the transform step) with power query, then convert these power query applied steps into SQL to apply to the larger dataset.
Awesome as alwys🎉 please can you make a video on snowflake database data load to power bi and best practices for handling huge database in power query... Thanks
Hi! Very nice video. I have one question, what is the best approach to work with excel files that are generated in different languages?
These tips are absolutely essential and apply way over importing data from multiple csv/excel files. That’s what any sql, bi, data analyst should know and apply🎉
how about savings excel files in binary - xlsb it help with refreshes
Nice work Chandeep :)
Good advice on an important topic 👌
Does PQ have some way to look how much space(?) does it take to execute each step?
You can do that with Query Diagnostics in the Tools Tab in Power Query (Power BI)
Great one!!! Loved it.
Exellent1!!
So basically he is saying how to make things unnecessarily complicated
Where are you getting that from?