This is exactly what I needed to know! I had all those errors you warned about. Thank you for the very good step-by-step process. I am sure we will see improvements to the wizards soon.
Don't have a firm answer for you. It depends of course. What transformations are being done? What's the compute to handle that transformation. In general, my gut feel is the code/notebook option may be cheaper, but you should do some tests against your data to validate based on numbers.
You actually could have done all of that in just one step, I mean unless of course you wanted to move the zip file as well, but you could have just done the zip deflate and move the contents of the file in one activity rather than two.
how can he do that? I did the equivalence of the two activities in one notebook function and i was thinking on how can we do it in pipeline solution. My data is very large, i could have a couple of hundreds of GB's in one folder.
NO CODE is great until you need to make it resilient and multi-environment. It is hard to see what was created, so you can automate it or put it in source control, and if you use dataflow, you still cannot put these in source control at all (without hacky workaround using json extracts files). When will Microsoft wake up to the fact that source control should be property No1 on all features? Gen 1 dataflow was created in 2019 and still has not source control integration I'm not saying this is a bad video or a bad idea for people who need something quick and dirty, rather I am answering that question: Why do Deveopers always head for the notebook? The answer is: Because the process will be scripted and therefore fit nicely into the SDLC.
This is exactly what I needed to know! I had all those errors you warned about. Thank you for the very good step-by-step process. I am sure we will see improvements to the wizards soon.
Very good and helpful video Adam. Thank you so much.
Great video Q. Can you explain what checking/unchecking enable staging does in the dataflow?
Good video, but image frustration of business user trying to figure this out. :)
Is there difference in performance doing it this way or coded way in the spirit of not being lazy but efficient?
However great video lol, loved it thanks. Looks really similar to ADF
Wat would be the cheaper option if you do this in large amounts of data?
Don't have a firm answer for you. It depends of course. What transformations are being done? What's the compute to handle that transformation.
In general, my gut feel is the code/notebook option may be cheaper, but you should do some tests against your data to validate based on numbers.
Why did you choose Deflate over ZipDeflate?
You actually could have done all of that in just one step, I mean unless of course you wanted to move the zip file as well, but you could have just done the zip deflate and move the contents of the file in one activity rather than two.
how can he do that? I did the equivalence of the two activities in one notebook function and i was thinking on how can we do it in pipeline solution. My data is very large, i could have a couple of hundreds of GB's in one folder.
Hmmm, zip file names have the same prefix, files inside are all data.csv
NO CODE is great until you need to make it resilient and multi-environment. It is hard to see what was created, so you can automate it or put it in source control, and if you use dataflow, you still cannot put these in source control at all (without hacky workaround using json extracts files).
When will Microsoft wake up to the fact that source control should be property No1 on all features?
Gen 1 dataflow was created in 2019 and still has not source control integration
I'm not saying this is a bad video or a bad idea for people who need something quick and dirty, rather I am answering that question: Why do Deveopers always head for the notebook? The answer is: Because the process will be scripted and therefore fit nicely into the SDLC.
Your prayers have heard and they fixed your feedback. Could you please redo your video ?
I'm thinking this "Wizard" didn't graduate from Hogwarts.
yikes. this is incredibly complicated for something so simple.