This is the most important video in the entire series. As a data engineer, having the ability to code the entire pipeline from data collection, staging, processing and then uploading to final target (in this case redshift) is the key. Do this for fifteen datasets, and different targets using Jupyter notebooks. In no time (he he i am lying) you will be the master...
@Darshil Parmar hello Darshil can you please help me with this error on the jupyter notebook where i am getting AttributeError: 'iam.ServiceResource' object has no attribute 'get_role'
I realize this already use S3, EC2, and then Redshift Cluster. Will trying this be costly? Or same in previous video about creating redshift cluster, delete everything after using, to avoid cost?
I don’t understand this video, pls where did you get the command you got from VS code, you didn’t show us how to get it. For someone just listening to everything and doing everything you say it would help to slowly put us through how u generate certain codes. Thank you for all you do tho.
Generally in a job if we get any issues/ queries or got stuck somewhere while doing project , we have peers in the team to resolve. How can you handle this in freelancing? I'm curious to know.
@@DarshilParmar HI Darshil, i am following your data engineering prouject course ,this is your first project ,hope to see cluster config file in these series,will check .if you got he link handy pls post here
@@surendraagraharapu7061 You will have to build config file manually by following the video. In previous video th-cam.com/video/kbPlzXBAqzI/w-d-xo.html You will understand basics of redshfit
Hi Darshil, your video is simply awesome. Can you please share the python script that you created in this video? Otherwise it will take very long time to type in every single line.
That is the task, is to perform everything hands on rather than just copy paste. Please watch from 1st video of data model and you’ll understand everything
Hello Darshil,, I also want to become data engineer but there is language problem ,,I come from a small town of uttarakhand and my english speaking is not good ,,, is there any chance for me,,
Hi, your contents are always on point. Please I have some issues. I have pycharm installed on my laptop and I am currently taking a python course where the instructor is using anaconda. I wants to setup anaconda too but it's telling me "this directory is not empty, choose a new one". How can I go about it. Thank you for the good work always.
@@newagegenre that's great to know dear. At that time it didn't work out for me. And even Darshil confirmed that for this tutorial we need some prerequisite...Happy learning!!!
This is the most important video in the entire series. As a data engineer, having the ability to code the entire pipeline from data collection, staging, processing and then uploading to final target (in this case redshift) is the key. Do this for fifteen datasets, and different targets using Jupyter notebooks. In no time (he he i am lying) you will be the master...
How to get the data which is inside cluster.config I mean should we know all that before hand or can we Google it
Hi Darshil :) Could you please share the link to the video about Jupyter's notebook? I really want to learn about it.
Really amazing series man. Thank you Darshil
Dude whats that cluster.cinfig you should mention that from where you are taking reference...
@Darshil Parmar hello Darshil can you please help me with this error on the jupyter notebook where i am getting AttributeError: 'iam.ServiceResource' object has no attribute 'get_role'
When I am trying to create role I don't get option of S3fullaccess or s3readonly.
Where is the Solution Architecture diagram for this project ?
please can you provide docs from where you get to know about options to use in config file. I am not able to get it
Hey Darshil, now aws interface changed and i can only see Amazon Redshift Serverless. What is a difference between the new one and the one you showed?
can we use it? I have same question as yours with this video
Yes we can use it
@@pulkitdikshit9474
@@pulkitdikshit9474 same but i just did all the steps to understand it heh
I realize this already use S3, EC2, and then Redshift Cluster. Will trying this be costly? Or same in previous video about creating redshift cluster, delete everything after using, to avoid cost?
Yes, always delete it after work it done, check AWS Free tier to understand what services are included and also add billing alert
@@DarshilParmarThank you very much!
I am getting permission error while accessing config parser can u help me in this
I don’t understand this video, pls where did you get the command you got from VS code, you didn’t show us how to get it. For someone just listening to everything and doing everything you say it would help to slowly put us through how u generate certain codes. Thank you for all you do tho.
This video is a part of entire playlist - th-cam.com/video/QHIOX_RqjLI/w-d-xo.html
If you directly watched this video then you might not understand it
@@DarshilParmar hi bro can you share the code from vs code and jupyter notebook code ?
Generally in a job if we get any issues/ queries or got stuck somewhere while doing project , we have peers in the team to resolve. How can you handle this in freelancing? I'm curious to know.
I have answered this already in this video
Job vs Freelance | What is the difference? Money? Working Hours?
HI Darshil,From where did the cluster config file suddenly came out
Watch previous videos
@@DarshilParmar HI Darshil, i am following your data engineering prouject course ,this is your first project ,hope to see cluster config file in these series,will check .if you got he link handy pls post here
@@DarshilParmar Could you please share the video link exactely
@@DarshilParmar Pls respond ,where is the config file
@@surendraagraharapu7061 You will have to build config file manually by following the video.
In previous video th-cam.com/video/kbPlzXBAqzI/w-d-xo.html
You will understand basics of redshfit
Hi Darshil, your video is simply awesome. Can you please share the python script that you created in this video? Otherwise it will take very long time to type in every single line.
That is the task, is to perform everything hands on rather than just copy paste.
Please watch from 1st video of data model and you’ll understand everything
@@DarshilParmar watched the data model and I don’t understand it
Hello Darshil,,
I also want to become data engineer but there is language problem ,,I come from a small town of uttarakhand and my english speaking is not good ,,, is there any chance for me,,
Yes, if you can understand written English and work on your speaking then you are good to go, you just need to know basic English
Hi, your contents are always on point. Please I have some issues. I have pycharm installed on my laptop and I am currently taking a python course where the instructor is using anaconda.
I wants to setup anaconda too but it's telling me "this directory is not empty, choose a new one". How can I go about it.
Thank you for the good work always.
Try changing the directory
and just search your error on google you will get 100s of results
@@DarshilParmar thank you
Darshil how long does it take you to complete one freelance work/project?
Depends but it’s 1 week to 6 months
👍👍
It seems like you are teaching to yourself....consider to us tooooo
There are few videos of cloud computing before this you need to watch to understand it, all of this in series
@@DarshilParmar ok Darshil... Can you please tell me which are the services are important for data enginee？？？please mention it here for aws and azure
what do you mean? this is by far the best tutorial I have seen on this topic
@@newagegenre that's great to know dear. At that time it didn't work out for me. And even Darshil confirmed that for this tutorial we need some prerequisite...Happy learning!!!
Darshil so here last you used
Python
Pandas
Boto3
Configpraser
Right
Yes