Building ETL Pipelines Using Cloud Dataflow in GCP

GitHub url: github.com/vig...
This demo reads a csv file from cloud storage buckets, transform using apache beam sdk and finally load the json schema of the intended output into BigQuery.
Follow us in LinkedIn: lnkd.in/gDT3ESdm
Follow us in Instagram: lnkd.in/gZ278ShA
Follow us in Facebook: lnkd.in/gQGF_3Eb
Follow us in Twitter: lnkd.in/gh7dZACW
Join in Telegram channel: lnkd.in/guFt2sAg
Join in WhatsApp group: lnkd.in/gAqkuDPA
Connect with me here:
Instagram: / vignesh909_ss
LinkedIn: / vignesh-sekar-sujatha-...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU NEED TO DO BELOW THINGS to support my channel
1. LIKE
2. SHARE
&
3. SUBSCRIBE
TO MY TH-cam CHANNEL
#gcpcloud #datafusion #bigdata #dataengineer #cloudplatform #dataflow #etl #gcpdataengineer #bigquery #cloudstorage

[@rrafaelpaz]

Very nice mate! Very well explained! Cheers from Brazil brotha!

[@cloudaianalytics6242]

Thanks!!!

[@venkatvlogs07]

too hurry not able to understand it as you are switching tabs and doing all the things and not mentioning where you are writing the code. The course should be designed so that even beginner should be able to understand it. please make a pin to pin point to point explanation video so that everyone can understand it. Thanks in advance ❤

[@cloudaianalytics6242]

Sorry about that....Ill keep this in mind. Thanks a lot

[@JS-kj1rc]

Very helpful. Thanks

[@cloudaianalytics6242]

Glad it was helpful!

[@nathaniasantanigels]

how can i do if the data from gheet?

[@cloudaianalytics6242]

Not sure

[@ashishvats1515]

Great video, i want to take input from JDBC connection a table and load to bigquery… could you please share any document related to this, to how take table as an input from JDBC and load to bigquery

[@cloudaianalytics6242]

beam.apache.org/releases/pydoc/2.24.0/apache_beam.io.jdbc.html
beam.apache.org/releases/pydoc/current/apache_beam.io.jdbc.html

[@ashishvats1515]

@@cloudaianalytics6242 thanks, if i’m facing any issue can i ping u on linkdin or telegram?

[@cloudaianalytics6242]

@@ashishvats1515 😊 sure

[@ashishvats1515]

@@cloudaianalytics6242i’m tried but facing some errors… could you please share a example code of this or make a video on it…

[@pournimaambikar5857]

I am getting below error while trying to run dataflow job:
import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
on both cloud sdk and cloud shell, wheras apache_beam is installed

[@RajDas-uy2ro]

pip install apache-beam[gcp]

[@cloudaianalytics6242]

pip install apache-beam[gcp] or try createing a virtual environment in cloud shell and run dataflow jobs from there after installing apache beam

[@chaithuchinna94]

is there any course available sir to learn gcp ?if so pls help me provide the details

[@cloudaianalytics6242]

Course Link: www.udemy.com/course/gcp-professional-dataengineer-certification-a-complete-guide
Reach for Coupon Code - www.linkedin.com/in/vignesh-sekar-sujatha-02aa9b125/

[@ashwinjoshi3331]

Thanks for the video. One question - in case the source is oracle on premise and sink is BigQuery then what changes are required to do ?

[@cloudaianalytics6242]

Need to do bit research on this. definitely we can use some JDBC, ODBC connectors

[@neharas]

what is on premise ? is it traditional computers? or some type of cloud

[@sumitdwivedi9474]

can you create this pipeline and do transformations within gcp dataflow itself?

[@cloudaianalytics6242]

It is in GCP dataflow

[@sanketgurnalkar5813]

How to give runtime parameters? can you give the code

[@cloudaianalytics6242]

Sure, Ill make a video on it. Meanwhile you can get it from my GitHub repo
github.com/vigneshSs-07?tab=repositories

[@honeylokesh2340]

How to enroll your training???

[@cloudaianalytics6242]

Please drop a mail to [e-mail]
If you are interested in self paced. take a look at this self paced course in Udemy
www.udemy.com/course/gcp-professional-dataengineer-certification-a-complete-guide/

[@ashraf_isb]

thanks man!

[@cloudaianalytics6242]

Glad it was helpful!

[@tommedcouk]

Dataflow isn’t the most widely used component in the Google Cloud Platform. Even if you Google this question, the sensible response is Compute Engine because it runs under pretty much all the other services, but also because a lot of companies do a lift and shift to cloud before integrating with the other services. You claim this twice at the beginning of the video, but it’s incorrect

[@klgulen650]

What about airflow ?

[@Rajdeep6452]

Can’t integrate airflow (cloud composer) with vm instances on gcp.

[@cloudaianalytics6242]

Apologies for the wrong information. Yes Compute engine is base for all, I agree. It really depends on the business use cases.

[@cloudaianalytics6242]

It is widely used to orchestrate big data pipelines..In GCP airflow is in built with Composer but you can run independently as well.

[@AnantPradhan-y7m]

Couldn't understand. Complicated...

[@cloudaianalytics6242]

Sorry to hear that. Ill try to break it down in upcoming videos. Please keep an eye on it

[@1itech]

make little bit slow

[@cloudaianalytics6242]

Sure

[@pm4306]

very confusing ......as you keep jumping from 1 screen to another.....

[@cloudaianalytics6242]

Sorry to hear. Can you use playback speed option in youtube to reduce the speed of video. Hope it helps

[@shamilak1]

head_usa_names share the file

[@cloudaianalytics6242]

you can find it in github