Building ETL Pipelines Using Cloud Dataflow in GCP

GitHub url: github.com/vig...
This demo reads a csv file from cloud storage buckets, transform using apache beam sdk and finally load the json schema of the intended output into BigQuery.
Follow us in LinkedIn: lnkd.in/gDT3ESdm
Follow us in Instagram: lnkd.in/gZ278ShA
Follow us in Facebook: lnkd.in/gQGF_3Eb
Follow us in Twitter: lnkd.in/gh7dZACW
Join in Telegram channel: lnkd.in/guFt2sAg
Join in WhatsApp group: lnkd.in/gAqkuDPA
Connect with me here:
Instagram: / vignesh909_ss
LinkedIn: / vignesh-sekar-sujatha-...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU NEED TO DO BELOW THINGS to support my channel
1. LIKE
2. SHARE
&
3. SUBSCRIBE
TO MY TH-cam CHANNEL
#gcpcloud #datafusion #bigdata #dataengineer #cloudplatform #dataflow #etl #gcpdataengineer #bigquery #cloudstorage

[@venkatvlogs07]

too hurry not able to understand it as you are switching tabs and doing all the things and not mentioning where you are writing the code. The course should be designed so that even beginner should be able to understand it. please make a pin to pin point to point explanation video so that everyone can understand it. Thanks in advance ❤

[@rrafaelpaz]

Very nice mate! Very well explained! Cheers from Brazil brotha!

[@cloudaianalytics6242]

Thanks!!!

[@student_voice]

And also, one more request, when you using a gcp Service, also explain required its access privilege for a user

[@cloudaianalytics6242]

sure..ill definitely make video on it

[@ashraf_isb]

thanks man!

[@student_voice]

Can you make a video on CI/CD for from oracle to bigquery using tools like jenkins bitbucket sonarqube checkmarks, airflow Composer..
If u can, this will be very helpful.. 🤝

[@cloudaianalytics6242]

Ill try to do this lab and post

[@ashishvats1515]

Great video, i want to take input from JDBC connection a table and load to bigquery… could you please share any document related to this, to how take table as an input from JDBC and load to bigquery

[@cloudaianalytics6242]

beam.apache.org/releases/pydoc/2.24.0/apache_beam.io.jdbc.html
beam.apache.org/releases/pydoc/current/apache_beam.io.jdbc.html

[@ashishvats1515]

@@cloudaianalytics6242 thanks, if i’m facing any issue can i ping u on linkdin or telegram?

[@cloudaianalytics6242]

@@ashishvats1515 😊 sure

[@ashishvats1515]

@@cloudaianalytics6242i’m tried but facing some errors… could you please share a example code of this or make a video on it…

[@chaithuchinna94]

is there any course available sir to learn gcp ?if so pls help me provide the details

[@cloudaianalytics6242]

Course Link: www.udemy.com/course/gcp-professional-dataengineer-certification-a-complete-guide
Reach for Coupon Code - www.linkedin.com/in/vignesh-sekar-sujatha-02aa9b125/

[@tommedcouk]

Dataflow isn’t the most widely used component in the Google Cloud Platform. Even if you Google this question, the sensible response is Compute Engine because it runs under pretty much all the other services, but also because a lot of companies do a lift and shift to cloud before integrating with the other services. You claim this twice at the beginning of the video, but it’s incorrect

[@klgulen650]

What about airflow ?

[@Rajdeep6452]

Can’t integrate airflow (cloud composer) with vm instances on gcp.

[@pournimaambikar5857]

I am getting below error while trying to run dataflow job:
import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
on both cloud sdk and cloud shell, wheras apache_beam is installed

[@RajDas-uy2ro]

pip install apache-beam[gcp]

[@cloudaianalytics6242]

pip install apache-beam[gcp] or try createing a virtual environment in cloud shell and run dataflow jobs from there after installing apache beam

[@ashwinjoshi3331]

Thanks for the video. One question - in case the source is oracle on premise and sink is BigQuery then what changes are required to do ?

[@cloudaianalytics6242]

Need to do bit research on this. definitely we can use some JDBC, ODBC connectors

[@neharas]

what is on premise ? is it traditional computers? or some type of cloud

[@sumitdwivedi9474]

can you create this pipeline and do transformations within gcp dataflow itself?

[@cloudaianalytics6242]

It is in GCP dataflow

[@honeylokesh2340]

How to enroll your training???

[@sanketgurnalkar5813]

How to give runtime parameters? can you give the code

[@cloudaianalytics6242]

Sure, Ill make a video on it. Meanwhile you can get it from my GitHub repo
github.com/vigneshSs-07?tab=repositories

[@AnantPradhan-y7m]

Couldn't understand. Complicated...

[@pm4306]

very confusing ......as you keep jumping from 1 screen to another.....

[@cloudaianalytics6242]

Sorry to hear. Can you use playback speed option in youtube to reduce the speed of video. Hope it helps

[@1itech]

make little bit slow

[@cloudaianalytics6242]

Sure

[@shamilak1]

head_usa_names share the file