Database replication(CDC) || Replicate CloudSQL data to BigQuery in real-time using Datastream

Looking to get in touch?
Drop me a line at [e-mail], or schedule a meeting using the provided link topmate.io/vishal_bulbule #gcp #googlecloud #googlecloudtraining #googlecloudplatform Associate Cloud Engineer -Complete Free Course
• Associate Cloud Engine...
Google Cloud Data Engineer Certification Course
• Google Cloud Data Engi...
Google Cloud Platform(GCP) Tutorials
• Google Cloud Platform(...
Terraform Tutorials
• Terraform Associate C...
Learn Python Basics
• Playlist
Linkedin
/ vishal-bulbule
medium
/ vishalbulbule
personal Blog
www.techtrapture.com/
Email - [e-mail] #googlecloud #devops #python #devopsproject #kubernetes #cloudcomputing #video #tutorial

[@ashwinjoshi3331]

Thanks for the video. It helps a lot. One question, how to import oracle or mysql data using cloud data fusion ? Is there any material available to perform that scenario ?

[@techtrapture]

In Data Fusion you can create connections by adding jdbc drivers for respective Database. You will easily find documentation on data fusion on connecting to MySQL/oracle.
Here one sample article I found on medium
rspraneethkumar.medium.com/oracle-to-bigquery-create-a-pipeline-to-extract-data-from-oracle-and-load-it-to-bq-using-data-69ac855f473f

[@ashwinjoshi3331]

Thanks . I actually referred to same document . While creating connections string for data source , there are connection arguments to be mentiond . As per documentation , It mentions as "
Connection Arguments: A list of arbitrary string tag/value pairs as connection arguments. These arguments will be passed to the JDBC driver, as connection arguments, for JDBC drivers that may need additional configurations. This is a semicolon-separated list of key-value pairs, where each pair is separated by a equals ‘=’ and specifies the key and value for the argument. For example, ‘key1=value1;key2=value’ specifies that the connection will be given arguments ‘key1’ mapped to ‘value1’ and the argument ‘key2’ mapped to ‘value2’. "
What does it mean ?
These details I am not able to find.

[@techtrapture]

@@ashwinjoshi3331 thanks for letting me know...Let me try it my environment. I will get back to you. Also let me know your exact requirements so I will try to find suitable solution.

[@ashwinjoshi3331]

Vishal, I really appreciate for your efforts .
I am from MSBI (SSIS,SQL Server) background. So I am novice over here.
So my doubts could be simple but hope you will consider it.
On my laptop , I need to do a simple task of creating a flow
i.e. a) take data from RDBMS source (Oracle (preferred/mysql /postgres/SQL server)
b) Push the data in staging/ final table (BigQuery)
c) Schedule a job
It's a typical kind of data warehouse scenario .
This can be done by various ways- Dataflow,Cloud Data Fusion, Datastream (current video),Python (as an intermediate).
To my knowledge , Cloud Data Fusion almost does these tasks and it's much more simpler also.
But there I faced issue related to connection arguments (explained above) . So I am unable to proceed with other tasks related to POC.
It would be really helpful if you can suggest here.

[@techtrapture]

Thanks for details. I got your query now. Yes Data fusion you can setup , that I will try in my local.
But if you need real time replication then Datastream is more easy but it support MySQL oracle and postgress .

[@ramashankarmaurya4161]

I like your vdo , I got job from your vdo 6 LPA

[@techtrapture]

Thanks for watching and congratulations !!!

[@ramashankarmaurya4161]

Sir please upload dr and backup tool in gcp

[@techtrapture]

I will upload backup and DR video in this week for sure.

[@ramashankarmaurya4161]

Good 👍