Data Warehousing With BigQuery: Best Practices (Cloud Next '19)

Take an in-depth look at modern data warehousing using BigQuery and how to operate your data warehouse in the cloud. During this session, we'll give lessons learned and best practices from prior implementations to give you the playbook for implementing your own modern data warehouse.
"BigQuery ML → bit.ly/2Khkqoq
BigQuery for data warehouse practioners → bit.ly/2TWh6P9"
Watch more:
Next '19 Data Analytics Sessions here → bit.ly/Next19DataAnalytics
Next ‘19 All Sessions playlist → bit.ly/Next19AllSessions
Subscribe to the GCP Channel → bit.ly/GCloudPlatform
Speaker(s): Ryan McDowell, Alban Perillat-Merceroz
Session ID: DA307
product:BigQuery; fullname:Ryan McDowell; event: Google Cloud Next 2019; re_ty: Publish; product: Cloud - Data Analytics - BigQuery; fullname: Ryan McDowell;

[@kunfang2457]

the best 'best practice' sharing session i ve ever had

[@anandakumarsanthinathan4740]

Beautifully explained. Many thanks. Now, why wouldn't anybody want to move to the sunny side of France and land a job at Teads !! Both of the presenters did an excellent job.

[@3rdinnings326]

Excellent session!

[@TheMitali_]

How to perform sum of columns value in big query. And no of columns are notfixed , at runtime we need to decide no of columns need to consider for sum. Depends on user inputs

[@ThisIsAli_Off]

Very useful!

[@jennwng]

At 13:36, why do we need to load into GCS first for batch loads? Can't we use Dafaflow/BigQuery BatchLoad / Data Fusion directly from GoldenGate?

[@krishnabg1350]

Can you share slides?

[@LuckyHongTJ]

At 13:35, what is the benefit of loading data from Oracle into GCS first and then BigQuery (i.e. won't directly loading into BigQuery from Oracle be faster)? I know GCS can serve as a staging area and we get another copy of the data for fault tolerance, but is there any other benefit? Thanks :)

[@amieewright4417]

H thanks u so much love you amiee Wright and Bella daws Xxxxx

[@mzamanmintu3694]

Congratulations

[@jennwng]

At 11:58 why DataFlow / Data Fusion can speed up pipelines? Like, the essence is either to do the transformation in BigQuery or in DataFusion. Even if the query is complex, won't it still be faster to do the transformation directly in BigQuery, rather than connecting to DataFusion and transforming data there?

[@vishnureddys4801]

It is because they can bill you in both Services, for using BigQuery and also dataflow.

[@Thiago280690]

6:30 do nada aparece uma nota de dez conto hahaha adorei

[@Vinch157]

Hello, will you also share the slides? Thank you

[@vishnureddys4801]

Has he yet?

[@JOPINC]

Dedicado a JMB!!!! :)

[@LouisChiaki]

How is this different from Spark?

[@jean4j_]

Well it's auto-managed and much simpler to work with.
That's just SQL. You don't need to fine-tune the configuration like you would for a Spark job I think.

[@sunilpipara]

Kafka to S3 and S3 to Cassandra via Spark is wrong approach.

[@anandakumarsanthinathan4740]

Yes, @Sunil Jain. I feel the same too. The source application would directly write it out to S3 for batch processing. Kafka would be required only for the streaming jobs.

[@GreenPower4ever]

P