วีดีโอ

stop acting like a clown, get to the point and make an efficient presentation.

I have a stupid question. If we write the data by processing time and not landing time. Then wouldn't it automatically get written in the right partition irrespective of which hour it got processed. Esp. in case of stateless eg: signup events pipeline.

Im curious to know instead of handshake mechanism for finding the when was the last updated time for the given profile, why not use redis cache for the last updatedTime ?

Very good talk . Always useful to hear these talks from practitioners.

Interesting idea to adopt design patterns for data engineering

Quite interesting that a survey was performed so that the points were grounded by data

How can we connect with Betty Li? Any LinkedIn please?

Can you enable captioning of the videos?

Whats the point of posting this without proper capture

This is really cool. Great high level summary of what there is to know about modelling and AI. Thanks Jide

Thanks for sharing the internals at such details level. QQ as per initial design it was mentioned that for quick response request goes to key value data store and then towards the end it was mentioned those requests are catered by Cassandra.Also nowhere in actual design the request is going to impression table as shown in initial design

Buffet of choices also confusing and fattening. There is beauty in a few lean options. Who likes a complex menu of this or that?

Too few details on how the model was actually trained. One of the slides says OpenAI -> Weaviate. How does it actually happen? Weaviate is just a database after all: how the queries towards it are built? A blogpost with details will be highly appreciated. The idea is great but some additional technical details have to be disclosed.

Exceedingly high level

I'd expect Netflix's channel to upload better resolution and size of the slides. :/

"my ducatti is current broken again" lollll

Can you share the paper/blog link please? thx.

Very detailed talk, extremely informative.. Great work Tulika!

amazing

awesome presentation and perfect timing for me. i have to give a presentation in a few days explaining all this new composability in data/database word and why developers should care about it.

Good job keep it up 👍

Veri nice 👍

Ahhh I wish we had the slides in this one 😢

Very nice presentation. Keep it up.

Very informative, keep up 👍

Great

Great ! ❤

Great! Informative 👍👍

The Struggle of Enterprise Data Modeling: A Data Architect's Journey [03:27](th-cam.com/video/DtzIIVJq8wA/w-d-xo.html) Transitioning from data architect to generative AI expert - Discussing journey as a data professional over 16 years, focusing on data modeling and architecture roles at various companies - Detailing challenges faced as a data architect in managing data schemas, infrastructure, and collaborating with developers on data placement [06:54](th-cam.com/video/DtzIIVJq8wA/w-d-xo.html) Challenges with disparate data and maintaining consistency in large organizations. - Data was duplicated and scattered across different teams, leading to difficulties in answering questions. - Complex processes of pulling and joining data from disparate systems and writing code for data consistency and unification. [10:21](th-cam.com/video/DtzIIVJq8wA/w-d-xo.html) Automating data discovery, mapping, and integrations for a unified and accessible data view. - The AI agent automates data mapping, integrations across multiple organizations, and discovers data and relationships. - It also interprets metadata, infers data types and constraints, builds an ontological model, and continuously updates the model. [13:48](th-cam.com/video/DtzIIVJq8wA/w-d-xo.html) Automating data architecture through generative AI - Data modeling involves logical and physical perspectives including entity attributes, relationships, inventory, structure definition, and data population. - Data collection sources range from Postgres, S3, Data Lake, operational systems like Salesforce and Zendesk, involving querying, schema inference, and reverse engineering SQL code. [17:15](th-cam.com/video/DtzIIVJq8wA/w-d-xo.html) Using generative AI to enhance data modeling and querying - The process involved building a data ontology and pushing it into a vector database, specifically Weavio, to enable querying and building multiple levels of relationships - The aim was to provide a user-friendly experience by enabling free text search without the need to build a separate model [20:42](th-cam.com/video/DtzIIVJq8wA/w-d-xo.html) Generative AI interprets queries for quick data access - Generative AI interprets user queries accurately - Feedback loop ensures data accuracy and user satisfaction [24:09](th-cam.com/video/DtzIIVJq8wA/w-d-xo.html) Automated model updates and data tracking for improved decision-making - Ensures agents can learn and adapt by monitoring ontology and updating models with new data sources. - Removes the need for explicit database specifications, enabling intuitive free text search for better decision-making. [27:35](th-cam.com/video/DtzIIVJq8wA/w-d-xo.html) Automating Data Architect with Generative AI - Implemented Snowflake data warehouse for executives to improve data queries and comparisons - Considering enhancing system with knowledge graph and open AI integration for better results

Quite an elaborate insights into inpressions🎉

Informative, good job!

Great content, learnt a lot....I wanted to know [ any viewer can answer as well if they got the answer] , how they ensured that in their SQS has no duplicates ? also, if batches are 10 mins apart, can't we use HWM table in OLTP systems to ensure we get ACID complaint ???

Hello netflix, i would like to be hired by you guys. I am a slightly below average software engineer. Maybe you guys could let me sweep the floors or something? I can be a FAANG janitor! Would just ask for maybe like $11 an hour plus a salad from the cafeteria maybe. I look forward to hearing back from you guys.

Wtf is this guy wearing?

Thanks for sharing this to public!

Gonna be in the team soon!!

Wow, Must see video for every reliable data engineer!

What is High play starts in the example for Context specific Audits @11:30

How I wish

Intro dope afff

how you avoiding too high tide of a changes? meaning - is any late data arriving triggers Psyberg? even just few thousand of rows? or you accumulating changes at some sort of gates/elevators and process when enough late data accumulated to justify downstream reprocessing?

Thanks for sharing. For the comparison between Extractor pattern and DRY principle, it stands but it's not exactly the same driver: DRY principle in programming is to avoid to replicate the same logic - as code - multiple times (to avoid repetitions and incoherences). And this logic can be applied multiple times during the run. Here, the goal is to avoid repetitions for the run itself.

Is there any way you can enable transcript on the youtube video

will they open source Maestro like Airbnb/Airflow??

very smart, using new Acronyms for old Audit tables.

Straight up talk 👏

Where can I download this slide?

Great job Pallavi and Lee! Just had a couple questions: 1. Iceberg and Spark: do you have any challenges running these? Spark Shell not working after adding support for Iceberg, dependency issues w/ AWS and Iceberg, etc.? 2. Why use SQS instead of Kafka? 3. How do you overcome interpreting sessions / sessionization in real time?

Great presentation! Thank you for sharing this! 4:56 - why use Iceberg instead of Delta Lake and Hudi? 8:26 - how do data engineers verify quality data? Isn't that the business office's or data science team's responsibility? 10:08 - DE isn't always told when data source context changes/is updated. 100% true. 12:33 - sometimes = always, in my experience. :) 13:42 - why use python? perhaps due to the schedules? wouldn't scala be faster? 17:55 - Janitor sounds like an incredibly helpful tool!

So insightful. Thanks!