- 242
- 63 782
DataEngBytes
Australia
เข้าร่วมเมื่อ 29 พ.ค. 2020
The official TH-cam Channel of DataEngBytes, and the Data Engineering meetup.
DataEngBytes website: dataengconf.com.au/
Meetups:
- Sydney: www.meetup.com/Sydney-Data-Engineering-Meetup/
- Melbourne: www.meetup.com/Melbourne-Data-Engineering-Meetup/
- Brisbane: www.meetup.com/Brisbane-Data-Engineering-meetup/
- Perth: www.meetup.com/Perth-Data-Engineering-meetup/
- Auckland: www.meetup.com/Auckland-Data-Engineering-meetup/
- Hobart: www.meetup.com/Hobart-Data-Engineering-meetup/
DataEngBytes website: dataengconf.com.au/
Meetups:
- Sydney: www.meetup.com/Sydney-Data-Engineering-Meetup/
- Melbourne: www.meetup.com/Melbourne-Data-Engineering-Meetup/
- Brisbane: www.meetup.com/Brisbane-Data-Engineering-meetup/
- Perth: www.meetup.com/Perth-Data-Engineering-meetup/
- Auckland: www.meetup.com/Auckland-Data-Engineering-meetup/
- Hobart: www.meetup.com/Hobart-Data-Engineering-meetup/
Data Ethics Panel Auckland
Our Data Ethics Panel will explore critical issues at the intersection of technology, data, and ethics. Distinguished experts will delve into pressing topics such as safeguarding privacy in the age of big data, ensuring algorithmic fairness and transparency, addressing bias in AI systems, and the ethical implications of emerging technologies. The panel will discuss strategies for responsible data collection and usage, the challenges of informed consent in the digital age, and the societal impacts of data-driven decision-making. Attendees can expect thought-provoking debates on data ownership, the ethical dimensions of data sharing, and the balance between innovation and individual rights. Join us for an insightful discussion that will shape the future of ethical data practices in our increasingly connected world.
Lauren Peate
CEO & founder at Multitudes
Lauren Peate is the CEO and founder of Multitudes, which provides an ethical AI coach to improve engineering efficiency and wellbeing. Before founding Multitudes, she ran a diversity, equity, and inclusion consultancy that worked with product and engineering teams at companies like Automattic and Xero. Previous experience includes working with Fortune 500 tech companies while at Bain & Co in San Francisco, working with an eCommerce startup in Jordan, advising startups across the Middle East, and conducting research as a Fulbright Scholar in Morocco. Lauren graduated from Stanford University.
Emma MacDonald
Director for the Centre for Data Ethics and Innovation
Emma was bought into Stats in April 2023 to establish up a Centre for Data Ethics and Innovation. CDEI is an initiative to help government agencies achieve a secure and trusted data environment through good data practices. CDEI works to get things done by bringing the right people with the right skills together. Emma has spent her career work in the policy across numerous agencies, including at the Department of Corrections, the Department of Internal Affairs, the Ministry of Transport, and Ministry of Business, Innovation and Employment.
Jessica Chunyao Zhang
Les Mills International, Senior Data Engineer
Jess is a Senior Data Engineer at Les Mills International, where they design and implement data platforms with a strong emphasis on governance, security, and customer-centric solutions. With a passion for data ethics, Jess has led initiatives such as enhancing data security with role-based access policies to better secure customer data. Their previous roles include migrating Trade Me’s Data Warehouse to the cloud, automating marketing pipelines, and managing data lifecycle and model validation at Narrative. Additionally, they have experience in software development for healthcare from their time at Orion Health. Driven by a commitment to ethical data practices and prioritizing customer impact, Jess brings valuable insights to the Data Ethics Panel on responsible data usage, collection, and storage.
Sean Falconer
Head of Developer Relations at Skyflow
Sean Falconer, PhD in Computer Science, with a Postdoc in Bioinformatics from Stanford University, brings over 15 years of experience in research, engineering, product, developer relations, and marketing. Prior to Skyflow, he contributed to projects with the World Health Organization, founded Proven.com, and led developer relations engineering for Google's Business Communication products. At Skyflow, Sean leads marketing and developer relations, actively engaging with communities through building, writing, speaking, and fostering discussions on engineering and data privacy.
Find out more about DataEngBytes at our website:
dataengbytes.com/
Follow us on LinkedIn:
www.linkedin.com/company/dataengbytes/
Proudly brought to you by Cloud Shuttle:
cloudshuttle.com.au/
Lauren Peate
CEO & founder at Multitudes
Lauren Peate is the CEO and founder of Multitudes, which provides an ethical AI coach to improve engineering efficiency and wellbeing. Before founding Multitudes, she ran a diversity, equity, and inclusion consultancy that worked with product and engineering teams at companies like Automattic and Xero. Previous experience includes working with Fortune 500 tech companies while at Bain & Co in San Francisco, working with an eCommerce startup in Jordan, advising startups across the Middle East, and conducting research as a Fulbright Scholar in Morocco. Lauren graduated from Stanford University.
Emma MacDonald
Director for the Centre for Data Ethics and Innovation
Emma was bought into Stats in April 2023 to establish up a Centre for Data Ethics and Innovation. CDEI is an initiative to help government agencies achieve a secure and trusted data environment through good data practices. CDEI works to get things done by bringing the right people with the right skills together. Emma has spent her career work in the policy across numerous agencies, including at the Department of Corrections, the Department of Internal Affairs, the Ministry of Transport, and Ministry of Business, Innovation and Employment.
Jessica Chunyao Zhang
Les Mills International, Senior Data Engineer
Jess is a Senior Data Engineer at Les Mills International, where they design and implement data platforms with a strong emphasis on governance, security, and customer-centric solutions. With a passion for data ethics, Jess has led initiatives such as enhancing data security with role-based access policies to better secure customer data. Their previous roles include migrating Trade Me’s Data Warehouse to the cloud, automating marketing pipelines, and managing data lifecycle and model validation at Narrative. Additionally, they have experience in software development for healthcare from their time at Orion Health. Driven by a commitment to ethical data practices and prioritizing customer impact, Jess brings valuable insights to the Data Ethics Panel on responsible data usage, collection, and storage.
Sean Falconer
Head of Developer Relations at Skyflow
Sean Falconer, PhD in Computer Science, with a Postdoc in Bioinformatics from Stanford University, brings over 15 years of experience in research, engineering, product, developer relations, and marketing. Prior to Skyflow, he contributed to projects with the World Health Organization, founded Proven.com, and led developer relations engineering for Google's Business Communication products. At Skyflow, Sean leads marketing and developer relations, actively engaging with communities through building, writing, speaking, and fostering discussions on engineering and data privacy.
Find out more about DataEngBytes at our website:
dataengbytes.com/
Follow us on LinkedIn:
www.linkedin.com/company/dataengbytes/
Proudly brought to you by Cloud Shuttle:
cloudshuttle.com.au/
มุมมอง: 6
วีดีโอ
Data, It’s a People Problem: Understanding the Sociotechnical Elements of a Data-Driven Organization
มุมมอง 4316 ชั่วโมงที่ผ่านมา
In the evolving landscape of data-driven organisations, success hinges on more than just technology - it's about how we structure and interact with our teams and data. This talk explores the sociotechnical components required to thrive in a data-centric world. We’ll delve into how organising teams using concepts from Empowered, Team Topologies, and Domain-Driven Design can enhance collaboration...
Who Really Uses Dataform? Spoiler: We Do, and Here’s Why!
มุมมอง 334 ชั่วโมงที่ผ่านมา
Ever wondered, "Who really uses Dataform?" Dive into our hands-on experience of implementing Dataform at Pet Circle. For those wondering, Dataform is a service in Google to develop, test, version control, and schedule complex SQL workflows for data transformation in BigQuery. And no, this isn’t a sales pitch - we’re here to share the good, the bad, and the ugly of our journey. In this beginner-...
From Start-Up to Scale-Up: Building a Data Team at a High-Growth Global Ed-Tech Company
มุมมอง 644 ชั่วโมงที่ผ่านมา
In four short years, Kami has grown from 8 million to over 40 million users and from 23 to over 100 employees, exporting New Zealand-made software to support teachers and students in classrooms all over the world. Such rapid growth saw Kami recognised in the TIME100 Most Influential Companies 2022 and as the NZ Deloitte Fast 50 Fastest Growing Company for 2021. It also came with a commensurate ...
Supercharge Information Extraction with Vision-Based LLMs
มุมมอง 527 ชั่วโมงที่ผ่านมา
Join us for an insightful session on how Vision-Based Large Language Models (LLMs) are transforming document information extraction. We'll start by exploring how these advanced models enable faster and more accurate data extraction compared to traditional methods. Next, we'll delve into a project conducted for an insurance provider aimed at reducing claim processing time. Discover the challenge...
Measuring Success - Observability and More!
มุมมอง 2177 ชั่วโมงที่ผ่านมา
Unlocking the Key Metrics That Matter to Data Professionals. Have you ever wondered how best to report / discuss / visualise how the data team's performing? If you tried, you'd know this is much tricker than measuring performance of other functions in the company. The deliverables of the Data team are of two types. one is visible work, like the analytical dashboard or some streamlit app to disp...
Data Platforms Transformation - Why and How
มุมมอง 1379 ชั่วโมงที่ผ่านมา
Data use cases and technologies around data processing and serving are evolving rapidly. The session focuses on why and when one should consider transforming data platforms to meet the future scale and demand and how to execute it. Using practical examples we will explore how to identify the problem space relative to current and future needs and different ways using which we can transform data ...
Unified dataframe API for different data backend
มุมมอง 22916 ชั่วโมงที่ผ่านมา
As data use cases become more complex, data platforms evolve to meet their needs. Data workloads run in different environments (local and production), in different modes (batch and streaming), and across different hardware (CPU and GPU). This talk introduces Ibis, an open-source project that allows users to work with different backends in different settings. We’ll go into how Ibis works under t...
Stop Making Your Data Team the 'Data Police'
มุมมอง 12916 ชั่วโมงที่ผ่านมา
When was the last time you performed a mathematical operation on an email address? Or multiplied a credit card number by a passport number? It's absurd, this would be an insane thing to do, yet we keep storing sensitive customer information in our data warehouses, risking PII exposure, as if we need to perform operations like this. This forces our data teams to act as data police, controlling a...
Navigating Streaming Infrastructure
มุมมอง 8216 ชั่วโมงที่ผ่านมา
When designing a distributed system architecture there will always be contradictory requirements. Sometimes we'll be referred to as constraints. Like the CAP theorem, where developers designing a system have to choose between what is physically possible between consistency, availability and partition tolerance. The same applies to streaming infrastructure systems. Optimizing for one will interf...
How open source is re-shaping the cloud data warehouse landscape
มุมมอง 33916 ชั่วโมงที่ผ่านมา
In the last decade, the rise of the proprietary cloud data warehouse, led by platforms like Snowflake, BigQuery, and Redshift, has helped modernize data warehousing by providing scalability, convenience, and most importantly flexibility and openness to a very important class of data workloads. Once this data was available in the cloud, it was possible to use it for more use cases, including use...
Mixed Model Arts
มุมมอง 12121 ชั่วโมงที่ผ่านมา
For decades, data modeling has been fragmented by use cases: applications, analytics, and machine learning/AI. This leads to data siloing and “throwing data over the wall.” With the emergence of AI, streaming data, and “shifting left" are changing data modeling, these siloed approaches are insufficient for the diverse world of data use cases. Today's practitioners must possess an end-to-end und...
Advanced Enterprise RAG Systems
มุมมอง 22214 วันที่ผ่านมา
Advanced Enterprise RAG Systems The need for accurate, contextually relevant, and timely information has never been greater. Retrieval Augmented Generation (RAG) systems combine the power of sophisticated information retrieval with the dynamic capabilities of generative AI. All this helps enterprises move beyond traditional search and query methods to enable the generation of responses that are...
How I built an entire data platform by myself that thousands of data engineers use in one year
มุมมอง 16K14 วันที่ผ่านมา
How I built an entire data platform by myself that thousands of data engineers use in one year Zach built an entire data platform by himself that thousands of data engineers use each year. In this talk, he goes through how he did it and the choices and lessons he learned along the way! Zach Wilson Founder @ DataExpert.io I love to teach. I post about #dataengineering and #mentalhealth daily. Ma...
DataEngBytes 2024 CFP Promo
มุมมอง 2234 หลายเดือนก่อน
DataEngBytes is back and better than ever! Last year was an absolute blast, and we're gearing up for another incredible journey. Check out what our attendees, speakers, , sponsors and organisers had to say about their experiences. Join us in Sydney, Perth, Melbourne, and Auckland for a day of learning, networking, and engaging data talks. Don't miss out on the fun and insights-get your tickets ...
Building an Analytical System of Record by Ananth Gundabattula
มุมมอง 755 หลายเดือนก่อน
Building an Analytical System of Record by Ananth Gundabattula
Harish Suresh, Demystifying Image Generation GenAI models
มุมมอง 446 หลายเดือนก่อน
Harish Suresh, Demystifying Image Generation GenAI models
Sean Beath and Peter Vandale, Reinvent Recap - Data and Analytics
มุมมอง 356 หลายเดือนก่อน
Sean Beath and Peter Vandale, Reinvent Recap - Data and Analytics
Andrew Ridgway, Duckdb and Metabase - a containerised reporting solution
มุมมอง 3206 หลายเดือนก่อน
Andrew Ridgway, Duckdb and Metabase - a containerised reporting solution
San Tran, Security in ClickHouse Cloud
มุมมอง 756 หลายเดือนก่อน
San Tran, Security in ClickHouse Cloud
Johnny Mirza, A Journey through Command vs Event Driven Architectures
มุมมอง 256 หลายเดือนก่อน
Johnny Mirza, A Journey through Command vs Event Driven Architectures
Tom Watson, Supercharging Billing Efficiency with ClickHouse
มุมมอง 2436 หลายเดือนก่อน
Tom Watson, Supercharging Billing Efficiency with ClickHouse
Johnny Mirza, An Introduction to ClickHouse
มุมมอง 857 หลายเดือนก่อน
Johnny Mirza, An Introduction to ClickHouse
Adam Malone, Building Hasura's Observability Infrastructure with ClickHouse and GraphQL
มุมมอง 1067 หลายเดือนก่อน
Adam Malone, Building Hasura's Observability Infrastructure with ClickHouse and GraphQL
Sandamali De Zoysa, Snowflake Data Quality Monitoring
มุมมอง 6017 หลายเดือนก่อน
Sandamali De Zoysa, Snowflake Data Quality Monitoring
Antony Southworth, Data for Dairy: Lessons Learned Building Halter's Data Platform
มุมมอง 457 หลายเดือนก่อน
Antony Southworth, Data for Dairy: Lessons Learned Building Halter's Data Platform
BUILD YOUR OWN ELECTRIC VEHICLE CHARGING MAP WITH PostGIS
มุมมอง 111ปีที่แล้ว
BUILD YOUR OWN ELECTRIC VEHICLE CHARGING MAP WITH PostGIS
PROCESSING 40 TB OF CODE FROM ~10 MILLION PROJECTS WITH A DEDICATED SERVER AND GO FOR $100
มุมมอง 101ปีที่แล้ว
PROCESSING 40 TB OF CODE FROM ~10 MILLION PROJECTS WITH A DEDICATED SERVER AND GO FOR $100
DataEngBytes 2023 - SYD-T3-04 - Abhinav Goyal
มุมมอง 52ปีที่แล้ว
DataEngBytes 2023 - SYD-T3-04 - Abhinav Goyal
Tanya, thank you very much for such a wonderful talk. Easy to follow, right amount of information. What do you think of ‘lake house’?
It sounds like the people attracted to your services missed the ethics class.
This guy is the Andrew Tate of Tech Bootcamps, good for him but I just don't think he does anything that matters
Zach
Thanks bro
Great!
Love zach
Nice!
i want to become data scientist as you what is the one thing that will help me in that plz tell me
You've come so far from our days at ucsd. I'm so proud of you
Hi
It would be really helpful if the presentation slide was shared somehow. It is difficult to follow the presentation.
Thank you for your interest! You can access the presentation slides from the session through this link: github.com/ylashin/dataengbytes-bne-2023. Feel free to explore the slides for a better view of the content. Enjoy reviewing the presentation! 👍
I just want to take it upon myself to apologize to everyone & the camera person for not standing in a single place. 🤣
Promo>SM
Excellent presentation thank you!
👌 P r o m o S M
Thanks for sharing the recording. Great to learn about the dynamic column masking on Redshift.
Great talk!
First (also great talk)
Find the APL nerd!
Oihh
Stoked for this!
Enjoyed talk #2. Learnt a lot. Thanks for posting the video.
Hi guys, how can i join into slack group? Thanks
Hi @cyln90, you can do that here: goo.gl/forms/DVNazDmNBg1FFm2X2
Supberb
The Chinese guy too much “you know “
This is pretty cool. #1 view provider Promo-SM!!
This is really awesome
Really helpfull information..
Looking forward to the event
Look at Gladys Berejiklian's face. The elite have threatened her with Gang Stalking and microwave weapons. No doubt. Do this to your App: Every day add new First Name=$coVid$ is a LIE, Last Name=vaccines for $PROFIT$. You can add really long messages. Then every time you check in (even with a qr from internet anywhere) and immediately check out. Keep doing this until you have 50 "friends" of all the illegal things the GOVT (actually the elite controlling the GOVT) are doing. Learn about Propaganda. Turn off TV. TV = brain poison. Take a stand, you feel like a human been again!
Hi sir how-to I am data scientist please guide me let me know
Its like Grammarly for a word document. Nice to know about SQL linting.
That's a good analogy tbh Mei! :)
G'day folks, exciting to see you all!
Thank you for sharing. Very well articulated !
around 27 minutes when talking about ”control plane” not seeming a good name how about “meta-data plane”. that might better describe that its about discovery, lineage, cross-cutting security, light weight governance etc. That is all meta-data.
if possible CDB in Oracle without Goldengate for Debezium
Good Explanation !
54:38 Gian Merlino Some Like It Hot 1:33:06 Caito Scherr Data-driven development in stream processing 2:11:14 Joel Roland & Steve Lee ETL and Data Ingestion Made Easy 2:40:24 Sam Harley How MongoDB Enables Real-Time Data with Event-Driven Architecture 3:16:35 Larene Le Gassick Inclusive Storytelling with Data 3:49:47 Aiko Klostermann Artificial Intelligence? - more like Artificial Stupidity! 4:31:40 Kerry McRae Building data integration services for real-time on AWS
The timestamps in the descriptions are invalid because the hosts are visible after 50 minutes of the video th-cam.com/video/ZQmAVotrzUI/w-d-xo.html
Thanks we will get them updated - those times were valid for the livestream but they need to be updated
can you please enable captions?
2:55:25 DataEngBytes Team - Welcome 3:00:50 Zhamak Dehghani - Introduction to Data Mesh 3:49:14 Charles Feddersen - Developing end-to-end analytics solutions 4:16:32 Dom Colyer - Building a Code Generated Data Platform 4:43:26 Vidya Venugopal - Ever changing data model - Schema management for the future 5:12:19 Louis Lee - Snowflake Cloud Data Platform - Building a Governed Data Lake 5:47:54 Mike Gouline - Data Team as an Optimisation Problem 6:16:13 Yana Segal - Automating ML pipelines with Kubernetes and Airflow 6:44:19 Marta Paes Moreira - Change Data Capture with Flink SQL and Debezium 7:17:51 Robin Moffatt - Building a Telegram bot with Apache Kafka and ksqlDB Your welcome :))
Great video, could you add in the description the links to the correct timing of different presentations?
Hey folks, we are taking questions - on our Slack group :)