- 14
- 1 016
DataForge
เข้าร่วมเมื่อ 12 มี.ค. 2024
At DataForge, our mission is to make data management, integration, and analysis faster and easier than ever. DataForge, the Declarative Data Management platform, automates data transformation, orchestration, and observability. By bringing functional programming to data engineering, DataForge introduces the next generation of data transformation solutions. Avoid the pitfalls of procedural scripting and take advantage of modern software engineering principles to automate orchestration, promote code reuse, and maximize observability. Experience a new era of data engineering with DataForge, where functional programming and automation pave the way for scalable data platforms.
DataForge Academy: Custom Loopback
In this video, we walk through a demo of how to create custom loopbacks to automatically ingest a customized dataset after output runs using the DataForge SDK. This video is part of a series of training demos included in the DataForge Academy: learn.dataforgelabs.com
Chapters:
0:00 Intro
0:25 Creating a virtual output
1:33 Setting up an interactive cluster with the DataForge SDK
2:30 Creating a new custom loopback notebook
3:54 Creating the loopback source
4:51 Custom loopback notebook logic
8:33 Incorporating custom parameters
11:00 Pointing the notebook to the custom loopback source
12:10 Running the custom loopback
Chapters:
0:00 Intro
0:25 Creating a virtual output
1:33 Setting up an interactive cluster with the DataForge SDK
2:30 Creating a new custom loopback notebook
3:54 Creating the loopback source
4:51 Custom loopback notebook logic
8:33 Incorporating custom parameters
11:00 Pointing the notebook to the custom loopback source
12:10 Running the custom loopback
มุมมอง: 4
วีดีโอ
DataForge Academy: Custom Post Output
มุมมอง 74 ชั่วโมงที่ผ่านมา
In this video, we walk through a demo of how to create custom post output processes using the DataForge SDK. This video is part of a series of training demos included in the DataForge Academy: learn.dataforgelabs.com Chapters: 0:00 Intro 0:16 Creating a new output 1:09 Setting up an interactive cluster with the DataForge SDK 1:58 Creating a new notebook for the custom post output 6:20 Incorpora...
DataForge Academy: Custom Parse
มุมมอง 44 ชั่วโมงที่ผ่านมา
In this video, we walk through a demo of how to use custom parse processing within DataForge. This video is part of a series of training demos included in the DataForge Academy: learn.dataforgelabs.com Chapters: 0:00 Intro 0:13 Setting up an interactive cluster with the DataForge SDK 1:55 Creating a custom parse source and notebook 4:27 Custom parse notebook logic 10:00 Incorporating custom par...
DataForge Academy: Custom Ingestion
มุมมอง 74 ชั่วโมงที่ผ่านมา
In this video, we walk through a demo of how to create custom ingestions to ingest data from source systems that DataForge doesn't have pre-built connectors or customized datasets using the DataForge SDK. This video is part of a series of training demos included in the DataForge Academy: learn.dataforgelabs.com Chapters: 0:00 Intro 0:26 Setting up an interactive cluster with the DataForge SDK 4...
Engineering Choices and Stage Design with Traditional ETL
มุมมอง 2112 ชั่วโมงที่ผ่านมา
Join Joe Swanson, Co-founder and Lead Developer at DataForge, as he demonstrates a BI data model build using the Coalesce ETL platform, covering essential decision points and the different approaches when using a more traditional toolset. 📄 Blog: www.dataforgelabs.com/blog/stage-choices 🌐 Website: www.dataforgelabs.com 🎥 Free Trial: accounts.cloud.dataforgelabs.com/signup Video Chapters: 0:00 I...
Data Transformation at Scale with DataForge Cloud: Rule Templates & Cloning Explained
มุมมอง 80วันที่ผ่านมา
In this video, Vadim Orlov, CTO and Co-founder of DataForge, provides an in-depth demonstration of how DataForge Cloud streamlines data transformation by promoting reusable code architecture. He covers the use of rule templates and the cloning feature, explaining how these tools save time and reduce complexity in managing data across different platforms. With step-by-step walkthroughs, Vadim de...
Mastering Schema Evolution & Type Safety with DataForge
มุมมอง 8721 วันที่ผ่านมา
In this video, Vadim Orlov, CTO and co-founder of DataForge, demonstrates how to handle complex schema evolution and ensure type safety in your data pipelines. Learn how DataForge simplifies the management of data changes, from schema modifications to type validations, ensuring robust protection against runtime failures. Watch as Vadim walks through a live demo showcasing real-time validation, ...
Introduction to DataForge Part 2: Understanding Declarative Object Models for Data Transformation
มุมมอง 19521 วันที่ผ่านมา
In Part 2 of our DataForge Framework series, Matt Kosovec, CEO and co-founder of DataForge, takes you deeper into the framework's object model, designed to simplify data transformation. This session builds on Part 1, so be sure to check that out first for essential context on foundational concepts like column-pure and row-pure structures. In this video, we explore the specifics of DataForge's r...
Introduction to the DataForge Column-Pure Data Transformation Framework
มุมมอง 121หลายเดือนก่อน
Welcome to DataForge! In this video, Matt Kosovec, CEO and co-founder of DataForge, walks us through the fundamentals of the DataForge framework, including its core concepts, modern approaches to data engineering, and how it addresses traditional challenges. Learn how DataForge simplifies complex data pipeline management by leveraging column-level transformations to reduce scope, streamline dev...
Stream Processing in DataForge: Real-Time Data Integration and Enrichment
มุมมอง 100หลายเดือนก่อน
In this video, Joe Swanson, co-founder and lead software engineer at DataForge, walks you through the exciting new Stream Processing capabilities in DataForge. Learn how to integrate real-time data from Confluent Cloud and Kafka with batch data for enriched, real-time analytics. Joe demonstrates how to pull data from a Kafka topic, enrich it with historical data in DataForge, and then push the ...
Simplifying Complex Data Structures with DataForge: Subsources Feature Demo
มุมมอง 88หลายเดือนก่อน
In this video, Vadim Orlov, CTO and Founder of DataForge, walks you through the latest DataForge feature - Subsources. This feature simplifies working with complex data structures such as nested JSON arrays without the need for normalization. Join Vadim as he demonstrates how to streamline data transformations and generate outputs using DataForge, enhancing your data engineering workflows. Try ...
DataForge vs. Databricks Delta Live Tables for Change Data Capture
มุมมอง 1022 หลายเดือนก่อน
This video compares two approaches to Change Data Capture (CDC), a data engineering pattern used to capture and track changes in a database table. Databricks Delta Live Tables (DLT): Requires manual configuration of the pipeline for pooling and updating table snapshots from the source database. Involves writing Python code to define the source table, target table, and how to apply snapshots. Ca...
DataForge Cloud Demo
มุมมอง 1666 หลายเดือนก่อน
DataForge, the Declarative Data Management platform, automates data transformation, orchestration, and observability. By bringing functional programming to data engineering, DataForge introduces a new paradigm for building data solutions. Avoid the pitfalls of procedural scripting and take advantage of modern software engineering principles to automate orchestration, promote code reuse, and max...
The Problem with Pipelines: Building Extensible Data Code - Matthew Kosovec - DataTune 2024
มุมมอง 578 หลายเดือนก่อน
With modern tools and technologies, it is easier than ever to quickly develop data logic to deploy advanced ETL/ELT solutions from scratch. Adding or modifying logic on an existing platform of thousands of pipelines and buried business rules? Nearly impossible. Join us for a deep dive into foundational challenges with modern data engineering development and a proposed solution for enabling exte...
Great insights on managing data transformation at scale! The overview of rule templates and schema evolution really helped clarify some complex concepts. Thanks for sharing this, DataForge team!
Amazing product! I think this would be a strong competitor to DBT