- 57
- 1 751 349
Prefect
United States
เข้าร่วมเมื่อ 26 ก.พ. 2020
Prefect Cloud is a workflow orchestration and observability tool, designed for modern infrastructure and powered by the open-source Prefect engine.
How Relativity Prefects: Orchestrating Generative AI Models
In this video, Relativity shares how they Prefect. Their organization has used Prefect to #orchestrate their #generativeai models.
มุมมอง: 168
วีดีโอ
Introducing Prefect 3.1.0: Using prefect.toml
มุมมอง 905หลายเดือนก่อน
Introducing prefect 3.1.0! This release comes with many bug fixes and enhancements, but here we'll go over the prefect.toml file. This is used for declaratively defining runtime settings in a way that's easy to grok, share, and even safe to commit to version control (as long as you're not setting any secret values of course). Fun fact: the setting PREFECT_TASK_RUNNER_THREAD_POOL_MAX_WORKERS use...
Engineering Robust Data Pipelines: Rec Room's Approach with dbt and Prefect
มุมมอง 126หลายเดือนก่อน
Explore Rec Room's innovative data orchestration using dbt and Prefect on Azure. They'll showcase their end-to-end workflow integrating data ingestion, quality checks, ETL, and ML inference. Learn about their FlowFactory for streamlined dbt usage, core job architecture, and how they leverage Prefect's Automations. Gain insights on scaling workflows while maintaining flexibility and reliability.
Orchestrating Fashion's Data Runway: How Rent the Runway Leverages Prefect and dbt
มุมมอง 107หลายเดือนก่อน
Discover how Rent the Runway, a tech-driven fashion rental company, revolutionized its data pipeline using Prefect and dbt. We'll explore how the company orchestrates models, optimizes resource allocation through containerization, and streamlines data ingestion from diverse sources. Join us as we share key learnings from our journey to enhance visibility and efficiency in our data transformatio...
Supercharging dbt: 3 Ways Prefect Enhances Your Data Workflows
มุมมอง 142หลายเดือนก่อน
Discover how Prefect amplifies dbt's in-warehouse prowess by managing crucial processes beyond transformations. We'll explore event-driven workflows, data freshness-based job management, and seamless integration with downstream processes like ML jobs. Learn how combining Prefect with dbt creates a more robust, efficient, and interconnected data ecosystem for your organization.
Better together, dbt and Prefect
มุมมอง 247หลายเดือนก่อน
Explore the powerful combination of dbt (data build tool) and Prefect in creating efficient, reliable data workflows. Join Alex Welch, Head of Data at dbt Labs, as he demonstrates how dbt's strengths in data transformation, modeling, and accessibility complement Prefect's capabilities in workflow orchestration and scheduling. By integrating these tools, data teams can achieve greater visibility...
ControlFlow: Orchestrating, Observing and Trusting Agentic AI Workflows
มุมมอง 322หลายเดือนก่อน
Large Language Models have completely redefined how we can build and orchestrate workflows. AI Agents can be dynamically dispatched to solve tasks with tools or choose what task to execute next. But there’s no free lunch: agentic workflows, while powerful, are notoriously hard to orchestrate, observe, and trust. In this talk, we’ll introduce ControlFlow - a framework powered by Prefect that mak...
Building Blocks for Orchestration Platforms by Wendy Tang, Cash App
มุมมอง 113หลายเดือนก่อน
Building Blocks for Orchestration Platforms by Wendy Tang, Cash App
Prefect Summit Industry Keynote by Chris Riccomini
มุมมอง 523หลายเดือนก่อน
Prefect Summit Industry Keynote by Chris Riccomini
Building Data Pipelines for Workforce Analytics by John Kang, Cox Automotive
มุมมอง 74หลายเดือนก่อน
Building Data Pipelines for Workforce Analytics by John Kang, Cox Automotive
From Chaos to Clarity: Managing Data Lakes with Prefect
มุมมอง 148หลายเดือนก่อน
From Chaos to Clarity: Managing Data Lakes with Prefect
Prefect 3.0: A Framework to Build Resilient Pipelines
มุมมอง 949หลายเดือนก่อน
Prefect 3.0: A Framework to Build Resilient Pipelines
Task Failed Successfully: Navigating Failure with Transactional Orchestration
มุมมอง 182หลายเดือนก่อน
Task Failed Successfully: Navigating Failure with Transactional Orchestration
Trigger your code to run when a webhook fires
มุมมอง 3.2Kปีที่แล้ว
Trigger your code to run when a webhook fires
Dynamic Event-driven Workflows with Prefect Cloud
มุมมอง 173Kปีที่แล้ว
Dynamic Event-driven Workflows with Prefect Cloud
Getting Started with Workflow Orchestration
มุมมอง 1.5Mปีที่แล้ว
Getting Started with Workflow Orchestration
How to Deploy & Schedule Data Workflows w/ Prefect
มุมมอง 9Kปีที่แล้ว
How to Deploy & Schedule Data Workflows w/ Prefect
An Introduction to Your First Prefect Flow
มุมมอง 17Kปีที่แล้ว
An Introduction to Your First Prefect Flow
How to create multiple deployments from one project in Prefect
มุมมอง 3.7Kปีที่แล้ว
How to create multiple deployments from one project in Prefect
The Orchestration Education Station at the Modern Data Stack Conference 2023
มุมมอง 702ปีที่แล้ว
The Orchestration Education Station at the Modern Data Stack Conference 2023
How to run Prefect in Azure Container Instances (ACI) with Prefect agents
มุมมอง 2Kปีที่แล้ว
How to run Prefect in Azure Container Instances (ACI) with Prefect agents
An intro into setting up ECS and Prefect
มุมมอง 3.3Kปีที่แล้ว
An intro into setting up ECS and Prefect
commands used in the video no longer works, seems to be outdated. The video also skip over important details on the setup, it is no clear where the kubernetes is hosted in the videos as well as the docker image registry
Hello @tonylim_nvd! We have plans to update the content on Kubernetes and Docker early in 2025. Our team would be happy to help you get started in the meantime. If you want to join our Slack community, our team will jump on any questions that you have: communityinviter.com/apps/prefect-community/prefect-community
Thanks for the video
Glad you enjoyed it!
come one come all to conquer ETL-ish flows! 😅
There is a project shown in the video, but at least for me, it's hard to read. Is there a chance this code or project is available online as a reference? I am particularly interested in the integration with Slack.
Hey Datamonkey! We don't have that project available as a reference currently. However, controlflow.ai/ has some code snippets to help you get started.
Why you remove deployments, why? Is serve() right way to run python flows in production containers?
hi Pravin! we have not removed deployments! you can `your_flow.deploy()` to create deployments or use `.serve` which is like a deployment combied with a process worker (easy mode!) I would encourage you to checkout this playlist on getting started with prefect if you want to see more details th-cam.com/play/PLWkgBUKPlwvCV5FdBGsDE16K2DSelOy9i.html
@@natefromprefect5895thanks let me see this, I am running prefect in my org on containers in kubernetes, running flows within containers
Do you recommend using serve method in production, for the context I'm deploying flows as pods in kubernetes cluster
For most use cases, serve is sufficient. You can read more about serve and deploy here: docs.prefect.io/3.0/deploy/infrastructure-concepts/work-pools
Is it cloud only feature?
Webhooks are a cloud-only feature.
What is the version?
This video is running on an older version of Prefect. The newest version of Prefect is 3.0.10. You can check it out here: github.com/PrefectHQ/prefect
If you are using a git repo as the code source like the demo here, it seems that the Python script needs to live in the parent directory of the repo and not in a subdirectory. Otherwise, the Work Pool throws an error "FileNotFoundError". If this is truly the only way to do it, it's definitely not the neatest implementation. But I'd guess there must be a way to direct Prefect to the subdirectories somehow, which I haven't figured out yet. If anyone knows, please share your knowledge.
Yeah same is really frustrating
Have you found the solution??
I found the solution you should run your deployment (prefect deploy) in the root directory of your project, I have also do the same for the worker pool
We are building a similar but different solution, we are in early beta, its called abysshub. If anyone is looking to become an early adopter and provide some feedback :)
Thanks for sharing Chris! This video was really well done and helped solidify the blog post, email, and docs online.
I only wish i could downvote this multiple times.
amazing!
it's outdated now
ء
Not fair
Hi
Hi
Hi
attn:@FCc intent from#agencysqued thisgenerationmineashouseholdcares,whileSuspectsamongstfamilias other of therein previous generations fiscal theirs minofamiluasthese ayudas,gsthersprecinterrorssitemaps
does this work without prefect cloud? My company does not allow using third-party cloud solution due to data privacy
Blocks are repos that you can install and initiate, for instance, the prefect-kubernetes library: github.com/PrefectHQ/prefect-kubernetes. You can find them in our public repos.
0:56
😂🎉ci0
Sorry
I wisk i could downvote this multiple times. Ill just block
Got the damned ad AGAIN!
❤
Iiiiiioou
Leeeee
youtube.com/@user-kt4vu5gm4t?feature=shared@@stephaniesalb4518
@@stephaniesalb45181:34
Can you please re-upload this so that it is possible to see the screen?
What are some of the major "tasks" that can be done. What data does it collect. Can it process information in another app to mask functions. I believe someone has used it to full their locations on Google, is this possible, what are the main reasons for this.
Hempseed.boymjk 0:23
The problem with this approach is that now your parameters are static. How can I have multiple deployments in a single github repository where the params are dynamic?
Thank you 60 blessings
Thanks a lot, been readying the documentation the whole day and couldn't find an example how to deploy a flow on a process work pool, all examples are to deploy on a docker work pool
Hi! Have you seen our newer flow.deploy() method? Check it out here: docs.prefect.io/latest/getting-started/quickstart/
That is all right
I have questions
Would be nice to have the code so we can follow with the tutorial :)
That's the ticket lolololol
Thanks for the video Jeff! I didn't realize you could hit a webhook from github.
Hey pal there Is no gear icon to click on
Yes
//
Is this real?
Seems like there’s overlap between prefect deployment apply and prefect deploy. Still trying to figure out the difference
Are you able to self-host a work pool in a long running container, or do you have to use Prefect Cloud for the work pool hosting?
I'm only 9 minutes in, but looking at the description, yeah that's my question as well. Is all of this relevant to Open Source Prefect?
@mdso
Even more lightweight and better separation of concerns and cloud best practices would be that it requires no code at all (other than to set defaults). That the ENVIRONMENT vars are just available as an environment variable. Does it work that way also?
Sweet. So what is happening under the covers when you “prefect deploy” in this case with one worker and two entry points? Is there one process to execute both (so they can’t run simultaneously or ?
Regardless of which worker type you choose, flow runs started from deployments run independently, so there's no restriction on the quantity of entrypoints. When that independent flow run starts, it'll just look for the entrypoint function in the specified path and execute it if found.
@@KevinGrismorePrefect Thx! Will a worker spawn multiple simultaneous processes if two entry points are executed simultaneously? How do you allow/restrict this?
@@PatrickSteil Yep, if on a process worker, it'll run multiple subprocesses. You can constrain the number of concurrently running flows on a work pool or one of the work queues within the pool using its concurrency limits. If you were to reach a point where you need more concurrent compute resources than a process worker can provide, switching to a worker type that utilizes dynamically scaling infrastrucutre, like Kubernetes, ECS, Cloud Run, or ACI, is recommended.
Terrible video for the following reasons: - doesn't show which directory Prefect is installed (not covered in the documentation Quickstart) - doesn't show where to drop the Python script with Flow functions (also not covered in the documentation Quickstart) I honestly think you could do better.
You should probably watch a beginners python course instead. Packages are installed in the environment that is activated in the terminal.
@@jpet11 I think he's a beginner based on his comment
❤
What does a Chanel offer tou
Nice 👍👍
th-cam.com/video/AjYHBwH2Mtc/w-d-xo.htmlsi=jO36tisO2dvLwV8T
Hey there is just one more thing my account to prove it to me so much for my new job investing goods and services produced by the support of the year edition for sale near the time change and payment for sale near the year edition