วีดีโอ

Did I miss something? The triggers came out of nowhere

I have the Issue with the connection string!

your head covers the dependency tree :(

Where did you copy your connection string from in the export phase at 0:57 in the video?

Is this an unreleased version? I don’t have AI pipeline capabilities in my Mage deployment

Thank you for shared

Hi! somebody can help me to solve this error "magic-1 exited with code 132" please

Great product. Will definitely check it out and see if this could be integrated in my current work.

is it possible (and a good practice) to create like 400 blocks?

Matt, would you say that this would be the most efficient way to implement dependency between two mage data pipelines? I mean, I need to schedule a pipeline in mage to run after another mage data pipeline. What I think is that, in my first pipeline, in the last block, I could run an API call to trigger the second pipeline. This seems like a super handy tool in mage for that purpose.

OMG, that was exactly what I needed. Thanks, Matt.

For people who experience the error "NotFound: 404 Not found: Dataset {project_id}:{schema_name} was not found in location US Location: US". My conclusion is that, I don't think Mage by itself has ever had the ability to create the cloud resource on the fly. This video just forgot the creation part and misled us , at 4:15, to think that Mage would create the BigQuery dataset for us, but there was no before-and-after comparison to verify this claim. It is likely that Matt just created this dataset manually beforehand. I spent an hour checking the videos, slack messages and FAQs for this error. Notice that, even for GCS, we have to manually create the bucket in the early section of the "2.2.4 Configuring GCP" video before we upload anything to it. Similarly, we should create the BigQuery dataset manually.

Ben's face while Tommy was tweaking during the first 30 seconds is priceless.

custom sink in Streaming pipeline 💪💪

Thanks for this very instructive series Matt! Very enjoyable! Hope you keep on doing such a great job at Replit. All the best!

Great stuff! Just a quick note: I think the UI changed a bit since the video was created. In my case, I found the button to enable split view under the "View" menu in the top left menu.

Is this still the default versioning mode or has it been replaced by the git versioning app? I cannot find the view that is shown in this video anywhere.

Hi matt, can you drop the link on how to whitelist the desired IPs?

There are a lot of features in this release! A lot of hidden ones too 😉

Thanks for the lections! 0:08 At first why do we initialize GDP block based on data_preparation pipeline while we are building new sklearn_training pipeline? Secondly launching GDP block returns an error AttributeError: 'NoneType' object has no attribute 'to_dict' in trigger_and_check_status function: @safe_db_query 135 def trigger_and_check_status( 136 global_data_product: GlobalDataProduct, (...) 149 remote_blocks: List[Union[Dict, RemoteBlock]] = None, 150 ): 151 tags = merge_dict(logging_tags, dict( 152 block_uuid=block.uuid if block else None, --> 153 global_data_product=global_data_product.to_dict(), 154 round_number=round_number, 155 ))

I'm not trying to be mean and I appreciate your effort but I want to leave a honest review. In this video I've seen musical intro and outro (about 10 secs which I personally don't understand), and an explanation on how to create files in the Mage text editor. It's a little bit uninformative, taking into account that the title mentions "data preparation". Again, I mean no disrespect - I'm just drawing your attention to what seems obvious and strange to me

Hey there, What I noticed is that when some exception is thrown (let's say, `TypeError: Cannot cast DatetimeArray to dtype float64`) on my attempt to build a time series chart, the UI doesn't show anything. So I as a user must go to the logs and figure out what went wrong

Hey. Thank you. I have a question about the line code X_val.shape[0] == 51492 within the @test . the number 51492 could be varied from 1 dataset to another one. so in this situation the test wouldnt pass?

wtf xD

how backfills is different from schedular in mage data pipeline , please answer me

what am I doing wrong, I downloaded the json file and moved it to my magezoomcamp folder but within mage and the test pipeline it says file note found... I literally copied its path how is that possible

ConnectionError: ('Connection aborted.', TimeoutError('The write operation timed out')) :/

EXcellent section!!! Thank you Matt and DTC

hello team as i am unable to preare the branching mentined in the video can you please update this video according to the version v0.9.70

When I tried to export the data into BigQuery using either SQL or Python data exporter, I got the error, "NotFound: 404 Not found: Dataset<project_id>:<schema_name> was not found in location US. Location: US Job ID: <job_id>" I'll appreciate any help on how to resolve this. PS: I selected US multi-region when I created my bucket. My GOOGLE_LOCATION in my io_config.yaml file is US. My default code region in BigQuery is set to us-central1 (Iowa) (there's no option to select only US)

The 30MB file size for the NY taxi parquet file seems really low for 1 million+ rows - is there an issue here? It also says that the file type is "text/plain". When I try to load it in the next video it is saying there is no data. Can you confirm?

can you demo a streaming pipeline with clickhouse sink?

This will be used in fallout shelters... to return magic into the world!

Why they use "{{}}" for the postgre configuration? I am begineer so i don't know what happen exactly

excuse me, i have error, "Error loading project configuration", how to fix this? thanks

Can we all have an example with pyarrow, this would be great for projects in the DTC course and personally it would really help as I cannot make it work ...plz plz

This feature saves me when I need to securely transfer multiple media files from one location to another (SCP), especially when the files are spread across different directories and across the network. Typically, one would have to loop through each file, but secure copy (SCP) operations can fail for various reasons, and dealing with exceptions can be error-prone. An alternative approach would be to use an API trigger (like the one provided by Mage AI) to receive POST requests for each file that needs to be securely copied. However, this method can be highly resource-intensive, particularly when dealing with thousands of pipeline runs. Dynamic block, which allows me to manage source and destination specifications in a DataFrame and then dynamically execute SCP operations without having to handle exceptions manually (thanks to the "retry incomplete block" feature), is a game-changer. Not only does it save me from the hassle of exception handling, but it also keeps computational resources in check by avoiding unnecessary pipeline triggers. Thank you for your world-class project. Keep up the excellent work.

Looks amazing! Jupyter notebooks + airflow on steroids))

Thanks Matt! What a great way to introduce into mage ai!

Hi, I tried to pull from the website for NY taxi data and all appears to be in Parquet, when I try using read_parquet the kernel crashes, any suggestions Also how do you parse dates in Parquet read

Hello, I modified the file io_config.yaml as instructed. Then, I tried to create an SQL data loader and I got the error message below : while parsing a block mapping in "<unicode string>", line 137, column 3: POSTGRES_CONNECT_TIMEOUT: 10 ^ expected <block end>, but found '<scalar>' in "<unicode string>", line 138, column 22: POSTGRES_DBNAME: ""postgres"" ^ Traceback (hide traceback) Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1786, in _execute result = await result File "/usr/local/lib/python3.10/site-packages/mage_ai/server/api/v1.py", line 61, in get return await execute_operation(self, resource) File "/usr/local/lib/python3.10/site-packages/mage_ai/api/views.py", line 63, in execute_operation raise err File "/usr/local/lib/python3.10/site-packages/mage_ai/api/views.py", line 56, in execute_operation response = await base_operation.execute() File "/usr/local/lib/python3.10/site-packages/mage_ai/api/operations/base.py", line 92, in execute result = await self.__executed_result() File "/usr/local/lib/python3.10/site-packages/mage_ai/api/operations/base.py", line 459, in __executed_result return await self.__create_or_index() File "/usr/local/lib/python3.10/site-packages/mage_ai/api/operations/base.py", line 544, in __create_or_index return await self.__resource_class().process_collection( File "/usr/local/lib/python3.10/site-packages/mage_ai/api/resources/BaseResource.py", line 192, in process_collection res = await res File "/usr/local/lib/python3.10/site-packages/mage_ai/api/resources/DataProviderResource.py", line 51, in collection profiles = list(yaml.safe_load(template_render(content)).keys()) if content else [] File "/usr/local/lib/python3.10/site-packages/yaml/__init__.py", line 125, in safe_load return load(stream, SafeLoader) File "/usr/local/lib/python3.10/site-packages/yaml/__init__.py", line 81, in load return loader.get_single_data() File "/usr/local/lib/python3.10/site-packages/yaml/constructor.py", line 49, in get_single_data node = self.get_single_node() File "/usr/local/lib/python3.10/site-packages/yaml/composer.py", line 36, in get_single_node document = self.compose_document() File "/usr/local/lib/python3.10/site-packages/yaml/composer.py", line 55, in compose_document node = self.compose_node(None, None) File "/usr/local/lib/python3.10/site-packages/yaml/composer.py", line 84, in compose_node node = self.compose_mapping_node(anchor) File "/usr/local/lib/python3.10/site-packages/yaml/composer.py", line 133, in compose_mapping_node item_value = self.compose_node(node, item_key) File "/usr/local/lib/python3.10/site-packages/yaml/composer.py", line 84, in compose_node node = self.compose_mapping_node(anchor) File "/usr/local/lib/python3.10/site-packages/yaml/composer.py", line 127, in compose_mapping_node while not self.check_event(MappingEndEvent): File "/usr/local/lib/python3.10/site-packages/yaml/parser.py", line 98, in check_event self.current_event = self.state() File "/usr/local/lib/python3.10/site-packages/yaml/parser.py", line 438, in parse_block_mapping_key raise ParserError("while parsing a block mapping", self.marks[-1], yaml.parser.ParserError: while parsing a block mapping in "", line 137, column 3: POSTGRES_CONNECT_TIMEOUT: 10 ^ expected , but found '' in "", line 138, column 22: POSTGRES_DBNAME: ""postgres"" I've also submitted the error on the "bugs-trouble-shoot-questions" Slack channel Can you please help me ?

The instructor video hides the down left corner of the slides 😞

Wanted to get familiar with Mage some time ago, after using airflow for a while. Great Intro! Will the code be available for all of the videos in the DTC course?

Thanks for the demo, is there an API to check the status of an API triggered pipeline other than the one that returns the logs?

0:11 I deleted Discord XD XD XD

Can we run dbt with mage locally?

Question: You're using docker to run an image of Mage. Within Mage you're setting up this ETL pipeline to get things uploaded to GCP. If you kill the Docker container where you're running Mage, won't your Mage files (i.e., your ETL pipeline) be lost? So you'd have to rewrite them again if you want to do the exact same thing right? How do you circumvent this?

I refactored transform section this way: cond = data['passenger_count'] > 0 print(f'Rides with zero passengers: {len(data[~cond])}') return data[cond]

There's a lot to be desired in this last section on deployment.