Thank you for the training :-) In cases that DAG runs on every 10 minutes and use XCom, I assume Airflow will end up with many XCom values that are from the past executions. Is there a way to limit the number of XCom values for one task or DAG to only keep the X last values? I was thinking of using external storage (Like GCS or S3) for keeping values between DAGs instead of using XCom, but wondering what do you suggest to use?
Good remark! Indeed, XComs are not automatically removed for you so you have to take care of them. I don't think there a way to limit the number of XCom values. Usually, I recommend to create a DAG in charge of cleaning your XComs that you will trigger once a week for example. Good idea, I'm actually working on a tutorial for setting uo XCom with AWS S3 :) This is useful if you need to share data exceeding the hard limit of your DB. Marc Lamberti
Thank you for the training :-) In cases that DAG runs on every 10 minutes and use XCom, I assume Airflow will end up with many XCom values that are from the past executions. Is there a way to limit the number of XCom values for one task or DAG to only keep the X last values?
I was thinking of using external storage (Like GCS or S3) for keeping values between DAGs instead of using XCom, but wondering what do you suggest to use?
Good remark! Indeed, XComs are not automatically removed for you so you have to take care of them. I don't think there a way to limit the number of XCom values. Usually, I recommend to create a DAG in charge of cleaning your XComs that you will trigger once a week for example.
Good idea, I'm actually working on a tutorial for setting uo XCom with AWS S3 :)
This is useful if you need to share data exceeding the hard limit of your DB.
Marc Lamberti