Scheduling in Airflow

แชร์
ฝัง
  • เผยแพร่เมื่อ 21 ก.ค. 2024
  • The flexibility and freedom that Airflow offers you is incredible, but to really take advantage of it you need to master some concepts first, one of which has just been released in Airflow 2.2 By the end of the webinar, you will be able to define schedule intervals that you thought were impossible before.
  • บันเทิง

ความคิดเห็น • 6

  • @maximilianrausch5193
    @maximilianrausch5193 2 ปีที่แล้ว +4

    Is it possible to use time tables to schedule tasks at the start of the interval now instead of the end ?

  • @ayushikhanna1094
    @ayushikhanna1094 5 หลายเดือนก่อน

    Is there any option available in airflow ui to auto trigger.

  • @1UniverseGames
    @1UniverseGames 2 ปีที่แล้ว

    I have a question, can you show me like I have a DAG trained model which done in deep learning technique. Now how can I integrate my model into Airflow for dag job scheduling. Can you show a demo using such model.

  • @cloudlover9186
    @cloudlover9186 6 หลายเดือนก่อน +1

    I am running in to below problem , will it be acheived by time table concepts . i have same dag which should satisfy the below schedule intervals
    schedule interval = '30 1,4,7,10,13,16,19,22 * * *' & '00 3,6,12,15,18,21,00 * * *' , Please help and guide.

    • @Astronomer
      @Astronomer  6 หลายเดือนก่อน

      Would you mind sharing info on what type of scheduling interval you'd like to achieve? Not sure what it is based on that string unfortunately!

    • @cloudlover9186
      @cloudlover9186 6 หลายเดือนก่อน

      @@Astronomer Hi , we are in process of changing a daily schedule to 90 mins frequency dag and expectation of dag to run at 00:00, 01:30 , 03:00 so on and also another new dag which is of same 90 mins frequency should run at
      00:20, 01:50, 03:10 etc.., point is if i have hard coded start date as future date for example today is 01/10 i will hard code as 01/11 (2024/01/11,00,00) any future change is not impacting the start date schedule , having said we have advised to research more not to hard code start date .FYI we are using timedelta(minutes=90) in schedule interval attribute.
      if we use current date logic , during deployment time ,(deployment time > start date time) it is executing immediatley , how we can over come this , please help.