Apache Airflow for Data Science #7 - How to Work with REST APIs

Learn to send and receive data between Airflow tasks with XComs, and when you shouldn't use it.
ARTICLE: betterdatascience.com/apache-...
00:00 Introduction
00:38 Configure Airflow for communication with REST APIs
04:06 Write the Airflow DAG
10:28 Outro
FOLLOW BETTER DATA SCIENCE
Website: betterdatascience.com
Medium: / radecicdario
LinkedIn: / darioradecic
FREE “LEARN DATA SCIENCE MASTERPLAN” EBOOK
eepurl.com/g7Ot3L
GEAR I USE
M1 Pro MacBook Pro 16”: amzn.to/3flD5wb
LG 27UL850 4K (Monitor): amzn.to/34QYVpq
Logitech MX Keys (Keyboard): amzn.to/3tH2V6r
Logitech MX Master 3 (Mouse): amzn.to/3Kd2Vk2
Rode NT-USB (Microphone): amzn.to/3FmIgq9
Audio Technica ATH-M40X (Headphones): amzn.to/3Fqj4iM

[@DataNgineering]

Amazing! Thank you for keeping it sweet 'n' simple

[@paleface_brother]

Thanks a lot, Dario! Your lessons are really useful.

[@BetterDataScience]

My pleasure!

[@joa0liveira92]

Great video!

[@AmitSharma-yi9dr]

great explanation, I have been struggling to find a resouce on how to trigger a dag run through Stable Rest Api which will trigger a dag placed in different airlow environment.

[@forumpost8118]

Hi Dario,
Thanks for this. I would like to know what advantages does this simplehttpsoperator offer compared to pulling data from API using Python operator. thanks.

[@WEST8802]

If the method save_posts is failing it's because you are getting no response. Set up the dependencies like this
task_is_api_active >> task_get_posts >> save_reports
You can test the entire DAG with the command: airflow dags test dag_id 2023-03-6
Date should be in the past.

[@jaganmohan4388]

Hi, How to use timeout option in httpsensor task. I have tried in extra_options ={'timeout':120 } also I have set poke_interval=30 means for every 30sec it should poke the endpoint and get the response and wait for 120sec of timeout and exit. But this scenario is not working in my case. Can anybody help.? Much Appreciated and thanks in Advance.

[@jonimatix]

Thanks a lot for the video!
A question, do you must use xcom to fetch api output to save to disk? Is there maybe a more efficient way?

[@BetterDataScience]

It’s up to debate. Return value from a function is automatically pushed to xcoms, so you can access it in other tasks.
You could do everything in a single task, but it’s not the best design approach. One airflow task should do one job only.

[@LuisFelipe-mb9dq]

Hello! First of all thanks for the video! It is very good. I am doing a GET to a public API, when I run it outside Airflow it runs perfectly. But when I do it inside Airflow, just like the video, it returns me a 403 error. What could it be? I am using Airflow with Docker.

[@zeki7540]

same here?

[@praveenreddy4648]

I have a question, how to do 200 asynchronous api calls to different end points?

[@herashak]

try grequests for your calls?