Platform for Genomic Processing Using Airflow and ECS

Presented by Zohar Donenhirsh & Alina Aven at Airflow Summit 2023.
High-scale orchestration of genomic algorithms using Airflow workflows, AWS Elastic Container Service (ECS), and Docker. Genomic algorithms are highly demanding of CPU, RAM, and storage. Our data science team requires a platform to facilitate the development and validation of proprietary algorithms. The Data engineering team develops a research data platform that enables Data Scientists to publish docker images to AWS ECR and run them using Airflow DAGS that provision AWS’s ECS compute power of EC2 and Fargate.

[@talshor5198]

Excellent presentation, thank you!

[@oklander1]

Zohar and Alina, great job!!!

[@ozzyoz6824]

Thank you!

[@KHUNANONWONGOUAN-f8d]

thailand 54000