Airflow Docker Set Up Guide

แชร์
ฝัง
  • เผยแพร่เมื่อ 19 ต.ค. 2024

ความคิดเห็น • 25

  • @dtsleite
    @dtsleite 2 ปีที่แล้ว

    Perfect tip, works like a charm! Thanks!

  • @vjin3129
    @vjin3129 ปีที่แล้ว

    This is perfect, I also got the same error message with unexpected character "-" in variable name near "-e \"AIRFLOW_UID=$(id -u) \
    AIRFLOW_GID=0\"
    " when running docker-compose up airflow-init. After change the .env file. it works like a charm.

    • @maybenew7293
      @maybenew7293 ปีที่แล้ว

      how did you change it?

    • @vladdank9158
      @vladdank9158 ปีที่แล้ว +2

      @@maybenew7293
      The issue is that the echo command actually adds the "-e" and the quotes to the file. Maybe a bug with cmd or something Microsoft changed since 2021 and the echo command.
      The .env file text should look like this:
      AIRFLOW_UID=50000
      AIRFLOW_GID=0
      Nothing else.
      but then this didn't work so I looked for an hour and found this solution to launch it as root user:
      AIRFLOW_UID=50000
      AIRFLOW_GID=0
      I tried to get my own proper UID and GID but that didn't work out.

  • @Omzodijacky
    @Omzodijacky ปีที่แล้ว

    thanks for the video . if you want to install additional python packages to use in your airflow container , how can you do that ? can you add a requirement.txt in docker-compose ?

  • @niluparupasinghe8581
    @niluparupasinghe8581 ปีที่แล้ว

    Simple and awesome...

  • @yashsalvi3653
    @yashsalvi3653 ปีที่แล้ว

    Could you please share how long does it take for the installation process to complete? I have been waiting for an hour for the installation process to complete. I am in the installation step where container airflow-redis-1 and airflow-postgres-1 indicate running.

  • @rahmansahinler1
    @rahmansahinler1 ปีที่แล้ว +1

    If you encounter following error:
    curl: (3) URL using bad/illegal format or missing URL
    you need to use double quotes("). I copied the current code from official airflow webpage and they have single quotes.

  • @umaphanim8894
    @umaphanim8894 2 ปีที่แล้ว +6

    hi vincent i am getting an error unexpected character "-" in variable name near "-e \"AIRFLOW_UID=$(id -u) \
    AIRFLOW_GID=0\"
    " when running docker-compose up airflow-init can you please help how to resolve this error do i need to correct anything in yaml file.

    • @ParvezAlam-tr6zj
      @ParvezAlam-tr6zj 2 ปีที่แล้ว +4

      yes it's because of the .env file.
      to resolve it.
      open .env file on any editor. you will find -e \"AIRFLOW_UID=$(id -u) \
      AIRFLOW_GID=0 something like text written on it which should not be like that.
      replace the whole text with following
      AIRFLOW_UID=50000
      AIRFLOW_GID=0
      Hope this helps.

    • @pravinbaste009
      @pravinbaste009 2 ปีที่แล้ว +1

      @@ParvezAlam-tr6zj Thank you so much..!! It help me..!!👏

    • @osamaelgendy7807
      @osamaelgendy7807 ปีที่แล้ว

      @@pravinbaste009 I wrote this and it still gives me the same error
      unexpected character "\x00" in variable name near "\xff\xfeA\x00I\x00R\x00F\x00L\x00O\x00W\x00_\x00U\x00I\x00D\x00=\x005\x000\x000\x000\x000\x00
      \x00
      \x00A\x00I\x00R\x00F\x00L\x00O\x00W\x00_\x00G\x00I\x00D\x00=\x000\x00"

    • @pravinbaste009
      @pravinbaste009 ปีที่แล้ว +1

      @@osamaelgendy7807 then create. env in existing directory. In.env file direcly mentioned below things only... It will surely work....!!
      AIRFLOW_UID=50000
      AIRFLOW_GID= 0

    • @osamaelgendy7807
      @osamaelgendy7807 ปีที่แล้ว

      @@pravinbaste009 yes I did this it worked surprisingly, now I have Permission denied: '/opt/airflow/logs/scheduler/2022-11-03' error and I can't grant permission, apparently the password is not airflow

  • @kenjskjsk
    @kenjskjsk 3 ปีที่แล้ว

    I have been using this official image since 1 month but my problem is how to share csv file. When I write simple read.csv command in my Python function and run the dag I get error which is file not found. I have been trying to solve it since many days but couldn’t able to find the answer. Please let me know if you have any clues on it.

  • @OvoOkpubuluku
    @OvoOkpubuluku 2 ปีที่แล้ว

    what do I do, getting an error message "no matching entries in password file"? Here's the console message in detail:
    Error response from daemon: unable to find user $(id -u)
    AIRFLOW_GID=0: no matching entries in passwd file

  • @samsal073
    @samsal073 ปีที่แล้ว

    Hi Everyone,
    Can someone help me please,
    I got the docker compose and I got the yaml file but when I try to run Docker-Compose up airflow-init , I receive the following error:
    no matching manifest for windows/amd64 10.0.14393 in the manifest list entries

  • @nghianguyen9439
    @nghianguyen9439 3 ปีที่แล้ว

    How can I add a Python module on the docker container in Windows? I need to use a Python library that is not inside the airflow.

  • @GordoCantaPosta
    @GordoCantaPosta ปีที่แล้ว

    the only missing thing is pgadmin I guess

  • @PravinMaske1
    @PravinMaske1 ปีที่แล้ว +2

    echo -e "AIRFLOW_UID=$(id -u)
    AIRFLOW_GID=0" > .env worked for me