We have already storaged metadata of Airflow in postgres. But When I tried to connect to collect metadata, it did not work. Can you give me some tips to do it?
In 16.08 I'm not able to click the test connection and showing "To set up metadata extraction through UI, you first need to configure and connect to Airflow" can you please help me on that
How can i assess the correct sizing of my virtual linux machine for moderate workloads, if i want to install airbyte for data ingestion (around 10 to 20mb daily), dbt tool with simple sql queries, airflow to start dbt queries, openmetada to handle the meta data of my infrastructure and apache superset for dashboard visualization? i was wondering that for testing env, i could install all these tools on a vm, while for postgres i would take another vm with 3 docker container of postgresql each for raw,curated and publish data. I was watching the hw requirements for each tool and considering that they are not running togheter, maybe a 4vcpu and 12 gb of ram would be good, what do you think about that?
Thanks a lot for your work! It is very helpfull!
Amazing video! Thank you very much!
Would you recommend a 8GB or 16GB VPS for personal lab projects?
16gb
We have already storaged metadata of Airflow in postgres. But When I tried to connect to collect metadata, it did not work. Can you give me some tips to do it?
In 16.08 I'm not able to click the test connection and showing "To set up metadata extraction through UI, you first need to configure and connect to Airflow" can you please help me on that
How can i assess the correct sizing of my virtual linux machine for moderate workloads, if i want to install airbyte for data ingestion (around 10 to 20mb daily), dbt tool with simple sql queries, airflow to start dbt queries, openmetada to handle the meta data of my infrastructure and apache superset for dashboard visualization? i was wondering that for testing env, i could install all these tools on a vm, while for postgres i would take another vm with 3 docker container of postgresql each for raw,curated and publish data. I was watching the hw requirements for each tool and considering that they are not running togheter, maybe a 4vcpu and 12 gb of ram would be good, what do you think about that?
You host it yourself on your own pc right?, can i ask for the information of the machine that is running the project on
32GB RAM and CPU AMD R7 5800X3D
wow i like your video
But S3 is free to use?
Minio is an opensource alternative, you can use it on local development for free
does not working today.
what you mean? the code still works . It was only tested on Windows.