I am facing an issue near max worker, from the set up it is set as 4, but while i am trying to link the requirements file then it is throwing an error as the number of max worker is not in th range (range 1-0) , i could not resolve this. I tried to create the env couple of times but still the same error is showing up.
Even if i configure requirements.txt correctly , still getting import pandas as pd modulenotfounderror: no module named 'pandas' in airflow and dag is broken error . Thanks
Broken DAG: [/usr/local/airflow/dags/openweather_api.py] Traceback (most recent call last): File "", line 241, in _call_with_frames_removed File "/usr/local/airflow/dags/openweather_api.py", line 8, in import pandas as pd ModuleNotFoundError: No module named 'pandas'
Hi, very nice project and video!!! I am having some problems, when I upload airflow-redshift-template.yaml to cloudformation -- Create stack, it shows the following error, could you tell me how to fix it? Template format error Follow the standard JSON or YAML spec to format your template.Learn more Parser error duplicated mapping key (323:3) 320 | WebserverLogs: 321 | LogLevel: !Ref Webserve ... 322 | Enabled: true 323 | SecurityGroup: ---------^ 324 | Type: AWS::EC2::SecurityGroup 325 | Properties: Thank you very much, sir!
The issue is as a result of a duplicated mapping key - the key "SecurityGroup" is defined twice. Just rename the second SecurityGroup to SecurityGroup1. "SecurityGroupIngress" is also duplicated so you'll need to do the same.
WARNING ... I created a FREE AWS account and completed this training exercise during my free time, which I completed over two days. I was shocked to receive a bill from AWS for nearly $60.00. and an estimate of $600 for a month, even though the system was not running anything. I contacted AWS and it is taken another two days of correspondence to manually shut down all the services, and disestablish connections, and I still am arguing over the charges. I am very unlikely to recommend AWS to any future client
The services used in this demo are not covered under free tier.. so it's the user responsibility to check before hand..however aws is usually wave out this kind of billing easily
Im got this error Broken DAG: [/usr/local/airflow/dags/openweather_api.py] Traceback (most recent call last): File "/usr/local/airflow/dags/openweather_api.py", line 26, in "appid": Variable.get("key") File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/models/variable.py", line 142, in get raise KeyError(f"Variable {key} does not exist") KeyError: 'Variable key does not exist'
![[LIVE] : ONE ลุมพินี 79 | คู่เอก "ก้องชัย vs อาเมียร์"](http://i.ytimg.com/vi/WfPSA1A0NHc/mqdefault.jpg)
Please post more content like this, its very useful for me to understand a project on the whole and how each tool/product work together! great work!
Will do!!
Thanks for the explanation, More of these videos please
Good work mate
Hi Nice informative and knowledgible video..
You rocked it Bro
Thank you very much
Thanks man, this is helpful👍
Well explained sir 👍🏻
Really appreciate it
hello that yaml file is giving error on AWS
I am facing an issue near max worker, from the set up it is set as 4, but while i am trying to link the requirements file then it is throwing an error as the number of max worker is not in th range (range 1-0) , i could not resolve this. I tried to create the env couple of times but still the same error is showing up.
hello, what if i do not wish to use redshift here? how can i deploy it
Even if i configure requirements.txt correctly , still getting import pandas as pd modulenotfounderror: no module named 'pandas' in airflow and dag is broken error . Thanks
Broken DAG: [/usr/local/airflow/dags/openweather_api.py] Traceback (most recent call last):
File "", line 241, in _call_with_frames_removed
File "/usr/local/airflow/dags/openweather_api.py", line 8, in
import pandas as pd
ModuleNotFoundError: No module named 'pandas'
Just redeploy the airflow instance .
is it necessary to create the bucket (airflowoututtos3bucket) manually
?
Not necessary
can you make project on EMR?
Hi, very nice project and video!!!
I am having some problems, when I upload airflow-redshift-template.yaml to cloudformation -- Create stack, it shows the following error, could you tell me how to fix it?
Template format error
Follow the standard JSON or YAML spec to format your template.Learn more
Parser error
duplicated mapping key (323:3)
320 | WebserverLogs:
321 | LogLevel: !Ref Webserve ...
322 | Enabled: true
323 | SecurityGroup:
---------^
324 | Type: AWS::EC2::SecurityGroup
325 | Properties:
Thank you very much, sir!
I agree, I am having same issue. Did you get it resolved?
The issue is as a result of a duplicated mapping key - the key "SecurityGroup" is defined twice. Just rename the second SecurityGroup to SecurityGroup1.
"SecurityGroupIngress" is also duplicated so you'll need to do the same.
Is this service free to use? If not can I work on this project and disable the service?
Yeah, you can work and disable the services .
WARNING ... I created a FREE AWS account and completed this training exercise during my free time, which I completed over two days. I was shocked to receive a bill from AWS for nearly $60.00. and an estimate of $600 for a month, even though the system was not running anything. I contacted AWS and it is taken another two days of correspondence to manually shut down all the services, and disestablish connections, and I still am arguing over the charges. I am very unlikely to recommend AWS to any future client
The services used in this demo are not covered under free tier.. so it's the user responsibility to check before hand..however aws is usually wave out this kind of billing easily
can we use any other service instead of airflow in aws ?
We can use step functions.
@@DataTechByAnandKumarmwaa will not be required right?
There are alternative ways to do things but if you don't wanna use airflow then sure.
Im got this error
Broken DAG: [/usr/local/airflow/dags/openweather_api.py] Traceback (most recent call last):
File "/usr/local/airflow/dags/openweather_api.py", line 26, in
"appid": Variable.get("key")
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/models/variable.py", line 142, in get
raise KeyError(f"Variable {key} does not exist")
KeyError: 'Variable key does not exist'
Me too!! Have you resolved your problem anyway?