small tip for anyone running into this issue. If you are uploading multiple .csv files they must be in their separate folder and you need to point glue to each of them in the step at 12:55 you would click "YES" and then point glue to the next folder
Thanks so much bro for this tutorial. I have been having problem creating the output query bucket until I watched this video, thank you so much. Pls do more.
Partitions are most important thing when considering Athena as a Big Data querying tool. You can’t go to production without propers partitions in place as per your usage patter. I would suggest going through official AWS documentation.
Very nice explanation sir. Would you mind please make vedios on How to move data from on premises to Aws And how to build data lake in AWS. I know individual services in AWS but bit confusing about how to move data to AWS(means which service is best) Please sir
Hi Ajay, first of all, many thanks for such a detailed hands-on. Great!. How about a part 2 of the video by exploring further? I mean take the data from Athena table and put it into Quicksight and then to a BI report? That will close the circle. Looking forward to this request.
So detailed & informative. You didn't skip any single point to cover. Really loveed it & could follow every single points that you covered. Good job brother. Keep uploadoing such videos.
Great video! Do you know if when using a crawler, can it automatically detect data partitions? Meaning if you specify the S3 path to a data folder, assuming we have year subfolders within that data folder, will it load the partitions for you or you have to do that manually? Thanks!
Hi Ajay, thanks for the great video. I wanted to ask, if we could capture more metadata information in AWS glue catalog apart from just data types, can I manually feed in more metadata , and will it still hold when next time AWS glue updates the catalog. Thanks
Thank you its very nice explanation . my query is when run query from athena on Glue catalog table , the data will fetch from catalog table or in back end it will connect to S3 bucket and fetch data ?
Nice video, but just one question.In real time scenario, do they use this Athena manually or will they encapsulate with any script to run as part of whole process, e.g. Lambda or anything ?
Kundan, this Glue Catalog creation is a one time activity and you can set up crawlers manually. Otherwise you can use Cloudformation (cftemplate) to set up Athena automatically.
Glue is used for table metadata....Structure of table that you refer in Athena is stored in Glue. Glue is AWS' centralized Data Catalog. You can create tables in Athena by yourself, but consider other scenarios such as changing data in s3 (new columns , columns data types), federated queries. For all these scenarios we create Glue Data Catalog and in Athena we don't indulge into pain of DDL changes.
Hey Ajay, superb video. Just want to ask that, can we use all the syntax available in SQL in Athena. Like creation of temp table, conditional & looping statements and dynamic query generation. Thanks in advance
Hi Ajay, we are exactly doing the same but our requirement is we dump data related multiple events in s3 path and create a multiple athena tables for selective fields .. this filter condition how can we enable.. if u can help me with this..
You can do segregation of your S3 files using Lambda function or use regex while giving path in Athena so that only files matching with the regex would be scanned
Hi Ajay, When I try to execute the sql query in Athena for the first time, it gives proper formatted results. However, when I run it the second time, it gives a weird looking output which has words like "hive", "varchar2", etc.....in the output alongwith the main records which somehow is displayed in quotes. (I have a csv file stored in S3 folder) Not sure why this is happening. My objective is to fetch the data from S3 using Athena query and access the same from Tableau for visualization/analysis.
Hi Ajay I have created a table in Athena but data is not coming It is giving message like zero records returned Can you please help me how to proceed to get the data
![AWS Glue Tutorial for Beginners [FULL COURSE in 45 mins]](http://i.ytimg.com/vi/dQnRP6X8QAU/mqdefault.jpg)
Hi Ajay
Your teaching is superbbbb and please make more videos on analytics services in aws
Tqq so much
I am glad you liked it. I will be making more videos on AWS server less . Subscribe for updates
@@AjayWadhara Thanks Ajay...This is the best piece I have found on you tube.
Thanks
Very valuable to see this process visually. Thank you
Crisp and clear 👌👌
Great job Ajay!
Nice tutorial with clear explanation 🎉
Best tutorial on Athena, cleared all my doubts, highly recommend
Excellent video, I learnt Athena from here 👍
Very informative for a newbie - ty!
small tip for anyone running into this issue. If you are uploading multiple .csv files they must be in their separate folder and you need to point glue to each of them in the step at 12:55 you would click "YES" and then point glue to the next folder
Very nice tutorial.
Thank you for the video it helped me a lot . Thank you Ajay bhaiya .
I’m glad you liked the video. Don’t forget to subscribe 🚀
Hi Ajay,
Very informative video...Thank you so much..
Thanks Saurabh, do check out other videos also.
Don’t forget to subscribe, this keeps me motivated to publish more🚀
excellent video with superb explanation
Thanks 🙏🏻
Excellent tutorial!
Great stuff and easy to understand.Thanks alot Ajay
It's a very good on AWS Athena. Neat explanation and the demonstration is great.
Thanks , I am glad you liked it.
Explaination was very clear. Thank you AJAY
Glad you liked it
Better than most of the paid tutorials out there! Please keep posting more videos.
I will ✅✅
Don’t forget to subscribe 🚀
Thanks so much bro for this tutorial. I have been having problem creating the output query bucket until I watched this video, thank you so much. Pls do more.
Thank you so much. This is an awesome video. Much appreciated!
Nice video Ajay
it is really helpful to keep going on.
awasome
Excellent !!!!! Explanation you covered end to end :-) , Thanks a lot
Thanks for this!
Great explanation. Thanks for sharing your knowledge. I found it really helpful
Thank You :)
Great Tutorial. Can you share some more briefs about partitions and where it using in athena.
Partitions are most important thing when considering Athena as a Big Data querying tool. You can’t go to production without propers partitions in place as per your usage patter.
I would suggest going through official AWS documentation.
its great and you should make a video series of aws.
Superb tutorial.... Plz make more videos in depth on AWS
Do subscribe and share ✅
Hi Ajay Kumar!! It’s possible up to mdb files to S3 and work ETL with Glue ?
Greeting from Chile.
Thanks, a great tutorial!
This is great thanks so much Ajay!
Glad you liked it!
Nice explanation 👌 👍
I’m glad you liked it. Don’t forget to subscribe ✅
Very informational for a newbie. Thanks, Keep Sharing :)
Great explanation, to the point and very clear!!
Great job Ajay
Yes, 10/10!
Thank you very much for your time, it was really helpful....
Nice Demo Ajay, help me a lot. Please let me know all your Video path for S3-Athena-Glue-Hive related.
Awesome explanation @ajay
Thanks 😊
Don’t forget to subscribe and share 🚀
Thanks for the knowledge transfer! it is to the point and very clear!
Glad you liked it. 🤘🏻
Hey Ajay,
I completed your course on Udemy and it was good man!!
Thank you for the valuable information
Glad you liked it...Don't forget to subscribe and share
Very nice explanation sir. Would you mind please make vedios on How to move data from on premises to Aws
And how to build data lake in AWS. I know individual services in AWS but bit confusing about how to move data to AWS(means which service is best)
Please sir
Nice one brother! 🔥🔥🔥
Thanks for such an awesome demo and explanation. just an query how can be specify the query result location before getting that error in athena
Very useful
Hi Ajay, first of all, many thanks for such a detailed hands-on. Great!. How about a part 2 of the video by exploring further? I mean take the data from Athena table and put it into Quicksight and then to a BI report? That will close the circle. Looking forward to this request.
Hey Sujoy, definitely I will come up with Athena Integration with QuickSight..Stay tuned
@@AjayWadhara Looking forward to the video as well, I know it's been a year but will be great if you can post. Many thanks!!
Hi Ajay, Your explanation was too good. Can you please make some videos on Aws Glue architecture and hands on that would be great..
Hi Ajay Your Explanation is awesome can you please add more videos on Analytics and S3 to cloudfront Thanks
Definitely Ashwith...Stay tuned for more videos ✅
Great video. Can you please talk about Workgroup as well. ?
Great job bro
Good presentation. How to you add PARTITIONS/modify to a table which was created used GLUE?
Can we use glue for xml, rdbms souce types
Hi Ajay,
Very informative video.
Can you please post a video on Athena connecting with different data source other than S3.
Sure Dipanjan, I will surely create one video for this.
Good one
I'm glad you like it.
So detailed & informative. You didn't skip any single point to cover. Really loveed it & could follow every single points that you covered. Good job brother. Keep uploadoing such videos.
Nice explanation bro
Thank you 🙂
Very impressive Bro!!!
The beneficial of using glue is that we don't need to create all column manually isn't it? or Is there something more to using glue?
This is just the tip of iceberg. Glue has all the features what an ETL tool provides.
Great video! Do you know if when using a crawler, can it automatically detect data partitions? Meaning if you specify the S3 path to a data folder, assuming we have year subfolders within that data folder, will it load the partitions for you or you have to do that manually? Thanks!
It will load automatically
Don't forget to subscribe and share ✅
The function of the crawler will provides the automated schema for our data. correct if I'm wrong...
Correct
DO you know where we can find an example with terraform to setup a simple environment? And use API to consume data from Athena?
Not sure Bill
Hi Ajay, Could you Please provide the video on how to query Postgresql data in Athena (Postgres as a data source in Athena)
Yeah..I can work on that...✅
@@AjayWadhara Thanks
Now how to reflect the output into the quicksight?
Nice
Do subscribe 🚀
10 out of 10 ,you covered the topics in best and simplest way
Hi Ajay, thanks for the great video. I wanted to ask, if we could capture more metadata information in AWS glue catalog apart from just data types, can I manually feed in more metadata , and will it still hold when next time AWS glue updates the catalog. Thanks
Thank you its very nice explanation .
my query is when run query from athena on Glue catalog table , the data will fetch from catalog table or in back end it will connect to S3 bucket and fetch data ?
Catalog table is just metastore, it contains information about schema but actual data is in S3.
Query will fetch data from S3 only
Hello!
I did same as you did but while running query in athena, it shows "zero records returned".
Kindly help in this.
how to run a query that has tables in more than 1 database ? i.e a query that joins the table across various databases ?
Hi Ajay,
May I know how do we connect the glue crawler to Athena. You haven’t showed it. Should we click on the ‘Connect data store’ option?
You don't have to connect Glue and Athena
Glue is centralized data catalog.
Database ,tables, views are automatically visible in Athena console.
@@AjayWadhara thanks a lot. It was really helpful ✌🏻😁
Subscribe and share...🎯✅
Good one bro
Thank you!!
Good job buddy
Nice video, but just one question.In real time scenario, do they use this Athena manually or will they encapsulate with any script to run as part of whole process, e.g. Lambda or anything ?
Kundan, this Glue Catalog creation is a one time activity and you can set up crawlers manually. Otherwise you can use Cloudformation (cftemplate) to set up Athena automatically.
Is it possible to query Athena from within Postgresql's sql session? Basically, I want to do SQL joins between Athena tables and RDS tables
Hey, Thats possible
Check Athena Federate queries.🚀
Hi Ajay, Can we use Athena without Glue catalog, incase I want to use informatica EDC instead of Glue catalog?
How to copy multiple files into folder in crawler job using python and Athena glue
Good presentation 👍
Can we have a complete course about AWS data engineering ? Ready to pay for it.
Hi Saurabh, this is on my bucket list for long but not finding time for this :(
Nice one. That means we can use Athena without using Glue. So why should we use Glue in Athena?
Glue is used for table metadata....Structure of table that you refer in Athena is stored in Glue.
Glue is AWS' centralized Data Catalog.
You can create tables in Athena by yourself, but consider other scenarios such as changing data in s3 (new columns , columns data types), federated queries. For all these scenarios we create Glue Data Catalog and in Athena we don't indulge into pain of DDL changes.
Hi Ajay please help me in providing topic wise questions for aws associate certification
How did you specify that the tables will be created in athena
Hi Ajay - can you do one on one training. I will pay for it of course.
Thank you....
Hey Ajay, superb video.
Just want to ask that, can we use all the syntax available in SQL in Athena. Like creation of temp table, conditional & looping statements and dynamic query generation.
Thanks in advance
I will say.. Syntax is almost SQL like.
For more details check this link
docs.aws.amazon.com/athena/latest/ug/querying-athena-tables.html
@@AjayWadhara thanks
Can I request you for some detailed videos on Glue ETL transformations using pyspark as scripting language.
Is that a yes I can take it as? Will you kindly make some videos on py spark coding in glue transformation examples?
Hi Ajay, we are exactly doing the same but our requirement is we dump data related multiple events in s3 path and create a multiple athena tables for selective fields .. this filter condition how can we enable.. if u can help me with this..
You can do segregation of your S3 files using Lambda function or use regex while giving path in Athena so that only files matching with the regex would be scanned
Hello. New subbie here. I love your teaching. May I ask if it is possible to do preprocessing of many text files with anthena?
Hi, you can use Lambda to pre process files on S3 put object or use Glue
if the file type is parquet, not CSV. Will the process is going to be the same and how the crawler is going to read a parquet file.
Process would be exactly same. Crawler supports Parquet also. You don't need to handle anything explicitly
After running query how to transfer that in S3 (in target system)
Output of the query is stored automatically in the Output folder specified, otherwise you can manually handle that part.
@@AjayWadhara thank you❤
Hi Ajay,
When I try to execute the sql query in Athena for the first time, it gives proper formatted results. However, when I run it the second time, it gives a weird looking output which has words like "hive", "varchar2", etc.....in the output alongwith the main records which somehow is displayed in quotes. (I have a csv file stored in S3 folder)
Not sure why this is happening.
My objective is to fetch the data from S3 using Athena query and access the same from Tableau for visualization/analysis.
Hi Ajay
I have created a table in Athena but data is not coming
It is giving message like zero records returned
Can you please help me how to proceed to get the data
Make sure your Glue schema is correct and you are specifying correct bucket
Hello sir please upload new more videos of aws
Sure...Btw I uploaded one tutorial on SNS today, do check here th-cam.com/video/BFUuyAyQYWM/w-d-xo.html
Can anyone help in providing this csv?
Ur voice is too low.... But ur video is great buddy
Thanks 😊
I have improved my voice in later videos. Do check the latest tutorials on Lambda✅
Don’t forget to subscribe 🚀
Sry for this but we have trouble to listen due to low voice ur voice is proper to get understand..... Thank for
👌👌👌👌
It is showing labelled error
I tried in AWS account bro, but table is not created in glue ...help me
Hey Ashok, if you follow the steps, you should be able to see the table
@@AjayWadhara not only my account ,i tried in my freds account ,logs are created successfully but table is not created