- 3
- 31 788
myCloud Uni
United Kingdom
เข้าร่วมเมื่อ 26 ก.ย. 2021
AWS Glue & Athena Tutorial To Search IMDB Movie Ratings
In this video, we will look at a couple of AWS Services (Glue and Athena) and build a simple application to query movie ratings.
Objective of this session is to get a basic understanding of the AWS Services (Athena and Glue) and also use the public datasets. After the completion of this exercise, do remember to delete all the AWS resources created to avoid costs.
We use publicly available datasets from IMDb Movie titles & ratings. Visit the IMDb site (datasets.imdbws.com) for terms and conditions of using the IMDb data sets.
We store the data in S3 and use Glue to crawl and catalogue the data, which can be further queried by Athena. By the end of this video, you will learn about AWS Athena, Glue and CloudShell and when and how to use them.
Appreciate, if you can provide feedback / comments on this video and subscribe to the channel for receiving updates.
IMDb Datasets are available at : datasets.imdbws.com
CLI Commands in Cloud Shell to upload data files to S3 (Replace {your bucket name} with your S3 bucket name)
aws s3 mb s3://{your bucket name}
wget datasets.imdbws.com/title.basics.tsv.gz
wget datasets.imdbws.com/title.ratings.tsv.gz
aws s3 cp ./title.basics.tsv.gz s3://{your bucket name}/basics/basics.tsv.gz
aws s3 cp ./title.ratings.tsv.gz s3://{your bucket name}/ratings/ratings.tsv.gz
Athena Sample Query :
SELECT
bas.primarytitle, rat.averagerating
FROM
"athenademo"."basics" bas
JOIN
"athenademo"."ratings" rat
ON
bas.tconst = rat.tconst
WHERE
primarytitle like '%Shawshank%Redemption%'
Objective of this session is to get a basic understanding of the AWS Services (Athena and Glue) and also use the public datasets. After the completion of this exercise, do remember to delete all the AWS resources created to avoid costs.
We use publicly available datasets from IMDb Movie titles & ratings. Visit the IMDb site (datasets.imdbws.com) for terms and conditions of using the IMDb data sets.
We store the data in S3 and use Glue to crawl and catalogue the data, which can be further queried by Athena. By the end of this video, you will learn about AWS Athena, Glue and CloudShell and when and how to use them.
Appreciate, if you can provide feedback / comments on this video and subscribe to the channel for receiving updates.
IMDb Datasets are available at : datasets.imdbws.com
CLI Commands in Cloud Shell to upload data files to S3 (Replace {your bucket name} with your S3 bucket name)
aws s3 mb s3://{your bucket name}
wget datasets.imdbws.com/title.basics.tsv.gz
wget datasets.imdbws.com/title.ratings.tsv.gz
aws s3 cp ./title.basics.tsv.gz s3://{your bucket name}/basics/basics.tsv.gz
aws s3 cp ./title.ratings.tsv.gz s3://{your bucket name}/ratings/ratings.tsv.gz
Athena Sample Query :
SELECT
bas.primarytitle, rat.averagerating
FROM
"athenademo"."basics" bas
JOIN
"athenademo"."ratings" rat
ON
bas.tconst = rat.tconst
WHERE
primarytitle like '%Shawshank%Redemption%'
มุมมอง: 943
วีดีโอ
AWS DMS (Data Migration Service) Step by Step Tutorial (Migrate data from Aurora to S3 & DynamoDB)
มุมมอง 19K2 ปีที่แล้ว
In this hands-on step by step tutorial, we will see as to how we can use AWS DMS (Data Migration Service) to migrate data from Aurora mySQL to S3 as well as to DynamoDB. Also, demonstrated is the Chandra Data Capture (CDC) features of DMS to stream source data changes to the DMS target(s). Things that you will learn in this tutorial : 1. How to create a DMS Replication Instance 2. How to create...
AWS DMS (Data Migration Service) Introduction
มุมมอง 12K3 ปีที่แล้ว
This video includes a brief introduction of AWS DMS (Data Migration Service) followed by a brief demo. The demo includes moving data from a mySQL Aurora database to DynamoDB. Also included in this video is a demonstration of CDC (Change Data Capturing) with DMS. Please note that DMS with DynamoDB as target would automatically create at least 3 extra DynamoDB tables each with 200 WCU and 200 RCU...
Hi This is a very usefull video, and i request could you please provide me steps to migrate on prem sql server to aws rds postgres Could you please make an demo for this. Thanks
Thank you so much brother. for detailed explanation
Thank you so much sir!, this video saved my hours and hours of work. Plus I have better understanding of soo many concepts. Really thankful sir. Please create more such videos on vaious AWS topics
Wow! What a great video that touches upon many topics (security, bastion server, setting bin logs on MySQL, etc.) that many other videos do NOT cover. All these details are important steps that you NEED to know when you actually implement a DMS pipeline. Thank you SO MUCH for taking the time to create this video with so many details and publishing it!
Thank you! It is very useful session. Lovely.
Valuable presentation 👍🏻
While i am connecting to aurora through SQL electron it is showing connection error. Can u please tell me where I made a mistake
Sir, Excellent Working. Much appreciated. Sir Do you have videos explaining STEP function and Lambda for the same use case?
Great job ! Appreciated !
Excellently explained. Expecting more videos from on channel.
Any training offered?
Please share email
Nicely explained
Please upload videos of on-premises application and their database migrate to AWS cloud
awesome, thanks
Useful Video, I have a doubt DMS will work for structural changes as well? Suppose, I added one column at source will it move to target? MS SQL -> Mysql Aurora
Hi Nani, Thank you. Yes; the structural changes at source (i.e. DDL) does cascade to the target; but with some limitations. For example, if you alter a source table and add a new column towards the end of the table, then it would cascade to the target. But, if you add a new column in the middle of the table, then it won't cascade. Suggest you to see the AWS documentation for the supported DDL statements for specific sources & target databases. Regards
@@myclouduni2973 Thanks for your reply Sir..
@@myclouduni2973 do you give online training for AWS?
really nice work , detailing of each and every step is appreciated . looking forward for new videos .
Hello Sir. Nice work and it is really helpful! Could you tell me how much will it cost for the mentioned setup? And what will be the cost for keeping it running for 2-3 days?
Hi Soumya, Thank you very much. The running costs are mostly due to (a) DynamoDB tables and (b) DMS replication instance. As mentioned in the tutorial, the DynamoDB tables get created with 200 WCU & 200 RCU by default. Please note that DMS with DynamoDB as target would automatically create at least 3 extra DynamoDB tables each with 200 WCU and 200 RCU provisioned capacity. The table names are : DMSCDC_Controller, awsdms_apply_exceptions and awsdms_full_load_exceptions). You are typically looking at $10-$15/day running costs if you leave these tables as-is. Of course, this depends on your region and the size of DMS instance. You can reduce these costs by change the DynamoDB tables to "OnDemand" instead of Provisioned. That way, you are looking at about a couple of dollars of running costs per day at most. In any case, keep an eye on the AWS billing dashboard and delete unused resources. Hope this helps.
Fantastic tutorial!
Hi Sir ..very nice video..could you please share the link of the other video which you said about basic idea of DMS video
Hi Areeb, Thank you for your comments. Here is the link to the other video & please checkout the same. th-cam.com/video/YVIugsV6egQ/w-d-xo.html
Hi sir ..could you please share the link which you said is having detailed video?
Hi, Here is the link th-cam.com/video/73HSRWiBVsE/w-d-xo.html ...
Very informative and Useful... Expecting more learning videos from u🤝. Thanks in advance