วีดีโอ

Does anyone here knows if it is possible to use Spark to select/collect multiple Parquet files from s3 bucket ( all in "ABC" folder) and combined them in one Parquet file in ( "DEF") file in the same location? and if so what is the code , thanks

Isn't it redundant to call out to an Athena cluster from within an EMR cluster? Is the reason for using Athena its integration with Glue's Catalog?

❤ thanks ❤

I am in video two and looks like it would be beneficial to provide details on how to create vcluster

Amazing job. Thank you!! What is the best way to read this delta tables now? Data Catalog and then Athena? I would like to see this data in our QuickSight.

Hi Dacort, I can’t finish the “Reopen in Container” step, as after it downloads the docker image, an error pops up saying “yum doesn’t have enough cache data to continue” at step “RUN yum install -y sudo && “Hadoop ALL=(ALL) NOPASSWD:ALL”.. Wondering if you might know any reason behind? Thank you so much!

When reading "cargo crates", my mind immediately jumped into the rust ecosystem and the cargo package manager :) Great video man! Learning a ton from you especially when it comes to AWS and EMR! Please keep up doing the great work!

Hey @dacort, Thanks for the great video. - What about Glue? Can we say that Glue and EMR serverless do more or less the same thing? - Let's say we only have Spark jobs to run based on some triggers. Since it is a transient job, I should run it with EMR serverless. On the other hand, if I need a long-running cluster, I should go with EMR on EC2/EKS. Can I extract the formula like this :)

Thanks it was helpful, what about aws athena and aws s3 access grants? I am struggling with a PoC where data is in one aws account and athena in another account and I would like to use aws s3 access grants to manage permissions

I tried to work with notebooks.. I succesful attach serverless but It can't run.. always show error when I try to run..aws docs say nothing....for me could be role. maybe "interactive rol"... what is it?... any advice?

You made EMR fun. Good luck with everything.

Really nice content. You should do more of such Developer tooling and productivity boosts while working with EMR.

@dacort Thanks for the video. In my case as well I'm using EmrContainerOperator to submit jobs to EMR on EKS cluster which is working fine. Now to track the cost for each jobs I want to assign tags to each jobs being executed in EMR on EKS virtual cluster. While assigning tags to the EmrContainerOperator, the jobs are being executed by aws_default connection id and is ignoring the aws_conn_id I've provided to my operator but once I remove the tags from the operator it is perfectly using the custom aws_conn_id I've provided. Any help here would be greatly appreciated. For your reference: emr_spark_submit = EmrContainerOperator( task_id="task_id", virtual_cluster_id="VIRTUAL_CLUSTER_ID", execution_role_arn="emr-on-eks-job-execution-role", release_label="emr-6.7.0-latest", job_driver=job_driver_arg, configuration_overrides=configuration_overrides_arg, name="submit_pyspark_job", aws_conn_id="emr-on-eks", tags={'job':'test'}, dag=dag, ) FYI: I'm using from airflow.providers.amazon.aws.operators.emr import EmrContainerOperator and not the from emr_containers.operators.emr_containers import EMRContainerOperator

Thanks for the video @dacort. I tried submitting the Job to my EKS cluster but it got failed. I'm unable to even check the logs why it got failed. Any help here would be really appreciated

Hi, thanks for the great video 😊 I am searching for an article or any direction to build a development pipeline on top of emr-eks from local/dev to production.

the video talks about the advantages of using EMR on EC2 and EMR serverless, so what is benefit of using EMR on EKS?

If you use PySpark engine on EMR to read the data from S3, will the resultant dataframe be stored on EMR or S3?

why EMR serverless does not support Flink? and also why EMR on EKS does not support Hive?

Q would this work with a EMR on EKS cluster?

Thanks for this great tutorial. Question: After a detached my notebook from a running cluster and stop my workspace(idle)... am I going to be charged for something else? (of course, assuming I terminated my cluster but still want to keep my workspace in emr studio). Thanks

좋아요

Thanks for creating this nice tool. In the Glue catalog view, can we also see the partition keys of a table?

Is there a way to install custom Java versions without creating custom images?

Is there a way to run EMR serverless with GPU? I want to run pyspark jobs with NVIDIA RAPIDS

What happened if the ip addresses have changed? do we need to re create the studio?

great video

This was extremely helpful, thank you! One suggestion for future videos on EMR: It's sort of implied in this video, but I think it would be helpful to new people to really emphasize that this will NOT WORK if you are signed in as the root user. I imagine there's a long list of reasons why using root user is bad practice, but I also imagine it's very common for people just starting out, and cause many wasted hours. 😅

Great video!! , Is there any way to run a dbt project using emr serverless?, I have seen that they have the Thrift option to connect to EMR on EC2, but I am not sure if it is possible to connect it to EMR serverless :(

Is there any way to run a dbt project using emr serverless?, I have seen that they have the Thrift option to connect to EMR on EC2, but I am not sure if it is possible to connect it to EMR serverless :(

Hello, I am getting the following error while retrieving databases

Hi I am getting USER_ERROR everytime I use custom image. Any solution for that ?

Does any of these have a "vacuum" equivalent, or how do you do housekeeping / maintenance on these incremental data lakes?

Thats an excellent demonstration

Hi Great video - can you please also show steps on how to install external libraries on EMR - bootstrap script replacement?

Can you covering the set up in more detail? specifically the IAM roles, EMR service role etc?

Hi Decort. Thank you for the video but somehow I am unable to connect this with my aws account as we use the SSO (single Sing On) authentication. But I am able to connect via AWS toolkit using the 'aws configure sso' command. Could you please help what can I do in this case?

Amazing. Could you do a tutorial about using step function with EMR Serverless? Thanks.

nice work

great tutorial

Thanks for the video Dacort..Please share how to connect my own S3 from Docker container. I try to add AWS keys in docker file as ARG. but it did not worked after rebuild the container. Please advise..

Amazing video do you know if there is any chance to send parameters from airflow DAG to the called notebook? For example the DAG receives a random date&&number then when you trigger the DAG it send those parameters to the notebook. Thank you! :)

This is great Damon. Just a quick question. Does this need a connection to AWS similar to the AWS Toolkit in Visual Studio? Somehow even after having AWS local profile and the necessary permissions, the explorers does not load

Amazing Demo!!!

Thanks for the concise demo!

This is wonderful demo video and very helpful. We want to create & submit jobs from either Terraform or open source Airflow. But Terraform supporting only application creation where as Airflow support from V5. Could you please share list of ways to create and run jobs.

Can we customize pods other than driver and executor pods? I wish to mount files that need to be available on job-runner container

have you configured EMR STUDIO for EMR on EKS? Please shared cloud formation stack info and TH-cam video. Thanks!

Amazing job, Thank you Dacort

I am trying to create Endpoint for EKS cluster to attach with EMR Studio. Endpoint requires Certificate. I have no clue how should I create certificate. I tried with certificate manager to create public certificate but not sure what is the domain I should provide there. Could you please explain about it ?

Nice speech and manner! Clear mind!