Introducing Cloud Run Jobs

แชร์
ฝัง
  • เผยแพร่เมื่อ 23 ธ.ค. 2024

ความคิดเห็น • 56

  • @googlecloudtech
    @googlecloudtech  2 ปีที่แล้ว +3

    What do you think of Cloud Run Jobs? We’d love to hear from you in the comments below! Don’t forget to subscribe to Google Cloud Tech → goo.gle/GoogleCloudTech

    • @TheMomander
      @TheMomander 2 ปีที่แล้ว

      @@adigabaie1093 Thank you for the feedback! I will pass on your comment to the product team.

  • @JagdeepSinghKalsi
    @JagdeepSinghKalsi 2 ปีที่แล้ว +5

    Been trying to run jobs using different tools (mostly, too cumbersome, even within the realm of GCP), and this one just hits that sweet spot... time to remove all that unnecessary code for those "tricks" to run jobs for Cloud Run... thanks Martin - around 3 years of following you have really helped me stay ahead of the curve... thanks a ton man...

    • @TheMomander
      @TheMomander 2 ปีที่แล้ว +1

      Good to hear you found it useful, Jagdeep, and thank you for the kind words!

  • @santoshperumal129
    @santoshperumal129 2 ปีที่แล้ว +4

    Martin the contents that you post are really great with a demo it is so easy to understand and you take it like a story love your contents please keep posting more like this

    • @TheMomander
      @TheMomander 2 ปีที่แล้ว +1

      Happy to hear you find the videos useful, Santosh!

  • @StraightCoding
    @StraightCoding 9 หลายเดือนก่อน +1

    Hi, good video. I have a question: Leaving aside the container feature of Cloud Run Job, what will be the difference of executing the script using Cloud function 2nd or Cloud Run Job? Thanks

    • @TheMomander
      @TheMomander 9 หลายเดือนก่อน +1

      In my experience as a developer who's used both, these are the major differences:
      * Cloud Run Jobs let you run a single job for up to 24 hours. With Cloud Functions 2nd gen it's one hour for HTTP functions or 9 minutes for event-driven functions.
      * Cloud Run Jobs let you configure automatic retries in case of failure. That can't be done with Cloud Functions triggered by HTTP. It can be done for Cloud Functions triggered by other events.
      * Cloud Run Jobs can start multiple parallel tasks (workers) for you, to finish quicker. With Cloud Functions you'd have to orchestrate parallel work yourself.
      * Cloud Run Jobs make it easy to run shell scripts. Some tasks (like doing a database backup) are simpler to do with a gcloud command from a shell script, rather than writing code in a regular programming language.

    • @StraightCoding
      @StraightCoding 3 หลายเดือนก่อน

      I see, thanks

  • @elephantum
    @elephantum 2 ปีที่แล้ว +4

    Is it possible to provide run-specific arguments when triggering a job?

    • @karolinanetolicka5982
      @karolinanetolicka5982 2 ปีที่แล้ว

      Unfortunately not; we're looking into adding this in the future.

    • @elephantum
      @elephantum 2 ปีที่แล้ว

      @@karolinanetolicka5982 thanks for the answer. It would be super helpful for use cases like scheduling re-processing for a specific date or for a specific chunk of data.

    • @elephantum
      @elephantum 2 ปีที่แล้ว

      @@karolinanetolicka5982 another question: what are the reasonable limits for the number of job definitions?
      Is it OK to create new job definition for each hourly run? (I'm thinking of baking job run parameters into job definition)

    • @TheMomander
      @TheMomander 2 ปีที่แล้ว +1

      @@elephantum Karolina told me there are no limits on the number of jobs per project - so yes, a new job for each hourly run is reasonable. We do have limits on the number of concurrently running executions in the project and will be introducing garbage collection of old executions. To find the page where limits are documented, search "google cloud run quotas".

    • @TheMomander
      @TheMomander ปีที่แล้ว +1

      Run-specific arguments (the thing you asked for) have now been implemented. Do a web search for "google cloud run override job configuration for a specific execution" and you will find the documentation. Thank you for your feedback last year!

  • @TheGothicSecret
    @TheGothicSecret 2 ปีที่แล้ว +5

    Perfect solution just in time! Amazing

  • @obscurecloud5575
    @obscurecloud5575 2 ปีที่แล้ว +3

    This is great, thanks. I was wondering if it's possible to trigger jobs using http

    • @TheMomander
      @TheMomander 2 ปีที่แล้ว +4

      Yes, you can. See the URL that you enter in Cloud Scheduler to schedule a job:cloud.google.com/run/docs/execute/jobs-on-schedule. But be aware that you have to set the authentication header yourself. If you want to trigger your code with an HTTP call, consider putting it inside a regular Cloud Run service instead of a Cloud Run Job.

  • @franciscomerdot
    @franciscomerdot 2 ปีที่แล้ว +3

    I'm wondering why use Cloud Run Jobs and not Cloud Function for such background operation?

    • @TheMomander
      @TheMomander 2 ปีที่แล้ว

      Good question! Cloud Functions are great if you prefer deploying source code. Cloud Run is great if you prefer deploying containers.

    • @shubhamchaudhari6884
      @shubhamchaudhari6884 2 ปีที่แล้ว

      Cloud Functions do not allow you to determine your environment and does not have the ability to run Shell scripts, Cloud Run Jobs can do both.

  • @DUCKJAIIIII
    @DUCKJAIIIII 2 ปีที่แล้ว +1

    Hello Martin, I wonder if there are any plans to use CICD (via GitHub) like regular cloud run to deploy cloud run jobs? Wondering if CICD is deliberately not included or work in progress.

    • @ChrisJaydenBeats
      @ChrisJaydenBeats ปีที่แล้ว

      Pretty sure there's a GH action for it. I saw it a week ago

    • @TheMomander
      @TheMomander ปีที่แล้ว +1

      As you say, it's possible to add regular Cloud Run services with CI/CD via GitHub. Cloud Run Jobs are not there yet, but we will add it.

  • @katiefranz2101
    @katiefranz2101 2 ปีที่แล้ว +1

    A single cloud run job task has a max timeout of 60 minutes, but if you start a job with a cloud scheduler, the scheduler has a max timeout of 30 minutes. How can I get around the 30 minute max timeout from the scheduler? Or would the job continue to run to completion, even if the cloud scheduler may timeout?

    • @karolinanetolicka5982
      @karolinanetolicka5982 2 ปีที่แล้ว +2

      The Cloud Run job will continue to run until it's done. Make sure to monitor job success by looking at the status of the Cloud Run job, not the status of the Scheduler job - the Scheduler job will show as successful when the Cloud Run job is successfully started, not when it completes.

  • @Антон-б2л9у
    @Антон-б2л9у 11 หลายเดือนก่อน

    Can I make sure that only one instance of the job is running? I have a Telegram client (python script with Telethon package), so in NO CIRCUMSTANCE i can allow two processes to be at the same time. Otherwise Telegram bans my Session ID, and I have to re-log and redeploy. That's what happened today when a trigger started a new execution while the old one was still running. Ideally, I want to have some number of retires (in case it fails), and a trigger in the morning, but so that the trigger would kill the previous execution. Is Cloud Run Jobs a good choice at all?

  • @moulouelsalim
    @moulouelsalim 3 หลายเดือนก่อน +1

    Hello, I'm attempting to restrict the number of identical Google Cloud Run job instances running concurrently to prevent simultaneous writes to BigQuery tables. However, the --parallelism=1 parameter isn't achieving this goal, did anyone ever experience this situation ?

    • @TheMomander
      @TheMomander 3 หลายเดือนก่อน

      You're setting "--parallelism=1" but you are still seeing multiple workers running in parallel? That has only happened to me when I triggered a new job before the previous one had finished. Might that be true in your case too? How many jobs do you see running in Cloud Run Jobs section of the Cloud Console?

    • @moulouelsalim
      @moulouelsalim 3 หลายเดือนก่อน +1

      yes i use "--parallelism=1", I want to prevent multiple instances of the same job from running simultaneously. When I clicked 'Execute' ten times, I noticed in the Cloud Run logs that all ten jobs were executing independently, without waiting for the previous one to finish.

    • @TheMomander
      @TheMomander 3 หลายเดือนก่อน

      @@moulouelsalim Ah, got it. The "parallelism" setting determines how many parallel tasks start when you click the Execute button. For example, if you set "parallelism" to 10 and start a job, you start executing one job which contains 10 parallel tasks.

    • @moulouelsalim
      @moulouelsalim 3 หลายเดือนก่อน

      @TheMomander does that mean cloud run jub can't achieve my purpose of having only one instance of a cloud run job executing in an instant t, if it's the case I can't use it for containers writing in DB without using my own locks on databases.

    • @TheMomander
      @TheMomander 3 หลายเดือนก่อน

      @@moulouelsalim Well, you decide when to start a new job (you click the Execute button) and you decide when a job ends (your code completes and exits). So you are in control and you can make sure that only one job is running at a time.
      It sounds from your comments like you have started new jobs before your previous jobs finished? Maybe you could share what you're trying to accomplish, and I will let you know how I would approach it.

  • @shubhamchaudhari6884
    @shubhamchaudhari6884 2 ปีที่แล้ว

    Question: Is there a way to invoke these jobs from GCS events? or an integration with Eventarc?

    • @karolinanetolicka5982
      @karolinanetolicka5982 2 ปีที่แล้ว

      Not at the moment, but it's something we may add in the future.

    • @RodrigoFernandes-fg6me
      @RodrigoFernandes-fg6me ปีที่แล้ว +1

      @@karolinanetolicka5982 is there any plans to add continuous deployment to cloud run jobs?

    • @TheMomander
      @TheMomander ปีที่แล้ว

      @@RodrigoFernandes-fg6me Karolina is out of the office today, so I will respond. Today you can do continuous deployment of jobs with the command line (which means you can do it from a CI/CD pipeline defined in Cloud Build or GitHub Actions) or with Cloud Deploy.

  • @Babbili
    @Babbili 2 ปีที่แล้ว

    Nice, is the same k8s job, `apiVersion: batch/v1` if i want to create it from a yaml file

    • @knetolicka
      @knetolicka 2 ปีที่แล้ว +2

      The Cloud Run jobs API is inspired by the k8s jobs API, but it's not the same. You'll likely need to make a few changes to your yaml file.

  • @Gary21H
    @Gary21H 2 ปีที่แล้ว +2

    Excited to try this!

  • @kirankumarboddu655
    @kirankumarboddu655 2 ปีที่แล้ว

    hi how many dags runs in airflow in parelllel processing using cloud run jobs

    • @TheMomander
      @TheMomander 2 ปีที่แล้ว

      Not sure I understand your question. Would you elaborate?
      If you want to find out more about Airflow on Google Cloud, check out the Cloud Composer documentation.

  • @ianyoung_
    @ianyoung_ 2 ปีที่แล้ว

    Woah, this could be a game changer which removes the need for VMs or hacks for long-running tasks. Thanks for the demo!

    • @TheMomander
      @TheMomander 2 ปีที่แล้ว

      Happy to hear this is useful. Life becomes a little simpler with each eliminated VM.

  • @babusivaprakasam9846
    @babusivaprakasam9846 2 ปีที่แล้ว +3

    Good one. Small batch processing dataflows can be eliminated

  • @patricknelson
    @patricknelson 2 ปีที่แล้ว +1

    I wonder if there’s a way to dynamically compute the number of workers at startup time. I’d imagine not, that you’d use Cloud Scheduler to instead trigger a task that then cascades into scheduling the correct number of workers. Not that _I_ need this, but it did make me wonder.
    p.s. For me: “Serverless” is _great_ because I don’t have to worry about maintaining a server anymore for these simple easily containerized workloads and applications. Now maintenance and updates are just done at the application container level, which would have been necessary anyway, and is easily streamlined or even automated with CI/CD.

    • @MartinOmander
      @MartinOmander 2 ปีที่แล้ว +1

      Good question! I don't know of a way to compute the number of workers at startup time. One way of doing it would be to always start X workers, and each worker has the logic to terminate early if there isn't enough work for it.
      That approach would also decouple the logic in workers from the decision of how many workers to create. In other words, you can change the number of workers without having to worry about updating the worker code, and vice versa.

  • @anilmm2005
    @anilmm2005 2 ปีที่แล้ว +2

    Thanks for wonderful update

  • @divyaalok5538
    @divyaalok5538 ปีที่แล้ว +1

    Awesome 😊

  • @dailymeow3283
    @dailymeow3283 2 ปีที่แล้ว +3

    Awesome

  • @ramgopalnalluri
    @ramgopalnalluri 2 ปีที่แล้ว +1

    Cool