What do you think of Cloud Run Jobs? We’d love to hear from you in the comments below! Don’t forget to subscribe to Google Cloud Tech → goo.gle/GoogleCloudTech
Been trying to run jobs using different tools (mostly, too cumbersome, even within the realm of GCP), and this one just hits that sweet spot... time to remove all that unnecessary code for those "tricks" to run jobs for Cloud Run... thanks Martin - around 3 years of following you have really helped me stay ahead of the curve... thanks a ton man...
Martin the contents that you post are really great with a demo it is so easy to understand and you take it like a story love your contents please keep posting more like this
Hi, good video. I have a question: Leaving aside the container feature of Cloud Run Job, what will be the difference of executing the script using Cloud function 2nd or Cloud Run Job? Thanks
In my experience as a developer who's used both, these are the major differences: * Cloud Run Jobs let you run a single job for up to 24 hours. With Cloud Functions 2nd gen it's one hour for HTTP functions or 9 minutes for event-driven functions. * Cloud Run Jobs let you configure automatic retries in case of failure. That can't be done with Cloud Functions triggered by HTTP. It can be done for Cloud Functions triggered by other events. * Cloud Run Jobs can start multiple parallel tasks (workers) for you, to finish quicker. With Cloud Functions you'd have to orchestrate parallel work yourself. * Cloud Run Jobs make it easy to run shell scripts. Some tasks (like doing a database backup) are simpler to do with a gcloud command from a shell script, rather than writing code in a regular programming language.
@@karolinanetolicka5982 thanks for the answer. It would be super helpful for use cases like scheduling re-processing for a specific date or for a specific chunk of data.
@@karolinanetolicka5982 another question: what are the reasonable limits for the number of job definitions? Is it OK to create new job definition for each hourly run? (I'm thinking of baking job run parameters into job definition)
@@elephantum Karolina told me there are no limits on the number of jobs per project - so yes, a new job for each hourly run is reasonable. We do have limits on the number of concurrently running executions in the project and will be introducing garbage collection of old executions. To find the page where limits are documented, search "google cloud run quotas".
Run-specific arguments (the thing you asked for) have now been implemented. Do a web search for "google cloud run override job configuration for a specific execution" and you will find the documentation. Thank you for your feedback last year!
Hello Martin, I wonder if there are any plans to use CICD (via GitHub) like regular cloud run to deploy cloud run jobs? Wondering if CICD is deliberately not included or work in progress.
Can I make sure that only one instance of the job is running? I have a Telegram client (python script with Telethon package), so in NO CIRCUMSTANCE i can allow two processes to be at the same time. Otherwise Telegram bans my Session ID, and I have to re-log and redeploy. That's what happened today when a trigger started a new execution while the old one was still running. Ideally, I want to have some number of retires (in case it fails), and a trigger in the morning, but so that the trigger would kill the previous execution. Is Cloud Run Jobs a good choice at all?
Yes, you can. See the URL that you enter in Cloud Scheduler to schedule a job:cloud.google.com/run/docs/execute/jobs-on-schedule. But be aware that you have to set the authentication header yourself. If you want to trigger your code with an HTTP call, consider putting it inside a regular Cloud Run service instead of a Cloud Run Job.
A single cloud run job task has a max timeout of 60 minutes, but if you start a job with a cloud scheduler, the scheduler has a max timeout of 30 minutes. How can I get around the 30 minute max timeout from the scheduler? Or would the job continue to run to completion, even if the cloud scheduler may timeout?
The Cloud Run job will continue to run until it's done. Make sure to monitor job success by looking at the status of the Cloud Run job, not the status of the Scheduler job - the Scheduler job will show as successful when the Cloud Run job is successfully started, not when it completes.
Hello, I'm attempting to restrict the number of identical Google Cloud Run job instances running concurrently to prevent simultaneous writes to BigQuery tables. However, the --parallelism=1 parameter isn't achieving this goal, did anyone ever experience this situation ?
You're setting "--parallelism=1" but you are still seeing multiple workers running in parallel? That has only happened to me when I triggered a new job before the previous one had finished. Might that be true in your case too? How many jobs do you see running in Cloud Run Jobs section of the Cloud Console?
yes i use "--parallelism=1", I want to prevent multiple instances of the same job from running simultaneously. When I clicked 'Execute' ten times, I noticed in the Cloud Run logs that all ten jobs were executing independently, without waiting for the previous one to finish.
@@moulouelsalim Ah, got it. The "parallelism" setting determines how many parallel tasks start when you click the Execute button. For example, if you set "parallelism" to 10 and start a job, you start executing one job which contains 10 parallel tasks.
@TheMomander does that mean cloud run jub can't achieve my purpose of having only one instance of a cloud run job executing in an instant t, if it's the case I can't use it for containers writing in DB without using my own locks on databases.
@@moulouelsalim Well, you decide when to start a new job (you click the Execute button) and you decide when a job ends (your code completes and exits). So you are in control and you can make sure that only one job is running at a time. It sounds from your comments like you have started new jobs before your previous jobs finished? Maybe you could share what you're trying to accomplish, and I will let you know how I would approach it.
Not sure I understand your question. Would you elaborate? If you want to find out more about Airflow on Google Cloud, check out the Cloud Composer documentation.
@@RodrigoFernandes-fg6me Karolina is out of the office today, so I will respond. Today you can do continuous deployment of jobs with the command line (which means you can do it from a CI/CD pipeline defined in Cloud Build or GitHub Actions) or with Cloud Deploy.
I wonder if there’s a way to dynamically compute the number of workers at startup time. I’d imagine not, that you’d use Cloud Scheduler to instead trigger a task that then cascades into scheduling the correct number of workers. Not that _I_ need this, but it did make me wonder. p.s. For me: “Serverless” is _great_ because I don’t have to worry about maintaining a server anymore for these simple easily containerized workloads and applications. Now maintenance and updates are just done at the application container level, which would have been necessary anyway, and is easily streamlined or even automated with CI/CD.
Good question! I don't know of a way to compute the number of workers at startup time. One way of doing it would be to always start X workers, and each worker has the logic to terminate early if there isn't enough work for it. That approach would also decouple the logic in workers from the decision of how many workers to create. In other words, you can change the number of workers without having to worry about updating the worker code, and vice versa.
What do you think of Cloud Run Jobs? We’d love to hear from you in the comments below! Don’t forget to subscribe to Google Cloud Tech → goo.gle/GoogleCloudTech
@@adigabaie1093 Thank you for the feedback! I will pass on your comment to the product team.
Been trying to run jobs using different tools (mostly, too cumbersome, even within the realm of GCP), and this one just hits that sweet spot... time to remove all that unnecessary code for those "tricks" to run jobs for Cloud Run... thanks Martin - around 3 years of following you have really helped me stay ahead of the curve... thanks a ton man...
Good to hear you found it useful, Jagdeep, and thank you for the kind words!
Martin the contents that you post are really great with a demo it is so easy to understand and you take it like a story love your contents please keep posting more like this
Happy to hear you find the videos useful, Santosh!
Hi, good video. I have a question: Leaving aside the container feature of Cloud Run Job, what will be the difference of executing the script using Cloud function 2nd or Cloud Run Job? Thanks
In my experience as a developer who's used both, these are the major differences:
* Cloud Run Jobs let you run a single job for up to 24 hours. With Cloud Functions 2nd gen it's one hour for HTTP functions or 9 minutes for event-driven functions.
* Cloud Run Jobs let you configure automatic retries in case of failure. That can't be done with Cloud Functions triggered by HTTP. It can be done for Cloud Functions triggered by other events.
* Cloud Run Jobs can start multiple parallel tasks (workers) for you, to finish quicker. With Cloud Functions you'd have to orchestrate parallel work yourself.
* Cloud Run Jobs make it easy to run shell scripts. Some tasks (like doing a database backup) are simpler to do with a gcloud command from a shell script, rather than writing code in a regular programming language.
I see, thanks
Perfect solution just in time! Amazing
Is it possible to provide run-specific arguments when triggering a job?
Unfortunately not; we're looking into adding this in the future.
@@karolinanetolicka5982 thanks for the answer. It would be super helpful for use cases like scheduling re-processing for a specific date or for a specific chunk of data.
@@karolinanetolicka5982 another question: what are the reasonable limits for the number of job definitions?
Is it OK to create new job definition for each hourly run? (I'm thinking of baking job run parameters into job definition)
@@elephantum Karolina told me there are no limits on the number of jobs per project - so yes, a new job for each hourly run is reasonable. We do have limits on the number of concurrently running executions in the project and will be introducing garbage collection of old executions. To find the page where limits are documented, search "google cloud run quotas".
Run-specific arguments (the thing you asked for) have now been implemented. Do a web search for "google cloud run override job configuration for a specific execution" and you will find the documentation. Thank you for your feedback last year!
I'm wondering why use Cloud Run Jobs and not Cloud Function for such background operation?
Good question! Cloud Functions are great if you prefer deploying source code. Cloud Run is great if you prefer deploying containers.
Cloud Functions do not allow you to determine your environment and does not have the ability to run Shell scripts, Cloud Run Jobs can do both.
Hello Martin, I wonder if there are any plans to use CICD (via GitHub) like regular cloud run to deploy cloud run jobs? Wondering if CICD is deliberately not included or work in progress.
Pretty sure there's a GH action for it. I saw it a week ago
As you say, it's possible to add regular Cloud Run services with CI/CD via GitHub. Cloud Run Jobs are not there yet, but we will add it.
Can I make sure that only one instance of the job is running? I have a Telegram client (python script with Telethon package), so in NO CIRCUMSTANCE i can allow two processes to be at the same time. Otherwise Telegram bans my Session ID, and I have to re-log and redeploy. That's what happened today when a trigger started a new execution while the old one was still running. Ideally, I want to have some number of retires (in case it fails), and a trigger in the morning, but so that the trigger would kill the previous execution. Is Cloud Run Jobs a good choice at all?
This is great, thanks. I was wondering if it's possible to trigger jobs using http
Yes, you can. See the URL that you enter in Cloud Scheduler to schedule a job:cloud.google.com/run/docs/execute/jobs-on-schedule. But be aware that you have to set the authentication header yourself. If you want to trigger your code with an HTTP call, consider putting it inside a regular Cloud Run service instead of a Cloud Run Job.
A single cloud run job task has a max timeout of 60 minutes, but if you start a job with a cloud scheduler, the scheduler has a max timeout of 30 minutes. How can I get around the 30 minute max timeout from the scheduler? Or would the job continue to run to completion, even if the cloud scheduler may timeout?
The Cloud Run job will continue to run until it's done. Make sure to monitor job success by looking at the status of the Cloud Run job, not the status of the Scheduler job - the Scheduler job will show as successful when the Cloud Run job is successfully started, not when it completes.
Hello, I'm attempting to restrict the number of identical Google Cloud Run job instances running concurrently to prevent simultaneous writes to BigQuery tables. However, the --parallelism=1 parameter isn't achieving this goal, did anyone ever experience this situation ?
You're setting "--parallelism=1" but you are still seeing multiple workers running in parallel? That has only happened to me when I triggered a new job before the previous one had finished. Might that be true in your case too? How many jobs do you see running in Cloud Run Jobs section of the Cloud Console?
yes i use "--parallelism=1", I want to prevent multiple instances of the same job from running simultaneously. When I clicked 'Execute' ten times, I noticed in the Cloud Run logs that all ten jobs were executing independently, without waiting for the previous one to finish.
@@moulouelsalim Ah, got it. The "parallelism" setting determines how many parallel tasks start when you click the Execute button. For example, if you set "parallelism" to 10 and start a job, you start executing one job which contains 10 parallel tasks.
@TheMomander does that mean cloud run jub can't achieve my purpose of having only one instance of a cloud run job executing in an instant t, if it's the case I can't use it for containers writing in DB without using my own locks on databases.
@@moulouelsalim Well, you decide when to start a new job (you click the Execute button) and you decide when a job ends (your code completes and exits). So you are in control and you can make sure that only one job is running at a time.
It sounds from your comments like you have started new jobs before your previous jobs finished? Maybe you could share what you're trying to accomplish, and I will let you know how I would approach it.
Excited to try this!
Woah, this could be a game changer which removes the need for VMs or hacks for long-running tasks. Thanks for the demo!
Happy to hear this is useful. Life becomes a little simpler with each eliminated VM.
Thanks for wonderful update
hi how many dags runs in airflow in parelllel processing using cloud run jobs
Not sure I understand your question. Would you elaborate?
If you want to find out more about Airflow on Google Cloud, check out the Cloud Composer documentation.
Nice, is the same k8s job, `apiVersion: batch/v1` if i want to create it from a yaml file
The Cloud Run jobs API is inspired by the k8s jobs API, but it's not the same. You'll likely need to make a few changes to your yaml file.
Question: Is there a way to invoke these jobs from GCS events? or an integration with Eventarc?
Not at the moment, but it's something we may add in the future.
@@karolinanetolicka5982 is there any plans to add continuous deployment to cloud run jobs?
@@RodrigoFernandes-fg6me Karolina is out of the office today, so I will respond. Today you can do continuous deployment of jobs with the command line (which means you can do it from a CI/CD pipeline defined in Cloud Build or GitHub Actions) or with Cloud Deploy.
Good one. Small batch processing dataflows can be eliminated
Awesome 😊
Awesome
I wonder if there’s a way to dynamically compute the number of workers at startup time. I’d imagine not, that you’d use Cloud Scheduler to instead trigger a task that then cascades into scheduling the correct number of workers. Not that _I_ need this, but it did make me wonder.
p.s. For me: “Serverless” is _great_ because I don’t have to worry about maintaining a server anymore for these simple easily containerized workloads and applications. Now maintenance and updates are just done at the application container level, which would have been necessary anyway, and is easily streamlined or even automated with CI/CD.
Good question! I don't know of a way to compute the number of workers at startup time. One way of doing it would be to always start X workers, and each worker has the logic to terminate early if there isn't enough work for it.
That approach would also decouple the logic in workers from the decision of how many workers to create. In other words, you can change the number of workers without having to worry about updating the worker code, and vice versa.
Cool