is it possible to run streaming in day time and stop at night time , as there is very less data in night time , so we can backfilling again business hour
100%. If you are using trigger(availableNow=True) on your pipelines, then you can trigger the jobs to run and always catch up before stopping. This can be done more aggressively during the day, and then you can relax the cron at night to run every 30 minutes to 1 hour - or really whatever works for the pipeline and the cadence of the data flow (or the expected SLA for data availability)
is it possible to run streaming in day time and stop at night time , as there is very less data in night time , so we can backfilling again business hour
100%. If you are using trigger(availableNow=True) on your pipelines, then you can trigger the jobs to run and always catch up before stopping. This can be done more aggressively during the day, and then you can relax the cron at night to run every 30 minutes to 1 hour - or really whatever works for the pipeline and the cadence of the data flow (or the expected SLA for data availability)
Bartozs Konie is explainig for himself.
We can't hear anything you say
agreed. I stopped watching it live due to lack of clarity.