One thing I forgot to cover but is super important in this video is to have a Dead Letter Queue in this design. When a file is failed to be processed, the finalised object event can be sent to a another PubSub topic which serves as a Dead Letter Queue. This can then be used for multiples purposes such as monitoring, alerting, as well as ingesting that data into BigQuery for further analysis (gives a lot of power on the debugging side) so you know exactly what can’t be processed and when.
One thing I forgot to cover but is super important in this video is to have a Dead Letter Queue in this design. When a file is failed to be processed, the finalised object event can be sent to a another PubSub topic which serves as a Dead Letter Queue. This can then be used for multiples purposes such as monitoring, alerting, as well as ingesting that data into BigQuery for further analysis (gives a lot of power on the debugging side) so you know exactly what can’t be processed and when.