There is no point if your goal is to reduce storage. The goal achieved here is to have a separate table with reduced sizing for eventual query/reduction in query cost/improved query performance. Azure Data Explorer is an APPEND-ONLY resource. It's not meant to be used for large scale data deletion/removal. To transform your data being streamed in beforehand (if your intent is to not maintain the original large raw data sample) would be to preprocess it before ingestion, such as by using Azure Stream Analytics, Azure Functions, or Azure Data Factory.
Great presentation and good workflow for update policies and materialized view creation.
I didn't know about the create table using query. Great tip!
what are the benefits of separating the deduplication step from the update policy? it consumes more storage.
There is no point if your goal is to reduce storage. The goal achieved here is to have a separate table with reduced sizing for eventual query/reduction in query cost/improved query performance.
Azure Data Explorer is an APPEND-ONLY resource. It's not meant to be used for large scale data deletion/removal.
To transform your data being streamed in beforehand (if your intent is to not maintain the original large raw data sample) would be to preprocess it before ingestion, such as by using Azure Stream Analytics, Azure Functions, or Azure Data Factory.
awesome that's exactly I needed... OPCUA data is really hard to model..
Excellent!
fabulous !!
👌👌