So we can use: 'table' -- for truncate and load 'incremental' -- for append or insert 'incremental' with 'unique_key' -- for upsert or merge.... Is the above statement is right?
Hey Anirvan, Thanks for clearly explaining. I am currently learding dbt and I came across this question whether we can keep multiple where conditions in incremental load
hello I have question dbt doesn't recognize my model as incremental I using incremental modling to take snap shot of table row count and insert it to build time serise table contain row conut for every day
Hello, i have a question how to do insert update and delete based on column other than a date. I am loading from Excel into postgres and generating a hashed column, every time a new record, or updated record a new hash key is generated for that column. I am trying to do an incremental update. Here is the select stmnt SELECT id, "name", alternate_name, description, email, website, tax_status, tax_id, year_incorporated, legal_status, logo, uri, parent_organization_id, hashed_value FROM organization; My CDC is based on hashed_column. Lets say name is changed in excel, when i load the data into postgres i get a new hashed key for the hashed_value column and similarly for a new record. How do i do my incremental load? any suggestion
that concept is called as change data feed , you need to first find out the records which have changed , There are different techniques , like in sql you can do SELECT * FROM table1 EXCEPT SELECT * FROM table2 to find what rows have changes then only insert those records.
So we can use:
'table' -- for truncate and load
'incremental' -- for append or insert
'incremental' with 'unique_key' -- for upsert or merge....
Is the above statement is right?
Yes that is correct, keep watching
5 completed ❤
Hey Anirvan, Thanks for clearly explaining. I am currently learding dbt and I came across this question whether we can keep multiple where conditions in incremental load
Yes, definitely. think it as a sql query with which you are filtering out the data.
@@anirvandecodes Hello Anirvan, any code snippet or any format suggestion from your end??
hello I have question dbt doesn't recognize my model as incremental
I using incremental modling to take snap shot of table row count and insert it to build time serise table contain row conut for every day
I will upload one video on snapshot soon , check that out !
Hello, i have a question how to do insert update and delete based on column other than a date. I am loading from Excel into postgres and generating a hashed column, every time a new record, or updated record a new hash key is generated for that column. I am trying to do an incremental update. Here is the select stmnt
SELECT id, "name", alternate_name, description, email, website, tax_status, tax_id, year_incorporated, legal_status, logo, uri, parent_organization_id, hashed_value
FROM organization; My CDC is based on hashed_column. Lets say name is changed in excel, when i load the data into postgres i get a new hashed key for the hashed_value column and similarly for a new record. How do i do my incremental load? any suggestion
that concept is called as change data feed , you need to first find out the records which have changed , There are different techniques , like in sql you can do SELECT * FROM table1
EXCEPT
SELECT * FROM table2 to find what rows have changes then only insert those records.
How to achieve incremental insert in dbt without allowing duplicates base on specific columns?
You can apply distinct in sql to remove the duplicates or use any other strategy to remove the duplicates
cool thanks
No problem!
Can you please share the dbt models as well
sorry i lost the model file
Hiii
hello