topic Re: Implementing Incremental Load in My Staging Layer in Talend Studio

Implementing Incremental Load in My Staging Layer

sushantk19 — Sat, 16 Nov 2024 02:38:38 GMT

Hi All,

I have to Implement Incremental Load in my Staging Layer. I have used the field Updated_at(date field) from source table to do this. After loading the data in my stage table, I update my Stage Incremental Table (externally created in database as mentioned in some of the posts) with latest update_at date.

Now i have to use this date for future purpose to extract incremental data. I want to create a context and pass this date value to context and then use this context in my source query. How can i do this? basically how do we define contexts which can store date value from my incremental Stage table?

Re: Implementing Incremental Load in My Staging Layer

manodwhb — Mon, 20 Apr 2020 12:26:05 GMT

@sushantk19, read the date value-form DB using tdbinput >>main connected tjavarow and you can set context variable like below. you need to create a context in context tab.

context.last_updated_date=row1.date;

from the above job you use on subjob ok and implement the main flow in the tDBInput that query.

"select * from where date>+context.last_updated_date"

Re: Implementing Incremental Load in My Staging Layer

sushantk19 — Mon, 20 Apr 2020 16:22:13 GMT

Please see the screenshots attached. I have tried to implement as you have mentioned above, but gives an error like I am passing NULL value to Context variable Last_run. what value should i put there?

Also, if i make the context variable as "String" instead of date and pass some date value to it, it doesnt pass that value through tJAVA component.can u please check and suggest?

context_defination.png
extracting_source.png
Source_Query.png
tjava transformation.png

Re: Implementing Incremental Load in My Staging Layer

sushantk19 — Tue, 21 Apr 2020 16:50:20 GMT

Hello Friends,

Please help. I have a deadline to meet 😞

Can you please share/show how to define context? and how to update tJavaRow component.

Presently, the tJavaRow component is :

context.last_run = input_row.Incremental_at;

and defined last_run as datatype "date" with no values assigned. Is this correct?