topic Incremental Load for the last 60 days data in Talend Studio

Incremental Load for the last 60 days data

Anonymous — Sat, 16 Nov 2024 02:41:33 GMT

Hi,
I'm a bit new to Talend and I have created some jobs to fetch data from APIs and posting into a Database. Now I need to modify the jobs where, when I run the job it should delete at 60 days of data and extract again and then merge with the historical data. Can anyone please help me to solve this issue.
Thanks in advance

Re: Incremental Load for the last 60 days data

billimmer — Tue, 14 Apr 2020 15:57:27 GMT

Use a global variable to hold the dates that you want to prune, and reload. Then use the global variable in your "WHERE" statement in your SQL. For example in a tjava component:

Date d_from;

Date d_to;

String s_from_date;

String s_to_date;

// get date range

d_to = TalendDate.getCurrentDate();

d_from = TalendDate.addDate(d_to,-60,"dd");

// convert to strings in the format of your data. In this

// example, dates are stored as YYYYMMDD in my DB.

s_to_date = TalendDate.formatDate("yyyyMMdd",d_to);

 
  s_from_date = TalendDate.formatDate("yyyyMMdd",d_from); 
 
  // Store in global variable 
 
  globalMap.put("myToDate", s_to_date); 
 
   globalMap.put("myFromDate", s_from_date); 
  
   // show your work 
  
   System.out.println(" To Date: " + (String)globalMap.get("myToDate")); 
  
    System.out.println(" From Date: " + (String)globalMap.get("myFromDate"));

Re: Incremental Load for the last 60 days data

Anonymous — Wed, 15 Apr 2020 09:10:43 GMT

Thanks for the reply. Can you please elaborate more o doing this task. I meant the components which I have to use and how to configure them. I'm pretty new to Talend that's why

Re: Incremental Load for the last 60 days data

billimmer — Wed, 15 Apr 2020 14:22:34 GMT

You will be using the t(your_db)sqlRow component to prune. To insert records you can use the t(your_db)Output or t(your_db)OutputBulkExec.

I recommend you read the on-line documentation on the database components for the database you are using. In that documentation there is an example for each.

Also, there is an e-book on Amazon called "Talend Open Studio Cookbook". This is a good book that will make you productive quickly.