topic Re: In-Memory processing variables in Talend Studio

In-Memory processing variables

McDon — Fri, 15 Nov 2024 23:16:37 GMT

Talend Studio 7.3

I am looking to define and update a list of processing variables.

My list would have the fields: source, destination, last updated.

I want to read the source and then only connect to, and update the destination if the source.lastUpdate is > destination.lastUpdate.

This is important as I can cut out connecting to the source and querying it by simply saving this information.

So my thinking is:

1 - read source

2 - if source.lastUpdate <= destination.lastUpdate OR destination.lastUpdate is null then...

3 - connect to destination

4 - update destination

5 - update destination.lastUpdate in memory

6 - sleep for interval

7 - repeat

This is obviously a significantly simplified representation. Connecting to the destination and querying it is bigger than what I've outlined, but this is what I'm trying to accomplish.

I currently have a tFixedFlowInput -> tFlowToIterate -> [connect to source]....

But I don't not know how to update the value in my tFixedFlowInput or something similar.

Any thoughts would be helpful

Re: In-Memory processing variables

McDon — Tue, 25 Jan 2022 23:42:14 GMT

So I thought I would try this:

tSetGlobalvar1.lastLoadDate = 1/1/1900

-> tInfiniteLoop

-> tRest

-> tExtractJsonFields -

-> tFilterRow.lastLoadDate < tRest.response.modifiedDate

--filter--> tSetGlobalVar2.lastLoadDate = tRest.response.modifiedDate

The problem is when I try and add the lastLoadDate to the filter,

it looks as if there are two instances of lastLoadDate (per the attached screenshot).

One instance defined within the scope of tSetGlobalVar1

and another entirely different instance of tSetGlobalVar2.

This defies the "Global" portion of the name of the component.

Have I done something wrong?

My goal is to update one instance of lastLoadDate and use it in the comparison.

Re: In-Memory processing variables

McDon — Tue, 08 Feb 2022 01:28:19 GMT

nothing on this?

I tMemorizeRows has some potential?

Struggling to see how I could get it into a loop to update a lastUpdatedDateTime in each iteration and compare that with the received lastModifiedDate in the JSON response.

Anyone have any examples?

Re: In-Memory processing variables

McDon — Wed, 09 Feb 2022 16:38:03 GMT

No response from anyone?

I will approach this in another way.

tMemorizeRows wasn't helpful, but now that I know it exists and what it is for, I do have other uses for it.

Re: In-Memory processing variables

gjeremy1617088143 — Fri, 11 Feb 2022 13:21:02 GMT

hi, to get the value of a global variable :

(type of your object variable)globalMap.get("variable name")

eg:

(String)globalMap.get("hello")

(int)globalMap.get("age")

to set it

globalMap.put("variable name", new variable value)

eg:

globalMap.put("hello","my name is jeremy")

globalMap.put("age",35)

Send me love and kudos

Re: In-Memory processing variables

McDon — Fri, 11 Feb 2022 17:41:05 GMT

Thank @guenneguez jeremy for your response.

I am currently using that method for a single variable.

However, in this particular case, I am doing

tFixedFlowInput -> tInfiniteLoop -> tFlowToIterate -> ... all sorts of processing stuff...

My tFixedFlowInput contains a list to operate on and processing parameters for each.

These are IDs that need to be used in a RESTAPI call along with other processing parameters that vary for each entry.

My goal is to make my flow based on an input list.

One of the things I want to accomplish is storing that last time I loaded that specific entry to my database.

Each call I make checks the last modified date, if the source data has not change, I terminate - no need to incur processing costs or time.

When I do load new data, I want to update my last load date ONLY FOR THAT ENTRY.

So I can successfully read the tFixedFlow input data.

How do I update it for the next iteration of the loop?

Note: I do not need to update the static entry defined in tFixedFlow input.

I only want to update the in-memory copy.

When the flow is terminated and restarted, I will begin again with whatever was originally defined in the tFixedFlowInput.

I would prefer to not use a database control table as that somewhat defeats the purpose. I want to check every 5 minutes and don't want to query my database 4 times (1 for each tFixedFlowInput entry) every 5 minutes.

Using a Webhook is not an option right now. My project does not have the web infrastructure to support that yet and I cannot hold up this implementation for that.

Thank you again for the interest and to anyone who gives this a look.