topic tMap with one source iterate tRestClient in Talend Studio

tMap with one source iterate tRestClient

dura45 — Tue, 19 Mar 2024 13:37:53 GMT

Hi, I'm trying to configure my job and these are my expectations:

- From one side I've a tFlowTolterate that iterates the id value from a db table( table 1), an use it for a tRestClient params

- From the other side I've another db table (table2) with the existing customers addresses informations

I would only add the new addresses from the tRestClient to the table2 and for reach this goal I'm using tMap component.

Maybe it works, but I'm sure that I can do better than now. It's possible to query all the tRestClient addresses and only after it is over, pass them to tMap component?

If I using "on Component OK", I'm not able to using it because I see a prohibition icon.

Anyone can help me? Thanks!

Re: tMap with one source iterate tRestClient

Shicong_Hong — Wed, 20 Mar 2024 00:49:06 GMT

Hello

Before all of them are passed to tMap, you can add a tUnite between tRestClient and tMap to merge all address information.

Regards

Shicong

Re: tMap with one source iterate tRestClient

dura45 — Wed, 20 Mar 2024 07:55:01 GMT

Hi @Shicong_Hong and thank you for you help.

I have a question about your answer: I want to load only the new addresses (and don't load existing ones), but before tMap I merge all address information with the tUnite, I doubt whether I will upload all the addresses and not just the new ones.

If I understood well I merge all the address before tMap and after that the job reload always all the addresses, even if they already exist. Right?

There a way for load only the newest? Thanks

Re: tMap with one source iterate tRestClient

Shicong_Hong — Wed, 20 Mar 2024 09:37:28 GMT

Maybe I don't understand your question well, you mentioned " It's possible to query all the tRestClient addresses and only after it is over, pass them to tMap component?", I thought the address information are returned from API called by tRestClient.

Can you please clarify your question to explain you are trying to achieve? What data is returned from API? I'm guessing you're using a tMap for the join to get only the new addresses and insert them to table 2?

Re: tMap with one source iterate tRestClient

dura45 — Wed, 20 Mar 2024 10:30:01 GMT

I'm sorry for the lack of clarity in my question.

I need to schedule an import into a database of all the addresses from an e-commerce site. To do this, I imported all the addresses in bulk a while ago, and now I want to add only the new addresses created to the database.

So, the data sources I have are:

S1 - an API call to extract all the existing addresses in an e-commerce
S2 - a snapshot of the addresses in a database from the first import

Now, I want to make sure that all the new addresses created on the e-commerce, which I get through the API call (S1), are added to S2, while the existing ones in S2 are not touched.

To make the API call, I need to iterate through all the account IDs present in the e-commerce (provided by a third table).

So, what I want to do is: download all the addresses via API and, only after doing so, use tMap to compare those that already exist in S2 and insert only the new ones.

Thanks

Re: tMap with one source iterate tRestClient

Shicong_Hong — Thu, 21 Mar 2024 01:05:00 GMT

Thanks you for your clarification in details. The existing addresses are loaded only one time from S2 (tDBInput_2), this is a lookup table. Use a tUnite before tMap to merge all the addresses returned by each iteration, so tUnite is the key component you are looking for.

Please try it and let me know if it works.

Regards

Shicong