topic Re: Data format after looped tRestClient in Talend Studio

Data format after looped tRestClient

JDuflot — Thu, 27 Jun 2024 14:27:01 GMT

Hello all,

I'm new to Talend and i'm trying to make a synchronization between two tools. First things first, i'm trying to make API calls to get objects from the first API. My problem is I can only get the objects 50 by 50. I tried to build a loop to get them all (90.000) and it's kind of working. (tJava to init the link -- (onSubjOk) -- tLoop --(Iterate)-- dummy tJava --- tRestClient -- ExtractJsonFields -- tJavaFlex to update the url).

My issue is the way i'm gathering the data. When i don't use this loop setup i get it the way I want :

ID | email | firstname | lastname

1 | email1 | fn1 | ln1

2 | email2 | fn2 | ln2

But when i work with the loop i get it like

url to the next api page |. id | firstname | lastname |

url | ["id1", "id2", "id3"] l ["fn1", "fn2", "fn3"] | ["ln1", "ln2", "ln3"] : these being strings not arrays (hehe). (yes the "[ ]" are part of the strings)

When i try to use the tNormalize component it sorts of multiplies the number of lines and is giving me all of the possibilities (like [id1, fn1, ln1], [id1, fn1, ln2]) which isn't right.

My questions are : is there a way to return the values after my ExtractJsonFields like i used to get them without the loop?
And if not, is there a way to normalize the data knowing that the number of objects returned by the API call will never be the same
After i normalize the data, how can i "store" all of the rows (several thousands) before executing the remaining of the circuit ? (i need to make comparisons with other responses, so i need to do it on all of the data)

I can provide screen shots if needed.

Thank you for reading and thanks even more for taking the time to answer !

Jean

EDIT : I could rebuild clean-formatted objects in a tJavaRow but it can only return one row (the last one, after all the previous ones have been overwritten). Is there a way i can return all of the rows ?

Re: Data format after looped tRestClient

Shicong_Hong — Fri, 28 Jun 2024 02:08:46 GMT

You need a tUnite component after tJavaRow to merge all rows.

Re: Data format after looped tRestClient

JDuflot — Fri, 28 Jun 2024 12:14:06 GMT

Hello and thank you for your answer,

Maybe I didn't explain it well but a tUnite can not hep me there, unless I don't get it right. When my only row containing all of my values like ["id1", "id2", "id3"] l ["fn1", "fn2", "fn3"] | ["ln1", "ln2", "ln3"] get into my tJavaRow, the only row I get in output is ["id3", "fn3", "ln3"]. All of the previous iterations are overwritten before getting out of the tJavaRow component. When i run it with my loop, and if I have for example 55 rows, I only get the 50th row and the 55th (the last one for each iteration) even with tUnite.

Thanks,

Jean

Re: Data format after looped tRestClient

Shicong_Hong — Mon, 01 Jul 2024 07:20:00 GMT

Do you print the output on the console using tLogRow or output them to file or database? I don't understand you said 'the previous row are overwritten？

Re: Data format after looped tRestClient

JDuflot — Mon, 01 Jul 2024 09:06:16 GMT

I print the output on the console using tLogRow. "When my only row containing all of my values like ["id1", "id2", "id3"] l ["fn1", "fn2", "fn3"] | ["ln1", "ln2", "ln3"] get into my tJavaRow, the only row I get in output is ["id3", "fn3", "ln3"]." The overwritten rows are ["id1", "fn1", "ln1"] and ["id2", "fn2", "ln2"], these ones are not printed in the tLogRow. I said overwritten because for every execution, my tJavaRows create a "clean" object and is supposed to return it, but instead it runs again without printing the output and that's why i'm only getting the last row as an output.

Here is how my tJavaRows returns the row (row6 is my output_row)