topic Re: Iterate tRestClient in Talend Studio

Iterate tRestClient

Anonymous — Sat, 16 Nov 2024 05:11:26 GMT

Hi,

I spotted another similar topic, but the advice in there didn't seem to work. I'm therefore posting my particular issue.

I am VERY new to Talend and have only being using it a few weeks, so please be patient

I have first used a tRESTClient to retrieve a list of IDs from an API call.

I now need to feed those IDs, one by one into another tREST call to retrieve some additional information.

I am storing the response from the first tREST call in a tHashOuput, and adding an unique URL relative path as a column called url

e.g.

If I tLogRow the first tREST call subjob, this is how it looks

|---------+------------------------|
|entity_id|url |
|=--------+------------------------|
|1234 |/apiresource/users/1234 |
|1235 |/apiresource/users/1235 |
|---------+------------------------|

What I now need to do is iterate through these rows, one-by-one, making a tREST call.

So I link my tHashOutput to a tHashInput to get the data back. I then do this:

tHashInput >>row7>> tFlowToIterate >>Iterate>> >>OnComponentOk>> tRESTClient >>row(Response)>> tExtractJSONFields >>tMap>> tLogRow

At the end I'm getting TWO tLog rows, both showing TWO rows with null in both output columns

This is my tRESTClient

This is the final output of this job.

Starting job UserMappingTest_2 at 17:04 23/07/2019.

.--+-----------.
| tLogRow_2 |
|=-+----------=|
|id|entity_name|
|=-+----------=|
|null|null |
|null|null |
'--+-----------'

[statistics] disconnected

Job UserMappingTest_2 ended at 17:05 23/07/2019. [exit code=0]

What am I doing wrong?

Thanks

Steve

Re: Iterate tRestClient

Anonymous — Wed, 07 Aug 2019 14:51:48 GMT

I am still struggling with this. Does anybody have any ideas?

Re: Iterate tRestClient

lennelei — Thu, 08 Aug 2019 08:23:17 GMT

Hello,

According to your screenshot and logs, it seems that either you send the wrong data to the tRestClient or your misconfigured the tExtractJSONFields.

You have two tLogRow_2 outputs which is correct as you have two iteration over two inputs row from your first api call.

In each tLogRow_2, you have two rows which may be correct or not depending on the result of the api call and the json parsing.

I would try one of those :

1) output the values of the tFlowToIterate in the tJava_1 for example:

System.out.println("url="+row7.url);

2) add a tLogRow right after the tRestClient_1 in order to check the returned values

3) eventually check what is done by the second line of your screenshot (the tHashInput > tMap > tHashOutput)

You may share the output of the tRestClient_1 and your json parsing as I presume this is where the issue is.

Regards

Re: Iterate tRestClient

Anonymous — Thu, 08 Aug 2019 09:43:36 GMT

Hi @lennelei , many thanks for your reply.

I have slightly re-configured the job now.

The first part of the job is working absolutely fine. It is making an API call via tRESTClient_3, extracting the data via a tExtraxtJSONFields_1 and feeding this data into a tHashOutput_1 component ready to be used in the second part of the job. This tLogRow_3 output showing what is in the hash is as below.

The "url" column is the relative path I with to use in the second tRESTClient component ( tRESTClient_1 )

I wish to process these rows each in turn through the tRESTClient component. So I feed this data from the tHashOuput_1 into the next part of the job by the tHashInput_2 component.

I added the bit of java to the tJava_1 component as you suggested and the output to the console was as below:

url=/resourceful/entity/user-v2/6070

This is the first record in the data, so assume that's correct. I therefore do not know why the tRESTClient component is reporting "path is null"?

I have attached some other pictures below showing the various component setups.

tFlowToIterate_1

tRESTClient_1

tExtractJSONFields_1

I have another job, created by a Talend Consultant, which uses this same method and it works perfectly fine, so I don't understand what I have done wrong here as I have essentially copied the other job and just changed a few components.

When I run the job now, this is the error I am getting.

Exception in component tRESTClient_1 (UserMappingTest_5)
java.lang.IllegalArgumentException: path is null
at org.apache.cxf.jaxrs.impl.UriBuilderImpl.doPath(UriBuilderImpl.java:517)
at org.apache.cxf.jaxrs.impl.UriBuilderImpl.path(UriBuilderImpl.java:512)
at org.apache.cxf.jaxrs.client.WebClient.path(WebClient.java:641)
at ukdm_talendtestproj.usermappingtest_5_0_1.UserMappingTest_5.tRESTClient_1Process(UserMappingTest_5.java:5118)
at ukdm_talendtestproj.usermappingtest_5_0_1.UserMappingTest_5.tHashInput_2Process(UserMappingTest_5.java:3967)
at ukdm_talendtestproj.usermappingtest_5_0_1.UserMappingTest_5.tRESTClient_3Process(UserMappingTest_5.java:3388)
at ukdm_talendtestproj.usermappingtest_5_0_1.UserMappingTest_5.tFileExcelWorkbookOpen_1Process(UserMappingTest_5.java:1033)
at ukdm_talendtestproj.usermappingtest_5_0_1.UserMappingTest_5.tDBConnection_1Process(UserMappingTest_5.java:882)
at ukdm_talendtestproj.usermappingtest_5_0_1.UserMappingTest_5.tPrejob_1Process(UserMappingTest_5.java:5922)
at ukdm_talendtestproj.usermappingtest_5_0_1.UserMappingTest_5.runJobInTOS(UserMappingTest_5.java:6307)
at ukdm_talendtestproj.usermappingtest_5_0_1.UserMappingTest_5.main(UserMappingTest_5.java:6009)

The error is with the tRESTClient_1 component, as adding a tLogRow directly after the tRESTClient doesn't change the result, the same error occurs.

Sorry if I sound dumb, but I'm very new to Talend. I've only been using it a few weeks.

Thanks

Steve

Re: Iterate tRestClient

Anonymous — Thu, 08 Aug 2019 11:56:47 GMT

I have now also gone back to my original configuration of the job, and added a tLogRow directly after the tRESTClient. These are the results.

It looks promising.

I'm getting a tlogrow for each of the two records in the tHashInput so it is definitely responding with the correct data.

However, I soon as I re-introduce the tExtraxtJSONFields_1 component and attempt to output the data to a tLogRow, I get the NULLs again

Here's the output

And here's the tExtractJSONFields component setup. I am trying to extract the "id" and "entity_name" fields from the tREST response (the "string" column from the tLogRow_3 in the first screenshot above)

Re: Iterate tRestClient

lennelei — Thu, 08 Aug 2019 13:58:32 GMT

Hello,

I didnt' read your last 2 posts entirely but I did spot a small mistake:

In one of your tFlowToIterate screenshot, you have unticked the "use default (key, value..." checkbox and used "url" as key for url values.

This means that the global var will be referenced by "url" and not "row7.url" as it would be with the checkbox ticked.

In your tRestClient, you have to use (String)globalMap.get("url") and not (String)globalMap.get("row7.url")

Could you check that (and also check you have the same value in the tJava System.out.println)?

In doubt, use autocompletion to retrieve the correct url global variable.

Regards

Re: Iterate tRestClient

lennelei — Thu, 08 Aug 2019 14:03:57 GMT

Now I've read the second post.

Try to configure the "Loop Json Path query" in your tExtractJSONFields simply to the value "$"

Your input data has no arrays: you don't need a "*"

Re: Iterate tRestClient

Anonymous — Thu, 08 Aug 2019 14:19:24 GMT

Hi @lennelei

I have tried that and get the same issue.

My 2 tLogRow outputs at the end are both null.

It's very odd. If I deactivate tExtractJSONFields_1 , tMap_1 and tLogRow_2 and instead just tLogRow the Response from tRESTClient_1, the data coming back is exactly correct.

So in the tExtractJSONFields_1 component I'm trying to read the "string" column, and extract the "id" and "entity_name" values into columns, then tMap them out to the tLogRow_2 (I will eventually be looking to output these to a CSV file for use elsewhere, but at the moment I just want to know I can extract the correct data).

This is the configuration of the tExtractJSONFields_1 component

I'm really stuck on this one

Thanks

Steve

Re: Iterate tRestClient

lennelei — Thu, 08 Aug 2019 14:21:54 GMT

I did made a second post: replace your json loop with simple "$"

Re: Iterate tRestClient

Anonymous — Thu, 08 Aug 2019 14:22:16 GMT

@lennelei

Just seen your response (coincided with my response).

You're a legend!!!! That's worked. Many thanks for your help

Thanks

Steve

Re: Iterate tRestClient

lennelei — Thu, 08 Aug 2019 14:29:01 GMT

Happy to help you!