topic Re: [resolved] How to pass data through a job ? in Talend Studio

[resolved] How to pass data through a job ?

Anonymous — Sat, 16 Nov 2024 11:01:46 GMT

Hello,
I'm new to Talend and I'm trying to achieve my first job for a network project.
Unfortunately I'm stuck at the moment because of my lack of knowledge of the software and its components.
Explanations
I'm collecting data from a network management console API (Cisco Prime Infrastructure) and I need to proceed with 4 API query to get all the necessary data : device id, ip address, interfaces and their descriptions. When the job has gathered everything together, the data are sorted and a last API query is made to the management console.
But the problem is that I don't know how to "link" each rows to the others through the different API calls. I made a pattern to make it clearer :

By now, I know how to get the data and parse them in order to get what I want from the XML files but I can't find how to get the data to "follow" through the job and be linked together.
For each device id should be linked an ip address, a list of its interfaces, the intefaces' descriptions... But how to proceed ?
Thank you.
Cardiganup.

Re: [resolved] How to pass data through a job ?

Anonymous — Mon, 28 Sep 2015 09:06:28 GMT

Hi
tRestClient requires a Document as input request or returns a Document as response, so you can use tXMLMap to extract data from the response or build the Document as input request. For example:
tRestClient1--tXMLMap--tRestClient2--tXMLMap...
Refer to the component manual and learn tXMLMap component.
Best regards
Shong

Re: [resolved] How to pass data through a job ?

Anonymous — Mon, 28 Sep 2015 12:38:13 GMT

Hi,
Thanks for your answer Shong.
I managed to parse the XML responses with a tExtractXMLFields, I can't get tXMLMap to work.
Here's how I did it :

1) "Get devices" Query : "
2) "Parse device_no" parsing the tRESTClient response.
3) "Get interfaces" Query : " + row16.device_no
4) "Parse interface" parsing the tRESTClient response.
I get the results I expect BUT I don't know how to link the devices' ID and the interfaces, how to get that :

So to each device id is linked its interfaces, ip, ... Is there a way to make like individual arrays to store the data for each device along the job?
Thank you.
Cardiganup.

Re: [resolved] How to pass data through a job ?

Anonymous — Tue, 29 Sep 2015 02:07:55 GMT

Hi
If I understand your requirement well, you just need to add a tFlowToIterate between Parse device_no and Get interfaces to iterate each device no. eg:
... Parse device_no--row16--tFlowToIterate--iterate-- Get interfaces--main--parse interfaces--main--tMap--tLogRow
change the URL in step 3 to:
3) "Get interfaces" Query : " + (String)globalMap.get("row16.device_no")
//Assuming the data type of device_no is string, if it is integer type in your case, change it to
"Get interfaces" Query : " + (Integer)globalMap.get("row16.device_no")
on tMap, add a new column called device_no in the output table, and set its expression as:
(String)globalMap.get("row16.device_no")

Best regards
Shong

Re: [resolved] How to pass data through a job ?

Anonymous — Tue, 29 Sep 2015 10:10:44 GMT

Hi Shong,
Many thanks ! I managed to make the job I wanted
Unfortunately, it's not completely done yet because I'm facing another issue...
I tried to switch from my test local environnement to the "real" production environnement I'm supposed to work with, but the tRESTClient can't connect to the server. Let me explain :

Server : Set on the Local Area Network, access via HTTPS, self-signed certificate.
Job :

I added the certificate to the "cacerts" Java file and I set its directory in the tSetKeyStore.
The tRESTClient tries to connect to a url that looks like : https://<certificate-attached-domain>.
I checked "Use authentification" with a "Basic HTTP" type and filled in the credentials.
But it's not working :

 connecting to socket on port 3530
 connected
Exception in component tRESTClient_1
 disconnected
javax.ws.rs.NotAcceptableException: HTTP 406 Not Acceptable
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Unknown Source)
	at org.apache.cxf.jaxrs.client.AbstractClient.convertToWebApplicationException(AbstractClient.java:495)
	at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:897)
	at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865)
	at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:428)
	at org.apache.cxf.jaxrs.client.WebClient.get(WebClient.java:611)
	at ciscoprime.cisco_prime_0_1.Cisco_Prime.tRESTClient_1Process(Cisco_Prime.java:656)
	at ciscoprime.cisco_prime_0_1.Cisco_Prime.tSetKeystore_1Process(Cisco_Prime.java:389)
	at ciscoprime.cisco_prime_0_1.Cisco_Prime.runJobInTOS(Cisco_Prime.java:929)
	at ciscoprime.cisco_prime_0_1.Cisco_Prime.main(Cisco_Prime.java:774)

And if I check "Follow redirects" and allow redirections to different hosts, here's what I get :

 connecting to socket on port 3349
 connected
: org.apache.cxf.transport.http.HTTPConduit - WWW-Authenticate response header is not set
Exception in component tRESTClient_1
javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.lang.reflect.Constructor.newInstance(Unknown Source)
	at org.apache.cxf.jaxrs.client.AbstractClient.convertToWebApplicationException(AbstractClient.java:495)
	at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:897)
	at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865)
	at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:428)
	at org.apache.cxf.jaxrs.client.WebClient.get(WebClient.java:611)
	at ciscoprime.cisco_prime_0_1.Cisco_Prime.tRESTClient_1Process(Cisco_Prime.java:662)
	at ciscoprime.cisco_prime_0_1.Cisco_Prime.tSetKeystore_1Process(Cisco_Prime.java:389)
	at ciscoprime.cisco_prime_0_1.Cisco_Prime.runJobInTOS(Cisco_Prime.java:935)
	at ciscoprime.cisco_prime_0_1.Cisco_Prime.main(Cisco_Prime.java:780)
 disconnected

I also tried to pass the authentification credentials through a Authorization header but it is not working as well.
Have you ever seen this behavior with a tRESTClient?
PS: Prior to using "tRESTClient", I was working with "tREST" components and it was going well.
Thanks.
Cardiganup.

Re: [resolved] How to pass data through a job ?

Anonymous — Wed, 30 Sep 2015 08:29:23 GMT

I solved the problem !
Apparently, the server didn't like to be forced to send XML back so I put "Any" in the type list :

Cardiganup.