topic Re: [resolved] TOS REST client iteratating and parsing in Talend Studio

[resolved] TOS REST client iteratating and parsing

TXAggie00 — Mon, 01 Jul 2013 22:21:24 GMT

Hello all!
I have recently been demoing talend as an ETL tool (currently SAP Data Services and Informatica developer). I am having some issues with a RESTful WS that returns only JSON data. The data returned is paged, so I need to be able to iterate through the rest calls until a key/value json field returns false. I have a Boolean flag in the globalmap that holds this value, but I can only attach a loop to the tRest component and not the tRESTClient component. I have yet been able to get the tRest component to return anything other than a string which I can't figure out how to process.
The other issue I have is that this particular CRM's WS is not particularly defined well. For instance, the main element that I will loop through is data. It has a child element that can either be a parent node or just an element. For example:
It can look like this:

"data" :

or this:

"data" :

Not quite sure how to best process this. In tRESTClient, I either have to define something as an element or a node, it can't be both.
Any idea?
Thanks,
Scott

Re: [resolved] TOS REST client iteratating and parsing

Anonymous — Thu, 04 Jul 2013 02:36:21 GMT

I have yet been able to get the tRest component to return anything other than a string which I can't figure out how to process.

After tRest, you can use a tExtractJsonField to extract the value from the response string, if the response format is not fixed, parse the response on a tJavaRow first, and then use different tExtractJsonField with different mapping base on the parse result to extract the value, for example:
tRest--main--tJavaRow--runIf_1--tFixedFlowInput--main--tExtractJsonField--main--tLogRow
--runIf_2--tFixedFlowInput--main--tExtractJsonField--main--tLogRow
Shong

Re: [resolved] TOS REST client iteratating and parsing

TXAggie00 — Fri, 05 Jul 2013 15:57:34 GMT

After tRest, you can use a tExtractJsonField to extract the value from the response string, if the response format is not fixed, parse the response on a tJavaRow first, and then use different tExtractJsonField with different mapping base on the parse result to extract the value...

Thanks for the reply Shong. I found a solution using the custom component tParseJSON. What I ended up doing for the elements that could also be nodes, was brought them in as Objects and then wrote the following custom code that parses the string passed in as that object:

    public static Integer getJSONValue(Object str) {
    	Integer myNullInt = null;
    	if(str == null || str.toString().trim().length()==0) {
    		return myNullInt;
    	}
    	String object = str.toString().trim();
    	int id;
    	try {
    		id = Integer.parseInt(object);
    		return id;
    	} catch(Exception e) {
    		String[] jsonArr = object.split(",");
    		for(int i=0;i<jsonArr.length; i++) {
    			if(jsonArr.contains("\"value\"")) {
    				String[] idVal = jsonArr.split(":");
    				idVal = idVal.replace('"', ' ').replace('}', ' ').replace('{', ' ').trim();
    				try {
    					id = Integer.parseInt(idVal);
    					return id;
    				} catch (Exception ex) {
    					return myNullInt;
    				}
    			}
    		}
    		return myNullInt;
    	}
    }

I attached the screenshot of the flow in case it helps anyone else. tParseJSON was found at https://github.com/ijokarumawak/tParseJSON
Thanks,
Scott