Getting Started with tRest and API's

[Anonymous]

Hi All,

Still pretty new to Talend but I need some help please.

I am currently using Talend Open Studio and I have managed to create a few jobs and task to make my life as DBA easier. Also managed to setup a job that actually connects to Google Analytics, retrieves stats and then updated local databases for reporting etc...

So although new, I think I am getting a good handle on this. (Considering I started using it 10 months ago and already have 33 active autonomous jobs running daily)

But... hahaha

I have received a new request from my manager to pull data, preferably via Talend, from a site and then process it to our local stores, much like the analytics data, except I am having a hard time getting my head around how this actually works using tRest or tRestClient.

Any tutorials or guidance would be helpful at this point.

So, what I have to use is this:

LSRC2 API

After tinkering with the settings and going through some forums and digging (a lot of digging), I managed to get it a response from the web service.

(Response has been attached.)

And this is fine, at least I know I am doing the call correctly. But now I am stuck. And I don't really understand how to move forward or where to begin.

I know that I am hitting the server correctly, but I have no clue as to how I am supposed to now call the other functions/methods to get the data that I am looking for. In particular, I am really only interested in the 'export_responses' function/method, nothing more and nothing less. Can anyone please provide some guidance on how this actually works? Not really understanding what to do after the initial REST call.


Any help and advice or guidance would be highly appreciated.

[Anonymous]

Looks like the endpoint you hit just returns a standard definition of all available functions.  You'll need to find the right endpoint and pass correct parameters to your tRestClient.  Typically what I do to parse out the data is first copy it to a file then import that file into a  tXMLMap if your response is XML

or tExtractJSONFields if it is JSON.  

[Anonymous]

Ok, I think you are saying that you have figured out how to authorise yourself (when you make contact), but do not know how to call subsequent functions. If that is the case, it is relatively straight forward.....but will require more details on the function calls. Essentially the functions will be further service calls. But with each of these you will be required to send the authentication supplied by the call I presume you are making already. So essentially you need to make that call, retrieve the returned String (I believe it is held as the "result" in your environment), then you will need to supply that value (possibly as a header) to your subsequent calls. 

Maybe take a look at this tutorial (https://www.rilhia.com/tutorials/using-oauth-20-talend-access-google-apis). Now this is MUCH more complicated than what you will need to do (OAuth2.0 is a little trickier than your requirement, but should give you an idea of the overall process you are after.

Essentially you need to authenticate and retrieve a token, then use that token in each of your function calls for that session.

Have a play and you will soon get to grips with it.

[Anonymous]

Ok, yes, that makes sense and I'll give it a try today. Thanks for both your input.

[Anonymous]

Hi,

So I am getting the process a bit better now. Still struggling with stuff that I honestly feel should be easier, but whatever.

Question on my next problem now.

The actual functions require that the session key gotten from the first tRest request be passed to the next request. After some tinkering and going through the link provide above I have setup a context variable that holds this key.

In the next tRest I have to pass the key.

"{
\"method\":\"list_questions\",
\"params\":[
\"+context.SessionKey+\", or \""+context.SessionKey+"\" or \"'+context.SessionKey+'\" or \"context.SessionKey\" (none make a difference at this point.)
\"378563\"],
\"id\":\"1\"
}"

But doing this only gives me a 'Invalid session key' message. To me this means either:

1) the first key is no longer valid (session closed after the request was logged)
2) the context variable isn't passed correctly or at all (I have tried numerous methods)

Any input or advice?

[Anonymous]

I believe your JSON should probably be something like below....

"{
\"method\":\"list_questions\",
\"params\":[
\""+context.SessionKey+"\",
\"378563\"],
\"id\":\"1\"
}"

....having said that, I have nothing to base this assumption on apart from what you have tried. The above is just correctly formatted for what you have given. Can you point us to where the service specification is by any chance? Also, what is 378563?

Unfortunately web services are notoriously complicated and Talend really can't make them any easier while enabling all of the functionality that is out there. However, you do tend to get used to the irritating issues. 

Can you also send the error message you get, as well as the specification you have been working to?

[Anonymous]

Hi,

Thanks. Service spec is in the original API doc I linked. Additional to that is the actual API functions.

LSRC2 API
API Functions

The 378563 is one of the surveys on Lime Survey. Before I do anything else I wanted to see if it would just send me the questions in this survey. (If that works I can try something more advance like getting the responses which has a bucket load of parameters to pass.)

export_responses(string $sSessionKey, int $iSurveyID, string $sDocumentType, string $sLanguageCode = null, string $sCompletionStatus = 'all', string $sHeadingType = 'code', string $sResponseType = 'short', integer $iFromResponseID = null, integer $iToResponseID = null, array $aFields = null)

As per the API docs, if I look at the string formats, if you where to do queries via Java/Python libraries:

Java Call;

post.setEntity( new StringEntity("{\"method\": \"list_groups\", \"params\": [ \""+sessionKey+"\", \"ID_SURVEY\" ], \"id\": 1}"));

Python Call;

data='{\"method\":\"get_question_properties\",\"params\":[\"'+skey+'\",'+QuestionID
                                    +',[\"gid\",\"type\",\"help\",\"language\",\"sid\",\"question_order\",\"question\",\"subquestions\"]],\"id\": 1}'

So if I consider this, and the fact that to get the actual Session key I do:

"{
\"method\":\"get_session_key\",
\"params\":[
\"***my_user_name***\",
\"***my_password_***\"],
\"id\":\"1\"
}"

And that actually works and gets the key... The other calls should work. Unless after the first call, the connection is closed. Currently, after I have the key, I do a 'On component ok' after setting the context.

[Anonymous · Accepted solution]

OK, there are few things that could be the cause of this. First of all, lets make sure your value is passed correctly. In Talend Context variables are usually set for the lifetime of the job. That is not always the case and they can be changed during the job, but using the tContextLoad to set a dynamic variable like this is not ideal. A slightly better way of doing this is to use the tSetGlobalVar component. This allows you to set a Java HashMap with a key/value pair that is available to the whole job. Take a look here to see how to use it (https://help.talend.com/reader/KxVIhxtXBBFymmkkWJ~O4Q/yjzLPbgCOJF~YJGNskhmMQ). You can simply add it as a replacement to your tContexrLoad component.

The other thing to change would be to remove the OnComponentOK link between the tContextLoad (which will be gone) and connect an OnSubJobOK from your tRest to a tJava and then another one from the tJava to the tRest below. This link is fired when a subjob is completely finished. They are far better at linking subjobs than OnComponentOK links. 

Let's say you call your generated  session key "session_key" in your tSetGlobVar. What we need to do is test that it is set OK. That is what we use the tJava for.  In the tJava component, use the following code....

System.out.println("Session key:"+((String)globalMap.get("session_key")));

You can use this tJava for other logging until you get this working. It might be a good idea to actually build your JSON String here, save it in the globalMap and then use the computed JSON in your tREST. To do that you would set your JSON into the globalMap in code as below...

String json = {JSON code};
globalMap.put("my_JSON", json);

Then in your tREST just put the below in your Body window....without quotes....

((String)globalMap.get("my_JSON"))

You can then print out the precise JSON you are using in the tJava.

Once you have done the above, you will be in a better condition to assess where you are going wrong....or give us more info.

[Anonymous]

Ah, yes, haven't thought of it that way. Will definitely give it a shot. Would be a way better structure as well.
Thanks for the tip on the On Subjob ok. I have used it before, along with the tPre/Post Job components in some of my other projects (call stats, data mart updates and queries etc...) Just thought I'd use that as a quick stop along the way.

[Anonymous]

Thanks,

Don't know where the typo was, but restructuring and cleaning everything up seems to have solved the issue. Getting the feeds that I am looking for and able to process most.

Golden info received here, thanks a million.