topic Re: CURL tRest/tRestClient/tHttpRequest JSON in Talend Studio

CURL tRest/tRestClient/tHttpRequest JSON

raoh — Sat, 16 Nov 2024 02:34:11 GMT

Hello,

I have an API which give me some information about my IT Infrastructure.

Here it's an example :

curl -i -k -H USER:plop -H PASSWORD:plop https://mySite.com/index.php/ApiGetInventory/nna//serveur/Linux_001"

API name is : ApiGetInventory

there are 2 mandatory parameters which are :

nna (void is allowed)
serveur (void is allowed too,but for my example i've put a name of one server. In the near future, I will leave the value at null and thus retrieve the information from the computer park)

In my Studio i've used tSystem which give me :

VALEUR  HTTP/1.1 200 OK
Date: Tue, 28 Apr 2020 07:52:35 GMT
Server: Apache
Set-Cookie: PHPSESSID=xyzazertyuiiop; path=/; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: _pk_ses.5.8853=%2A; expires=Tue, 28-Apr-2020 08:22:35 GMT; Max-Age=1800; path=/
Set-Cookie: _pk_id.5.8853=f67b95e276009301.1588060355.1.1588060355..; expires=Wed, 26-May-2021 07:52:35 GMT; Max-Age=33955200; path=/
Set-Cookie: _pk_cvar.5.8853=false; expires=Tue, 28-Apr-2020 08:22:35 GMT; Max-Age=1800; path=/
Vary: Accept-Encoding
Content-Length: 731
Connection: close
Content-Type: application/json

{
  "nom_api": "ApiGetInventory", 
  "skip": 0, 
  "resultats": [
    {
      "servicePack": "", 
      "uptime": "243 jour(s), 09:58", 
      "version": "Red Hat Linux Server 5.4", 
      "zone": "Area-51", 
      "type": "Physical", 
      "hostname": "Linux_001", 
      "site": "Baie-4", 
      "nna": [], 
      "application": [], 
      "osComplet": "Red Hat Enterprise Linux Server 5.4", 
      "env": "Sandbox", 
      "kbs": [], 
      "dateInventaire": "2020-03-27 07:06:06", 
      "os": "Red Hat Enterprise Linux Server 5.4"
    }
  ], 
  "nb_serveurs_traites": 1, 
  "requete": {
    "os": "Linux", 
    "nna": "", 
    "serveur": "Linux_001"
  }, 
  "code_retour": 0, 
  "information_erreur": "", 
  "nb_resultats": 0
}

Is it the right component to use ? Can i do it with tRest/tRestClient/tHttpRequest ?

How can I easily exploit these results (all information in "resultats": [ { ..........}]) ?

Do i need to create a JSON file ? and exploit it, with tFileInputJson, (to put it in database) ?

I thank you in advance

Re: CURL tRest/tRestClient/tHttpRequest JSON

raoh — Thu, 30 Apr 2020 14:55:18 GMT

i will answer myself :

Here it's my job design :

I do not know if it's the correct way but it's almost working

I connected my tSystem component to a tJava via OnSubjobOK.

Tjava : Empty code

tJavaRow :

tExtractJSONFields :

tJavaRow : Transform String in String[]

i did that, bc length of my string is > 255. So i've created a column as array in postgreSQL

String strName = input_row.kbs;
String[] strArray = new String[] {strName};
//System.out.println(strArray[0]);
output_row.kbs = strArray;
output_row.servicePack 		= input_row.servicePack ;
output_row.uptime        	= input_row.uptime;
output_row.version       	= input_row.version;
output_row.zone          	= input_row.zone;
output_row.type          	= input_row.type;
output_row.hostname      	= input_row.hostname;
output_row.site          	= input_row.site;
output_row.osComplet     	= input_row.osComplet;
output_row.env           	= input_row.env;
output_row.dateInventaire	= input_row.dateInventaire;
output_row.os            	= input_row.os;
output_row.nna           	= input_row.nna;

Is this the right way to proceed ? All suggestions are welcome

weird behaviour : when i put this url in my web broswer, i've got the answer in 30sec (80.000+ records)

https://mySite.com/index.php/ApiGetInventory/nna//serveur//

But via Talend after 2H it's not finish and there is an error : SyntaxError: JSON Parse error: Unexpected EOF

Can I expect to get some answers/suggestions now that most of the work has been done?

Re: CURL tRest/tRestClient/tHttpRequest JSON

raoh — Thu, 30 Apr 2020 15:29:33 GMT

I will answer myself :

First of all to remove header just remove option -i

Here my (final ?) job design :

I do not know if it's correct, but it's almost working ...

I connected my component tSystem to a tJava with "OnSubjobOk". is it correct ?!

In my tJava, code is empty

In my tJavaRow

 row4.json = (String)globalMap.get("tSystem_1_OUTPUT");

in my tExtractJSONFields

in my tJavaRow_3 :

I did that because the value of my KBS column exceeds 255 characters. and I'm inserting, for the time being, as is...

"kbs": [
        {
          "kb": "KB3192137",
          "installedBy": "",
          "description": "Update",
          "dateInstall": "9/12/2016"
        },
        {
          "kb": "KB4091664",
          "installedBy": "",
          "description": "Update",
          "dateInstall": "10/24/2019"
        }]

CODE :

String strName = input_row.kbs;
String[] strArray = new String[] {strName};
//System.out.println(strArray[0]);
output_row.kbs                  = strArray;
output_row.servicePack 	        = input_row.servicePack ;
output_row.uptime        	= input_row.uptime;
output_row.version       	= input_row.version;
output_row.zone          	= input_row.zone;
output_row.type          	= input_row.type;
output_row.hostname             = input_row.hostname;
output_row.site          	= input_row.site;
output_row.osComplet            = input_row.osComplet;
output_row.env           	= input_row.env;
output_row.dateInventaire       = input_row.dateInventaire;
output_row.os            	= input_row.os;
output_row.nna           	= input_row.nna;

it works for my example

strange thing :

when i put this URL in my web browser :

https://mySite.com/index.php/ApiGetInventory/nna//serveur//

I've got the anwser within 30 seconds (same when i use cmd prompt in Windows). But when I do it via Talend, it takes me 2 hours and I have an error of the type : net.minidev.json.parser.ParseException: Unexpected End Of File position 12910591: null

There are, only, 80.000+ records

any idea ? Can I expect to get some answers/suggestions now that most of the work has been done?