Fetch a file using a REST API from one Application and Upload it back to another Application using a REST API (IFS / Salesforce / REST / PATCH / BINARY / PNG / PDF)

[aarimaz]

I had a niche requirement where I had to fetch a file from Salesforce and Upload it back to the IFS Document Management system (DOCMAN). I would like to explain the solution I have created here in the community so that someone could make use of it as well as whomever knows a better solution could give their feedback.

Even though it was explicitly Salesforce and IFS in my case, you might have the same scenario between other applications. Here the focus is on fetching a file (especially non-txt format) via REST API as a binary (Stream file type) and sending it back to another REST API.

I would really appreciate it if you could go through my solution and give your feedback if this is good or if there is a better solution only using talend components

Scenario:

• GET REST API to fetch the file data from one Application. (Content-Type: application/octet-stream) (Salesforce)
• PATCH REST API to upload the file data back to an Application (Content-Type: application/octet-stream) (IFS)
• I want to retrieve the file from salesforce using the first API and upload it back to IFS using the 2nd API.
• In this process, it would be best if we can make sure that we are not writing the file to a disk.
• Instead, keep it in cache in a variable and then send it back instantly. (Not sure if this is recommended but I felt this will be a clean solution. any of your feedback would be appreciated)

Components Available to use and issues I faced with each component:

1. tRest and tRestClient -
   1. These are the two main components that will come to your mind when you are dealing with REST APIs.
   2. I tried tRestClient to implement the job and I managed to successfully implement the job.
   3. But it was only working for TXT files. PNG or PDFs were not getting uploaded properly since the stream data that comes from the first REST API gets converted to String within tRestClient.
   4. Again if we manage to get the Stream data somehow, when we pass the stream back to another REST API using tRestClient regardless if it was in byte[] array or not, still it gets converted back to String before it is sent as the body of the REST call.
   5. After reading some posts on the community I concluded that tRest and tRestClient can only handle String-type bodies. (Please correct me if I am wrong and suggest me a better solution)
2. tFileFetch -
   1. After reading more posts on the community, I learned that we could use tFileFetch to retrieve as well as upload files via REST APIs.
   2. But the catch was it can only handle HTTP methods "GET" and "POST"
   3. In my case, I only have a PATCH REST endpoint to work with to upload the file back to IFS.
   4. So I used tFileFetch to retrieve the file as an inputStream (octet-stream / binary / byte[])
      1. then another interesting feature of this component is that you can keep the file data in cache in an inputStream variable and later access in the job.
      2. You don't have to write it to a file when you do this.
   5. but for uploading the file back to IFS, tFilefetch had limitations.
      1. I am unable to use the cache file data to upload.
      2. instead, I can only specify a file location to fetch the file from a disk location
      3. next tFileFetch only has GET and POST. In my case, I had to use a PATCH REST endpoint.
3. tFileInputRaw -
   1. tFileInputRaw or any other component I came across had similar issues as above where it will not support processing an inputStream variable instead, you need to specify a file location to read the file from.
4. Talend Routine (my own java code) -
   1. since I was unable to find a solution on the community or internet, next I explored the option of writing a routine to take care of the upload process.
   2. So now, I can use the tFileFetch to get the file data in its purest form (without converting it to string) and cache it in a variable without saving it in a disk location.
   3. next, I need to take this variable and send it back to IFS using a PATCH REST API endpoint
   4. I came up with a very simple java code block and put that in a routine with a couple of parameters I needed
   5. then I called this routine via a tJavaRow component

Summary:

1. Finally, I managed to implement the job the way I wanted.
2. I used tFileFetch to retrieve the file data and cache it without writing it to the disk
3. then I used a java routine I wrote to take that cached data and upload it back to a REST endpoint using a PATCH, HTTP-method.

Below are some screenshots from the job for a better understanding:

tFileFetch:

 

tFileFetch Parameter to access the cached data:

 

tJavaRow calling the Java Routine and converting the file data from InputStream to a byte[] array:

 

Java Routine:

[Anonymous · Accepted solution]

Thanks for sharing. This is a nice way of demonstrating that no matter the requirement, there is always a way with Talend......even if you need to write a little Java. You can send files using a content type of "multipart/form-data", but the API you are calling needs to support that. I see the issue you have had here and maybe the tRestClient can be adjusted to make this easier. But as I said, this is a great workaround and easily followed by anyone else who may want to do this.

[aarimaz]

Hello rahll!

Thanks for your feedback and suggestions! Yes, I tried multipart/form-data as well but, as you said, it seems the API is not supporting it.

I too wish that it would be nice to have this included in tRestClient. Till then, someone could find this post useful 🙂

Thanks!