The art of the possible , with an ESB

[SteveSalt]

Hi,

I am very new to the Talend ESB Studio, so bear with me. I have been tasked with researching a solution to a business ​need. I am not looking for others to write a solution for me, just pointers where I should be looking.

We will be receiving a JSON payload from a Partner, and ultimately, it will store the contents of the payload in a MS SQL DB, then transform the content into a suitable form to be ingested by our ERP.  

We don't have the API piece in place, but I'm assuming I can configure the API to direct the payload to the input of a Talend job, using the API Designer.

What I was hoping to do is put together a series of jobs that have small, but defined processes, and then join these together to form a larger system,  as we have to also send data back to the Partner, in a prescribed form (based on the layut described by their XSD).

To me, this seems like a normal sort of transaction that most businesses would undertake, I'm just trying to fathom out what Talend objects I need, to achieve this.

I have configured a basic Metadata > File Xml , using the XSD to configure the columns from that schema, what I don't know is whether I can use example Payloads for each of the different paths that would be taken, to map the elements required for each differing setup.

E.g. Payload 1 for Customer B, will hit Input 1 on Job 1 and be processed accordingly, even though Job 1 used an example payload for Customer A

Hope that makes sense.

Regards,

Steve

[David_Beaty]

Hi,

I'd consider decoupling the API bit from the MS SQL DB with a message queue, say something like ActiveMQ. Doing this would allow the API to be reasonably lightweight; to just receive the message and put it onto the queue. A 2nd more heavyweight process can take the messages off the queue, validate its in the format you expect and pull them apart and process.

If you have Talend Cloud, which it sounds you do, I'd strongly recommend the use of the API Designer and API Tester in the process of development and testing. The API designer is a fancy front end to designing the calls you can make to the API, which you then import into your Repository metadata.

Kind regards

David

[SteveSalt]

Hi David, Thanks for your response – the only Talend software we currently have is the free ESB Studio. We are considering our options, but I thought that with the ESB studio, I could demonstrate the transformation of the payload as an initial POC of the broader solution. I know the CTO will ask whether the problem can be solved using just Talend software, I think as an attempt to reduce the number of touchpoints. It’s been well over a decade since I’ve had involvement with an ESB – we had listeners configured to a specific URL, and our partners would post their data to that – using content-based-routing we would transform the message into our standard and direct it to an appropriate endpoint, when our systems would process it downstream. We don’t have such systems where I work now, but we are looking to automate areas of the business – so I latched onto the ESB product, as I thought this may fit our requirements. Regards, Steve