topic Re: Talend Development in Talend Studio

Talend Development

SreejithSL — Fri, 10 Jun 2022 14:05:10 GMT

Hello All ,

I have a requirement to pull data from the amazon SQS queue and load to the Redshift table.I already used tsqsInput to connect the SQS queue and print the message to the output console with the help of tlogRow component. In fact I am seeking your support to load the body part of the message to load to a table in Redshift. In internet I didn't find similar use cases to refer . It would be pleasure if anybody could help me to suggest which component can be used this to achieve. Below is a reference to the mapping which I already have.

Re: Talend Development

Anonymous — Sat, 11 Jun 2022 10:28:16 GMT

It provides various software and services for data integration, data management, enterprise application integration, data quality, cloud storage and Big Data. Talend first came into the market in 2005 as the first commercial open source software vendor of data integration software.

Re: Talend Development

Anonymous — Sun, 12 Jun 2022 02:47:47 GMT

@Sreejith Sreedharan Leela , I'm not an expert of SQS queue. I see the tSQSInput component has a default schema, one of column is Body,

This field output the body part of queue?

To load data into Redshift DB, you can use tRedshitOutput component. A simple Job looks like:

tSQSInput--main--tRedshitOutput

Regards

Shong

Re: Talend Development

SreejithSL — Mon, 13 Jun 2022 10:47:29 GMT

@shong

Thanks for your response, in this case the body field consists of json data , I need to load only the body field data into multiple columns in redshift , it seems tExtractJson worked for this use case , I could extract the key value if I used tExtractJson after the SQS input.

Still I appreciated your attention to this query , in fact you are the person who always addressed the queries and support to rethink. Thank you.

May be the below solution would help our community

Re: Talend Development

Anonymous — Mon, 13 Jun 2022 17:34:33 GMT

Yes, tExtractJsonField is the right component that used to extract the value from a Json string.

Re: Talend Development

SreejithSL — Thu, 30 Jun 2022 07:47:58 GMT

Hi All,

I would like to re-open the discussion since the progress to extract data from SQS facing many obstacles . Actually my input queue have an average of 25K messages per day,these messages would need to load to a redshift table for analysis, the challenging part is this need to be achieved through a single load ,but the problem is tSQSInput component allows to extract a maximum of 10 messages at a time. It would be pleasure if anybody share your idea about this.