topic Re: how do we read gzip data coming from kinesis stream using talend in Talend Studio

how do we read gzip data coming from kinesis stream using talend

rrohit1_r — Sat, 16 Nov 2024 00:51:22 GMT

currently we are using tKinesisInput component to consume data from kinesis stream. it works when data is in readable format ex- json, but with aws cloud watch log data , subscription filters gzip's data and publishes into the stream. so when

tKinesisInput component reads this data, it cannot unzip it ( field type for payload is string, and it cannot be modified to byte[]) , we see unreadable data and unzip java code says , payload is not a zipped file/format.

we raise a support ticket to give flexibility to change payload data type from string to byte[] , so to unzip using java code, but its been a while and no ETA yet form talend r&d.

im checking in here to see, if there are any alternative solutions to unzip data ( I tried to place data in file and use tunarchive comoponent but it gives same error - not a zipped file/format.

Re: how do we read gzip data coming from kinesis stream using talend

Anonymous — Tue, 19 Jan 2021 07:37:58 GMT

Hello,

Could you please give us your support case Id? In this way, we will check your case status.

Best regards

Sabrina

Re: how do we read gzip data coming from kinesis stream using talend

rrohit1_r — Tue, 19 Jan 2021 14:44:00 GMT

Hello, below are talend and its jira tickets

talend case -

For Case 00202441, How to read and decompress data from AWS Kinesis in Big Data Streaming job, currently the JIRA ( TCOMP-1822 (https://tlnd-prod--zsfjira.na119.visual.force.com/apex/zsfjira__ZIssueDetails?issueKey=TCOMP-1822) ) Unable to decompress data from AWS Kinesis in Big Data Streaming job has been assigned under review for approval with no ETA Date on Fix or Release.

JIRA:

https://jira.talendforge.org/browse/TBD-11753

Unable to decompress data from AWS Kinesis in Big Data Streaming job

Re: how do we read gzip data coming from kinesis stream using talend

Anonymous — Wed, 20 Jan 2021 08:55:27 GMT

Hello,

Our colleagues are working on your feature request and we will keep you up to date about it.

Best regards

Sabrina