topic Re: Find out if a ZIP / GZ file is corrupt in Talend Studio

Find out if a ZIP / GZ file is corrupt

kleinmat — Mon, 31 Aug 2015 14:32:24 GMT

Hi there,
I have this very "funny" scenario where I receive .GZ files from a partner system - but some of the files are corrupt.
So I need to have a process which checks the integrity of the file - and if it is corrupt, skip it.
The process may not abort, though, which is the default behavior of Talend.
So how can I make sure I only process "good" files and skip "bad" ones?
Thanks
Matt

Re: Find out if a ZIP / GZ file is corrupt

willm1 — Mon, 31 Aug 2015 16:07:36 GMT

I expected the 'check integrity' to skip bad files and proceed gracefully. As alternative, how about using a tSystem component and executing the unzip command per file. That way you can also capture info about what files returned errors.

Re: Find out if a ZIP / GZ file is corrupt

kleinmat — Mon, 31 Aug 2015 16:21:16 GMT

That's what I expected, too. Unfortunately, it does not act that way. Instead, it crashes as if the checkmark was not set. Well "crashing" is not the right word: it throws an exception and jumps out of the iterator. So it does not continue after detecting a bad file.
Plus, the integrity check does not even work properly (e.g. it does not recognize incomplete files; for instance files that have been properly compressed but were only transmitted to a target system incomplete)

Re: Find out if a ZIP / GZ file is corrupt

willm1 — Mon, 31 Aug 2015 16:29:36 GMT

I had faced a similar issue in the past... Agree with you. I've created a Jira ticket to have this looked at and hopefully resolved soon - https://jira.talendforge.org/browse/TDI-33802

Re: Find out if a ZIP / GZ file is corrupt

kleinmat — Mon, 31 Aug 2015 16:37:08 GMT

Thank you willm

Let's hope Talend does something about it soon.
In TOS DI 6.0, the issue is still there, though

Re: Find out if a ZIP / GZ file is corrupt

Anonymous — Mon, 31 Aug 2015 17:13:36 GMT

Thanks for reporting this issue in our bugtracker.

Re: Find out if a ZIP / GZ file is corrupt

Anonymous — Tue, 01 Sep 2015 08:41:46 GMT

hi all,

for job that I expect 'normal' failure - corrupted zip file are a good example - I put them in a 'child' job and uncheck the 'Die on child error' option.

The father job manage loop and/or iteration and children jobs only the threatment.
Could be a workaround ...

hope it help

regards
laurent