topic Re: tBufferOutput/Input components are changing the value of the byte array in Talend Studio

tBufferOutput/Input components are changing the value of the byte array

Metikosh — Fri, 17 Dec 2021 13:52:42 GMT

Hi,

As described in the subject, I am having an issue with the tBufferOutput/Input components. I have a table in a MariaDB database which contains a column which is BINARY datatype (in this column I have a hash value calculated with SHA256). For this example I used a single row so it would be easy to notice the difference.

Here is a screenshot of my job:

Following are screenshots of the schemas of the job components:

And finally here is the result (testDataHash1 is the source table and testDataHash2 is the destination table - both have the same definition - only one column BINARY with length 64).

This is a simplified scenario to demonstrate the issue I am facing - I actually have complex parent-child jobs where I need to propagate data from the child to the parent job (including the DataHash column which I am later using in a tMap component as a lookup; the lookup fails because the values are different because of the tBuffer components).

There is no workaround in my job logic (in terms of not using the tBufferOuput/Input), so I would appreciate any advice on how to tackle this issue!

Thanks!

Re: tBufferOutput/Input components are changing the value of the byte array

gjeremy1617088143 — Tue, 21 Dec 2021 08:54:20 GMT

Hi, it seem you have an encoding problem,

check the encoding used in the advanced parameters of tBuffer components .

Send me love and Kudos

Re: tBufferOutput/Input components are changing the value of the byte array

Metikosh — Tue, 21 Dec 2021 09:32:09 GMT

Hi gjeremy1617088143,

Thank you for your answer but I'm afraid that is not the case.

I forgot to mention it in my question but I have already tried executing the job with different encoding selected in the advanced settings of the tBufferInput component (I used UTF-8, ISO-8859-15, ASCII, Cp1252, UTF16, UTF32 and some other encodings that I thought would make a difference) and all I got was multiple different values for the DataHash column - none of them is the same as the original.

If you have some other suggestions please share them with me.

Thanks.

Re: tBufferOutput/Input components are changing the value of the byte array

gjeremy1617088143 — Tue, 21 Dec 2021 09:53:39 GMT

you can add additionnal parameters in your tDB components to force encoding

Re: tBufferOutput/Input components are changing the value of the byte array

Metikosh — Tue, 21 Dec 2021 10:13:12 GMT

I don't think that the encoding would affect a binary column - but I tried this as you suggested and it does not fix the problem.

I have tested this also without the tBuffer components (just passing the row from tDBInput to tDBOutput) with different encodings in the additional JDBC parameters and it did not affect the value at all (I used latin1 and UTF8). So I think that the encoding in the source-destination is not the problem, the problem is with the tBuffer components (and again not with the encoding because changing it did nothing to fix the issue).

Update: I also tried force encoding in the source as latin1 and mapping it in a destination with UTF8 (without the tBuffer components) - the value of DataHash stayed the same.

Re: tBufferOutput/Input components are changing the value of the byte array

gjeremy1617088143 — Tue, 21 Dec 2021 10:34:22 GMT

have you tried with other component like tHashinput and tHashOutput to replace tBuffer component ?

Re: tBufferOutput/Input components are changing the value of the byte array

Metikosh — Tue, 21 Dec 2021 10:54:52 GMT

tHashinput and tHashOutput components work perfectly well with hashed values - and I use them in my jobs - but tHashinput and tHashOutput are not useful in this situation because I need to propagate data from a child to a parent job (what is shown in the screenshots in my question is just a simple example to demonstrate the issue and not my actual job) and as far as I know I can't achieve that with the tHash components, that is why I am using tBuffer components in the first place.

Re: tBufferOutput/Input components are changing the value of the byte array

gjeremy1617088143 — Tue, 21 Dec 2021 13:18:57 GMT

hi you could use this :

https://help.talend.com/r/i6eFKBuNsRD2KzBCYnXHhw/4jPcdaVw7eaDvMyLDjdYfQ

instead of a String you declare a byte[], then you can get the byte[] or set it in all your job

Re: tBufferOutput/Input components are changing the value of the byte array

Metikosh — Tue, 21 Dec 2021 16:15:08 GMT

This would probably work if I wanted to pass a single value - because for multiple rows the variable value would be reset for every row and in the parent job I would get only the last value of millions of rows. Maybe my example job confused you because in the example I am moving a single row (single value), but I am not actually working with a single value - I will have millions of rows for the DataHash column.

Thanks a lot for your time, I really appreciate your input!

Re: tBufferOutput/Input components are changing the value of the byte array

gjeremy1617088143 — Tue, 21 Dec 2021 16:52:00 GMT

so you can use the method with private variable and getter setter with a List<byte[]> instead of byte[]

Re: tBufferOutput/Input components are changing the value of the byte array

gjeremy1617088143 — Wed, 22 Dec 2021 15:06:18 GMT

for example you can create a private static List<byte[]> in a routine.

on the child job you use tJavaFlex : in the begin part you instanciate the liste, in the main you add the byte[] of the current row in the ArrayList on the end part you set the private var with the list you filled.

on the father job you use another tJavaFlex with a foreach clause inside to send the list to a flow.

here for eg I use a list of string and globalVar for put or get the list :

you just have to transpose it to list<byte[]> and use getter setter of private variable in a routine instead of globalVar

Re: tBufferOutput/Input components are changing the value of the byte array

Metikosh — Fri, 25 Feb 2022 15:06:05 GMT

Hi @guenneguez jeremy ,

Thank you for your input! I am sorry for the late reply, but priorities shifted on my side and I didn't have the time to test your approach.

So finally I have tested it and it works great! However, I would like to inform you that the mistake was on my side - the tBuffer components work as they should, my data was actually incorrect (I won't go into details, let's just say that it was incompatible...). But thank you for introducing the global routine approach to me, as I was unfamiliar with the usage of global routines.

I will select your answer as best because of your valuable input from which I have learned a lot, but since there is no problem with the tBuffer components - my question, and this whole discussion, is nullified.

Best regards!