topic Re: tWriteJSONField seems to cause a roadblock in the data flow in Talend Studio

tWriteJSONField seems to cause a roadblock in the data flow

theronrohr — Wed, 17 Oct 2018 21:44:42 GMT

Hi there... new Talend developer here... I have a data flow (picture hopefully inserted below) where I am querying a table for some rows (3 in this case) and I want to process each one in turn. This was working as I would expect in an earlier version where the steps were: (1) parse the records into fields, (2) map several fields into a combined comma delimited message field, (3) send the message to a queue, (4) update the relevant database row with a processed date.

However when I added a tWriteJSONField component the behavior has changed. Now all 3 rows are processed as before up until the tWriteJSONField component. At that component the flow seems to wait until all rows are processed before proceeding. Then after the tWriteJSONField component the remaining components are run 3 times but all 3 using the final rows values. This of course is no good. Is there somethign I am doing wrong in the setup of the tWriteJSONField component? Or can I not use this inside a flow? I'm confused!

Note that the diagram below is on step1 in debug mode. It shows that 3 rows have been processed but not beyond the tWriteJSONField component.

Re: tWriteJSONField seems to cause a roadblock in the data flow

theronrohr — Wed, 17 Oct 2018 21:55:38 GMT

Me again... to clarify: what I want is for each of the 3 records to go all the way through the flow until the end before the next record is processed.

Re: tWriteJSONField seems to cause a roadblock in the data flow

nfz11 — Thu, 04 Apr 2019 04:07:38 GMT

I have exactly the same problem and have spent a lot of time trying all sorts of workarounds to get the flow to pass each row through tWriteJSONField without queuing them up first.

As in the OP, the rows halt at tWriteJSONField, then the flow proceeds row by row after it. This behavior messes up my setting global variables with the row values before the tWriteJSONField; the global variables always reflect the last row of data.

Why was the tWriteJSONField component designed this way? Also, as a side effect of this, or as a different anti-pattern, the variables in the schema get nulled out, so you can't expect to persist schema variables through the tWriteJSONField.

Unless someone can explain to me the reason for this design and a workaround, I would consider this a bug. Somebody developed a custom component called tRowToJSON probably because of the stock component's maladjusted behavior.

Thanks in advance for any help.

Re: tWriteJSONField seems to cause a roadblock in the data flow

theronrohr — Fri, 05 Apr 2019 18:23:52 GMT

Oh, someone replied to my post!

In the end I just worked around this by writing out the json formatting in a java component. But it's good to know there is a custom component for when this comes up again.

I also find it annoying that a lot of components do not flow the schema variables across them. There must be some design philosophy behind this that will become apparent to me one day.

Re: tWriteJSONField seems to cause a roadblock in the data flow

theronrohr — Fri, 05 Apr 2019 18:25:26 GMT

Maybe if tWriteJSONField was renamed to tWriteJSONReport there would be less confusion?

Re: tWriteJSONField seems to cause a roadblock in the data flow

nfz11 — Thu, 11 Jul 2019 04:36:34 GMT

Hi @theronrohr , I figured out a workaround for this, see my response in my thread here:

https://community.talend.com/t5/Design-and-Development/tWriteJSONField-and-tWriteXMLField-queue-unexpectedly-and/