tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output.

nfz11 — Sat, 04 May 2019 03:38:06 GMT

The flow pauses at tWriteJSONField and tWriteXMLField until all rows have hit these components. Then the flow continues on each row after the component. I have not seen this behavior with any other component. I would expect, like other components, that the flow would proceed through these components one row at a time with each row continuing on.

Also, I get different ordering of the output in an arbitrary way. For example, see the two outputs below. The only thing I changed is the formatting of the tLowRow_4. I have also seen inconsistent behavior even without changing anything, possibly between different compilations. For the tWriteXMLField sub-job, one row is processed all the way through but the other rows queue up.

Here is a screenshot of my simple test job with two sub-jobs, the first for JSON the second for XML:

Here are the two outputs that shows both the queuing and the inconsistent output:

1. All three rows queue before continuing for both JSON and XML:

|#1. tLogRow_1 |
+------+-------+
| key  | value |
+------+-------+
| id   | 1     |
| name | Jim   |
+------+-------+

.--------------.
|#2. tLogRow_1 |
+------+-------+
| key  | value |
+------+-------+
| id   | 2     |
| name | Bob   |
+------+-------+

.----------------.
| #3. tLogRow_1  |
+------+---------+
| key  | value   |
+------+---------+
| id   | 3       |
| name | Charles |
+------+---------+

.---------------------------------------------.
|                #1. tLogRow_2                |
+--------+------------------------------------+
| key    | value                              |
+--------+------------------------------------+
| string | {"employee":{"id":1,"name":"Jim"}} |
+--------+------------------------------------+

.---------------------------------------------.
|                #2. tLogRow_2                |
+--------+------------------------------------+
| key    | value                              |
+--------+------------------------------------+
| string | {"employee":{"id":2,"name":"Bob"}} |
+--------+------------------------------------+

.-------------------------------------------------.
|                  #3. tLogRow_2                  |
+--------+----------------------------------------+
| key    | value                                  |
+--------+----------------------------------------+
| string | {"employee":{"id":3,"name":"Charles"}} |
+--------+----------------------------------------+

.--------------.
|#1. tLogRow_3 |
+------+-------+
| key  | value |
+------+-------+
| id   | 1     |
| name | Jim   |
+------+-------+

.--------------.
|#2. tLogRow_3 |
+------+-------+
| key  | value |
+------+-------+
| id   | 2     |
| name | Bob   |
+------+-------+

.----------------.
| #3. tLogRow_3  |
+------+---------+
| key  | value   |
+------+---------+
| id   | 3       |
| name | Charles |
+------+---------+

.----------------------------------------------------------------------------------------------------------------.
|                                                 #1. tLogRow_4                                                  |
+--------+-------------------------------------------------------------------------------------------------------+
| key    | value                                                                                                 |
+--------+-------------------------------------------------------------------------------------------------------+
| string | <?xml version="1.0" encoding="ISO-8859-15"?>

<employee>
  <id>1</id>
  <name>Jim</name>
</employee>
 |
+--------+-------------------------------------------------------------------------------------------------------+

.----------------------------------------------------------------------------------------------------------------.
|                                                 #2. tLogRow_4                                                  |
+--------+-------------------------------------------------------------------------------------------------------+
| key    | value                                                                                                 |
+--------+-------------------------------------------------------------------------------------------------------+
| string | <?xml version="1.0" encoding="ISO-8859-15"?>

<employee>
  <id>2</id>
  <name>Bob</name>
</employee>
 |
+--------+-------------------------------------------------------------------------------------------------------+

.--------------------------------------------------------------------------------------------------------------------.
|                                                   #3. tLogRow_4                                                    |
+--------+-----------------------------------------------------------------------------------------------------------+
| key    | value                                                                                                     |
+--------+-----------------------------------------------------------------------------------------------------------+
| string | <?xml version="1.0" encoding="ISO-8859-15"?>

<employee>
  <id>3</id>
  <name>Charles</name>
</employee>
 |
+--------+-----------------------------------------------------------------------------------------------------------+

2. Two tWriteXMLField rows queue, then the first row proceeds, then the third row queues, then the second and third row proceed.

|#1. tLogRow_1 |
+------+-------+
| key  | value |
+------+-------+
| id   | 1     |
| name | Jim   |
+------+-------+

.--------------.
|#2. tLogRow_1 |
+------+-------+
| key  | value |
+------+-------+
| id   | 2     |
| name | Bob   |
+------+-------+

.----------------.
| #3. tLogRow_1  |
+------+---------+
| key  | value   |
+------+---------+
| id   | 3       |
| name | Charles |
+------+---------+

.---------------------------------------------.
|                #1. tLogRow_2                |
+--------+------------------------------------+
| key    | value                              |
+--------+------------------------------------+
| string | {"employee":{"id":1,"name":"Jim"}} |
+--------+------------------------------------+

.---------------------------------------------.
|                #2. tLogRow_2                |
+--------+------------------------------------+
| key    | value                              |
+--------+------------------------------------+
| string | {"employee":{"id":2,"name":"Bob"}} |
+--------+------------------------------------+

.-------------------------------------------------.
|                  #3. tLogRow_2                  |
+--------+----------------------------------------+
| key    | value                                  |
+--------+----------------------------------------+
| string | {"employee":{"id":3,"name":"Charles"}} |
+--------+----------------------------------------+

.--------------.
|#1. tLogRow_3 |
+------+-------+
| key  | value |
+------+-------+
| id   | 1     |
| name | Jim   |
+------+-------+

.--------------.
|#2. tLogRow_3 |
+------+-------+
| key  | value |
+------+-------+
| id   | 2     |
| name | Bob   |
+------+-------+

<?xml version="1.0" encoding="ISO-8859-15"?>

<employee>
  <id>1</id>
  <name>Jim</name>
</employee>

.----------------.
| #3. tLogRow_3  |
+------+---------+
| key  | value   |
+------+---------+
| id   | 3       |
| name | Charles |
+------+---------+

<?xml version="1.0" encoding="ISO-8859-15"?>

<employee>
  <id>2</id>
  <name>Bob</name>
</employee>

<?xml version="1.0" encoding="ISO-8859-15"?>

<employee>
  <id>3</id>
  <name>Charles</name>
</employee>

This happens in both Talend 7.0.1 and 7.1.1.

I am attaching the zip export of the job if you want to look at all the details.

This is a related thread about the queuing:

https://community.talend.com/t5/Design-and-Development/tWriteJSONField-seems-to-cause-a-roadblock-in-the-data-flow/td-p/137690

If anyone can shed some light on this issue I thank you in advance.

Re: tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output.

vapukov — Sun, 05 May 2019 01:43:35 GMT

Hi,

you can test your jobs with tXMLMap instead of tWriteXMLField

Re: tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output.

nfz11 — Sun, 05 May 2019 03:02:32 GMT

I am working with JSON, I just included XML to show that the bug was there too.

Is there a way to use tXmlMap with JSON?

Thanks.

@vapukov wrote:

Hi,

you can test your jobs with tXMLMap instead of tWriteXMLField

Re: tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output.

nfz11 — Thu, 11 Jul 2019 04:16:22 GMT

I figured out a workaround for this. If you put in an extra tFlowToIterate and then tFixedFlowInput before the tWriteJSONField then the rows will pass one by one through the tWriteJSONField and will not queue up.

Here is the output of my new test job, which is what I want:

Starting job WriteJsonFieldQueueBug at 23:09 10/07/2019.

[statistics] connecting to socket on port 4066
[statistics] connected
.--------------.
|#1. tLogRow_5 |
+------+-------+
| key  | value |
+------+-------+
| id   | 1     |
| name | Jim   |
+------+-------+

.---------------------------------------------.
|                #1. tLogRow_8                |
+--------+------------------------------------+
| key    | value                              |
+--------+------------------------------------+
| string | {"employee":{"id":1,"name":"Jim"}} |
+--------+------------------------------------+

.--------------.
|#1. tLogRow_5 |
+------+-------+
| key  | value |
+------+-------+
| id   | 2     |
| name | Bob   |
+------+-------+

.---------------------------------------------.
|                #1. tLogRow_8                |
+--------+------------------------------------+
| key    | value                              |
+--------+------------------------------------+
| string | {"employee":{"id":2,"name":"Bob"}} |
+--------+------------------------------------+

.----------------.
| #1. tLogRow_5  |
+------+---------+
| key  | value   |
+------+---------+
| id   | 3       |
| name | Charles |
+------+---------+

.-------------------------------------------------.
|                  #1. tLogRow_8                  |
+--------+----------------------------------------+
| key    | value                                  |
+--------+----------------------------------------+
| string | {"employee":{"id":3,"name":"Charles"}} |
+--------+----------------------------------------+

[statistics] disconnected

Job WriteJsonFieldQueueBug ended at 23:09 10/07/2019. [exit code=0]

Here is the screenshot of the new test job. The extra components I needed were the tFlowToIterate_4 and tFixedFlowInput_5

I still think that this queuing behavior is a bug and should not be the intended behavior of the tWriteJSONField and tWriteXMLField components. At least this workaround exists though.

Re: tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output.

theronrohr — Tue, 16 Jul 2019 00:56:26 GMT

Good work- I'll try this out next time!

topic tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output. in Talend Studio

tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output.

Re: tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output.

Re: tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output.

Re: tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output.

Re: tWriteJSONField and tWriteXMLField queue unexpectedly and produce inconsistent output.