Rows missing from output when appending file

[JM39]

Hello.  I have a simple job where I have one input file, one tmap and then two outputs.  I want to append the second output to the first.  None of my outputs have a header row and I don't need a header.  My issue is this

Each output should produce 23016 rows as per the input.  When I run it to produce two separate output files they both produce the 23016 rows.  So I'm expecting it to produce a combined output of 46032.  However, when I append the second output to the first there are two issues

1. After the first 23016 rows of data, it inserts a blank row

2. It is appending only 17 rows out of the 23016, even although both components are showing 23016 rows generated in the job.

[akumar2301]

Which componet you are using to merge?

Tjoin , you will have only unique records from both set

[JM39]

Hi, so I'm just checking the append box on the second output file.  That way works fine for another job I have, so not sure what's going wrong with this one.

[akumar2301]

Can you share your job design

[JM39]

Thanks - I've attached screenshots

talend issue.docx

[TRF]

It's unusual to have 2 or more outputs from the same tMap connected to the same physical output file.

As output buffer size may cause this kind of problem, you should try to change it for both output files.

Go to "Advanced settings", tick the option "Custom the flush buffer size" and set the "Row number" value to 1.

This way you'll force the buffer to be flushed for each record.

For huge volume, it 's better to have separates output files and to merge them on the next subjob.

[JM39]

Hi, thanks I tried that suggestion and it actually produced less combined rows than without it!  Seems like I need another solution.  What would be the easiest way to merge the individual output files?

[akumar2301]

Hello

You would have defined some columns as key.

If you need to append out2 to out1 , while creation of flow out2 in tmap ,
you just join it to Out1 (instead of new output). And then you will have
only one output with all records.

Let me know if that works.

[JM39]

Thanks - is it possible you could show me how to do that in a screen shot?  I'm a total beginner with Talend (and I mean a beginner!) and I haven't had any training at all.  Thank you!

[TRF · Accepted solution]

The simplest way is to assign each output flow from the tMap to a dedicated file.

Then on next subjob, merge both files content using a tUnite and store the result to the desired file.

Here it is: