topic Re: Rows missing from output when appending file in Talend Studio

Rows missing from output when appending file

JM39 — Sat, 16 Nov 2024 02:13:04 GMT

Hello. I have a simple job where I have one input file, one tmap and then two outputs. I want to append the second output to the first. None of my outputs have a header row and I don't need a header. My issue is this

Each output should produce 23016 rows as per the input. When I run it to produce two separate output files they both produce the 23016 rows. So I'm expecting it to produce a combined output of 46032. However, when I append the second output to the first there are two issues

1. After the first 23016 rows of data, it inserts a blank row

2. It is appending only 17 rows out of the 23016, even although both components are showing 23016 rows generated in the job.

Re: Rows missing from output when appending file

akumar2301 — Fri, 05 Jun 2020 21:19:27 GMT

Which componet you are using to merge?

Tjoin , you will have only unique records from both set

Re: Rows missing from output when appending file

JM39 — Sat, 06 Jun 2020 10:26:09 GMT

Hi, so I'm just checking the append box on the second output file. That way works fine for another job I have, so not sure what's going wrong with this one.

Re: Rows missing from output when appending file

akumar2301 — Sat, 06 Jun 2020 11:13:43 GMT

Can you share your job design

Re: Rows missing from output when appending file

JM39 — Sat, 06 Jun 2020 11:42:43 GMT

Thanks - I've attached screenshots

talend issue.docx

Re: Rows missing from output when appending file

TRF — Sat, 06 Jun 2020 11:44:03 GMT

It's unusual to have 2 or more outputs from the same tMap connected to the same physical output file.

As output buffer size may cause this kind of problem, you should try to change it for both output files.

Go to "Advanced settings", tick the option "Custom the flush buffer size" and set the "Row number" value to 1.

This way you'll force the buffer to be flushed for each record.

For huge volume, it 's better to have separates output files and to merge them on the next subjob.

Re: Rows missing from output when appending file

JM39 — Sat, 06 Jun 2020 15:13:49 GMT

Hi, thanks I tried that suggestion and it actually produced less combined rows than without it! Seems like I need another solution. What would be the easiest way to merge the individual output files?

Re: Rows missing from output when appending file

akumar2301 — Sat, 06 Jun 2020 17:00:37 GMT

Hello

You would have defined some columns as key.

If you need to append out2 to out1 , while creation of flow out2 in tmap ,
you just join it to Out1 (instead of new output). And then you will have
only one output with all records.

Let me know if that works.

Re: Rows missing from output when appending file

JM39 — Sun, 07 Jun 2020 10:45:40 GMT

Thanks - is it possible you could show me how to do that in a screen shot? I'm a total beginner with Talend (and I mean a beginner!) and I haven't had any training at all. Thank you!

Re: Rows missing from output when appending file

TRF — Sun, 07 Jun 2020 12:07:27 GMT

The simplest way is to assign each output flow from the tMap to a dedicated file.

Then on next subjob, merge both files content using a tUnite and store the result to the desired file.

Here it is:

Re: Rows missing from output when appending file

JM39 — Mon, 08 Jun 2020 10:37:12 GMT

Thank you so much - that's worked perfectly! I'm honestly eternally grateful for your help. Thank you 🙂

Re: Rows missing from output when appending file

akumar2301 — Mon, 08 Jun 2020 11:08:48 GMT

Other way to duplicate the data ,

talend-example.PNG