topic Re: Split a positional file into multiple positional files depending on number of rows in Talend Studio

Split a positional file into multiple positional files depending on number of rows

Anonymous — Sat, 16 Nov 2024 04:19:41 GMT

Hello,

I'm needing to split a Positional file into multiple positional files with a maximum of rows.

Like a number of rows equal to 400000.

The tfileoutputdelimiter has the option to split files, but the positional file doesn't have.

Anyone can help me?

Re: Split a positional file into multiple positional files depending on number of rows

TRF — Sun, 20 Oct 2019 22:41:02 GMT

Use this design:
tFileInputPositional-->tJavaRow-->tFlowToIterate-->tFixedFlowInput-->tFileOuputPositional
In the tJavaRow you need a sequence from 1 to 400000 and a global variable corresponding to the current file number. Each time the sequence reach the value of 400000, reset it and set the file number to the next value.
tFlowToIterate is here to break the flow and tFixedFlowInput is here to create a new one to populate the output file (this sequence is requires to change the filename on the fly).
Finaly, use the global variable for the filename construction to make it dynamic.

Re: Split a positional file into multiple positional files depending on number of rows

Anonymous — Mon, 21 Oct 2019 09:14:47 GMT

Hello, thanks, I will try it.

I have another question/issue. I have a header and footer that each file has to have also. The problem is that the footer has the number of rows, I'm thinging to count it to hash file and then write it.

What is the best way?

Re: Split a positional file into multiple positional files depending on number of rows

TRF — Mon, 21 Oct 2019 10:15:03 GMT

Using tFileRowCount you can the number of records included into a given file.

Based on this information, you can set the Header to the right value to get the last record.

Is that what you want?

Re: Split a positional file into multiple positional files depending on number of rows

Anonymous — Tue, 22 Oct 2019 13:19:42 GMT

Hello,

I'm implementing it by another way, limiting the rows at the DB.

So what I want is a loop where the number of rows is not major of 400000. From the DB is reading a block of data until the function has 0 records. So every time that we read the data from DB, that data is deleted, until it has no rows.

LOOP

From: 0

To: 400000

Step: ((Integer)globalMap.get("tDBInput_1_NB_LINE"))

however the information in DB sometimes has less than 400000, and the loop still goes. How can I do this? I want a maximum of 400000 but sometimes the DB has less.

I don't know if you are understanding.

Re: Split a positional file into multiple positional files depending on number of rows

TRF — Tue, 22 Oct 2019 13:31:49 GMT

Initially you were talking about an input file. From where comes this DB?
Anyway, if you want a really simple way, redirect the output to a tFileOutputDelimited with the option "Split output in several files" ticked and value for "Rows in each output file" set to 400000. Then, use a tFileList to iterate over each output file created.

Re: Split a positional file into multiple positional files depending on number of rows

Anonymous — Wed, 23 Oct 2019 12:08:49 GMT

Hello, thanks,

"Initially you were talking about an input file. From where comes this DB?"

yes, because I'm testing solutions. I have a job and this is a change to a job that is already working.

The goal is that now, I have to divide a positional if the number of rows is more than 400000. But this process is creating a file depending on the name, and have header and footer (personalized footer and header).

I'm thinking now in ssh, because, with iterate is taking to much time to running.

I'm testing solutions, but the only one that I'm seeing that will really work is making a shell script.