topic Re: Split Files in Talend Studio

Split Files

mani1304 — Sat, 16 Nov 2024 00:54:30 GMT

Hi All,

I have a file having several records, but few records are of same id, now need to separate records which have same id. After every same set of records we have ENDORDER as keyword, this can be taken into consideration to split the file. Can someone suggest on it.

Re: Split Files

Anonymous — Sat, 26 Dec 2020 23:02:38 GMT

Can you take a look at tUniqueRow component? This component is used to get the unique rows and duplicated rows. If it does not meet your need, please describe your requirement with some example data.

Regards

Shong

Re: Split Files

mani1304 — Wed, 30 Dec 2020 09:30:30 GMT

Thanks Shong, but here is no duplicate record. We have a file which has several orders and now we need to split file into each order and each order ends with ENDORDER and then need to convert it to json file.

e.g.

A|1|2|3|4

B|1|2|3|4|5|6

C|1|2|3

D|1|2|3|4|5

ENDORDER

A|1|2|3|4

B|1|2|3|4|5|6

C|1|2|3

D|1|2|3|4|5

ENDORDER

A|1|2|3|4

B|1|2|3|4|5|6

C|1|2|3

D|1|2|3|4|5

now it has 3 orders as below

A|1|2|3|4

B|1|2|3|4|5|6

C|1|2|3

D|1|2|3|4|5,

A|1|2|3|4

B|1|2|3|4|5|6

C|1|2|3

D|1|2|3|4|5,

A|1|2|3|4

B|1|2|3|4|5|6

C|1|2|3

D|1|2|3|4|5

Also please suggest how to convert such files in json/xml. A,B,C,D are having different schema and can come multiple/no times in an order.

Thanks in advance.

Re: Split Files

mani1304 — Sun, 03 Jan 2021 07:41:17 GMT

Can anyone please suggest on it.

Re: Split Files

Anonymous — Wed, 06 Jan 2021 03:20:25 GMT

Here is a scenario showing how to split the file to multiple text files.

This job will split the file to multiple text files, like filename1.txt, filename2.txt etc.

If you want to generate XML or Json file, you need to use a tFileList to iterate these text files in next step. Can you show an example of XML/Json file you want to generate?

Regards

Shong

Re: Split Files

mani1304 — Wed, 06 Jan 2021 08:18:59 GMT

Thank you very much Shong, The files which we separated from the above need to convert into a XML/JSON message.

e.g.

A|1|2|3|4

B|1|2|3|4|5|6

C|1|2|3

D|1|2|3|4|5

here row starting from A,B,C,D are having different schema and B,C,D can come in multiple/no times. Hope this clarifies the question.

Re: Split Files

mani1304 — Mon, 11 Jan 2021 10:08:34 GMT

@Shicong Hong Could you please suggest on it.

Re: Split Files

Anonymous — Mon, 11 Jan 2021 11:07:36 GMT

What does the XML/Json file look like? There is no header columns in the file, what will be the element name in the XML/Json file?

Re: Split Files

mani1304 — Mon, 11 Jan 2021 11:53:58 GMT

Currently I don't have any output file xsd, ony doing POC where we need to convert a flat file like above into an XML file. there will be 4 types of records A, B, C, D where B,C,D can come multiple/no times as below. A,B,C,D are having different schema not dynamic schema means A will have fix number of fields and so on on.

A|1|2|3|4

B|1|2|3|4|5|6

C|1|2|3

D|1|2|3|4|5

Hope you get the requirement.@Shicong Hong

Re: Split Files

Anonymous — Mon, 11 Jan 2021 12:21:07 GMT

We usually use tAdvancedFileOutputXML to generate a XML file. There is no header columns in your file, I just show a simple example, set the schema to have 7 columns, c1,c2...c7.

It will generate a XML file as below:

<?xml version="1.0" encoding="ISO-8859-15"?>

<c6/>

<c7/>

</person>

</c1>

</person>

</person>

</c1>

<c5/>

<c6/>

<c7/>

</person>

<c5/>

<c6/>

<c7/>

</person>

<c5/>

<c6/>

<c7/>

</person>

</c1>

<c7/>

</person>

</c1>

</rootTag>

Re: Split Files

mani1304 — Mon, 11 Jan 2021 12:45:43 GMT

Thanks @Shicong Hong there is no fixed columns it may be 1 or many, means only A is constant but B,C,D can come many times.

Re: Split Files

Anonymous — Tue, 12 Jan 2021 03:47:49 GMT

To read the flat file, we must define the schema in design phase. In your case, there is no header line in the file,

I guess the max number of columns is 7 and I define 7 columns on the schema, the value will be null for the rest columns if the row don't have 7 columns. You don't show us an example of XML file you expected or you don’t know what the XML file looks like;(, I’m afraid I can’t continue to help you.

Regards

Shong

Re: Split Files

mani1304 — Tue, 12 Jan 2021 12:58:20 GMT

@Shicong Hong As a work around I split the file into 4 types and then merged the files into tFileXMLOutput Component and this way I am able to do it, but I am looking for a different way if the Talend provides.