topic File with 3K columns in Talend Studio

File with 3K columns

EmmanuelTrejo — Fri, 08 Jul 2022 19:42:14 GMT

Hi everyone

I've a file with almost 3K columns, when I create metadata it sends me this message as an error, what can I do to load file??

Regards.

Re: File with 3K columns

Anonymous — Mon, 11 Jul 2022 03:54:14 GMT

Hello,

it's caused by log4j settings, the workaround is to

in studio, Open the Project settings->Log4j-> uncheck the button 'Active log4j in components'

see https://community.talend.com/s/article/Compilation-error-The-code-of-method-is-exceeding-the-bytes-limit-doIL7

Re: File with 3K columns

Anonymous — Mon, 11 Jul 2022 07:22:46 GMT

Hello,

How many columns do you have?

On which talend build version you got this issue?

It's suggested that there should not be a lot of columns. There is a 65,535-byte limitation in java method. If you use a lot of columns, there are generated a large of Assignment statements.

Best regards

Sabrina

Re: File with 3K columns

EmmanuelTrejo — Mon, 11 Jul 2022 15:18:46 GMT

Hi Sabrina,

File has 2835 columns, TOS DI 7.3,

Understand, however the file has this structure I cannot delete any column

Regards

Re: File with 3K columns

Anonymous — Tue, 12 Jul 2022 04:00:09 GMT

Hello,

The limitation comes from the Java class file specification and isn't specifically related to any Talend component -- although they are related, since the generated code must respect the limitation. It usually occurs with jobs with large schemas, and the workaround is to manually force the job into several distinct steps.

Maybe we could reduce the number of columns that need to be passed around, What does your whole job design look like? Which operations are you trying to do in your further processing?

Best regards

Sabrina

Re: File with 3K columns

EmmanuelTrejo — Thu, 14 Jul 2022 15:08:51 GMT

Hi Sabrina

Ok, so those parameters cannot change?

Design is input-->tmap-->rowaggregate-->tmap-->output

Or any change to separate the file by columns from beginning?

Re: File with 3K columns

Anonymous — Fri, 15 Jul 2022 08:14:30 GMT

Hello,

All 2835 columns will be processed in your workflow?

Here is a component tExtractDelimitedFields in talend which can split and extract a column data separated via Field Separator into multiple columns.

https://help.talend.com/r/en-US/8.0/processing/textractdelimitedfields

Best regards

Sabrina