File with 3K columns

[EmmanuelTrejo]

Hi everyone

I've a file with almost 3K columns, when I create metadata it sends me this message as an error, what can I do to load file??

Regards.

[Anonymous]

Hello,

it's caused by log4j settings, the workaround is to

1. in studio, Open the Project settings->Log4j-> uncheck the button 'Active log4j in components'

see https://community.talend.com/s/article/Compilation-error-The-code-of-method-is-exceeding-the-bytes-limit-doIL7

[Anonymous]

Hello,

How many columns do you have?

On which talend build version you got this issue?

It's suggested that there should not be a lot of columns. There is a 65,535-byte limitation in java method. If you use a lot of columns, there are generated a large of Assignment statements.

Best regards

Sabrina

[EmmanuelTrejo]

Hi Sabrina,

File has 2835 columns, TOS DI 7.3,

Understand, however the file has this structure I cannot delete any column

Regards

[Anonymous]

Hello,

The limitation comes from the Java class file specification and isn't specifically related to any Talend component -- although they are related, since the generated code must respect the limitation. It usually occurs with jobs with large schemas, and the workaround is to manually force the job into several distinct steps.

Maybe we could reduce the number of columns that need to be passed around, What does your whole job design look like? Which operations are you trying to do in your further processing?

Best regards

Sabrina

[EmmanuelTrejo]

Hi Sabrina

Ok, so those parameters cannot change?

Design is input-->tmap-->rowaggregate-->tmap-->output

Or any change to separate the file by columns from beginning?

[Anonymous]

Hello,

All 2835 columns will be processed in your workflow?

Here is a component tExtractDelimitedFields in talend which can split and extract a column data separated via Field Separator into multiple columns.

https://help.talend.com/r/en-US/8.0/processing/textractdelimitedfields

Best regards

Sabrina