limit on number of columns in delimited input output file
Hi All,
I am trying to determine if there is any length constraint(for each row) or number of columns constraint while reading/ writing to a delimited input/output file using Talend.
I had tried to create a sample input file with 10k columns (comma delimited), while reading the file and writing to the output delimited file, I am getting the following errors.
"The code of method writeData(ObjectOutputStream) is exceeding the 65535 bytes limit.
The code of method readData(ObjectInputStream) is exceeding the 65535 bytes limit.
The code of method toLogString() is exceeding the 65535 bytes limit.
The code of method toString() is exceeding the 65535 bytes limit."
My goal is to determine if Talend puts any constraining on record length for a row or number of columns while writing to a file into hdfs.
On high level, I will be unloading data from multiple tables and normalize these data and join them using key and spit out one big reocord with normalized data .
Hi, Have you already checked the KB article about:TalendHelpCenter:Exception+Exceeding+the+65535+bytes+limit? Best regards Sabrina
hi Sabrina,
Now I know why this exception is being thrown.
Is there a simpler work around for this kind of scenario where the input/output delimited file would have a lot of columns(unlimited number).
The weird thing is that while running the job, I am getting this warning but the job runs successfully and writes the data into output file.
For one of the project, we are trying to assess if we can write a big copybook data into hdfs after converting them into ascii using Talend.
This copybook data has a lot of columns and repeating elements which we would write as a row of data and there is no limit on the number of columns in the copybook it can be in the order of 1k to 10k.
Thanks,
Ghufran
Thanks,
Ghufran
