limit on number of columns in delimited input output file

One Star

limit on number of columns in delimited input output file

Hi All,
I am trying to determine if there is any length constraint(for each row) or number of columns constraint while reading/ writing to a delimited input/output file using Talend.
I had tried to create a sample input file with 10k columns (comma delimited), while  reading the file and writing to the output delimited file, I am getting the following errors.
"The code of method writeData(ObjectOutputStream) is exceeding the 65535 bytes limit.
The code of method readData(ObjectInputStream) is exceeding the 65535 bytes limit.
The code of method toLogString() is exceeding the 65535 bytes limit.
The code of method toString() is exceeding the 65535 bytes limit."
My goal is to determine if Talend puts any constraining on record length for a row or number of columns while writing to a file into hdfs.
On high level, I will be unloading data from multiple tables and normalize these data and join them using key and spit out one big reocord with normalized data .
Moderator

Re: limit on number of columns in delimited input output file

Hi,
Have you already checked the KB article about:TalendHelpCenter:Exception Exceeding the 65535 bytes limit?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: limit on number of columns in delimited input output file

Hi,
Have you already checked the KB article about:TalendHelpCenter:Exception+Exceeding+the+65535+bytes+limit?
Best regards
Sabrina

hi Sabrina,
Now I know why this exception is being thrown.
Is there a simpler work around for this kind of scenario where the input/output delimited file would have a lot of columns(unlimited number).
The weird thing is that while running the job, I am getting this warning but the job runs successfully and writes the data into output file.
For one of the project, we are trying to assess if we can write a big copybook data into hdfs after converting them into ascii using Talend.
This copybook data has a lot of columns and repeating elements which we would write as a row of data and there is no limit on the number of columns in the copybook it can be in the order of 1k to 10k.
Thanks,
Ghufran
Thanks,
Ghufran