tFileOutputDelimited - Header

One Star

tFileOutputDelimited - Header

I am suppose to be going to production asap, just found out there is a limit on how many records can be loaded in one file in the destination system. So great Talend provides an option to split the file out.
.
My issue I am struggling with is to get the header in each file.
.
My header takes up rows 1 to 7. I build it first by reading the headings from a text file and writing out using tFileOutputDelimited.
.
Then I read the data and map it and append to the same file.
.
Works great, except when I need to split it.
.
My question is when you indicate 'Include Header' what is defined as a Header Row? Can I somehow indicate that the first seven rows is the header? Is there a different component I should be using.
.
I am short on time so any help would be great.
.
Derek
One Star

Re: tFileOutputDelimited - Header

Yes. In talend, you can specify the number rows which need to be skipped to treat as header.
One Star

Re: tFileOutputDelimited - Header

I understand that on the input. But this is on the output to a text file. I am creating 7 header rows initially then reading from the database to get the data. Everything is fine except the size of the files. I am pushing over 100,000's. I found the destination can only handle 5000 in a file. So I switch on the setting on tFileOutPutDelimited to limit the size of the file but I don't have away to get my 7 lines of custom headings to be written to each file. I looking for ideas on the best approach to be able to write the headers also break the file into chunks.
One Star

Re: tFileOutputDelimited - Header

ok. So this is the approach I have put in place. write out the data first, so create multiple files in a work folder all with 5000 rows. Then when the data is processed, use tFILELIST to read the file names and create the a file in the Final folder with the headings only. Then have a second tFILELIST that iterates over the files and using tFILEINPUTDELIMITED reads the data and appends it to the files with the headings. Works, just don't like having to process the data twice but couldn't figure a better way. Still would love to hear any other ideas.
One Star

Re: tFileOutputDelimited - Header

Talend doesn't have a 5000 row limit.
One Star

Re: tFileOutputDelimited - Header

I know Talend does not have a limit, the destination system that will load the data has, thats is why I need to break the files into chunks of data of 5000 rows.