File not closing using tFileOutputDelimited component

Six Stars SD
Six Stars

File not closing using tFileOutputDelimited component

Hi:

  I am fetching records from DB and creating multiple files based on number of records each file can hold, thru a talend job using tloop, till the total number of records in the DB table is over. I see that the last file is not created in the folder and can see all the previous files.

And then if I have to do a subsequent processing of the files, I get error.

How do I close the last file.

Thanks.


Accepted Solutions
Sixteen Stars

Re: File not closing using tFileOutputDelimited component

I'm guessing here, so you will have to try this out. But I think this is actually a slightly different problem. I *think* your first file is either not being written or being written somewhere else (due to an empty filename) and your last file is being written with your second to last filename.

 

Let me explain. The tFileOutput components need their filenames at the beginning of the subjob they sit in. I'm guessing you are setting the filename (in a context variable?) in the tJava_3 component. This component, while running before the tFileOutput, is not setting the name in time. So the first file is written (or attempted to be written) to an empty filename and the second file is written to the first filename....and so on.

 

To solve this, find a way to set the filename via an iterate link or straight after the the tLoop BEFORE the db query. Ensure that the DB query is the beginning of the subjob and ending is the FileOutput. 

 

Another point, don't keep opening a connection on the loop. Do that before the loop. It doesn't need to be recreated each time. So actually, maybe it is best to have....

 

tLoop----> tJava (create filename) --OnComponentOK--->DB Query---rest of job---->

 

I hope that makes sense.


All Replies
Sixteen Stars

Re: File not closing using tFileOutputDelimited component

Can you give us a screenshot of your job and maybe show us your file component configuration please?

 

Six Stars SD
Six Stars

Re: File not closing using tFileOutputDelimited component

Attaching the job. As per the current setup, it is supposed to create 6 files. I can see all files name being correctly printed in tJava component. But the last one is not created in the directory (only five are created).

 

Sixteen Stars

Re: File not closing using tFileOutputDelimited component

Where are the filenames printed (Java_2)? What is the tLoop config? Do you get any errors?

Six Stars SD
Six Stars

Re: File not closing using tFileOutputDelimited component

tLoop_1 is looping as long as the total number of records fetched from database is more then the counter that I am maintaining in tJava_1 (thru while condition)

This is what I have in tJava_1:

context.C1 = context.C2+1;
context.C2 = context.C2+5;
context.fileCounter =context.fileCounter+1;

System.out.println("context.C1 ***" + context.C1 + "***");
System.out.println("context.C2 ***" + context.C2 + "***");

Long size=(Long)globalMap.get("size");
System.out.println("Size ***" + size + "***");

=================================================

tjava_1 is printing the filenames:

Starting job CSVFileCreation at 11:11 22/02/2018.

[statistics] connecting to socket on port 3875
[statistics] connected
context.Total_Rec::33
******oFilename::C:/CSVFiles/AutoBulkLeads1.csv
context.C1 ***6***
context.C2 ***10***
Size ***null***
******oFilename::C:/CSVFiles/AutoBulkLeads2.csv
context.C1 ***11***
context.C2 ***15***
Size ***null***
******oFilename::C:/CSVFiles/AutoBulkLeads3.csv
context.C1 ***16***
context.C2 ***20***
Size ***null***
******oFilename::C:/CSVFiles/AutoBulkLeads4.csv
context.C1 ***21***
context.C2 ***25***
Size ***null***
******oFilename::C:/CSVFiles/AutoBulkLeads5.csv
context.C1 ***26***
context.C2 ***30***
Size ***null***
******oFilename::C:/CSVFiles/AutoBulkLeads6.csv
context.C1 ***31***
context.C2 ***35***
Size ***null***
[statistics] disconnected
Job CSVFileCreation ended at 11:11 22/02/2018. [exit code=0]

Sixteen Stars

Re: File not closing using tFileOutputDelimited component

I'm guessing here, so you will have to try this out. But I think this is actually a slightly different problem. I *think* your first file is either not being written or being written somewhere else (due to an empty filename) and your last file is being written with your second to last filename.

 

Let me explain. The tFileOutput components need their filenames at the beginning of the subjob they sit in. I'm guessing you are setting the filename (in a context variable?) in the tJava_3 component. This component, while running before the tFileOutput, is not setting the name in time. So the first file is written (or attempted to be written) to an empty filename and the second file is written to the first filename....and so on.

 

To solve this, find a way to set the filename via an iterate link or straight after the the tLoop BEFORE the db query. Ensure that the DB query is the beginning of the subjob and ending is the FileOutput. 

 

Another point, don't keep opening a connection on the loop. Do that before the loop. It doesn't need to be recreated each time. So actually, maybe it is best to have....

 

tLoop----> tJava (create filename) --OnComponentOK--->DB Query---rest of job---->

 

I hope that makes sense.

Six Stars SD
Six Stars

Re: File not closing using tFileOutputDelimited component

Thanks so much for explaining that.

yes, I was just playing moving the components and I happened to get the creation of filename before the component itself. It worked and then when I see your explanation that 

"The tFileOutput components need their filenames at the beginning of the subjob they sit in."- that makes perfect sense.

It is little strange though that having a filename in any variable (happens to be a context variable here) has any kind of connection as to when they are opened.

Thanks!