Wait the end of several jobs to archive files

One Star

Wait the end of several jobs to archive files

Hi everybody,
I have three jobs, each one containing several jobs. These jobs are creating CSV output files.
I want to run these jobs in parallel, and to archive CSV files when all the jobs are ended, but I don't know how to do that.
Anyone can help me ?
Tags (1)
Employee

Re: Wait the end of several jobs to archive files

(reading your previous posts, I suppose you run with Perl)
"multithread" was added in Perl project in TOS 2.4, see details on my blog and the tPostJob was added (Perl/Java) in TOS 2.4 also. So I've used a 2.4.0 (to be released very soon on the website) to show you how this can be done.
To make "multithread" execution work, you have to enable the option in the "Job settings" view and you have to remove links between subjobs (warning : "subjob" doesn't mean "child job", "subjob" is what you see surrounded by blue frames, "child job" is launched with a tRunJob).
One Star

Re: Wait the end of several jobs to archive files

I built my job as you told me (cf. screenshot). (Note : I didn't find any documentation about the component tPostJob).
The jobs seems to be running (CPU use is 100%), but I have some errors with archiving (I use TOS 2.4.0.RC1 with Perl):
Starting job verifDataChimiotheque at 10:14 09/06/2008.
connecting to socket on port 3486 ...
connected
IO error: reading data :
at C:/Perl/lib/Archive/Zip/NewFileMember.pm line 60
Archive::Zip::NewFileMember::_readRawChunk('Archive::Zip::NewFileMember=HASH(0x4ac7fcc)', 'SCALAR(0x4a7a1b0)', 32768) called at C:/Perl/lib/Archive/Zip/Member.pm line 663
Archive::Zip::Member::readChunk('Archive::Zip::NewFileMember=HASH(0x4ac7fcc)', 32768) called at C:/Perl/lib/Archive/Zip/Member.pm line 931
Archive::Zip::Member::_writeData('Archive::Zip::NewFileMember=HASH(0x4ac7fcc)', 'IO::File=GLOB(0x4abd1f8)') called at C:/Perl/lib/Archive/Zip/Member.pm line 905
Archive::Zip::Member::_writeToFileHandle('Archive::Zip::NewFileMember=HASH(0x4ac7fcc)', 'IO::File=GLOB(0x4abd1f8)', 1, 1794) called at C:/Perl/lib/Archive/Zip/Archive.pm line 280
Archive::Zip::Archive::writeToFileHandle('Archive::Zip::Archive=HASH(0x4abd024)', 'IO::File=GLOB(0x4abd1f8)', 1) called at C:/Perl/lib/Archive/Zip/Archive.pm line 257
Archive::Zip::Archive::writeToFileNamed('Archive::Zip::Archive=HASH(0x4abd024)', './workspace/incoherences_data_chimiotheques.zip') called at C:\Talend\TOS-All-r13964-V2.4.0RC1\workspace\.Perl\TEST_INCOHERENCES.job_verifDataChimiotheque_0.1.pl line 332
main::tFileArchive_1_subprocess() called at C:\Talend\TOS-All-r13964-V2.4.0RC1\workspace\.Perl\TEST_INCOHERENCES.job_verifDataChimiotheque_0.1.pl line 232
main::tPostjob_1_subprocess() called at C:\Talend\TOS-All-r13964-V2.4.0RC1\workspace\.Perl\TEST_INCOHERENCES.job_verifDataChimiotheque_0.1.pl line 694
===
execution time: 8992 milliseconds
===
Job verifDataChimiotheque ended at 10:14 09/06/2008.

EDIT : This error is displayed only when "Statistic" option is checked.
EDIT2 : In fact, despite the error the result seems to be correct...
Another problem is when I kill my "super-job", it doesn't kill the child jobs : CPU is still used 100 % during at least one minute.
And a last one (solved) : TOS doesn't manage spaces in directories. I had an error because my Talend directory was on the desktop : C:\Documents and Settings\etc...
The problem is solved by moving the directory on C:\
Employee

Re: Wait the end of several jobs to archive files

I built my job as you told me (cf. screenshot). (Note : I didn't find any documentation about the component tPostJob).

That's right, it will be in the next version of the documentation, published before the end of the week.
I use TOS 2.4.0.RC1 with Perl

You should consider upgrading to 2.4.0 :-)
The jobs seems to be running (CPU use is 100%), but I have some errors with archiving
EDIT : This error is displayed only when "Statistic" option is checked.
EDIT2 : In fact, despite the error the result seems to be correct...

When you mean "seems to be correct", can you check it realy is correct? The error doesn't seems to be related to the "multithread" option. I have simulated a "warn" and a "die" in the tPerl of the father job described in my previous post and I have the information in red in the console, without statistics activated. I'm running GNU/Linux, your problem may be related to Microsoft Windows.
Another problem is when I kill my "super-job", it doesn't kill the child jobs : CPU is still used 100 % during at least one minute.

I've also checked and I don't reproduce your problem. So it may be related to Microsoft Windows. Can you create a bug notification in the bugtracker please?
One Star

Re: Wait the end of several jobs to archive files

I compared the data produced by the super-job and the data produced by the child-jobs, and it's the same. Furthermore the archiving is working, so I really don't understand the error :-/
I will add a post in the bugtracker for the kill button that kills only the super-job and not the child-jobs.
About the new version of TOS, are the 2.4.0RC1 projects compatibles with the version 2.4 ? I guess they are, it's just to confirm ;-)
Employee

Re: Wait the end of several jobs to archive files

About the new version of TOS, are the 2.4.0RC1 projects compatibles with the version 2.4 ? I guess they are, it's just to confirm ;-)

They should be compatible. We don't have "RC to final" as development constraint. It means that sometimes a project created with TOS 2.4.0RC2 may not be imported in a 2.4.0.
I can confirm that a project created in 2.3.x will be importable in 2.4.y.
One Star

Re: Wait the end of several jobs to archive files

OK, I will test, and tell you if it's not working.
EDIT : It works with v2.4, the archiving error doesn't appear anymore, but I still have the problem with the kill button that doesn't kill child-jobs