Windows Task Scheduler - Talend DI job does not run, but returns success to Windows batch file

[YuriB]

Hello everyone

I have a Windows .BAT batch generated by Tqalend DI 6.4.1 file which I run interactively from Command Shell DOS windows and it works.

This is on Windows 10 latest version.

The Talend jobs take several minutes and up to 30 min to run.

When I schedule the same batch file in Windows Task Scheduler the task stars OK but it immediately returns code 0 (i.e. success) and finishes.

The Talend job inside that batch file does not fun at all (i.e. completes immediately, returning success to the .BAT file).

If there is an error returned from the JAR file I can not see it inside scheduler, it gets swallowed and reported as success back to Windows.

Googling around - the only issue I found was due to mapped disk drive letters and network drives using \\server\share\folder.

To the best of my knowledge my package only use C: drive.

It also connects to HTTP, uses tREST, uses AWS cloud.

I put some debugging code in BAT file which creates a local folder after the JAR file is run just to see if the batch file runs until the last line.

It certainly does run, the folder (MKDIR command) gets created immediately after Talend JAR file is run.

So I am certain that the batch file runs but the Talend JAR file exits immediately.

The PATH is correctly set up, because I put compile and run a Hello World Java program immediately before the Talend JAR file and it compiles, creates .class file successfully.

Have anyone encountered this problem? Any ideas what could be the problem?

thanks

[YuriB · Accepted solution]

Found the problem.

I searched the Java generated code, there was a reference to the Redshift JDBC driver LOG file which was located on a mapped network drive U:\workspace....

It was something I did not set, Talend DI picked it up from my workspace default settings, so it was a land mine ready to explode, which it did once I run in Windows task scheduler.

Given this happened "behind my back" I did not check this.

I changed the JDBC log file to point to C:\ instead of mapped drive U:\ and it worked.

This is a problem/bug with Java generated by Talend, IMHO.

If Talend Java code not access a mapped drive such as U:\workspace drive running from task scheduler then it must report ERROR code back to the Windows Command shell.

Instead it just exits and it reports exit code 0, success, which task scheduler reports in job history.

Windows task scheduler is also an evil piece of software, because it does not allow the use the mapped disk drive letters but this is the devil we know, it has always been this way.

I hope this helps the next poor unfortunate who is using Windows Task Scheduler with Talend DI Java.

[vapukov · Accepted solution]

if exclude - path and folders, on my jobs - no additional changes

just not forget change shell file mode

chmod u+x jobname_run.sh

before execute

[vapukov]

generally it could depend from Your Job design

but first of all - try (if not already) to edit task in scheduler and set start folder to folder where .bat file located
Start in (optional)

[YuriB]

thanks, done that, as I said the job runs and it creates files in same batch file where Talend DI JAR file is called, which i used for debugging to see if the batch file executes at all.

so task scheduler finds the batch file, run it, but the actual Talend JAR file execution either crashes but still returns success to the batch file (code=0) or something else prevents JAR file from running at all inside Task scheduler but the error code is still returned success (0). 

I was wondering of some Java JVM -D settings needs to be set up when it runs inside Windows Task Scheduler?

As I said, I am using REST API, AWS Cloud.

[vapukov]

generally not IF job set in scheduler with same user credentials as it tested in CMD windows

I have many Jobs with AWS, REST and any other functional run from Windows (and 10 and server) - no one of them was not require additional changes.

in other case need check - is this user have access to AWS, folders and etc

[YuriB]

thanks, yes, I am running batch job under my own credentials and ticked "use highest privilege" and I have local admin on my Windows 10 PC.

[vapukov]

You can add in Job few "dummy" checkpoint - as variant tFileTouch (each with sequence filename)

at the begin, before and after most important steps, and check - where Your job stop

[YuriB]

re this:

"I have many Jobs with AWS, REST and any other functional run from Windows (and 10 and server) - no one of them was not require additional changes."

Please clarify one thing for me: do you run the above Talend jobs inside Windows Task Scheduler ?

Thanks!!

[vapukov]

---

@YuriB wrote:

re this:

"I have many Jobs with AWS, REST and any other functional run from Windows (and 10 and server) - no one of them was not require additional changes."

Please clarify one thing for me: do you run the above Talend jobs inside Windows Task Scheduler ?

Thanks!!

---

Yes, my primary Jobs run under Linux (and from TAC and standalone), but I have many tuning on windows, and yes some of them call REST Services, some of them work with AWS

[YuriB]

thanks!
This helps a lot because it confirms that I must be doing something wrong.

I will have a look inside generated Java code to see if there is anything accessing mapped disk drives instead of UNC paths etc.... This is the most typical problem with windows task scheduler - disk drive letters and network path names.....

[vapukov]

may be You are right, my Jobs located on disk C: and work with remote locations over sftp, so I not test - how it work with UNC

would be good to know

[YuriB · Accepted solution]

Found the problem.

I searched the Java generated code, there was a reference to the Redshift JDBC driver LOG file which was located on a mapped network drive U:\workspace....

It was something I did not set, Talend DI picked it up from my workspace default settings, so it was a land mine ready to explode, which it did once I run in Windows task scheduler.

Given this happened "behind my back" I did not check this.

I changed the JDBC log file to point to C:\ instead of mapped drive U:\ and it worked.

This is a problem/bug with Java generated by Talend, IMHO.

If Talend Java code not access a mapped drive such as U:\workspace drive running from task scheduler then it must report ERROR code back to the Windows Command shell.

Instead it just exits and it reports exit code 0, success, which task scheduler reports in job history.

Windows task scheduler is also an evil piece of software, because it does not allow the use the mapped disk drive letters but this is the devil we know, it has always been this way.

I hope this helps the next poor unfortunate who is using Windows Task Scheduler with Talend DI Java.

[vapukov]

I meet with Redshift JDBC "problem" before, I use for this context parameter - talend_logs+jobName+".csv"

but at least on Linux, I was saw error this in logs

not sure - what information it save, need to test when find time - but You can enable logging for scheduler
but at least for debug purpose You can always use

YourProgram.bat >> log.txt

it will redirect all output to file

[YuriB]

Hello 

re this: "Yes, my primary Jobs run under Linux (and from TAC and standalone), but I have many tuning on windows, and yes some of them call REST Services, some of them work with AWS"

We are also thinking of running Talend jobs under Linux (considering either TAC and Linux standalone "free" options).

Windows Server costs license money and Linux Server is much cheaper/free and given Talend is all about Java/JVM we are thinking of  moving scheduled tasks over to Linux Server OS.

Question: how do you run Talend jobs on Linux standalone, specifically? cron? other ?

Are there any Talend Linux scheduled jobs tips, traps, issues - you can share with us on this forum please?

[vapukov]

---

@YuriB wrote:

Hello 

Question: how do you run Talend jobs on Linux standalone, specifically? cron? other ?

Are there any Talend Linux scheduled jobs tips, traps, issues - you can share with us on this forum please?

---

Most of Jobs run from cron, it is specific, because most of instances have not a lot running jobs (up to 10-15 per server)

but I also have a pilot installation with scheduler from SOS Berlin, not position my self as expert in this tools, but it work

http://www.sos-berlin.com

another alternatives - https://www.quartzdesk.com , http://www.schedulix.org/en

[YuriB]

thank you!

for Linux cron solution - are there any "traps" or "tips" running Talend jobs that is good to know ?

We develop Talend DI jobs on Windows 10 desktops and then we will migrate the JAR files and the shell batch files to run on scheduled tasks on Linux in production so I am wondering how well this will work ? I will have to deal with file paths/folders/directory names between Windows and Linux, this is a known thing.

Anything else?

[vapukov · Accepted solution]

if exclude - path and folders, on my jobs - no additional changes

just not forget change shell file mode

chmod u+x jobname_run.sh

before execute

[YuriB]

Thank you very much! 8^)

[migcunha]

@YuriBI have a maped drive in my tFileList node. Would this create the same problem you mentioned?