Cannot build standalone jobs since upgrade from 6.3.1 to 6.4.0

[jonasdb]

Hi there,

We upgraded to the latest Talend Data Integration (6.4.0) this week, and I've been getting issues with building standalone jobs.

Basically I can't build any standalone job anymore (except for two days ago, when it would build the first job I would try to build after opening Talend, but would fail afterwards. Hooray for consistency!). The build dialogue will run, list through what it's doing, disappear with no errors (no errors in logs either) and no zip is created.

I've:

• checked jdk setting (using latest 1.7 build, set compiler to 1.7)
• checked all jobs build successfully
• checked no routines have any errors in them
• been able to build war files
• unchecked unzip files in the build dialogue
• reinstalled java 1.7 and reapplied settings
• reinstalled Talend
• restarted my computer
• reloaded .m2 folder, and loaded all optional packages
• deleted .Java workspace folder, so it reloads
• created a new project, and tried to build a simple job from there. Nothing.

After some amount of googling, I came across someone talking about the .Java folder. In here, I've found the actual error in the 'lastGenerated.log' file:

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] MI_NEW Codes Master
[INFO] MI_NEW Routines
[INFO] MI_NEW Pingdom_To_Redshift (Job Designs)
[INFO] 
[INFO] Using the builder org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder with a thread count of 8
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building MI_NEW Codes Master 6.4.0
[INFO] ------------------------------------------------------------------------
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building MI_NEW Pingdom_To_Redshift (Job Designs) 0.1.0
[INFO] ------------------------------------------------------------------------
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building MI_NEW Routines 6.4.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ routines ---
[INFO] 
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ Pingdom_To_Redshift ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] Copying 43 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ Pingdom_To_Redshift ---
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ routines ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.7:copy-resources (copy-running-resources) @ Pingdom_To_Redshift ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ Pingdom_To_Redshift ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ Pingdom_To_Redshift ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ Pingdom_To_Redshift ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ routines ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ routines ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ routines ---
[INFO] Tests are skipped.
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ Pingdom_To_Redshift ---
[INFO] 
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ routines ---
[INFO] Building jar: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/target/pingdom_to_redshift_0_1.jar
[INFO] Building jar: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/target/routines-6.4.0.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.5.5:single (default) @ Pingdom_To_Redshift ---
[INFO] Reading assembly descriptor: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/src/main/assemblies/assembly_Pingdom_To_Redshift_0.1.xml
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] MI_NEW Codes Master ............................... SUCCESS [  0.002 s]
[INFO] MI_NEW Routines ................................... SKIPPED
[INFO] MI_NEW Pingdom_To_Redshift (Job Designs) .......... FAILURE [  1.659 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.764 s (Wall Clock)
[INFO] Finished at: 2017-05-19T12:24:06+00:00
[INFO] Final Memory: 50M/245M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single (default) on project Pingdom_To_Redshift: 
Failed to create assembly: Error adding file 'org.talend.code.mi_new:routines:jar:6.4.0' to archive: 
/Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/target/classes isn't a file. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :Pingdom_To_Redshift

So maven can't find the routines jar for some reason.

I've looked for it and have found it in '.Java/target'

Does anyone have any ideas?

I'm running on OSX 10.11.6

[xdshi]

Hello,

Please verify the java settings, that it points to a JDK (preferences / Java / Installed JREs)
(JDK is not needed to run job especially in the studio, but needed to export the job).

Verify in case the the structure of installation, in case there is something wrong.
folders of JDK must be like:
http://docs.oracle.com/javase/8/docs/technotes/tools/windows/jdkfiles.html

And the java settings must point to the root JDK folder, and not the JRE.

Have you tried to use JDK 1.8 to see if it works?

Best regards

Sabrina

[jonasdb]

Hi Sabrina, thanks for your reply.

• I've always had it pointing to the jdk (/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home)
• Would war files be able to build using only jre? Or is this a good indication that the jdk setup is fine?
• I have tried 1.8, but this makes no difference unfortunately

I also just did another reinstall, and removed all hidden files in the workspace this time in case this could have an effect but sadly it has not.

Note I have seen a few different people have had this issue in the past, and I've attempted all solutions to no avail.

Thanks

Jonas

[jonasdb]

Hi again,

any further suggestions for solving this issue?

Is the jdk setup usually the reason for this kind of problem? I've seen a few people with similar problems, but solutions seem to vary quite a lot.

Thanks,

Jonas

[jonasdb]

Ok issue seems to have gone after doing the following:

1. Delete the install folder

In the workspace folder:

1. delete .Java
2. delete .metadata

In the project folder:

1. delete talend.project (and copy this over from a colleague's install, as we do not store this in our git repo)
2. delete .settings
3. delete *.log

Reinstall from zip (did not rename the folder just in case), load all optional libs

[jonasdb]

Ok this only seems to have fixed the issue temporarily.

Reinstalling/refreshing everything seems to make some jobs run once the first time, but not others. Still trying to find out what could be causing the differences (perhaps routine dependencies? However, removing all user created dependencies doesn't make any more jobs build)

I've looked inside the routine.jar files created in the .Java/target folder, and when the build fails, extracting the jar gives an error (but still extracts):

java.io.EOFException: Unexpected end of ZLIB input stream
	at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
	...

The last file which gets created inside this is NoHeaderObjectOutputStream.class, which is zero bytes in size. So it seems that when trying to create this class there's an error, which means java doesn't correctly mark the end if the file of routine.jar.

Is there anywhere which would highlight why this error occurred?

For now my work around is to use the jars created in the target folder for the standalone job.

Cheers.

[pickle]

I'm having the same error, any help would be great.   (I uninstalled every JRE/JDK from my machine then downloaded the newest JDK 8, same error.   I blew away all of the jars under the lib directory and re-downloaded everything, same error) 

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] REPORTING Codes Master
[INFO] REPORTING Routines
[INFO] REPORTING report101 (Job Designs)
[INFO]
[INFO] Using the builder org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder with a thread count of 8
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building REPORTING Codes Master 6.4.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Building REPORTING report101(Job Designs) 0.8.0
[INFO] Building REPORTING Routines 6.4.0
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ routines ---
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ report101---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 44 resources
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ report101---
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ routines ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.7:copy-resources (copy-running-resources) @ report101---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ routines ---
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ report101---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO] Copying 0 resource
[INFO]
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ report101 ---
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ routines ---
[INFO] Nothing to compile - all classes are up to date
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ report101 ---
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ routines ---
[INFO] Tests are skipped.
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ report101 ---
[INFO]
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ routines ---
[INFO] Building jar: C:\tos64\TOS_DI-20170510_1410-V6.4.0\workspace\.Java\target\report101_0_8.jar
[INFO]
[INFO] --- maven-assembly-plugin:2.5.5:single (default) @ report101 ---
[INFO] Reading assembly descriptor: C:\tos64\TOS_DI-20170510_1410-V6.4.0\workspace\.Java/src/main/assemblies/assembly_report101_0.8.xml
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] REPORTING Codes Master ............................ SUCCESS [ 0.000 s]
[INFO] REPORTING Routines ................................ SKIPPED
[INFO] REPORTING report101 (Job Designs) ......... FAILURE [ 11.154 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.311 s (Wall Clock)
[INFO] Finished at: 2017-06-26T15:59:02-07:00
[INFO] Final Memory: 12M/245M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single (default) on project report101 : Failed to create assembly: Error adding file 'org.talend.code.reporting:routines:jar:6.4.0' to archive: C:\tos64\TOS_DI-20170510_1410-V6.4.0\workspace\.Java\target\classes isn't a file. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :report101

[jonasdb]

Hi Pickle, sort of glad I'm not the only person in the world with this issue now

Some extra info:

Setting maven to debug mode in talend (somewhere) gives some more specifics in the lastGenerated.log file (bellow), mostly about file permissions?

Also a semi work around, in case it helps – if you're just updating a job's jar, you can either get it out of the ./Java/target folder, or build as a war and extract it from there. You don't get the working routines.jar or .sh scripts from this though, so no good for fresh jobs.

Note: some jobs build sometimes, but I've yet to figure out what sets these semi working jobs apart from the others.

Jonas

[DEBUG] FileSet[Harvest_Load_Configs/src/main/templates/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] adding entry routines/system/SharedDBConnectionLog4j.class
[DEBUG] FileSet[Harvest_Load_Configs/src/main/resources/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/src/main/resources/xmlMappings/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/xmlMappings/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/src/main/resources/rules/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/src/main/assemblies/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/src/main/java/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] NOT reformatting any files in /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/src/main/java
[DEBUG] Adding file-set in: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/src/main/java to archive location: Harvest_Load_Configs/src/main/java/
[DEBUG] FileSet[Harvest_Load_Configs/src/main/java/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/src/main/resources/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] NOT reformatting any files in /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/src/main/resources
[DEBUG] Adding file-set in: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/src/main/resources to archive location: Harvest_Load_Configs/src/main/resources/
[DEBUG] FileSet[Harvest_Load_Configs/src/test/java/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/src/test/resources/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[sunfire-reports/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] NOT reformatting any files in /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/src/main/resources
[DEBUG] Adding file-set in: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/src/main/resources to archive location: Harvest_Load_Configs/
[DEBUG] FileSet[Harvest_Load_Configs/src/main/resources/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] FileSet[Harvest_Load_Configs/items/mi_new/metadata/survivorship/] dir perms: -1 file perms: -1
[DEBUG] The archive base directory is 'null'
[DEBUG] Processing binary artifact for module project: org.talend.code.mi_new:routines:jar:6.4.0
[DEBUG] adding entry routines/system/SharedDBPreparedStatement.class
[DEBUG] Adding artifact: org.talend.code.mi_new:routines:jar:6.4.0 with file: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/target/classes to assembly location: lib/routines.jar.
[DEBUG] Adding file: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/target/classes to archive location: lib/routines.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] MI_NEW Codes Master ............................... SUCCESS [  0.003 s]
[INFO] MI_NEW Routines ................................... SKIPPED
[INFO] MI_NEW Harvest_Load_Configs (Job Designs) ......... FAILURE [  3.019 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.132 s (Wall Clock)
[INFO] Finished at: 2017-06-27T09:27:30+00:00
[INFO] Final Memory: 13M/245M
[DEBUG] adding entry routines/system/SparkRunStat$StatBean.class
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single (default) on project Harvest_Load_Configs: Failed to create assembly: Error adding file 'org.talend.code.mi_new:routines:jar:6.4.0' to archive: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/target/classes isn't a file. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.5.5:single (default) on project Harvest_Load_Configs: Failed to create assembly: Error adding file 'org.talend.code.mi_new:routines:jar:6.4.0' to archive: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/target/classes isn't a file.
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
	at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:188)
	at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:184)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.maven.plugin.MojoExecutionException: Failed to create assembly: Error adding file 'org.talend.code.mi_new:routines:jar:6.4.0' to archive: /Users/jonas/workspace/dash/ETL/talendWorkspace_6/.Java/target/classes isn't a file.
	at org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute(AbstractAssemblyMojo.java:541)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
	... 11 more

[jonasdb]

actually it's probably because of lines

The archive base directory is 'null'

Seems this could be an issue as highlighted in this stackoverflow

[xdshi]

Hello,

Have you tried to create a new job in V 6.4.0 and build the standalone job to see if it works? Does this issue only repro on imported project(created on 6.3)?

Best regards

Sabrina

[jonasdb]

Hi Sabrina,

Thanks for your message.

I initially experienced the issue in an imported project. I've since tried both a new job in this project, and a new job in a newly created project, to no avail.

Also I'm now running the latest 6.4.1 version if DI but this has had no effect unfortunately.

Best,

Jonas

[xdshi]

Hi jonas,

Could you please create a jira issue on talend bug tracker? Our developer will check it to see if it is a bug.

https://jira.talendforge.org/secure/Dashboard.jspa

Best regards

Sabrina

[jonasdb]

Hi Sabrina,

I've submitted the bug here.

Thanks,

Jonas