In my workflow, i iterate through each row in a delimited file. The file can be quite large at times. One of the things I need to do to each row is look up an exchange rate that is stored in a database. I wrote a Java routine that I call from within a tMap component to do this. I have attempted to re-use the same DB connection by pulling it out of the globalMap and passing it as a parameter to my method. However, I'm still getting weird errors after a couple thousand rows. "java.lang.OutOfMemoryError: unable to create new native thread at .....". I have also tried playing with the JVM arguments, setting Xmx and all the others without any luck. thanks
I've ruled out it has anything to do with re-using a DB connection. I took that out of my workflow and the same thing is still happening. Altering JVM arguments only changes how long it takes before I get the OutOfMemoryError. Usually it's between 500-3000 rows depending on the JVM arguments. A bit more info: My Talend job consists of a main job and a subjob. Main job has a fileInputPositional iterate through each record and go to a tRunJob to call my subjob. The tRunJob passes about 100 context parameters, but the checkboxes for "Use an independent process to run subjob", "die on child error" and "transmit whole context" are all unchecked. In my subjob I get each row via a tFixedFlowInput. It doesn't matter what else I have in my subjob - even de-activing it all won't eliminate the error.
The tRunJob component in my workflow spawns a new thread each time it is called. Is this the intended behavior? I have a tFileInputPositional -> Iterate -> tRunJob. My input file is thousand rows long so each time a row is passed to the subjob a new thread is created (and never stopped!) leading to the out of memory errors after approx 3000 rows. What I can't figure out is that an earlier version of this exact same Talend project worked fine and it did the exact same thing (even processing files hundreds of thousands of rows long!) What is going on? Anybody?
I tried making a different subjob that was completely empty and it was still spawning new threads each time it was called. Then I uninstalled and reinstalled Talend. I started up the program again and ran it using my empty subjob. THIS time it worked. It did not create thousands of zombie threads. I next deactivated that and re-activiated my original tRunJob calling my original subjob. It went back to creating zombie threads and crashed. I then switched it back to calling my empty subjob and it still created the zombie threads, despite working the first time I tried it after reinstalling Talend. Pleeeeease help!
When I delete the .metadata, .JETEmitters, and .Java folders from my workspace and re-open Talend and re-import my project it will run the first time. The second time I try running it I get an "Update Detection" window that pops up. Here's what the window says... The properties used in the following items have been modified. Choose and update what you have checked. (*) : This job is not opened. Then it has my main job listed with Components -> Components | Reload components underneath it. The components checkbox is checked by default. If i leave the box that way and hit ok and run, it'll go back to spawning zombie threads until its out of memory.
Alright Talend people, Talend 3.2.1 does not have this problem, but Talend 4.0.2 (TOS-Win32-r43696-V4.0.2) does. Talend 4.1M4 also does. So either its a bug in Talend 4, or the way subjobs work has been changed. Even though I finally have a way forward, I'd like to know which it is....
Hello Matt Sorry for delay, I did't see your topic before. Please up the topic or notify it to me if you don't get an answer later. About your question, - The tRunJob component works the same behavior on the version 3.2.1 and 4.0.2 if the box "Use an independent process to run subjob" is unchecked. - Please check if you check the box 'Muti thread execution' box on the job setting of main job and subjob, check if you check the box' Enable parallel exection' of iterate. - Can you send me your job via email? Best regards Shong
---------------------------------------------------------- Talend | Data Agility for Modern Business