We are using Talend Big data Enterprise edition. In our jobs, volume of data subject to change. Sometime, we will process 1 million and other time 10 million. Job is getting failed due default JVM memory settings. I want to change JVM memory settings in runtime, depending on the amount of data we process. Triggering the job through unix. How to handle this case while running through shell? Kindle guide me. Thanks
Thanks Sabrina. Right now, our .sh has default settings as java -Xms256M -Xmx1024M But this need to be changed in runtime, while running the job depending on the amount of data, we are processing. Thanks
You cannot change this setting while the job is running! It is not a lack of ability of Talend, it is simply the fact, these settings are initial settings for the JVM right at the start. It is not a problem to allow much more memory with this parameters. The memory will only be allocated if it is needed.
Thanks jlolling. If i run with default settings in .sh file, job fails due to out of memory issue for huge dataset. java -Xms256M -Xmx1024M You want us to allocate more memory(maximum anticipated depends on data in future)?. Any other way to handle this? Kindly help me.
Hi, Have you tried to allocate more memory to your job to deal with your out of memory issue before running? You can open the Job which you want to allocate more memory to and in the Run view, open the Advanced Settings tab and select theUse specific JVM arguments box. Then allocate more memory to the active Job by double clicking the default JVM arguments and editing them. Best regards Sabrina
-- Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.