One Star

[resolved] java.lang.OutOfMemoryError: Java heap space

Hi All.
One Table have 4million rows.
I am Exporting 4million rows to csv file i am getting below error.
"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"
How to increase Java heap space?
Thanks
kumar.talend

  • Data Integration
30 REPLIES
Four Stars

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

You can increase the heap from the Run->Advanced settings tab. You can also pass parameters etc, if not running in the dev environment. This is not always the best option.
What's your source? If it's MySQL, you can stream the data, for example.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

HI tal00000
Thanks for your feedback.
My Source is postgressql.
I increased JVM size : -Xmx2048M
Again I am getting below error:
"Could not create the Java virtual machine.
Error occurred during initialization of VM
Could not reserve enough space for object heap"
Thanks
Kumar.talend
Four Stars

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

I think that's your own local issue. You can Google to find why you're getting this message.
tPostgressInput->Advanced settings supports "Use cursor" and a solution may lie there.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

HI tal00000
Thanks for your feedback.
tPostgressInput->Advanced settings supports "Use cursor".
check the Use cursor check box and enter 5000000 in Box.
But same problem.
"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"
Any ideas.......
Thanks
kumar.talend
Four Stars

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

try 1000 and see where that gets you
HI tal00000
Thanks for your feedback.
tPostgressInput->Advanced settings supports "Use cursor".
check the Use cursor check box and enter 5000000 in Box.
But same problem.
"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"
Any ideas.......
Thanks
kumar.talend
Moderator

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi,
Hope these articles on talend help center will be useful for your job.
Exception+Could+not+reserve+enough+space+for+object+heap
Exception+outOfMemory.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi Sabrina,
Thanks for your feedback.

I am modify the "TOS_DI-win32-x86.ini" configuration file based on your suggested article.
No changes, i am getting same error.
"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"

Thanks.
kumar.talend
Community Manager

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi
I am modify the "TOS_DI-win32-x86.ini" configuration file based on your suggested article.

The "TOS_DI-win32-x86.ini" configuration file is for Talend Studio application itself, not for the job execution, don't know which article or blob have you read, please refer to the KB article Exception+outOfMemory to learn how to allocate more memory to the job execution.
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi Shong,
Thanks for your feedback.

I am modify the "TOS_DI-win32-x86.ini" configuration file through talend help center reference article.
and also I tried to increase heap size as per your suggested article.
I tried to increase heap size like below steps:
1)In tMap I select the option Store on disk and given file path.
2) tPostgressInput->Advanced settings supports "Use cursor".
check the "Use cursor" check box and i given 4000000 value.
In Both the case i am getting below error:
"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space".
3)In the Run view, open the Advanced Settings tab and select the Use specific JVM arguments box
and i set the JVM arguments like below:
-Xms512M
-Xmx2048M
In the above case i am getting below error.
"Could not create the Java virtual machine.
Error occurred during initialization of VM
Could not reserve enough space for object heap"

I tried to fix the OutofMemory issue all possible way.But it is escaping.
My job design.
tPostgressInput-->tMap-->tFileOutputDelimeted
Thanks to all for giving your valuable suggestions.
Thanks,
kumar.talend
Employee

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

You re running Talend on a 32 bits OS so you cant exceed 1536Mo for the JVM memory.
Please use the cursor option, but not with 5M value. Lower it to 100 000 rows. Should work.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi guys,
I am having exactly the same error with the user above in Talend. Did someone find a solution to this as yet?
Your help is much appreciated as I am in the middle of a project.
Thanks and Regards,
AG
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi,
I had the same error and these posts were helpful.
In the Postgres input, 'Advanced Settings' I set "use cursor" to 100 000.
In the Run 'Advanced Settings' I put both JVM arguments to Xms1024M and Xmx1024M.
It worked.
Community Manager

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi telena
Glad to see you figure it out and thanks for your feedback!
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi,
Hope these articles on talend help center will be useful for your job.
Exception+Could+not+reserve+enough+space+for+object+heap
Exception+outOfMemory.
Best regards
Sabrina
Hey I want to generate data using java progam but i'm getting an error Exception in thread "main" java.lang.OutOfMemoryError: Java heap space. Can you plz help me
Moderator

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi mukuljain015,
Hey I want to generate data using java progam but i'm getting an error Exception in thread "main" java.lang.OutOfMemoryError: Java heap space. Can you plz help me


Have you already changed your JVM arguments in  'Advanced Settings' of Run tab? Did this issue only repro on your specific job? Could you please give us more information about your current situation?

Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi, i have the same problem but it happens when procesing a Delimited File with 381996 records. I tried setting the VM arguments of the job to -Xms1024M and Xmx1024M but still throws the Java Heap Space Exception. 

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2367)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
at java.lang.StringBuilder.append(StringBuilder.java:132)
Moderator

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi sbravo,
What's your whole job design? Is there any DB component?Did you use any tMap, tUniqRow, tSortRow component in your work flow? What's the RAM of your machine?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

other option you can try is selecting 'stream the file' in component view of the input file. (only for file based inputs)
Regards
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi,
I am using two source files (delimited .dat files ). Both the files have 2 million records . I am joining these files in tmap and sending the output to a flat file (.dat). but its giving me Java heap memory error after reading 1 lakh records.
I have tried all the options written in above comments ( tried increasing Xmx parameter, Store temp data =Yes), but still I am getting the error.
Can someone please suggest what should I Try.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi,
I am using two source files (delimited .dat files ). Both the files have 2 million records . I am joining these files in tmap and sending the output to a flat file (.dat). but its giving me Java heap memory error after reading 1 lakh records.
I have tried all the options written in above comments ( tried increasing Xmx parameter, Store temp data =Yes), but still I am getting the error.
Can someone please suggest what should I Try.

You need to increase it again, or you just need more memory.
Check the task manager to see if you're RAM is full when you're running the job
Moderator

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi ummed,
Did you increase Xmx parameter in the Run 'Advanced Settings'which works for your current job not Talend Studio?
What's your OS and RAM?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi Sabrina,
Thanks for your reply. Yes, I increased the Xmx parameter value in Run   'Advanced Settings' as well. I tried all the combinations (increasing it at Talend studio level, increasing at Run job level and increasing for both of them). My OS is windows 7 and RAM is 4 GB.
When I put this job on client server, it is running fine.
Q-1)  so I guess its RAM issue on my system, Please confirm .
But another problem is, its taking 15 minutes to join 2 flat files (3 million records in each file ) and write to target flat file.
Q-2) is there any way I can increase the performance?
One way I can think of is Parallelism , but it seems in Talend open studio we do not have option to do Parallelism ( its available in Talend Data integration ).
So before moving to Talend Data Integration, I want to do testing if how much it will improve the performance.
Q-3) Can you please let me know if there is any option to download trail version of Talend Data Integration so that I can do initial testing before moving to Talend Data Integration.
Moderator

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi ummed,
As suggested, you need to optimaize the job deisgn and allocate more memory to the job execution.
Would you mind showing us your current job design screenshots?
Q-3) Can you please let me know if there is any option to download trail version of Talend Data Integration so that I can do initial testing before moving to Talend Data Integration.

You can download a free trial Talend Enterprise for Data Integration product from Talend Official Website:https://www.talend.com/enterprise-data-integration
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi,
I have installed Talend Enterprise edition from above link. And i got below icons on my computer in talend folder but all of them are only services. How do i start the  Talend to build any jobs( i don't see any icon for job developer) ? is there something missing in the installation?
Please suggest.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

attaching the screenshot
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

I am not able to attach the screenshot. but in my windows- all programs one folder "Talend 5.6.2 " is created and below is the structure of this folder:
Talend 5.6.2 ( this is main folder )
 
 Talend Administration Center ( this is 1st option in main folder)
 Manager Services     ( this is sub folder which has below options)
      Start Remote Job Server
      Start Talend Administration Center
      Start Talend Command Line Service
      Start Remote Job Server
      Start Talend Administration Center
      Start Talend Command Line
      
How can i start building the job from here as the only options i see are to start the services.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi,
I was able to find the window where to create job. But i have another questions now:
1. how to login into Talend Enterprise Data Integration Administration Center ( i don't remember setting up any id/password for this while installing )

2. When i right click on my toracleinput , i don't see option for set parallelism  . Can you please let me know why.  I was trying to do as explained at below link.



Thanks,
Ummed
Moderator

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

Hi ummed,
Here is a response in your another topic https://www.talendforge.org/forum/viewtopic.php?id=45006.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] java.lang.OutOfMemoryError: Java heap space

I upgraded the Talend from 5.5.1 to 6.1.1 recently. I am getting this "java.lang.Exception: java.lang.OutOfMemoryError: Java heap space" when I build my job in Talend 6.1.1 Studio. This never happened in 5.5.1 Studio. This is the setting for my 6.1.1 Studio.
-vmargs
-Xms1024m
-Xmx2048m
-XX:MaxPermSize=512m
-Dfile.encoding=UTF-8