One Star

Getting error while processing large amount of data

Talend is installed in a 32 bits system(4GB RAM). I was migrating 20GB data with some transformation. It was showing error like that

connecting to socket on port 3968
connected
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
disconnected
disconnected
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Unknown Source)
at java.lang.Class.getDeclaredField(Unknown Source)
at java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl.<init>(Unknown Source)
I have made  the following changes 
1) In tmap  set "Store temp data" option as true and also set a directory for temp data
2)Also set JVM arguments as per my system configuration
Still I am getting that error.
Please help me on that issue.
Thanks in advance.
14 REPLIES
Moderator

Re: Getting error while processing large amount of data

Hi,
Does this issue only repro on your specific job?
Did you edit the JVM parameters in .ini file for studio itself or open the Run view, click Advanced settings tab and then edit the JVM parameters for your current job?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Getting error while processing large amount of data

Hi Sabrina,
I have set it in run view > Advanced settings tab.
Still getting error :rolleyes:
Please help me
Moderator

Re: Getting error while processing large amount of data

Hi,
What's your current JVM parameters? Could you also show us your job design? Which components are you using in it?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Getting error while processing large amount of data

Hi Sabrina,
Could you please share your email address so that I can send you all the details about that job.
Thanks
Babul
Five Stars

Re: Getting error while processing large amount of data

Hi Babul,
What component (tMysqlInput?, tPostgresInput?, ???) are you using to read the data?  If you are using the tMysqlInput component, there is an 'Enable stream' option on the component's Advanced settings tab that may be of help to you.
Hope this helps.
 
One Star

Re: Getting error while processing large amount of data

Hi,
I am using postgres database  as source database as well as target database.
Is there any such option ?
Thank
Babul
Five Stars

Re: Getting error while processing large amount of data

Let's see...
the tPostgressqlInput component has a 'use cursor' option on the 'Advanced settings' tab which allows you to set the cursor size and may help you better manage the amount of data you are bringing in on your select. 
the PostgressqlBulkExec component may also be of use to you though I do not see any options that let you determine different ways to handle the incoming data. 
Not having a Postgres database to test with, I can only guess on what I see.  That said, experiment a little with the above two and see how that works. 
One Star

Re: Getting error while processing large amount of data

Hi,
Now I am facing a new issue. In my 25 GB data migration, execution speed is very low (260 rows/s).
I am using postgres database as source database as well as target database.
Is there any way to improve the execution speed ??
Thanks 
Babul
Moderator

Re: Getting error while processing large amount of data

Hi,
Here is an option "Use Batch Size" in tPostgresqloutput which is used to activate the batch mode for data processing.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Getting error while processing large amount of data

Hi,
I already set the batch size. In spite of that, Job is taking 20 hours to migrate 25 GB of data.
Is that issue related to Talend or Database ?
Thanks 
Moderator

Re: Getting error while processing large amount of data

Hi,
What's the row rate? Would you mind sharing your job design screenshot with us?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Getting error while processing large amount of data

Hi,
I am not able to upload any screenshot.
(tinput1,tinput2) --> tMap -->tfilter --> toutput
tOutput is being loaded with 1200 rows/s. Total data volume is 25 GB.
Thanks
Babul
One Star

Re: Getting error while processing large amount of data

Hi Sabrina,
Please help me if there is any way to enhance the processing speed.
Thanks 
Babul
Moderator

Re: Getting error while processing large amount of data

Hi Babul,
Performance issue is usually caused by the DB connection or the job design. You can register and log in as a Community member first after that  "UPLOAD"  button should be available on your end or could you please export the job and email it to us?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.