From Thursday, July, 9, 3:00 PM Pacific,
our Community site will be in
read-only mode
through Sunday, July 12th.
Thank you for your patience.

OutOfMemoryError on tFileInputJson

Highlighted
Seven Stars

OutOfMemoryError on tFileInputJson

Hi,

 

I am in trouble with a tFileInputJson from an url.

I have the following error : 

 

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at org.json.simple.JSONArray.toJSONString(Unknown Source)
at org.json.simple.JSONArray.toJSONString(Unknown Source)
at org.json.simple.JSONArray.toString(Unknown Source)
at p_flux1.api_grand_debat_0_1.API_Grand_Debat.tFileInputJSON_1Process(API_Grand_Debat.java:779)
at p_flux1.api_grand_debat_0_1.API_Grand_Debat.tFixedFlowInput_1Process(API_Grand_Debat.java:1920)
at p_flux1.api_grand_debat_0_1.API_Grand_Debat.runJobInTOS(API_Grand_Debat.java:2171)
at p_flux1.api_grand_debat_0_1.API_Grand_Debat.main(API_Grand_Debat.java:2020)

 

The job is running for like 15min, the tFileInputJSon is stuck at Starting, and the error comes.

 

I dont find any solution, and i can't limit the data to dowload from the url Json doc.

 

Any help is welcome.

 

Regards,

 

Pierre

Regards,
Pierre
Highlighted
Employee

Re: OutOfMemoryError on tFileInputJson

Hi,

 

    A quick method is to increase the JVM parameters associated to your job to increase the memory.

 

     You can change them at Run tab -> Advanced Settings 

 

image.png

But it is just one side of memory management. You need to verify the flow why it is taking more memory. If you are using a tMap after reading the data from source, you need to use disk operation than full memory based operation.

 

image.png

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

 

Highlighted
Seven Stars

Re: OutOfMemoryError on tFileInputJson

Hi,

 

sorry for delay,

 

No it's not working, i have tried that trick already and i still get the same error, the file is really huge :

http://opendata.auth-6f31f706db6f4a24b55f42a6a79c5086.storage.sbg5.cloud.ovh.net/2019-01-31/LA_TRANS...

 

I guess i should use a trest component but i could not make it work.

 

Pierre

Regards,
Pierre

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now