memory error using tfileinputJson

Hello,
   When i am extracting around 200 MB of file using tFileInputJson it's erroring out as: Out of memory exception 
where as i have configured JVM to run on 
-Xms5230M -Xmx12240M
and i am having 20GB of RAM where i have scheduling this job.
But still i am facing this problem of memory.
Can you please suggest how optimizely i can do this using talend.
If any other conponent is there who can read line by line and load it rather than reading it whole and string in memory.
Or if whiler reading i can use temp disk space rather than caching ito memory.
What should be the configuration to run a >200 MB file using the tFileInputJson Component.
Thanks,
Jyotiranjan
4 REPLIES
Moderator

Re: memory error using tfileinputJson

Hi,
Have you already checked KB article about:TalendHelpCenter:ExceptionoutOfMemory? Did you configure JVM in Advanced Settings of Run view? What does your whole job design look like?
What's your OS and JDK version?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: memory error using tfileinputJson

Hi All,
I'm facing the same issue, I have a 144Mb Json file and the job just doesn't start processing the JSON file.
I already configure the job memory (-xmx8192M which should be far enough for that kind of processing) and currently my job consist of a TfileInputJSON followed by a tLogRow which is as simple as it gets.
I tried with a small extract (25kb) and it works fine so the job isn't the problem here.
Do you have some guidance to follow for "big" Json file processing?
I'm using Talend DI 6.0.0 and also tried Talend BD 6.1.1 with the same result.
Best,
Nicolas
Four Stars

Re: memory error using tfileinputJson

Using Xpath isn't the right solution, to process large json files one has to use JsonPath.
It creates other issues but using the jsonPath read the file element by element without first parsing the whole file.
Moderator

Re: memory error using tfileinputJson

Hi,
I'm facing the same issue, I have a 144Mb Json file and the job just doesn't start processing the JSON file.

What's the error message you are getting?
So far, there is no documentation to follow for "big" Json file processing.
Have you tried to set up input json file metadata to see if it works?
TalendHelpCenter:Centralizing JSON file metadata
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.