Four Stars

java.lang.OutOfMemoryError: Java heap space

Hi all,

I'm new to the Talend Cloud Real-Time Big Data Platform.

 

We have the following scenario: a connection to a remote Azure Data Lake, where we try to write a file of 600MB of size. 

The component we use is a tFileInputRaw. When the job starts, after a couple of seconds it generates the following error: "java.lang.OutOfMemoryError: Java heap space". 

 

Of course it is a memory related issue. But changing the JVM settings didn't help.

 

As far as I can see the component itself has an option that says "Stream the file" and I feel like that this option would solve the issue. 

 

Is there anyone who implemented it? The documentation looks incomplete. 

 

Also other suggestions are welcome. 

1 ACCEPTED SOLUTION

Accepted Solutions
Moderator

Re: java.lang.OutOfMemoryError: Java heap space

Hello,

When you are streaming a file from a component like tFileInputRaw, it is intended to load the whole file into memory.

In other words, the full raw component reads the full raw file and put it into memory.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
3 REPLIES
Moderator

Re: java.lang.OutOfMemoryError: Java heap space

Hello,

When you are streaming a file from a component like tFileInputRaw, it is intended to load the whole file into memory.

In other words, the full raw component reads the full raw file and put it into memory.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: java.lang.OutOfMemoryError: Java heap space

Hi Sabrina,

 

thank you! Exactly, the full raw component is what we used and solved the issue! 

 

 

Moderator

Re: java.lang.OutOfMemoryError: Java heap space

Hello,

Great it works. Thanks for marking this topic as resolved.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.