One Star

What will be the Maximum Size Can be taken into tFileInputXML

Does tFileInputXML reda 5GB XML File???What will be the Maximum Size for XML file that Can be taken read through tFileInputXML....
6 REPLIES
Seventeen Stars

Re: What will be the Maximum Size Can be taken into tFileInputXML

hi praveen,
ther is no just only answer
Do you want to read the entire file or some elements ?
For huge file use Sax Parser because DOM save a "snapshoot" of the document in 'live' memory available by Jvm.
So for entire file with Dom parser ,you 'll have to give a lot of memory for JVM
http://www.oracle.com/technetwork/articles/javase/monitoring-141801.html#Insufficient_Memory
so you'll have to manage -Xmx and -Xms size to have enought space jvm memory (heap)
regards
laurent
One Star

Re: What will be the Maximum Size Can be taken into tFileInputXML

I want read the entire file and it is of size 1 GB,i have increased the size aswell,still i am getting the same java heap space issue....for that purpose i want to check the max size of xml file with max memory consumption for 4GB RAM Machine.
Seventeen Stars

Re: What will be the Maximum Size Can be taken into tFileInputXML

you can find some issue using jvm monitoring console
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
regards
laurent
One Star

Re: What will be the Maximum Size Can be taken into tFileInputXML

Can you please let me know the maximum size of the XML file that Talend Integration Suite can Process if i want to process entire file.
It will be great if somebody can give direct answer in numbers....like 5GB or 10 GB???...
Thanks,
KPK
Seven Stars

Re: What will be the Maximum Size Can be taken into tFileInputXML

There is no direct answer. It depends on your job design: which parser you use, how many columns are in your schema, what processing is done with the file etc..
One Star

Re: What will be the Maximum Size Can be taken into tFileInputXML

i have a very simple xml file which is of size 1gb and iam trying to split using tfileinputxml---->Tfileadvancedoutputxml
very simple job and i used dom4 and SAX as well..still iam getting the same issue...will talend allow more than 1 GB files...why it is not working for even simple job..somebody put an end to my issue.....