I create the XML metadata for it but I can't enter more than 175-200 vars? I could just copy some fields over and run it 3 times to have all fields and join the 3 results. But when I load the first time (not even tried to merge/join/..) it takes half an hour to run the job. And it does nothing. I have an inputXML with the metadata (calling the vars that I want with the Xpath.-> works when defining the vars/xpath) connected with a tmap and a mssqloutput.
What could possibly be the problem? JOb starts and finishes but has no rows?
Hi, You have 3 Generation mode in the tFileInputXML component in the view Advanced Settings = Slow and Memory-consuming / Memory-consuming / Fast with low memory consumption. Considering the size of your XML file which is uncommon; I would suggest you turn the Generation mode in : Fast with low memory consumption. Then for that size of XML and requirement for Hierarchical file mapping and parsing; Talend Platforms (Data Management, BigData, DataServices, etc...) provide the Talend Data Mapper along with the tHMap which is strongly focus on large and complex XML or JSON dataset; or EDI or EBCDIC Copybook mainframe files. If this XML requirement is a recurrent requirement in your organization and very critical for your project; it could be a good idea to get introduced to our Sales team for a quick tour of our Platform offering and the Talend Data Mapper. Best regards;
Okay, Did some more investigation and tests: Filesize is not the issue. The issue is the number of variables for the metadata. i can only enter up to 211 in the metadata and he won't use them I don't have an output. If I only make the metadata for the same file with for example 30 vars it runs. Anybody having ideas of workarounds?
FrederikvdgTalend - are you trying to read the big XML or write to it? Let me know... I'm using Data Mapper for a similar complex XML file at the moment and will soon share how to set up and use it... Short of using Data Mapper, the way I've gotten around this issue in the past is leveraging database XML methods to generate XML and pass that to Talend for stitching. Here's an example on Oracle: SELECT KeyField1, KeyField2, XMLAGG ( XMLFOREST ( field1, field2, .... .... ).GETCLOBVAL()) AS DATAROW_XML FROM TableName What I'm doing above is pushing generation of a huge list of fields back to the DB, and returning just 1 field (Datarow_XML) that has a neatly formatted XML field. I then pass this to say a tFileOutputMSXML to build the XML. This is how I've typically gotten around the "is exceeding the 65535 bytes limit" (the 65k java method limit). Let me know if this helps... Else, share more details...
Do you mean 211 fields defined in the schema? Perhaps it has something to do with the size of the struct or method being generated as Will mentioned. An experiment: What if you use the shortest field names possible? Try to shorten all the field names in the schema to less than 8 characters.
539? Yes, I can see that might be a problem. You might try the old tSmooksInput component if you can find it and make it work with the current version. You would still have alot of Xpaths but you offload the XML processing to Smooks and can use separate schemas if you are looping, or you could also see if you can split it with Smooks or Twig. If you need ALL of them in one schema that's going to be tricky. I suppose you might be able to make multiple passes (gross)