tExtractXMl component in a Spark job

Five Stars

tExtractXMl component in a Spark job

The tExtractXMl compoent in a spark job is complaining with the following error for the  below section of code .Unable to build the job.

 

Error message: "the code of method call (Tuple2<NullWritable,row9Struct>) is exceeding the 65535 byte limit" 

 

public java.util.Iterator<scala.Tuple2<NullWritable, row3Struct>> call(
scala.Tuple2<NullWritable, row9Struct> data)
throws java.lang.Exception {
java.util.List<scala.Tuple2<NullWritable, row3Struct>> outputs = new java.util.ArrayList<scala.Tuple2<NullWritable, row3Struct>>();
row3Struct row3 = new row3Struct();
row9Struct row2 = data._2;

 

PLease help me understand this issue .

Highlighted
Forteen Stars

Re: tExtractXMl component in a Spark job

hi,

 

65535 it is a limit for java method code (maybe I not 100% correct describe, but it is knowing error)

 

source of error could be a complicated structure (with long XPath and many columns)

there are no single solution, but often possible resolve it if:

  • exclude not used tags (if any)
  • split into several steps (if possible) - parse half, then next half, then join

 

-----------
Five Stars

Re: tExtractXMl component in a Spark job

Thank you very much. Your sugggestion worked. I had 498 columns with xpath. reduced it to 300 columns and that worked.

 

Thanks ONce again

 

Badri Nair

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now