Six Stars

Using Custom Code in tJavarow

Hi,

 

Is it possible to write custom java code in tJavarow  for Big data jobs.

I am using tJavarow inside a spark job.

but the custom code is not executed.

The following is the custom code

context.Flag="YES";

System.out.println("###################### This output is from tJavaRow ###################");

the value is not assigned as well as the message is not printed.

 

Thanks

 

 

  • Big Data
  • Data Integration
5 REPLIES
Nine Stars

Re: Using Custom Code in tJavarow

not sure about something more serious, but as described - yes it work 

 

variables - not assigned

 

Screen Shot 2017-06-07 at 10.06.00 AM.pngScreen Shot 2017-06-07 at 10.06.08 AM.png

-----------
Six Stars

Re: Using Custom Code in tJavarow

Hi ,

 

is that a talend bigdata spark job?

If so , instead of running as local, run it in a spark cluster.

 

I am running my spark job in the cluster.

if i put custom in tJava instead of tJavaRow, its getting executed.

I am able to see the output in spark application logs.

but custom code in tJavarow is not getting executed.

 

Thanks

Nine Stars

Re: Using Custom Code in tJavarow

yes, will test, but may be You are right - it will not work 

 

let wait - what Talend staff answer :-)

-----------
Six Stars

Re: Using Custom Code in tJavarow

yeah..lets wait..Smiley Happy

Five Stars

Re: Using Custom Code in tJavarow

Hello,

 

Custom code components (tJava and tJavaRow) behave and have to be used differently depending on what type of job you are building.  For instance, Spark batch jobs you need to write with Spark Java API syntax to work with the input and output RDD (read the comments in the component when you first add it for help on how to do a test print on your input RDD, try that instead of your system.out*).  In Spark streaming job, you'll be working with RDD in Dstream.  tJava and tJavaRow behave differently too which tJavaRow uses Spark DataFrames API and tJava is purely working with RDDs.

 

See the documentation for the differences between them when using across various types of jobs:

https://help.talend.com/reader/KxVIhxtXBBFymmkkWJ~O4Q/y0Us7J_ukdgxhe9Jx_o_NQ?section=sect-components...

 

Hope that helps.