Not able to iterate files in Talend Spark using tLoop

Four Stars

Not able to iterate files in Talend Spark using tLoop

Is there a way to iterate files in Spark ? Like tfilelist . I know we can give multiple files as input in tfiledelimited input by giving wild card char "*". But my requirement is to iterate each xml file and create corresponding output file with same name. May i know if u have any suggestions on how to achieve this in spark ?

 

Here is the solution (Screenshot below) that i tried with tLoop which also not working. 

 

In the below code, context.FileList= concatenated list of files with ",", eg: file1,file2,file3 and so on...

 

tJava component Code ==>

List<String> FileList = Arrays.asList(context.FileList.split(","));
context.FileListN = context.FileList;

System.out.println("FileList -->"+ context.FileListN);

//String FileListArray[] = context.FileList.split(",");
int listsize = FileList.size();
int i;

globalMap.put("SizeofList", listsize);
globalMap.put("FileList", FileList);

System.out.println("FileList -->"+ globalMap.get("FileList"));
System.out.println("Size of List -->"+ globalMap.get("SizeofList"));

for(i=0;i < listsize; i++){
String filenm = ((List<String>)globalMap.get("FileList")).get(i);
System.out.println("Processing File -->"+ filenm);
Until_ConvertXMLtoJSON_V1.XmltoJson(context.FileDir, filenm);
}

 

 

tLoop ==> 

While loop

Declaration == int i=0

condition =i <((Integer)globalMap.get("SizeofList"))

Iteration =i++

 

Let me know if i am missing anything!

 

Moderator

Re: Not able to iterate files in Talend Spark using tLoop

Hello,

tLoop component in spark job has the same feature of the DI one, except the parallelization, which does not exist on Spark.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: Not able to iterate files in Talend Spark using tLoop

The tloop component did not work for me. Can you post a solution if u have
with tloop. I want same file names for input and output both.
Five Stars

Re: Not able to iterate files in Talend Spark using tLoop

HI, 

Do you have any solution for the same? I have the same requirement hence the same issue .

 

Thanks

BAdri Nair 


@xdshi wrote:

Hello,

tLoop component in spark job has the same feature of the DI one, except the parallelization, which does not exist on Spark.

Best regards

Sabrina


 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now