Four Stars

How to retrieve process flow record count in Talend Bigdata Spark jobs?

Hi,

 

I was trying to build  a Spark job in Talend, i was trying to look for global variable NB_LINE. I was not able to get that from Outline. Is there any option available in Spark?.. Please advice..

 

I also need some inputs regarding tSqlRow component, any sample SQL's that we can use in this component?.. Appreciate your inputs..

 

Regards

Sudhar

4 REPLIES
Moderator

Re: How to retrieve process flow record count in Talend Bigdata Spark jobs?

Hello,

Have you tried to utilize a DI Job to orchestrate the context of the Spark Job? Could you please have a look at this article:https://community.talend.com/t5/Architecture-Best-Practices-and/Spark-Dynamic-Context/ta-p/33038 to see if it can meet your needs.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: How to retrieve process flow record count in Talend Bigdata Spark jobs?

Thanks for sharing the link. My question was more towards how to get the record count of a link in Spark jobs. I am not able to see the options similar to regular DI job. Please advice

 

Thanks

Sudhar

Moderator

Re: How to retrieve process flow record count in Talend Bigdata Spark jobs?

Hello,

Could you please give us some description about your current bigdata spark jobs? The tflowmeter and tflowmetercatcher are not available in bigdata spark job, so far.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: How to retrieve process flow record count in Talend Bigdata Spark jobs?

Hi,

 

Our current requirement in Spark job is to get the job flow details..

We are trying to moving data from one layer to another layer, we would need to capture all the job metadata information (source file/target record count etc) from every job. This data need to stored for audit and reconciliation purpose.. I see that in Spark we are having limited option to get this information... I see that in regular DI we have all components that can help in getting all job information.. please provide your inputs for Spark execution.. will AMC be capturing the information that i am looking for..

 

Thanks

Sudhar