Five Stars

thashoutput/input not available in Big Data Framework

Hi all,

 

I have built a standard job which requires some intermediate results to be combined and stored in a database. Therefore I have used the combination of thashoutput and thashinput. However, when I changed the standard job to Spark job, the thashoutput/input could not be loaded. Can anyone suggest which component(s) can use to replace thashoutput/input in Spark framework?

 

Thanks in advanced.

 

Regards,

Sam

2 REPLIES
Moderator

Re: thashoutput/input not available in Big Data Framework

Hello,

So far, Cache components, tCacheIn and tCacheOut can be available in the Spark Batch and Spark Streaming Job framework.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: thashoutput/input not available in Big Data Framework

Hi Sabrina,

I notice that, unlike tHashOutput, tCacheOuput does not have the option to "Link with a tCacheOuput". Is it possible to append intermediate results with the same schema from different subjobs by using tCacheOuput and set the Storage Level to 'Memory only'?

 

Thanks in advance.