[resolved] What about having several tBufferOutput in a job?

One Star

[resolved] What about having several tBufferOutput in a job?

Hi,
I was wondering why it is not possible to use several tBufferOutput in a job.
I would to pre-load several data sets (store each of them in a buffer output) and reuse them (via a buffer input) many times.
This will avoid to have to load data each time I need it.
Do you have another solution to do that as buffer does not support it?
Cheers,
Seb

Accepted Solutions
Employee

Re: [resolved] What about having several tBufferOutput in a job?

Hi,
You can use the tHashInput and tHashOutput components. These components are disabled by default. To show them in the Palette:
- Click on File / Edit Project Properties.
- In Designer, select Palette Settings.
- Move the Technical Folder to the right side. Click on Apply then click on OK.
- Now you will have a technical folder in your palette. The components are in there.
You can link several tHashOutput together by selecting the Link with a tHashOutput option, so that even if your tHashInput only selects one tHashOutput to get data from, if all the other Output are linked to this Output, the Input will get all the data.
Hope that was clear enough.

All Replies
Employee

Re: [resolved] What about having several tBufferOutput in a job?

Hi,
You can use the tHashInput and tHashOutput components. These components are disabled by default. To show them in the Palette:
- Click on File / Edit Project Properties.
- In Designer, select Palette Settings.
- Move the Technical Folder to the right side. Click on Apply then click on OK.
- Now you will have a technical folder in your palette. The components are in there.
You can link several tHashOutput together by selecting the Link with a tHashOutput option, so that even if your tHashInput only selects one tHashOutput to get data from, if all the other Output are linked to this Output, the Input will get all the data.
Hope that was clear enough.
One Star

Re: [resolved] What about having several tBufferOutput in a job?

Thanks,
It works fine.
Why theses components are not directly available?
Is there a limitation or are they deprecated?
I can't find documentation of them. Do you have one?
Seb
Employee

Re: [resolved] What about having several tBufferOutput in a job?

Glad to hear it worked.
These components are usually reserved for use by advanced users. I guess we don't want them to be in bad hands!
I don't think there is a documentation for them.
One Star

Re: [resolved] What about having several tBufferOutput in a job?

Hi,
A further question.
It seems that If I want to drag a subjob into a further job,the output of subjob should be tBufferOutput,or else,I can't transfer the data from subjob to the father job directly.Is that right? However, my workmate use the tMSSqlOutputbulk as the target component of the subjob and then use main link linking the subjob to another is ok,but when I try, it just transfer one rows,even the data is wrong,from the subjob(tRunJob).He(my workmate) has tried it in my computer and using my workspace ,the same one-row-transfered result.
Community Manager

Re: [resolved] What about having several tBufferOutput in a job?

Hi Joe
Yes, you must use a tBufferOutput in sunjob to pass the output to father job. The schema of tBufferOutput and tRunJob should be the same.
Best regards
Shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: [resolved] What about having several tBufferOutput in a job?

Hello Shong
It seems that tBufferOutput will hold the memory and won't release it untill the job over.
So,if there are several subjobs and each needs a 'tBufferOutoput',it will waste a huge memory.
I have the experience: I run my job for three times in succession,and I find it takes 7G of the server memory.
What's more,my workmate use a 'tMSSqlOutputBulk' as the target component of the subjob,and it runs well.But using the same way in defferent computer and defferent workspace,I get a only-one-row-transfered result,just as I describe above.
One Star

Re: [resolved] What about having several tBufferOutput in a job?

subjob_1: ...?main?>tmap??>tMSSqlOutputBulk subjob_2: subjob_1(tRunJob)?main?>tmap??>
| |
-----------------------------------------
just like this,'without tBufferOutput',it runs well.However,when I try it like this ,there is just one row in the subjob_1 to tmap line.
We use TOS 4.1.1.

Re: [resolved] What about having several tBufferOutput in a job?

Joe, it sounds like you should change your design. If you truly need to pass large amounts (7GB!) of data between subjobs, you should probably just include the components of your subjob as a joblet or directly in your master job. When you use tBufferOutput it is pushing all variables to memory. Its much more efficent to deal with data as a stream. Eliminate your the need to buffer large numbers of rows and your ETL will be much better Smiley Happy
One Star

Re: [resolved] What about having several tBufferOutput in a job?

HI All,
I am working on web services using talend . I have a simple map tOracleInput ----->tBufferOutput ,
The performance of this map is very poor. it takes nearly 10mins to load 5k records.
Is there any other way to send output to web service .
Please advice me
Thanks in advance,
Naveen Kumar C