How to work in memory to avoid temporary files

One Star

How to work in memory to avoid temporary files

Hi
I would like to transform a file to an other file by using several intermediary steps (split, sort, merge, ...).
For the moment I must create some temporary files
1/
Is it possible to avoid temporary files ? By using Array ?
2/
If not,
Is it possible to use 1 ou several output files like input files for the next step directly (without using some new items "tInputFIle" that are a copy of the output files) ?
Thank you for your help
Tags (1)
Employee

Re: How to work in memory to avoid temporary files

Hello
1) It's probably not necessary to use temporary files, but it depends of your business problem.
Can you post your current job for a more precise answer ?
2) you can directly put another transformer or output component after a tFileOutput, but in this case you do not need this temp component.

Best regards,
One Star

Re: How to work in memory to avoid temporary files

screenshots :
One Star

Re: How to work in memory to avoid temporary files

(bis.... bad handling)
----
Hello,
(mhirt, thank you for your answer.)
Here is a lot of precisions of my need
File in input (called "HOST1"), type "positionnal" with a length=25
----------------------------------------------------------------------------------
A1DUPONT JEAN-EDOUAM
A2DURAND JEROME
B1ZARAI RICA MME
There is 4 fields by row (for the first row : "A1", "DUPONT ", "JEAN-EDOUA", and "M ")
File in output (called "IMX1"), type "delimited" (warning, this is not a "true CSV file")
---------------------------------------------------------------------------------------------------------
02;INDIVIDU;A1;DUPONT;JEAN-EDOUA
03;INDIVIDU;A1;M
02;INDIVIDU;A2;DURAND;JEROME
03;INDIVIDU;A2;
02;INDIVIDU;B1;ZARAI;RICA
03;INDIVIDU;B1;MME
For the first row of the input file :
- The fields "A1", "DUPONT ", "JEAN-EDOUA" are copied in the row n°1 of the output file (with the constants "02" & "INDIVIDU")
- The fields "A1", "M " are copied in the row n°2 of the output file (with the constants "03" & "INDIVIDU")
(...)

For the moment, I do those actions :
- split of "HOST1" in 2 temporary files with tMap
- delete temporary previously files (in Perl)
- merge of the 2 temporary files in one temporary file (by using tFileList + option "Append" of tOutputDelimitediIle)
- sort of the temporary file (because the order is important for my need)

The resut is ok, but I had to use some temporary useless files.
For this need, the main difficult - for me - is that I didn't succeed in :
- transforming one row to n rows (here n=2)
- using 2 output files in input of a tMap (without using temoporary files)
Thank you for your help or comments
Jerome
Employee

Re: How to work in memory to avoid temporary files

You're right, this type of transformation can't be currently done without temp files.
We are currently working on tNormalize/tDenormalize components which will exactly match this transformation.
They will generate n*m line in output for n line in input.
We hope to make those components available in next stable release (2.1)
Thanks for your support.
Michaël.