Processing output of "tSystem"

Highlighted
One Star

Processing output of "tSystem"

Hello,
I often have to do a query at the command line and processing the result.
Actual I do the following steps:
1. Delete possible existing temp-files
2. Execute the command via tSystem and pipe the output into a temporary file
3. Open the file with TFileInput* and handle the data
4. Delete the tempFile
Ist there better solution for this? I saw that it is possible to fetch the output of tSystem to a global variable. Can I use this variable as Input for TFileInput*?
Alternatively is it possible to create from the global variable a data stream (convert the (string?)-variable into a hash/list(?) with tPerl for example)?
Volker
Employee

Re: Processing output of "tSystem"

You current solution (reading the produced file with tFileInputDelimited) is fine in my opinion. But as you suggest, there are many ways to do it.
Can I use this variable as Input for TFileInput*?

No, you can't.
Alternatively is it possible to create from the global variable a data stream (convert the (string?)-variable into a hash/list(?) with tPerl for example)?

Yes, I suppose you can create a list from the tSystem output variable. It depends a lot on your Perl skill level and the output variable format.
To read the list, I use the tArrayIn invisible component (make it visible by checking the option in the TOS preferences). My tPerl contains:
our @lines = (
,
,
,
);

My job output is:
Starting job TEMP7 at 09:49 10/07/2007.
.---------------------.
| tLogRow_1 |
+----+----------+-----+
| id | username | age |
+----+----------+-----+
| 1 | plegall | 26 |
| 2 | fbonan | 36 |
| 3 | jgottero | 21 |
'----+----------+-----'
Job TEMP7 ended at 09:49 10/07/2007.
One Star

Re: Processing output of "tSystem"

Is this component (tArrayIn) available in 2.1.0RC1? I cant find any option to activate ist.
I found a different solution myself (thanks for the "perl-tip"): I tried to pipe the output in the open command like the example shown in the picture below.
But it seems that the generated code adds the fixed mode "<" to the open command.
open($input_FH_tFileInputDelimited_1, '<', $desc_tFileInputDelimited_1{filename})

Is this necessary? Or could it be changed to a option (with mapping list in the gui)?
Bye
Volker
One Star

Re: Processing output of "tSystem"

Hi plegall,
I found the option for showing the hidden components in Talend 2.1.0 (without RC). I'll try it next time. Are there any special things to know If I use them? Like lower stability, possibly not available in the next release, and so on?
Concerning my suggestion to allow to parse the output of a system command / program. Should I open a feauture request? Additional it may be possible to take the logic to read a file (or system command) into a independent module and connect them. But this may be a bigger change to Talend.
Example: (tFileInput | tCommandExecute | ... ) -> (tInputXml | tInputPositional | ...)
With this solution I could solve another problem: Sometimes I have to change files before I could use them. For example make same changes to well form a file to use it with tFileXmlInput. Actual I read it with tFileInputPositional, transform it with tPerlRow and write it back with tFileOutputPositional into a temporary file. This file is pocessed by tFileInputXml.
Altogether creating, writing, deleting temporary files is one of my main tasks. Maybe creating a module to handle them, could be another way...
Bye
Volker

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog