When I call the tSystem component and use it in a flow, I think that it probably opens a session with the OS, runs the command, and then closes the session for each row.
Does anyone know if it is possible to keep the session with the OS open so that a process using tSystem might run faster?
I'm afraid you cannot reuse the "connection" from one tSystem call to another.
However, are you able to estimate the cost to renew the "connection" at each call?
For that, you may have a test with an "empty" command for an estimation of the call cost.
You may also try to run the same command with the same parameters from the OS the get the operation net cost whithout any "connection" cost.
Hope this helps.
If your job isn't reliant on anything returned by the tSystem component, then you could simply build a batch/shell script (using a tOutputFileRaw) in your job, outputting the commands you need to execute in each iteration, and then executing this script with a single tSystem at the end of your job.
If you require anything returned by the tSystem at each iteration, then there's probably little that can be done to improve the situation.
Watch the recorded webinar!
Accelerate your data lake projects with an agile approach
Create systems and workflow to manage clean data ingestion and data transformation.
Introduction to Talend Open Studio for Data Integration.