Talend Remote Engines houekeeping

Four Stars

Talend Remote Engines houekeeping

So it would appear that the Remote Engines do not out of the box clean-up after themselves leaving large amounts of data in <install_dir>/TalendJobServersFiles/repository/, including the temp directory.

There are 2 major problems with this:

1) If your jobs stage data to disk you rapidly run out of space.

2) Data, possibly including sensitive data, is left "at rest" with all the GDPR, etc. implications of this.

 

So is there a switch or setting that I can change to get the Remote Engine to properly clean up after itself or will I have to work out a way to do it using CRON, etc?

 

 

Moderator

Re: Talend Remote Engines houekeeping

Hello,

Do you want to delete the files in tmp folder (TalendRemoteEngine\data\tmp) without affecting the operation of Talend Integration?

We can delete the content when the Remote Engine is ON but we would better suggest to stop the Remote Engine, DELETE the content in tmp folder and Restart the Remote Engine.

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: Talend Remote Engines houekeeping

Is there a way to delete the temp files automatically after the job finishes? like a cleanup step post job completion?

Four Stars

Re: Talend Remote Engines houekeeping

No I ideally want to delete the content of /TalendJobServersFiles/repository/<job_name and version>/<artefact Name>/temp automatically when a job ends or is terminated (ideally the behaviour should be switchable).

It's all very well having a tPostJob procedure that cleans up the files, but this won't run if the job crashes or is terminated leading to not only running out of disc space but also data protection compliance issues.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Talend Cloud Developer Series – Updating Context Variables

This video will show you how to add context parameters to a job in Talend Cloud

Watch Now

Talend Cloud Developer Series – Deploying First Job to Cloud

This video will show you how to run a job in Studio and then publish that job to Talend Cloud

Watch Now

Talend Cloud Developer Series – Fetching Studio License

This video will help someone new to using Talend Studio get started by connecting to Talend Cloud and fetching the Studio License

Watch Now