Employee

error in TIC while using /tmp space

Here is te scenario:

 

I am pulling data out of a cloud DB and then splitting into files (using /tmp dir ) in cloud and then zipping them uploading to AWS S3.

Until the data volume was less everything was working fine , but when i started handling 90+ M records , it started giving error without reasoning .

 

Any idea ?

 

Error:

Unable to calculate MD5 hash: /tmp/5173575443141637840patients8.csv.gz (No such file or directory) 

 

- Unable to calculate MD5 hash: /tmp/5173575443141637840patients9.csv.gz (No such file or directory)
tS3Put_1 - Uploading an object with key:talend-poc/XXXX-XXXX-06302017/5173575443141637840abc9.csv.gz

  • Talend Integration Cloud
Tags (5)
9 REPLIES
Employee

Re: error in TIC while using /tmp space

@PM

 

Perhaps you can provide some guidance to our consultant. Who is running into the issue noted above.  Below are some questions that support has not been able to address.

We will be having the APAC team reach out to gather the job and to record the Webex of the issue for an escalation.  If you could provide any guidance what would help this engagement it would be appreciated.

 

From Smrutiranjan Nayak in case  00079846:

Do you mean even if the real volume of data handled in one iteration is lets say 20GB to 60GB , we can still handle this by specifying multi part size to say 3 GB ?
Also I have this question on /tmp space of TIC cloud.

1. Is this the only space allowed for any staging activity?
2. For how long this /tmp will hold the files? What is the fluch out duration?
3. Can I use tFileList on /tmp if needed for any reason and will it read?


Also I have noticed that a smaller volume of data which succeeded in local run (in windows) has been failing in TIC showing same error I reported, that's the reason I have never gone to volume level test.

I will also do a quick experiment with multipart and report you back , if that is making any difference.

Volume that this job need to handle can range from 90M records to 14000 M records. The volume is huge.

Employee

Re: error in TIC while using /tmp space

Hi Tim,

 

first, you right that we provide about 200GB of temp space in a cloud engine. This temp space is shared between all jobs that will run on this engine during its lifetime.

 

Lifetime means: at a minimum 1 hour. If no second or 3rd job gets executed within the hour the cloud engine is shut down. New cloud engines have a complete blank state.

 

This means by implication that each job needs to do careful resource management (file storage) of the the (temp) files it creates. This is the main reason that we have introduced a context parameter resource_flow_temp_folder, which helps to automatically clean up space once a job finished. Refer to How to define temporary folder parameters in our help for more details.

 

To answer the questions:

1. Is this the only space allowed for any staging activity?

Yes, resource_flow_temp_folder is the only space you can and should use


2. For how long this /tmp will hold the files? What is the flush out duration?

You can only safely assume that these files are kept for the lifetime of a single job execution. You may be "lucky" that they survive longer in case the cloud engine doesn't get shut down. See the lifecycle discussion above.


3. Can I use tFileList on /tmp if needed for any reason and will it read?

You can but see 2.

 

HTH

Thomas Steinborn
Sr. Director Product Management
Employee

Re: error in TIC while using /tmp space

Hi Thomas.

That's a great piece of information.

 

I have some quick qs on /tmp area now :

 

1. if together multiple jobs running which is using /tmp space and if exceed 200GB at given moment in whether same job exec or different job exec will not work beyond 200GB usage?

2. If the job complicacy allows the job to run beyond 1hr , due to the restriction you mentioned - will it pose a RISK ?

3. Unable to calculate MD5 hash : when normally you get this error in /tmp TIC ? I keep getting this error in this scenario.

 

Regards

Nayak

 

Employee

Re: error in TIC while using /tmp space

1. if together multiple jobs running which is using /tmp space and if exceed 200GB at given moment in whether same job exec or different job exec will not work beyond 200GB usage?

 

That's correct. The limit is for all jobs running concurrently. E.g. if you have 3 running at the same time they need to share those 200GB

 

2. If the job complicacy allows the job to run beyond 1hr , due to the restriction you mentioned - will it pose a RISK ?

 

That is not a problem as the 1h applies to idle times only. If the engine is in use it doesn't get shut down.

 

3. Unable to calculate MD5 hash : when normally you get this error in /tmp TIC ? I keep getting this error in this scenario.

 

Most likely just a red herring as you seem to run out of disk space during gzip.

Thomas Steinborn
Sr. Director Product Management
Employee

Re: error in TIC while using /tmp space

@tsteinborn

The customer that was working with PS has created a new case as 00079326 and is stating that the 200GB limit is a show stopper for them.

 

Here is the use case from the customer:

 

We are facing issues with the anarchical process again and this time it's due to disk space issue in TIC server. 
We are currently unarchiving 25 GB file which would turn up to be ~140 GB while performing data cleansing on unarchived file,
The error occurred during moving this to archive as it will write back another ~140 GB data post cleansing. What is the maximum temp space allocated in TIC? I suspect this to be the root cause for the failure.

 

Is there anything that can be advised for this use case?  My only thought would be an RE but that may not be desirable.

 

Thank you.

Employee

Re: error in TIC while using /tmp space

@ttaylor . Can you loop me in the support ticket reply stating Nayak is already working on setting up remote engine instead of usng cloud engine.

The restriction on /tmp is however is 200GB on single job run instance.

However the archival issue is different and  have a ticket already running for it which you can refer to customer and close the customer ticket.

 

ticket : 00079928 

 

I don't think this has anything to do with /tmp space as customer mentioned 140GB which is still accommodated in /tmp space.

 

Please go throught the ticket I mentioned for back ground and I raised it on behalf of customer only (which they might not be aware of).

 

Regards

Nayak

Employee

Re: error in TIC while using /tmp space

Looks like in case 00079846 escalations suggested a RE but the customer did not respond if that was an acceptable solution.

Employee

Re: error in TIC while using /tmp space

my ticket is old than this new ticket I guess. You can always give my reference and infact I spoke to customer yesterday over phone and again from 4th Sep I am engaging with them.

However for research , I will upoad the job exported into ticket.

Employee

Re: error in TIC while using /tmp space

Thank you for the context.... we will close out case 00079326 and let the customer know that PS is working this with them.

Thank you.