Five Stars

Files being deleted Automatically post run

Hi Team,

 

I am facing an issue where my Talend job is deleting the files automatically post run.

The data is transferred to staging tables and post that it is deleting the files from my source.

Attaching here with the screenshot for reference.

 

 

 

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
Ten Stars

Re: Files being deleted Automatically post run

if you are using snowflake you may need to use something like set stage_copy_options = (purge = false)
regards,

Francois Denis

Don't forget to tag when it's "solved"!

7 REPLIES
Fifteen Stars

Re: Files being deleted Automatically post run

You need to give us more information than this I'm afraid. What files are you talking about? What components are using files? Can you show us the whole job because nothing you have shown will use a file.

Rilhia Solutions
Ten Stars

Re: Files being deleted Automatically post run

without a good view of each component we cannot help you.
can you give more detail of each components?

Francois Denis

Don't forget to tag when it's "solved"!

Five Stars

Re: Files being deleted Automatically post run

Hi,

The flow of my Talend job :

 

Connects to Snowflake --> Copies data from file Staging --- > Commit

 

EX: "Copy into abc.def.mytable
from @~/acc/Idds/loc
file_format = (
type= 'CSV'
FIELD_DELIMITER = ','
)
;"

 

Post this run.. Data is being inserted in my staging table. But the file present in that path is being deleted.

 

 

Ten Stars

Re: Files being deleted Automatically post run

if you are using snowflake you may need to use something like set stage_copy_options = (purge = false)
regards,

Francois Denis

Don't forget to tag when it's "solved"!

Five Stars

Re: Files being deleted Automatically post run

Hi Denis,

Thank you so much fro your response.
But I do have similar jobs which run with the same logic and does not cause any issue.
Is there something with Talend which may cause this issue.?
Ten Stars

Re: Files being deleted Automatically post run

on, in your case Talend is just sending sql query to snowflake runtime.

Francois Denis

Don't forget to tag when it's "solved"!

Five Stars

Re: Files being deleted Automatically post run

Thank you so much .. It worked !!