Files being deleted Automatically post run

Five Stars

Files being deleted Automatically post run

Hi Team,

 

I am facing an issue where my Talend job is deleting the files automatically post run.

The data is transferred to staging tables and post that it is deleting the files from my source.

Attaching here with the screenshot for reference.

 

 

 

 

 

 


Accepted Solutions
Eleven Stars

Re: Files being deleted Automatically post run

if you are using snowflake you may need to use something like set stage_copy_options = (purge = false)
regards,

Francois Denis

Tag as "solved" for others! Kudos to thanks!


All Replies
Sixteen Stars

Re: Files being deleted Automatically post run

You need to give us more information than this I'm afraid. What files are you talking about? What components are using files? Can you show us the whole job because nothing you have shown will use a file.

Eleven Stars

Re: Files being deleted Automatically post run

without a good view of each component we cannot help you.
can you give more detail of each components?

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: Files being deleted Automatically post run

Hi,

The flow of my Talend job :

 

Connects to Snowflake --> Copies data from file Staging --- > Commit

 

EX: "Copy into abc.def.mytable
from @~/acc/Idds/loc
file_format = (
type= 'CSV'
FIELD_DELIMITER = ','
)
;"

 

Post this run.. Data is being inserted in my staging table. But the file present in that path is being deleted.

 

 

Eleven Stars

Re: Files being deleted Automatically post run

if you are using snowflake you may need to use something like set stage_copy_options = (purge = false)
regards,

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: Files being deleted Automatically post run

Hi Denis,

Thank you so much fro your response.
But I do have similar jobs which run with the same logic and does not cause any issue.
Is there something with Talend which may cause this issue.?
Eleven Stars

Re: Files being deleted Automatically post run

on, in your case Talend is just sending sql query to snowflake runtime.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: Files being deleted Automatically post run

Thank you so much .. It worked !!