Files being deleted Automatically post run

Five Stars

Files being deleted Automatically post run

Hi Team,

 

I am facing an issue where my Talend job is deleting the files automatically post run.

The data is transferred to staging tables and post that it is deleting the files from my source.

Attaching here with the screenshot for reference.

 

 

 

 

 

 


Accepted Solutions
Twelve Stars

Re: Files being deleted Automatically post run

if you are using snowflake you may need to use something like set stage_copy_options = (purge = false)
regards,

Francois Denis

Tag as "solved" for others! Kudos to thanks!

View solution in original post


All Replies
Community Manager

Re: Files being deleted Automatically post run

You need to give us more information than this I'm afraid. What files are you talking about? What components are using files? Can you show us the whole job because nothing you have shown will use a file.

Twelve Stars

Re: Files being deleted Automatically post run

without a good view of each component we cannot help you.
can you give more detail of each components?

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: Files being deleted Automatically post run

Hi,

The flow of my Talend job :

 

Connects to Snowflake --> Copies data from file Staging --- > Commit

 

EX: "Copy into abc.def.mytable
from @~/acc/Idds/loc
file_format = (
type= 'CSV'
FIELD_DELIMITER = ','
)
;"

 

Post this run.. Data is being inserted in my staging table. But the file present in that path is being deleted.

 

 

Twelve Stars

Re: Files being deleted Automatically post run

if you are using snowflake you may need to use something like set stage_copy_options = (purge = false)
regards,

Francois Denis

Tag as "solved" for others! Kudos to thanks!

View solution in original post

Five Stars

Re: Files being deleted Automatically post run

Hi Denis,

Thank you so much fro your response.
But I do have similar jobs which run with the same logic and does not cause any issue.
Is there something with Talend which may cause this issue.?
Twelve Stars

Re: Files being deleted Automatically post run

on, in your case Talend is just sending sql query to snowflake runtime.

Francois Denis

Tag as "solved" for others! Kudos to thanks!

Five Stars

Re: Files being deleted Automatically post run

Thank you so much .. It worked !!

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog