AWS S3 to Snowflake Bulk load taking time using Talend

Highlighted
Six Stars

AWS S3 to Snowflake Bulk load taking time using Talend

Hello,

  I have designed a job to Load multiple files from AWS S3 to Snowflake table using Bulk Load components.

My Flow is:

1)tPrejob->tS3Connection

2)tS3list->tS3Get->tFileinputdelimited->tDBOutputBulk->tDBBulkExec->tDBROW

3)TPostJob->tS3Close

 

Where:

tDBOutputBulk has storage as "Internal" stage.

tDBROW has "Commit" command

 

There are total 2 files 450MB each on S3(total around 1GB data i.e 20 million records with 6 columns)

To load 1GB data, it is taking 25 min. I want to improve performance of my job.

 

Can anyone help in improving performance?

Also how to handle restartability in case of failure here?

 

thank you.

 

One Star

Re: AWS S3 to Snowflake Bulk load taking time using Talend

For this you can do this:

  1. Create a named file formats that clearly describe your data files. 
  2. Then, create name stage objects.
  3. Load the data of S3 bucket into Snowflake tables
  4. Data files error resolution.

Regards, 

192.168.0.254 192.168.2.1 192.168.1.2  192.168.178.1

Six Stars

Re: AWS S3 to Snowflake Bulk load taking time using Talend

Hello,

  I want to load data into snowflake using Talend Bulk components.

Any performance tips on my existing job design or any modifications?

 

Please let me know

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Talend Cloud Available on Microsoft Azure

An integration platform-as-a-serviceto help enterprises collect, govern, transform, and share data from any data sources

Watch Now

Self-service Talend Migration: Moving from On-Premises to the Cloud

Move from On-Premises to the Cloud by following the advice of experts

Read Now

How to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud reg...

Learn how to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud registries

Blog