java.sql.SQLException: Column not found: rows_loaded

Highlighted
Six Stars

java.sql.SQLException: Column not found: rows_loaded

Hello,

  I am trying to perform Bulk load from Snowflake External stage to Snowflake table.

Snowflake external stage refers to files from S3 bucket.

Attaching the Talend Flow with component properties.

Note: "S3_BUCK_STAGE" is external stage created.

While running the code through talend Studio, i am getting "Column not found" error. Screenshot is attached.

More information about code:

1)S3 file structure and snowflake table structure and its column names are same.

2)CSV file is staged from S3 in snowflake

3)All the column data types in table are String type

 

Could you please help me with the issue.

Please let me know for any more details.

 

Thank you

 

 

Employee

Re: java.sql.SQLException: Column not found: rows_loaded

Hi,

 

    Could you please double check whether all the column names have been given with Capital Letters?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Six Stars

Re: java.sql.SQLException: Column not found: rows_loaded

Hello,

  Yes the column names are in Capital letters both in File and Table.

The order of the columns are also same.

 

 

Employee

Re: java.sql.SQLException: Column not found: rows_loaded

Hi,

 

    Could you please try to load with default copy command options and see whether its working? It could be because the separator symbol from file is not getting recognized.

 

      Could you please also try to load some dummy data using tDBOutput component just to make sure that target column details in the schema are correct?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

 

Six Stars

Re: java.sql.SQLException: Column not found: rows_loaded

Hello,

  Thank you for your inputs.

I tried doing a one to one load(50 rows) from S3 file to Snowflake table and it worked fine without any errors.

 

Regarding Bulk load using talend, i observed the COPY command executed in snowflake history as:

"copy into "WORLD_DATE_20190808_005100_716" from '@~/@s3_buck_stage/' ON_ERROR='continue' FILE_FORMAT=(type=csv field_delimiter=',' compression=gzip)"

 

My requirement is to load data from External stage(Which points to AWS S3 bucket) created in Snowflake to snowflake table.

But Storage option in tDBBULKEXEC component has  "Internal" option only. 

 

How can we load data from external stage(AWS S3 files staged) to snowflake table?

 

Six Stars

Re: java.sql.SQLException: Column not found: rows_loaded

Hello,

  Thank you for your inputs.

I tried doing a one to one load(50 rows) from S3 file to Snowflake table and it worked fine without any errors.

 

Regarding Bulk load using talend, i observed the COPY command executed in snowflake history as:

"copy into "WORLD_DATE_20190808_005100_716" from '@~/@s3_buck_stage/' ON_ERROR='continue' FILE_FORMAT=(type=csv field_delimiter=',' compression=gzip)"

 

My requirement is to load data from External stage(Which points to AWS S3 bucket) created in Snowflake to snowflake table.

But Storage option in tDBBULKEXEC component has  "Internal" option only. 

 

How can we bulk load data from external stage(AWS S3 files staged) to snowflake table?

 

Employee

Re: java.sql.SQLException: Column not found: rows_loaded

Hi,

 

    Could you please try below option?

image.png

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Six Stars

Re: java.sql.SQLException: Column not found: rows_loaded

Hello, 

  Let me try and get back to you the queries Snowflake fires in background.

 

Thanks

 

Regards,

Rohit

Employee

Re: java.sql.SQLException: Column not found: rows_loaded

Hi Rohit,

 

      Once you get the details, please update the forum so that it will help others in our Talend community.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Six Stars

Re: java.sql.SQLException: Column not found: rows_loaded

Hi Nikhil,

  Definitely. 

 

Regards,

Rohit

Six Stars

Re: java.sql.SQLException: Column not found: rows_loaded

Hello,

 I think there is no option to load data from External Stage created in Snowflake which points to S3.

What i tried is using Outputbulk and bulkexec components where S3 data is staged into Internal Snowflake stage  using OutputBulk component and then loaded in Snowflake table using BulkExec component.

This solution worked fine but is taking time to load data.

 

Now i am trying to performance tune the existing design.

 

thank you for your help

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

Talend API Services Publish to Talend Cloud

Learn how to publish your API Services to Talend Cloud

Watch Now

Talend Studio Improvements for API Services

Taken a look at the Talend Studio improvements for API Services

Watch Now