topic Re: java.sql.SQLException: Column not found: rows_loaded in Talend Studio

java.sql.SQLException: Column not found: rows_loaded

Anonymous — Sat, 16 Nov 2024 05:04:47 GMT

Hello,

I am trying to perform Bulk load from Snowflake External stage to Snowflake table.

Snowflake external stage refers to files from S3 bucket.

Attaching the Talend Flow with component properties.

Note: "S3_BUCK_STAGE" is external stage created.

While running the code through talend Studio, i am getting "Column not found" error. Screenshot is attached.

More information about code:

1)S3 file structure and snowflake table structure and its column names are same.

2)CSV file is staged from S3 in snowflake

3)All the column data types in table are String type

Could you please help me with the issue.

Please let me know for any more details.

Thank you

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Wed, 07 Aug 2019 04:44:42 GMT

Hi,

Could you please double check whether all the column names have been given with Capital Letters?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Wed, 07 Aug 2019 06:49:00 GMT

Hello,

Yes the column names are in Capital letters both in File and Table.

The order of the columns are also same.

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Wed, 07 Aug 2019 15:32:02 GMT

Hi,

Could you please try to load with default copy command options and see whether its working? It could be because the separator symbol from file is not getting recognized.

Could you please also try to load some dummy data using tDBOutput component just to make sure that target column details in the schema are correct?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Wed, 07 Aug 2019 20:38:39 GMT

Hello,

Thank you for your inputs.

I tried doing a one to one load(50 rows) from S3 file to Snowflake table and it worked fine without any errors.

Regarding Bulk load using talend, i observed the COPY command executed in snowflake history as:

"copy into "WORLD_DATE_20190808_005100_716" from '@~/@s3_buck_stage/' ON_ERROR='continue' FILE_FORMAT=(type=csv field_delimiter=',' compression=gzip)"

My requirement is to load data from External stage(Which points to AWS S3 bucket) created in Snowflake to snowflake table.

But Storage option in tDBBULKEXEC component has "Internal" option only.

How can we load data from external stage(AWS S3 files staged) to snowflake table?

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Wed, 07 Aug 2019 20:45:02 GMT

Hello,

Thank you for your inputs.

I tried doing a one to one load(50 rows) from S3 file to Snowflake table and it worked fine without any errors.

Regarding Bulk load using talend, i observed the COPY command executed in snowflake history as:

"copy into "WORLD_DATE_20190808_005100_716" from '@~/@s3_buck_stage/' ON_ERROR='continue' FILE_FORMAT=(type=csv field_delimiter=',' compression=gzip)"

My requirement is to load data from External stage(Which points to AWS S3 bucket) created in Snowflake to snowflake table.

But Storage option in tDBBULKEXEC component has "Internal" option only.

How can we bulk load data from external stage(AWS S3 files staged) to snowflake table?

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Wed, 07 Aug 2019 21:46:05 GMT

Hi,

Could you please try below option?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Fri, 09 Aug 2019 12:19:22 GMT

Hello,

Let me try and get back to you the queries Snowflake fires in background.

Thanks

Regards,

Rohit

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Fri, 09 Aug 2019 14:44:59 GMT

Hi Rohit,

Once you get the details, please update the forum so that it will help others in our Talend community.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Sat, 10 Aug 2019 09:48:00 GMT

Hi Nikhil,

Definitely.

Regards,

Rohit

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Tue, 27 Aug 2019 07:57:20 GMT

Hello,

I think there is no option to load data from External Stage created in Snowflake which points to S3.

What i tried is using Outputbulk and bulkexec components where S3 data is staged into Internal Snowflake stage using OutputBulk component and then loaded in Snowflake table using BulkExec component.

This solution worked fine but is taking time to load data.

Now i am trying to performance tune the existing design.

thank you for your help

Re: java.sql.SQLException: Column not found: rows_loaded

Anonymous — Wed, 27 Nov 2019 09:17:10 GMT

Hi Rohit,

Congratulations, you seems to have cracked the reason for the rows_loaded column not found issue. Please manage to share the root cause. Does changing it staging to S3 resolves the issue or it is something else.

Thanks,

Shiv