topic Re: Talend/Snowflake issues with autocommit in Talend Studio

Talend/Snowflake issues with autocommit

LMelcher — Fri, 15 Nov 2024 21:52:52 GMT

Hi there,

I'm new to using Talend Studio Version 8.0.1, and I might have missed some things since I'm still learning.

I'm working on a project where I need to read data from a CSV file and map it to a Snowflake database table while using transactions.

At first, I tried using Talend's native Snowflake components, like tSnowflakeInput, tSnowflakeOutput, tSnowflakeConnection, tSnowflakeCommit, and tSnowflakeRollback. Unfortunately, every tSnowflakeOutput committed, even though I tried to turn off auto-commit in different ways, including deactivating it inside Snowflake. I made sure that I matched all components with the correct database connection, too.

To fix this problem, I switched to using JDBC, and everything is working as expected.

Do you have any ideas about what could have caused the issue with the native Snowflake components?

Check out the attached image to see the job working with JDBC.

Thanks!

Re: Talend/Snowflake issues with autocommit

Anonymous — Mon, 24 Apr 2023 03:47:28 GMT

Hello,

Starting from V 7.3.1, we have added new option "commit every" on tSnowflakeConnection, tSnowflakeCommit and tSnowflakeRollBack. You are able to to control transaction logic during reading/writing data to SnowFlake with the help of tSnowflakeCommit, tSnowflakeRollBack components.

Best regards

Sabrina

Re: Talend/Snowflake issues with autocommit

LMelcher — Mon, 24 Apr 2023 11:47:46 GMT

Hello,

thank you for your answer.

Could you please tell me, if the job design I provided should work with tSnowflake components?

Thank you.

Re: Talend/Snowflake issues with autocommit

Anonymous — Tue, 25 Apr 2023 06:25:04 GMT

Hello,

The job design should work with tsnowflakexxx components and tSnowflakeconnection has the option to deselect 'AutoCommit'.

Did you see the log in the snowflake History that it does auto commit to true and therefore, Rollback is not working?

Please try to check "Die on error" option to make it fail to see if there is any error message printed on console?

Best regards

Sabrina

Re: Talend/Snowflake issues with autocommit

LMelcher — Tue, 25 Apr 2023 08:05:03 GMT

Hello,

'AutoCommit' was deselected.

In the snowflake History I could still see a commit coming from Talend.

"Die on error" does not invoke any error messages.

Regards,

Lasse

Re: Talend/Snowflake issues with autocommit

Anonymous — Thu, 27 Apr 2023 06:23:30 GMT

Hello,

Is there any message in your CsvMverror?

"Rollback message: "+((String)globalMap.get("tDBRollback_1_ERROR_MESSAGE")))

Best regards

Sabrina