topic Re: Problem with tsalesforceoutput main|reject flow when Commit level >1 in Talend Studio

Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Sat, 16 Nov 2024 09:13:03 GMT

Hello,

I have an issue with tsalesforceoutput in talend 6.4.1.

While using the tsalesforceoutput with a commit level set above 1, the first batch (successfully updated within the salesforce org) isn't logged into the output files.

Commit lvl : 200 | Records updated : 2513

As you can see, 199 records (the first ones to be updated) were lost in the execution, but a quick check in salesforce shows that those records were successfully updated.

Could you point me onto the right direction to resolve this issue, or at least tell me which version of talend can be used without this error ?

Thanks.

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Thu, 12 Oct 2017 04:51:22 GMT

Hi,

Have you tried to clear "Extend Insert" check box to see if rejected rows can be logged into your excel file? on ? Did you get NULL values?

Best regards

Sabrina

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Thu, 12 Oct 2017 09:17:58 GMT

Hi Sabrina,

I've tried it, all the records are succesfully updated and logged in the excel file (no null value) but the time to do so is nearly 9 min, which is way too long.

Best regards,

Nicolas

extendInsertUnchecked.PNG

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Mon, 30 Oct 2017 08:24:08 GMT

Hi,

I've tried to resolve my problem for 15 days, but i am still stuck to the same point, have you found something ?

Thanks,

Nicolas

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Mon, 30 Oct 2017 13:29:41 GMT

Hello,

Try to add a tLogCatcher component to retrieve errors.

You can put the information in a file or sent them by email.

I think that it will help you.

Moreover, you can add a tLog component between the component tSalesforceOutput and the error file.

Best regards,

Marine

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Tue, 31 Oct 2017 07:44:13 GMT

Hi,

Please make sure that you have selected 'Cease on Error' check box to stop the execution of the Job when an error occurs in advanced settings of tSalesforceOutput.

Best regards

Sabrina

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Fri, 08 Dec 2017 15:00:03 GMT

Hi,

@MarineTiphon : I tried your method, it successfully logged all the records onto the excel (which is a good thing), but didn't separate the wrong ones from the good ones. It makes me wonder if the error is only on the Reject row ?

@xdshi: I selected the 'Cease on error' checkbox, it stops the job at the first wrong record, but doesn't tell me at exactly which row the error occurs, although it gives me the error's information

It kinda get closer from what i expect this component to do, but i would like (if possible) to get the import job
to go through the all of my records, then separate into two distincts files all the success / errors.

Best regards

Nicolas

ceaseOnErrorChecked.PNG
errorReturned.PNG
tLogCatcher_CeaseOnErrorUnchecked.PNG

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

TRF — Fri, 08 Dec 2017 18:03:49 GMT

Hi,

@Moe, the following will not solve your case but may help.

It sounds like a bug or at least a strange behaviour from the tSalesforceOutput component or the salesforce API.

I've realized the following tests with 253 records (plus 1 header line) from an input file with 1 expected rejected record (due to blank name).

Here are the results:

"Extend Insert" checked and "Commit level 200" : 54 record into the success flow and 54 into the error flow. Always the same record in this error flow but not the expected record and always the same error message!
252 records inserted into Salesforce as expected.
(200 - 1) + 54 = 253 ==> which corresponds to the number of input records.
"Extend Insert" checked and "Commit level 50" : 204 record into the success flow and 50 into the error flow. Always the same record in this error flow but not the expected record and always the same error message!
(50 - 1) + 204 = 253 ==> which corresponds to the number of input records.
"Extend Insert" checked and "Commit level 1" : 252 record to the success flow and 1 to the error flow (as expected).
Records are inserted 1 by 1 so response time is very bad and API consumption is high.
"Extend Insert" unchecked and "Retrieve Id" checked or not : 252 records to the success flow and 1 to the error flow!
Records are inserted 1 by 1 so response time is very bad and API consumption is high.

My conclusions:

When "Extend Insert" option is ticked, we know records are sended to the API by batches of n records where n depends on the value of "Commit Level" parameter (refer to Salesforce documentation).

With this option, as soon as an error occurs, result for success and reject flows is wrong (except when "Commit Level" = 1) but the operation on Salesforce side is rigth.

My question:

Where is this f**k**g bug?

My advices:

If you care about error messages, avoid options 1 and 2.

If you care about response time (with low data volume), choose option 1.

If you care about error messages and response time, consider Bulk option (should be reserved for high data volume, but sometimes...).

Feel free to complete or correct if necessary.

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

TRF — Sun, 17 Dec 2017 19:14:10 GMT

@Moe, does this help or not?

Please let us know and if it does, don't forget to give Kudo and accept the solution if you have solved your case thanks to the proposed solution.

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Thu, 11 Jan 2018 20:42:50 GMT

Hello,

I also faced this problem. What I noticed is that the header of the source file is not removed before the content is split into batches, but after. Consequently, the first batch of data to be sent to Salesforce will have number of data records = commit level -1. Probably an event is not triggered in this case because neither the main, nor the rejected flow continues. It is more obvious if you execute the job with a file that has less records than the batch size. You will see the status "Starting" on both flows.

By analyzing your 1st example:

"Extend Insert" checked and "Commit level 200" : 54 record into the success flow and 54 into the error flow. Always the same record in this error flow but not the expected record and always the same error message!
252 records inserted into Salesforce as expected.
(200 - 1) + 54 = 253 ==> which corresponds to the number of input records.

Commit level 200

Input file total number of records: 254 = 253 data records + 1 header record;

Batch 1: 200 data records - 1 header records = 199 (below commit level, so the records are not transferred to the success/rejected files);

Batch 2: 54 remaining data records. Probably there is a special behaviour implemented for the last batch since the total number of records will not always be a multiplier of the batch size. However, this behaviour does not apply if the file has ONLY one incomplete batch.

Please try to execute the job with a file that does not have a header and let me know if it works. For me it did.

Have fun,

Florentina

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Tue, 30 Jan 2018 09:31:56 GMT

Hello, first, i'd like to apologize for the time i took to respond, i was away frome the office. Then i'd like to thank you all for your response, although it didn't resolve my issue, it did help me figure out other tests to do.

Debugging :

Input : 2521 records

Commit Level : 200

Cf : debugOver200.png, debugUnder200.png
With this, i tried to understand where exactly the component was bugged. I put a duplicate record into the first batch. Before the row 200, no records go beyond the SalesforceComponent. After, they are put inside the success files the way i expected it to (But starting at the record 200, the 199 first one are lost). The error (always the same error message, the first one encountered) just add up to the size of the batch (200 lines).

Execution without header and without error :

Input : 2521 records

Commit Level : 200

Cf : executionWithoutHeaderAndErrorFree.png

I tried different version without header, like @Florentina said. In this one, we see 2322 rows in the success file.

2521 - 2322 = 199 records lost.

Execution with header and Error in the first batch :

Input : 2521 records

Commit Level : 200

Cf : executionWithHeaderAndErrorFirstBatch.png

With the header on, and a duplicate record in the first batch, we get 200 lines in the rejected (expected : 2), and 2322 in the success.

200+2322=2522. The error message are always the same.

Execution without header and Error in the first batch :

Input : 2521 records

Commit Level : 200

Cf : executionWithoutHeaderAndErrorFirstBatch.png

Same execution, but without header. The result is exactly the same than before.

The log can be seen in logExecutionWithoutHeaderAndErrorFirstBatch.png

Execution without header and Errors in multiple batches :

Input : 2521 records

Commit Level : 200

Cf : executionWithoutHeaderAndErrorsInSameBatchMultipleTimes.png

In this test, i tried to put errors in multiple batches, so i duplicated records in different batches. That's where it gets interesting. For every batch containing flawed records, you get 200 errors messages printed out. No matter how many error a single batch contains, it will always print the first occurence, 200 times. Plus, errors isn't separated from success. In the exemple i provided, you can see 400 errors + 2322 success = 2722 records (?!?). But it is only with 2 different batches flawed. I tried with more errors in more batches, and went up to 1200 records in the errors, and still 2322 in the success. Why ? Absolutely no idea.

Execution with small dataset :

No header, No error :

Input : 179 records

Commit Level : 200

Cf : executionWithoutHeaderWithoutErrorDatasetSmallerThanBatch.png

I tried with smaller dataset to see if i could reproduce the results of @Florentina. No success, No errors, nothing.

No header, Error :

Input : 179 records

Commit Level : 200

Cf:executionWithoutHeaderWithErrorDatasetSmallerThanBatch.png,

executionWithoutHeaderWithErrorDatasetSmallerThanBatchCeaseOnError.png

Same thing than previous one. I tried a cease on error to see if the job saw the error, and it did. But without it, no success, no errors.

@TRF

Your tests were really helpful, and your advices where pretty much the same thing i came up with on my own. The matter is that we need both the logs and the response time, but we can't use bulk and its asynchronous feature.

Let me be clearer. We use those talend jobs in workflows inside salesforce, meaning that we import data, then operate on it. We need a synchrone way to load them, because we need to be sure that all the records will be loaded before the operations on them start.

Thank you for your time

Nico

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

jdope — Sat, 24 Feb 2018 02:01:51 GMT

The output from tSalesforce components is not right, it hasn't been right for a long time...maybe never. Chaning the commit level to 1 to get proper logging is not a solution; This is a bug and honestly a pretty big bug if you are trying to reliably use tSalesforce components for anything serious...eg. it can't accurately / reliably report the outcome of an insert operation?...not good.

I've created a bug: https://jira.talendforge.org/browse/TDI-40145. Not sure what happens from here, but maybe if we get comments, or votes, or whatever to get the bug noticed might be a start to getting it fixed. It seems as though you all have put some indepth work into steps to repro, etc. Maybe you can add your notes to the existing bug?

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Mon, 26 Feb 2018 08:59:21 GMT

Hello @jdope,

I already tried to submit a bug, so far no response. I created it in october, although i didn't really fed it with info, considering the fact that i didn't get any return. However what i did get is an assignee. I provided the URL to your bug, and the URL to this topic, hoping that maybe it will catch some attention. I also started watching your bug, and voted for it.

But for the adding my notes part, i'm not really sure how i am supposed to do it. The tests i did are quite space consuming, and i don't think it can fit inside a comment (which seems to be the only way for me to add something meaningful to your issue).

Any idea ?

Nico

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

jdope — Mon, 26 Feb 2018 14:24:26 GMT

@Moe ok great. Yeah as far as "adding notes" I think a link to this community post is probably good enough. Thanks for cross-referencing the bug I created with your bug. What was that bug #? I'd like to add that as a comment to the one I created.

@xdshi can you elaborate on the process for getting jira items evaluated / worked? Is there something from a community level that we can do (votes, comments, etc) to get the bug's priority elevated and worked?

specifically we are interested in https://jira.talendforge.org/browse/TDI-40145 and the yet to be mentioned bug (we can combine them) from @Moe.

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

Anonymous — Mon, 26 Feb 2018 15:03:25 GMT

@jdope Bug TDI-40156

https://jira.talendforge.org/browse/TDI-40156?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel

Re: Problem with tsalesforceoutput main|reject flow when Commit level >1

jdope — Tue, 27 Feb 2018 14:04:24 GMT

https://jira.talendforge.org/browse/TDI-40145

(these can be merged in JIRA if a JIRA admin JIRA is looking at this)

Anyone else who may be experiencing this issue please login to JIRA and vote up (right side) these JIRA issues!