topic Re: Limitation on rows in TOS in Talend Studio

Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 11:24:04 GMT

Is there a limitation on the no of rows in trial version of TOS ? my job is not loading the data after 20000rows.. looks like it is not updating more than that.

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 11:58:54 GMT

No there is a no limitation on the number of rows that Talend will process. I have processed 100s of millions in the Open Source Edition. What are you trying to do? This could be a memory limitation or an issue caused by data.

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 12:05:57 GMT

when you say memory limitation , could you explain more on this.?
i am trying to load data from oracle to ms-sql..

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 12:14:24 GMT

The limit of what your system can handle and your JVM settings have been set to. Look up "Talend JVM settings" on line. Talend do not impose any limits on developers.

Re: Limitation on rows in TOS

TRF — Fri, 16 Feb 2018 12:17:18 GMT

Hi,

Do you have any error message?

Are you running the job from the Studio?

If so, how many rows are counted after tOracleInputComponent (should appear in green on the link after the component)?

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 12:25:24 GMT

can you comment on this job ? this has only 1642 rows , but it still does not complete... Attached screenshot

Screenshot_2.png

Re: Limitation on rows in TOS

TRF — Fri, 16 Feb 2018 12:28:57 GMT

You need to commit your transaction at the end using a tMSSqlCommit component (or use an existing connection which configured with autocommit)

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 12:39:09 GMT

please help me with this error.. in the component list i do not see any option.

Screenshot_3.png

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 12:40:33 GMT

Can you show your Output component's Advanced Settings?

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 12:42:05 GMT

here it is.

Screenshot_4.png

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 12:46:30 GMT

As a test, can you reduce the "Commit every" to 1000 and the "Use Batch Size" to 1000. Then try running it.

Re: Limitation on rows in TOS

TRF — Fri, 16 Feb 2018 12:54:35 GMT

OK, I forgot the option "Commit every"...

Can you share the basic settings too?

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 13:42:43 GMT

After reducing the "Commit every" to 1000 and the "Use Batch Size" to 1000, the data is loaded for one job which has 1642 rows,

the second job which has 52000 rows is still not updated.. I have hard coded it to 1000 as well.. please suggest

Re: Limitation on rows in TOS

TRF — Fri, 16 Feb 2018 13:45:38 GMT

Try to add an explicit tMSSqlCommit at the end.

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 13:49:00 GMT

what do i fill here ? attached screenshot

Screenshot_6.png

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 13:51:49 GMT

This is likely caused by your Input component. Can you show us your job and component config? Do you get an error? Are ANY rows being loaded to your database out of the 52000?

Re: Limitation on rows in TOS

TRF — Fri, 16 Feb 2018 13:55:07 GMT

You need a MS SQL Connection (tMSSQLConnection component to be added at the beginning then to be reused for tMSSQLOutput and tMSSQLCommit components)

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 13:55:21 GMT

No, rows are not loaded to the database. attached screenshot

Screenshot_7.png

Re: Limitation on rows in TOS

TRF — Fri, 16 Feb 2018 14:04:20 GMT

Are you able to use a tLogRow component after the tMSSQLOutput (connected with the Reject flow)?

Re: Limitation on rows in TOS

Anonymous — Fri, 16 Feb 2018 14:08:31 GMT

Ah, I see you are using the Oracle componenst :-).In your Advanced settings of your Input component, tick use cursor.