Really Slow load from Informix Database (version 7.3)

[Anonymous]

Hello!

I would like help with a really big problem that I am facing right now, I have load data from an Informix database but Talend can't load more than 70 rows per second, if I try to load in this same database from Excel or Qlik I can read it at least 8 x faster.

Could someone help me with that?

[Anonymous]

Hello,

I figured the problem could be the time to convert the local database format to Talend format if I take off varchar/string field the load speed increase more than 3 times.

but I can't find a solution to improve that, because of the varchar field is a key in this table.

=/

Someone, could me provide some help, please?

tks!

Best Regards!

[vapukov]

Hi,

there is must be no problem with the conversion but easily could be a problem with data size

reading speed depends from:

• query - how it complicated and how fast DB start returning data
• data size
• network latency
• JDBC driver performance

[Anonymous]

Thank you @vapukov,
But the query is a simple select from, if I use the same query in a odbc connection in Execel or Qlik for example I get a considered better reply.
I have made some test in talend, if I take off the varchar field the velocity increases to 140 rows/sec if I get just 3 fields (integer) I can load 3000 row/sec.
But I need all of the fields, if I consider all I can't pass than 70 rows/sec.

All the test have been made un the same server with the same environment.

[vapukov]

ok, do you really output all 50k+ rows to tlogrow?

is the speed same if output is CSV file?

[Anonymous]

I landing more than 200 k but I just output the last 100 rows.
I didn't test with csv, I am going to test it and I inform you.

[vapukov]

if you print just 100 rows, it is not important

[vapukov]

I afraid there no simple answer, need to check all what I mention above, because simple test with 3 int and 1 string (255 characters):

so, again:

- size (maybe your "string" is 32000 char 🙂 )

- network

- etc

add:

if you are IBM customer, try to download fresh JDBC drivers and test with tJDBCConnection / tJDBCInput

[Anonymous]

Actually this string there is 22 characters, it is a fixed alphanumeric field.
I have to check my jbdc driver, but the wired thing about it is if I retrieve different fields in the same table I have different performances.
I believe in the time when I load the data source formats Talend spend a large time to transform for a Telend data type, I don't know if has a better way to do that.

[vapukov]

my picture example from Informix, with 255 mixed string, so - Talend don't spend time for this