Retrieve the number of rows at a time T

[stephbzr]

Hello,

I am looking to implement something very useful in my Talend job. I was wondering if it was possible and I'm open to any ideas.

My job is classic, it inserts data in a database. I would like to get the number of rows inserted at a given time, and iteratively update the fields of a table (as the insertion progresses). 

For example, let's say I have 10,000 rows to insert and it takes 10 minutes. I want to be able to select a field in my table at any time between 0 and 10 minutes and know how many rows are already inserted. 

Thanks

[Anonymous]

You could build something to do this, but you would need to decide the minimum time units that you would want the results to be iterated over, you'd also need to decide whether you are talking about rows processed and sent to the DB or whether they are committed (this may result in a variation of your design depending on how you are committing to the DB). Doing this would almost certainly have a negative impact on your job's performance though.

A slightly easier way of achieving this (again, depending on how regularly the data is committed if you want to look at these results while the job is running) would be to add an "insertion_date" column to your DB table. Then, for every row, calculate the current time just before it is sent to the DB component. This won't tell you when the data is actually loaded (it will be just prior to that), but it will give you an idea of the performance over a large number of rows. You would then be able to query this column giving any "from" and "to" time periods to calculate the number of rows.

For your example, you mentioned 10,000 rows over 10 minutes. This seems very slow. Is that what you are getting or was that just an arbitrary example?