Load the data from MongoDB (tMongoDbInput) of recent 15 mints only

[raowaqasakram]

Background :

I have created a job which is reading data from MongoDB and loading that to MS-SQL.

Current Behaviour :

Whenever I run the job it is fetching all the data from MongoDB .

Expected Behaviour :

When the job will run it should fetch only the data of last 15 mints. I have a timestamp field in mongoDB document.

Example

Timestamp :2022-07-29T08:14:14.657+00:00

I am looking forward for such type of solution may be we can add some query in tMongoDBInput to load the rows of recent 15 mints?

How we can apply that?

Or is there any generic solution that if the job run first time then it will extract data of all time and load to SQL.

when the job run next time (let say after 15 mints) then it will automatically assume that these are newly created and will load only new rows?

[gjeremy1617088143]

Hi , if you insert the timestamp in sql server, you can check the max on the target table then query the source table with all the rows wich have timestamp > to your max mssql target table timestamp , so it will load only the new rows created since the last insert on mssql

[raowaqasakram · Accepted solution]

I have published an article on this.

https://blog.devgenius.io/fetch-latest-data-from-mongodb-talend-1f21ba7b98b5