topic Re: Load the data from MongoDB (tMongoDbInput) of recent 15 mints only in Talend Studio

Load the data from MongoDB (tMongoDbInput) of recent 15 mints only

raowaqasakram — Fri, 15 Nov 2024 22:41:46 GMT

Background :

I have created a job which is reading data from MongoDB and loading that to MS-SQL.

Current Behaviour :

Whenever I run the job it is fetching all the data from MongoDB .

Expected Behaviour :

When the job will run it should fetch only the data of last 15 mints. I have a timestamp field in mongoDB document.

Example

Timestamp :2022-07-29T08:14:14.657+00:00

I am looking forward for such type of solution may be we can add some query in tMongoDBInput to load the rows of recent 15 mints?

How we can apply that?

Or is there any generic solution that if the job run first time then it will extract data of all time and load to SQL.

when the job run next time (let say after 15 mints) then it will automatically assume that these are newly created and will load only new rows?

Re: Load the data from MongoDB (tMongoDbInput) of recent 15 mints only

gjeremy1617088143 — Tue, 02 Aug 2022 07:48:58 GMT

Hi , if you insert the timestamp in sql server, you can check the max on the target table then query the source table with all the rows wich have timestamp > to your max mssql target table timestamp , so it will load only the new rows created since the last insert on mssql

Re: Load the data from MongoDB (tMongoDbInput) of recent 15 mints only

raowaqasakram — Wed, 05 Oct 2022 12:20:49 GMT

I have published an article on this.

https://blog.devgenius.io/fetch-latest-data-from-mongodb-talend-1f21ba7b98b5