topic Re: To insert records from file to table in Talend Studio

To insert records from file to table

Soumya_M — Fri, 15 Nov 2024 22:29:55 GMT

Q. A job is already in place, input data - transformations - output data. I have a new file with some records, the moment tFileExists component detects the file in the given location, the file should be transformed into a table, all the records must go through the same transformation and give the output. How can i achieve this ?

Re: To insert records from file to table

Anonymous — Thu, 06 Oct 2022 10:10:32 GMT

The easiest way would be to convert the file's data into the format of your existing job's input data (and to the same source), then use the existing job.

Re: To insert records from file to table

Soumya_M — Thu, 06 Oct 2022 11:11:57 GMT

Can you elaborate it please?

I need to automate the job.

Re: To insert records from file to table

Anonymous — Thu, 06 Oct 2022 14:08:23 GMT

It is a little hard to elaborate as each of the decisions on how to do this depend on several factors. For example, what products are you using? Are you using Talend Open Studio or a subscription version? Do you want this to run in a batch way (a job that is started and stopped) or do you want this to be triggered by the presence of a file? Those are just a few of the questions.

If you want this to be triggered by the presence of a file, then you will probably be best served by using a route with the ESB product. If you are happy to have this process scheduled to run (maybe every hour or even every minute or 5 minutes), this can be done with a job (and would be a little easier). To reuse your existing job, you will need to know how to get your new data picked up by that job. Now, that will depend upon where your existing job picks its data up from.

Re: To insert records from file to table

Soumya_M — Fri, 07 Oct 2022 06:06:04 GMT

Okay, I'm using Talend Open Studio for Data Integration 8.0.1

The job has to be triggered by the presence of a file, it should automatically give the output without minimal or no user interaction.

I have given certain location in local system [in tFileExists component], from where it picks up the file.

Re: To insert records from file to table

Anonymous — Fri, 07 Oct 2022 10:52:49 GMT

OK, you have two choices. You either schedule the job to run every few minutes or hours (depending on the frequency of the file delivery) or you use the ESB product and build a route to continually monitor the folder to pick up the files as they arrive.