Developing an ETL framework using Talend

avrgal — Sat, 20 Oct 2018 16:46:08 GMT

Hi All,

my first message here.

I do have some DW design and ETL experience, however not with Talend.

I'm trying to understand what would be the best way to accomplish the below:

I would like every job to get variables from a master table based on feed_id.

If, for instance, I have a job called 'Web Extracts' which I've assigned feed_id =1 to,

I'd like to have a generic mechanism to get all records where feed_id = 1 from the variables table (i.e variables relevant across a feed only, filename for instance, plus global variables (i.e variables relevant across the whole project, for example, remote server hostname, which can have 999 as feed_id)

This is how the table should look like:

I'm assuming I'd need to create a joblet for this, with different feed_id depending on the job?

Or is there a better ideas?

On each case, could you please provide me with guidance and examples on how to achieve it?

In addition,

would I need to declare these variables for the job to compile, or is a way around doing it?

Next,

every time a job runs, a file_id (a sequence generated by an identity column on a db, see bellow)

will be generated in an etl_file_process table.

This table will record start time, end time, records processed, records rejected, and filename processed for every run.

Again the question is, what's the best way to go about it, and whether I would have to declare variables beforehand for the job to compile?

Thank you all in advance,

avrgal

Re: Developing an ETL framework using Talend

Anonymous — Mon, 05 Nov 2018 08:20:47 GMT

Hello,

Your job requirement is not very clear for us. The screenshot is a little indistinct to get the logic.

Could you please elaborate your case with an example with input and expected output values?

Best regards

Sabrina

topic Developing an ETL framework using Talend in Talend Studio

Developing an ETL framework using Talend

Re: Developing an ETL framework using Talend