Developing an ETL framework using Talend

One Star

Developing an ETL framework using Talend

Hi All,

 

my first message here.

I do have some DW design and ETL experience, however not with Talend.

 

I'm trying to understand what would be the best way to accomplish the below:

I would like every job to get variables from a master table based on feed_id.

If, for instance, I have a job called 'Web Extracts' which I've assigned feed_id =1 to,

I'd like to have a generic mechanism to get all records where feed_id = 1 from the variables table (i.e variables relevant across a feed only, filename for instance, plus global variables (i.e variables relevant across the whole project, for example, remote server hostname, which can have 999 as feed_id)

This is how the table should look like:

1.PNG

I'm assuming I'd need to create a joblet for this, with different feed_id depending on the job?

Or is there a better ideas?

On each case, could you please provide me with guidance and examples on how to achieve it?

In addition,

would I need to declare these variables for the job to compile, or is a way around doing it?

 

Next,

every time a job runs, a file_id (a sequence generated by an identity column on a db, see bellow)

will be generated in an etl_file_process table.

This table will record start time, end time, records processed, records rejected, and filename processed for every run.

2.PNG

 

Again the question is, what's the best way to go about it, and whether I would have to declare variables beforehand for the job to compile?

 

Thank you all in advance,

avrgal

 

Moderator

Re: Developing an ETL framework using Talend

Hello,

Your job requirement is not very clear for us. The screenshot is a little indistinct to get the logic.

Could you please elaborate your case with an example with input and expected output values?

Best regards

Sabrina

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog