High level project design, data studio, managing complexity

Six Stars

High level project design, data studio, managing complexity

I'm trying to integrate 10 disparate sources of CSV data into a database, which could be a source for an integration.

The problem is that the source data will need cleansing, and in some cases standardization.

I know how to do each task on its own, but the complexity is becoming a concern. How would u break up the jobs? How would you have it come back together?

Employee

Re: High level project design, data studio, managing complexity

Hi,

 

    Why don't you create each functionality in multiple sub jobs and call them from a parent job using tRunJob Component?

 

    In this way, the individual functionalities will be preserved in each job but still you can maintain the overall flow.

 

https://help.talend.com/reader/jomWd_GKqAmTZviwG_oxHQ/ICv6hRE2pgpUtQFvQDmxkg

 

    Please refer the above link for tRunJob properties along with sample scenarios. This detail will guide you to resolve your query.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: High level project design, data studio, managing complexity

That is a really good idea for once I get to how to execute the jobs in the project, but I'm at the design phase.   How would you organize this so I get some degree of reuse for common stuff between jobs?  How would you make this maintainable?

Employee

Re: High level project design, data studio, managing complexity

Hi,

 

    What I usually do is to mimic the SoA type of Architecture. This means I would take an epic and divide them to standalone jobs which can cover a full service. Within each service, you can add multiple subjobs which will each subject area of a service.

 

   A simple example of a Job is Customer Creation and sub jobs which needs to be identified is Addess Loading, Emil Loading, Telephone Loading, Adding Foreign keys of these tables back to Customer table etc.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog