architecture of TOS

One Star

architecture of TOS

Following are the questions associated with TALEND Architecture
1) You are saying that TALEND supports distributed architectute,is there any client server concept
I am developing jobs in talend in one machine it acts as client as well as server
but my requirement is that ,i developed project which should reside in server and accessed by many
user through client side.
2) You are also saying that TALEND supports grid computing .So how can i implement grid computing in
TALEND Is there any suggested or recommended software for implementing this grid computing
3) Can you explain in brief the diagram which you have given in link below
http://www.talend.com/solutions-data/architecture.htm
Tags (1)
One Star

Re: architecture of TOS

Hi Alankar!
Unlike most of traditional ETL, Talend is a code generator. With this approach, Talend generate autonomous data integration jobs. You can export your program (right clicking on your job in the repository - since V1.1) and put your generated program on any server. To run those jobs, you just need a Perl interpreter or a JRE (depending on the version you choose) but Talend Open Studio is not needed any more.
So right now you can run Talend jobs on as many servers as you want at the same time (this is not possible with traditional engine based ETL).
We?re currently working hard on a fully visual job distribution tool.
Stay tuned!
Regards,
Fabrice
One Star

Re: architecture of TOS

hi
please can you tell me how i can execute a job without TOS in server that had been develop at client side and export to server?
Employee

Re: architecture of TOS

please can you tell me how i can execute a job without TOS in server that had been develop at client side and export to server?

With TOS 1.1.1, in the repository view > Job designs, right click on any job and choose "Export Job Scripts". It will create a zip archive containing all what you need to run the job independantly.
One Star

Re: architecture of TOS

hi
can you tell
how this is possible that my job use perl in some t_map for some operation and this job is execute by only jvm ?
can you put some light on this and also expalin TOS architecture regarding grid computing?
thanks & regards
alankar
One Star

Re: architecture of TOS

how this is possible that my job use perl in some t_map for some operation and this job is execute by only jvm

In fact, we said that designed jobs are autonomous, they don't need Talend Open Studio (neither do they need to have the JVM) tu run but they still need a Perl Interpreter!
Regards
One Star

Re: architecture of TOS

Hello,
I have a question about architecture.
I use to work with two integration tools, ETI and Sunopsis. I also worked with DataStage in the past.
ETI is a code generator, and it generates code in the selected technology that you're working. To do that, you need to buy the correct DSL, or Data System Library. So, with the correct DSL, you can generate code in C, Cobol, Adabas, Teradata, Oracle, or whatever. ETI don't need to be installed or even know what's happening. It's a very fine tool.
Usage example:
I worked with ETI in variuous projects, but one of them is very interesting. It was a Datawarehouse project for a big Telco in Brazil, and everything was done with Flat Files. The whole project was with flat files, everything. We even customized the ETI to generate SyncSort, to do some operations faster. At the end, the Models (still in flat files) where loaded to DB2 in bulk load.
Sources: Flat files from SAP, Flat files from Microsiga, and ADABAS.
Targets: Flat files, and DB2 just for the final load.


Sunopsis is a SQL-Generator tool. It runs from Db to Db. Sunopsis can work with files also, but it needs a database to make the transformations. Sunopsis can be installed anywhere with a JVM, and connects to any database that haves a jdbc-4. Also a very fine tool.
Usage example:
I also worked with Sunopsis in various projects, even in Telefonica Empresas, here in Brazil. But on project was very interesting, and it's for a federal department. We needed to extract information in real time from various databases in different regions from Brazil, and different cities, and consolidate all of them in a single one. Everything was done in database to database.
Sources: Oracle, Ingres, Caché
Target: Oracle
Both of them are (like Talend), distributed, but each one in his special way. ETI generates the code in the selected language, and them you can send the codes to wherever it should run. Sunopsis installs "Agents" (which are unlimeted free) that can execute the SQL wherever it should execute.
How about Talend? How does it works?
It generates Perl codes? How talend would work in the projects I mentioned?
Where do I find detailed (very detailed) information about Talend architecture, if there is any?
Thanks in advance!
Employee

Re: architecture of TOS

Hello lfilipe
I have a question about architecture.
I use to work with two integration tools, ETI and Sunopsis. I also worked with DataStage in the past.

We work like ETI. Talend is a code generator.
Sunopsis needs a database to work, for example an Oracle Database. With Sunopsis, all your jobs must be perform by the Oracle Database. So, you must buy more and more CPU (and licences) because you need more and more powerfull (The flow of data absorbed by Corporate IT systems doubles every 12/18 months).
With Talend, your job can be deploy on a server, somewhere on your SI. You can reuse available servers to increase your calculating power!
How about Talend? How does it works?
It generates Perl codes? How talend would work in the projects I mentioned?

Talend generates Perl code (release TOS 1.0 or 1.1) or Java code (next release... it's only on the Subversion for the moment Smiley Wink ).
In your project, Talend doesn't need to load all your files to perform a job. TOS can access directly to each files and doing transformation to create output file. At the end, TOS can load the final file on your database.
To finish, see on attachment an architecture picture about TOS. This slide comes from the training slideshows.
One Star

Re: architecture of TOS

Thank you for the explanation.
It seems that to get all the information i want, I must do the training.
How can I do the training, here in Brazil?
Employee

Re: architecture of TOS

Thank you for the explanation.
It seems that to get all the information i want, I must do the training.
How can I do the training, here in Brazil?

It's right that a training should give you a lot of information about Talend : It's the goal of a training!
If you are on Brazil, we can use elearning applications to perform training. I will send you an email for more information.
One Star

Re: architecture of TOS

A nice feature of Sunopsis (now : Oracle Data Integrator) is the way connection to the physical schemas (databases) is handled :
Interfaces (programs generated from Sunopsis) connect to "logical schemas" (such as "Brazil Accounting Software", "Sao Paulo financial consolidation"...), that are bound lately, at run-time, to physical schemas (e.g. myserver.mybase). This binding is managed in a central "topology database".
I find it handy in failover scenarios, because when my prod server crashes (happens, sometimes :-)), I can redirect all connections to the failover server, simply by changing the topology database. Then all sunopsis packages automatically switch to this physical schema upon next run, without deploying anything.
In Talend, I suspect that connection strings are hardcoded into the generated program. Is it true ?
If yes, my failover scenario is getting a bit fat : we need to re-deploy all programs after changing the repository.
Would it be possible to build the connection strings dynamically at run-time, to query an ad-hoc "topology database" of our own (aiming at supporting this failover scenario).
One Star

Re: architecture of TOS

In Talend, I suspect that connection strings are hardcoded into the generated program. Is it true ?

No, in fact, you can use varaible for any parameter of any component.
For example, you can set variable for hostname, login, passwd... of you tXXXInput component.
You will be able to change the value of these parameters after (at run time).
Try-it pressing F5 on a field and take a look at the Context panel - on the bottom of T.O.S. - (see Doc also ;-)
You can create as many Context as you want (Test, Pre Prod, Prod... or in your case Server1, Server2...)
With this feature, you should be able to do what your are looking for.
By the way, I was thinking about some tVirtual (Input Output) components. We have to define it a little bit more but this is a definitly a good idea!
Stay tuned!
Regards,
Fabrice