I'm currently new to Talend and I'm learning through videos and documentation.
My goal is to integrate Magento and Quick Book using Talend.
Initially my first thought was I will setup direct DB connection for Magento and will take relevant data which I need and will process it and will send to QuickBook using REST API's(specifically bulk API's)
But then again I thought it would be little hectic for me to query Magento database(multiple joins) so I've another option to use Magento's REST API.
But as I'm not much familiar with the tool I'm struggling little, so any help is appreciated.
What I've done currently?
For now that's what I've built and my next step is send the data to QB using REST while getting access_token and saving it to context variable and again storing the QB reference into Magento DB.
Also I've decided to use QB bulk API's but I'm not sure how I can process data in chunks in Talend(I tried to check multiple resources but no luck) i.e. if the Magento is returning 500 rows I want to process them in chunks of 30 as QB batch max limit is 30, so I will be sending it using REST to QB and as I said I also want to store back QB reference ID in magento(so I can update it later).
Also this all will be on local, then how can I do same in production? how I can maintain development and production environment?
Resources I'm referring
To chunk your data into a number of records, this explanation could be of use (https://community.talend.com/t5/Design-and-Development/Generate-n-rows-of-XML-output/m-p/122222/high...). This is about an XML output, but the theory remains the same.
When using a db component to retrieve records, you can use any SQL you like. This can involve multiple tables, WITH clauses, you name it. Be aware that the code is essentially a Java String. So if you want to add dynamic clauses, you will need to use variables and some String manipulation. For example.....
"Select Table1.Column1, Table2.Column2, Table3.Column3 From Table1 JOIN Table2 ON Table1.Column1 = Table2.Column5 JOIN Table3 ON Table1.column1 = Table3.Column4 WHERE Table1.Column5 = '" + context.MyWhereValue + "'"
If you have used context variables for your db connections, you can simply adjust the context variable values by using a tContextLoad component after a tPreJob component, or you could use the implicit context load.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema
Pick up some tips and tricks with Context Variables