Data integration for Magento to Quick Book

[indrajeet183]

Hi,

I'm currently new to Talend and I'm learning through videos and documentation.
My goal is to integrate Magento and Quick Book using Talend.

Initially my first thought was I will setup direct DB connection for Magento and will take relevant data which I need and will process it and will send to QuickBook using REST API's(specifically bulk API's)

But then again I thought it would be little hectic for me to query Magento database(multiple joins) so I've another option to use Magento's REST API.

But as I'm not much familiar with the tool I'm struggling little, so any help is appreciated.

What I've done currently?

1. I've saved my auth(for QB) and db(Magento) data in file and using tFileInputDelimited and tContextLoad, I'm storing them in context variables so they can be accessible globally.
2. I've successfully configured database connection and input but I've not used metadata for connection(should I use that and if Yes how can I pass dynamic values there?). I've used my context variables data in db connection settings.
3. I've taken relevant fields for now but if I want multiple fields simple query is not enough as Magento stores data in multiple tables for Customer etc but it's not big deal I know but I think it might increase my work.

For now that's what I've built and my next step is send the data to QB using REST while getting access_token and saving it to context variable and again storing the QB reference into Magento DB.

Also I've decided to use QB bulk API's but I'm not sure how I can process data in chunks in Talend(I tried to check multiple resources but no luck) i.e. if the Magento is returning 500 rows I want to process them in chunks of 30 as QB batch max limit is 30, so I will be sending it using REST to QB and as I said I also want to store back QB reference ID in magento(so I can update it later).

Also this all will be on local, then how can I do same in production? how I can maintain development and production environment?

Resources I'm referring 

For REST - https://community.talend.com/t5/How-Tos-and-Best-Practices/Using-OAuth-2-0-with-Talend-to-Access-Goo...

[Anonymous]

To chunk your data into a number of records, this explanation could be of use (https://community.talend.com/t5/Design-and-Development/Generate-n-rows-of-XML-output/m-p/122222/high...). This is about an XML output, but the theory remains the same.

When using a db component to retrieve records, you can use any SQL you like. This can involve multiple tables, WITH clauses, you name it. Be aware that the code is essentially a Java String. So if you want to add dynamic clauses, you will need to use variables and some String manipulation. For example.....

"Select
Table1.Column1,
Table2.Column2,
Table3.Column3
From Table1 JOIN  Table2 ON Table1.Column1 = Table2.Column5
JOIN Table3 ON Table1.column1 = Table3.Column4
WHERE Table1.Column5 = '" + context.MyWhereValue + "'"

If you have used context variables for your db connections, you can simply adjust the context variable values by using a tContextLoad component after a tPreJob component, or you could use the implicit context load.