I'm new to the Talend Data Integration Studio. I would like to perform a migration from CSV files to the Oracle database. I have a couple of questions regarding the migration progress control and results reporting:
1) How can I check the progress during the runtime? Currently my grand job consists of several jobs, which also have subjobs, so in the grand job I can only see that one row was migrated but in fact the migration covered thousands of objects (see the snip below). 2) Can other Talend users check the current status of the execution of my job? 3) I would like to get the report containing information regarding migration like: a) how much time did it take b) how much time did each of the subjob take c) how much data was processed? d) with what context variables values was it executed? Is there any functionality in Talend, which enables to record and display such data? 4) Can the jobs be scheduled to be executed later? Is there any way to build a map of job in Talend DI and schedule them to run in a specific order? 5) How can you perform the data reconciliation after a migration in the Talend DI?
I know that I asked many questions in one post but I would appreciate if I got an answer to any of them. If similar issues were handled in other topics, then please redirect me to them. Best regards, Kapi
1- use as many log message as necessary and check log file during the process
2- why not if you share the place where logfiles are stored
3- activate statistics for tStatCatcher in the components you want to server then, having a subjob actived where statistics are genrated, you can trace the information you want
4- scheduling is part of enterprise edition - use an external scheduler if you are using the free edition
5- more details please
Regarding question no. 5:
Let's say, we have extracted records from a CSV file, transformed them (e.g. trim a column to have only zip code instead of zip code and city) and finally loaded them to the database table.
Afterwards, we would like to check not only the quantity of migrated records but their quality as well (e.g. whether all zip codes were properly trimmed). Is there any way to extract the data from both CSV file and database table and get a report presenting deviations between the desired and actual state? Is that feasible within the features of Data Integration application, or I should look for another one (most probably Data Quality)?
Sure DQ may help for this purpose.
However you can construct your own validation process with DI, probably with a higher cost but will respond exactly to your needs.
It depends of your expectations.
Regards,
TRF
