Process to follow in Data Management Project

[vidya821]

Hi,

I have a requirement wherein i need to import different flat files data in orcale data base in 3NF data model which would be then used to export to a datawarehouse.

However before importing to 3NF data model, i need to do data analysis, cleaning, transforming, etc on the data

Could you please let me know if my understanding is correct reg different talend products that can be used in process (Note: I am limited to use only free Talend products)

also to note that this is monthly process, has to be done every month files.

1) Data Integration - Load the initial flat file data in oracle flat table (combine all the different flat files data into one oracle flat table) (IT Responsibility)

2) Data Quality - Analyse the data inconsistency and data mismatch in the oracle flat table (IT/Data Quality Analyst Responsibility)

3) Data preparation - After data quality, the data prep tool will help to cleanse data and import the data back in the oracle flat table or may be in similar table (based on features available in free open source tool) (doubt- can we update the data in same oracle flat table ) (IT/Data Quality Analyst Responsibility)

4) Data Integration  - This cleansed data can be used to import in 3NF data model with applicable business rules and other transformations,etc (IT Responsibility)

Please do correct my understanding and help me with appropriate solution

Thanks

Vidya

[Anonymous]

Hi Vidya,

      Below link would give details about all different open source products of Talend in detail.

https://www.talend.com/products/talend-open-studio/

      I would suggest to start the data quality verification work as the starting point as it will help to identify the source system related issues at the earliest possible time.Once the quality issues are addressed, you can do data preparation and move further to integrate the flow from source to target system.

      If you would like to have a single software, Talend Data Fabric will be the one stop solution although you will have to purchase the license which is quite less compared to other competitors in the market.

Warm Regards,

Nikhil Thampi

[vidya821]

Thanks Nikhil for the inputs,

I have checked and gone through all the open source products and as of now i am limited to use only the free Tools and would not be using the Data fabric.
I would certainly love to use the data fabric 🙂

Thanks
Vidya

[Anonymous]

Hi Vidya,

      Thanks for the update. If you are happy with the details provided about the details of different Talend Open source products, could you please close the topic with solution accepted flag? 

Warm Regards,

Nikhil Thampi