topic Extract different record values from 2 csv files in Talend Studio

Extract different record values from 2 csv files

Anonymous — Mon, 25 Jul 2016 17:51:42 GMT

Hi there, I m trying to compare the records of 2 identical csv files that contain about 80 different fields. I want to list for each field the records that have different values and write the output to a single file. I've tried comparing the two files with a tMap (by joining the primary key of the two files), and creating a single output for each field, which is not very efficient for more than 10 fields. Is there any alternative approach that I can apply ?

Thanks,
Panos

Re: Extract different record values from 2 csv files

Anonymous — Tue, 26 Jul 2016 08:53:56 GMT

Hi,
Could you please take a look at Talend DQ profile feature which is used to analyze column, file and table structure
TalendHelpCenter:Comparing identical columns in different tables to see if it is satisfying your needs?
Best regards
Sabrina

Re: Extract different record values from 2 csv files

Anonymous — Tue, 26 Jul 2016 12:13:19 GMT

Hi, thanks for replying ! I don't think DQ will help me, as I have more than 5 pairs of identical csv files that I want to check, which are about 400mb each. I would prefer DI in order to automate the process for all the files I want to validate.
Regards,
Panos

Re: Extract different record values from 2 csv files

Anonymous — Thu, 28 Jul 2016 17:26:15 GMT

I managed to find a solution to my problem, so I' m posting it in case someone needs to do something similar with DI. I used a tMap component to join on the equal ID of the two tables, a custom tUnpivotRow component to transform the csv file to a key-value table and filters to filter out equal records. Finally I got the format I wanted by extracting the values from the unpivoted table.