I have one source_db and one destination_db. Currently, I have created dimension tables in my destination_db using data from source_db.
Now, I'm trying to populate a fact table using data from source_db and alongside looking up with dimension tables from destination_db. but I'm facing below error:
Duplicate entry '2859' for key 'PRIMARY' Duplicate entry '2859' for key 'PRIMARY' Duplicate entry '2859' for key 'PRIMARY' Duplicate entry '2859' for key 'PRIMARY'
I'm attaching screenshots of job design and tmap settings. Please look into it and let me know what's the issue.
Could you please do the following error validation steps?
a) Add a tLogrow instead of Output component and verify whether multiple output records are coming for same input record (to make sure that no cross joining is happening)
b) If its happening, double the data in source and lookup tables to make sure no duplicate records are present. It will also depend on how you are storing the dimension information. If you are storing multiple versions of dimension information, pick only relevant version
c) If the data after joining is fine in the tLogrow and the issue is occurring due to primary key error, double check the key value in DB. Again, if you are planning to handle multiple versions of same fact data, then make sure versioning is done. Else select the keys exactly based on the key in the DB.
Hope these steps will give you some clue. Please also share the DB table structure if the error persists along with some output data (by printing them using tLogrow).
Warm Regards, Nikhil Thampi
Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂
in the tMap, you did not connect the main input (row1) to the lookup inputs (row2, row3, ...). That way, Talend executes a cross join, combining each input row1 with all rows of each lookup. That generates multiple rows with the same JobID and SOID (assuming that these two or one of them should serve as the primary key of your output table). Hence the error "Duplicate key".
Thanks for your reply. In my fact table, the primary key is a combination of three columns( JobID + SOID + FranchiseeKey). So, like you mentioned I have my main input(row1) columns connected to the lookup dimension table columns....also I have checked the above 3 columns as my keys. But still I'm facing the error :
Duplicate entry '2859' for key 'PRIMARY' Duplicate entry '9443' for key 'PRIMARY' Duplicate entry '9403' for key 'PRIMARY' Duplicate entry '9527' for key 'PRIMARY'
.
.
.
I'm attaching screenshots again please have a look at them. I would like to know is there any way to ignore these errors and let the data pass through?
Could you please do the following error validation steps?
a) Add a tLogrow instead of Output component and verify whether multiple output records are coming for same input record (to make sure that no cross joining is happening)
b) If its happening, double the data in source and lookup tables to make sure no duplicate records are present. It will also depend on how you are storing the dimension information. If you are storing multiple versions of dimension information, pick only relevant version
c) If the data after joining is fine in the tLogrow and the issue is occurring due to primary key error, double check the key value in DB. Again, if you are planning to handle multiple versions of same fact data, then make sure versioning is done. Else select the keys exactly based on the key in the DB.
Hope these steps will give you some clue. Please also share the DB table structure if the error persists along with some output data (by printing them using tLogrow).
Warm Regards, Nikhil Thampi
Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂
