set Operation not working as expected in tMongoDBOutput
Hi,
This is regarding a data migration project, where we have to update data in Mongo DB.
We have a collection in Mongo with multiple arrays and objects. The total count of documents in that collection is 600 million. In that there is a particular object with 5 fields inside it. The objective is to build a pipeline to integrate data from multiple source tables and then update that particular object (2 existing fields and add a new field).
In Mongo, we use the $set operation to update values of specific fields without disturbing the other fields.
Similarly, in Talend, we tried using the set operation and it completely overwrites the object. The fields inside the object that are not part of the update are getting removed. The object is left with the new field and the 2 fields which were updated.
Is this expected behavior in Talend? Or is there any other way to perform update operation on Mongo through Talend.
Yes, this behavior is expected when updating MongoDB documents through Talend if the update is performed at the nested object level.
MongoDB natively supports partial updates using $set with dot notation to update individual fields within an object. However, in Talend, if a nested object is mapped and updated as a single structure, the object is treated as a full replacement, and any fields not included in the update are removed.
Recommended approaches:
Use dot notation to update individual fields (for example: object.field1, object.field2, object.newField) instead of updating the full object.
Avoid mapping the complete nested object in the update schema unless a full replacement is intended.
For complex or large-scale updates, consider using tMongoDBRow with a native MongoDB $set statement for precise field-level control.
Best practice: Always perform field-level updates for nested objects to prevent unintended data loss.
