Question

TALEND : while moving employee data from source to target I want to remove duplicates and move them to separate file

question from:https://stackoverflow.com/questions/65883149/talend-while-moving-employee-data-from-source-to-target-i-want-to-remove-dupli

Answer 1

Exactly if the full record is same its a duplicate

like there could be 2 scenarios

1. duplicate records in the source file
2. Duplicate records in the source file when compared with existing table.

duplicate in source file : if there are 3 duplicate records catch the first record and move it to target and other 2 move them to another file (may be u shd add a surrogate key to identify them uniquely)

duplicate rec compared with table data : compare each rec with table data and if its duplicate move it to another file