Remove Duplicate Items

Data Sync can be configured to remove duplicate items from a List with a little configuration.

Data Sync can be configured to remove duplicate items from a List with a little configuration.

Image you have a DataSet like this where ProductID has been duplicated i.e. ProductID=1 appears more than once.

Data Set

ProductID is our key and is supposed to be unique however because of this error Data Sync flags it as a Duplicate like this.

Duplicates

To remove the invalid row we create a new project where source and target point to the same SharePoint List. This project should include only the ID column since we know this will always be unique and therefore will not be automatically excluded from the Data Sync Results.

Schema

Then we need to use a Data Sync Function ISDUPLICATE to return only those rows that are not duplicates so we get a delete action for the duplicates. In Dynamic Columns return the inverse of ISDUPLICATE against the column we're testing i.e. !ISDUPLICATE(ProductID) in the BeginRow() method.

partial class DataSourceRowOverride : Simego.DataSync.DynamicColumns.DataSourceRowInternal 
{
    public override bool BeginRow()
    {                
        return !ISDUPLICATE(ProductID); 
    }   
}

When we now Compare A->B we get the duplicate items to delete. We just need to Synchronise to clean all this UP.

Compare Results


Can't find the answers you're looking for? Contact our support team and we'll personally get back to you as quickly as we can.