Incremental Synchronisation

Data Sync now includes a new compare mode that allows you to easily implement incremental synchronisation scenarios.


Incremental Synchronisation mode of Data Sync improves the efficiency when dealing with large data sets. Data Sync has a unique approach whilst maintaining the state-less operation ensuring a reliable synchronisation process. Incremental mode allows data sync to load a subset of records from the Source and match these against the Target and run the normal reconciliation to generate the change set. Because of this mode of operation is it not possible to detect DELETE operations with Incremental Mode enabled. Not all providers support incremental mode however the following list of providers have incremental support enabled.

  • Microsoft SQL Server
  • ODBC Connections
  • OleDb Connection
  • Microsoft Access
  • Microsoft SQL Compact Edition 4.0
  • Microsoft Dynamics CRM
  • Podio
  • Simego WebStore List
  • Intercom

Incremental mode supports a single key column i.e. you cannot use a composite key column with Incremental Mode.

To enable incremental mode is as simple as setting the option to SyncAtoBIncremental in Project -> Properties

Properties

This then changes the behaviour of Data Sync to load only matching records from the Target based on the Key column selection in the schema map. Because of this Incremental mode should only be used when.

  • Target Data Set Contains many thousands or millions of records
  • Source Data set is relatively small > 20k records.

If the source returns a large data set then the performance of incremental mode is not optimal due to the round trips required to the target to retrieve the records, Data Sync does optimize this by using batching and multiple threads where it can.

This mode of operation would usually be used with a View or CRM FetchXMLFilter Expression that limits the source records.

When Incremental mode is used with DS3 Automation Server (Ouvvi) you can dynamically adjust the source filtering based on a value supplied by the Server. The server will inject some Project Properties into the Project Properties collection. The property Auto_LastSuccessfulRun value is the Timestamp from DS3 Server when the project was last run.

Property Collection

If you combine this with Project Automation you can adjust your data source properties at runtime, for example to update the Where clause of a SQL Provider you might use the code below in Project Automation.

Project Automation


Can't find the answers you're looking for? Contact our support team and we'll personally get back to you as quickly as we can.