Team Blog

Updating the Active Directory Manager Attribute

info@simego.com (Rebecca Allen) — Fri, 18 Feb 2022 08:49:45 +0000

The Active Directory connector has built in columns that handle all the hard work when mapping to attributes that need to lookup values, these are the Data Sync columns (DS-) that will only show in Data Sync.

When updating or adding a manager attribute to a user in Active Directory you need to lookup the manager within your OU by providing the distinguished name. With the Data Sync columns you can choose to use either their Email, their Employee ID, their SAMAccountName or the Distinguished Name. Data Sync will then lookup the value in the OU you are connected to and return the DN, this is what AD requires internally when writing to the manager attribute.

The following page discusses how you would go about configuring your project to update the manager attribute in Active Directory for existing users, and then the options you have to automate and schedule your project. We cover how to add the manager attribute to new users in the Creating AD Users blog.

Requirements
Connect to your Source and Active Directory
Connect to Active Directory
Schema Mapping
Preview the Compare Results and Sync
Miscellaneous

Adding Additional Attributes

Automate & Schedule

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
Have a source data set containing your user and manager details
Have access to Active Directory

If you do not have Data Synchronisation Studio you can get a free evaluation edition. To test the automation using Ouvvi please contact us to upgrade your trial key.

Connect to your Source and Active Directory

Start by opening Data Sync and connect to your source data. This can be a SQL table, excel spreadsheet or a CSV as a few examples. For this scenario we will be using a SQl table, but the process will be the same for any of our available connectors.

You then need to connect to your Active Directory users. To do this click onto Connect Datasource in the target window and go to Active Directory > Active Directory V2 - Users/Contacts/Groups/Computers.

By default Users will be selected from the DefaultAttributes drop-down menu, this is what you need so you don't need to change this. Then enter in the LDAP path to your Active Directory instance, and enter in any credentials you need to access AD.

The LDAP Path can be just your server name or the path (including the server name) to a specific OU. For example connecting with just a Server Name would look like this: LDAP://dc01, and connecting to a specific OU would look similar to: LDAP://dc01/OU=Demo,DC=lab,DC-simego,DC=com. You can find more details on how to find your LDAP path here.

You can then click Connect & Create Library Connection to save the connection to the connection library. Just enter in a name for the connection and click OK. Then refresh the connection window and your Active Directory connection will be visible in the list.

This only needs to be done once per OU as you will be able to access other objects such as Users and Contacts from the connection window.

If you have already saved your connection to AD you can select the Users object from your AD connection.

Mapping

We then need to map the source columns to their corresponding attributes in AD.

For the manager attribute you need to have listed either the email address, employee ID, SAMAccount name or distinguished name of the manager in your source data.

Map this source column to the corresponding DS- column from the target window. For example we are using the manager's email address, so this is mapped to the DS-Manager-Email column. Data Sync will then do the lookup for you and return the DN of the manager when writing to AD.

Preview the Compare Results and Sync

Once the mapping is configured you can compare the source data against the target and preview the results.

In this example we can see we have 5 updates to make, and the changes to be applied to Active Directory are highlighted in yellow.

Please note deletes are disabled by default but will still show in the results if records are not present in the source but are in the target.

To apply these changes click onto Synchronise and then Start to begin the sync.

Your Active Directory users will now have the manager attribute updated with the value you supplied. Make sure to save your project so that you can use it again in the future.

Automate & Schedule

You may want to automate the manager update project so that it runs on a regular basis so that your Active Directory is always kept up to date. To do this you can use either the Run Tool or Ouvvi.

Ouvvi gives more options for scheduling, with both time and event based triggers. So you could configure a trigger to start your project whenever a change is detected in your SQL table or SharePoint List. Note that this requires using a modified date time stamp in your SQL table.

Alternatively you can use the run tool to schedule the project to run using Windows Task Scheduler.

Run via the Run Tool

The Run Tool is an additional program that comes linked to Data Sync and enables you to build out your data integration jobs.

You can use this to group Data Sync projects that need to run in order and add additional step types such as adding an status report to email you when the project has run and if there were any failures. Each step is run in the order it appears in the list and you can apply conditional rules so that the next step can run if the previous one succeeded or had data changes for example. To find out more please see our Run Tool Documentation.

To open your project in the Run Tool, in your Data Sync project go to Tools >Open in Run Tool.

You can then click onto the green Run button to check it runs as expected. If there are data changes you will be provided with a count of how many items were added updated or deleted

Run via the Command Line

Another option is to Run the Run Tool project or single Data Sync project from the command line. To do this pass the path to the project file like this:

Run Tool Project

%DATASYNCINSTALLPATH%\Simego.DataSync.Run.exe -execute "D:\DSProjects\myproject.dsrun"

Data Sync Project

%DATASYNCINSTALLPATH%\Simego.DataSync.Run.exe -execute "D:\DSProjects\myproject.dsprj"

Automate with Windows Task Scheduler

You can then schedule your Run Tool project to run when you need it to using Windows Task Scheduler. For the full details on how to do this see our Task Scheduler Documentation.

Automate with Ouvvi

An alternative option, with more scheduling capabilities is to use Ouvvi Automation Server. Ouvvi enables you to fully schedule and manage all of your Data Integration Projects, from Data Sync projects to SQL Statements to Powershell Scripts.

It provides full logging and documentation capabilities to fully manage your integration operations.

To find out more see our Ouvvi Documentation or send us an email.

To learn about creating new Active Directory Users, setting default passwords and adding them to groups you can see our Create AD Users blog.

Creating New Active Directory Users

info@simego.com (Rebecca Allen) — Tue, 22 Feb 2022 10:00:00 +0000

If you have a series of users to provision in your Active Directory then you can make the process simple and automated using Data Sync and Ouvvi.

The following page will cover how to configure a Data Sync project to create Active Directory user accounts, set the manager, assign the users to groups and set a default password in the same project. We then cover a few options you have in order to automate the project so that it can be run automatically.

You can create Active Directory users from any data source, for example a CSV, Excel Spreadsheet, SQL Table or a SharePoint list to list a few.

Requirements
Connect to your Source
Connect to Active Directory
Map the Source & Target Columns

Set a Default Password
Assign a Manager
Enable User Accounts
Add to Groups

Preview and Sync the Results
Automate & Schedule
Video - Adding Users

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
A data set with your user details listed
Access to Active Directory

If you do not have Data Synchronisation Studio you can get a free evaluation edition. To test the automation using Ouvvi please contact us to upgrade your trial key.

Connect to your Source

First you need to prepare your source data to make sure you have all the data you want to add. You want your source data to look like you want your Active Directory to be.

If you want to add a manager you need to ensure you have either a manager email, employee ID, distinguished name, or sAMAccountName listed against your users. In this example we have the manager email address.

If you want to add the users to groups then you need to ensure the groups are listed with a semi colon separating them. The dataset shown below has users added to three different groups and these are listed as: Group A;Group B;Group C

A preview of the dataset being used in this example can be seen below:

Once you have your data together connect to this as your source in the Data Sync Project by clicking into the data source A window. In this example our data is stored in a SQL Table, but you can use any of the built in providers to connect to your source data.

Connect to Active Directory

You now need to connect the target data source to your Active Directory OU. To do this click onto Connect Datasource in the target window and go to Active Directory > Active Directory V2 - Users/Contacts/Groups/Computers.

You can enter in the full LDAP path (including the server name) or just the server name. If you just use the server name your path would look similar to: LDAP://dc01.
Otherwise an example path connecting to a specific OU could be: LDAP://dc01/OU=Test,DC=demo,DC=simego,DC=com. We have more details on finding the LDAP Path in our documentation.

Then click Connect & Create Library Connection to save the connection to the connection library. Just enter in a name for the connection and click OK. Then refresh the connection window and your Active Directory connection will be visible in the list.

This only needs to be done once per OU as you will be able to access other objects such as Groups and Contacts from the connection window.

If you have already saved your connection to AD you can select the Users object from your AD connection.

Map the Source & Target Columns

The next step is to configure the schema map so that your source columns are linked to the correct attribute in Active Directory.

You need to specify a key column to identify each user as unique. We find that the AccountName is a good field for this so map this to the DS-SAMAccountName column another alternative is LogonName (DS-UserPrincipalName).

DS- columns are Data Sync attribute columns that handle the addition of that field for each user. These usually have to be handled differently to add the values, which in previous versions would have needed you to write code. These columns take away the need to add code to project automation.

Your basic schema map might look similar to the screen capture below:

You can then follow each of the sections below to add the manager details, set a password, enable the user account and add the user to groups.

Set the Default Password

To set a password you need to map the password column from your source to the DS-SetPassword column in the target.

If you don't have a password listed in your source data you can supply a password, such as !Password123, to be used to all new accounts using calculated columns.

To do this click onto the fx button in the data source A toolbar to open the calculated column window.

Your calculated column should then be configured to look similar to:

Add your column to the schema map and match this to the DS-SetPassword column in the target.

To make the user have to change the password at their next logon, as the default is not secure, we need to set pwdlastset in the target connection properties with the OnCreateChangePasswordNextLogon property. By default this is set to true, however if you do not want users to change the password then set this to false.

So then the password does not cause an update action each time the project runs (on existing user accounts) you need to change the schema property Ignore for the Password column to True.

NB. The password can only be set on new user accounts

Set the Manager Attribute

To set the manager attribute you just need to map one of the following columns to it's corresponding data sync column.

Distinguished Name -> DS-Manager-DN
EmployeeID -> DS-Manager-EmployeeID
Email -> DS-Manager-Email
sAMAccountName -> DS-Manager-SAMAccountName

In this example the manager is listed by the Email Address, so for this we would use the target column DS-Manager-Email.

Please note that if the manager does not exist in Active Directory it will return null for the manager field. For example a new manager may be in the list of users you are adding. If you have this scenario then you will need to run the project again to update the manager column. Make sure to set EnableUpdate to True if it is not already.

For a more detailed description on updating the manager attribute for existing users please see our other blog here.

Enable User Accounts

You can also set the user account to be enabled or disabled by returning a boolean value. Return True to disable the account and False to enable the account.

Then map this column to DS-UAC-Disabled.

Add to Groups

To add the users as members in groups you need to have the groups they are to be members of listed in an array separated by a semi colon. Your groups listed must already exist in Active Directory when adding users. E.g. The source data in this example has a value of Group A;Group B;Group C for the groups column.

If your groups do not already exist checkout our groups documentation which covers how to create groups and manager the assigned users.

If your group does not exist in Active Directory then Data Sync will throw an error when you try to synchronise.

To add the users to these groups, map your source Group column to the target column DS-User-MemberOf and Data Sync will handle the rest for you at the point on synchronisation.

At the end of this your schema map should look something like the following image.

Preview and Sync the Results

To add the data to your Active Directory you need to click the Compare A > B button. This will enable you to preview the additions or changes being made to your Active Directory.

If it shows delete actions do not panic, these are disabled by default. If you want to enable deletes you will need to set EnableDelete to True on the target.

In this example we have 9 users to add, clicking onto the Add line will show you the data that is to be added to Active Directory.

If you want to run a test on one account rather than the whole dataset then deselect the check-boxes using the clear all button and then check the check-box against the record you want to add.

Now click the Synchronise button and then start to begin the sync to add your user accounts to Active Directory.

To check the data has been synchronised successfully you can either browse your AD for a user account or you can click onto the Preview Datasource in the target window.

You should now have all your users added to Active Directory with their managers assigned, them being members of the groups they need to be in, and have a password set so that they can log in.

Automate & Schedule

You may want to automate the creating of user records in active directory so that it runs on a regular basis, to do this you can use either the Run Tool or Ouvvi.

Alternatively you can use the Run Tool to schedule the project to run using Windows Task Scheduler.

Run via the Run Tool

The Run Tool is an additional program that comes linked to Data Sync and enables you to build out your data integration jobs.

To open your project in the Run Tool, in your Data Sync project go to Tools >Open in Run Tool.

You can then click onto the green Run button to check it runs as expected. If there are data changes you will be provided with a count of how many items were added updated or deleted

Run via the Command Line

Another option is to Run the Run Tool project or single Data Sync project from the command line. To do this pass the path to the project file like this:

Run Tool Project

%DATASYNCINSTALLPATH%\Simego.DataSync.Run.exe -execute "D:\DSProjects\myproject.dsrun"

Data Sync Project

%DATASYNCINSTALLPATH%\Simego.DataSync.Run.exe -execute "D:\DSProjects\myproject.dsprj"

Automate with Windows Task Scheduler

You can then schedule your Run Tool project to run when you need it to using Windows Task Scheduler. For the full details on how to do this see our Task Scheduler Documentation.

Automate with Ouvvi

It provides full logging and documentation capabilities to fully manage your integration operations.

To find out more see our Ouvvi Documentation or send us an email.

Managing Active Directory User Group Membership

info@simego.com (Rebecca Allen) — Fri, 25 Feb 2022 10:04:44 +0000

Data Sync can be useful to assign users to groups in Active Directory and also export a list of users that are members of groups in your Active Directory.

You might want an export to review current group membership, check that the correct users are in the correct groups, or be used to update another business system e.g. a SharePoint list.

Requirements
Group Connector Options
Export Group Membership
Manage Group Membership
Adding Groups to User Records

Set a Default Password
Assign a Manager
Enable User Accounts
Add to Groups

Preview and Sync the Results
Automate & Schedule
Video - Adding Users

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
A data set with your user details listed
Access to Active Directory

If you do not have Data Synchronisation Studio you can get a free evaluation edition here.

Active Directory Group Connector Options

When connecting to Active Directory Groups you have a few options available, which you use depends on how your data is structured and whether you want to create groups at the same time.

The Group Membership connector returns a list of all the groups within the connected Active Directory environment or organisational unit and the users that are members as separate rows. This uses the Active Directory V2 - Group Members connector, and can only be used to add and remove users from existing groups. An example of the dataset this produces can be seen below:

The Groups connector returns a list of all the groups and a column with an array of the users that are members. This uses the Active Directory V2 - Users/Contacts/Groups/Computers connector with the Groups connection and can be used to create groups if they do not already exist and add users to those groups. An example of how this looks as a dataset is:

If your data has a list of users with an array column containing the groups they are members of, then you can use the Active Directory V2 - Users/Contacts/Groups/Computers connector with the Users connection. An example of how this looks as a dataset is:

We cover using each of these in different scenarios in the next sections.

If you need to get your data from group members to an array of members then you can do so with a SQL statement. The statement below aggregates the members for each group into an array separated by a semi colon, grouped by the group name.

SELECT [Group],
STRING_AGG([Members], ';') WITHIN GROUP (ORDER BY [Members] ASC )
FROM [Groups]
GROUP BY [Group]

You can then use this data set with Active Directory V2 - Users/Contacts/Groups/Computers connector.

Export Group Membership

To export the Group Membership you need to first connect to Active Directory. Decide how you would like your data returning either separate rows for each member in a group or an array of members against each group.

For this example we will use the Group Membership connector to return a list of all the groups and all the users contained in the groups.

Start by connecting to your Active Directory Groups Members. To do this click open the connection window and go to Active Directory > Active Directory V2 - Group Members.

Then decide if you want to export it to a business system or as a simple CSV, Excel, XML or JSON file.

Export to a File

To export your members list to a file, you can either use the create functions from the tools menu or you can use the export options on the preview tab.

Start by adding the columns you want to include in your export to the schema map.

Then to export to a file, click onto Preview A. You can then choose from the export options which file type you would like and then save the file. Choose between an Excel spreadsheet, a CSV, XML or JSON file.

Alternatively you can use the tools menu to create a new CSV or XML file. Just go to Tools and select the option you would like. Select the location to save the file and click save.

Then check the schema mapping, and make sure you have defined a key column. The key column for the group membership is going to be a composite key as the group is not unique. Your mapping might look similar to this:

To synchronise the data just click Compare A > B and sync the results.

Export to a Business System

To export your members list to another business system, for example a SharePoint List or a SQL Table, you can simply connect your target to the list or table if it already exists.

If you need to create a new list or SQl table you can make use of the quick create functions to create them. Just open the Tools menu and select the option you would like to use. For example to create a new SQL table you can select Create Sql Table and then follow the instructions in the wizard to create it. The columns will be populated based upon the columns/attributes added to the schema map.

Make sure to set your key columns, this needs to be identifying so that each record appears as unique. In this example we need to use a composite key, so both the Group AccountName and the User AccountName.

To add the data to the table click Compare A > B followed by Synchronise and then Start. You can preview the results before synchronising to make sure the data presents as expected. Below you can see an example of the preview results, where we have 11 records to add.

Make sure to save your project for future use. You might want to schedule it to run on a regular basis using either the Run Tool or Ouvvi Automation Server.

Manage Group Membership

To manage group membership in Active Directory you can use a source dataset to add and remove users from groups. It depends on how your data is presented in your source system as to which Active Directory connector you will use in Data Sync.

If your dataset has the users listed against the groups, for example:

Group 1 | User A
Group 1 | User B
Group 2 | User A

Then you will use the Active Directory V2 - Group Membership connector.

If your dataset lists the groups and the members within that group as an array (separated by a semi-colon) then you can use the normal Active Directory V2 - Users/Contacts/Groups/Computers connector.

Group 1 | User 1;User 2
Group 2 | User 1
Group 3 | User1;User3;User5

To create new groups you will need to ensure you are connected using the Groups option in the Default Attributes connection property as all the other types do not support group creation.

In the following example we have groups with the members listed as an array, this will be being updated with data stored in a SQL Table. There will be a few groups listed in the source that are not yet in Active Directory.

Connect to your Source Data

Start by opening Data Sync and connecting to your source data. You can use any of the built in connectors in Data Sync or build your own connector if we do not have one readily available.

For this example we are connecting to a SQL Table that has the group AccountName and a column containing an array of the users that should be members. This dataset looks like this:

Connect to your Active Directory Groups

You now need to connect to Active Directory. To do this click onto Connect Datasource in the target window and go to Active Directory > Active Directory V2 - Users/Contacts/Groups/Computers.

As this project is specifically updating groups we need to change the DefaultAttributes property to Groups. If you wanted to modify other attributes alongside the group membership then you could leave the DefaultAttributes property set to Users, but we discuss this in more detail in the next section.

Then enter in the LDAP path to your Active Directory instance, and enter in any credentials you need to access AD. You can enter in the full LDAP path (including the server name) or just the server name.

If you just use the server name your path would look similar to: LDAP://dc01.
Otherwise an example path connecting to a specific OU could be: LDAP://dc01/OU=Test,DC=demo,DC=simego,DC=com.

We have more details on finding the LDAP Path in our documentation.

Then click Connect & Create Library Connection to save the connection to the connection library. Enter in a name for the connection into the windows that pops up and click OK. You can refresh the connection library window (to the left of the Data Sync window) and your Active Directory connection will now be visible in the list.

This only needs to be done once per OU as you will be able to access other objects such as Users and Contacts from the connection window.

If you have already saved your connection to AD you can select the Groups object from your AD connection.

Map the columns

Now map the source columns to their corresponding attributes in Active Directory. Make sure to map the user array to the DS-MemberNames column as this will handle the lookup of the user names for you.

Make sure to set the Group SAMAccountName as the key column so that each group can be identified.

Preview the Results & Sync

Once you have configured the schema map you can preview the results before synchronising.

To do this click on the Compare A > B button in the toolbar, this will display the number of records to be added, updated and deleted. Please note that deletes are disabled by default but will still show in the compare results. To enable them please set EnableDelete to True in the target connection properties.

You can preview the data changes by clicking onto each section. In this example we have 2 groups to create and 2 groups to update. Clicking onto the result rows will show the data to be changed.

For updates any changes are highlighted in yellow.

Please note that deletes are disabled by default but will still show in the results if the group is not present in the source data. To enable deletes set EnableDelete to True on the target connection properties.

Once you are ready to apply the changes to Active Directory click onto the Synchronise button and then click Start.

Your users will now be added to their relevant groups, and if the group did not exist it will now be created.

Make sure to save your project so that you can use it again in the future.

If you would like to schedule your project to run automatically you can do so using either the Run Tool or Ouvvi Automation Server with both event and time based triggers.

Adding Groups to User records

You can also manage group membership when adding and updating existing users in Active Directory. Your user records need to have the groups they are a member of listed in a column separated by a semi-colon.

An example of how this might look in a dataset is:

Connect to Active Directory as normal using the Active Directory V2- Users/Contacts/Groups/Computers connector and make sure the DefaultAttributes is set to users.

Then map your group array column to the schema map and make sure to include a key column to identify the user. We recommend using the SAMAccountName as this will always be unique. In this example we are also updating the manager email address so this is also added to the schema map.

Then run the compare and preview the results. Once you are satisfied that the columns are mapped correctly click Synchronise and then Start to begin the sync.

Please note that if your groups do not already exist in Active Directory then an error will be thrown. You will need to use the Groups connector to be able to create new groups.

If you're looking to synchronise Active Directory group members to a SharePoint security group take a look at our other blog here.

Bulk Update Active Directory

info@simego.com (Rebecca Allen) — Wed, 02 Mar 2022 10:14:11 +0000

You can use Data Sync to apply large updates or additions to your Active Directory records with a few easy steps.

You may need to change the enabled or disabled status of a large quantity of users, or you might have added a new metadata column for all users stored within a SharePoint list and this needs to be applied to each Active Directory record. With Data Sync you simple connect, map, compare and sync, and your changes will be applied.

Below we discuss setting up a project to add 800+ new users, and update 160 records with an additional column of data and an account status change.

Requirements
Connect to your Source and Target
Mapping
Preview the Results and Sync

Requirements

Before you get started there are a few requirements:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
Access to Active Directory
A data set containing the records you want to add

If you do not have Data Synchronisation Studio you can get a free evaluation edition here.

Connect to your Source and Target

To get started open Data Sync and connect to your source data, you can connect using any of the inbuilt connectors or build your own.

Then connect the target window to Active Directory. The default attribute to select in the connection will depend on the data you want to add. In this example we are adding to user records but you can change this as you need to and each type will follow the same process.

To connect to the user records in Active Directory we need to click onto Connect Datasource in the target window and go to Active Directory > Active Directory V2 - Users/Contacts/Groups/Computers.

By default Users will be selected from the DefaultAttributes drop-down menu, enter in the LDAP path to your Active Directory instance, and enter in any credentials you need to access AD.

You can enter in the full LDAP path (including the server name) or just the server name.
If you just use the server name your path would look similar to: LDAP://dc01.
Otherwise an example path connecting to a specific OU could be: LDAP://dc01/OU=Test,DC=demo,DC=simego,DC=com.

We have more details on finding the LDAP Path in our documentation.

This only needs to be done once per OU as you will be able to access other objects such as Groups and Contacts from the connection window.

If you have already saved your connection to AD you can select the Users object from your AD connection. To do this right click onto Users object and select Connect to Target (B) or drag the object onto the target window.

Mapping

The next step is to map the columns you want to update and make sure to add a key column that can be used to identify each user. For users we recommend using either the SAMAccountName or the Email Address as these will always be unique.

Add the source columns to the schema map and match them to their corresponding Active Directory attribute.

Preview the Results and Sync

Now we need to run the compare to preview the results before synchronising. To do this click onto the Compare A > B button in the toolbar, and then select a row to view either the adds, updates or deletes.

NB. Deletes are disabled by default but will still show in the results if the records do not exist in the source data set. To enable deletes set EnableDelete to True in the target connection properties.

In this example we have 840 records to add to the target and 160 records to update. Below you can see that the update actions are highlighted in yellow.

Make sure to take a quick glance through to ensure the data looks right and then click Synchronise and Start to begin the sync.

Your Active Directory records will then be added and updated where necessary.

Make sure to save your project in-case you need to use it again in the future. You can always modify it to include more columns.

Creating Database Scripts from the Schema Map

info@simego.com (Rebecca Allen) — Thu, 24 Mar 2022 10:00:00 +0000

A feature available in Data Sync, that you may not already know about, is the ability to generate script to create new tables in your Database based upon the columns you add to the schema map window.

You can edit the configuration before generating the script; such as changing the column names, data types and data lengths.

To generate the script you simply click a button.

We then go a step further and enable you to connect to your database and run the script from Data Sync. Before running the script if you want to add to it you can.

This works in a similar way to the SQL Query Tool.

Requirements
Connect to your Source
Generate the Script
Set the New Table as the Target
Map the Source & Target Columns
Compare the Results and Sync

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
A dataset to base the script off
Access to a database (either SQL Server, MySQL or PostgreSQL)

If you are connecting to PostgreSQL or MySQL please make sure you have the relevant ADO.net drivers installed.

MySQL: https://dev.mysql.com/downloads/connector/net/
PostgreSQL: https://github.com/npgsql/npgsql/releases/tag/v4.0.12

If you do not have Data Synchronisation Studio you can get a free evaluation edition.

Connect to your Source

You need to start by connecting to a datasource that you can base your schema off. This might be the dataset you want to ultimately import into your new table once it has been created.

In this example we have a simple CSV file with a number of columns describing contacts. You can use any of the providers built into Data Sync or you can write your own to connect to your business systems.

To connect to a CSV file you can either drag and drop the file from the file explorer onto the source window or you can browse for the file in the connection window. Then click onto Connect Datasource > Text Files > CSV File and browse for your CSV by clicking onto the ellipsis in the FileName field.

Then add the columns you want to create in the new table to the schema map and move onto the next step to configure and generate the script.

Generate the Script

In the schema window toolbar you can find the Generate SQL Script button. Click on this to open the configuration window.

You are first presented with a window to define the table name and select a connection to the Database you want to create the table in. If you want to run the script outside of Data Sync then you can leave this blank. You can also define the type of database and script you want to generate. These settings can also be changed later on.

You can only select connections saved in your connection library. If you have not saved your connection you will need to open a new Data Sync project, create and save the database connection to the connection library before you can connect in the SQL Script Generator.

The Script window will then be opened and you can now add additional columns and change column names, data types and data lengths. If you defined a key column in the schema map this will be identified as the primary key column in the configuration. If you need to set a composite key, you can do so here.

To generate the script click onto the Script Create Table in the window toolbar.

This will generate the script in a window below the columns.

You can make edits and additions to this script, copy it or run it directly from Data Sync. To run the script on the database connection you listed at the start click onto the Execute Query button.

Your table will now be created in the Database specified. If you refresh your connection library and navigate to your connection. You should now be able to see your table in the list.

Set the New Table as the Target

You most likely now want to synchronise data to the new table. To do this you need to connect your target to the table.

As this is stored in the connection library you can find your database connection, open the tables folder and either drag and drop or right click and select Set as Target.

Map the Source & Target Columns

Now check that the source and target columns all match and ensure you have selected your key column to identify each record.

Compare the Results and Sync

You can then click the Compare A > B button in the toolbar to preview the results. As this is a new table we will only see additions, but if you run the project again at a later date we may also see updates and deletions.

Deletes are disabled by default but will still show in the results. To enable deletes set EnableDelete to True on the target.

To synchronise the results click Synchronise and then Start to begin the sync.

Your data will then be visible in your new table. Make sure to save the final project so that you can run the future synchronisation again in the future.

Miscellaneous

Whilst using the SQL Table Script Generator you can edit the configuration you initially determined at the beginning. You can change the script language, change the name of the table to be created, and edit the database connection details.

Edit the Script Language

You can edit the language the script is generated by selecting between SQL Server, MySQL, and PostgreSQL in the drop down in the toolbar window.

Edit the Table Name

To edit the name of the table to be generated, and listed within the script, click onto the current name field in the toolbar of the window and simply make the changes you need.

Change the Database Connection Details

To change the database connection details you can click onto the connection in the script editor. This will open the Connection Properties for the connection you selected at the start. Make the changes you need and then click OK to save them.

Execute Selected Script

If you add additional script to the script generated by Data Sync, for example a SELECT query, then you can run just that script by highlighting it and then clicking Execute Query.

Taking the example we used above if I wanted to preview the records within that table after I had run the synchronisation I could use the query:

SELECT * FROM MyContact

Highlighting this query and clicking the Execute Script button will then bring up my data in the preview window below the script.

In the preview window you can also edit the records manually by double clicking onto the field you want to edit. Clicking out of that field will apply the changes.

If you are looking to run other SQL queries in Data Sync then you can do so using the SQL Query Tool. Check out our documentation for more information.

Integrating with Salesforce

info@simego.com (Rebecca Allen) — Wed, 30 Mar 2022 10:00:00 +0000

With Data Sync you can quickly and easily import and export data to and from your Salesforce objects.

Below we cover importing contact records into the contact object from a SQL table, and then we cover how to export the Salesforce account object into a SQL table.

Although we use the example of a SQL table in this article, you can use any of the available connectors to import and export data. If there is not currently a connector for your system then you can always build a custom connector.

Note that when designing your integration with Salesforce you need to think about the relationship between objects. There may be relational fields that are dependant on values being added in another object first before they can be used in the one your project is updating.

For example the contact object has the Account ID which links it to the account object. If you try to add a contact for an account that doesn't exist in the account object then it is likely to fail. To get around this issue you should make sure that the account object is updated before the contact object.

Requirements
Importing Data into Salesforce

Connect to your Source Data
Connect to Salesforce
Map the Source & Target Columns
Preview the Results and Sync

Exporting Data from Salesforce

Connect to Salesforce
Connect to your SQL Table
Map the Source & Target Columns
Preview the Results and Sync

Additional Options

Filtering the Results
Automate & Schedule
Troubleshooting

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
Access to Salesforce
A dataset to connect to
Access to a SQL table, or another system to export your data to.

If you do not have Data Synchronisation Studio you can get a free evaluation edition.

Importing Data into Salesforce

You can quickly and easily synchronise your business data into Salesforce when you use Data Synchronisation Studio. The example below connects to a SQL table with contact records that need to be imported into Salesforce and linked to an account. The account records have already been synchronised prior to this.

Although we use a SQL table in this example you could use any of the other connectors as your source. This might be Active Directory, a SharePoint List, a CSV file or a Dynamics Entity to mention a few. Just select the connector you need, enter the required connection details and you're ready to go.

Connect to your Source Data

In this example we have a handful of contacts in a SQL table that we want to get into a Salesforce Contact object.

To do this we need to start by connecting our Source window to our SQL table. To do this click onto Connect Datasource to open the connection window. Then navigate to SQL Database > Microsoft SQL Server (SQL Client).

Enter in the network name of your SQL Server, any credentials you might need to connect and set encryption or trusted certificate as required.

Click Connect & Create Library Connections to save this database connection to the connection library. This only needs to be done once per database and will make future project setup easier.

Select the database and table from the tree view to connect to, and if you are saving to the connection library another window will open where you need to enter a name in for the connection.

This will load the columns in your SQL table into the source window.

Connect to your Salesforce Object

To connect to Salesforce in Data Sync you need to have first configured the app registration in Salesforce so then you have all the details needed to connect via OAuth.

You will need the redirect URL port number for your app registration which can be found in the connection window of Data Sync. To get to this click onto the target window and then go to Salesforce > Salesforce Object.

Once you have configured the app in Salesforce you can now enter the Consumer Key and Consumer Secret into the connection window in Data Sync and click Authorize Connection to validate the app details.

If the authorisation is successful you will be able to select an object to connect to from the object name dropdown list.

Click Connect & Create Library Connection to save the connection to the connection library. This only needs to be done once per site as you will be able to access the other objects from the connection library.

If you have already saved your connection to the connection library then you can expand the connection and select the object you want to connect to.

Map the Source & Target Columns

You now need to configure the schema map so that the source columns are mapped to their corresponding column in Salesforce. Make sure that the data types are correct and that you have selected a unique key column that can be used to identify each record.

If your source data doesn't have the Salesforce Account ID then you can do a lookup based on the internal account number.

Lookup Account Number

In order to write to the account number column in salesforce you need to return the correct salesforce id for the account. If you have synchronised your internal account id's to the account number column within the salesforce account object then you can do a lookup to return the correct account id so that salesforce links the contact to the account.

Just drag the account object from the connection library onto your account number column to create the lookup, and configure the connection to use the account number column to match the values.

You can then map the id column returned by salesforce to the lookup account id column.

Preview the Results and Sync

Once the mapping is complete you can click Compare A > B to preview the changes that need to be made. Here you can preview additions, updates and delete actions.

Please note that deletes are disabled by default but will still show in the results. To enable deletes please set EnableDelete to true in the target connection properties.

Click onto each option to view the changes that will be applied to Salesforce. For updates the changes will be highlighted in yellow.

To synchronise the results click onto the Synchronise button and then Start to begin the sync.

Your contact records will then be added to Salesforce and available to view. Make sure to save your project so that you can use it again in the future.

Exporting Data from Salesforce

If you need to get data out of Salesforce and integrate it with another business system then Data Sync makes this into another simple task.

You can export the data previews to files such as XML, CSV, Excel and JSON for one off exports by using the buttons in the export window.

Alternatively you can create a new SQL table, CSV File or XML dataset for example from the tools menu, or you can connect to an existing system using any of the connectors available.

In the example below we are setting up a project that will export the salesforce account records to an existing SQL table on a regular basis.

Connect to your Salesforce Object

To connect to Salesforce in Data Sync you need to have first configured the app registration in Salesforce so then you have all the details needed to connect via OAuth.

You will need the redirect URL port number for your app registration which can be found in the connection window of Data Sync. To get to this click onto the source window and go to Salesforce > Salesforce Object.

Once you have configured the app in Salesforce you can enter the Consumer Key and Consumer Secret into the connection window in Data Sync and click Authorize Connection to validate the app.

If the authorisation is successful you will be able to select an object to connect to from the object name dropdown list.

If you have already saved your connection to the connection library then you can expand the connection and select the object you want to connect to.

Connect to your SQL table

To connect to a SQL table click onto Connect Datasource in the target window to open the connection window. Then navigate to SQL Database > Microsoft SQL Server (SQL Client).

Enter in the network name of your SQL Server, any credentials you might need to connect and set encryption or trusted certificate as required.

Click Connect & Create Library Connections to save this database connection to the connection library. This only needs to be done once per database and will make future project setup easier.

Select the database and table from the tree view to connect to, and if you are saving to the connection library another window will open where you need to enter a name in for the connection.

This will load the columns in your SQL table into the target window.

Map the Source & Target Columns

You now need to configure the schema map so that the columns in Salesforce are mapped to their corresponding column in your SQL table. Make sure that the data types are correct and that you have selected a unique key column that can be used to identify each record.

Compare the Results and Sync

Once the mapping is complete you can click Compare A > B to preview the changes that need to be made. Here you can preview additions, updates and delete actions that need to be made.

Please note that deletes are disabled by default but will still show in the results. To enable deletes please set EnableDelete to true in the target connection properties.

Click onto each option to view the changes that will be applied to your SQL Table. For updates the changes will be highlighted in yellow. In this example we have 80 records to add and 3 to update.

To synchronise the results click onto the Synchronise button in the toolbar and then Start to begin the sync.

Your contact records will then be added and available to view in your table.

Make sure to save your project so that you can use it again in the future.

Additional Options

Below you will find a couple of other topics such as troubleshooting, filtering the records and how to run the projects on a schedule.

Filtering the Results

When exporting results from Salesforce you can filter the results returned before you export, using either a SOQL Query or the filter box built into Data Sync.

SOQL Query

To use a SOQL query you need to have first connected to Salesforce and saved the connection to the connection library. You can then either drag and drop or right click and connect to the SOQL Query connector within your Salesforce connection. To add the query to the project click onto the ellipsis in the connection properties window in the query field and hten enter in the statement. You can import a query statement by using the Open From File button to browse for the query.

An example of a SOQL Query you could use could be, to select all records in the contact object that have a first name of "Dave":

SELECT Id, FirstName
FROM Contact
WHERE FirstName = "Dave"

To use the filter box built into Data Sync, start by connecting to Salesforce. Then enter in a function to filter your records. This function needs to be in a C# friendly format i.e. no spaces in the column names and no special characters.

An example to filter the results for a specific company is: Name == "MyCompany"

The expression needs to return true in order to return any results. So for the example above if there is not an exact match for MyCompany in the Name column in Salesforce then no results will be returned. If MyCompany exists then the record will be returned.

To preview the results click onto the Preview A button in the schema map window.

Automate and Schedule

If you want these projects to run on a regular basis then you can automate them and schedule them to run using either the Run Tool & Windows Task Scheduler or Ouvvi Automation Server.

The Run Tool will enable you to collect together multiple projects to run in a sequence and then schedule them to run at a regular time or interval using Windows Task Scheduler. The Run Tool can be accessed either through the start menu or through the Tools menu in Data Sync.

There a multiple step types available so that you can build out your integration project further. For example you can add an email report step to send an email every time the project runs or if there is a failure. You will find more information on how to use the Run Tool here.

If you want a fully documented integration manager and scheduling system then Ouvvi will be the tool to use.

Ouvvi enables you to bring together multiple projects, steps and even Run Tool projects into the workspace and then schedule these to run using either time based or event based triggers. The whole environment can be documented at a click of a button and you can add additional description to each of your projects and steps to help other users understand their function.

You can find more information on how to get started with Ouvvi in our training pages.

Troubleshooting

Connection Erroring a day later

If you find the connection errors on the second day that the project runs then it is likely you didn't save it to the connection library. The connection library handles getting the refresh token for you whenever it is needed.

Alternatively it might be that you didn't add the correct scope when creating the app in Salesforce. Please take a look at your app registration in Salesforce to ensure that the Perform requests on your behalf at any time (refresh_token, offline_access) scope has been added.

What API and API version does the connector use?

The Salesforce connector uses the REST API and binds to the latest version available, so you remain ahead of any deprecations.

If you have any questions on how to use Data Sync, the Run Tool or Ouvvi, or need more help connecting to Salesforce please reach out to us at support@simego.com.

Synchronising Data with Dynamics Navision and OData

info@simego.com (Rebecca Allen) — Tue, 05 Apr 2022 10:00:00 +0000

With Data Sync you can import and export data to and from your Navision objects/cards.

The following article covers a couple of considerations you need when planning your Navision integration, we go through using Ouvvi to manage and schedule the whole project, and how to configure an example Data Sync step to get data from a SQL table into Dynamics Navision.

Although we use the example of a SQL table in this article, you can use any of the available connectors to import and export data from Dynamics Nav. If there is not currently a connector for your system then you can always build a custom connector.

Here we cover how to connect using OData V4 however check out our other blog if you are looking to connect via SOAP Web Services.

Requirements
Considerations when Importing into Navision
Ouvvi Configuration

Connect to your Source Data
Connect to Navision
Map the Source & Target Columns
Preview the Results and Sync
Schedule
Exporting Data from Navision

Connect to Navision
Connect to your SQL Table
Map the Source & Target Columns
Preview the Results and Sync

Miscellaneous

Filtering the Results
Troubleshooting
Clearing the Cache in Data Sync

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
Installed Ouvvi Automation Server
Access to Dynamics Navision
Access to a SQL table

If you do not have Data Synchronisation Studio you can get a free evaluation edition. To test Ouvvi during a trial please contact us to discuss upgrading your evaluation license.

Importing Data into Navision

You can quickly and easily synchronise your business data into Dynamics Nav when you use Data Synchronisation Studio.

You need to consider which columns you want to be updating within the cards and if those columns relational fields that lookup their values in a different entity. For example, invoice needs the customer to already exist in the customer entity. If the value doesn't exist then an error will be thrown The remote server returned an error: (400) Bad Request. So the customer entity needs to be updated before the invoice entity.

To get around this type of issue you need to look at the structure of your Navision and plan your integration accordingly. This is where Ouvvi becomes invaluable as you can contain each Data Sync project as a step, add descriptions to the overall project and individually on each step, and execute the whole process in a sequence. This ensures cards are updated in the correct order to prevent errors.

We came across a useful site for finding out the relationships between tables in Nav, if you want to take a look you can find it at this link: https://dynamicsdocs.com/nav/2018/w1/.

Below we cover creating your project and steps within Ouvvi and how to configure an example Data Sync step to connect to update data in Dynamics Navision.

Although we use a SQL table in this example you can use any of the other connectors as your source. This might be Active Directory, a SharePoint List, a CSV file or a Dynamics Entity to mention a few. Just select the connector you need, enter the required connection details and you're ready to go.

Ouvvi Configuration

We need to start by creating our Ouvvi project to hold each Data Sync step in the integration. To do this open Ouvvi and go to Projects > All Projects > New. Enter in a Name for your project and click Save to continue.

You then need to add a Data Sync step. For this integration project there are going to be multiple Data Sync steps that will need to be configured to connect to the source data and the relevant Navision entity.

For the purpose of this article we will only focus on the configuring of one step, but the process should be the same for each step.

To add a Data Sync step go to Add Step > Data Sync Project and then enter in a name for the project and click Save.

Then open the project by clicking Open Project.

When Data Sync has downloaded from Ouvvi and you open the file, you should find that it shows as connected to Ouvvi. You can check this in four locations:
1- The title bar
2- The connection library
3- The output window
4- The bottom right of the window

You can now configure the project within Data Sync.

Connect to your Source Data

In this example we will be configuring the project to import customers from a SQL table into the Navision Customers object.

Enter in the network name of your SQL Server, any credentials you might need to connect and set encryption or trusted certificate as required.

Click Connect & Create Library Connections to save this database connection to the connection library. This only needs to be done once per database and will make future project setup easier.

Select the database and table from the tree view to connect to, and if you are saving to the connection library another window will open where you need to enter a name in for the connection.

This will load the columns in your SQL table into the source window.

Connect to Navision

When connecting to Navision you need to have first published each entity you want to connect to.

You will need the OData V4 endpoint URL which will look something like: http://nav17:7048/DynamicsNAV110/ODataV4/Company('CRONUS%20UK%20Ltd.')/Customers and you need to edit this to remove the entity from the end of the URL.

Your URL should now look like http://nav17:7048/DynamicsNAV110/ODataV4/Company('CRONUS%20UK%20Ltd.')/. Enter this into ServiceURL field.

Be aware that if the account does not have permission to access the Nav server then the connection will fail and error.

Once these details have been entered select the entity/card to connect to from the dropdown. In this example we select Customers as we are exporting the Customer Card in Navision.

Click Connect & Create Library Connection to save the connection to the connection library. This only needs to be done once per site as you will be able to access the other entities/cards from the connection library.

As the project has been opened from Ouvvi, the connection will be stored in the Ouvvi connection library.

If you have already saved your connection to the connection library then you can expand the connection and select the object you want to connect to.

NB. If you publish more Nav entities after you have connected you may need to delete the cache file and refresh your connection. You can find the cache folder by going to Tool> Open Schema Cache Folder, then delete the cache file for your Navision connection.

Map the Source & Target Columns

You now need to configure the schema map so that the source columns are mapped to their corresponding column in Navision.

Some nav objects will have multiple key columns so be sure to map these where required.

Make sure that the data types are correct and that you have selected a unique key column that can be used to identify each record.

Navision can be tricky to integrate with so the initial project setup may have a bit of trial and error. You may find that some columns are not possible to write to or they may be expecting the data to be presented in a specific format. To diagnose try removing columns you suspect could be causing the error and add them back one at a time. You can also preview the records in Nav using the Preview B button to see how the data is being returned.

Preview the Results and Test

Once the mapping is complete you can click Compare A > B to preview the changes that need to be made. Here you can preview additions, updates and delete actions that need to be made. In this example there are 90 records to add and 1 to update.

Please note that deletes are disabled by default but will still show in the results. To enable deletes please set EnableDelete to true in the target connection properties.

Click onto each option will enable you to view the changes. For updates the changes will be highlighted in yellow.

You can either wait to synchronise everything in sequence or you can run the synchronisation now. We do recommend running a test to make sure there are no issues as Navsion can be temperamental to write data to.

To test a few records start by deselecting the update checkbox and click onto the adds, then click Clear ALL to clear the checkboxes and select one or two records to add. Then click Synchronise and Start to begin the sync.

Be aware that if you are testing within the Cronus environment there are data restrictions. For example you can only add dates that are in Nov, Dec, Jan or Feb. Anything outside of this will cause an error to be thrown. In a fully licensed Nav environment you will not have this issue.

Once you are confident the columns are mapped correctly and Navision will allow you to write to the columns you have selected you can save the project and return to Ouvvi.

You can then continue configuring the other projects, add a trigger to schedule it or manually run the Ouvvi project with the Start Project button.

Your Ouvvi project might look similar to the below image once you have added all of your required steps.

Schedule

You can now add a trigger to your project to run it on a schedule, for this example we are using the start of day trigger and the end of day trigger. This will run the project at the times configured in the Ouvvi Settings.

By default these are 8am and 6pm but you can always change them to suit your needs. To do this go to Settings > System Settings > Edit and then change the start and end times.

Please be aware of any time zone settings and if your Navision or Server is using a different timezone eg. one might be in UTC and the other in local. This can make the trigger look like they are starting at the wrong time but are actually just running in the wrong time zone.

Please see our training page to learn how to create a new trigger.

Exporting Data from Dynamics Navision

If you need to get data out of Navision and integrate it with another business system then Data Sync makes this simple.

You can export the data previews to files such as XML, CSV, Excel and JSON for one off exports by using the buttons in the preview window.

Alternately you can create a new SQL table, CSV File or XML dataset for example from the tools menu, or you can connect to an existing system using any of the connectors available.

In the example below we are setting up a project that will export the Navision item card to an existing SQL table on a regular basis.

Connect to your Navision Object

To connect to Dynamics Navision in Data Sync you need to have first published the card in Navision so then you access the records via the OData V4 API.

In our documentation we cover how to publish entities in Navision..

You will need the OData V4 endpoint URL which will look something like: http://nav17:7048/DynamicsNAV110/ODataV4/Company('CRONUS%20UK%20Ltd.')/Lists and you need to edit this to remove the entity from the end of the URL.

Your URL should now look like http://nav17:7048/DynamicsNAV110/ODataV4/Company('CRONUS%20UK%20Ltd.')/. Enter this into the ServiceURL field.

Make sure to enter any credentials into the relevant credentials field. i.e. Windows credentials to access your Navision Server need to be entered into the Windows field. Leaving this blank will use the current user crednetials to access. If the project is being run in Ouvvi, this will be the service account. Be aware that if the account does not have permission to access the Nav server then the connection will fail and error.

Once these details have been entered select the entity/card to connect to from the dropdown. In this example we select Items as we are exporting the items from Navision.

If you have opened the project from Ouvvi then the connection will be stored in the Ouvvi connection library.

If you have already saved your connection to the connection library then you can expand the connection and select the entity you want to connect to.

Connect to your SQL table

To connect to a SQL table click onto Connect Datasource in the target window to open the connection window. Then navigate to SQL Database > Microsoft SQL Server (SQL Client).

Enter in the network name of your SQL Server, any credentials you might need to connect and set encryption or trusted certificate as required.

Click Connect & Create Library Connections to save this database connection to the connection library. This only needs to be done once per database and will make future project setup easier.

Select the database and table from the tree view to connect to, and if you are saving to the connection library another window will open where you need to enter a name in for the connection.

This will load the columns in your SQL table into the target window.

Map the Source & Target Columns

You now need to configure the schema map so that the columns in Navision are mapped to their corresponding column in your SQL table. Make sure that the data types are correct and that you have selected a unique key column that can be used to identify each record.

Compare the Results and Sync

Once the mapping is complete you can click Compare A > B to preview the changes that need to be made. Here you can preview additions, updates and delete actions that need to be made.

Please note that deletes are disabled by default but will still show in the results. To enable deletes please set EnableDelete to true in the target connection properties.

Click onto each option to view the changes that will be applied to your SQL Table. For updates the changes will be highlighted in yellow. In this example we have 226 records to add.

To synchronise the results click onto the Synchronise button in the toolbar and then Start to begin the sync. Your items will then be added to your SQL table and available to use.

Miscellaneous

Below you will find a few additional points on filtering the results, a few troubleshooting tips if your project is throwing an error, and how to clear the cache if you make changes to the connection.

Filtering the Results

You can filter the results before exporting by using the filter box built into the source window.

Just enter in a C# Expression that returns true to only return a specific set of results. For example we could filter the items list to return only items of a specific type using the following expression:

Type == "Inventory"

You can then preview the data using the preview A button in the schema map before synchronising to check the correct results are being returned.

Troubleshooting

Method Exception Failure

If you are getting a Method Exception Failure then it is most likely down to not publishing the correct object. For example for Items you need to publish and connect to the Items card.

Bad Request Error

If you keep getting Bad request as an error then you need to take a look at what is being written to as there are a couple of reasons for this error.

There is a field that cannot be written to included in the schema map
A field mapped in the schema map throws an action in Navision which cannot be handled on the API
There is a required field missing from the schema map so the record cannot be created.

One way to diagnose which field is throwing the error is to remove fields and add them back one at a time, synchronising one record at a time.

Another way to work out what is wrong is to create a simple record in Nav and see what data is required in the UI. This will help you work out what is needed when writing to Navision.

You can also enable trace in the connection properties to see the data being sent in the output window.

Clearing the Cache in Data Sync

If you have published a new entity or made a change to the connection, you will need to clear the cache file before the new changes will be visible.

To remove the cache file go to Tools > Open Cache Folder and then delete the Navision file. Alternatively the cache folder can be found at C:\ProgramData\Simego\DataSyncStudio30\Cache.

You can then go back to Data Sync and try connecting to Navision again. Your new entity should now be visible in the list.

If you have any questions please reach out to us at support@simego.com.

Scheduled Import and Export for Dynamics 365

info@simego.com (Rebecca Allen) — Tue, 03 May 2022 12:48:01 +0000

When integrating with Dynamics 365 (aka. CRM, dataverse or CE) you want to be able to set your projects up once and then be able to repeat them at regular intervals. You may have other systems relying on Data from Dynamics 365 or need to update Dynamics 365 with data from another source.

For example you might have a SQL Database where account records are updated and these need feeding into Dynamics. Alternatively you might need to get the contact records, that are updated regularly by the sales team, into another system to ensure it is kept up to date.

For the following example we are going to use Ouvvi to schedule, manage and automate a series of projects that update Dynamics on a regular basis. We cover how to configure the projects so that you can link two entities together, i.e. account and contact records.

We will also cover configuring projects to export data from Dynamics into a SQL table, XML file and Excel file and having these run at the start of the day.

Requirements
Considerations when Importing into Dynamics
Ouvvi Configuration

Connect to your Source Data
Connect to Dynamics
Create the Lookup to Account
Map the Source & Target Columns
Preview the Results and Sync
Schedule
Exporting Data from Dynamics

Connect to Dynamics

Export Options

Single Export to File
Export to a New Data Source
Export to an Existing Data Source

Troubleshooting

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
Installed Ouvvi Automation Server
Access to Dynamics 365

If you do not have Data Synchronisation Studio you can get a free evaluation edition. To test Ouvvi during a trial please contact us to discuss upgrading your evaluation license.

Considerations when Importing Data into Dynamics

You can quickly and easily synchronise your business data into Dynamics 365 (CRM / CE/ Dataverse) when you use Data Synchronisation Studio.

You need to consider which columns you want to be updating within the entity and if those columns are relational fields that lookup their values in a different entity.

For example, contact needs the account to already exist in the contact entity. If the value doesn't exist then an error will be thrown The remote server returned an error: (400) Bad Request. So the account entity needs to be updated before the contact entity.

To get around this type of issue you need to look at the structure of your Dynamics 365 site and plan your integration accordingly.

This is where Ouvvi is invaluable as you can contain each Data Sync project as a step, add descriptions to the overall project and individually on each step, and execute the whole process in a sequence. This ensures entities are updated in the correct order to prevent errors.

Below we cover creating your project and steps within Ouvvi and how to configure an example Data Sync step to connect to update data in Dynamics 365.

Ouvvi Configuration

For the purpose of this article we will only focus on the configuring of one step, but the process should be the same for each step in the project.

To add a Data Sync step go to Add Step > Data Sync Project and then enter in a name for the project and click Save.

Then open the project by clicking Open Project.

You can now configure the project within Data Sync.

Connect to your Source Data

In this example we will be configuring the project to import customers from a SQL table into the Dynamics Customers entity. Please note that you will need to have configured the Accounts import first to ensure this runs without errors. As if you try to lookup the account and it doesn't exist in Dynamics an error will be thrown.

To do this we need to start by connecting our Source window to our SQL table. Click onto Connect Datasource to open the connection window, then navigate to SQL Database > Microsoft SQL Server (SQL Client).

Enter in the network name of your SQL Server, any credentials you might need to connect and set encryption or trusted certificate as required.

Click Connect & Create Library Connections to save this database connection to the connection library. This only needs to be done once per database and will make future project setup easier.

Select the database and table from the tree view to connect to, and if you are saving to the connection library another window will open where you need to enter a name in for the connection.

This will load the columns in your SQL table into the source window.

Connect to Dynamics

Now we need to connect to Dynamics. If you haven't configured your connection to Dynamics please see our documentation pages for the full connection guidance details to create your Azure Application.

Once you have the OAuth connection details from your Azure app click onto the target window and go to Microsoft Dynamics > Microsoft Dynamics 365. Enter int he URL to your Dyanmics site and Data Sync will add the relevant parameters to connect to the SOAP/WCF Service. Select the custom OAuth Application type from the drop-down and then add your connection details.

In this example we are using the client credentials flow so we add the ClientID, Client Secret, Token URL V2 endpoint and change the grant type to client_credentials. Make sure to clear the Redirect URL of any text and then click Authorise Connection

You can then select the entity to connect to, in this case Contact and then save the connection to the connection library by clicking Connect & Create Library Connection. As we are connected to Ouvvi, this will save the connection to the Ouvvi Connection Library. You only need to save the connection to your Dynamics site once as you will be able to access all of the entities from the one connection.

If you already have saved your connection to the connection library, you can either drag and drop the contact entity onto the target or right click and select Connect to Target (B)

Create the Lookup to Account

Now we need to lookup the account IDs in the account entity of Dynamics. To do this we can drag and drop the account entity from the connection library onto the account id column in the source.

Alternatively you can lookup the account by selecting the column to lookup and then clicking onto the lookup button in the source window. You then select the connection to Dynamics from the connection library tab and select the entity to connect to.

Then to configure the lookup select the target column that will contain the values to be looked up and click OK.

You can then access all of the columns from the entity you looked up the values for and add these to the schema map as needed. For this scenario we need the AccountID to link the contact to the right account in Dynamics.

Map the Source & Target Columns

The next step is to configure the schema mapping to make sure the columns you need are added and mapped to the corresponding fields in Dynamics.

You also need to make sure that a key column has been selected that can be used to identify each record as unique. In this case this is the ID column as this will be unique. You could also use email, or use a composite key such as the lookup1_accountid and the email column if you don't have an external identifier available.

Preview the Results and Sync

We now need to preview the results and test synchronise a couple of records to make sure the project is configured correctly.

To do this click onto the Compare A > B button in the toolbar. This will open the comparison results tab where we can preview the changes that need to be made to make the target (Dynamics) the same as the source (SQL).

You will be able to preview any adds, updates or deletes that need to be made by clicking onto each option. Please note that deletes are disabled by default but will still display in the results. If you want to enable deletes then please set EnableDelete to true in the target connection properties.

For this example we have 546 records to add, 0 to update, and 1 record to delete. As deletes are disabled this record will not be included in the sync.

If there are no linked accounts for your contacts, then the account does not currently exist in Dynamics. The corresponding accounts should be added in the account project that will be run before this one. So when the whole integration project is run from Ouvvi, the accounts will be added first and the correct AccountID will be added to these records. If the account is added at a later date then this will be updated in a future run of the project.

To test that the sync will work as expected we can deselect the updates and additions and just add a single record from each. To do this click onto the additions (or updates) and select deselect all. Then check the checkbox against a record.

You can then click Synchronise and Start to sync the record selected.

As that went through without an error, we can now save the project and go back to Ouvvi. Refreshing the page in Ouvvi will show the project details, and you can now configure your remaining projects for the integration.

Once all the projects are added you can run the whole sequence by clicking onto the Start Project button.

Next we will configure the trigger to schedule the project automatically.

Schedule

Once all the projects have been built and added to Ouvvi you can schedule it to run on a regular basis. For this example I want the project to run at the start of the day.

To do this we need to add the Start of Day Trigger by going to Add Trigger > Start of Day.

The start of day time is configured in the system settings of Ouvvi, and by default this is set to 8am. To change this to a different time please change the time in the system settings.

Once the trigger is added to the project it will run everyday at the specified start of day.

Your final Ouvvi project might look something like this:

Exporting Data from Dynamics

As well as being able to import data into Dynamics it is likely you also want to export data from your Dynamics entities. Below we will briefly cover the options you have to setup a regular export of your Dynamics data.

You will want to contain your export projects within Ouvvi so that they can be automated and run at a set schedule, please see the details in the previous section on how to add and configure Ouvvi projects. You can see the details on adding a trigger here or for more detailed documentation please take a look at our training pages.

Below I am assuming you have already created an Ouvvi project to hold your Data Sync projects and just need to configure an export project.

Connect to Dynamics

If you haven't configured your connection to Dynamics please see our documentation pages for the full connection guidance details to create your Azure Application.

If you already have saved your connection to the connection library, you can either drag and drop the contact entity onto the target or right click and select Connect to Target (B)

Then add the columns you want to be included in the export to the schema map.

Export Options

You have a couple of options to export your data. You can either do a one off export and export to a CSV, Excel, XML or JSON file from the preview tabs; you can use the quick functions from the tools menu e.g. Create CSV File to create a new data source based off the schema properties; or you can connect to an existing data source.

Single Export to File

To do a one off export from the preview just click onto the Preview A button in the schema map to preview the columns added to the schema map. Alternatively you can export every column from your entity by clicking onto the Preview button in the data source window.

In the preview window you can then choose between exporting to an XML, CSV, Excel or JSON file. Just click onto the corresponding button and choose where you want the file to be saved. Your data will then be exported to this file.

Export to a New Data Source

To create a new data source using the tools menu, just click onto Tools and select the option you want to use. For this example we will create a new CSV file so can use the Create CSV File option.

You just need to define a name for the file and select where it should be created and this will be loaded into the data source B window. The columns created will be based upon the columns you added to the schema map previously.

To synchronise the data to the file just click Compare A > B > Synchronise > Start. You can then save and schedule this project to run in Ouvvi whenever you need it to.

Export to an Existing Data Source

To export to an existing Data Source you need to connect the target to that source. For example this might be a SharePoint list.

Click onto the target window to open the connection window and go to Microsoft SharePoint > Microsoft Office 365 SharePoint Online and add the required connection details. Full connection details can be found in our documentation pages.

Map the columns from Dynamics to the corresponding columns in SharePoint and make sure to define a key column that can be used to uniquely identify each record. You can select a combination of columns to make a composite key if you do not have a column in SharePoint that can take the Dynamics ContactID.

You can then preview the results by clicking Compare A > B and either run the sync now or save the project and return to Ouvvi to schedule it to run at a later time.

Troubleshooting

Below are a couple of errors/issues you might come across and how you can go about resolving them.

Permissions - Execution Context

If you are finding that your project is failing in Ouvvi but not when you run it locally, then it is likely that your service agent does not have permission to access the resource you are connecting to.

You can see what the error is by clicking onto the log instance and viewing the log output. For example, this project connects to a SQL database but the user LAB\REBDEV$ does not have permission to access the database so the project has failed.

To fix this you can either add the service account to the SQL database or you can change the execution context of the project. To do this click onto the execution context to add specific user details for the project to run under.

Greyed Out Start Button

Another common issue is that the start button of the project is greyed out and cannot be clicked. This is usually because the services are not running or the instance does not have an active license key.

To start the services open the deployment manager, select your Ouvvi instance and click the green play button. Alternatively right click and select Start Service.

To check if there is a license key activating the instance please go to Settings > Register License Key and you should see a green tick next to the license key and a support and maintenance date on the page.

If you have any questions or want to know more please email us at support@simego.com or checkout our other blog posts.

Importing Documents as Attachments in Dynamics 365

info@simego.com (Rebecca Allen) — Fri, 06 May 2022 10:00:00 +0000

Data Sync has many uses and if you are already integrating with Dynamics 365 (CRM) then you can quickly import documents as attachments into your Dynamics environment in a few steps.

The following page will show you an example of how you can import files from your file systems into the attachments entity in Dynamics and link these to a corresponding account.

Requirements
Connect to your Source Data
Create the Metadata Columns

Connect to Dynamics
Lookup Account
Configure the Schema Map
Compare and Sync

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
Access to Dynamics 365
Configured your connection to Dynamics 365 in Data Sync

If you do not have Data Synchronisation Studio you can get a free evaluation edition.

Connect to your Source Data

The following example is using the file explorer as the source and assumes you have your files arranged into folders. Where the folder name is the name of the account that you want to link the file to.

To get started you need to open Data Sync and connect your source to your file system. To do this open the connection window and go to File System > Files, then click onto the ellipsis in the path field to browse for your folders.

Then click Connect to add the connection.

Create the Metadata Columns

In order to pass the file data to Dynamics this needs to be in the format of a Base64 encoded string. With Data Sync this can be converted using calculated columns and a simple expression.

Don't forget to escape the backslashes in the path otherwise your expression will not work

TOBASE64(System.IO.File.ReadAllBytes("C:\\Users\\Rebecca\\Documents\\ExampleData\\Documents\\" + FullFileName))

We also need to include an Object ID Type Code of account, as we will be returning the id of the account objects that are specified in the path to link the documents to each account. This can be done with another calculated column where we just return the string account.

Connect your target to Dynamics

Next you need to connect your target to the annotation entity of your Dynamics site. As you have already pre-configured your connection and saved this to the connection library you can simply expand your connection and right click on the annotation entity and select Connect to Target (B).

Lookup Account

You then need to lookup the account name in the account entity of Dynamics using calculated columns. The expression we use for this is:

LOOKUPB("accountid", "account", KV("name", Path))

This will lookup the account id in the account entity when the account name matches the folder name in the path.

Configure the Schema Map

Now add the columns to the schema map and map them to their corresponding column in Dynamics. Make sure to set the composite key columns so that the records will return as unique. In this example we use FileName and the lookup column objectid.

The schema map should then look similar to:

Compare and Sync

Now run the compare by clicking onto the Compare A>B button in the toolbar and preview the results.

As we are adding documents we can see that we have X many documents to add to Dynamics by clicking onto the add item in the results list. If we were updating an existing attachment document then this would show in the update item.

Please note that deletes are disabled by default but will still show in the results if the record is not present in the source system.

To enable deletes please set EnableDeletes to True in the target connection properties.

If the data is presenting as expected then you can synchronise the results. To do this click Synchronise and then Start to begin the sync.

Your documents will now be added to Dynamics as attachments to each account specified.

If you want more ideas on how you can integrate with Dynamics take a look at our other blog posts. As always if you have any questions please send an email to support@simego.com and we will be happy to help.

Setting Content Types on SharePoint List Items

info@simego.com (Rebecca Allen) — Tue, 10 May 2022 10:00:00 +0000

When trying to update SharePoint Document Libraries or SharePoint lists on a regular basis it is useful to be able to set sepcific metadata.

One type of metadata that you might be interested in setting is Content Types. Content types allow you to set templates, retention policies and define a group of columns for that type.

The following page will show you how you can set the Content Types on SharePoint list items and Document Library items using Data Synchronisation Studio.

Requirements
Getting Started
Calculated Columns

Lookup to a Custom List
Preview & Sync

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
Access to SharePoint Online
Configured your connection to SharePoint Online in Data Sync

If you do not have Data Synchronisation Studio you can get a free evaluation edition.

Getting Started

To set the content types for an item you need to return the content-type id value from SharePoint.

If you do not know the ID value that SharePoint is expecting we recommend setting content types for a couple of items manually and then preview the results returned from SharePoint in Data Sync. You can then make a note of each ID and update your source data accordingly.

Some common ID types are:

Item: 0x0100860DBE824AC9614589A21DCD29DB5448007A823F8834051D4F986EA19A5812FF09
Issue: 0x01030066DC9B12B8DE924CBD18288FA706EA1C
Message: 0x010700B796C8AF0D82994A86E8DAF1B01A3FFB
Task: 0x0108004931F959CF5D224F8A156604BB7B6005

You have a couple of options when updating the source data:

Manually update the source data with the correct ID
Use calculated columns to replace the text value with the ID that matches
Create a list with the types and IDs and lookup the ID

The manual process is self explanatory, but you can find descriptions on how you might configure the other options below.

You need to start by configuring your Data Sync project so that you are connected to your source data and the target is connected to your SharePoint list you want to update. Then you can choose between creating a calculated column or creating a lookup.

Calculated Columns

You can use calculated columns and a CASE function to return the relevant ID when the content type name in the source data matches the case in the function.

The expression below returns the ID for message, item, issue, and task and if the source column contains something that doesn't exist in the expression it will return a default value of the issue ID.

CASE(ContentTypeName, "0x0100860DBE824AC9614589A21DCD29DB5448007A823F8834051D4F986EA19A5812FF09",
KV("Message", "0x010700B796C8AF0D82994A86E8DAF1B01A3FFB"),
KV("Item", "0x0100860DBE824AC9614589A21DCD29DB5448007A823F8834051D4F986EA19A5812FF09"),
KV("Task", "0x0108004931F959CF5D224F8A156604BB7B6005"),
KV("Issue", "0x01030066DC9B12B8DE924CBD18288FA706EA1C"))

The syntax for this is:

CASE(ColumnName, DefaultValue, KV(Value, ValueReturned), KV(Value2, ValueReturned2), KV(Value3, ValueReturned3))

Clicking OK will create the column in your source window and you can then add this column to the schema map and map this to the Content Type ID column in SharePoint

Lookup to a Custom List

If you have a list of content types you want to use within your list or library then you could store these and their corresponding ID's in another list or datasource and perform a lookup on each type.

Please be aware that if there is a type that does not exist in the list you create and therefore has no ID being returned SharePoint will default the type to Item.

Here I have created a SharePoint list to hold the content type names and their corresponding IDs and I will lookup the type name I have listed in my source data against this list to return the relevant ID. This list has been named ContentTypes.

To create the lookup just drag and drop the ContentType list onto the column in the source data that contains the name of the content type, in this example it is ContentTypeName.

This will then open the lookup configuration window where you can define that it is the content type that is being looked up. This should look similar to the below:

Click OK to create the lookup and now you can access all of the columns within the ContentTypes list. I'm adding the Content Type ID to the schema map by expanding the lookup columns and checking the checkbox.

You can then map this column to the Content Type ID in your SharePoint list, and add any additional columns you need. Please make sure to select a key column that can be used to identify each record, in this case it is Title.

To show you that the lookup is working I have added in the column from the source that returns the content type name and previewed the data to show the different IDs that are returned.

Make sure you remove the Content Type Name column before synchronising as this will throw an "Invalid Request Error" if you try to synchonise to it.

Preview & Sync

We can then preview the changes to be made to add these to SharePoint by clicking Compare A > B.

This will show us any add, update or delete actions that need to be performed to make the SharePoint list match the source data.

Please note that deletes are disabled by default but will still show in the results if the record does not exist in the source data set.

To enable deletes please set EnableDelete to True in the target connection properties.

There are 6 records to add and we can now synchronise the results by clicking Synchronise and then Start.

The records will then be added and updated in SharePoint with the corresponding content type.

You can now save the project to be used again in the future or you can schedule it to run on a regular basis using either Ouvvi or the Run Tool.

Creating Dynamics 365 Activity Records

info@simego.com (Rebecca Allen) — Tue, 17 May 2022 12:55:33 +0000

To create activities such as phonecalls, appointements, tasks and emails in Dynamics you need to provide the GUID of the item you are adding and sync this to the corresponding entity. For example tasks will need to be synced to the Task entity, phone calls to the Phonecall entity, appointments to the Appointment entity and so on.

Below we will cover a few of the considerations for each activity type and how you might map the columns

Requirements
Considerations
Adding Phonecall Activities
Adding Appointment Activities
Adding Email Activities

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
Access to Dynamics 365
Configured your connection to Dynamics 365 in Data Sync

If you do not have Data Synchronisation Studio you can get a free evaluation edition.

Considerations

If you are regularly updating activity records in Dynamics then we would recommend containing all of your projects within an Ouvvi project. This will enable you to run each project on a schedule and run them in sequence. You can additionally add a step to notify you that the project has completed and the details of what changed.

You can find more details on the project status report step in our documentation. Alternatively you could also use the Run Tool to schedule your projects.

If your source data is a selection of different activity types then you can filter the source records to only return the activity type you are trying to update. For example you could filter the source with the expression activitytype == "phonecall" to filter the results to return only phonecalls.

The below examples assume you have already connected to Dynamics and saved the connection to the connection library, and that you have your source connected to your source data.

Adding Phonecall Activities

To write phone call activities you must connect the target to the Phonecall entity and provide the data in the format Entity Name| GUID of the Item. We recommend adding a record manually so then you can preview the format that Dynamics is expecting.

For phone call activities you need to supply the following fields at a minimum:

from - The user that made the call in the format systemuser|systemuserid
to - The account or contact that the call relates to in the format account|accountid or contact|contactid
subject - The subject of the call
activityid - A guid to uniquely identify each activity

You can then additionally add in other metadata such as a description, duration, date, ownerid, ownertype etc.as you need.

A sample of the data we are returning for this can be seen below:

You will need to lookup the account name in your Dynamics site to return the account id so that we can present the to field in the format expected in a few moments.

If your data is linking to a contact record rather than account then you can follow the same details as below but lookup the contactid in the Contact entity and return the id in the format contact|contactidGUID e.g. contact|14daefde-d3d1-ec11-a7b5-00224880a494.

Drag and drop the account entity onto the source column that contains the account name, e.g. Account.

This will open the lookup configuration window where you will need to specify the column within the account entity to lookup the source values. In this case name. Click OK to save this and create the lookup.

Now repeat this for the system user entity to lookup the user by fullname.

We can now use calculated columns to return the entity name and GUID in the format we mentioned earlier.

To do this click onto the fx button in the datasource toolbar, enter in name for the column and make sure the data type is set appropriately.

CONCAT("account|", STR(Lookup1_accountid))

Alternatively you can do the lookup in the function rather than creating it separately:

CONCAT("account|", STR(LOOKUPB("accountid", "account",WHEN("name", Account))))

You need to do this for both the to and from fields.

Once they have been created we can then map these columns to the schema map and link them to their corresponding field in Dynamics. You can now add any additional columns you might need and make sure to set a unique key column that can be used to identify each phone call activity.

To run the sync click Compare A > B to view the changes that need to be made, and check the preview to ensure the data looks as you expect. Here we have 5 phonecalls to add, note that deletes show in the results but are disabled by default. To enable them please set EnableDeletes to True in the target connection properties.

Once you are ready click Synchronise and Start to begin the sync.

You can now save you project and move on to creating the projects to sync other activities.

Adding Appointment Activities

Adding appointments as activities to your Dynamics site is very similar to adding phonecalls, the main difference being you connect to the appointment entity as your target and there are different required fields.

At a minimum the columns you need are:

activityid - A guid to uniquely identify the activity.
organiser - The user the appointment is with in the format systemuser|systemuserid
requiredattendiees - The contact or account that the appointment is with in the format contact|contactid or account|accountid
scheduledstart
scheduledend

If you want to change the status of the appointment you can do so by adding statecode: 1 is completed, 2 is cancelled and 3 is scheduled.

A preview of the data we are using for this example can be seen below:

We need to lookup the contactid and the systemuserid and then format the values.

First start with the lookups. Drag and drop the contact entity onto the contact column in the source, and then configure the lookup to target the contact full name as this is the data we have in the source file.

We can then repeat this for system user. Drag and drop the systemuser entity onto the organiser column and then configure the lookup to target the full name of the user.

Now we can create the calculated columns to format the values. Lets start with the organiser. Create a calculated column and use the following expression to concatenate systemuser| with the systemuserid.

For the contact, create a calculated column and use the following expression to concatenate contact| with the contactid.

You can now add these new column to the schema map alongside the other required fields and any other fields you want to include. Your schema mapping should now look similar to this:

To finish simply compare, preview & check the results, and then synchronise the changes to import your appointment activities.

Adding Email Activities

To synchronise emails as activities in Dynamics you need to connect your target to the email entity and include a few more required columns. The example below uses a document to contain the message details rather than connecting the source to exchange.

The columns you need to include are:

activityid - A guid to identify each activity as unique
from - The system user who sent the email in the format systemuser|systemuserid
to - The contact that the email is being sent to in the format contact|contactid
subject - The email subject
description - The body of the email in either HTML or plain text
regardingobjecttype - The type of object the email relates to in this case a "contact"
regardingobjectid - The ID of the contact it relates to

By default the email will be set to draft status, however if you want to define this then you can do so by setting the statuscode: 1 is draft and 3 is sent.

Below is a preview of the data we are using in this example.

We need to start by looking up the contact id and the sender id and then format this into the format object|objectid.

To add the lookup to contact drag the contact entity from the connection library onto the contact column in the source.

Then configure the lookup so that the value is being looked up in the correct column. In this example as we have the full name we join the lookup on fullname.

To add the lookup to the system user drag the systemuser entity from the connection library onto the sender column in the source.

Then configure the lookup so that the value is being looked up in the correct column. In this example as we have the email of the sender we join the lookup on domainname.

We now need to use the lookups to format the to and from columns. This can be done using calculated columns and the functions described below.

To format the to column we can use calculated columns with the expression:

CONCAT("contact|",STR(Lookup1_contactid))

To format the from column we can use calculated columns with the expression:

CONCAT("systemuser|", STR(Lookup2_systemuserid))

These can now be added to the schema map and linked to their corresponding column in Dynamics. Your schema map should look something similar to below with a key column selected.

You can get the contactid to map to the regardingobjectid column from the lookup. Just expand the lookup columns and select the Lookup1_contactid column.

If you need to add multiple recipients you can do so by supplying them separated by semi colon e.g. contact|77962dc3-fcd5-ec11-a7b5-000d3abdf148;contact|969924e0-7c2c-4e85-be9d-000d3abdf148

You can now compare and sync as normal to import your email activities.

Don't forget to save your project so that you can use it again at a later date. If you upload it into Ouvvi or the Run Tool it can be scheduled to run on a regular basis.

If you have any questions please send us an email at support@simego.com.

Performance and Optimisation Strategies for Dynamics Synchronisations

info@simego.com (Rebecca Allen) — Thu, 26 May 2022 12:55:33 +0000

Data Sync is fantastic at getting data in and out of Dynamics, however it is limited by the Dynamics API. When uploading large volumes of data you need to optimise your settings so then you don't hit the Dynamics threshold limits and then get throttled, which will slow your synchronisation to a snails pace.

The following page covers some considerations and tips on how to make the most out of Data Sync and optimise your Dynamics 365 integrations.

API Limits
Threads & Batch Size
Real-Time Data

API Limits

The Dynamics / PowerPlatform implements request limits to maintain a reliable service for customers. This is a complex area and is based on the license type and other factors this may also change at any moment due to Microsoft changing the service. You can read more on their current limit allocations within the Microsoft Documentation pages here.

Typically you will see the following errors in the log if your hitting the limits:

RateLimitExceeded
TimeLimitExceeded
ConcurrencyLimitExceeded

Each of these error are a temporary error that occurs when the Dynamics API is throttled. We have built into Data Sync a feature so that it will automatically back off and try the request again up to 5 times.

In general its better to avoid hitting the API limits and use lower batch and thread settings. If you keep hitting the limits then this will be slower overall due to continually having to back off the requests.

Threads & Batch Size

Within the Dynamics connector you can define the number of threads and batch size to use when synchronising the changes.

If you increase the Batch size to be greater than 1 then the requests are sent as ExecuteMultipleRequests messages. You can send a maximum of 1000 requests in a single message. We recommend using a batch size of 10 or less if your using a high thread count otherwise 50 should work well. Using the maximum is generally really slow and will not help your synchronisation performance.

If you increase the Thread count to be greater than 1 then requests will be sent in parallel. If the batch size is larger than 10, alongside an increased thread count, you may see ConcurrencyLimitExceeded errors. This is because by default you can only run two ExecuteMultipleRequests calls at a time unless you contact Microsoft Support and get this limit increased.

A baseline configuration to start with is a thread count of 4 and a batch size of 10. If your not seeing any errors after say 20K records then you can try increasing the thread count.

Real-Time Data

Getting real-time integrations is hard, however you can get very close by using Ouvvi in combination with Data Sync in incremental mode with an appropriate filter and a Trigger to detect the changes. Using this method will typically apply changes to the target within 1 minute.

You would start by creating a project in Ouvvi to hold your Data Sync project(s), then add a Data Sync step and configure the project to do your Synchronisation.

You can then set incremental mode by going to File > Properties > SyncOption and setting the value to SyncAtoBIncremental.

Alternatively you can set incremental mode from the main toolbar.

You would then configure your Data Sync project as normal applying any filters to reduce the datasets being returned, and save your project back to Ouvvi. By reducing the dataset being returned you will speed up the synchronisation thus allowing you to get closer to real-time updates. You could use the modifiedon column from Dynamics to filter for records that were modified today, perhaps within the last hour.

Once your project has been created and saved back to Ouvvi you can configure either a time trigger, to run your project at a regular interval, or you can use an event trigger to run your project when a change is detected e.g. in a Dynamics entity. Event triggers are called every 30 seconds by Ouvvi to check if there have been any changes, this is how you can get close to real-time synchronisations. Every time a change is detected in an entity (the modified timestamp changes) the project will be automatically run.

Making Dynamics 365 Items Inactive rather than Deleting them

info@simego.com (Rebecca Allen) — Mon, 27 Jun 2022 12:55:33 +0000

When integrating with Dynamics you might need to mark items that are no longer needed as inactive rather than deleting them completely. Logically making something inactive is easy but the physical difference when synchronising items is more involved.

The following blog will discuss how you can set entity items to inactive rather than deleting them. We will be synchronising a Supplier table from a SQL database into the Account Entity in Dynamics 365.

Requirements
Considerations
Project 1 - Adding New and Updating Existing Records

Filter Target for Active and Supplier Records using FetchXML
Set the CustomerTypeCode
Configure the Schema Map
Compare and Synchronise

Project 2 - Setting the Inactive Status

Create the Lookup
Filter to only show items to be deleted
Set StateCode to Inactive
Set StatusCode to Inactive
Configure the Schema Map
Compare and Synchronise

Requirements

Before getting started you need to ensure you have the following:

Windows 10 or Windows Server
Downloaded & Installed Data Synchronisation Studio
Access to Dynamics 365
Configured your connection to Dynamics 365 in Data Sync

If you do not have Data Synchronisation Studio you can get a free evaluation edition.

Considerations

You can find more details on the project status report step in our documentation. Alternatively you could also use the Run Tool to schedule your projects.

These are the parameters we need to consider and set to make an entity item inactive or active:

When a Supplier exists in the SQL Table but is not active or Inactive in the Account entity, create it and set it to Active.
When a Supplier exists in the SQL Table but Inactive in the Account entity, set to Active and update any fields.
When a Supplier exists in the SQL Table and exists as Active in the Account entity, update required fields.
When a Supplier does not exists in the SQL Table and exists as Active set to Inactive
When a Supplier does not exists in the SQL Table and exists as Inactive, do nothing.

To do this we will create two Data Sync projects, which will allow us to create, update and mark items as inactive rather than deleting them.

The first project is a simple synchronisation between the SQL Supplier table and Active Accounts of type Supplier (customertypecode of 10) with deletes disabled (EnableDeletes=False). This adds any new entities and updates any details that might have changed.

The second project maps the Account entity to itself but uses a lookup to see if the underlying record still exists in the SQL Supplier table. If the lookup returns no data then it is missing and the status should be changed to InActive.

Project 1 - Adding New and Updating Existing Records

We need to start by creating the project that will add and update records within the Account entity.

To do this open Data Sync and connect the source to the Supplier table (SQL) and the target to the Account entity in Dynamics. Make sure to leave EnableDelete set to false, which is the default value.

Filter Target for Active and Supplier Records Only

We now need to use a FetchXML filter on the target data source (Dynamics) to remove any inactive records and to only return the supplier customer type (code 10).

You can add a FetchXML filter to the target by entering the filter into the FetchXMLFilter property field found in the target connection properties (below the columns). Click onto the ellipsis to open the editor and enter your FetchXML statement.

An example of the FetchXML filter you could use is:

<filter type="and">
<condition attribute="statecode" operator ="eq" value="0"/>
<condition attribute="customertypecode" operator="eq" value="10"/>
</filter>

This filter will be applied when we run the compare later on. You can check it is correct by previewing the target data using the preview button in the datasource toolbar.

Set the CustomerTypeCode

The next step is to set the customertypecode to 10 to set the records as Suppliers within the account entity. To do this we can create a calculated column in the source by clicking onto the fx button. Then enter in a name for the column e.g.customertypecode, set the data type to System.Int32 and enter in 10 for the expression.

Clicking OK will create the column which can then be added to the schema map.

Configure the Schema Map

We now need to configure the schema map so that the calculated column we just created is linked to the corresponding column in Dynamics (customertypecode). Then continue to add and map the columns from the source table to the Accounts entity in Dynamics as you need, and make sure to add a column that is unique to each record e.g. in this example is the SupplierID which is linked to the accountnumber field.

Your mapping should now look similar to this:

Compare and Synchronise the Results

The next step is to run the comparision to preview the differences between the source SQL table and the target Account entity.

Do this by clicking onto the Compare A > B button in the toolbar. In the screen capture below we can see that there are 29 records to add.

Now click Synchronize and Start to apply the updates to Account Entity.

Make sure to save the project so that you can use it again.

Project 2 - Setting the Inactive Status

We will now create the second project which will set the status to inactive on records that do not exist in the SQL supplier table.

Start by setting the Account entity from Dynamics as the source and target in your Data Sync project, and keep EnableDelete set to False. This will prevent any accidental deletion of your data.

We need to add the FetchXML statement we used in the first project, so that only the supplier records are targeted in the Account Entity. If we didn't filter the records then it would set all account records not included in the supplier table to inactive.

<filter type="and">
<condition attribute="statecode" operator ="eq" value="0"/>
<condition attribute="customertypecode" operator="eq" value="10"/>
</filter>

Make sure to add this to the source and target datasets.

Create the Lookup

The next step is to create a lookup on the account name in the source by dragging the supplier table from the connection library onto the name column in the source.

You then need to define the lookup configuration setting the target lookup column to be the CompanyName from the SQL table.

Here we have essentially joined the Account entity in Dynamics to the SQL Table.
If there is no corresponding record in the SQL table then we know it has been deleted and we need to set entity to inactive.

Filter to only show deleted items

We need to then add a filter to the source data to only return rows that have nothing returned from the lookup.

ISNULL(Lookup1_SupplierID)

Set State Code to InActive

We are now going to use calculated columns to set the statecode and statuscode to the correct values depending on whether the lookup can find the existing items in the SQL Suppliers table.

The state code field is expecting a string value and the value for inactive is 1, so we can do this by adding a new calculated column that returns 1.

Set Status Code to InActive

Then repeat the same steps to create a calculated column to return the statuscode. The status code field is a Int32 field and the value for inactive is 2.

Configure the Schema Map

We now need to configure the schema map and add the calculated columns we just created making sure to join them with their corresponding target columns statecode and statuscode.

Make sure to define a key column that can be used to uniquely identify each record, in this case accountid. You should only need these three columns and your schema map should now look similar to this:

Compare and Synchronise the Results

To finish we need to compare the source to the target by clicking the Compare A > B button in the toolbar. We can then preview the data to check the results by clicking onto the different result lines.

In this example we can see that there is one row missing in the SQL Supplier table (showing as an update) so that record needs to be updated to inactive status. The 29 records showing as delete items are the suppliers that exist in the SQL table but have been filtered out for the synchronisation. As deletes are disabled they will not be affected by this synchronisation.

To make this record inactive click Synchronize and Start to apply the changes.

You can now save this project and schedule both projects to run in sequence using either Ouvvi or the Run Tool.

For more examples on integrating with Dynamics take a look at our other blog posts. Alternatively if you have any questions please send us an email at support@simego.com.

Upcoming Changes to Data Sync and Ouvvi

info@simego.com (Rebecca Allen) — Wed, 07 Dec 2022 12:55:33 +0000

In 2023 we will be releasing a new version of Data Synchronisation Studio and Ouvvi Automation Server, these will both be labelled as Version 6.0. Data Sync has jumped from V3 to V6 to keep the version numbers in-line with Ouvvi, as it there have been many releases over the years and it was time to start afresh.

There are a number of changes that have been made to improve the user experience of Ouvvi and the integration of Data Sync and Ouvvi, as well as updating the backend to use .NET Standard. The following page will cover the changes that have been made so far and how you use each feature, as you can download a functioning preview version now.

Considerations
Data Sync Changes

Copy your Connection Library

Opening Ouvvi Steps in Data Sync

Create an Ouvvi Project
Create a New Data Sync Step

Ouvvi UI changes

Overlay Windows
Button Functionality

Configuring Ouvvi

Authentication Options

Service Configuration

OAuth Connections

IP Filtering

Miscellaneous

Back End Changes

Considerations

Before getting started we recommend you take a backup of your Ouvvi Database and any local Data Sync projects. Once you have updated your Ouvvi database you will be unable to downgrade back to version 4 without using a backup to restore your site settings.

You can install and run Data Sync V6 alongside V3, so that you can choose between the versions depending on the project you are working with.

Any local Data Sync projects will open using Version 6 and once changes have been made and saved you may not be able to open all of your projects in Data Sync V3 (if you decide to downgrade). You will still be able to access Data Sync V3 from the start menu, but the default version will be V6.

Your license key can be used to activate both Data Sync V3 and V6 so long as they are both installed on the same machine, as they will both have the same Client ID.

Your local connection library will need copying to the new V6 location to be used in your projects, but this will be covered in a section below.

The following page assumes you have taken all the backups you need and you have downloaded and installed the latest version from the preview page.

Data Sync Changes

From the moment of downloading you will notice there are a couple of visual differences to Data Sync, the first being the new colours for the splash screen and Icon, and the second being the new icon. We decided to change the colour and icon to make it easier to tell the difference between the versions from a quick glance.

Alongside this minor visual change there have been a number of changes in the architecture of Data Sync.

The Data Sync core libraries have been updated to run on .NET Standard in preparation for the move to .NET Core in the future and the code base reduced and tidied to contain only what is needed. We've also updated the UI and Data Connectors framework to run .NET v4.8 which was released back in 2019 so should be compatible with most systems.

The code complier has been upgraded to be based on Roslyn. This is now running on C# Version 7.3, whereas previously the code compiler was running C# Version 4.0 This means you can now use the latest C# syntax in Calculated Columns, Dynamic Columns and Project Automation. For example you can use the null-coalescing operator?? and functions such as $"Hello{ID}" to replace values in a string with data from a specified column.

Currently the code editor is a few versions behind the compiler and so does not recognise the new C# syntax, but we are hoping to update this in the new year. For the time being you will get the squiggly line under any of this syntax but the code will still build.

At the moment the custom connectors on github and any connector that uses the Install custom connector function will not work as they need reworking to reference the correct code base. This will be handled before the final release but is currently not a functioning feature. If you have built your own custom connector for using in Data Sync you will need to rebuild it to get the latest code base.

An additional feature we have added is to make it possible to open your Ouvvi Data Sync steps within the Data Sync designer. This is covered in more detail in the section below.

Copy the Local Connection Library

When you first open Data Sync V6 you will find your connection library isn't automatically imported. You will need to copy your current connections over or re-create them in the designer.

To copy your current local connections open the registry folder for Data Sync V3, by default this is usually C:\ProgramData\Simego\DataSyncStudio30\Registry . Then copy the connection files and paste them into the registry folder for Data Sync V6, by default this is C:\ProgramData\Simego\DataSyncStudio60\Registry.

Your connections will then be available to use from the connection library window in Data Sync.

For Ouvvi, if you have upgraded your version using the deployment manager, your connections should still be in the database. However there may be some connection types that no longer function and will need updating.

Opening Ouvvi Steps in Data Sync

You will be able to connect and open your Data Sync projects stored within your Ouvvi site from Data Sync. We're hoping that this will make it easier and more obvious on how to edit your projects when they are stored within your Ouvvi Database. To access the projects from Data Sync go to the File menu and select Open Ouvvi Project.

This will open the Ouvvi Connection Window where you need to enter in the URL to your Ouvvi site (by default this is usually http://localhost:8080) and click Connect. You will then be able to see your groups, projects and steps.

From this connection window you will be able to open your Data Sync steps, create new projects within your Ouvvi site and provision/create Data Sync steps within your projects. We will cover how to do this in a moment.

To open your Data Sync step simply double click and it will be opened in the Data Sync window. This will download a local copy to your download folder but whilst it shows as connected to Ouvvi (see toolbar, connection library, output window and status bar), you will be using the Ouvvi connection library and saving any changes back to your Ouvvi site.

You can connect to different Ouvvi sites by entering each URL, and can switch between them by selecting the URLs from the dropdown.

Create an Ouvvi Project

Within the Connection Window to Ouvvi in Data Sync you can design your Projects by creating new Projects and Data Sync Steps. This can either be to quickly mock up your project or to fully build and devleop it.

To create a new project in Ouvvi right click onto the site URL in the tree view to add a project outside of a group, or right click onto the group you want to add the project to and select Add Project.

This will open a window where you can define the project name, then click OK to create it.

This will then add the project to the tree and you can add Data Sync Steps as needed.

Create a New Data Sync Step

To create a new Data Sync Step, right click onto the project you want to add it to and select Add DataSync Step.

This will open a new window where you can enter in the name for hte step, the retry count and the run condition.

The step will then be added to the tree and you can double click it to open it in Data Sync. The project will be connected to Ouvvi and any changes you make and save will be uploaded into Ouvvi automatically. You can tell when the project is connected to Ouvvi in 4 locations:

The Title Bar : This will show the URL to your Ouvvi project
The Connection Library : This will show [Ouvvi] in the title bar of the window
The Output Window : This will show the URL to your Ouvvi project in the output for the connection library
The Footer Bar : To the right of the footer you will find the URL to the Ouvvi registry (connection library)

If none of the above are showing then you are not connected to Ouvvi and you will need to follow the steps again to re-open the project.

Ouvvi UI Changes

When you now browse to your Ouvvi site you will notice that the UI has been changed. This follows a similar style to Ouvvi V2.5 however uses a few different styles and has better navigational options. We also have dark themes available under system settings.

Our key reasoning behind the UI change was to improve your experience when using Ouvvi. We found it seemed difficult to find functions on the page, and explanations of how to do something became cumbersome. We also noticed that you can quickly loose your position when performing tasks and if you wanted to carry on editing a project you would need to re-navigate back to it.

You will now notice all of the navigation appears at the top of the page, there is a main toolbar and a small navigation menu that will appear when necessary. Within the small toolbar you will find the functions relating to the page you are currently on. Taking the project overview page shown below, you can find the navigation back to the view all projects page along with the add step and trigger functions within the small sub-toolbar..

For the steps within the project you can now choose to open or edit them from the dropdown, this reduces the number of pages you need to navigate and then number of clicks to perform the actions. If the step is a Data Sync step you will also have the option to open the project in Data Sync to allow you to edit the proejct as you need.

On some pages you will find quick function buttons, and some functions will now be performed in overlay windows instead of on a new page.

In the sections below we will cover a few of these and where you might find them.

Overlay Windows

To further help with navigation, we felt that there were too many page changes when you wanted to do something simple such as edit a user setting, copy a step and deleting a step etc.

To achieve this we have now made simple actions an overlay popup window so that you can quickly perform these tasks and do not loose where you were in Ouvvi.

You will find these interspersed throughout the UI and can either perform the task, click the cross or click away from the overlay to close the window.

Button Functionality

Another feature that has been added is a series of quick function buttons. These range from ending a process, reseting the queue, adding a service agent and so on.

The idea behind this is to remove the need for you to access the database directly and be able to perform everything you need to from the browser. This will allow users to resolve these issues, as the database is not always accessible for all users.

For example, in the past if the Server got unexpectedly restarted and the queue got stuck, you would need to access the Ouvvi database to drop the items in the queue table to get your projects back up and running. Now you simply navigate to the queue table and click Reset Queue.

Note that you need to have stopped your services before clicking reset to prevent other issues.

Another button like this is the Reset Trigger Clock which can be found on the Triggers page.

If you can think of any other quick functions you would find useful in your Ouvvi site, get in touch with us and we can see if we can add them to the build.

Configuring Ouvvi

Understanding authentication options and service user accounts can be confusing, but in this update we have tried to make the Ouvvi setup and configuration as straight forward as we currently can, trying to lead users through the steps with a next, next, next pattern.

The deployment manager has also been changed so that it disables editing the configuration unless it has been uninstalled, as none of the changes will be applied until it is uninstalled and reinstalled. This is all to make it clearer to users when items can and cannot be changed.

Ouvvi Authentication Options

In Ouvvi V6 we have made it possible to choose your preferred authentication type. You can choose between Windows and Forms authentication during the initial configuration and then you can further choose to use Azure AD credentials.

Windows Authentication uses local machine users, whereas Forms Authentication allows you to define your own usernames and passwords or use Azure AD Credentials.

You can specify the authentication type in the deployment manager configuration window by changing the drop down option.

If you selected Windows Authentication it will continue to use Windows Accounts via Integrated Security managed by IIS as before.

If you selected Forms Authentication then you can choose between using Windows Accounts (Local Domain Accounts) or Internal Ouvvi Accounts to sign into Ouvvi on the setup page. If you select Internal accounts you can later configure Ouvvi to use Azure AD Credentials to sign in.

Azure AD Accounts

To make Ouvvi use your Azure AD credentials for users you need to first browse to Ouvvi and make sure the Authentication Mode drop-down is set to Internal Accounts. Then enter in a set of credentials you want to act as the default account. This will be used to sign in to Ouvvi until you have configured the Azure AD connection.

Now go to Settings > System Settings and click onto the Azure AD button in the sub toolbar menu.

You then need to setup an Azure AD app to enable the connection, making sure to use the redirect URL listed on the Ouvvi Azure AD Setup page. From the app you need to enter the Tenant (Directory) ID and Client (Application) ID, then click Continue.

The setup will then guide you to sign in to your Microsoft account and accept the permissions.

You will then need to enter the Client Secret for your Azure App, and click Save to finish the setup.

You will then be signed out and will need to sign in again with the account you used to accept the permissions. This will be the default account with admin permissions. You can then specify additional users under the user profiles section by adding their logon name and setting the permission level.

Internal Accounts

Another option is to specify your own user accounts/credentials. To do this change the drop down to Internal Accounts and set the user credentials to act as the default system admin. You can then add additional user accounts to Ouvvi under Settings > User Profiles.

Windows Accounts

To do this, browse to Ouvvi and make sure the Authentication Mode drop down is set to Windows Accounts. Then enter in the local domain user account you want to use as the default admin.

Service Configuration

One of the biggest problems we have is users not completing the Ouvvi setup and wondering why the services don't start. So the main focus was trying to make the service setup and configuration a more guided experience. Ouvvi will now redirect you to the setup page if you have not completed the setup and guide you through adding a Service User to the Ouvvi Profiles.

Forms Authentication

If you click Start Service before finishing the configuration then you will be redirected to the setup page where you will need to configure the default system admin account and finish setting up the database. Once you have completed the setup you can go back to the deployment manager and try to start the service again. If it is successful you will be redirected to the services page and green flags will show to show that they are running.

If you do not have a service user specified then it will redirect to the services page and you can create a new service user by using the button in the browser.

When using Forms Authentication you can also change the Service User Credentials from the deployment manager. To do this right click onto your instance and select Set Service Ouvvi Credentials.

You then specify the Username and Password for the user and click OK.

If you have configured your Instance to use Azure AD credentials then clicking Start Service with redirect you straight to the Configure Ouvvi Agent Services page where you can change the username of the agent or leave the default name. This will create a service agent for the machine you are currently on.

Windows Authentication

If you are using Windows authentication, when you have installed Ouvvi and right click to proceed with the setup, if you click onto the Start Service button you will be redirected to the setup page to complete the Ouvvi database setup.

If you have already completed the database setup then clicking Start Service will redirect you to the services page where you can create a new Service agent or see that your services are up and running.

When using windows authentication you need to ensure you enter in the machine name to the Service User if you are not using the default NETWORKSERVICE user.

Connection Library OAuth Connections

A number of the connectors and their UIs have also been extended to support OAuth, this is to ensure that you can keep connecting to the providers you need to.

You can now view, edit and create OAuth connections in Ouvvi without needing to use the Data Sync designer. For a while the UI for editing certain connectors was not available, however all connector types can now be edited within the Ouvvi UI. The connectors now supporting OAuth are: Dynamics, Salesforce, SharePoint, Podio, OData.

Please note that for Dynamics, Salesforce and SharePoint connections in Ouvvi, it is limited so that only the Simego default apps can authenticate on localhost for the redirect URL. For custom apps you will need to specify the redirect URL to use your own host name. This is usually only an issue if you are exposing Ouvvi to the internet (not on localhost).

If you prefer to create your connections in Data Sync then you can continue to do this, additionally you can still also upload and export connection files as normal.

IP Filtering

Another feature now available in Ouvvi is the ability to filter access to Ouvvi by IP address. This works for restricting access when you have exposed your Ouvvi site to the internet. You should still use an SSL certificate when exposing Ouvvi outside of the local machine.

If an IP is not included in the list then access to Ouvvi is disabled and not possible. Users will have the message Access to [OUVVISITENAME] was denied

Note that localhost is always allowed by default to prevent being completely locked out.

To add IP addresses to the list click onto the Edit button and add all the IP addresses you need.

We will continue working on Version 6 into the start of 2023, so if you have any comments or suggestions please send us a message and we can look at adding additional features to the new release.

Advanced: Building a Custom Data Sync Connector

info@simego.com (Sean Cleaver) — Tue, 13 Feb 2024 12:55:33 +0000

What is a Custom Connector?

A Data Sync custom connector is compiled connector built with .NET Framework and C# and this gives you full control to build the connector and is typically used in more advanced scenarios. Examples of custom connectors might be the SharePoint and Dynamics CRM connectors, as these are complex and not something which can be built with a simple markup description. By building a connector for Data Sync you can leverage all of the Data Sync features, have the full debugging experience with F5 run and the ability to step through your code with Visual Studio.

What do you need?

To build a custom connector you will need a licensed copy of Data Synchrionisation Studio and either Visual Studio 2022 or Rider .NET Framework Development Environment.

How do Custom Connectors work?

Custom Data Sync connectors are loaded at runtime by Data Sync and we use .NET reflection to find meta-data about the connector. After this we use the Interfaces IDataSourceReader and IDataSourceWriter to call methods on the connector.

The classes DataReaderProviderBase and DataWriterProviderBase provide a quicker implementation as they handle most of the default behavior and you only need to implement a few methods.

For example a basic read-only custom connector we need to implement the following methods when inheriting from DataReaderProviderBase: GetDefaultDataSchema, GetDataTable, GetInitializationParameters and Initialize methods.

GetDefaultDataSchema - return the schema for the connector i.e. the columns and data types for the table.
GetDataTable - return a table of data based on the configured SchemaMap.
GetInitializationParameters - return a list of configuration parameters to save in the project file.
Initialize - Load the connector configuration from a list of parameters stored in the project file.

Building a basic Data Sync connector

In the example below we create a basic custom connector that downloads currency exchange rates from https://open.er-api.com/v6/latest/USD and converts the results in to a DataTable with the columns CCY and RATE.

First we define a configuration property for the Url we need to call. Public properties exposed on the class will be shown in the Data Sync property grid.

public string ServiceUrl { get; set; } = "https://open.er-api.com/v6/latest/USD";

Next we have methods to Load and Save the configuration properties so we can save the project. Initialize is called to load the configuration and GetInitializationParameters is called to save the configuration.

//Load the Provider Settings from the File.
public override void Initialize(List<ProviderParameter> parameters)
{
    foreach (ProviderParameter p in parameters)
    {               
        if(p.Name == nameof(ServiceUrl))
        {
            ServiceUrl = p.Value;
        }               
    }
}       
        
//Return the Provider Settings so we can save the Project File.
public override List<ProviderParameter> GetInitializationParameters()
{
    return new List<ProviderParameter>
                {
                    new ProviderParameter(nameof(ServiceUrl), ServiceUrl)
                };
}

Then we need to let Data Sync know what columns this data source exposes. In GetDefaultDataSchema we return a DataSchema object with a list of columns in the Map property which represents the data source schema.

//Return the Data source default Schema.
public override DataSchema GetDefaultDataSchema()
{        
    DataSchema schema = new DataSchema();

    schema.Map.Add(new DataSchemaItem("CCY", typeof(string), true, false, false, 3));
    schema.Map.Add(new DataSchemaItem("RATE", typeof(decimal), false, false, false, -1));
            
    return schema;
}

Finally we need to return the data by filling the passed DataTable (dt) with rows based on the configured SchemaMap.

The SchemaMap defines a column mapping of ColumnA -> ColumnName <- ColumnB, therefore depending which side your connected to the map can be different. We can use DataSchemaMapping to help with this column mapping to map columns to the source name. Further to this by using a callback function on the dt.Rows.Add method we can make it easier by just returning the column value for each column asked for in the callback.

// Return Table of Results.
public override DataTableStore GetDataTable(DataTableStore dt)
{                                    
    var helper = new HttpWebRequestHelper();
    var response = helper.GetRequestAsJson(ServiceUrl);

    var mapping = new DataSchemaMapping(SchemaMap, Side);
    var columns = SchemaMap.GetIncludedColumns();
            
    //Loop around your data adding it to the DataTableStore dt object.
    foreach (JProperty item_row in response["rates"])
    {                
        dt.Rows.Add(mapping, columns,
            (item, columnName) =>
            {
                if (columnName == "CCY") return item_row.Name;
                if (columnName == "RATE") return item_row.ToObject<decimal>();
                return null;
            });
    }
            
    return dt;
}

Putting it all together

If we put all this together in a Data Connector class it would look like this.

using Newtonsoft.Json.Linq;
using Simego.DataSync;
using Simego.DataSync.Providers;
using System.Collections.Generic;

namespace ExampleConnector
{
    [ProviderInfo(Name = "ExampleConnector", Description = "ExampleConnector Description")]
    public class ExampleConnectorDatasourceReader : DataReaderProviderBase
    {
        public string ServiceUrl { get; set; } = "https://open.er-api.com/v6/latest/USD";

        // Return Table of Results.
        public override DataTableStore GetDataTable(DataTableStore dt)
        {                                    
            var helper = new HttpWebRequestHelper();
            var response = helper.GetRequestAsJson(ServiceUrl);

            var mapping = new DataSchemaMapping(SchemaMap, Side);
            var columns = SchemaMap.GetIncludedColumns();
            
            //Loop around your data adding it to the DataTableStore dt object.
            foreach (JProperty item_row in response["rates"])
            {                
                dt.Rows.Add(mapping, columns,
                    (item, columnName) =>
                    {
                        if (columnName == "CCY") return item_row.Name;
                        if (columnName == "RATE") return item_row.ToObject<decimal>();
                        return null;
                    });
            }
            
            return dt;
        }
        
        //Return the Data source default Schema.
        public override DataSchema GetDefaultDataSchema()
        {        
            DataSchema schema = new DataSchema();

            schema.Map.Add(new DataSchemaItem("CCY", typeof(string), true, false, false, 3));
            schema.Map.Add(new DataSchemaItem("RATE", typeof(decimal), false, false, false, -1));
            
            return schema;
        }

        //Return the Provider Settings so we can save the Project File.
        public override List<ProviderParameter> GetInitializationParameters()
        {
            return new List<ProviderParameter>
                       {
                            new ProviderParameter(nameof(ServiceUrl), ServiceUrl)
                       };
        }

        //Load the Provider Settings from the File.
        public override void Initialize(List<ProviderParameter> parameters)
        {
            foreach (ProviderParameter p in parameters)
            {               
                if(p.Name == nameof(ServiceUrl))
                {
                    ServiceUrl = p.Value;
                }               
            }
        }        
    }
}

Example Connector loaded in the Data Sync Designer

Once this has been built and loaded into Data Sync, you can build a project to use it like the image below.

Getting Started

To get started with your own connector we have created a function in DataSync to create a default project for you. To get to this go to Tools > Create Data Provider VS Project, which will open a window where you can define the assembly name and output path.

Clicking OK will create a Visual Studio Project for a new connector configured ready to start your development.

For connector development you need a reference to Simego.DataSync.Core.dll from the Data Sync install directory and if your going to use the HttpWebRequestHelper you will also need to add a Nuget package reference to Newtonsoft.Json version 13.0.3.

To be able to debug your project in Visual Studio the project needs to be configured to start an external program which is the Data Sync Designer from the install directory i.e. "C:\Program Files\Simego\Data Synchronisation Studio 6.0\Simego.DataSync.Studio.exe" and you need to pass the command line parameter -debug.

When you run your project from Visual Studio it will start Data Sync and Data Sync will look in the Visual Studio output directory for connectors and load them from the folder ready to debug your code.

Identifier Columns

The first concept to understand is the role of the Identifier Column with Data Sync connectors. These values are attached to the row internally within Data Sync and are a way to maintain the data source PrimaryKey even when its not part of the Schema Map. By maintaining this Identifier Column value it makes it easier to handle UPDATE/DELETE actions on the target by the PrimaryKey value.

If your connector is going to be read-only then Identifier Columns are not necessary.

During the call to GetDataTable you can add Identifier Columns once only. After you add Identifier Columns you must add the rows to the Data Table with the method call AddWithIdentifier

public override DataTableStore GetDataTable(DataTableStore dt)
{            
    //This data source has a single integer ID value we use this to store the ID to assist with UPDATE and DELETE actions.
    dt.AddIdentifierColumn(typeof(int));
            
    var mapping = new DataSchemaMapping(SchemaMap, Side);
    var columns = SchemaMap.GetIncludedColumns();

    // Returns a basic Dictionary with some data you would replace this with calls to your target system to get the data.
    var result = TestData();

    //Loop around your data adding it to the DataTableStore dt object.
    foreach (var item_row in result)
    {
        dt.Rows.AddWithIdentifier(mapping, columns,
            (item, columnName) =>
            {
                return item_row[columnName];
            }
            , item_row["item_id"]);
    }
            
    return dt;
}

Now within your writer class it is easy to get the identifier value for a row to be used with UPDATE and DELETE actions.

Custom Configuration UI

You can provide a custom UI component to be displayed in the Data Sync connection dialog. This is used to assist in creating the connection and is especially useful if you need to guide the user or choose different data sources to connect to.

To use a custom UI component your connector must implement the IDataSourceSetup interface.

public interface IDataSourceSetup
{
    void DisplayConfigurationUI(IntPtr parentHwnd);
    IDataSourceReader GetReader();
    bool Validate();
}

DisplayConfigurationUI is called when we show the UI component it will pass a handle to the parent window where you can add your control.
Validate() is called when the user clicks the OK button in the connection dialog. Here you can throw an exception to the user or return true to the accept the configuration settings.
GetReader() is then called to return an instance of the configured DataSource Reader. Typically this is just this.

Below is an example which loads a ConnectionInterface form which contains a PropertyGrid control, and is then loaded into the connection dialog.

public void DisplayConfigurationUI(IntPtr parent)
{
    var parentControl = Control.FromHandle(parent);

    if (_connectionIf == null)
    {
        _connectionIf = new ConnectionInterface();
        _connectionIf.PropertyGrid.SelectedObject = new ConnectionProperties(this);
    }

    _connectionIf.Font = parentControl.Font;
    _connectionIf.Size = new Size(parentControl.Width, parentControl.Height);
    _connectionIf.Location = new Point(0, 0);
    _connectionIf.Dock = DockStyle.Fill;

    parentControl.Controls.Add(_connectionIf);
}

public bool Validate() => true;
        
public IDataSourceReader GetReader() => this;

The ConnectionProperties class is used here to create a wrapper around the reader object to expose the configuration properties to display in the PropertyGrid control.

class ConnectionProperties
{
    private readonly ExampleDatasourceReader _reader;
        
    [Category("Settings")]
    public string ExampleSetting { get { return _reader.ExampleSetting; } set { _reader.ExampleSetting = value; } }

    public ConnectionProperties(ExampleDatasourceReader reader)
    {
        _reader = reader;
    }        
}

Lookups

To support Lookup functions (LOOKUPA and LOOKUPB) in Data Sync you need to implement the interface IDataSourceLookup. This requires that you return a DataTable of results based on the Lookup configuration.

public interface IDataSourceLookup
{
    DataTableStore GetLookupTable(DataLookupSource source, List<string> columns);
}

Here is an example of a Lookup implementation. This method would be called when a user uses calculated columns with an expression like this.

LOOKUPA("supplierid", "suppliers", WHEN("companyname", value))

What you do is create a new instance of your connector and set the appropriate properties from your current instance. In this case we need to copy the database connection string. DataLookupSource contains parameters passed by the user. So really in the example below we are looking to change the SourceTable property to the name passed in the source (suppliers).

You then create a default schema map including only the columns from the columns list ("supplierid", "companyname") and fill and return the DataTable.

public DataTableStore GetLookupTable(DataLookupSource source, List<string> columns)
{
    var reader = new SqlDataSourceReader
                        {
                            ConnectionString = ConnectionString,
                            CommandTimeout = CommandTimeout,
                            SourceTable = source.Config.ContainsKey("SourceTable") ? source.Config["SourceTable"] : source.Name
                        };

    reader.Initialize(SecurityService);

    var defaultSchema = reader.GetDefaultDataSchema();
    reader.SchemaMap = new DataSchema();

    foreach (var dsi in defaultSchema.Map)
    {
        foreach (var column in columns)
        {
            if ( dsi.ColumnName.Equals(column, StringComparison.OrdinalIgnoreCase))
                reader.SchemaMap.Map.Add(dsi.Copy());
        }
    }

    return reader.GetDataTable();
}

Connection Library

The connection library is used to store connection configuration which can be updated to maintain the connection. This is typically endpoints, username and passwords. These are then stored separate from the project file in either the local library on disk or the Ouvvi library in the Ouvvi database.

To support the connection library you need to implement the interface IDataSourceRegistry

public interface IDataSourceRegistry
{
    string RegistryKey { get; set; }        
    void InitializeFromRegistry(IDataSourceRegistryProvider provider);
    List<ProviderParameter> GetRegistryInitializationParameters();
    object GetRegistryInterface();
}

RegistryKey - The name of the connection library entry. This must be managed by the default methods Initialize and GetInitializationParameters so that this key is set during project load and saved when you save the project.
InitializeFromRegistry - Used to Initialize your connector from settings stored in the connection library.
GetRegistryInitializationParameters - Returns the connection parameters to store in the connection library.
GetRegistryInterface - Returns a version of the connector where you can hide the connection library parameters from the default Data Sync UI.

public void InitializeFromRegistry(IDataSourceRegistryProvider provider)
{
    var registry = provider.Get(RegistryKey);
    if (registry != null)
    {
        foreach (ProviderParameter p in registry.Parameters)
        {
            if(p.Name == nameof(ServiceUrl))
            {
                ServiceUrl = p.Value;
            }  
        }
    }
}

TIP: If you need to update the connection library at runtime, save a reference to the IDataSourceRegistryProvider so you can write back to the library later.

public List<ProviderParameter> GetRegistryInitializationParameters()
{
    return new List<ProviderParameter>
                {
                    new ProviderParameter(nameof(ServiceUrl), ServiceUrl)
                };
            
}

GetRegistryInterface requires a new class which wraps your connector to expose only those parameters which can be changed. For example you can use the helper base class DataReaderRegistryView<T> and then add the properties to the class you want to be available in the data source property grid like this.

public class ExampleConnectorDatasourceReaderWithRegistry : DataReaderRegistryView<ExampleConnectorDatasourceReader>
{
    [ReadOnly(true)]
    public string ServiceUrl => _reader.ServiceUrl;

    public ExampleConnectorDatasourceReaderWithRegistry(ExampleConnectorDatasourceReader reader) : base(reader)
    {
            
    }
}

Then return and instance of this class on a call to GetRegistryInterface.

object GetRegistryInterface() => new ExampleConnectorDatasourceReaderWithRegistry(this);

Connection Library Server Explorer

If your connector can return multiple data sources i.e Tables in a Database then you can implement the Connection Library Server Explorer. This allows for quick connection directly from the TreeView and lists the available data sources.

To do this you need to implement the IDataSourceRegistryView interface.

public interface IDataSourceRegistryView
{
    RegistryConnectionInfo GetRegistryConnectionInfo();
    RegistryViewContainer GetRegistryViewContainer(string parent, string id, object state);
    object GetRegistryViewConfigurationInterface();
}

GetRegistryConnectionInfo - returns meta data about this connection i.e name and icon.
GetRegistryViewContainer - return the Folder and items list to display in the TreeView.
GetRegistryViewConfigurationInterface - currently unused and should return null.

Below is a simple implementation of the IDataSourceRegistryView interface to return a Folder List to display in the TreeView.

Within the RegistryFolderType the ParameterName represents the configuration property name which will receive the object name from the list in the TreeView. In this example "Item 1" and "Item 2".

public RegistryConnectionInfo GetRegistryConnectionInfo()
{
    return new RegistryConnectionInfo { 
        GroupName = "Example Provider", 
        ConnectionGroupImage = RegistryImageEnum.Folder, 
        ConnectionImage = RegistryImageEnum.Folder 
    };
}

public RegistryViewContainer GetRegistryViewContainer(string parent, string id, object state)
{
    var folder = new RegistryFolderType
    {
        DataType = InstanceHelper.GetTypeNameString(GetType()),
        Image = RegistryImageEnum.Table,
        Preview = true,
        ParameterName = nameof(Table))
    };

    folder.AddFolderItems(new string [] { "Item 1", "Item 2" });

    return new RegistryViewContainer(folder, null);
}

public object GetRegistryViewConfigurationInterface() => null;

You can add multiple folders and they can be lazy loaded. This GetRegistryViewContainer method would be called multiple times as the user expands the tree whilst passing the parent folder name.

Writing Back

To create a connector which can write back to your data source you need to create a new class which implements IDataSourceWriter or ideally derives from DataWriterProviderBase. You then implement the methods AddItems, UpdateItems and DeleteItems.

Within your reader you return an instance of this class in the GetWriter whilst setting the SchemaMap property.

public override IDataSourceWriter GetWriter()
{    
    return new ExampleConnectorDatasourceWriter {SchemaMap = SchemaMap};
}

Within the DataWriter class the Execute method is called during the write operation and its up to you how you process this. Typically you would call each of the methods AddItems, UpdateItems and DeleteItems.

public override void Execute(List<DataCompareItem> addItems, List<DataCompareItem> updateItems, List<DataCompareItem> deleteItems, IDataSourceReader reader, IDataSynchronizationStatus status)
{
    //Process the Changed Items
    if (addItems != null && status.ContinueProcessing) AddItems(addItems, status);
    if (updateItems != null && status.ContinueProcessing) UpdateItems(updateItems, status);
    if (deleteItems != null && status.ContinueProcessing) DeleteItems(deleteItems, status);            
}

An example implementation for AddItems might look like the code below, where you would fill in the code in the TODO: section to write the data to the target to create a new record from the data in the targetItem dictionary object.

These examples also handle hooks for Project Automation Item events. If you do not plan to support those then the code can be simplified by removing the Automation Hooks.

public override void AddItems(List<DataCompareItem> items, IDataSynchronizationStatus status)
{
    if (items != null && items.Count > 0)
    {
        int currentItem = 0;

        foreach (var item in items)
        {
            if (!status.ContinueProcessing)
                break;

            try
            {
                var itemInvariant = new DataCompareItemInvariant(item);

                //Call the Automation BeforeAddItem (Optional only required if your supporting Automation Item Events)
                Automation?.BeforeAddItem(this, itemInvariant, null);

                if (itemInvariant.Sync)
                {                    
                    //Get the Target Item Data
                    Dictionary<string, object> targetItem = AddItemToDictionary(Mapping, itemInvariant);

                    //TODO: Write the code to Add the Item to the Target

                    //Call the Automation AfterAddItem (pass the created item identifier if possible)
                    Automation?.AfterAddItem(this, itemInvariant, null);

                }

                ClearSyncStatus(item); //Clear the Sync Flag on Processed Rows

            }
            catch (SystemException e)
            {
                HandleError(status, e);
            }
            finally
            {
                status.Progress(items.Count, ++currentItem); //Update the Sync Progress
            }

        }
    }
}

For UpdateItems its slightly different. We use a call to GetTargetIdentifier<T> to get the identifier value of the row we want to update. This time the targetItem dictionary object only contains those columns which were changed and you need to fill in the code in the TODO: section to write the data to the target.

public override void UpdateItems(List<DataCompareItem> items, IDataSynchronizationStatus status)
{
    if (items != null && items.Count > 0)
    {
        int currentItem = 0;

        foreach (var item in items)
        {
            if (!status.ContinueProcessing)
                break;

            try
            {
                var itemInvariant = new DataCompareItemInvariant(item);

                //Example: Get the item ID from the Target Identifier Store 
                var item_id = itemInvariant.GetTargetIdentifier<int>();

                //Call the Automation BeforeUpdateItem (Optional only required if your supporting Automation Item Events)
                Automation?.BeforeUpdateItem(this, itemInvariant, item_id);

                if (itemInvariant.Sync)
                {
                    //Get the Target Item Data
                    Dictionary<string, object> targetItem = UpdateItemToDictionary(Mapping, itemInvariant);

                    //TODO: Write the code to Update the Item in the Target using item_id as the Key to the item.

                    //Call the Automation AfterUpdateItem 
                    Automation?.AfterUpdateItem(this, itemInvariant, item_id);
                }

                ClearSyncStatus(item); //Clear the Sync Flag on Processed Rows
            }
            catch (SystemException e)
            {
                HandleError(status, e);
            }
            finally
            {
                status.Progress(items.Count, ++currentItem); //Update the Sync Progress
            }

        }
    }
}

Finally for DeleteItems, we can usually just delete the item with the Identifier value.

public override void DeleteItems(List<DataCompareItem> items, IDataSynchronizationStatus status)
{
    if (items != null && items.Count > 0)
    {
        int currentItem = 0;

        foreach (var item in items)
        {
            if (!status.ContinueProcessing)
                break;

            try
            {
                var itemInvariant = new DataCompareItemInvariant(item);

                //Example: Get the item ID from the Target Identifier Store 
                var item_id = itemInvariant.GetTargetIdentifier<int>();

                //Call the Automation BeforeDeleteItem (Optional only required if your supporting Automation Item Events)
                Automation?.BeforeDeleteItem(this, itemInvariant, item_id);

                if (itemInvariant.Sync)
                {
                    //TODO: Write the Code to Delete the Item in the Target using item_id as the Key to the item.

                    //Call the Automation AfterDeleteItem 
                    Automation?.AfterDeleteItem(this, itemInvariant, item_id);
                }

                ClearSyncStatus(item); //Clear the Sync Flag on Processed Rows
            }
            catch (SystemException e)
            {
                HandleError(status, e);
            }
            finally
            {
                status.Progress(items.Count, ++currentItem); //Update the Sync Progress
            }

        }
    }
}

Incremental Data Load

To support Incremental Data Load you need to set SupportsIncrementalReconciliation=true in the constructor and implement the method GetDataTable(DataTableStore dt, DataTableKeySet keyset).

In Incremental mode we pass a list of keys (keyset) that are the unique key values of the loaded source connection. In response you need to return a DataTable of the rows with matching keys, each key is expected to be unique and return only 1 result.

Packaging and Deployment

To package your connector for deployment you need to compile your connector and compress the required components (typically just your .dll file) into a zip archive file. You can then use the File > Install Data Connector feature in Data Sync to install the connector into the DataSync custom connector library.

Here is an example bat file script which you could use to compile and package into a zip archive.

set p=ExampleConnector
"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\msbuild.exe" /t:Build /p:Configuration=Release /p:NoWarn=1591
rmdir ..\dist\ /S /Q
mkdir ..\dist\files\%p%
xcopy ..\src\%p%\bin\Release\net48\*.* ..\dist\files\%p%\*.* /y
cd ..\dist\files\
del .\%p%\Simego.DataSync.Core.dll
tar.exe -acf ..\%p%.zip *.*
cd ..\..\src

NOTE: During development you must not have the connector in both places. So if your currently developing your connector make sure to delete it from the Data Sync connector library.

Further Information

You can checkout some of our open-source connectors in our GitHub repository here github.com/simego-ltd for more ideas.

SharePoint File Data Source Connector

info@simego.com (Sean Cleaver) — Wed, 10 Apr 2024 12:55:33 +0000

Data Sync can now connect to a file stored in a SharePoint Document Library and access the data stored within it, without needing to download a copy and reupload it.

We've had a number of enquiries over the years asking for this functionality and now it is available! So hopefully this fills a gap, and you will all find this connector useful.

With this connector you can preview, write to and create CSV, XML and Excel files within SharePoint. It will also handle version control, and you can specify the type of check in to be performed i.e. Minor, Major or Overwrite.

Below we will cover a brief overview of how the connector can be used and a few example use cases. For the full documentation please see our docs site here

Connect to the File

To connect open the data source window, locate the Microsoft SharePoint folder, expand the node and select SharePoint File Data Source. Then select the connection type, by default this will use the Simego OAuth option but you can also choose to use a custom OAuth connection.

Enter in the base URL to the SharePoint site. This can be either the root site or a sub-site and should be in the format https://<domain>.sharepoint.com/<site1>/<site2>.

Then click Authorise Connection to enable the connection to SharePoint. Once the connection has been authorised you need to enter in the path to the file you want to connect to. This should include the folder paths, the file name and the file extension. For example the path could be /Documents/ContactRecords/Contacts_04-2024.csv.

You will need to build this path yourself by navigating through your SharePoint site to the file, making a note of each folder and the order you pass through. The link that can be gathered from the file in SharePoint directly is not user friendly, so you should use a combination of the URL and the filename and extension.

The final connection should look similar to the image below:

The connector also supports the connection library. So you can save the connection to the site and just need to specify the path and file type in future connections.

You can find the documentation pages on this connector on our Docs site here.

Use Case Examples

Below we discuss a few examples of how this connector could be used in day to day business scenarios to help prompt you with some ideas. For example creating a backup of a SAAS system, making data available outside of systems that require licenses, feeding data into Power BI models and so on.

Create New Files Within SharePoint for a Daily Sync

You could use the SharePoint file connector to add a copy of the data edited that day to a folder within SharePoint. For example you could run a daily sync to capture the contacts added to a contact list in Dynamics today into an XML file.

To do this you would connect to the Dynamics Contact entity as the source, and a template target XML file that has the column names of the columns you want to capture within SharePoint.

We can then add a filter to filter for only the contacts that have been modified today using modifiedon >= DATEADD("day",-1,TODAY()). Next make sure to map the source and target columns in the schema map and ensure a key column has been selected.

Then using project automation you can override the filename to add today's date to the file name, and specify the path it should be saved to.

If the folder doesn't exist yet, Data Sync will create this in SharePoint for you. We also need to remember to escape the backslashes in the path with another backslash as it is using C#.

public override void Start()
{
	DataSourceB.Path = $"\\Contacts Backup\\UpdatedContactRecords-{DateTime.Today.ToString("yyyyMMdd")}.xml";
}

The project configuration should then look similar to the below image:

Now when you compare and sync the results, the file will be created in SharePoint and the data uploaded so that you have a full record of the data from that day.

Template File Doesn't Exist

If the template file you are connecting to doesn't exist yet, you can easily make a template document using the export options in Data Sync. Add the source columns you want to include in the file to the schema map and then use the Preview A export options from the schema map to export a copy of the data set as an XML/CSV/Excel file.

You would then need to upload this file into SharePoint and this will act as a template file to connect the target data source window going forward.

Update the base file for a PowerBI model

You could use Data Sync and the SharePoint file connector to keep a PowerBI model up to date. If you set up a regular sync so that the file has the most up to date data, then Power BI can pull data from it when the model refreshed as and when it is needed.

In short you would use Data Sync to keep the file data up to date and then just refresh the model to get the updates.

In Data Sync the project would be configured so that your source data feeds into your file within SharePoint on the target. You can set this project to run on a regular basis e.g. at the beginning and end of the day to capture the changes, and the file be set as the data source in PowerBI.

You then just need to connect your Power BI model to the file and refresh it as needed. We have a quick bit of guidance on connecting the file to Power BI below.

Connecting to the file in Power BI

Sometimes connecting the document in SharePoint to Power BI can throw a few errors. Most of these revolve around getting the right path to the file, so we will briefly cover how to do this.

To connect to the file in Power BI click onto Get Data. Then select Web from the options list and click Connect.

You now need to enter in the path to the file in SharePoint.

The best way to get the right path is to open the file in the desktop app from SharePoint. Then go to File > Info and select Copy Path. You then need to remove web=1 at the end and the connection should authenticate without any issues.

This should look something like the following:

Then follow through the wizard to sign in and you should now be able to load or transform the data as needed.

Once you have created your model/report in PowerBI you can simply refresh the dataset to get the latest updates whenever you need an updated report.

Create a Backup of your SAAS Systems

Another option is that you could use the SharePoint File Data Source connector to create a backup of your SAAS systems.

For example you could have an XML file that holds all of your contact records from Salesforce. You can make this possible by connecting your source to the Salesforce Contact object and the target to the XML file in SharePoint that will hold the records.

Map the fields you want included in the sync in the schema map and set a key column that link the results (this is usually an ID but could be something different so long as it is unique). Then simply compare the results and sync the changes. Remember that by default Data Sync disables deletes so any records that have been removed from your Contact object will still exist in the XML file unless you enable deletes.

We can use project automation to add the date onto the file name and set the path for the backup. You might want use Data Sync to add the file into a specific folder if you are running it multiple times during the day.

public override void Start()
{
	DataSourceB.Path = $"\\Salesforce Backup\\ContactObject-{DateTime.Today.ToString("yyyyMMdd")}.xml";
}

You can make use of Ouvvi or the Run Tool to schedule the project to run on a regular basis. As a backup you might just want to run it at the end of the day, or every hour if the changes are frequent.

If the file doesn't exist yet you can easily make a template document by adding the fields you want to include in the file to the schema map, and then use the preview export options to export a copy of the data set as an XML file. You would then need to upload this file into SharePoint and this will act as the template file going forward.

Make Data Available outside of Dynamics for Other Users

A slightly different use case is to expose data outside of Dynamics so users can view without a dynamics license - e.g. Exporting a client list into an excel spreadsheet in SharePoint.

You can connect your source to the client list in Dynamics and the target to the file you want to synchronise the data to. Map the columns you want to include, run the compare and sync the results.

Now your employees/ users can view the data without needing to sign into Dynamics. If you also schedule the synchronisation to run every 30 minutes to 1 hour, using either Ouvvi or the Run Tool, then your file will remain up to date for the users. You can set the schedule to meet your needs depending on how frequently the data changes.

It really is that easy!

Make Employee To Do Lists to Split Tasks

If you have a small team that needs specific tasks (e.g. a call list) splitting up into separate files for each employee, then you could use multiple Data Sync projects to do this.

To do this you would have one project per employee that creates a spreadsheet they can work from without conflicting with the other users. You would start by connecting to your source data, which for this example we are saying is a call list that has a user assigned against each account. Then connect the target to a template excel spreadsheet found in SharePoint, that has the columns you want to be included.

Next you would add a filter on the source to only return records for the first user you are wanting to create the file for, then map the columns between the source and target, and make sure to select a key column to identify each row as unique. This might be an ID column or a composite key of the account name and date (if there is only 1 record per account per day). Finally you can use Project Automation to write the file to a specific path and add the employee name to the file being created in SharePoint. This would use the same code explained in the previous example where we created new files with today's date only this time it has the date, an employee name, and a specific folder defined in the path. You can see an example shown below:

public override void Start()
{
	DataSourceB.Path = $"\\To Do List\\Employee123\\CallList-Employee123-{DateTime.Today.ToString("yyyyMMdd")}.xlsx";
}

Alternatively for a quick and simple option just export the data to the excel file, include the employee names and let the employee filter the excel document to find their tasks.

Snapshot SharePoint List Data

An alternative use case is to take a snapshot of some SharePoint list data, so that you can look back and see the data that was contained in that list on a particular day. This could be for audit purposes, or simply as a backup.

To do this you would connect your source to the SharePoint list and connect the target to a template file in SharePoint that has the columns you want to include

If the file doesn't exist yet you can easily make a template document by adding the columns you want to include from the source, to the schema map and then use the preview export options to export a copy of the data set as an XML/CSV/Excel file. You would then need to upload this file into SharePoint and this will act as a template file to connect the target data source window going forward.

Make sure the mapping is correct in the schema map and select a key column to identify each row as unique.

You will then need to use project automation to add the date onto the filename and determine the path the file should be created at. You can do this by adding the following line of code but make sure the extension matches your file type. The below example is for an excel file so has xlsx listed.

public override void Start()
{
	DataSourceB.Path = $"\\SharePoint SnapShots\\MyListSnapShot-{DateTime.Today.ToString("yyyyMMdd")}.xlsx";
}

This will now create a new file at the path you defined when you run the synchronisation, and connect to it as the target file.

Export from Active Directory

LDAP filter for admin or who logged on today for audit purposes.

A final example could be to export data from Active Directory. This might be a file for auditing the administrators of a group in Active Directory or for who logged in that day.

This Data Sync project would be connected to Active Directory as the source and export the data to a file within SharePoint to act as an audit file for that day. You could then add in date to the file name using project automation to create new file.

public override void Start()
{
	DataSourceB.Path = $"\\Active Directory Audit Data\\UsersLoggedIn-EOD-{DateTime.Today.ToString("yyyyMMdd")}.csv";
}

To get the logged in users you could simply apply a filter to your source data (Users in AD) to only return those with a value of Todays date for the DS-LastLogon attribute.

This filter could look like this: DSLastLogon >= DATEADD("day",-1,TODAY())

Then map the columns between the source data and target file in SharePoint, run the compare and Sync the results.

To get the admin members into a file you could connect to the users list and use an LDAP filter to only return the only return Administrators. Then follow the same steps as above and you will get the file created in SharePoint.

We would then recommend scheduling the project to run so that it runs every day at the end of the day to give you a full audit history. You can do this with either Ouvvi or the Run Tool.

If you have any questions about how to make a specific scenario work for you, send us an email and we will be happy to help get you started.

Simego Updates

info@simego.com (Sean Cleaver) — Tue, 24 Sep 2024 12:55:33 +0000

Microsoft.Data.SqlClient

Version 6.0.3396

Since Microsoft have deprecated the System.Data.SqlClient library we have updated Data Sync to use the newer Microsoft.Data.SqlClient library. This enables more authentication options especially with SQL Azure which can be configured via the 'Additional Connection Properties' property pane.

The change should be seamless however since these connections are now Encrypted by default if you have an older connection string you may need to update it to Trust the Server Certificate.

Set Sync Mode in Project Automation

Version 6.0.3388

We have made it possible to adjust the Sync Mode at runtime within the Start() or GetDataSchema() methods of Project Automation.

SyncMode = SyncModeEnum.SyncAtoBIncremental;

This allows for custom synchronisation scenarios where you can adapt the sync based on some condition. See Dynamics to SQL Export for an example.

OAuth Certificate Authentication

Version 6.0.3372

We have extended our OAuth2 support to include Client Certificate Support in place of a Client Secret.

Certificates are loaded from the Windows Certificate Store and therefore the process will need access to the Private Key. If you use the LocalMachine certificate store then ensure that the Ouvvi Service user has permission to read the private key. Additionally if you want to manage the connection from Ouvvi ensure that the Ouvvi AppPool user also has permission to the private key.

In this version 6.0.3372 we have support for using certificates with authorization_code and client_credentials flows with the following connectors.

SharePoint Online
SharePoint FileStore (used to access data inside Excel/CSV files in a document library)
Dynamics 365
SQL Azure
JSON API Tool

You can specify either the thumbprint or subject name as the certificate setting in the connection.

You can also specify the full path i.e. LocalMachine\My\DataSync-SharePoint or just DataSync-SharePoint and we will look for the certificate in the My store location in both the LocalMachine and CurrentUser stores.

Since we now support certificates its possible to connect with SharePoint using client_credentials flow and use application level permission rather than a delegated user. This prevents your integration projects being tied to a user account.

Ouvvi Time Trigger - Window

Version 6.0.3370

We had a customer request to prevent Time Triggers to running after a certain time. By default if a trigger time is missed due to the service not running because if an IT issue or similar, The trigger runs when the service is restarted. Generally this is what you want, However for this customer they did not want to run resource heavy overnight tasks during the day if this occurred.

The solution to this is a new EndTime setting on the Time Trigger which defaults to '23:59:59'. Now if the End Time period has passed the trigger does not evaluate and therefore start the attached projects. So you can now configure a Time Trigger to run between say 03:00:00 and 05:00:00 to prevent the trigger running after 5AM.

If you have any feedback or would like us to consider a new feature please reach out to us at support@simego.com.

Dynamics to SQL Data Export

info@simego.com (Sean Cleaver) — Wed, 30 Oct 2024 12:39:33 +0000

In this guide we show how you can use Data Synchronisation Studio to build a Dynamics/Dataverse to SQL Server Data Export solution. Whilst only creating a single Data Sync project and using Project Automation to enable automatic creation of SQL Tables from the Dynamics schema.

You could also use this approach with other Datasources like Salesforce or SharePoint.

Overview

We will build a single Data Sync project which will automatically update itself based on the requested entity. If the schema of the entity changes we will drop the existing table and re-create it.

The steps required for this to work are as follows.

Connect to Entity named in Project Property.
Discover Dynamics Entity Schema.
Create matching SQL Table in SQL Database from the Dynamics Schema.
Check if the schema has changed and re-create the table if necessary.
Setup Source and Target in the Data Sync project and update Schema Map to match.

Command Line

Once this project is configured you can call the sync for different entities right from the command line like this.

"%DATASYNCINSTALLPATH60%\Simego.DataSync.Run.exe" -e project.dsprj -p:entity=account
"%DATASYNCINSTALLPATH60%\Simego.DataSync.Run.exe" -e project.dsprj -p:entity=contact

Setup Project

First we need a SQL Table in our target database used to track which entities have been created and a hash value of the schema columns.

Use the script below to create your table for tracking the created entity tables.

CREATE TABLE [dbo].[_dsTableMetadata] (
	[ID] int IDENTITY(1, 1) NOT NULL,
	[Name] nvarchar(255) NOT NULL,
	[Hash] nvarchar(100) NOT NULL,
	[Script] nvarchar(MAX) NULL,
	[Created] datetime DEFAULT(getutcdate()) NOT NULL,
	[Updated] datetime DEFAULT(getutcdate()) NOT NULL
)
GO

ALTER TABLE [dbo].[_dsTableMetadata] ADD CONSTRAINT [PK__dsTableMetadata_BCAF3B7C] PRIMARY KEY CLUSTERED ([ID] ASC) 
GO
ALTER TABLE [dbo].[_dsTableMetadata] ADD CONSTRAINT [IX__dsTableMetadata_FDE2506A] UNIQUE NONCLUSTERED ([Name] ASC)

Next we need a DataSync Project configured with connection from the Connection Library for Dynamics and SQL. At this point it doesn't really matter about the Schema Map and if the project works its about configuring the source and target systems.

Start with a new Project and connect the source to the Dynamics 'account' entity from the Connection Library. Then connect the Target to a SQL Table from the Connection Library.

Under File->Properties->Properties add a new Project Property named 'entity' with a value 'account'.

Project Automation Code

This is the code which dynamically creates a SQL Table matching the source Data Schema and sets up the project and schema map to match.

These are the steps we need the code to complete.

Setup the source and target to point to the Entity value stored in the Project Property entity.

Then when the Schema Map is requested in the GetDataSchema override function.

Get the source Schema.
Check if the SQL Table already exists in the SQL Database or whether to re-create the Table.
Converts the source schema into a SQL Table script.
Create the SQL Table on the target SQL Server.
Return the updated Schema Map.

Enable Project Automation on your project and replace the code with this version below.

using System;
using System.Collections.Generic;
using System.Data;
using System.Globalization;
using System.Linq;
using System.Text;
using Simego.DataSync;
using Simego.DataSync.Engine;

class ProjectAutomationOverride : Simego.DataSync.Automation.ProjectAutomationShim //Do Not Change This Line
{    	
    public override void Start()
    {
        // Set the Entity on DataSource A
        DataSourceA.Entity = Properties["entity"];		
        // Set the Entity in DataSource B
        DataSourceB.SourceTable = $"[dbo].[{DataSourceA.Entity}]";
        // AutoMap the Schema		
        AutoSchemaMap = true;		
    }
    
    public override DataSchema GetDataSchema(DataSchema schema)
    {
        var schemaA = DataSourceA.GetDefaultDataSchema();
        
        // Get the Key columns for the PK.
        var keyColumns = string.Join(", ", schemaA.GetUniqueColumns().Select(p => $"[{p.ColumnName}]"));
        
        // Get the column list
        var columns = string.Join(",\r\n", schemaA.Map.Where(p => isValidColumn(p)).Select(column => $"\t[{column.ColumnName}] {getSqlColumnType(column)} {(column.AllowNull ? "NULL" : "NOT NULL")}"));
        
        // Generate a Hash of the columns to detect if any have changed.
        var columnHash = getHash(columns);
                
        // Check if the entity Table already exists in the DB. By getting the existing Hash from our _dsTableMetadata table.
        var existingHash = DataSourceB.ExecuteScalar($"SELECT Hash FROM _dsTableMetadata WHERE Name = '{DataSourceB.SourceTable}'") as string;
        if (existingHash == null)
        {									
            // Create Table Script
            var tableScript = getTableScript(keyColumns, columns);
            
            // Create the Target Table
            DataSourceB.ExecuteNonQuery(tableScript);	
            
            // Update the _dsTableMetadata Table
            DataSourceB.ExecuteNonQuery("INSERT INTO [dbo].[_dsTableMetadata] (Name, Hash, Script) VALUES (@p0, @p1, @p2)", DataSourceB.SourceTable, columnHash, tableScript);
            
            // Write to Log
            Trace.WriteLine($"Created Table '{DataSourceB.SourceTable}'");			
        } 
        else if(existingHash != columnHash)
        {			
            // Create Table Script
            var tableScript = getTableScript(keyColumns, columns);
            
            // Update the _dsTableMetadata Table
            DataSourceB.ExecuteNonQuery("UPDATE [dbo].[_dsTableMetadata] SET Hash=@p0, Script=@p1, Updated=getutcdate() WHERE Name=@p2", columnHash, tableScript, DataSourceB.SourceTable);
            
            // Drop the existing Table
            DataSourceB.ExecuteNonQuery($"DROP TABLE {DataSourceB.SourceTable}");	
            
            // Create the Target Table
            DataSourceB.ExecuteNonQuery(tableScript);	
                                    
            // Write to Log
            Trace.WriteLine($"Updated Table '{DataSourceB.SourceTable}'");			
        }
        
        // Get the Target Schema
        var schemaB = DataSourceB.GetDefaultDataSchema();
        
        // Return an Auto-Map
        return AutoSchemaMap ? schemaA.ToAutoMap(schemaB) : schema;
    }
    
    string getTableScript(string keyColumns, string columns)
    {
        return string.Concat(
                $"CREATE TABLE {DataSourceB.SourceTable} (\r\n",
                // Append the Columns
                columns,
                ")\r\n",
                // Add the Primary Key
                $"ALTER TABLE {DataSourceB.SourceTable} ADD CONSTRAINT [PK_dbo_{DataSourceA.Entity}] PRIMARY KEY CLUSTERED ({keyColumns})\r\n");
            
    }
    
    bool isValidColumn(DataSchemaItem column)
    {
        // Use to Filter out invalid columns.
        return true;
    }
        
    string getSqlColumnType(DataSchemaItem column)
    {			
        switch(column.DataType.FullName)
        {
            case "System.Guid": 
            {
                return "uniqueidentifier";
            }
            case "System.Int64": 
            {
                return "bigint";
            }
            case "System.Int16": 
            case "System.Int32": 
            {
                return "int";
            }
            case "System.Single": 
            case "System.Double": 
            {
                return "float";
            }
            case "System.Decimal": 
            {
                return "decimal(19,4)";
            }
            case "Simego.DataSync.Date": 
            case "System.DateTime": 
            {
                return "datetime";
            }
            case "System.Boolean": 
            {
                return "bit";
            }
            case "System.String": 
            {
                if(column.Length > 0 && column.Length < 4000)
                {
                    return $"nvarchar({column.Length})";
                }	
                if(column.ColumnName.EndsWith("codename"))
                {
                    return "nvarchar(100)";	 
                }
                break;
            }
        }
            
        // Default Column value
        return "nvarchar(4000)";
    }
    
    static string getHash(string input)
    {
        using (var sha1 = System.Security.Cryptography.SHA1.Create())
        {
            // Convert hash bytes to hex string
            return string.Concat(sha1.ComputeHash(Encoding.UTF8.GetBytes(input)).Select(b => b.ToString("x2")));            
        }
    }
}

You can adjust this code to work best for you its just a guide on the process you may want look at how the SQL Columns are defined and potentially implement some column filtering via the isValidColumn function.

Test Project

You can now test this project by running in the designer you should see an 'account' table is created and there is a result in the 'Compare Results' window to write to the SQL Table.

Note: The Designer doesn't always update the UI when running dynamically like this only when re-opening the Data Sync project file.

Command Line

You can now run this on the command line to process each entity you want to sync.

"%DATASYNCINSTALLPATH60%\Simego.DataSync.Run.exe" -e project.dsprj -p:entity=account
"%DATASYNCINSTALLPATH60%\Simego.DataSync.Run.exe" -e project.dsprj -p:entity=contact

Ouvvi

You can use the Run Tool Project step in Ouvvi to run these and pass the entity name in the App Settings.

By running these via the RunTool step means they are run in a new process which has a performance hit on loading the Dynamics Schema. We are building a new Step Handler for Ouvvi to allow you to run these steps In-Process from a single project file and pass the project property configuration.

The Ouvvi Step should be a RunTool Project Step where we can pass the Entity name via the App Settings. By using the value {{StepName}} we can inject the name of the Ouvvi Project Step as the Entity name.

Improving Performance - Incremental Sync

Requires an update to Data Sync Version 6.0.3388

We can improve the performance of the synchronisation process by using an incremental sync where possible and using a new Ouvvi Step handler to ensure the steps run In-Process with the Ouvvi Service.

To implement Incremental Sync we need to change the Data Sync Mode to return those records which have a greater version number since the last sync. Dynamics entities have a column versionnumber which we can use with a Fetch XML Filter to return only those changed records.

We need to store the last version number in our metadata SQL Table and then use this to filter the source records on the next sync. Use the SQL below to add a new column to the metadata table to store the version number.

ALTER TABLE [dbo].[_dsTableMetadata] ADD [VersionNumber] bigint NOT NULL DEFAULT(0)

By changing our code to check for this column we can switch the sync mode in Project Automation and apply this filter at runtime. The code below includes the changes to switch to incremental mode and update the Fetch XML Filter at runtime to only return those changed records.

using System;
using System.Collections.Generic;
using System.Data;
using System.Globalization;
using System.Linq;
using System.Text;
using Simego.DataSync;
using Simego.DataSync.Engine;

class ProjectAutomationOverride : Simego.DataSync.Automation.ProjectAutomationShim //Do Not Change This Line
{    	
	public override void Start()
	{
		// Set the Entity on DataSource A
		DataSourceA.Entity = Properties["entity"];		
		// Set the Entity in DataSource B
		DataSourceB.SourceTable = $"[dbo].[{DataSourceA.Entity}]";		
		// AutoMap the Schema		
		AutoSchemaMap = true;					
	}
		
	public override DataSchema GetDataSchema(DataSchema schema)
	{
		// Set the Sync Mode to Normal
		SyncMode = SyncModeEnum.SyncAtoB;
		// Reset the Fetch Expression
		DataSourceA.FetchXmlFilterExpression = null;
				
		// Load the Source Schema
		var schemaA = DataSourceA.GetDefaultDataSchema();
	    
		// Get the Key columns for the PK.
		var keyColumns = string.Join(", ", schemaA.GetUniqueColumns().Select(p => $"[{p.ColumnName}]"));
		
		// Get the column list
		var columns = string.Join(",\r\n", schemaA.Map.Where(p => isValidColumn(p)).Select(column => $"\t[{column.ColumnName}] {getSqlColumnType(column)} {(column.AllowNull ? "NULL" : "NOT NULL")}"));
		
		// Generate a Hash of the columns to detect if any have changed.
		var columnHash = getHash(columns);
				
		// Check if the entity Table already exists in the DB. By getting the existing Hash from our _dsTableMetadata table.
		var existingHash = DataSourceB.ExecuteScalar($"SELECT Hash FROM _dsTableMetadata WHERE Name = '{DataSourceB.SourceTable}'") as string;
		if (existingHash == null)
		{									
			// Create Table Script
			var tableScript = getTableScript(keyColumns, columns);
			
			// Create the Target Table
			DataSourceB.ExecuteNonQuery(tableScript);	
			
			// Update the _dsTableMetadata Table
			DataSourceB.ExecuteNonQuery("INSERT INTO [dbo].[_dsTableMetadata] (Name, Hash, Script) VALUES (@p0, @p1, @p2)", DataSourceB.SourceTable, columnHash, tableScript);
			
			// Write to Log
			Trace.WriteLine($"Created Table '{DataSourceB.SourceTable}'");			
		} 
		else if(existingHash != columnHash)
		{			
			// Create Table Script
			var tableScript = getTableScript(keyColumns, columns);
			
			// Update the _dsTableMetadata Table
			DataSourceB.ExecuteNonQuery("UPDATE [dbo].[_dsTableMetadata] SET Hash=@p0, Script=@p1, Updated=getutcdate() WHERE Name=@p2", columnHash, tableScript, DataSourceB.SourceTable);
			
			// Drop the existing Table
			DataSourceB.ExecuteNonQuery($"DROP TABLE {DataSourceB.SourceTable}");	
			
			// Create the Target Table
			DataSourceB.ExecuteNonQuery(tableScript);	
									
			// Write to Log
			Trace.WriteLine($"Updated Table '{DataSourceB.SourceTable}'");					
		}
		else
		{	
			// If this Entity has a 'versionnumber' column set the project to an incremental sync.
			if(schemaA.Map.Any(p => p.ColumnName == "versionnumber"))
			{				
				var previousVersionNumber = DataSchemaTypeConverter.ConvertTo<Int64?>(DataSourceB.ExecuteScalar($"SELECT VersionNumber FROM _dsTableMetadata WHERE Name='{DataSourceB.SourceTable}'")) ?? 0;
		
				if(previousVersionNumber > 0)
				{				
					// Set the Sync Mode to Incremental
					SyncMode = SyncModeEnum.SyncAtoBIncremental;
					
					// Filter based on checkpoint.
					var fetchXml = $"<order attribute='versionnumber' descending='false'/><filter type='and'><condition attribute='versionnumber' operator='gt' value='{previousVersionNumber}'/></filter>";

					// Set the DataSource Filter
					DataSourceA.FetchXmlFilterExpression = fetchXml;
				}
			}
		}
		
		// Get the Target Schema
		var schemaB = DataSourceB.GetDefaultDataSchema();
		
		// Return an Auto-Map
		return AutoSchemaMap ? schemaA.ToAutoMap(schemaB) : schema;
	}
	
	
	public override void End(ProjectAutomationResult result)
	{
		if(result.Success)
		{
			if(result.CompareResult.Schema.Map.Any(p => p.ColumnName == "versionnumber"))
			{
				// Update the VersionNumber in the Metadata Table
				DataSourceB.ExecuteNonQuery($"UPDATE _dsTableMetadata SET VersionNumber = (SELECT COALESCE(MAX(versionnumber), 0) FROM {DataSourceB.SourceTable}) WHERE Name='{DataSourceB.SourceTable}'");
			}
		}	
	}
	
	string getTableScript(string keyColumns, string columns)
	{
		return string.Concat(
				$"CREATE TABLE {DataSourceB.SourceTable} (\r\n",
				// Append the Columns
				columns,
				")\r\n",
				// Add the Primary Key
				$"ALTER TABLE {DataSourceB.SourceTable} ADD CONSTRAINT [PK_dbo_{DataSourceA.Entity}] PRIMARY KEY CLUSTERED ({keyColumns})\r\n");
			
	}
	
	bool isValidColumn(DataSchemaItem column)
	{
		// Use to Filter out invalid columns.
		return true;
	}
		
	string getSqlColumnType(DataSchemaItem column)
	{			
		switch(column.DataType.FullName)
		{
			case "System.Guid": 
			{
				return "uniqueidentifier";
			}
			case "System.Int64": 
			{
				return "bigint";
			}
			case "System.Int16": 
			case "System.Int32": 
			{
				return "int";
			}
			case "System.Single": 
			case "System.Double": 
			{
				return "float";
			}
			case "System.Decimal": 
			{
				return "decimal(19,4)";
			}
			case "Simego.DataSync.Date": 
			case "System.DateTime": 
			{
				return "datetime";
			}
			case "System.Boolean": 
			{
				return "bit";
			}
			case "System.String": 
			{
				if(column.Length > 0 && column.Length < 4000)
				{
					return $"nvarchar({column.Length})";
				}	
				if(column.ColumnName.EndsWith("codename"))
				{
					return "nvarchar(100)";	 
				}
				break;
			}
		}
			
		// Default Column value
		return "nvarchar(4000)";
	}
	
	static string getHash(string input)
    {
        using (var sha1 = System.Security.Cryptography.SHA1.Create())
        {
			// Convert hash bytes to hex string
            return string.Concat(sha1.ComputeHash(Encoding.UTF8.GetBytes(input)).Select(b => b.ToString("x2")));            
        }
    }
}

Ouvvi

We can now use a new Ouvvi Step handler to run a Data Sync project from a file and pass the name of the entity as a project property just like the Run Tool Step. Use the Data Sync Project File step type with a User Setting pointing to the actual file path so it can be easily changed.

To add additional entities to the export you only need to use the Copy Step function and name it the same as the Dynamics entity.

Now you should see that the sync can run through much quicker as those with no changes will be processed quickly.

Data Sync Connector for JSON Web APIs

info@simego.com (Sean Cleaver) — Tue, 30 Sep 2025 10:55:33 +0000

We have come up with a new way to easily connect Data Sync V6 to virtually any JSON based Web API.

Features

Read/Write Data Access (New! Write access as of version 6.0.3300)
Define Multiple Datasources
Automatic API Definition Discovery
Parametrized configuration variables
Authentication
- OAuth2 (authorization_code and client_credentials flows)
- Basic Authentication
- Http Headers
- Authorization Header (Bearer Authentication)
For Each Loops
Paging Requests
HTTP Fetch (GET, POST, PUT, PATCH, DELETE)
Read Json File
Lookups
Connection Registry
XML based API definition
API GUI Designer

JSON API Designer

The JSON API Designer Tools->JSON API Tool in Data Sync makes it easy to create the XML API definition for the API. The Designer also allows for quick API configuration by way of an Automatic configuration from a URL, We will load the JSON from the URL and attempt to define a Schema and Fetch operation automatically. From the Designer you can quickly connect your API to the source or target of your project.

In the screenshot below we have loaded an API definition for multiple datasources which also use a Dynamic Schema and OAuth2 based authentication.

Auto Generate API Specification

You can automatically generate the API specification from a URL using the Add Datasource from URL function. This will attempt to download the json response and build the datasource specification from the response.

Simple API Definition

The connector is based on a simple XML Schema which defines the operations required for the connector to function. Data Sync connectors require 2 main functions GetDataSchema which must return the datasource schema to define the schema map. GetDataTable which returns the data as a Data Table.

The API XML definition below describes a simple API and defines a Static Schema Map, URL to fetch the response and a Data Transform to fill the Data Table from the response.

<Connector name="ODataV4 Sample" description="Connect to an OData V4 Feed" version="1.0">
  <Datasources>
    <Datasource name="People">
      <Actions>
        <Action name="GetDataSchema">
          <StaticSchemaMap>
            <Column name="UserName" type="System.String" unique="true" allownull="false" />
            <Column name="FirstName" type="System.String" length="255" />
            <Column name="LastName" type="System.String" />
            <Column name="Emails" type="JArray" />
          </StaticSchemaMap>
        </Action>
        <Action name="GetDataTable">
          <Fetch url="https://services.odata.org/TripPinRESTierService/People">
            <DataTableTransform path="value" />
          </Fetch>
        </Action>
      </Actions>
    </Datasource>
  </Datasources>
</Connector>

Data Paging

Many JSON APIs implement some form of Data paging where they limit the number of records returned in the response. These can be implemented in many random ways and we have built a PagingRequest function to handle the most common.

Page Counter

The most simple is the Page counter where we keep incrementing a page number until the response returns an empty response.

<PagingRequest path="data" start="1">
  <Fetch url="{URL}/app/list/{Datasource.Name}?limit={PageSize}&amp;page={PagingRequest.Page}">
    <DataTableTransform path="data" />
  </Fetch>
</PagingRequest>

Next Page Token

The next page token method is based on extracting the value for the next page and passing it in the next request until the next page token is empty.

<PagingRequest path="data" start="1" next-path="nextpage">
  <Fetch url="{URL}/app/list/{Datasource.Name}?limit={PageSize}&amp;page={PagingRequest.Next}">
    <DataTableTransform path="data" />
  </Fetch>
</PagingRequest>

Alternatively if the Next Page token returns a URL to the next page of results then you can use it like this. Where you pass the first page in the next token which sets up the initial request and the next page URL is returned from next-path token. The paging process will complete when the value of the next-page token returns null or empty string.

<PagingRequest path="data" start="1" next="{URL}/app/list/{Datasource.Name}?limit={PageSize}" next-path="next">
  <Fetch url="{PagingRequest.Next}">
    <DataTableTransform path="data" />
  </Fetch>
</PagingRequest>

Next Page Token with Continuation Parameter

Requires DataSync Version: 6.0.3532+

This is a variation on the Next Page token for when the last page returns the current page token instead of null to signal the end. Instead it uses another value to tell you whether to continue or not.

Here we use while-path and while-path-value which are used with a string equality check to decide if the loop should continue. In the example below the loop continues all the time meta.has_more equals 'True'.

<PagingRequest path="data" start="1" next="{URL}/app/list/{Datasource.Name}?limit={PageSize}" next-path="next" while-path="meta.has_more" while-path-value="True">
  <Fetch url="{PagingRequest.Next}">
    <DataTableTransform path="data" />
  </Fetch>
</PagingRequest>

Skip & Take

If your API is based on a Skip and Take page counter then you can use the PagingRequest like this. Where you can use the internal PagingRequest.Count counter to manage the row index. start is used to set the initial counter value. The process continues until the API returns an empty response.

<PagingRequest start="0" path="data">
	<Fetch url="{URL}/app/list/{Datasource.Name}?skip={PagingRequest.Count}&amp;take={PageSize}">
		<DataTableTransform path="data" />
	</Fetch>
</PagingRequest>

Indexed Paging

Indexed paging is based on paging through rows using an Index. Each iteration of the page we move the index along according to the page size parameter value. The paging stops when the API returns an empty response or HTTP status code 204.

So you will see requests like this

GET /app/list/products?startIndex=0&endIndex=10
GET /app/list/products?startIndex=11&endIndex=20
GET /app/list/products?startIndex=21&endIndex=30
GET /app/list/products?startIndex=31&endIndex=40

<PagingRequest page-size="{PageSize}">
    <Fetch url="{URL}/app/list/{Datasource.Name}?startIndex={PagingRequest.StartPageIndex}&endIndex={PagingRequest.EndPageIndex}"> 
        <DataTableTransform path="data" />
    </Fetch>
</PagingRequest>

Parent Child

Perhaps you need to call the API multiple times based on the results of an API call. In this case we have the ForEach function which can be used to loop over the results. In the example below we get a List of Suppliers and then Get a List of Products for each of those Suppliers whilst still using a PagingRequest.

<Fetch url="{URL}/app/list/suppliers">
  <!-- For Each SupplierID get the related Products -->
  <ForEach path="data">
    <!-- Handle Paging for the Product List -->
    <PagingRequest path="data" start="1" next-path="nextpage">
      <Fetch url="{URL}/app/list/products?filter=SupplierID%20eq%20{$SupplierID}&amp;page={PagingRequest.Page}">
        <DataTableTransform path="data" />
      </Fetch>
    </PagingRequest>
  </ForEach>
</Fetch>

Stacked Response

Perhaps you need to extract multiple parts from the Json Response, in this example we pull out the all the bank holidays from UK gov feed and assign a static value to the region column.

<Fetch url="https://www.gov.uk/bank-holidays.json">
  <DataTableTransform path="england-and-wales.events">
    <ColumnMap value="england-and-wales" to="region" />
  </DataTableTransform>
  <DataTableTransform path="scotland.events">
    <ColumnMap value="scotland" to="region" />
  </DataTableTransform>
  <DataTableTransform path="northern-ireland.events">
    <ColumnMap value="northern-ireland" to="region" />
  </DataTableTransform>
</Fetch>

Post Request

If you need to Post data to your endpoint rather than a Get request. You can specify the method of the request and add a body.

<Action name="GetDataTable">
    <Fetch method="POST" url="{URL}">
        <Body content-type="application/x-www-form-urlencoded">
            <Item name="Item1" value="value1" />
            <Item name="Item2" value="value2" />
        <Body>
        <DataTableTransform path="value" />
    </Fetch>
</Action>

You can also specify the Body as a raw body value by using the BodyContent element.

<Action name="GetDataTable">
    <Fetch method="POST" url="{URL}">
        <BodyContent content-type="application/x-www-form-urlencoded">
            Item1=value1&Item2=value2
        <BodyContent>
        <DataTableTransform path="value" />
    </Fetch>
</Action>

Open File

You can read json from a local file as well by using the OpenFile action and specifying the path to the file.

<Datasource name="fxrates">
  <Action name="GetDataSchema">
    <StaticSchemaMap>
      <Column name="Currency" data-type="System.String" key="true" />
      <Column name="Rate" data-type="System.Decimal" />
    </StaticSchemaMap>
  </Action>
  <Action name="GetDataTable">
    <ReadFile path="fxrates.json">
      <DataTableTransform />
    </ReadFile>
  </Action>
</Datasource>

Authentication

We have several Authentication options where you only need to specify the authentication scheme to use in the Authentication section. All subsequent requests will then use the authentication details.

Basic Authentication

You can specify that the connector uses Basic Authentication by adding the BasicAuth authentication scheme.

<Authentication>
  <BasicAuth username="username" password="password" />
</Authentication>

OAuth2 Authentication

You can specify that the connector uses OAuth either client_credentials flow or authorization_code flow by adding the OAuth2 authentication scheme.

<Authentication>
  <OAuth2 authorise-url="{authorize-url}" token-url="{token-url}" client-id="{client-id}" client-secret="{client-secret}" grant-type="authorization_code" scope="{scope}" prompt="select_account" />
</Authentication>

You can now use certificate based OAuth2 flows via the certificate-thumbprint attribute.

<Authentication>
    <OAuth2 token-url="{token-url}" client-id="{client-id}" grant-type="client_credentials" scope="{scope}" certificate-thumbprint="{thumbprint}" />
</Authentication>

Http Headers

You can add HttpHeaders to the request by adding multiple HttpHeader types to the authentication scheme.

<Authentication>
  <HttpHeader name="OData-Version" value="4.0" />
  <HttpHeader name="OData-MaxVersion" value="4.0" />
</Authentication>

Authorization Header

You can specify an Http Authorization Header like this.

<Authentication>
 <AuthorizationHeader type="Bearer" token="{api-token}" />
</Authentication>

Using Project Automation

Should you require calling an API to get an access token but the API is not OAuth2 but some home grown API then you can use Project Automation to call the API manually and setup the Authorization Header.

Enable Project Automation and implement the Start() method like this. This will call the Authentication API and add the access token to the API definition at runtime.

public override void Start()
{
	var helper = new  HttpWebRequestHelper(); 
	var response = helper.PostRequestAsJson(new { email = "username", password = "password" }, "https://localhost:2026/api/login"); 
	
	DataSourceA.AuthorizationHeader.Token = response["access_token"]?.ToObject<string>();            
}

Or you could manually call an OAuth endpoint like this to set the Token on the AuthorizationHeader.

public override void Start()
{
	var helper = new HttpWebRequestHelper();
	var result = helper.PostRequestAsString(new Dictionary<string, string> {  { "client_id", "client_xyz_" }, { "client_secret", "secret_xyz_"}, { "grant_type", "client_credentials" } }, HttpWebRequestHelper.MimeTypeApplicationWwwFormUrlEncoded, "http://localhost:8080/oauth/token");
		
	DataSourceA.AuthorizationHeader.Token = (string)HttpWebRequestHelper.FromJsonToDictionary(result)["access_token"];
}

Column Mapping

The DataTableTransform action will automatically map columns from the json response to the DataTable based on the schema map mapping the columns by name (case-sensitive). If you want to manage the mapping or add additional mappings then you can add ColumnMap nodes to the DataTableTransform.

For example mapping a Json node Currency to a Schema Column named CCY. The Path here is a JsonPath expression and therefore can select complex nodes.

<DataTableTransform>
 <ColumnMap path="Currency" to="CCY" />
</DataTableTransform>

If you want to map a static value to a column you can also do this with the value attribute like this.

<DataTableTransform>
 <ColumnMap value="1" to="version" />
</DataTableTransform>

If you need to pull a value from the parent of the current json context. You can use the parent-path attribute.

<Action name="GetDataTable">
  <!-- Fetch the list of invoices from a json file -->
  <ReadFile path="invoices.json">
   <!-- For each invoice, get the related Line Items -->
   <ForEach path="invoices">
    <DataTableTransform path="line_items">
     <!-- Get the Invoice ID from the parent element -->
     <ColumnMap parent-path="id" to="invoice_id" />				
      </DataTableTransform>
   </ForEach>
 </ReadFile>
</Action>

If you need to select part of a value from a string inside an json element you can use a Regular Expression with the expression attribute. The first group match will be returned as the value for the column.

<DataTableTransform path="contacts">
    <ColumnMap path="address" to="_internalID" expression="R:(\d{3,})" />
</DataTableTransform>

To select the OData ETag element use this format because of the @ in the name.

<ColumnMap path="$.['@odata.etag']" to="etag" />

Debugging

You can add a column to the StaticSchemaMap to hold a copy of the row json object so you can see the raw json per row in the Data Sync data preview. You then add a ColumnMap to the DataTableTransform to map the json to the column like this below.

<Datasource name="fxrates">
  <Action name="GetDataSchema">
    <StaticSchemaMap>
      <Column name="Currency" data-type="System.String" key="true" />
      <Column name="Rate" data-type="System.Decimal" />
      <Column name="__json" data-type="JToken" />
    </StaticSchemaMap>
  </Action>
  <Action name="GetDataTable">
    <ReadFile path="fxrates.json">
      <DataTableTransform>
        <ColumnMap path="$" to="__json" />
      </DataTableTransform>
    </ReadFile>
  </Action>
</Datasource>

You can also trace to the Data Sync Output window the entire Json Document by using the TraceElement action type like this.

<Datasource name="fxrates">
  <Action name="GetDataSchema">
    <StaticSchemaMap>
      <Column name="Currency" data-type="System.String" key="true" />
      <Column name="Rate" data-type="System.Decimal" />
    </StaticSchemaMap>
  </Action>
  <Action name="GetDataTable">
    <ReadFile path="fxrates.json">
      <DataTableTransform />
      <TraceElement />
    </ReadFile>
  </Action>
</Datasource>

Putting it all together

The example below puts everything together to create a OAuth2 authenticated JSON API connection to return a Products dataset by Looking up the Products for Each Supplier and paging the returned results.

<!--
An Example REST API connector which will use 2 requests
 * First to return a List of Suppliers
 * Second to loop around the Suppliers and return a List of Products for each Supplier.
-->
<Connector name="Simego Web API Connector" description="Connect to Simego Web API" version="1.0">
  <Parameters>
    <Parameter name="URL" value="https://demo.online.simego.com" />
    <Parameter name="PageSize" value="10" />
    <Parameter name="ClientID" value="" />
    <Parameter name="ClientSecret" value="" />
  </Parameters>
  <Authentication>
    <OAuth2 token-url="{URL}/oauth/token" client-id="{ClientID}" client-secret="{ClientSecret}" grant-type="client_credentials" />
  </Authentication>
  <Datasources>
    <Datasource name="Products">
      <Action name="GetDataSchema">
        <!-- Define Static Schema for this Datasource -->
        <StaticSchemaMap>
          <Column name="ProductID" data-type="System.Int32" />
          <Column name="SupplierID" data-type="System.Int32" />
          <Column name="ProductName" data-type="System.String" />
        </StaticSchemaMap>
      </Action>
      <Action name="GetDataTable">
        <!-- First get a List of Supplier ID values -->
        <Fetch url="{URL}/app/list/suppliers?columns=SupplierID&amp;limit=1000">
          <!-- For Each SupplierID get the related Products -->
          <ForEach path="data">
            <!-- Handle Paging for the Product List -->
            <PagingRequest path="data" start="1" next-path="nextpage">
              <Fetch url="{URL}/app/list/products?filter=SupplierID%20eq%20{$SupplierID}&amp;limit={PageSize}&amp;page={PagingRequest.Page}">
                <DataTableTransform path="data" />
              </Fetch>
            </PagingRequest>
          </ForEach>
        </Fetch>
      </Action>
    </Datasource>
  </Datasources>
</Connector>

Or in this example we can call the Microsoft Graph API whilst using OAuth2 authorization_code flow.

<Connector name="MS Graph API" description="Test quering Graph API via JSON API" version="1.0">
  <Parameters>
    <Parameter name="url" value="https://graph.microsoft.com/beta" />
    <Parameter name="authorize-url" value="https://login.microsoftonline.com/cfb0271d-de7d-4732-9414-957ca405620b/oauth2/v2.0/authorize" />
    <Parameter name="token-url" value="https://login.microsoftonline.com/cfb0271d-de7d-4732-9414-957ca405620b/oauth2/v2.0/token" />
    <Parameter name="client-id" value="" />
    <Parameter name="client-secret" value="" />
    <Parameter name="scope" value="https://graph.microsoft.com/.default offline_access" />
  </Parameters>
  <Authentication>
    <OAuth2 authorise-url="{authorize-url}" token-url="{token-url}" client-id="{client-id}" client-secret="{client-secret}" grant-type="authorization_code" scope="{scope}" prompt="select_account" />
    <HttpHeader name="OData-Version" value="4.0" />
    <HttpHeader name="OData-MaxVersion" value="4.0" />
  </Authentication>
  <Datasources>
    <Datasource name="users">
      <Actions>
        <Action name="GetDataSchema">
          <StaticSchemaMap>
            <Column name="id" type="System.String" />
            <Column name="givenName" type="System.String" />
            <Column name="mail" type="System.String" />
          </StaticSchemaMap>
        </Action>
        <Action name="GetDataTable">
          <Fetch url="{url}/users">
            <DataTableTransform path="value" />
          </Fetch>
        </Action>
      </Actions>
    </Datasource>
  </Datasources>
</Connector>

Or in this example we call the Dynamics CRM/Dataverse API whilst using OAuth2 authorization_code flow with a certificate. This example shows how to implement data paging with an ODataV4 endpoint.

<Connector name="Dynamics365" description="" version="1.0">
	<Authentication>
		<HttpHeader name="OData-Version" value="4.0" />
		<HttpHeader name="OData-MaxVersion" value="4.0" />
		<HttpHeader name="Prefer" value="odata.maxpagesize=10" />
		<OAuth2 token-url="https://login.microsoftonline.com/cfb0271d-de7d-4732-9414-957ca405620b/oauth2/v2.0/token" client-id="9a598195-082a-4d67-99b9-392c6c3e727d" grant-type="client_credentials" scope="https://simego.api.crm4.dynamics.com/.default" thumbprint="84FC24B07BFBCD08F0D5E5B45F" />
	</Authentication>
	<Datasources>
		<Datasource name="accounts">
			<Action name="GetDataSchema">
				<StaticSchemaMap>
					<Column name="accountid" data-type="System.Guid" />
					<Column name="name" data-type="System.String" />
					<Column name="etag" data-type="System.String" />
				</StaticSchemaMap>
			</Action>
			<Action name="GetDataTable">
				<PagingRequest path="value" next="https://instance.api.crm4.dynamics.com/api/data/v9.2/accounts?$select=accountid,name" next-path="$.['@odata.nextLink']">
					<Fetch url="{PagingRequest.Next}">
						<DataTableTransform path="value">
							<ColumnMap path="$.['@odata.etag']" to="etag" />
						</DataTableTransform>
					</Fetch>
				</PagingRequest>
			</Action>
		</Datasource>
	</Datasources>
</Connector>

Writing Back

We have created a few different ways you can write back to the API. Since its likely you might need extra flexibility in how the data is formatted or sent to the API endpoint.

Identifier Column

The first concept to understand is the role of the Identifier Column with Data Sync connectors. These values are attached to the row internally within Data Sync and are a way to maintain the datasource PrimaryKey even when its not part of the Schema Map. By maintaining this Identifier Column value it makes it possible to call UPDATE/DELETE actions on the target by the PrimaryKey value.

To ensure that the Identifier Column is set you need to specify the column from the Datasource which contains the Primary Key value like this within the DataTableTransform element.

<DataTableTransform path="data">
  <IdentifierColumn data-type="System.Int32" path="ProductID" />
</DataTableTransform>

Default Automatic Mode

This is the default no-code option where the data is serialized to JSON and sent to the API endpoints using the URLs and Http Method(s) defined within the Writer element.

<Datasource name="products">
	<Action name="GetDataSchema">
		<StaticSchemaMap>
			<Column name="ProductID" data-type="System.Int32" />
			<Column name="ProductName" data-type="System.String" />
			<Column name="SupplierID" data-type="System.Int32" />
			<Column name="CategoryID" data-type="System.Int32" />
			<Column name="QuantityPerUnit" data-type="System.String" />
			<Column name="UnitPrice" data-type="System.Decimal" />
			<Column name="UnitsInStock" data-type="System.Int32" />
			<Column name="UnitsOnOrder" data-type="System.Int32" />
			<Column name="ReorderLevel" data-type="System.Int32" />
			<Column name="Discontinued" data-type="System.Boolean" />
		</StaticSchemaMap>
	</Action>
	<Action name="GetDataTable">
		<PagingRequest start="0" path="data">
			<Fetch url="{URL}/ListApi?skip={PagingRequest.Count}&amp;take={PageSize}">
				<DataTableTransform path="data">
					<IdentifierColumn data-type="System.Int32" path="ProductID" />
				</DataTableTransform>
			</Fetch>
		</PagingRequest>
	</Action>
	<Writer name="Default">
		<AddItem method="POST" url="{URL}/ListApi" />
		<UpdateItem method="PUT" url="{URL}/ListApi/{Identifier0}" />
		<DeleteItem method="DELETE" url="{URL}/ListApi/{Identifier0}" />
	</Writer>
</Datasource>

For example this mode would send a HTTP response like this when Adding an item to the datasource.

POST http://localhost:2026/ListApi
Content-Type:application/json
Accept:application/json
{"ProductID":74,"ProductName":"Longlife Tofu","SupplierID":4,"CategoryID":7,"QuantityPerUnit":"5 kg pkg.","UnitPrice":10.0,"UnitsInStock":0,"UnitsOnOrder":20,"ReorderLevel":5,"Discontinued":false}

Should the API require the data within another object the you can also specify a single path element via the path attribute.

<Writer name="Default">
	<AddItem method="POST" url="{URL}/ListApi" path="data" />
	<UpdateItem method="PUT" url="{URL}/ListApi/{Identifier0}" path="data"/>
	<DeleteItem method="DELETE" url="{URL}/ListApi/{Identifier0}" />
</Writer>

The Default writer will only include the changed column values in UPDATE actions in the Json Body. If you need to send all the column data again you can configure the writer by setting a mode value of AllColumns this will send all column values as part of the JSON body in the request.

<Writer name="Default">
    <AddItem method="POST" url="{URL}/ListApi" />
    <UpdateItem method="PUT" url="{URL}/ListApi/{Identifier0}" mode="AllColumns" />
    <DeleteItem method="DELETE" url="{URL}/ListApi/{Identifier0}" />
</Writer>

To send only the Key column values from the Schema Mapping as part of the JSON body for UPDATE set the mode to AllKeys.

<Writer name="Default">
    <AddItem method="POST" url="{URL}/ListApi" />
    <UpdateItem method="PUT" url="{URL}/ListApi/{Identifier0}" mode="AllKeys" />
    <DeleteItem method="DELETE" url="{URL}/ListApi/{Identifier0}" />
</Writer>

To send only the Identifier column values set the mode to AllIdentifiers and provide a comma separated list of column names in the identifier-columns attribute to use for the Identifier column values.

<Writer name="Default">
    <AddItem method="POST" url="{URL}/ListApi" />
    <UpdateItem method="PUT" url="{URL}/ListApi/{Identifier0}" mode="AllIdentifiers" identifier-columns="ProductID" />
    <DeleteItem method="DELETE" url="{URL}/ListApi/{Identifier0}" />
</Writer>

Project Automation Callbacks

This mode allows for full control of how the API is called via a C# delegate callback on each item. In this mode we setup the Writer element with a name of Callback and then configure the callback function in Project Automation.

<Datasource name="products">
	<Action name="GetDataSchema">
		<StaticSchemaMap>
			<Column name="ProductID" data-type="System.Int32" />
			<Column name="ProductName" data-type="System.String" />
			<Column name="SupplierID" data-type="System.Int32" />
			<Column name="CategoryID" data-type="System.Int32" />
			<Column name="QuantityPerUnit" data-type="System.String" />
			<Column name="UnitPrice" data-type="System.Decimal" />
			<Column name="UnitsInStock" data-type="System.Int32" />
			<Column name="UnitsOnOrder" data-type="System.Int32" />
			<Column name="ReorderLevel" data-type="System.Int32" />
			<Column name="Discontinued" data-type="System.Boolean" />
		</StaticSchemaMap>
	</Action>
	<Action name="GetDataTable">
		<PagingRequest start="0" path="data">
			<Fetch url="{URL}/ListApi?skip={PagingRequest.Count}&amp;take={PageSize}">
				<DataTableTransform path="data">
					<IdentifierColumn data-type="System.Int32" path="ProductID" />
				</DataTableTransform>
			</Fetch>
		</PagingRequest>
	</Action>
	<Writer name="Callback">
		<AddItem url="{URL}/ListApi" />
		<UpdateItem url="{URL}/ListApi/{Identifier0}" />
		<DeleteItem url="{URL}/ListApi/{Identifier0}" />
	</Writer>
</Datasource>

Within the Start() method we setup the callback methods for ADD/UPDATE/DELETE. These will then be called during the synchronisation process for you to call the API.

For convince we pass you an object for calling HTTP methods the HttpWebRequestHelper with any authentication headers set and within the ProjectAutomationItemInfo object we have the target URL in the Target property. The Data property contains a Dictionary<string, object> of the data converted to match the target schema.

Therefore a simple implementation which would match the Default mode would be implemented like this.

public override void Start()
{		
	DataSourceB.SetWriterCallback(AddItem, UpdateItem, DeleteItem);
}

public void AddItem(HttpWebRequestHelper helper, ProjectAutomationItemInfo info)
{
	helper.PostRequestAsJson(info.Data, info.Target);
}

public void UpdateItem(HttpWebRequestHelper helper, ProjectAutomationItemInfo info)
{
	helper.PutRequestAsJson(info.Data, info.Target);
}

public void DeleteItem(HttpWebRequestHelper helper, ProjectAutomationItemInfo info)
{
	helper.DeleteRequestAsJson(null, info.Target);
}

Project Automation Batch Callback

Requires Data Sync Version 6.0.3398

This mode allows for full control of how the API is called in batches via a C# delegate callback on each batch of items. In this mode we setup the Writer element with a name of BatchCallback and then configure the callback functions in Project Automation.

In this example the API expects an Array of items like this below. Therefore we will need to wrap the array of results into an anonymous object to add the items element.

{
    "items": [
        { ... },
        { ... }
    ]
}

Since we need to include the item identifier in the body of the request rather than the URL we use the mode option of AllIdentifiers to ensure that the items ID is included in UPDATE and DELETE operations.

To Adjust the size of the Batch include a Parameter named UpdateBatchSize where you can specify the size of each batch of items.

<Connector name="Simego Web API Connector" description="Connect to Simego Web API" version="1.0">
  <Parameters>
    <Parameter name="URL" value="http://localhost:8080" />
    <Parameter name="PageSize" value="10" />
    <Parameter name="UpdateBatchSize" value="10" />
  </Parameters>
  <Datasources>
    <Datasource name="Products">
      <Action name="GetDataSchema">
        <StaticSchemaMap>
          <Column name="ProductID" data-type="System.Int32" />
          <Column name="ProductName" data-type="System.String" />
          <Column name="SupplierID" data-type="System.Int32" />
          <Column name="CategoryID" data-type="System.Int32" />
          <Column name="QuantityPerUnit" data-type="System.String" />
          <Column name="UnitPrice" data-type="System.Decimal" />
          <Column name="UnitsInStock" data-type="System.Int32" />
          <Column name="UnitsOnOrder" data-type="System.Int32" />
          <Column name="ReorderLevel" data-type="System.Int32" />
          <Column name="Discontinued" data-type="System.Boolean" />
        </StaticSchemaMap>
      </Action>
      <Action name="GetDataTable">
        <PagingRequest path="data" start="1" next-path="nextpage">
          <Fetch url="{URL}/app/list/{Datasource.Name}?limit={PageSize}&amp;page={PagingRequest.Page}">
            <DataTableTransform path="data">
              <IdentifierColumn data-type="System.Int32" path="WS_ID" />
            </DataTableTransform>
          </Fetch>
        </PagingRequest>
      </Action>
      <Writer name="BatchCallback">
        <AddItem url="{URL}/app/list/{Datasource.Name}" />
        <UpdateItem url="{URL}/app/list/{Datasource.Name}" mode="AllIdentifiers" identifier-columns="WS_ID" />
        <DeleteItem url="{URL}/app/list/{Datasource.Name}" mode="AllIdentifiers" identifier-columns="WS_ID" />
      </Writer>
    </Datasource>
  </Datasources>
</Connector>

public override void Start()
{		
	DataSourceB.SetWriterBatchCallback(AddItem, UpdateItem, DeleteItem);
}

public void AddItem(HttpWebRequestHelper helper, ProjectAutomationBatchInfo info)
{
	helper.PostRequestAsJson(new { items = info.Data }, info.Target);
}

public void UpdateItem(HttpWebRequestHelper helper, ProjectAutomationBatchInfo info)
{
	helper.PutRequestAsJson(new { items = info.Data }, info.Target);
}

public void DeleteItem(HttpWebRequestHelper helper, ProjectAutomationBatchInfo info)
{		
	helper.DeleteRequestAsJson(new { items = info.Data }, info.Target);
}

Since we are processing items in Batch this mode does not support Project Automation Item events. If you need to do further processing we would suggest you do that in the callback function instead.

Conclusion

In closing we hope you will be able to see how easy it will be to connect to JSON style APIs with Data Sync and this new connector.

Introduction to MirraSQL

info@simego.com (Rebecca Allen) — Fri, 27 Feb 2026 12:39:33 +0000

MirraSQL is a Windows command-line utility that imports Dataverse entities directly into SQL Server tables. It handles the connection to both systems, creates or updates your SQL schema to match Dataverse, and imports your records — all from a single command.

MirraSQL is currently in beta. Download it here and send us your thoughts to help shape what gets built next.

How It Works

MirraSQL connects to your Dataverse environment and your SQL Server database using a saved configuration profile. When you run an import, it:

Reads the entity schema from Dataverse
Creates or updates the matching SQL Server table automatically
Imports the records and reports progress in real time

You don't need to pre-create your tables or define schemas. MirraSQL takes care of that, just make sure you have a database ready to hold the tables.

Use Cases

Below you can see a couple of use cases where we think MirraSQL will prove beneficial.

Use Case	Benefit
Reporting and BI	Run Power BI, SSRS, or Excel reports against SQL instead of your live Dataverse — meaning faster reports and lower API usage
Data Backup and Archiving	Maintain a queryable SQL snapshot of your Dataverse data that doesn't depend on Dataverse availability
Feeding a Data Warehouse	Get your raw Dataverse entities into a staging schema ready for transformation into your dimensional model
Custom Reporting	Give your teams access to Dataverse data via standard SQL queries, without Dataverse API access or additional licences
Audit and Compliance	Keep a SQL-based historical record of Dataverse data that can be queried and reported on with standard database tooling
Cross-System Analytics	Join Dataverse data with ERP, HR, and finance records by getting everything in the same SQL database
Performance Offloading	Move heavy analytical queries off Dataverse entirely to avoid API throttling and reduce load on your live environment
Supporting Legacy Systems	Feed Dataverse data into older internal systems that can query SQL Server but cannot connect to Dataverse directly
Development and Testing	Quickly populate DEV or UAT databases with a copy of production Dataverse data for integration testing or report development
Disaster Recovery Preparation	Ensure a recent, queryable copy of your critical entities is always available in SQL Server if Dataverse becomes temporarily unavailable

System Requirements

.NET Framework v4.8
Windows Server 2016 - 2025
Windows 10/11
SQL Server 2016 or later (including Express edition)
2 GB RAM
1 CPU Core
50 GB Disk Space

Installation

MirraSQL is available as a downloadable package for Windows Machines, making installation quick and easy. Dowload the installer package here to get up and running straight away.

Once installed open a command prompt and type MirraSQL to confirm it's working. You should see the help output with usage instructions and available commands. If it doesn't just search for the MirraSQL.cmd program to open it.

Setting Up Your Connections

Before running your first import you need to tell MirraSQL where to connect. You'll need two things:

Your Dataverse Connection String
Your SQL Server Connection String

If you're already running DataSync then you can grab these connection details from your Data Sync Connections.

1. Dataverse Connection String

For the Dataverse Connection String you will need:

AuthType=ClientSecret
The root URL of your Dataverse environment, for example https://yourorg.api.crm4.dynamics.com
The Client ID and Client Secret from the Azure Active Directory (EntraID) App used to access Dataverse. See the Microsoft Webpage here for more information on how to create this app.

The final connection string should look like the following:

AuthType=ClientSecret;url=https://yourorg.api.crm4.dynamics.com;ClientId=[CLIENT_ID];ClientSecret=[CLIENT_SECRET]

You can find more examples of connection strings for different authentication types in the Microsoft Dataverse documentation.

2. SQL Database Connection String

For the SQL Database Connection String you will need:

The server name enterd as data source
Database name entered as initial catalog
Integrated Security=true;encrypt=True;trustservercertificate=True

To get your SQL Server database connection string details open SQL Server Management Studio and go to database properties -> Connection details. Alternatively speak to your DBA who will be able to get it for you.

This should look something like the following:

data source=.\SQLEXPRESS;initial catalog=MirraSqlDb;Integrated Security=true;encrypt=True;trustservercertificate=True

The above example is where the server is a local SQL Express instance and the database "MirraSqlDb".

Set the Connections

Once you have both connection strings, set them in MirraSQL with these two commands:

MirraSQL config set source="AuthType=ClientSecret;url=https://yourorg.api.crm4.dynamics.com;ClientId=[CLIENT_ID];ClientSecret=[CLIENT_SECRET]"
MirraSQL config set target="data source=.\SQLEXPRESS;initial catalog=MirraSqlDb;Integrated Security=true;encrypt=True;trustservercertificate=True"

To confirm your settings were saved correctly:

MirraSQL config show

This will present the source and target connection strings with the Client Secret being hidden.

That's all the configuration you need to get started.

Your First Import

With your connections set up, importing entities is a single command. Pass the logical names of the Dataverse entities you want to import:

MirraSQL import account contact systemuser

MirraSQL will connect to both systems, create the SQL tables if they don't exist, and import the records.

The codeblock below shows the output for the first time sync of the account, contact and systemuser entities. MirraSQL will fetch the schema, create the tables, and then import the records to these tables:

MirraSql Version 1.0.0 - Demo Mode (Expires 10/05/2026)
Copyright 2026 Simego Ltd

Using profile: [default]

√ Connected to Dataverse
√ Connected to SQL Server

Importing 3 entities...

[1/3] [account] 186 inserted, 0 updated, 0 deleted (4.6s)
[2/3] [contact] 570 inserted, 0 updated, 0 deleted (3.14s)
[3/3] [systemuser] 202 inserted, 0 updated, 0 deleted (2.13s)

√ Import complete
√ 3 entities, total records 958 inserted, 0 updated, 0 deleted
√ Total time: 9.87 seconds

Note: Deletes are hard deletes as it is replicating the Dataverse environment into your Database.

You can pass as many entity names as you need in a single command:

MirraSQL import account contact lead opportunity product systemuser team

Each entity is then processed in sequence with its own record count and timing, and a summary is shown at the end.

MirraSql Version 1.0.0 - Demo Mode (Expires 10/05/2026)
Copyright 2026 Simego Ltd

Using profile: [default]

√ Connected to Dataverse
√ Connected to SQL Server

Importing 7 entities...

[1/7] [account] 0 inserted, 3 updated, 0 deleted (3.91s)
[2/7] [contact] 1 inserted, 4 updated, 0 deleted (2.5s)
[3/7] [lead] 0 inserted, 0 updated, 0 deleted (0.29s)
[4/7] [opportunity] 2 inserted, 0 updated, 0 deleted (2.57s)
[5/7] [product] 1 inserted, 0 updated, 0 deleted (1.18s)
[6/7] [systemuser] 0 inserted, 0 updated, 0 deleted (1.28s)
[7/7] [team] 41 inserted, 0 updated, 0 deleted (2.53s)

√ Import complete
√ 7 entities, total records 45 inserted, 7 updated, 0 deleted
√ Total time: 14.26 seconds

Incremental Imports

Once your tables are populated from a full import, you can switch to incremental mode to pull only new and modified records by keeping track of the 'versionnumber' in the MirraSql metadata table rather than reloading everything. Use --incremental or -i:

MirraSQL import account contact systemuser --incremental

MirraSql Version 1.0.0 - Demo Mode (Expires 10/05/2026)
Copyright 2026 Simego Ltd

Using profile: [default]

√ Connected to Dataverse
√ Connected to SQL Server

Importing 3 entities...

[1/3] [account] 0 inserted, 0 updated, 0 deleted (1.74s)
[2/3] [contact] 0 inserted, 0 updated, 0 deleted (0.2s)
[3/3] [systemuser] 0 inserted, 0 updated, 0 deleted (0.2s)

√ Import complete
√ 3 entities, total records 0 inserted, 0 updated, 0 deleted
√ Total time: 2.15 seconds

Incremental mode is particularly useful when you want to minimise the time and load of each run. The typical pattern is to run a full import first to populate your tables, then use --incremental for all subsequent runs.

To extend the functionality you can combine it with a profile and log file:

MirraSQL import account contact lead opportunity --incremental --profile prod --log-file "C:\Logs\mirrasql_log"

Note the log file will be overwritten if the command is run again writing to the same output.

Setting Up Profiles

Once you're comfortable with the basics, profiles let you store multiple named connection configurations. This is useful when you're working across different environments such as development, UAT, and production.

Create a named profile by adding --profile [PROFILENAME] to the config set commands. For example:

MirraSQL config set source="AuthType=ClientSecret;url=https://yourorg.api.crm4.dynamics.com;ClientId=[CLIENT_ID];ClientSecret=[CLIENT_SECRET]" --profile dev
MirraSQL config set target="Server=DEVSQL;Database=MirraSqlDb;Integrated Security=true" --profile dev

MirraSQL config set source="AuthType=ClientSecret;url=https://yourorg.api.crm4.dynamics.com;ClientId=[CLIENT_ID];ClientSecret=[CLIENT_SECRET]" --profile prod
MirraSQL config set target="Server=PRODSQL;Database=MirraSqlDb;Integrated Security=true" --profile prod

To check what's stored in a profile:

MirraSQL config show --profile prod

To see all configured profiles:

MirraSQL config list

Switching Profiles

You can then use --profile or -p on any import command to target that environment:

MirraSQL import account contact --profile dev

MirraSQL import account contact --profile prod

This makes it easy to test imports against a development database before running against production.

Additional Features

Targeting a Specific SQL Schema

By default MirraSQL creates tables in the dbo schema. Use --schema to target a different one:

MirraSQL import account contact systemuser --schema crm

This creates tables as crm.account, crm.contact, and crm.systemuser. Useful if you want to keep your Dataverse data isolated from other tables in the same database.

Controlling Page Size

By default MirraSQL fetches records from Dataverse in pages of 5,000. For very large entities you may want to reduce this:

MirraSQL import account --source-page-size 1000

Reducing the page size can help with entities with large string data like attachments and files.

Skipping Schema Synchronisation

If your SQL tables already exist and you don't want MirraSQL to check or modify the schema, use --skip-schema-sync:

MirraSQL import account contact --skip-schema-sync

This can speed up repeated imports once your tables are set up correctly.

Quick Reference

Arguments:
  <entities>   One or more Dataverse entity logical names to import (required)

Options:
  -s, --schema <schema>           Database schema [default: dbo]
  --skip-schema-sync              Skip schema synchronisation
  -i, --incremental               Incremental mode — only import new and modified records
  --source-page-size <number>     Records per Dataverse API page [default: 5000]
  -p, --profile <profile>         Configuration profile [default: default]
  -l, --log-file <path>           Path to write log output
  -?, -h, --help                  Show help and usage
  --version                       Show version information

MirraSQL and Data Sync — Complementary Tools

MirraSQL and Data Synchronisation Studio serve different purposes and work well alongside each other.

Use MirraSQL for Dataverse -> SQL imports for reporting, backup, or analytics. It is optimised for bulk entity imports.
Use Data Sync when you need complex field mappings, transformations, or integration between any combination of systems.

A common pattern is to use MirraSQL for nightly full Dataverse imports into a reporting database, while Data Sync handles specific integrations that require custom logic.

Try It Out

MirraSQL is currently in beta, but you can download it here and give it a try against your own Dataverse environment. We're actively collecting feedback on which features matter most, so your input can directly shape what gets built next.

If you have questions or run into anything unexpected, get in touch at support@simego.com.

Team Blog

Updating the Active Directory Manager Attribute

Contents

Requirements

Connect to your Source and Active Directory

Mapping

Preview the Compare Results and Sync

Automate & Schedule

Run via the Run Tool

Run via the Command Line

Run Tool Project

Data Sync Project

Automate with Windows Task Scheduler

Automate with Ouvvi

Creating New Active Directory Users

Contents

Requirements

Connect to your Source

Connect to Active Directory

Map the Source & Target Columns

Set the Default Password

Set the Manager Attribute

Enable User Accounts

Add to Groups

Preview and Sync the Results

Automate & Schedule

Run via the Run Tool

Run via the Command Line

Run Tool Project

Data Sync Project

Automate with Windows Task Scheduler

Automate with Ouvvi

Managing Active Directory User Group Membership

Contents

Requirements

Active Directory Group Connector Options

Export Group Membership

Export to a File

Export to a Business System

Manage Group Membership

Connect to your Source Data

Connect to your Active Directory Groups

Map the columns

Preview the Results & Sync

Adding Groups to User records

Bulk Update Active Directory

Contents

Requirements

Connect to your Source and Target

Mapping

Preview the Results and Sync

Creating Database Scripts from the Schema Map

Contents

Requirements

Connect to your Source

Generate the Script

Set the New Table as the Target

Map the Source & Target Columns

Compare the Results and Sync

Miscellaneous

Edit the Script Language

Edit the Table Name

Change the Database Connection Details

Execute Selected Script

Integrating with Salesforce

Contents

Requirements

Importing Data into Salesforce

Connect to your Source Data

Connect to your Salesforce Object

Map the Source & Target Columns

Lookup Account Number

Preview the Results and Sync

Exporting Data from Salesforce

Connect to your Salesforce Object

Connect to your SQL table

Map the Source & Target Columns

Compare the Results and Sync

Additional Options

Filtering the Results