View All Blog Posts

Supporting Mongo DB Connections

Data Sync is now able to support Mongo DB connections to add, update, delete and extract data to and from Mongo DB

We've been being asked by a number of customers to support Mongo DB connections in Data Sync. So we are pleased to announce that we have listened and have developed a new connector that supports MongoDB.

This is currently a custom connector available on our Simego Git Hub account, but will most likely become part of our default providers in the future.

Mongo DB

Some examples of how this connector could be used are :

  • Syncing data between two Mongo Databases.
  • Syncing data from a data source to Mongo to act as a system backup.
  • Syncing data to Mongo so that it can be processed. For example your source might have API limits or run slow, so by adding the data to Mongo you can process it and synchronise it to other data sources.
  • Exporting data from Mongo to another system.

Requirements

Before getting started you need to ensure you have the following:

  • Windows 10 or Windows Server
  • Downloaded & Installed Data Synchronisation Studio
  • Have Visual Studio or another IDE installed
  • Have access to a Mongo Database

If you do not have Data Synchronisation Studio you can get a free evaluation edition.

Get the Connector

To use this connector you need to copy the code from Github into Visual Studio and build the code. To do this browse to the MongoDb code on our Simego Git Hub account, click onto the green code button in Github and copy the web url.

Copy Link Github

Now open Visual Studio, select Clone a repository, enter in the URL to the connector to the Repository location and then set the path for the repository to be created in.

Visual Studio - Clone Repo

The next step is to open the solution file (double click onto the .sln file).

Open Solution File

You then need to build the project by clicking F5 or going to Debug > Start Debugging. This will run Data Sync and load the Mongo Connector into the provider list.

Build the Code

To access the connector, open the connection window and go to Others > Mongo DB.

Mongo DB Blank Connection

Connect to Mongo DB

To connect to your Mongo databases you will need your connection string. You can find this in Mongo by going to Databases > Connect > Connect your Application.

Copy the connection string and make sure to edit it so that you change <password> to use your actual password.

Mongo Connection String

Enter this connection string into the ConnectionString field. If the connection string is valid you should now be able to view a list of your Databases in the Database drop down. Select the database to connect to and then select the Collection.

Once you are done click Connect to connect to your Mongo DB.

Mongo DB Connector

Getting Data into Mongo

If you have a clean collection in your database with no data in it, you will need to ensure that one document is added to Mongo so then Data Sync can discover a schema (the columns).

Luckily we have made it easy to get a JSON file in Data Sync, you can now preview your data and export it as a JSON file.

In this case we only need one record to upload. To get this filter the source data to return one record e.g. ProductID == 1.

Then go to the schema map and click onto Preview A, then click onto the Export JSON button in the preview tab.

Export JSON

We can then open this file copy the data from within the array. We now need to go to Mongo to import it.

JSON Data to Copy

In Mongo, click onto your collection and then select INSERT DOCUMENT. Click onto the curly braces to switch to the JSON view, remove the text already there and paste the JSON we copied a moment ago into the window. Click Insert to insert it.

Insert Document

Now in Data Sync, refresh your connection by clicking onto the refresh button in the data source toolbar. You should now have columns available and ready to map.

Columns Available

Map the Columns

The next step is to map your source columns to the columns in Mongo and make sure to set a key column. The key column should be unique and be able to be used to identify each record. If this is not unique, your data will show as having duplicates which will be excluded from the sync.

Mapping

Compare, Preview and Sync

Once you have mapped your columns you can run the compare to view the differences between your source and target. Click onto the Compare A > B button in the toolbar and the results will then be shown on screen. You can view the additions, updates and deletes to be made if there are any.

Please note that deletes are disabled by default. To enable them please set EnableDelete to True on the target properties.

In this example when we run the compare there are 76 records to add to Mongo DB.

Preview Data

If you have a large data set to add or delete and the project takes a while to run, you should find that setting UpdateBatchSize to 25 in the target properties gives you a better performance. Please note that updates will still do 1 record at a time.

Once you are ready click Synchronise and then Start to begin the sync. The documents will then be imported to Mongo DB.

Synchronise

Make sure to save the project so that you can use it again in the future.

Miscellaneous

Debugging

Whilst you are running the code in Visual Studio you can set a break point in the code and step through it to actively de-bug the code.

Set your break point, in this example I have it set on the Data Reader at the GetDataTable method, so that I can check what is being returned when I click Preview Data in the data source window.

Once I have clicked the Preview Data button I can now step through the code and when I get to the itemRow I can now see one of the records being returned.

Debugging

Deploying the Connector to Data Sync

Once you are confident that it is working as you need you can install the connector to your local Data Sync.

Right click onto the project in the solution window and go to Show in File Explorer. Then click into the foldersbin > Debug > net472. This will show you all the DLL files, you need. Copy across the one's that do not currently exist in the program files, to the Data Sync program files at the path C:\Program Files\Simego\Data Synchronisation Studio 3.0.

The DLLs you need should be :

  • DnsClient.dll
  • MongoDB.Bson.dll
  • MongoDB.Driver.Core.dll
  • MongoDB.Driver.dll
  • SharpCompress.dll
  • Simego.DataSync.Providers.MongoDb.dll
  • System.Buffers.dll
  • System.Runtime.CompilerServices.Unsafe.dll
  • System.ValueTuple.dll

Now when you open Data Sync from your machine you will be able to access the Mongo DB connector.

This example helps to show you how you can create, debug and build your own custom connectors for Data Sync. For more guidance on building your own custom connector take a look at our Connector SDK documentaion or send us an email.

| Wednesday, February 9, 2022 |