Keep upto date with everything going on at Simego.
20 February 2014
We have a new Data Provider available for Intercom, which provides full Read/Write Capability. Allowing you to Import, Export and Synchronise data between many different systems and Intercom.
For example with this you could set up an Intercom to Dynamics CRM Lead Capture integration, or use Data Sync to decorate your Intercom data with data from your own internal databases.
The Full Source Code for the Intercom Provider is available in the Data Sync Installation Directory so that you can study it to build similar REST API providers or to adapt it to your own needs.
19 February 2014
We’ve added a new feature to make it super easy to start building your own data providers for Data Sync. This feature is currently in beta so you need release 3.0.808 or greater.
The feature from a Click of a Button creates a Visual Studio Project with a Template Project that can be opened in Visual Studio and give you F5 run and Debug.
This new Project Opened in Visual Studio 2012
Full F5 Debugging via Visual Studio to help you build the provider.
This allows you to code up whatever you want and then deploy the assembly to Data Sync and use it just like the providers we have built.
We often get asked how long does it take to write a provider for Data Sync. It really depends on the system you connect to and the API it exposes, however you can generally be up and running with a simple read-only provider in a couple of hours. We recently created a complete read-write provider for Intercom which is a JSON REST based API and this took approximately 8 hours over a weekend to complete.
6 February 2014
Quick video showing how easy it is now to connect to your source or target data, preview the data and attach a Lookup to the data source.
3 February 2014
We’re getting ready to release a major productivity boost for Data Synchronisation Studio. It will now be possible to store connection details to your systems and then use the NEW Connection Library schema explorer to connect to your data sources.
You simply browse to the object you want to connect to and just drag and drop the object onto the connection source or target and that’s it!
You can even create lookups by just dragging the related object to the Column in the Datasource to create the Lookup connection.
The connection Library explorer will support all the major providers, SQL Server, ODBC, OleDB, SharePoint, Dynamics CRM, Podio, Active Directory and Simego Online WebStore.
15 January 2014
At Simego we have been thinking and working hard with our clients to utilise emerging web based API standards to enable simple B2B and departmental integration. At its core is Simego WebTable and surrounding it is a feature complete cloud data and integration environment.
What will B2B look like using Simego Online and Web Table
Essentially, your business analyst selects a set of data from a system or file and publishes to your clients using modern web standard and security.
Using a simple a Desktop Application (Simego DS3), the analyst choose the source and creates a new Web Table.
The Simego WebTable exposes the data in a modern web API, which is accessible through all of the emerging standards. In addition Simego WebTable makes the data accessible through excel and the client can even use Simego DS3 to integrate their WebTable data with other systems.
Hedge Fund A has several clients, all of whom want access to ‘their data’ in as real time as possible, in this case trades. Classically their options would be to create a complex integration involving FTP sites, servers, certificates which would be time consuming and risky. We would recommend using Simego Online and a ‘Web Table’ to synchronise client data in near real-time. This would allow each client to choose what format they would like to download the data and you could choose the level of security.
Using Web Table they could quickly give their client’s access to their trades create a strong and unique relationship:
Each client relationship contained in a separate container:
The data con be synchronised up to the client trades Web Table:
Separate cloud based projects can be set up to synchronise just the client portion of data to their
trades web table. Each project has its own details audit and change control:
This project moves only the specific data relating to that client to their Web Table. The move is triggered by any change in the source data. This makes the updates to the data available near real-time. Alternatively, a synchronisation can be scheduled periodically. In addition the system can send notification emails when a synchronisation is finished.
To be able to manage any integration it is critical to have transparent reporting, clear dashboards and deep audit details.
Once the data is synchronised with the clients WebTable, they can access it as if it was a download link or use it as an integration point for more advanced integration scenarios using xml, json, csv or as a source for DS3 to synchronise with internal client systems:
If you would like a demonstration on how WebTable and Simego can make your business relationships unique and more profitable then please ask to see a demo or a trial invite from email@example.com.
13 January 2014
As part of Simego’s new Online services you can publish data online to share with your clients and partners. To create a Web Table is really easy with Data Synchronisation Studio.
First connect to your Source Data in DS3.
Then from the Tools menu select Create Web Store List
Then enter the credentials to your Online Account, choose your Workstream and give your List a Name. (If you do not have a Workstream you will need to create a Workstream in your online account first.)
This will then create the structure of your List/Table in your Workstream under the Storage section.
Now you just need to Compare A->B and synchronise your Data into the Online List. Once this is done your Data is now available online in multiple formats.
We currently support CSV, XML and JSON formats for the data depending on the URL you use.
You can also filter the data with simple URL parameters i.e. to filter records where Supplier ID = 1 we apply a filter parameter SupplierID eq 1
Now we have a filtered view of the Data.
13 January 2014
Today we’re Announcing availability of Simego Online.
Simego Online is an advanced hybrid-cloud service for Data Integration allowing you to run simple and complex integrations in whatever location suits your business. You can choose whether to execute integrations on-premise or in the cloud from a single platform.
Included in the new platform is the new Simego WebTable technology. Simego WebTable enables business to publish and consume data via standard methods such as CSV, XML and more advanced API publishing technology JSON using industry standard security methods using simple desktop applications and NO CODE.
Both on-premise and cloud integrations can utilise these ‘WebTables’ to create simple and powerful integration solutions enabling secure, near real-time and advanced B2B in minutes.
You can choose where to run your workloads either OnPremise via our familiar products Data Synchronisation Studio and Simego Automation Server (Ouvvi) or In our Online Environment hosted in Microsoft Azure.
Enabling B2B solutions and internal integrations could not be easier. Quickly register with Simego Online and either evaluate the platform using a 'Trial' account or create a new package. Each package contains integration minutes, a DS3 Cloud Edition key for your business and the ability to create integration Workstreams.
Data Synchronisation Studio – Online Edition
Data Synchronisation Studio – Online Edition is a version of the full DS3 client that integrates with the Online Platform.
The Online Edition of DS3 links to your Online account and uses credit from your account each time you run a sync. Your can DS3 online license can be installed on multiple workstations and servers and used by multiple users.
Workstreams are the Online equivalent of Simego Automation Server (Ouvvi). Create a Workstream to host your integration projects and steps and run them in the cloud via our hosted environment. Create Schedules and trigger projects from external events or via HTTP REST API.
Web Tables allow you to create lists of Data in the Cloud this is a great way to share data with third parties without having to stand up new servers, databases etc. Simply use DS3 online to publish your data to the Cloud and allow your clients to access the Data securely via JSON, XML or CSV end points.
To find out more and for a demo account please contact our Team via our firstname.lastname@example.org
20 November 2013
Often it is necessary to keep a SharePoint Document Library synchronised with the local file system including sub-folders and Created Dates. This could not be easier using DS3!
Step 1. Connect the Source to a SharePoint Document Library
Step 2. Connect the Destination to the file system
Step 3. Map the fields and Compare A -> and Synchronise
Delete all other columns and create the following map:
Click the Compare A –> B button, then the Synchronise Button and then the Start Button and you will now have a SharePoint Document Library including subfolders and files on the file system.
18 November 2013
One of the challenges with Dynamics CRM Online, and the cloud in general, is integrating cloud data with all your internal touch points and requirements.
If you want to do some detailed analysis of your account, contact or any entity data from Dynamics CRM, exporting a complete entity can take time and be laborious. Some companies like to have a ‘backup’ in case of extended outages.
Recently a client asked if he could download his 925000 row account table!
Step 1: Download and install the trial (you can currently do this using the trial edition up to 10k rows without registering!)
Step 2: Open DS3 Developer and connect to Dynamics CRM Online
Click the green ‘Connect Data Source’ on the left.
Select the Dynamics CRM provider
Enter in Dynamics CRM Online Details:
- Authentication: Federated
- URL: https://crm4.Dynamics.com
- Pick your organisation then the entity name.
Step 3: Select the columns we want to download
We must tell DS3 which columns we want to extract from the entity. The more columns we include, the more data that has to be downloaded which is therefore slower (Dynamics CRM will bring down about 1000 records per second). If you want ALL of the available columns, check the tick beside the ‘Columns’ item in the columns tree view on the left.
Step 4: Use DS3 to create a CSV file for you.
In this step we use DS3 to create a CSV file with the same shape (columns) as your source entity.
Click tools on the menu bar, then Select the ‘Create CSV File’. Enter in the destination of the CSV file.
This creates a new CSV file and points the destination of the download to it.
Step 5: Compare and download the data
DS3 is more than just a download tool and is designed to synchronise two systems. In this case the CSV file is empty but we still have to compare it to the online entity.
For performance you can change the Page Size in the CRM settings to be 5000 (the max permitted).
1. Compare. Click the Compare A –> B so that DS3 can determine what to add to the CSV File.
DS3 will now download all of the data form the Dynamics Online Entity.
2. Synchronise. DS3 will prepare the synchronisation and present you with the ‘Synchronise Data’ Dialog.
3. Start. Click the start button and DS3 will now write the data to the CSV files.
When it has finished synchronizinga, all of the data will be written to the csv file.
All your records will now be in the CSV files, for some more interesting downloads you can download to a SQL Table, an XML File , an XML Dataset, an Excel Sheet etc etc.
This can be automated now using the DS3’s Server component to just incrementally download the changes.
Get Started and download here it is only 8 MB and no registration for the trial:
1 November 2013
It is more and more important to businesses to have up to date data to act upon. Customer service representatives informing their clients ‘If you wait a couple of minutes you should be able to log-in’ or ‘Your balance has not been updated yet, can I take a number or can you call back?’ are not what successful businesses are made of.
Earlier on this year an Aberdeen Group report discovered that business leaders wanted more and more access to real-time data.
David Linthicum, the author wrote “The report noted that 89 percent of enterprises that use real-time integration have the power to provide managers with accurate information when it is needed, as opposed to only 73 percent of organizations that do not use real-time services,”
This on face value looks like a 16% difference in capability (or competitive advantage). In reality, it is a huge difference due to the source of data no longer being from clean, managed data store but from core transactional systems.
“This includes updating the real time data that is flowing to the decision maker, as the data changes over a given period of time. For instance, the ability to track factory production over an afternoon, as the production data changes minute-to-minute.
“In other words, we are moving from a report-oriented mentality to a dashboard-oriented mentality.”
Who really needs Real-Time?
Finance and banking solutions are very reliant on systems being in-sync due to the timely nature of balances, pricing and transactions. Information such as available balances, current status of accounts and the actual cost of a transaction are required seconds after their change. As mentioned above, businesses who want to be able to build new processes should think hard of the benefits of not having to wait until later or worse an overnight update to make decisions.
What is involved in integrating in Real-Time?
There are obviously three critical stages to integrating data from one system into Dynamics CRM in ‘Real-Time’:
- Detection of the change in the source data.
- Identifying if this change impacts on a Dynamics CRM destination entity.
- Updating or Inserting data in the Dynamics CRM destination entity.
In addition there critical dimensions to the data changes:
- The amount of entities changed in the source entity (eg Start of Day balance updates) in a time window.
- The time it takes to process an update or inserts vs the next required update window .(eg If there are 100 changes in 30 seconds and the integration solution takes 0.8 seconds to process a change serially then they system will never be in sync).
Critically Real-Time integrations have to be ‘in sync’ before starting the real-time data integration process. This is possible with preparation and good data management but the most difficult moment is when the source system or the Dynamics CRM system are unavailable. There must be a process by which once reconnected, the system will bring themselves back in sync. Recovery from unavailability, is the hardest problem in real-time synchronisation.
An example of Dynamics CRM Integration
As a demonstration I am going to configure a Real-Time synchronisation between a Microsoft SQL Server Table and a Dynamics CRM entity.
For the example to be realistic we are going to test two scenarios and record the shortest and longest time from the change in the source systems to the update completing in the target system:
- Simulate a ‘Large Single Update’ such as a balance update in the morning or mass change in data. Updating 3000 records in the SQL Table a Dynamics CRM entity with 6000 items.
- Simulate a ‘Regular Change’ of data in the source system: 5 updates every 5 Seconds from a table with 6000 records targeting a Dynamics CRM entity with 6000 items .
Detection of the change in the source data
To detect the change in the source table I attached a DS3 Monitoring Trigger (this is not a SQL ‘Trigger’ attached to the table). This monitoring trigger activates a DS3 Integration project when it detects a insert or update to the source table. There are many Monitoring triggers built into DS3 Server (File Monitor, SharePoint List, Http result etc etc). This should table about 1 minute to set up .
Identifying if this change impacts on a Dynamics CRM destination entity.
Create a new DS3 project mapping between the source SQL table and the target entity.
Updating or Inserting data in the Dynamics CRM destination entity.
I uploaded the DS3 project into DS3 Server. I can now connect the Trigger to the project.
We don’t have to worry about recovery in this example as DS3 detects the last successful sync and automatically synchronises from there. This technique is only possible if you have a high performance insert/update process as you have to ‘catch-up’ i.e. Insert/update 100x faster that required to keep the system in sync.
Scenario 1: 6000 Instant Updates
|Source SQL Table Updated||16:11:23||Start|
|First Update Appears in Dynamics CRM||16:11:25||3 Seconds from Start|
|Last Update Completes in Dynamics CRM||16:11:42||9 Seconds from Start|
Scenario 2: 5 Updates every 5 Seconds
|Source SQL Table Updated||16:51:18||Start|
|First Update Appears in Dynamics CRM||16:11:21||3 Seconds from Start|
|Last Update Completes in Dynamics CRM||16:11:22||4 Seconds from Start|
If we are having updates every 5 seconds we would probably do away with the change detection and just schedule the updates every 5 seconds, thus reducing the load on the source. DS3 Server is also doing some clever stuff like stopping the trigger running another insert when detecting changes and making sure any updates that happened during a sync were processed.
Real-Time Dynamics CRM synchronisation is possible and manageable, I set this up in about 15 minutes and ran the tests over an hour. When we implement these systems with our clients, we also run a reconciliation process which verifies all the updates and delete have been completed. This is a simple job we schedule nightly and allows us to also manage deletes elegantly and gives the business managers a solid reconciliation position.
If you would like to discuss with us how you can increase the productivity of you Dynamics CRM solution through Real-Time integration, give us a call or email : email@example.com.