Team Blog

Calculated Columns UI Enhancements

info@simego.com (Sean Cleaver) — Mon, 20 May 2013 15:36:55 +0100

Just a quick post to show that we now have code highlighting and intellisense type support for our new calculated column feature.

For example below is a screenshot of our CASE function which can be used to transform a value into 1 of several possibilities.

We also have a full list of our supported functions here http://www.simego.com/Help/Functions

Calculated Columns

info@simego.com (Sean Cleaver) — Wed, 15 May 2013 20:54:25 +0100

We have a great new feature for Data Sync that we just have to tell you about, previously if you wanted to transform some value we had “Dynamic Columns” where you could write your own .NET C#. Now we have a simplified yet very powerful solution for when you just need something simple or if your not very good with .NET. We’ve tried to make it similar to other scripting solutions whilst keeping the ability to be really advanced if you need to.

For example below is a little snippet that adds 30 days to Today.

As with everything we do we’ve made it really easy just hit the (Fx) button on your Data Source to add a new column based on your Function.

We’ve added lots of useful functions so you can virtually do anything you need, plus you can even mix it with the existing Dynamic Columns feature.

Here’s a List of Current Functions with more on the way!

AND(params bool[])
BITWISEAND(params int[])
BITWISECOMPLEMENT(params int[])
BITWISEOR(params int[])
BITWISEXOR(params int[])
CHOOSE(int?, params string[])
CHOOSE(int?, params int?[])
CHOOSE(int?, params double?[])
CHOOSE(int?, params object[])
CONCAT(params string[])
CONCATENATE(params string[])
DATALENGTH(byte[])
DATEVALUE(string)
DAY(System.DateTime?)
FALSE()
FROMBASE64(string)
HOUR(System.DateTime?)
IF(bool, object, object)
IF(bool, string, string)
IF(bool, int, int)
INT(object)
ISINT(string)
ISNULL(object)
ISNULLOREMPTY(string)
ISNUMBER(string)
ISO8601DATE(System.DateTime?)
ISO8601DATETIME(System.DateTime?)
JOIN(string, params string[])
LEFT(string, int)
LEN(string)
LENGTH(string)
LOWER(string)
LTRIM(string)
MID(string, int, int)
MILLISECOND(System.DateTime?)
MINUTE(System.DateTime?)
MOD(int?, int)
MONTH(System.DateTime?)
NEWGUID()
NOT(bool)
NOW()
NULL()
NUMBER(object)
OR(params bool[])
PROPER(string)
REPLACE(string, string, string)
RIGHT(string, int)
RND()
RTRIM(string)
SEARCH(string, string)
SEARCH(string, string, int)
SECOND(System.DateTime?)
SPLIT(string, string)
STR(object)
TOBASE64(byte[])
TODAY()
TRIM(string)
TRUE()
UPPER(string)
UTCNOW()
WEEKDAY(System.DateTime?)
WEEKNUM(System.DateTime?)
YEAR(System.DateTime?)

How to insert a large amount of records into Dynamics CRM Online

david@simego.com (David Evans) — Mon, 13 May 2013 15:09:45 +0100

As more and more value is being derived from enhancing Dynamics CRM with integrated data, there has been an issue with the rate at which you can create and then maintain records with in your Dynamics CRM online environment. It is considered slow, expensive and difficult to achieve, this is more down to the implementations of the third party vendors rather than some failing of the infrastructure or Dynamics API design.

For this demonstration we are going to use a default Dynamics Online instance with no optimisation, I did not need:

Microsoft Server Products
Microsoft SQL Server Products including DTS or SSIS
Any type of message queuing
Any type of xml experience.

I am going to import 5000 simple contact records into the contact entity in a Dynamics CRM Online instance but this solution scales directly to 100,000 records or more (a later blog will introduce the how to run DS3 in parallel for high volume inserts). I did require:

The 5000 contact file can be found here (http://www.briandunning.com/sample-data/).
Data Synchronisation Studio v3 found here (http://www.simego.com/Products/Data-Synchronisation-Studio 8mb download runs on the desktop or server)

I connected the Source to the CSV file and selected the 5000 record contact file.

I then connected the destination to our dynamics CRM Online instance.

I mapped contact and address details to make sure the sync reflected a realistic insert.

The compare only takes about 2 seconds and DS3 predicts 5000 inserts and previews the records:

Clicking the synchronise button synchronises the 5000 records from the Source to the Destination in 4:06 seconds (20 records per sec)

This in itself is faster than most competing solutions but what is more important is the ability to alter the source system data and re-synchronise. This is where much migration pain is felt. For example we could decide that the web address is completely incorrect we are better to calculate it by using the last portion of the email address: ie. david.evans@simego.com –> http://www.simego.com.

We can achieve this by using a dynamic column which is just a calculated field:

I can then just map this Dynamic Field onto the websiturl in the contact entity.

Comparing the 5000 locally and the 5000 on Dynamics CRM Online takes a maximum of a couple of seconds. DS3 then shows you what it will be updated within Dynamics online. It will ONLY update columns that have been changed, meaning it will update other fields with may trigger workflows or plug-ins inadvertently.

We can now synchronise the updated column in less than 2 minutes (1:59) which is 42 rows per second. I could have stopped and restarted the sync at any time and it would have just set off where it stopped.

Whilst it is possible for other products to use parallel servers with message queues, it soon becomes very expensive and time consuming.My next post will demonstrate using DS3 in parallel which is only limited by the bandwidth and speed that Dynamics Online can process the incoming requests.

If you would like a quick demonstration of DS3's Dynamics CRM integration (or any other Sharepoint, Active Directory integration) please feel free to drop an email to support@simego.com. We really are always happy to show of our solutions and share some of our clients successes.

Exporting Data from Podio into SQL Server

info@simego.com (Sean Cleaver) — Thu, 14 Mar 2013 12:12:01 +0100

Here’s an example where were using Podio to allow users to enter details on forms upload pictures and then use Data Sync to integrate this data with SQL Server.

Data Sync will Synchronise the Form Data with a SQL Database and download the attached files into a Folder on the file system.

Here’s a typical form in Podio and we want to export this Meta Data to SQL and download the picture files.

First we connect Data Sync to this App in Podio and include the columns were interested in the schema map.

Using the Tools Create SQL Table Menu we automatically create a SQL Table from this Schema to import the Data.

Now everything is Mapped as we Expect.

The Next Step is to use the Project automation feature of Data Sync to add your own code to download the files from Podio and write them to the Local Disk.

The Podio Provider exposes a special “Podio” object that allows us to interact directly with the Podio API, we added a special GetFile Method to this object to make it easy to download files from Podio.

At the End of the Synchronisation process we iterate through the New items collection and get each file and write them to the disk with the code below.

#region Usings
using System;
using System.Collections.Generic;
using System.Data;
using System.Globalization;
using System.Text;
using System.Linq;
using Simego.DataSync;
#endregion

using System.IO;

/// <summary>
/// </summary>
class ProjectAutomationOverride : Simego.DataSync.Automation.ProjectAutomationShim //Do Not Change This Line
{    
    
    public override void Start()
    {
        
    }

    public override void BeforeCompare(ProjectAutomationLoadResult loadResult)
    {

    }

    public override void BeforeSynchronisation(ProjectAutomationCompareResult compareResult)
    {
            
    }

    public override void End(ProjectAutomationResult result)
    {
        if ( result.HasChanges ) 
        {
            foreach(var item in result.CompareResult.New) 
            {
                string path = Path.Combine(
                    @"C:\Temp\PodioExport", 
                    item.SourceRow[5].BeforeColumnValue.ToString());
                
                File.WriteAllBytes(path, 
                
                    DataSourceA.Podio.GetFile(item.SourceRow[6].BeforeColumnValue.ToString()));    
            }
        }
    }

    public override void Error(ProjectAutomationException e)
    {
        
    }

}

[Note: The Index values in the SourceRow collection above relate to the row position in the Schema Map]

Now after the Synchronisation the SQL Table is populated from Podio and the Local Disk contains copies of the Files that were attached to the Form Files.

This is just a simple example of how Data Sync can integrate with Podio to provide business value quickly and easily.

Connecting to Podio

info@simego.com (Sean Cleaver) — Thu, 14 Mar 2013 11:26:07 +0100

We currently have two providers for Podio one for Space Contact Items and one for Space Apps.

To connect to Podio you will need to locate the Podio providers in the list of Providers and the easiest way is to use the search function.

[Note: Leaving the OAuth settings blank will allow Data Sync to use our default application settings with Podio, you can override these with your own Podio API keys if you like.]

Click on the Credentials property and this will then bring up a web page where you need to authenticate with Podio and grant Data Sync permission to connect to your account.

Once connected you can then pick the App in Podio to connect to from the App Drop-down list.

The Silent property prevents updates to the app INSERT/UPDATE from being reported to your Activity Stream.

Clicking OK will then popup the Connection Library option which is a new feature that allows you to save credentials for use again later. The OAuth credentials will expire after 28 days so it’s a good idea to save these in the connection library so you can easily update them again later.

That’s it now your connected to Podio and you can start designing your integration or migration.

Integration with Podio

info@simego.com (Sean Cleaver) — Fri, 22 Feb 2013 14:33:20 +0100

Here at Simego we love Podio so it was natural that we developed a connector for Podio that would allow us to integrate Data from our LOB systems into Podio.

Out of the Box Podio lets you import Excel sheets and a few other options. But we need something that we can run on a Schedule which can take information captured through our Website and ensure it ends up in Podio.

We’ve made this all a bit too easy, simply connect Data Sync to your Source data and use the new Podio data provider. If your working with Contacts then you should use the Contacts provider. Set the Credentials which goes through the OAuth song and dance and then choose the Space in Podio to connect.

Then Map your data just as you would any other system, you need a Unique Key to relate records and Podio exposes a nice external_id column you can use to store the ID from your source system.

Then run through the normal Compare and Sync process and your new data will then be synchronised with data in Podio.

EMC Documentum Integration

david@simego.com (David Evans) — Thu, 21 Feb 2013 15:45:30 +0100

We have recently been working with some of our partners to allow them to integrate their client’s ‘Documentum’ environments using Data Sync Studio. Currently, this is still in development but early signs are that we will offer the integration as a product.

We have developed the product to be purely api based and it needs no connections to the underlying Database or content servers.

I am going to run through moving files between the desktop and a Documentum environment. We could just as easily move files from a Sharepoint Library to Documentum as well.

As usual we use the following simple integration steps in DS3:

Connect to a source which in this case is a folder on my desktop
Connect to the destination which is a Documentum folder within a Repository
Map any fields we require. This mapping is a feature built into DS3 and allows us to lookup information to update Documentum metadata about the document entity.
Press the ‘Synchronise’ button and the documents are sent up to Documentum with the meta data correctly set.

For this demonstration we are going to use some simple text files as examples but we can target more complex types in Documentum.

Step 1: Connecting to the file system source:

Select the folder containing the files you would like in Documentum.

Step 2 : Connecting to a Documentum Repository and Selecting in Folder

We have several options:

Credentials: Username, password and optionally Domain

DocumentType: The connector displays a dropdown listing all types within the repository.

DocumentumServerURL: The http address of the Documentum DFC url.

Timeout: A timeout in of any request.

ContentUpdateBehaviour: Under some circumstances we are not interested in integrating the content of a document just the attributes (contentless documents). We can choose to synchronise the content or not.

Folder: The connector allows you to browse the folders within the Repository to find a folder to sync.

RepositoryName: The Repository Name.

Step 3: Map the columns between the 2 Data Sources

There are several things to note about the connector:

It will automatically bring back the attributes for the selected ‘type’ In this case we are just using the vanilla ‘dm_document’ type in Documentum.
It will use the ‘dm_formats’ table to map files to their correct ‘Format’ in Documentum.
It supports repeating attributes with a string[], int[] attribute type.
Dynamic Columns are fully supported if we need to lookup values to write to attributes in the target types.

I added a new Dynamic Field called ‘Subject’ to map to the dm_document attribute Subject:

Step 3: Run the Synchronisation:

Comparing A->B (File Systems –> Documentum) and then synchronising moves the documents and updates the metadata accordingly:

When I delete some files from the source file system and change the Subject in the Dynamic Column to be ‘Food and Beverage’ we can update the new files.

Comparing Comparing A->B (File Systems –> Documentum) again DS3 shows us the updates it will make to Documentum:

You might notice the performance times, there is no caching and the Documentum Server (Data Source B) is 3000 miles away over the internet.

We can then see the update to the Subject in the Document properties:

As we progress through the development of this connector I will post some more blogs.

If you would like more information about this Documentum connector please just drop an email to sales@simego.com and we can organise a chat over tea.

Install Data Sync Team Server

info@simego.com (Sean Cleaver) — Mon, 07 Jan 2013 12:37:58 +0100

System Requirements

Microsoft Windows Server 2008/R2/2012
IIS 7.0 or Higher
Windows Authentication for IIS
.NET Framework 4.0 Full Profile
SQL Server 2008 or Higher configured for SQL Mixed Authentication Mode.

Step 1 – Data Synchronisation Studio

Install Data Synchronisation Studio as normal selecting either 64 bit or 32 bit as required for your system. Then activate Data Sync with your Server Edition License Key.

Step 2 – Data Sync Team Server

Install Data Sync Team Server as normal by running the install program this will install the IIS WebSite for Ouvvi and install the Ouvvi Processing Service.

Step 3 – DB Setup

The next step is to create an Ouvvi Database that this instance of Ouvvi will connect to. When you start the Ouvvi Website if there’s no configuration file it will start the setup process. Here you need to enter your Network Credentials these are your Windows Username & Password that has Admin rights for your SQL Server. The SQL Server Name and Instance Name such as LOCALHOST\SQLEXPRESS the name of the Database you wish to create and a SQL User account name and password (Setup will create this user).

Step 4 – License

Once the DB is configured which takes a few seconds you will be redirected to the home page with further setup instructions. The first thing is to enter your Server Edition license key.

Selecting “Projects” from the Menu will re-direct you to the License Page. Enter your Key and Click Activate.

Step 5 – Windows Service

Next you need to open the Windows Service Control Manager (services.msc) to configure and Start the Ouvvi Processing Service.

Locate the “Ouvvi Processing Service” set the user to a domain service account you want to use to run the Data Sync integration Tasks and press Start.

Now if you switch to the Services –> System Services Tab you will see all the background Tasks have started and are running.

NOTE: If you need to re-start the setup process you should delete the Ouvvi.config file from the Directory C:\ProgramData\Simego\Ouvvi and restart the OuvviWeb AppPool in IIS.

SQL Change Trigger - DS3 Team Server Feature

info@simego.com (Sean Cleaver) — Tue, 11 Dec 2012 11:05:14 +0100

Yesterday we blogged about the CRM Trigger that enables real-time synchronisation with Ouvvi / Data Sync Team Server. Today we’re going to cover off the other side where you can trigger a project to start from a change in your SQL Database (SQL Server, Oracle, DB2, MySQL, ODBC, OleDB etc).

This works in a similar way to the CRM Trigger except this time we query your SQL Database every 30 seconds and ask for the latest DateTime value of any changes.

To do this you need to write a query that would return the latest change timestamp from your database for the items your monitoring.

For example

SELECT MAX(modifiedon) FROM dbo.AccountBase

To set this up you Add a new Trigger to your Data Sync Team Server environment.

Choose the SQL Trigger Option

Then configure the Trigger, choosing the Network Library and connection string to your database and the Query to execute. (Yes you can use other database servers i.e. Oracle, IBM DB2, etc).

That’s it Ouvvi will start to Query your SQL DB every 30 seconds and when there’s a change projects linked to this trigger will be queued to start.

Now that the project is using a Trigger you can see under Reporting how the project is started through distribution of actual changes rather than a schedule.

If your Table does not have a valid DateTime column that gets updated when the record is changed then you can implement a solution based on SQL Triggers to maintain an updated timestamp value like this.

Create a new SQL Table like this in your Database

   1: CREATE TABLE [dbo].[OuvviTriggers] (   2:     [ID] int IDENTITY(1, 1) NOT NULL,   3:     [Name] nvarchar(255) NOT NULL,   4:     [Updated] datetime DEFAULT(getdate()) NOT NULL   5: )   6: GO

Then you can create a SQL Trigger on any Table that you want to monitor like this where you update the ‘TableA’ to match the name of the Table your monitoring.

   1: CREATE TRIGGER TableA_OuvviTrigger   2: ON [TableA]   3: FOR INSERT, UPDATE, DELETE   4: AS   5:     UPDATE [dbo].[OuvviTriggers]   6:         SET [Updated] = GETDATE()   7:     WHERE [Name] = 'TableA'   8: GO

Then your Ouvvi Trigger just becomes SELECT [Updated] FROM [OuvviTriggers] WHERE [Name] = ‘TableA’

CRM 2011 Real Time Synchronisation - DS3 Team Server Feature

info@simego.com (Sean Cleaver) — Mon, 10 Dec 2012 10:28:04 +0100

DS3 Team Server or Ouvvi has a Triggering system that can be used to provide almost real-time synchronisation. Internally Ouvvi has a 30 second clock that evaluates configured Triggers and when they need to be fired any projects associated with that trigger will be queued for execution. This feature is completely independent of your CRM system and does not require any special installation onto your CRM Server.

One Trigger that Ouvvi has is the CRM Entity Trigger, this trigger queries an Entity in CRM for any updates since the last change (createdon, modifiedon). If this changes then the Trigger is fired. The benefit to this method is that changes are replicated quickly and unnecessary scheduled syncs do not need to occur.

To create a new CRM Entity Trigger go to the Triggers view and choose Add Trigger

Then choose the CRM Entity Trigger

Then configure the Trigger by entering your CRM Server Details and the Entity you want to monitor. (leave credentials blank to use the Ouvvi service credentials)

That’s it …. you simply then attach this trigger to the Ouvvi projects you want to execute when changes are made to this entity. It’s important that your projects are configured to run in the Single Instance mode so that they do not run in Parallel.

Connect to SharePoint 2013

info@simego.com (Sean Cleaver) — Tue, 04 Dec 2012 16:49:13 +0100

For those of you planning to upgrade to SharePoint 2013 we thought we would let you know that DS works just fine with SharePoint 2013.

Connect as normal – we recommend you start using the Client API Provider for new SharePoint projects.

Browse your Site Schema to find the List you want to connect to.

Map your source to your target as normal

Once you synchronise your Data is now presented in SharePoint 2013.

Handling SharePoint Lookup Columns

info@simego.com (Sean Cleaver) — Mon, 03 Dec 2012 15:29:58 +0100

The SharePoint 2010 (Client API) Provider has an inbuilt feature to automatically match and link Lookup Columns based on the value associated with the List.

In this example we have created an Orders list and added one column that is a multi-value column that will contain the “Products” that relate to this Order.

To handle the lookup data we simply need to provide the name of the lookup value separated with a semi-colon(;) i.e. here we created an Excel file with the order details.

We then simply create a DS Project to Map this to our Orders List

Then once we synchronise this data the lookup is automatically connected.

During the Synchronisation we download the list items from the related SharePoint List and index into this list to get SharePoint’s internal identifier number to assign to the item being linked. Therefore items must be an exact match and they must exist for the link to be successfully created.

If your Lookup is simply a single item then you just need to return the Name/Title of that item.

Connect to Dynamics CRM 2011 Online

info@simego.com (Sean Cleaver) — Fri, 30 Nov 2012 15:19:57 +0100

To Connect to Dynamics CRM 2011 Online with Data Synchronisation Studio you will need the URL to your CRM Environment.

For example one test environment of ours is at https://simegoltd.crm4.dynamics.com for data sync you need to remove the Organisation name part so this becomes https://crm4.dynamics.com

This environment is pre-Office 365 Integration therefore if you have CRM Linked to your Office 365 Environment on one of the E plans then the Url will most likely be https://crm.dynamics.com

You then Enter this into the CrmServerUrl property of the Provider, For CRM Online with LiveId choose the LiveId Authentication Provider, for Office 365 Choose Federated.

Next Enter your credentials under Authentication and then Choose your Organisation name from the Drop-down this might take a little while as it discovers the Organisations that are linked to your account.

Next Choose an Entity to work with from the Entity Drop-down.

Then Click OK to connect.

Issues

If you get an error message when you try to connect that the system cannot load the Microsoft.IdentityModel assembly then you need to install the Windows Identity Framework (WIF) then re-start Data Sync for the changes to take affect.

Getting Starting with a Trial Evaluation

david@simego.com (David Evans) — Fri, 30 Nov 2012 09:41:55 +0100

Introduction - DS3 is so simple you can just 'Get Started'!

Data Synchronisation Studio v3 (DS3) is moves data between computer systems without the cost, effort and risk associated with developing a system integration solution. You point DS3 at the data source and then the target and DS3 makes one side equal to the other.

In more advanced cases DS3 will make both sides the same! Make change on either side and it synchronises with the other.

Starting an Evaluation

Step 1: Download the latest trial from http://www.simego.com/Install/Data-Synchronisation-Studio

Step 2: Click the zip file and then click on the Data Sync installer

Step 3: Approve running the installer

Step 4: Install the application and Finish

Step 5: Run Data Sync for the first time:

Getting a Trial License for DS3

Once started DS3 will ask for a key. Just send an Email to us requesting a Trial/Evaluation Key to support@simego.com, we will send you back a trial key within a few minutes and if you give us some detail on what you need to integrate eg, CRM + SQL, we have more quick start guides for many situations.

Step 1: Request Key.

Step 2: When you have your response back from support it will contain a key like this (this is not a valid Key!!):

QETSC3-SWSEZH-SRS4XM-S4EQHX-1YW4U6-HQ1BWA-U

Step 3: Click on the ‘Enter License Key’ button and paste in the Key.

Step 4: Click the OK Button to enter Data Synchronisation Studio

Using Lookups to Transform Values

info@simego.com (Sean Cleaver) — Tue, 27 Nov 2012 10:42:33 +0100

In this simple example we will demonstrate how you can use the Lookup functionality in Data Synchronisation Studio to Transform/Convert a value via an Lookup Data source.

Our source data contains the numbers 1-10 and we want to convert these into the names of the Numbers.

We need then to create a Transformation Data source and for this we’re going to use Excel. We could use virtually anything we like, SQL, CRM, SharePoint, XML, CSV etc but Excel is ideal for this.

This Excel file contains the From and To values for the Transformation, we need to create Column headers in the First row and then the Data below i.e.

Now we add the Lookup to the column (select the ID column, and press the ‘Add Lookup’ toolbar button) that we want to Transform, in our Example this is the ID column. First we select the Data source which for this example is Excel and browse to the Path of the Excel file.

Now we define the Lookup connection details, so we want to join the Lookup based on the Value in the From Column to the Value in the ID column in our source data. We also specify the Data Type as Int32 since this is a Number 1-10.

We then add the “Lookup1_To” column from the Data source to our Schema Map so that we return the Value of the Lookup Transformation.

There we have it a Transformation from an External Data source via Lookups.

This simple Transformation could be created in code via Dynamic Columns but this shows a no-code solution and has the added benefit that you can easily extend the transformation values by updating the Excel data source.

Connect to Office 365 Exchange Server

info@simego.com (Sean Cleaver) — Tue, 27 Nov 2012 10:17:26 +0100

To connect to Exchange Server with Data Synchronisation Studio you need to get the URL for your Exchange Server. First go to your Outlook Web Access and the URL for your Exchange server.

For example this one is https://dbxprd0710.outlook.com/owa/?exsvurl=1&realm=office365.com&wa=wsignin1.0

This URL need to be modified so that it points to the Exchange Web Services URL so we take the server part and change it to https://dbxprd0710.outlook.com/ews/Exchange.asmx

We then take this URL and enter it into the ExchangeUrl property of the “Exchange Web Services (Appointments)” provider.

We then enter our Exchange Credentials

We then add our Mailbox name to the Mailboxes collection, this details which mailbox data we wish to return.

And that’s it now you can return data from Office 365 Exchange Server.

Connect to 365 SharePoint Online

info@simego.com (Sean Cleaver) — Tue, 27 Nov 2012 10:04:01 +0100

To connect to a List/Document Library with Data Synchronisation Studio on SharePoint Online you need to get the URL to your site.

For example in the screenshot below our Team Site URL would be http://crmonline4u.sharepoint.com/TeamSite

You then enter this URL into the SharePointUrl Property of the “Microsoft Office 365 SharePoint Online” Data Provider in Data Sync.

You then need to enter the credentials for your SharePoint Online environment. Note that we do not currently support environments where you have federated authentication enabled to your own AD, standard 365 Authentication is fine.

Next select the List that you wish to connect to from the ListName drop down.

And click OK to connect and there you have it!

Issues

Importing CRM 2011 Connections

info@simego.com (Sean Cleaver) — Mon, 26 Nov 2012 12:20:33 +0100

To import/synchronise connection data in Dynamics CRM 2011 with Data Synchronisation Studio you need to define your schema map like this below.

record1idobjecttypecode and record2idobjecttypecode should return the names of the related entities for example.

CRM will automatically create the reverse connection which is a problem for Data Sync as it will see an extra record on the next sync. Therefore we filter this out by using an FetchXmlFilterExpression and just select the rows where ismaster is true as below.

   1: <filter type="and">   2: <condition attribute="ismaster" operator="eq" value="true"  />   3: </filter>

Empty String and NULL Behaviour

info@simego.com (Sean Cleaver) — Mon, 26 Nov 2012 11:00:43 +0100

By Default Data Sync detects NULL and an Empty String as a difference, however certain Data Sources such as CSV and SharePoint cannot differentiate between NULL and Empty String. Therefore we automatically override this behaviour for you when using these providers.

If you want to force NULL and Empty String to be considered Equal during the comparison you can do this within the Project Automation feature. Simply set the Property EmptyStringAndNullAreEqual to True on one of the Data source Tables using the BeforeCompare Method.

For Example

   1: public override void BeforeCompare(ProjectAutomationLoadResult loadResult)   2: {   3:     loadResult.TableA.EmptyStringAndNullAreEqual = true;   4: }

Ouvvi Data Sync Team Server - Configuration Management

info@simego.com (Sean Cleaver) — Fri, 31 Aug 2012 17:32:35 +0100

With Ouvvi Data Sync Team Server you can manage Data Sync project settings from a Central Location with the ‘User Settings’ page in Ouvvi.

This allows you to easily switch Databases from Test to Live, specify credentials etc.

When the Step is run inside Ouvvi or downloaded the configuration variables are expanded so that the value in the settings list is replaced with the value in the Data Sync project.

To enable a setting on a project to be managed this way you must edit the Data Sync project file before uploading into Ouvvi.

Open the project in Notepad and locate the configuration variables.

For example to enable the SharePointUrl to be managed this way add a value in ConfigKey below.

so this

becomes this

Then when you upload the project into Ouvvi it will create a new User Setting for OurSharePointSite with the current value that allows you to easily switch the value at runtime.

Importing Active Directory Contacts

info@simego.com (Sean Cleaver) — Fri, 31 Aug 2012 14:35:22 +0100

This guide shows how you can Import/Export or Synchronise Contact Data with Active Directory with Data Synchronisation Studio. (Requires Data Synchronisation Studio Release 3.0.586 or greater)

In this Example we have created a new OU called ‘Contacts’ that we will Synchronise our Contact Data into.

Connect Data Source A

We have created some sample Data and connected it to Data Source A to Synchronise with Active Directory. In this example we use the UserCode column as our Unique Key and we will store this value in the EmployeeId attribute in Active Directory to maintain the link between records.

Connect Data Source B

We need to specify a correctly formed LDAP Connection such as the one below for this.

LDAP://OU=Contacts,DC=corp,DC=simego,DC=com

We also need to modify the LDAPFilter property to return objects of Type ‘Contact’ for example (&(objectCategory=contact))

Next we need to adjust a few properties on the Active Directory provider to enable the creation of new records in your AD.

Set the following properties

SchemaClassName to ‘Contact’

SchemaItemFormat to ‘$FirstName$ $Surname$’

EnableAdd to ‘True’

EnableDelete to ‘True’ (Optional)

Then Setup your Schema Map to Map your Source Columns to Active Directory and in this case using the EmployeeID to Store an Identifier to the Contact record.

Now when you Compare and Synchronise the records are created in your AD.

Dynamics CRM 2011 N:N Relationships

info@simego.com (Sean Cleaver) — Fri, 31 Aug 2012 10:45:32 +0100

This guide will demonstrate how to create N:N (Many to Many) Relationships in CRM 2011 with Data Synchronisation Studio. (requires Data Sync release 3.0.586 or greater)

For this example we will use out of the box entities and show how to relate 1 or more Competitors to Opportunities.

Below shows an existing Opportunity linked to 3 Competitors in CRM we will create a Data Sync project that can create this automatically.

Configure Source Data

First up we need to configure what it is were going to import/synchronise with CRM. To keep things simple we will create a CSV File containing the Target Relationships.

CSV Data

Opportunity,Competitor
Opportunity 1, Competitor 1
Opportunity 1, Competitor 2
Opportunity 1, Competitor 3
Opportunity 2, Competitor 1
Opportunity 2, Competitor 2

This Data shows we are Mapping ‘Opportunity 1’ to Competitor 1,2 and 3 and ‘Opportunity 2’ to Competitor 1 and 2.

You will notice that we do not have the Target CRM Guids for the Competitor and Opportunity here so we will use the Data Sync Lookup Function to provide this transformation.

Configure Data Source A

First connect Data Source A to the CSV File that contains the relationships.

Next we define a Lookup for Opportunity –> CRM Opportunity.Name by adding a Lookup to the Opportunity Column linking to CRM Entity Opportunity and setting the Join Target column to the Name of the Opportunity.

Next we define a Lookup for Competitor–> CRM Competitor.Name by adding a Lookup to the Competitor Column linking to CRM Entity Competitor and setting the Join Target column to the Name of the Competitor.

Now we can test the Lookups by creating a Schema Map of OpportunityId and CompetitorId from the Lookup Columns.

Then Check that the Data Looks good.

Configure Data Source B

Connect Data Source B to the new “Microsoft Dynamics CRM 2011 – N:N” provider and choose the Relationship “opportunitycompetitors_association”.

Now configure your Schema Map so that you map the Lookup1_OpportunityId to OpportunityId and Lookup2_CompetitorId to CompetitorId and ensure that the Key Column is checked for both columns as were creating a Composite Key based on both values.

Now you can Compare and Sync to update CRM with the Data and when you check CRM you will see the Relationship is now configured.

Manually Setup Ouvvi : Data Sync Team Server

info@simego.com (Sean Cleaver) — Wed, 15 Aug 2012 08:53:16 +0100

This guide shows how to setup the default Ouvvi Database manually from a Script which maybe your only option when installing the database onto a remote SQL Server.

Ouvvi requires a Database on SQL2008, 2008R2 or 2012 and also runs with the free SQLEXPRESS editions. Your SQL Server must be configured to accept Standard SQL Accounts.

When Ouvvi starts it first offers the automatic option and this will connect to your Database server using Integrated Security from the Logged on user of the Browser. This can fail against remote servers or if this user does not have the required permissions.

Note: If you get an error trying to access the Ouvvi Web Page check that you have Windows Authentication Feature in IIS installed and activated on the Ouvvi Web Site.

To configure manually, select the option to generate a SQL Script of the database.

Now you need to enter some details for the script generation, specifically the SQL Database and SQL Username (SQL Stored Procedures will be granted Execute Permission with this Username) and the Ouvvi server that is running the Notification Service (Usually the current machine).

This will now create a SQL Script that you need to execute on your SQL Server to create the Database Structure and default Static Data.

Now you need to create a configuration file so that Ouvvi can connect to your database.

Create a new XML Text File called Ouvvi.config and place it in directory C:\ProgramData\Simego\Ouvvi

This File should contain the SQL Connection String to Ouvvi an example is shown below.

<?xml version="1.0"?>
<configuration>  
  <connectionStrings>    
    <add name="ConsoleDB" connectionString="data source=.\SQLEXPRESS;initial catalog=Ouvvi;integrated security=false;persist security info=true;User ID=Ouvvi_User;Password=password;" />  
  </connectionStrings>
</configuration>

You can then Click Home in your browser to take you back to Ouvvi home page.

You may get a Schema Error where the database needs upgrading and by following the same procedure you can generate a Script to bring the Database up to date.

Then once the Database is all configured and correct you can then access the Ouvvi Home Page. Which details the last few steps to start the Ouvvi Processing service and start creating projects.

Workaround API Limitations eg CreatedBy and ModifiedBy

info@simego.com (Sean Cleaver) — Thu, 09 Aug 2012 11:43:28 +0100

This is a little trick you can use with Data Synchronisation Studio Project Automation Feature to workaround limitations with certain system API’s such as Dynamics CRM.

For example you can’t directly update the CreatedOn, ModifiedOn fields in Microsoft CRM via the Web Service API. However if you ignore the warnings about modifying the Database directly you can implement a little function at the end of the Sync to update the Database directly.

This example uses a slightly modified version of the Massive SQL Library by Rob Conery to simplify the SQL required and make it really simple.

Essentially at the end of the Synchronisation we run through the change set and update the fields in the Database Directly.

In this example the CreatedOn and ModifiedOn fields are part of the Sync operation as Updating them doesn’t error it simply ignores their values this way they are part of the Changed row data.

    public override void End(ProjectAutomationResult result)
    {
        if ( result.HasChanges && result.Success ) 
        {
            string connection = "Data Source=.;Initial Catalog=SimegoLtd_MSCRM;Integrated Security=True;";    
            
            var model = new Simego.DataSync.Helpers.Massive.DynamicModel(connection, "AccountBase", "AccountId", "System.Data.SqlClient");
            
            foreach(var update in result.CompareResult.Updated) 
            {
                model.Update(
                    new { 
                        CreatedOn = (DateTime)update.SourceRow[1].BeforeColumnValue, 
                        ModifiedOn = (DateTime)update.SourceRow[2].BeforeColumnValue 
                    }, (Guid)update.SourceRow[0].BeforeColumnValue);        
            }
            
        }
    }

This trick could be used with many different systems to workaround limitations in the API. For example the Same trick could be used with SharePoint to update the Database directly this way too.

Dynamics CRM Import Limits

david@simego.com (David Evans) — Wed, 08 Aug 2012 12:02:34 +0100

As ‘Customer Relationship Solutions’ become more and more utilised in industry, IT departments are going to be migrating/importing more and more data. Dynamics CRM is an excellent CRM solution and as such will continue to be the focus of many import/export scenarios.

The Dynamics CRM Import Wizard has several limits due to it ‘uploading’ the source data and fetchsql results.

Importing that flat data will be easy peasy, I promise! (this includes the reason and fix!)
CRM 2011 Data Import Wizard

I though I would just post a few screenshots to illustrate how easy it is to import and export data to and from Dynamics CRM 2011 (and 4.0) with Data Synchronisation Studio.

What is more important is that we can change the source re-sync, ie make a mistake, fix and re-sync:

What may be more interesting is the speed:

For a 35 Second video of developing the import from SQL to CRM:

Watch as we sync from start to finish in 35 seconds!

CRM Change Trigger-Ouvvi : Data Sync Team Server

info@simego.com (Sean Cleaver) — Tue, 07 Aug 2012 10:29:29 +0100

We have created a CRM Entity change tracking trigger for Data Sync Team Server that operates in a similar way to the SharePoint List Change Trigger.

Internally the trigger system calls each trigger at a 30 second interval to evaluate it’s state, the CRM trigger will then call a Web Service in CRM to see when the last item was created or updated. This is a quick call that only returns a single data item the DateTime of the last change. If this has changed since the last time time a change was detected then any projects associated with this trigger will be started.

Rather than synchronising on a timed schedule you can now achieve almost real-time updates (max 30 second delay) by implementing the Entity Change Trigger with your CRM environment.

This feature is currently only supported with OnPremise AD Authenticated CRM 4.0/2011 installations.

The feature requires no change to your CRM environment and can therefore be easily deployed.

To configure the Trigger add a new Trigger of type ‘Crm Entity Trigger’

Then enter the URL, Organisation Name and Entity Name you want to monitor you may also need to enter some credentials if Ouvvi is not running under an account that has access to your CRM environment.

The only other configuration item required is to ensure that your projects are configured for ’Single Instance’ so that multiple synchronisation operations do not occur in parallel.

That’s all it takes you can now simply associate this Trigger with any projects you want to run when a change is detected.

This new feature is in release 2.0.226 http://www.simego.com/Install/Ouvvi-Beta

HTTP Change Trigger-Ouvvi : Data Sync Team Server

info@simego.com (Sean Cleaver) — Tue, 07 Aug 2012 10:07:04 +0100

The latest Ouvvi Data Sync Team Server update includes a new Trigger for integration with custom solutions. This trigger calls an external HTTP service to evaluate whether the Trigger should be fired.

The HTTP service should simply return the Date Time value of the last change in the format yyyy-MM-ddThh:mm:ss. Ouvvi will then use this value to determine whether the Trigger should fire. As this is a triggering system your service should return very quickly so that it does not tie up the internal Ouvvi trigger list.

To configure this trigger simply add the ‘Http Trigger’

Configure the URL and HTTP Method ‘GET’ or ‘POST’ if your service is authenticated you can enter Windows Credentials here as well.

Now simply associate this trigger with any projects you want to start when the Trigger fires.

Updating Custom Geckoboard Widget

info@simego.com (Sean Cleaver) — Fri, 27 Jul 2012 10:57:24 +0100

This is a quick guide that demonstrates how to update a custom Widget on Geckoboard with the result of a Data Synchronisation using the Project Automation Feature.

Note: Requires Data Synchronisation Studio Release 3.0.570 or greater.

First you need an account at Geckoboard you can signup for a free trial to test it out.

Once you have your first Dashboard add a Custom Widget and configure it for Push, here I selected the Text widget that were just going to push a Single Text value to.

You need to make a note of the Push URL and get your API key form your Account that we need in Data Sync Later.

Now that you have the widget on your dashboard we need to send it some data.

This example uses a demo Data Sync project you can download but you can use any Data Source you like such as CRM, SharePoint, SQL etc.

First we setup our Data Source and preview the Data from Data Source A.

This is a single item that were going to send to Geckoboard, next Export this result to an XML file from the Toolbar so we have a Datasource that we can connect Datasource B to and handle change events.

Connect Datasource B to the .NET XML Dataset provider by browsing to the XML file we just created.

You now have a simple Data Sync project set up that Updates the XML file when there are changes to the Source Data.

Now for the clever bit, switch to the Project Automation Tab and enable Project Automation so that we can send the results to Geckoboard.

We need to write the code that generates the Json payload to send to Geckoboard this is quite simple with .NET Anonymous types.

Within the End method of the Project Automation which occurs after the synchronisation to the XML file and therefore only updates Geckoboard when changes occur add the following code. [Update widgetUrl and apiKey with your values].

    public override void End(ProjectAutomationResult result)
    {
        if ( result.Success && result.HasChanges ) 
        {
            string widgetUrl = "https://push.geckoboard.com/v1/send/aaabbbccc";
            string apiKey = "aaabbbcccc";
            
            var data = new 
            { 
                    api_key = apiKey, 
                    data = new 
                    {                         
                        item = new []  
                        { 
                            new 
                            { 
                                text = result.CompareResult.Updated[0].SourceRow[1].BeforeColumnValue, 
                                value = result.CompareResult.Updated[0].SourceRow[0].BeforeColumnValue 
                            } 
                        } 
                    }
            };
                    
            var client = new System.Net.WebClient();
            client.UploadString(widgetUrl, Json.Encode(data));            
        }
    }

The CompareResult object contains the results of the Comparison and in this case were only interested in Updates. the Updated object is an Array of Updated Rows and the SourceRow object links back to the Datasource A row. The index of the Columns match the Schema Map layout.

And then after the Sync you will see your Geckoboard updated.

Import File Share or Folder into SharePoint 2010 Document Library

info@simego.com (Sean Cleaver) — Wed, 25 Jul 2012 14:15:00 +0100

To setup a File/Folder/Share Synchronisation to a SharePoint 2010 document library is quick and easy with Data Synchronisation Studio.

Connect Data source A to your Source Folder and ensure you set the property “WebFriendlyPaths” to True.

Connect your Target Data source B to SharePoint using the SharePoint Client API provider this is for SharePoint 2010 only.

Map your Source to your Target like this

And that’s it Compare A->B and then Synchronise to copy the files from your Folder/Share into your SharePoint Document Library. Even the Created and Modified fields will match the File timestamps.

If you want to update Metadata columns at the same time you can by either using Dynamic columns to return values for the Metadata Columns or use a Lookup function against another Data Source to provide the values for your metadata.

Transform Any Data with Data Sync

info@simego.com (Sean Cleaver) — Tue, 17 Jul 2012 17:55:48 +0100

We have a database with order information where we just have 2 columns ID and Data where the Data column holds a Blob of strings like this below. I wanted to extract these strings into separate columns so that they could be stored in the database.

FIRST_NAME:Sean
LAST_NAME:Cleaver
COMPANY:Simego Ltd
EMAIL:sean@simego.com
PHONE:
FAX:
COUNTRY:United Kingdom
STATE:East Susses
CITY:St Leonards-on-Sea
ZIP:TN38 9NP
ADDRESS:34 Stirling Road
LICENSE_CODE:5E366AAFBB
EXPIRATION_DATE:2012-06-23
DATE_UPDATED:
CHANGED_BY:VENDOR
LICENSE_TYPE:REGULAR
DISABLED:0
RECURRING:0a
LICENSE_PRODUCT:4559088
START_DATE:2012-05-23
LICENSE_LIFETIME:0
PARTNER_CODE:
PSKU:
ACTIVATION_CODE:DCDA2M-MEX6DR-Q
EXPIRED:0
HASH:7b9dfb715ae1a703e12603aa

With Data Sync it took about 5 minutes to write a few Dynamic Columns to transform this into the right shape to import virtually anywhere. (CRM, Sharepoint, SQL etc)

For each row we basically read each line creating a Dictionary of the Values and then update the Dynamic Properties with the values from the Dictionary. The main task here was Cut & Paste

    public string FIRST_NAME { get; set; }
    public string LAST_NAME { get; set; }
    public string COMPANY { get; set; }
    public string EMAIL { get; set; }
    public string PHONE { get; set; }
    public string FAX { get; set; }
    public string COUNTRY { get; set; }
    public string STATE { get; set; }
    public string CITY { get; set; }
    public string ZIP { get; set; }
    public string ADDRESS { get; set; }
    public string LICENSE_CODE { get; set; }
    public string EXPIRATION_DATE { get; set; }
    public string DATE_UPDATED { get; set; }
    public string CHANGED_BY { get; set; }
    public string LICENSE_TYPE { get; set; }
    public string DISABLED { get; set; }
    public string RECURRING { get; set; }
    public string LICENSE_PRODUCT { get; set; }
    public string START_DATE { get; set; }
    public string LICENSE_LIFETIME { get; set; }
    public string PARTNER_CODE { get; set; }
    public string PSKU { get; set; }
    public string ACTIVATION_CODE { get; set; }
    public string EXPIRED { get; set; }
    public string HASH { get; set; }
    
    public override bool BeginRow()
    {        
        Dictionary<string, string> items = new Dictionary<string, string>();
        
        using(var sr = new StringReader(Data)) 
        {
            string line = string.Empty;
            while((line = sr.ReadLine()) != null)
            {
                string [] split = line.Split(':');
                if ( split.Length == 2 )
                {
                    items.Add(split[0], split[1]);    
                }                
            }
        }
        
        FIRST_NAME = items["FIRST_NAME"];
        LAST_NAME = items["LAST_NAME"];
        COMPANY = items["COMPANY"];
        EMAIL = items["EMAIL"];
        PHONE = items["PHONE"];
        FAX = items["FAX"];
        COUNTRY = items["COUNTRY"];
        STATE = items["STATE"];
        CITY = items["CITY"];
        ZIP = items["ZIP"];
        ADDRESS = items["ADDRESS"];
        LICENSE_CODE = items["LICENSE_CODE"];
        EXPIRATION_DATE = items["EXPIRATION_DATE"];
        DATE_UPDATED = items["DATE_UPDATED"];
        CHANGED_BY = items["CHANGED_BY"];
        LICENSE_TYPE = items["LICENSE_TYPE"];
        DISABLED = items["DISABLED"];
        RECURRING = items["RECURRING"];
        LICENSE_PRODUCT = items["LICENSE_PRODUCT"];
        START_DATE = items["START_DATE"];
        LICENSE_LIFETIME = items["LICENSE_LIFETIME"];
        PARTNER_CODE = items["PARTNER_CODE"];
        PSKU = items["PSKU"];
        ACTIVATION_CODE = items["ACTIVATION_CODE"];
        EXPIRED = items["EXPIRED"];
        HASH = items["HASH"];
        
        return true; // return false to remove row from results.
    }

And the results in Data Sync Data Preview

Duplicate Key Values Behaviour Change

info@simego.com (Sean Cleaver) — Mon, 16 Jul 2012 12:47:40 +0100

Data Sync Release: 3.0.568

We have an update to the way Data Sync handles Duplicate Key values, previously during the load when Data Sync hit a duplicate key value it caused an internal exception and we displayed a list of keys showing you the exception. You would then need to either adjust the Key column in the schema map or fix the data to ensure all rows were unique.

This would keep happening for every duplicate key in your data source and it could be quite time consuming correcting the data to ensure that the key values are unique.

We have now changed this behaviour so that duplicate keys no longer cause the application to stop but rather the duplicate key rows are excluded from the comparison data set and flagged as a duplicate.

When you preview the Data from the Schema Map PreviewA/B functions you will see a error icon next to the duplicate rows and a message at the bottom indicating how many duplicates there are in the result set. This provides a quick visual indication to whether your Key selection is correct.

likewise the comparison results will show that some rows were excluded but it will not prevent the comparison from running.

It is possible to force an error if there are duplicates via the Project Automation feature. You can simply test for Duplicates before the Compare and throw an exception to stop the process like this below.

When you run the compare with this check Data Sync will simply show the error or log the error when running from the Run Tool.

Partner Program

info@simego.com (Sean Cleaver) — Fri, 13 Jul 2012 13:14:33 +0100

Are you an Application Integrator deploying solutions with Microsoft Dynamics or SharePoint?

We're very pleased to announce the new Simego’s Partner Programme. This allows qualified partners to earn money by packaging their integration consultancy with Simego’s products?

To learn more and find out if you are already qualified contact us today.

Data Synchronisation Studio-Fails to Start

info@simego.com (Sean Cleaver) — Tue, 10 Jul 2012 14:27:36 +0100

If your copy of Data Synchronisation Studio fails to open the main window when you run the program it could be that the window location file is corrupt.

The easiest thing to do is to remove your local settings for Data Sync and let it start from a clean start.

Delete the following directory replacing ‘Sean’ for your profile name.

C:\Users\Sean\AppData\Local\Simego\DataSyncStudio30

Now when Data Sync starts it will start with default settings.

Note: This directory maybe hidden on your system so you might need to show hidden files.

Yammer Integration with Sharepoint

david@simego.com (David Evans) — Tue, 26 Jun 2012 09:09:11 +0100

Yammer stands to make an even bigger splash in the Enterprise sector with Microsoft leading it. At Simego we have always developed solutions that allowed IT staff to integrate their systems with ease and Yammer will be no exception.

All scenarios DO NOT NEED Yammer Admin permissions

Scenario: Send an update Notification to Yammer when a Status Changes in a Sharepoint list

Many IT departments need to notify their users when there are issues with email, file systems, Line-of-Business apps etc. This scenario shows how to connect to Yammer, connect to a Sharepoint ‘Status Board’ and respond to changes in the list by sending a notification to Yammer.

Prerequisites:

A Yammer account for your service to talk through : systems@yourcomany.com
Data Sync Studio Download a 7 day full Version trial
Sharepoint 2007 & 2010 (Online or On-Premise)
MS SQL Express (or any to compare the list for changes).

Step 1: Create a new List in Sharepoint similar to the one below

This is essentially a ‘Status Board’ for the IT Department. Each ‘Service’ can have the following states:

Operating with no issues
The service is back up for most, but some users may be experiencing issues
We are experiencing a serious problem with this service
A small percentage of users are experiencing issues

We want these Status updates to appear in Yammer

Step 2: Open Data Sync and connect to the list created in Step 1

Step 3: Sync the SharePoint list with a SQL List (Data Sync can create the SQL Table for you)

Data Sync has a feature that left you create a new SQL Table based on the contents of your source. We are going to use that as a target. Create a SQL Table using the wizard from the Tools Menu.

You can now compare the Sharepoint List to the SQL Table and Synchronise by pressing the :

and then

…this will synchronise all the items in the Sharepoint List into the SQL Table.

So, if you change an item in the Sharepoint list and then Compare and Synchronise a single update will be make to the SQL Table.

It would be so useful to send that change notification to Yammer!!

Luckily for us Data Sync is unbelievable flexible, and uses the .net Framework with a built in development environment to hook into synchronisation events. (Sounds hard but it is automagic).

Step 4: Enable ‘Project Automation’

On the ‘Project Automation’ Tab click on the ‘Enable Project Automation’ link.

This is the bit that makes or breaks any web based solution like Yammer….Integration.

Step 5: Set up an Application in Yammer.

Waltz on over to Microsoft Yammer’s application creation page:

http://www.yammer.com/client_applications/new

and complete the following details obviously replacing your fake name for Simego’s!

and you should get a set of OAuth Keys and Secrets:

Data Sync has a built in solution for authenticating with Yammer through their standard protocols but it is never going to be plain sailing!

Step 6: Do a Song and Dance to get OAuth working

If Yammer approve ‘Data Sync’ as a Global Application you will not need to do such a Song and Dance to get all these keys (Hint! Hint!)

We are going to use Data Sync’s development environment to do a neat trick for us by doing the OAuth dance and handing out the secure tickets so we can post to Yammer!

Back to Data Sync and to the Project Automation editor window. We are now going to use the Consumer Key and Consumer Secret to get a Token Key and Token Secret using DataSync’s YammerHelper:

private Simego.DataSync.Providers.OAuth.YammerHelper yammer;
    public override void Start()
    {
        //Configure the helper
        yammer = new Simego.DataSync.Providers.OAuth.YammerHelper(
            "OHdHeOvWcpODC69Q58ecg", //Consumer Key
            "GbXNAYk8YwTTNDal4ZHwo6Pviph0nzJ3hPkhA4v5w"//Consumer Secret
            );

        //Get the yammer helper to negotiate with Yammer for our OAuth Keys
        Trace.WriteLine(yammer.Authorise());
    }

Click the ‘Compare A-B’ to start the OAuth negotiation.

Your web browser will launch and as you to authorise the application (Pro-Tip : If you are not signed in to Yammer then you have to sign-in on this screen)

Click the ‘Authorize’ button to progress to the next stage. You should be presented with a code number :

Data Sync should have shown you a little dialog to enter this code into:

The magic happens in the background and back on the Data Sync Project Automation page in the bottom of the window Data Sync will be displaying the TokenKey and TokenSecret:

Next we just need to copy those keys next to the Consumer Keys and Secrets and remove the yammer.Authorise() because we have completed the Dance!

We can even send a Test Message using the yammer.SendMessage() method!

In Yammer we will see the message:

Step 7: Publish Changes when the Status Changes

All we need to do is add a little code to the ‘End’ method:

public override void End(ProjectAutomationResult result)
    {
        if ( result.Success && result.HasChanges ) {

            foreach(var item in result.CompareResult.Updated) {
                yammer.SendMessage(string.Format("{0}, {1}",
                    item.SourceRow[3].BeforeColumnValue,
                    item.SourceRow[2].BeforeColumnValue));
            }
        }
    }

Note: The SourceRow columns are the columns in the Data Sync Sharepoint results referenced by number starting with 0.

And we are Done!!

Step 8: Test

To test just update the status of one of your Systems in the Sharepoint List and the changes with appear in Yammer:

Data Sync Project as a Data Source

info@simego.com (Sean Cleaver) — Wed, 20 Jun 2012 14:34:36 +0100

Imagine you have a reasonably complex Data Source configured that uses Dynamic Columns and possibly Lookups to define your Source Data Transformations. Currently if you wanted to use this again in another Sync job then you would have to copy it all again which isn’t ideal.

But now …with the new Data Sync Project File provider you can reference an external Data Sync Project as a Data Source.

So this Source appears just like a normal Data Source where you can then create new Dynamic Columns or Lookups against the already Dynamic Source.

You can see from this that you could easily define a set of Data Sources in Data Sync Project Files and then simply attach them to new projects.

This new feature is in Data Sync Release 3.0.560

Importing Currency Rates from Avangate

info@simego.com (Sean Cleaver) — Wed, 13 Jun 2012 14:16:00 +0100

If you use Avangate as your payment processing partner you may need to download their current currency rates so that you can present the same rate conversion to your customers on your site.

Avangate implement a simple XML based API where the following URL returns an XML Document containing the rates.

https://secure.avangate.com/content/exchange-xml.php?CURRENCY=USD

With Data Synchronsiation Studio you can implement a Synchronisation Import job to import this in about 2 minutes.

Simply connect with the XML Provider and Specify the URL and Item XML XPath of //currency

Add 2 Columns csymbol and crate

And that is it preview the Data and setup the Synchronisation job to your Database.

Using Automation to Send Email

info@simego.com (Sean Cleaver) — Wed, 13 Jun 2012 11:45:10 +0100

With the new Automation feature of Data Synchronisation Studio it is now incredibly easy to write custom actions in the pipeline. For example suppose you wanted an email sent whenever a synchronisation has run and you want to know the changed counts.

You simply implement the End method of the Automation Class and send an SMTP Email message like this.

    public override void End(ProjectAutomationResult result)
    {
        if ( result.Success && result.HasChanges ) 
        {            
            var smtpClient = new System.Net.Mail.SmtpClient("smtp.gmail.com", 587);
            
            smtpClient.EnableSsl = true;
            smtpClient.Credentials = new System.Net.NetworkCredential("username", "password");
            
            //Message to Send
            string message = string.Format("New {0}, Updated {1}, Deleted {2}", 
                result.CompareResult.New.Count, 
                result.CompareResult.Updated.Count, 
                result.CompareResult.Deleted.Count);
        
            //Send Message
            smtpClient.Send("from@simego.com", "to@simego.com", "Synchronisation Result", message);                                                    
        }
    }

Data Synchronisation Studio : New Features

info@simego.com (Sean Cleaver) — Fri, 08 Jun 2012 11:34:30 +0100

With the release of Data Sync 3.0.550 we have a few new features to tell you about.

Integrated Screen Sharing

Form the Help Menu you can now initiate a Screen Sharing session with us to help you with issues that you may have. Please contact support first if you have a problem and we will see how we can help.

Project Automation

With Data Sync you can now hook the internal process to handle certain events as they happen. This allows you to call another Data Sync project of example or report the actual individual data changes.

    public override void Start()
    {
        
    }

    public override void BeforeCompare(ProjectAutomationLoadResult loadResult)
    {

    }

    public override void BeforeSynchronisation(ProjectAutomationCompareResult compareResult)
    {
            
    }

    public override void End(ProjectAutomationResult result)
    {
        
    }

    public override void Error(ProjectAutomationException e)
    {
        
    }

Custom Assemblies

If you want to use external assemblies that are not part of Data Sync with Dynamic Columns you can now tell Data Sync to load these Assemblies from the Project Properties menu.

Start a Synchronisation from Code

You can now easily start a Synchronisation Operation directly from your code like this.

var op = new SynchronisationOperation();                
op.Execute(@"C:\Temp\CSV.dsprj");

If your sync is 2-way then you need a little more

var op = new SynchronisationOperation();                
op.Key = "AAA";
op.SolutionName = "CSV";
op.SolutionPath = @"C:\Temp");

op.Execute(@"C:\Temp\CSV.dsprj");

Then if you want to control logging you can provide your own implementation for IDataSourceLog

Load a Data Source from an existing Project

In dynamic columns you may need to load data from a remote source to build some kind of Lookup or other Transformation. Rather than accessing the underlying objects you can now easily load the data source from a pre-configured Data Sync Project file.

var store = DataTableStore.LoadFrom(@"C:\Temp\CSV.dsprj");
        
foreach(var row in store.Rows) 
{
    var id = (int)row["ID"];    
}

You can also create a Dictionary from this data source easily like this

var dict = store.ToDictionary<int>("ID");

Ouvvi

Ouvvi release 2.0.220 has also been updated to support these new Data Sync features.

Updating Active Directory Manager Attribute

info@simego.com (Sean Cleaver) — Thu, 10 May 2012 09:49:44 +0100

With Data Synchronisation Studio you can now update the Manager attribute with the AD Provider if you modify the Provider configuration slightly.

By default we parse out the Manager field so that it’s more suitable for input to SharePoint or something. However you can now switch off this parsing to get to the raw AD attribute value.

To update the manager you need to supply the full Distinguished name value of the AD Object you want to set which would be something like this below.

CN=Sean Cleaver,CN=Users,DC=xxx,DC=xxx,DC=com

Disable the TryParseValue and ReadOnly properties of the Manager field in the Property Collection on the AD Provider.

Here I created a Dynamic Column to return the Distinguished Name of the Manager to set against the AD Users.

Now we just create a simple Schema Map mapping the X_Manager to Manager and using DSID as the Key column.

Now you just need to compare the data and synchronise the changes.

You can download and use a fully featured version of Data Sync Studio for 15 Days to see how this AD solution works.

Click here to go to the Downloads

Create an RSS Feed

info@simego.com (Sean Cleaver) — Wed, 09 May 2012 16:10:03 +0100

This post is an example of how you can use the Data Sync RSS Provider to create an Exchange Rate RSS feed. You can use a similar approach to create an RSS feed from virtually any kind of data.

First we connect Data Source A to the FX Rate provider.

We want a couple of additional columns for our RSS File so we create them using Dynamic Columns.

We then connect the Target Data source B to the RSS File provider, enter a name for the File and click OK.

We then create the Map between the FX Rates and the RSS File like this.

Schema Map

Once you sync up the Data you will have a nice new RSS File that contains the current currency exchange rates from Yahoo.

Dynamics CRM 2011 Import Contacts and Accounts

info@simego.com (Sean Cleaver) — Wed, 09 May 2012 15:39:05 +0100

This (no code) post shows how to use Data Sync Studio and Lookup Columns to import Accounts and Contacts from SQL into Dynamics CRM and keep the relationship between Account and Contact.

Our SQL Export Database consists of 2 tables “Account” and “Contact” these 2 tables are linked via a Foreign Key Contact.AccountID –> Account.ID.

Account SQL Table

Contact SQL Table

Dynamics CRM doesn’t work with Integer keys it uses Guids for the Primary Key. In many cases it may be impossible to change the source database to update each record with a Guids. We are using Data Synchronisation Studio which can manage the GUIDs without making any changes to your SQL Database or Dynamics CRM.

Step 1 - Import your Accounts from SQL

This is an easy Data Sync project where you simply import your accounts table into the Accounts entity mapping name to name and id to accountnumber.

Project in Data Synchronisation Studio

Schema Map

We now just sync the changes into the CRM Account entity.

Step 2 – Import your Contacts

This step requires mapping the Contact table to the Contact entity in CRM mapping the ID value from the SQL Table to the ExternalUserIdentifier column in CRM.

Project

SQL Schema Map

We need one more trick to automatically associate the Contact with the imported Account record in Dynamics CRM. For this we define a Lookup between AccountID and the accountnumber in CRM.

Select the AccountID column in the Data source window and click the “Add Lookup” button.

Next configure the Lookup Data source to connect back to the Dynamics CRM account entity.

Next define the Lookup relationship by selecting accountnumber in the Target Column field.

Now you can use all the columns from the related Account CRM entity in your project.

We now need to map Lookup1_accountid to parentcustomerid to create the relationship in Dynamics CRM.

and that's it you can sync up the data and see that your contacts are correctly related to your accounts.

You can download and use a fully featured version of Data Sync Studio for 15 Days to see how this SQL and CRM solution works.

Click here to go to the Downloads

Top Ten System Migration Traps

info@simego.com (Sean Cleaver) — Mon, 23 Apr 2012 14:31:26 +0100

Considering the impact on delivery dates, testing, training, regulatory compliance and budget, it is always amazing how little focus there is on migration. There are many many traps to fall into when implementing a new system but migration is one stick that the underlying business will beat you with.

1. Your project plan has one task that says ‘Migrate Data’.

This is very easily done as either the team developing the new system or the vendor, can’t estimate the effort so it is just left as a single item. Don’t do it. Do some analysis that will at least give you a consideration of risk, effort and cost. Without these, managing the expectation of the business will be impossible.

2. Not using an automation tool to migrate data.

In the past I have conducted migrations based on SQL scripts, batch files, custom code etc. Quickly you realise that there are real benefits to automation. For example the new system has 5 new fields in a table. Rather than going back and redeveloping the migration code, you just point the tool at it and click update. 1 hour saved and no errors introduced, commit it to source control.

3. Migrating old data that might better be placed in a reporting system.

You should always consider that some data might be better placed in a reporting or document management solution. If you have a 'end of year' balance brought forward into a new system, do you need all of last years transactions or can you just lookup a report showing them?

4. You let the vendor leave out migration or worse make it your entire problem.

Migrating data is risky and can be expensive; your development team knows how their current system works better than anyone. Make sure you have some migration commitment from the vendor before agreeing the deliverables and costs.

5. Assume that the migration with be straightforward because the old system and the new system do the same thing.

The architectural requirements of systems mean that they may have completely differing internal structures and security models. These factors influence the effort of migration by factors of 10. Even simple upgrades between versions can be the same as moving to a different vendor’s.

6. Underestimate the effect of the new security model on migration.

Security models are there to make sure that data is not viewed or manipulated by an unauthorised individual. During migrations it is often necessary to circumvent or assume the identity to extract and insert data. The developers of the current systems could make it extremely difficult for you to move data by using encryption, logic in the user screens and even more esoteric (smart at the time) tricks.

7. Thinking you can sort out the data quality once it is in the new system.

Often a driving reason the move to a new system is that it has better data quality control. For example, we often see telephone numbers saved as ‘TBC’ in an old system and the new system will only allow number to be saved. A significant phase in migration is cleaning the underlying system so that clean data can be stored easily into the new system.

8. Underestimating the involvement from the business to clean the data.

As with 7, it is important to realise that users will have to assist in cleaning data. You can make this easier by creating lists of broken records and request rules for your automation tools to follow.

9. Not realizing the impact of month/quarter/year ends on the migration.

Many accounting systems require journaling and retrospective adjustment. Additionally, when picking a date for migration, make sure the business is available and not buried in month end, quarter end, tax year-end or year end.

10. Not documenting the migration process so no one knows where the data came from!

When migrating data it is often necessary to transform the data several times before it finally resides in the new system. Often a client will ask, why is the address used and not the billing address from the old system? You need to have a method of tracing the source to target and often you automation tools can help to generate documentation.

At Simego we have been involved in assisting clients who have undertaken system migrations.

Data Synchronisation Studio 2012

info@simego.com (Sean Cleaver) — Mon, 02 Apr 2012 10:15:01 +0100

The latest version of Data Synchronisation Studio has gone live today, this release has had major re-work to allow loading of much larger data sets and to significantly reduce the memory usage of the application. Our reliance on the .NET DataTable object is now gone and we created a new object to store the data which uses fast disk based storage instead. This also allowed us to to improve Dynamic Columns so that DBNull.Value is now gone and is just NULL and you can now use Nullable<T> in you Dynamic Column properties.

For example loading an comparing Data Sets of Millions of rows so now possible. For example below is a screenshot of Data Sync loading and displaying over 18 Million Rows from a CSV file.

We also now have a new Social Panel on the Start Page so that you can see our latest Blogs and Tweets right from within the application.

This release includes our new Data Generator Provider which with Dynamic Columns allows for the generation of Sample Test Data or for you to connect to other data sources we haven’t created a provider for. With the Test Data Generator you can further transform your data such as creating Pivots etc.

Using an RSS Feed as a Data Source

info@simego.com (Sean Cleaver) — Thu, 29 Mar 2012 11:06:11 +0100

If you have an RSS Feed you want to use as a Data source with Data Synchronisation Studio it’s really easy to use the XML File provider to read this feed and present it as a Data Source.

To configure the Data source you need to know the structure of the XML and a little XPath to define the Columns you want to return.

First enter the URL to your Feed for example http://www.simego.com/Blog/RSS

This is a standard RSS feed so each item/row is within an item element in the Xml therefore we set the XmlItemElementXPath to be //item

Next we need to define the columns to return, as the RSS is quite simple we just create columns for Link, Author, Title, Description, Source and PublishDate that map directly to the Xml element name in the XPathExpression field.

And that’s it now you have an RSS feed as a Data source that you can use to synchronise into some other system.

Download Example Data Sync Project

Import Csv File

info@simego.com (Sean Cleaver) — Tue, 06 Mar 2012 14:37:37 +0100

Overview

This article describes how to import a file such as CSV into a new SQL Server Table with Data Synchronisation Studio.

Data Synchronisation Studio can also import other files :-

CSV
TAB
XLS, XSLX (Microsoft Excel Files)
MDB (Microsoft Access Database Files)
SDF (SQL Server Compact)
XML
+ many more

Step 1 - Open the CSV Data File

Choose the Connect Data Source on Data Source A in Data Synchronisation Studio to open the connection dialog. Locate the CSV File provider and select it. Then enter the path to the CSV file in the Filename property.

Once Data Sync loads the CSV File to discover the columns in your file you will be presented with the schema and a default schema map containing your columns.

You can now preview the data to check that it loads ok.

Now we need to adjust the schema to include the columns we want and setup the data types. We adjust the map like this below where we define the ID column as a Key and set the Data Type as an Integer.

Step 2 - Create SQL Database Table

We now use the Create SQL Table Wizard from the Tools menu to create a SQL Table to import the data. You can also connect to an existing SQL Table or SharePoint List etc.

Step 3 - Import the Data

Now when the Table is created the Schema Map is mapped automatically ready to compare and load the data.

Selecting the Compare A->B toolbar option compares the Data and creates a Change Set to apply to the Target. Simply click Synchronise and the data will be copied into the SQL Table. You can now save the project and quickly and easily run it again at any time.

You can download and use a fully featured version of Data Sync Studio for 15 Days to see how this SQL CSV Import solution works.

Click here to go to the Downloads

Error Starting SQL Admin

info@simego.com (Sean Cleaver) — Tue, 06 Mar 2012 14:17:18 +0100

This problem can occur when you "Refresh Server" and the process does not complete properly or you end task the application during some save operation of the core configuration files.

The application stores a cache of your database list and the server list in XML documents under your profile

On Windows XP

C:\Documents and Settings\<ProfileName>\Local Settings\Application Data\Simego\SQLToolsExplorer

On Windows Vista/7/2008

C:\Users\<ProfileName>\AppData\Local\Simego\SQLToolsExplorer

The quickest way to resolve the issue is to delete all the files in this folder however you can resolve it manually by checking that the XML Documents are not corrupt. Generally you will find that the file is missing an end tag.

Filename	Purpose
settings.xml	Records your server list with connection details
{2BF0F4EA-1510-4143-82FB-968692943ECE}.xml	Application Settings
{7FD68FA9-E96C-4607-A2EC-F380255CB1D9}.xml	Application Layout Settings
Server Name.xml	Database List Cache for each server connection

You can safely delete any of the files however if you delete the settings.xml file you will need to delete all the server database cache files as well and then re-connect all your servers.

TIP: If you connect to a Server on a Shared Host and your database list is large then you can edit the Database Cache List to only display your database(s).

SharePoint List Data Replication with Ouvvi

info@simego.com (Sean Cleaver) — Fri, 17 Feb 2012 15:10:41 +0100

It’s possible to setup a near real-time bi-directional SharePoint List replicator between different SharePoint Sites/Farms etc with Ouvvi and Data Synchronisation Studio. Additionally there is nothing to install into your SharePoint environment so it can also work against hosted SharePoint or SharePoint which is locked down by your admins.

Ouvvi 2.0 has a new Trigger for SharePoint Lists which fires whenever items in the list change therefore if you configure a bi-directional Data Sync project and load it into Ouvvi and assign the triggers to the project you can build a replication system.

Here we define a new project in Ouvvi and create a single step which is the 2-Way Data Sync project. (You need to know how to build the 2-Way project with DSID replication columns).

We then define 2 triggers to query each list for changes, by default the triggers are executed every 30 seconds so you should expect the project to be kicked off within 30 seconds of a change (this is what I mean by near real-time).

You then assign the 2 triggers to the project so that whenever each one fires the project is started.

To prevent the project running sync jobs in parallel we define the project as Single Instance this way each start request is queued and only one instance is ever running.

You can also see in the logs when the sync runs how many items were changed.

And view real-time reports of the execution process.

Using Data Generator to get Public Holidays

info@simego.com (Sean Cleaver) — Tue, 14 Feb 2012 16:43:03 +0100

You can find several public holiday feeds on the internet such as this one here

http://ical.mac.com/ical/UK32Holidays.ics

Now with the Data Generator Data Provider in Data Synchronisation Studio 2012 and Dynamic Columns it’s easy to turn this into a Data Source.

The first step is to connect Data Source A to the Data Generator.

The Data Generator simply returns a number between 1 and the number you set in the properties.

All the magic happens in the Dynamic Column code where we override the Setup() method and load the Data from the holiday service into a List of Events. Then for each Row Data Sync calls BeginRow() and we simply return whether there are any more events and return the calendar data as properties of the Row.

#region Usings
using System;
using System.Collections.Generic;
using System.Data;
using System.Globalization;
using System.Text;
using System.Linq;
using System.IO;
#endregion

class DataSourceRowOverride : Simego.DataSync.DynamicColumns.DataSourceRowInternal //Do Not Change This Line
{    
    private List<dynamic> Events = new List<dynamic>();
        
    public DateTime StartDate { get { return Events[ID-1].Date; } }
    public DateTime EndDate { get { return StartDate.Add(new TimeSpan(23, 59, 0)); } }
    public string Title { get { return Events[ID-1].Title; } }
    public bool AllDayEvent { get { return true; } }
    
    public override bool BeginRow()
    {        
        return ID <= Events.Count; // return false to remove row from results.
    }

    public override void Setup()
    {
        var request = System.Net.HttpWebRequest.Create("http://ical.mac.com/ical/UK32Holidays.ics");
                
        using(var reader = new StreamReader(request.GetResponse().GetResponseStream()))
        {            
            DateTime date = DateTime.MinValue;
            string title = null;
            string line = null;
            
            while((line = reader.ReadLine()) != null) 
            {                        
                if ( line.StartsWith("DTSTART") )
                    date = DateTime.ParseExact(line.Substring(19), "yyyyMMdd", CultureInfo.CurrentCulture);
                
                if ( line.StartsWith("SUMMARY:") )
                    title = line.Substring(8);
                
                if ( line.StartsWith("END:VEVENT") ) 
                    Events.Add(new { Date = date, Title = title });
            }
            
            //Order the Events by Date
            Events = Events.OrderBy(p => p.Date).ToList();
        }
    }
}

And there you have it, Calendar entries from a Public Calendar feed that you can import right into a SharePoint List.

Download Sample Project File

Test Data Generator Download for Data Sync

info@simego.com (Tremayne Christ) — Tue, 14 Feb 2012 16:20:10 +0100

Need hundreds, thousands, millions, hundreds of millions?! of rows of test data? Well you probably didn’t think to use Data Synchronisation Studio. Why would you, it’s a synchronising tool right?

Well, yes, this is true. However, DS 2012 now includes a new provider that can generate as many rows as you need. Utilising this, we created a test data generator using dynamic columns. It can create you an entire user base in seconds!

You can download and use a fully featured version of Data Sync Studio for 15 Days to see how this solution works and generate some data!

Click here to go to the Downloads

If you have used Data Synchronisation Studio before, you will be familiar with data source sides A and B. You may also be familiar with the term Dynamic Columns.

This test generator works by creating a faux data source on side A using Dynamic Columns. You can then use this to create faux data for Dynamics CRM, SQL, SharePoint etc. to give yourself an instant user base for testing.

Do you have Data Synchronisation Studio 2012 (v3.0.500) or above?

Yes I do, Download Test Data Generator Project
No, I need to get the 15 day trial Download the Latest Version of Data Synchronisation Studio

Features

Generates random name combinations
Generates dates of births
Chooses from five different countries
Chooses a region based on the country
Chooses a city based on the region
Generates random road names
Creates phone numbers based on the country’s number format
Creates user identification codes
Generates email address based on the users names
Creates a password hash
Generates ‘Date Joined’ ‘Previous Login’ & ‘Last Login’
Randomly selects a security question and answer
Creates a security PIN

Columns

ID (Int32)
UID (Guid)
UserCode (String)
FirstName (String)
MiddleInitial (String)
LastName (String)
DateOfBirth (DateTime)
HouseNumberName (String)
AddressLine1 (String)
AddressLine2 (String)
TownCity (String)
Region (String)
PostalCode (String)
Country (String)
PhoneNumber (String)
MobileNumber (String)
WorkNumber (String)
Email (String)
PasswordHash (String)
Role (Int32)
DateJoined (DateTime)
PreviousLogin (DateTIme)
LastLogin (DateTime)
AccountStatus (Int32)
SecurityQuestion (Int32)
SecurityAnswer (String)
SecurityPin (Int32)
NewsletterSubscription (Boolean)
AgreedToTerms (Boolean)
Other (String)

How do I use this generator?

Make sure you have the correct version of Data Synchronisation Studio, as this project will only be supported by version 3.0.500 and above.
Download the project, using the link above
Open the project, making sure you are connected to the internet. This is so the project can download the names database (only happens once)
Set the row count to the amount of rows you would like to create and set the seed to zero for completely random data.
Click the ‘Preview A’ button to check the data (optional)

This is optional, because it can take a few minutes to generate the rows. This depends on the machine spec and the row count chosen. Below is a basic guide to timings.

10,000 = 1.5s
100,000 = 12s
1,000,000 = 2m
Now that you have setup the generator to your liking, we can go about inserting the data into a usable SQL table. You can choose other data sources, but for this post we’ll use SQL.
Go to the tools menu and select create SQL Table.
Follow the instructions of the SQL Table Wizard
Once you’ve created your new SQL Table, Click ‘Compare A->B’
This will compare the generated data with your new table. Obviously your new table is empty, so once the sync has completed you will see a list of items to add to SQL.
Click the ‘Synchronise’ button and then click ‘Start’
This will fill your SQL Table with all of your new generated data. You now have a test database to work with!

Connecting to Firebird Database via ODBC

info@simego.com (Sean Cleaver) — Thu, 19 Jan 2012 12:28:32 +0100

You can use Data Synchronisation Studio with Firebird via the ODBC connection provider and the Firebird ODBC Driver.

Ensure you have the Firebird/InterBase ODBC Driver installed and if you have Data Sync 64 bit edition you also have the 64 bit version of the ODBC Driver.

Select the ODBC Driver and enter the connection string to Firebird like this.

DRIVER=Firebird/InterBase(r) driver;UID=SYSDBA;PWD=masterkey;DBNAME=C:\Temp\EMPLOYEE.FDB

Then select the Object in the database or switch to the SQL Tab and enter your own SQL.

Once the Schema Loads you can start defining your Project.

Or just preview the Data to check the connection.

Data Sync Studio and Lookup Columns

info@simego.com (Sean Cleaver) — Wed, 18 Jan 2012 16:56:50 +0100

Data Synchronisation Studio has a new feature to make it unbelievably easy to link data by way of Lookup Columns. A column can now link to any other data via a special Lookup column, you could do this with Dynamic Columns before but now it’s just click…click…click.

You can connect SQL Data to SharePoint data for example or Join SQL Data across servers like SQL and Oracle or whatever you get the idea anything to anything.

To get started connect your data source as before and select the column that represents the Link or Join

Then click the Icon in the Data source Toolbar to Add a Lookup

Once you configure the connection details you need to confirm the Lookup Join details which is essentially the Data Type and Column.

Now once the Lookup loads you will see all the columns from the related Data source listed and you can add any of them to the Schema Map as if they were already in the source.

That’s it really easy lookup columns in Data Synchronisation Studio, this feature is in release 3.0.444

Set Exchange 2010 Folder Permissions via Powershell

info@simego.com (Sean Cleaver) — Thu, 08 Dec 2011 15:41:45 +0100

For the Data Sync Exchange Providers to read Calendars,Contacts or Tasks from other mailboxes you will need to add Reviewer permission to the mailboxes for the user that is used in Data Sync. Fortunately you can do this with a PowerShell script like this.

Add-MailboxFolderPermission -Identity ayla@contoso.com:\Contacts -User Ed@contoso.com -AccessRights Reviewer

-Identity is mailbox and folder your targeting and –User is the user to grant access.

Synchronise Documents between SharePoint 2007 and 2010

info@simego.com (Sean Cleaver) — Fri, 02 Dec 2011 13:17:12 +0100

This blog article describes how you might synchronise Documents between SharePoint 2007 and SharePoint 2010 with Data Synchronisation Studio including custom Metadata columns whilst preserving the Created and Modified Timestamps and user accounts.

Here we have a Document Library in SharePoint 2007 with a few Documents and a Folder where going to Synchronise this to a new SharePoint 2010 document library.

Here’s the Empty Document Library in SharePoint 2010 which will be the Target

To define the project in Data Synchronisation Studio we need to use the SharePoint provider for SharePoint 2007 and the new SharePoint Client API provider for SharePoint 2010.

Connect Data Source A to the SharePoint Provider

Connect Data Source B to the SharePoint 2010 Client API Provider

Configure the Schema Map like this and add any other meta data columns you also wish to synchronise.

Ensure that the Document Library Update Behaviour is set for FileAndMetaData on each side to that the Documents are uploaded.

Run a Compare A->B and Synchronise the Documents

And now the documents have been synchronised including the Created/Modified Timestamps and User accounts.

Data Synchronisation Studio Connection Dialog

info@simego.com (Sean Cleaver) — Wed, 30 Nov 2011 17:33:29 +0100

As the number of Providers has increased that Data Synchronisation Studio now supports it was getting increasingly harder to locate the provider in the list. Therefore we have now re-worked the Connection Dialog into groups and allowed you to select your own Favourites which are easy to find under the Favourite Tab.

This update is in release 3.0.434

Sync Active Directory Group Members to SharePoint Security Group

info@simego.com (Sean Cleaver) — Tue, 29 Nov 2011 16:20:47 +0100

We have created a new Data Synchronisation Studio Provider that allows you to synchronise the Members in a SharePoint Security Group. You may want to use this to keep the Members in sync with Members in an Active Directory Group.

This new Provider is in release 3.0.432 of Data Synchronisation Studio.

Example

Synchronise Members from AD Group to SharePoint Group

Connect Data Source A to Active Directory

In this example we use a LDAP Filter to return only those users of a specific AD group, for example

(&(objectClass=User)(memberOf=CN=SharePoint Team Users,CN=Users,DC=corp,DC=litware,DC=inc))

Connect Data Source B to the Microsoft SharePoint Security Groups provider

Choose the SharePoint Group to synchronise from the dropdown.

Adjust the schema map like this so that the AD Account Name is mapped to Login name and Display Name is mapped to Name. Account Name should be the Key Column.

Your completed project will look like this.

Running a Compare A to B shows the differences and you can now synchronise the member lists.

And now after synchronisation in SharePoint the Group Members is populated.

Now that the users have been correctly added to SharePoint you can also setup a standard SharePoint List sync with the User Information List to populate other meta data from AD into SharePoint.

You can download and use a fully featured version of Data Sync Studio for 15 Days to see how this AD solution works.

Click here to go to the Downloads

Ouvvi Azure Blob Upload

info@simego.com (Sean Cleaver) — Tue, 15 Nov 2011 11:57:02 +0100

Now with Ouvvi you can upload and download files into Azure Blob Storage, this is especially useful if you need to backup some files off site such as Database Backup images as part of your automated daily system processes.

Ouvvi Reporting Features

info@simego.com (Sean Cleaver) — Fri, 11 Nov 2011 15:53:43 +0100

Here’s a screenshot of the new Reporting Views were creating for Ouvvi. The view shows Tasks running in Ouvvi, how long they took and their status.

Ouvvi Refresh

info@simego.com (Sean Cleaver) — Tue, 08 Nov 2011 10:04:37 +0100

Were working on a refresh of Ouvvi and creating better integration between Data Sync and Ouvvi so that eventually they will become one product together.

The UI has been refreshed with a more modern look and feel.

Opening Data Sync Projects from Ouvvi now integrate with Data Sync and can simply be saved directly back to the Ouvvi Step without having to re-Import.

Changes to Ouvvi Include

View Queue and Queue Failure Tables
Re-Submit a Queue Failure back to the processing Queue
Stop/Abort a Running Instance
Aborting a Running Instance now injects a ThreadAbortException
Queue Peek support to prevent multiple instances of the Same Project from the UI
New REST URL’s to remotely start a Project i.e. http://localhost:4000/Project/Start/10
Simple Step re-ordering
Change the Order of Projects on the Project View
Control the Re-try count per Step
Control the Failure Mode per Step
Enable/Disable Steps
Import/Export Project definitions
Synchronise Running Threads back to main Project
Step Handler for SQL Batch
Support for OleDb, SQLClient and ODBC Data sources in SQL Commands
Custom Step Handlers are loaded in the Database
Updated Log Views
Cross browser support

Were also working on new features as we get ready for Ouvvi 2.0

Were creating report views so you can see statistics about processes running in Ouvvi. Were also working on a solution which will submit log data to the “Ouvvi-Cloud” and allow us to create iPhone/Web Apps so that you can view the status of your running system from outside of your corporate network. (Handy for Remote Support)

Active Directory Computers

info@simego.com (Sean Cleaver) — Tue, 01 Nov 2011 16:41:10 +0100

Using the Active Directory Users provider it is possible to return the computer information stored within your Active Directory.

To do this you need to adjust the LDAPFilter to

(&(objectClass=Computer))

And remove all the auto-generated Properties and add the following list :-

objectguid (DSID)
adspath (AD Path)
name (Name)
operatingSystem (OperatingSystem)
operatingSystemVersion (OperatingSystemVersion)
operatingSystemServicePack (ServicePack)
description (Description)

Once this is done refresh the Schema on your Data Source and Preview the data like this.

Data Sync Ignore Flag Change

info@simego.com (Sean Cleaver) — Tue, 23 Aug 2011 09:42:49 +0100

In the latest build 3.0.414 we have made a change to the behaviour of the "Ignore" flag on the schema map. Previously this flag just excluded the column from the comparison routine and therefore the value was never passed to the update routine.

However now by setting this flag to true the value is not part of the comparison but if any other columns trigger an update then these fields marked "Ignore=True" are also passed to the update routine.

So in this example we have set the columns UnitsInStock, UnitsOnOrder and ReorderLevel to Ignore=True. When we modify the ProductName we get a change detected like this.

Additionally the Ignored columns are flagged as changed even though the values are the same and all these values are passed to the Update routine.

You would typically use this behaviour when you want to only compare certain columns to trigger an update but should there be a change you want to include values in the other columns. For example you may have audit columns UpdatedBy, UpdatedAt, etc but you don't want to trigger the update based on these values but trigger on actual data changes and include these columns when there is an update.

Integrate Data Sync with 3rd Party Systems Part 2

info@simego.com (Sean Cleaver) — Thu, 18 Aug 2011 10:35:05 +0100

See :- Integrate Data Sync with 3rd Party Systems Part 1

For users that do not have Visual Studio 2010 you can download the free Visual Web Developer Express 2010 from Microsoft and develop the Web Service with this version of Visual Studio.

Download http://www.microsoft.com/web/downloads/platform.aspx

Once installed simply copy the web service code from the download into your my documents folder to start working with the web service code.

Then double click on the solution file to open the project in Visual Studio which will then open like this.

The sample project includes a very basic data source called "DefaultDataSource" under the "DataSources" folder you can open this and see how it works.

If you put breakpoints on the GetDefaultSchema() and GetData() methods you can then step into the code as it runs. Press F5 to run the project and IE will start showing the default Web Page for the service.

Now connect Data Sync to the service

When you click OK your first breakpoint GetDefaultSchema() will be hit where Data Sync requests information from your Data Service about the Schema.

Hit F5 to continue and the Data Source is loaded into Data Sync like this.

Now in Data Sync click the Preview A toolbar button on the Schema Map and then the GetData() method will be hit.

Hitting F5 again returns you to Data Sync and the Data is displayed from the service.

As you can see developing a custom web service to connect to your data source is quite simple. We used a simple SOAP WebService rather than WCF to keep it simple and we also kept the code simple rather than use any fancy code.

You can add additional Data Sources to the Service by copying the DefaultDataSource class and adding it in the DataSourceList class. These are then available on the DataSource Dropdown when you connect in Data Sync.

using System;
using System.Collections.Generic;

namespace DataSyncService.DataSources
{
    class DataSourceList
    {
        public static Dictionary<string, Type> GetDataSources()
        {
            Dictionary<string, Type> result = new Dictionary<string, Type>();

            //Add Datasources here....
            result.Add("Default", typeof(DefaultDataSource));

            return result;
        }
    }
}

Integrate Data Sync with 3rd Party Systems

info@simego.com (Sean Cleaver) — Tue, 16 Aug 2011 10:37:11 +0100

We now have an alternative way for you to integrate other systems that we do not have providers for with Data Sync. This could be used with virtually anything and support 2 way synchronisations.

The process involves a Data Sync provider that connects to a Web Service and you would place your code in the web service. We have abstracted all the schema mapping code that you needed to contend with when writing providers directly so you simply detail with connectivity to the system your connecting to. Also as it's a separate Visual Studio project you can easily debug the Web Service end in Visual Studio.

Overview

Web Service

The Web Service exposes a few methods that the Data Sync provider will interact with we are supplying a template Visual Studio Project that you can use to build your project.

GetSchemas - Return a List of DataSources/Schemas from the Service
GetDefaultSchema - Returns Schema Information for a Schema/DataSource
GetData - Returns the Data from the DataSource
AddItem - Add's an Item
UpdateItem - Update's an Item
DeleteItem - Deletes an Item

Really Simple Connector

Below is some sample source code for a really simple read-only connector that is also supplied in the sample project. You can see there are just a few simple methods that you need to implement to get it working.

using System;
using System.Collections.Generic;
using System.Data;
using System.Linq;

namespace DataSyncService.DataSources
{
    public class DefaultDataSource : IDataSource
    {
        private readonly DataTable _dataTable = new DataTable();

        public DefaultDataSource()
        {
            //Setup Schema
            _dataTable.Columns.Add("ID", typeof (int));
            _dataTable.Columns.Add("FirstName", typeof (string));
            _dataTable.Columns.Add("LastName", typeof(string));
            _dataTable.Columns.Add("Email", typeof(string));
            _dataTable.Columns.Add("Phone", typeof(string));

            _dataTable.Columns["ID"].Unique = true;

            //Add Rows
            _dataTable.Rows.Add(new object[] { 1, "Sean", "Cleaver", "sean@simego.com", "555-1234" });
            _dataTable.Rows.Add(new object[] { 2, "Trem", "Christ", "trem@simego.com", "555-1235" });

        }

        public List<SchemaItem> GetDefaultSchema()
        {
            return (from DataColumn column in _dataTable.Columns select new SchemaItem {ColumnName = column.ColumnName, DataType = column.DataType.ToString(), Unique = column.Unique}).ToList();
        }

        public List<object[]> GetData(List<SchemaItem> detail)
        {
            var data = new List<object[]>();

            foreach(DataRow row in _dataTable.Rows)
            {
                var rowData = new object[detail.Count];
                for (var i = 0; i < detail.Count; i++)
                    rowData[i] = row[detail[i].ColumnName];
                data.Add(rowData);
            }
            
            return data;
        }

        public void AddItem(object key, DataItem[] values)
        {

        }

        public void UpdateItem(object key, DataItem[] values)
        {

        }

        public void DeleteItem(object key)
        {

        }
    }
}

Connecting to the Web Service

To Connect to the Web Service you need to install the Data Sync Integration Web Service Connector and then it's a matter of entering the URL to the Service and choosing a Data Source from the Drop down.

Once you connect to the Web service everything just works like any other Data Source in Data Sync. You can change the Data Source, Refresh the Schema etc all from within Data Sync.

Download the necessary connector and sample web service code from here.

Linking Contact to Account in Dynamics CRM 2011 Online

info@simego.com (Sean Cleaver) — Fri, 12 Aug 2011 15:47:03 +0100

If you want to link related records in Dynamics CRM 2011 Online with Data Synchronisation Studio you should create a Dynamic Column on the Source Data to return the ID of the related record.

For example to link Contacts to Account you need to lookup the AccountID from the Account entity to get the internal CRM ID for the Account.

Below is screenshot from CRM showing that my record relates to the Customer "Simego Ltd"

What we need to do is load a List of Account ID's against the Account Number in the Setup method of the Dynamic Column Code like below. The code creates an instance of a CRMDataSourceReader and configures it to return the ID and Account Number from the CRM Server.

//Get a DataTable object of the related List via a new CRM Reader
MSCrmDataSourceReader reader = new MSCrmDataSourceReader();        
reader.Username = ((MSCrmDataSourceReader)this.DataSourceB.Reader).Username;
reader.Password = ((MSCrmDataSourceReader)this.DataSourceB.Reader).Password;
reader.CrmServerUrl = ((MSCrmDataSourceReader)this.DataSourceB.Reader).CrmServerUrl;
reader.Organisation = ((MSCrmDataSourceReader)this.DataSourceB.Reader).Organisation;
reader.Entity = "account";
//Create a Schema Map to return the ID and account number
reader.SchemaMap = new DataSchema();
reader.SchemaMap.Map.Add(new DataSchemaItem("accountid", typeof(Guid), true, false, false, -1));
reader.SchemaMap.Map.Add(new DataSchemaItem("accountnumber", typeof(string), false, false, false, -1));
//Load the Data Table
DataTable accounts = reader.GetDataTable();        
//Create a Lookup Index to help speed-up lookups
foreach(DataRow r in accounts.Rows) 
    accountLookup.Add((string)r["accountnumber"], (Guid)r["accountid"]);

Then we create a Dynamic Column that looks into the created Dictionary for an Account ID against the Account Number we have in the source data.

/// <summary>
/// Return the Account ID via the Lookup Dictionary
/// </summary>
public object accountid
{
    get 
    { 
        if ( accountLookup.ContainsKey(this.OrganisationID.ToString()) )
            return accountLookup[this.OrganisationID.ToString()];
        
        return null;
    } 
}

When we preview this data you see that we get a Guid for the "accountid" field which is what we need to synchronise into the Parent Customer column.

The Schema Map might look something like this where the accountid links to the parentcustomerid and we store the ID of the contact from the source system into the externaluseridentifier field.

Complete Dynamic Column Code

#region Usings
using System;
using System.Collections.Generic;
using System.Data;
using System.Globalization;
using System.Text;
using Simego.DataSync;
using Simego.DataSync.Providers.MSCrm2011;
#endregion

/// <summary>
/// An object that defines the Dynaimc columns for this project.
/// To Create new Dynamic Columns simply expose them as properties in this class.
/// </summary>
class DataSourceRowOverride : Simego.DataSync.DynamicColumns.DataSourceRowInternal //Do Not Change This Line
{
    //Category Index
    Dictionary<string, Guid> accountLookup = new Dictionary<string, Guid>();
    
    public string accountname { 
        get { 
            
            if ( !string.IsNullOrEmpty(Name) )
                return Name;
            
            return EmailAddress; 
        } 
    }
    
    public string firstname { 
        get { 
            
            string [] parts = Name.Split(' ');
            if ( parts.Length > 0 )
                return parts[0].Trim();
            
            return Name; 
        } 
    }
    
    public string lastname { 
        get { 
            
            string [] parts = Name.Split(' ');
            if ( parts.Length > 1 )
                return parts[1].Trim();
            
            return ""; 
        } 
    }
    
    
    /// <summary>
    /// Return the Account ID via the Lookup Dictionary
    /// </summary>
    public object accountid
    {
        get 
        { 
            if ( accountLookup.ContainsKey(this.OrganisationID.ToString()) )
                return accountLookup[this.OrganisationID.ToString()];
            
            return null;
        } 
    }

    public override void Setup()
    {
        //Get a DataTable object of the related List via a new CRM Reader
        MSCrmDataSourceReader reader = new MSCrmDataSourceReader();        
        reader.Username = ((MSCrmDataSourceReader)this.DataSourceB.Reader).Username;
        reader.Password = ((MSCrmDataSourceReader)this.DataSourceB.Reader).Password;
        reader.CrmServerUrl = ((MSCrmDataSourceReader)this.DataSourceB.Reader).CrmServerUrl;
        reader.Organisation = ((MSCrmDataSourceReader)this.DataSourceB.Reader).Organisation;
        reader.Entity = "account";
        //Create a Schema Map to return the ID and account number
        reader.SchemaMap = new DataSchema();
        reader.SchemaMap.Map.Add(new DataSchemaItem("accountid", typeof(Guid), true, false, false, -1));
        reader.SchemaMap.Map.Add(new DataSchemaItem("accountnumber", typeof(string), false, false, false, -1));
        //Load the Data Table
        DataTable accounts = reader.GetDataTable();        
        //Create a Lookup Index to help speed-up lookups
        foreach(DataRow r in accounts.Rows) 
            accountLookup.Add((string)r["accountnumber"], (Guid)r["accountid"]);
    }

}

You can download and use a fully featured version of Data Sync Studio for 15 Days to see how this CRM solution works.

Click here to go to the Downloads

Connecting to Dynamics CRM 2011 Online

info@simego.com (Sean Cleaver) — Fri, 12 Aug 2011 15:03:38 +0100

A few customers have been struggling to connect Data Synchronisation Studio to their Dynamics CRM Instance.

What you need is to ensure that you have the Windows Identity Framework (WIF) components installed on your machine from here http://www.microsoft.com/download/en/details.aspx?id=17331

Then it is just a matter of entering your Live Credentials and the base URL to your hosted CRM environment. This will be something like https://crm4.dynamics.com or https://crm5.dynamics.com do not include your organisation name in the URL. You should then select an organisation and Data Sync will call the CRM Discovery Service to locate a list of Organisations that are available for your login. Finally choose the entity from the Entity drop down, you can choose built-in Entities or Custom entities from this list.

Once your connected you will see all the columns that are available from your selected Entity in Data Sync.

SharePoint List and SQL Table 2 Way Synchronisation

info@simego.com (Sean Cleaver) — Fri, 12 Aug 2011 11:39:00 +0100

This is a guide to show how you should go about configuring a 2 Way Synchronisation between a SQL Table and a SharePoint List with Data Synchronisation Studio.

Prerequisites

Data Synchronisation Studio 3.0 You can download the 7 day trial here which to try out this solution
SQL Server Compact Edition 4.0 via Web Platform Installer

Background

To successfully implement a 2 way sync you need to use a key value that will always be unique even when the new row is created from the other side. The simplest way is to use a GUID and this is what we recommend. Data Sync supports a special column we created called DSID in SharePoint that when Data Sync see's this column and it's blank it writes a new Guid back to the Row so that it always remains unique.

Setup Data Sources

SQL Server Table

In this example were setting up a simple table that contains contact records and the important column in here is DSID which is configured as a uniqueidentifier and has a default of newid() so that new rows automatically get a new Guid value.

CREATE TABLE [dbo].[Contacts] (
    [ContactID] int IDENTITY(1, 1) NOT NULL,
    [FirstName] nvarchar(255) NULL,
    [LastName] nvarchar(255) NULL,
    [MiddleName] nvarchar(255) NULL,
    [Suffix] nvarchar(50) NULL,
    [EmailAddress] nvarchar(255) NULL,
    [Phone] nvarchar(255) NULL,
    [DSID] uniqueidentifier DEFAULT(newid()) NOT NULL
)
GO

ALTER TABLE [dbo].[Contacts] ADD CONSTRAINT [PK_Contacts_0B234716] PRIMARY KEY CLUSTERED ([ContactID] ASC)

The table data will look like this below

SharePoint List

I have created a standard Contact List in a new SharePoint Site to be the Target for this Data. (This is SharePoint 2007 although you can use SharePoint 2010 as well).

This list does not have the column DSID yet so we need to add it and Data Sync has a special Wizard that lets you add new columns that are hidden from the user.

From the tools menu select the SharePoint Column Wizard

Step through the Wizard and Choose your SharePoint Server

Select the List on which to add the new Column

Add the New Column DSID to this List and make it hidden so the user cannot modify it.

Setup Data Sync Project

Start Data Sync and setup the project by connecting Data Source A to your SQL Table and Data Source B to the SharePoint List.

Your Project would look like this with SQL Table on the Left and the Target SharePoint List on the right.

The Schema Map would look like this with DSID as the Key and note that the SharePoint ID column is not included in this Map.

Now we should sync 1 way all our data into SharePoint by simply Compare A->B and Synchronise.

Sync the contact records into SharePoint

Run a Compare A->B again to check that all the contact records are in Sync with SharePoint.

And check SharePoint

Configure 2 Way Sync

Now to configure the 2 way sync we need to adjust the project properties, save the project and load it into the Run Tool.

Select File -> Properties

Change the Sync option to SyncAandB to enable 2 way synchronisation.

Then save the project.

Run Tool

Open the "Data Synchronisation Studio Run Tool" this is the program you use to configure unattended execution of Data Sync projects. With this you can create a Data Sync solution of many Data Sync projects and set the order of execution, send email and call other programs in the pipeline.

Under the File menu select Import and browse to your Data Sync project.

Save the Run Tool project and click Run in the Toolbar, if all is working you will get a nice green tick.

On the Log tab you can see the detailed log of event's, here you can see a new SQL Server Compact Database is created and loaded with data from each side and this is how it detects the changes made on each side.

The Snapshot Database is stored in the same path as the Run Tool solution project file like this.

Now if you run the project again you will see a shorter log and no changes.

Where as if you make a change in SharePoint you will see a change detected on B and applied to A like this.

And this is essentially all you need to do to create a 2 way sync between a SharePoint list and a SQL Table.

The only remaining step is to configure a schedule to run the project regularly.

Dynamic Lookups with Data Synchronisation Studio

info@simego.com (Sean Cleaver) — Thu, 11 Aug 2011 14:45:43 +0100

Data Synchronisation Studio's Dynamic Columns makes it really easy to lookup Data in a remote system to Lookup Data and decorate the source Row with the value.

For example we have this simple set of currency data, this contains an ID, Currency Name and Value. However our target Database requires an ID value for the Currency Name so we need to transform this with a Dynamic Column.

Data Sync includes the excellent Massive Data Access library and by using this it's really easy to create a Lookup Dictionary that you can use to lookup the Currency ID.

#region Usings
using System;
using System.Linq;
using System.Collections.Generic;
using System.Data;
using System.Globalization;
using System.Text;
using Simego.DataSync.Helpers.Massive;
#endregion

class DataSourceRowOverride : Simego.DataSync.DynamicColumns.DataSourceRowInternal //Do Not Change This Line
{
    private dynamic currencyLookup;
    
    public object CurrencyID
    {
        get { return currencyLookup.ContainsKey(CurrencyName) ? currencyLookup[CurrencyName].ID : null; }
    }
    
    public override void Setup() 
    {
        currencyLookup = new DynamicModel(DataSourceB.Settings["ConnectionString"], "Currency").All().ToDictionary(k => k.CurrencyName);    
    }

}

Then it's just a simple case of mapping the Dynamic CurrencyID column to the target.

Bulk update Active Directory

info@simego.com (Sean Cleaver) — Wed, 27 Jul 2011 11:14:11 +0100

Using Data Sync it's pretty easy to apply bulk updates to meta data stored in your Active Directory. One way is to export the whole AD to an Excel Spread sheet then sync changes from the Excel Sheet back to the AD.

However another way if you just want to apply a transformation is to use Dynamic Columns, here were updating the Company Name and Email Address fields of all our Users in the AD. We set up a project that has the Source and Destination set to the same data source and we use Dynamic Columns to calculate the changes we need.

Simple Dynamic Columns like this apply the Transformations we need and all we need do is Map the new Dynamic Columns to the AD Fields.

The completed project in Data Sync

Importing Data from QuickBooks with Data Sync Studio

info@simego.com (Sean Cleaver) — Thu, 09 Jun 2011 13:16:11 +0100

This solution shows how you might connect a remote QuickBooks installation to Data Sync to synchronise the data with SharePoint, CRM or a SQL Database.

In this example we have created a basic web server that gets installed onto the QuickBooks machine, this webserver exposes the QuickBooks data as an OData feed which can then be consumed by many applications such as Excel Power Pivot or Data Synchronisation Studio.

The Web Server uses the Inuit COM API to access data stored in QuickBooks as in the example code below.

We run the Web server on the QuickBooks machine from the command line.

To enable a external host url we have to run a little netsh command

netsh http add urlacl url=http://WIN-3KL7AJEADUK:4001/ user=WIN-3KL7AJEADUK\Sean

This Web server exposes a few admin / status screens so we can see what's going on

If we view the feeds in IE you get the default RSS view of the data from QuickBooks

Next we connect Data Sync to the Feed with the OData Provider

And Voila we have connected Data Sync to the OData feed and can now work with it like any other Datasource in Data Sync.

If you have a need for a solution like this to connect to your QuickBooks data please contact us.

Office 365 SharePoint Data Synchronisation

info@simego.com (Sean Cleaver) — Fri, 03 Jun 2011 10:11:04 +0100

We have built a new Data Provider for Data Sync that can be used to Synchronise Lists and Document Libraries including Metadata between some source and Office 365 SharePoint Online.

For Example Synchronising Documents between an OnPremise SharePoint Server Document Library and SharePoint Online is quite simple.

Supports

Lists
Document Libraries
Lookup Columns (Automatic linking)
Choice Columns
Large Lists (over 2000 items) with Paging
Folders in Document Libraries

Requirements

Data Synchronisaton Studio version 3.0.388 or higher
Windows Identity Framework - Download Here

Connection to SharePoint Online

Schema Map between SharePoint and SharePoint Online

Compare Results

We can maintain the Created and Modified values and any folders are also created in SharePoint Online.

Synchronise the Data

All done

With this new Data Provider you can either migrate to SharePoint Online from OnPremise or just publish some of your business data to the cloud. As with everything Data Sync you can setup a regular schedule to push content into SharePoint Online very easily.

Dynamic Columns - Back to Basics

info@simego.com (Trem) — Thu, 26 May 2011 15:04:42 +0100

Today we’re going to look at DS3's (Data Synchronisation Studio 3) dynamic columns, however this time, it’s from a beginners perspective. We will be looking at combining values, adding dates, and pulling information from other data sources.

So to start, we need to open DS3 and connect to a data source. I’m using the Microsoft Adventureworks Database and selecting the Contacts Table.

Once you have connected to your chosen data source, enable dynamic columns. You will see the following (or something like it). The part we’re interested in, is the section highlighted in yellow. This is the part that creates the new dynamic column.

Deconstructing the Code

public string MyNewColum

public means to make public/to make available for Data Sync to use.

public string MyNewColum

string basically means you can use numbers, letters and symbols

public string MyNewColum

MyNewColumn is the name of the dynamic column. This can be changed to anything baring in mind it cannot contain spaces or symbols, and must not start with a number.

MyNewColumn – My New Column – MyNewColumn1 – 1MyNewColumn

Adding Columns to Schema Map

If you now press the button at the top of the dynamic column window, you should notice a new column in ‘Data Source A’ (left)

Switch to your schema map and add this new column by either dragging it over or by double clicking.

Now press the button to preview the data from ‘Data Source A’ including your new dynamic column. This will be the last column in the table and as you can see, the data in the column is the text that is returned from the function in the dynamic column window.

Combining Values

Combining values is very handy when needing to publish data on different systems. For example, building a new database for a new web application and pulling in existing customers from an older database.

You may want to display a display name when a user is authenticated that will include their User ID, First Name and Last Name. In that table I’m using there is no such thing as User ID, so first I am going to create a User ID from the ContactID column.

public string UserID
    {
        get
        {
            return string.Format("SU{0}", this.ContactID.ToString("D4"));
        }
    }

The above code will return something like this SU1234

Remember… If you have added a new property (column), to press the ‘Build’ button to compile the code so that you can add it to the schema map before you click the ‘Preview’ button. If you change the name of the column, also remember to add it back to the schema map.

We’re using string.Format() so that we can add a prefix to the ContactID. The ‘{0}’ gets replaced by ‘this.ContactID.ToString(“D4”)’
We could just use this.ContactID, however for a ContactID of 1, this would return SU1 which doesn’t look like much of a User ID, so instead we prepend zeros to the ContactID by using ToString(“D4”).
Now the User ID looks like this SU0001.

Now we have our User ID, we can start to build our display name. We’re again going to use string.Format() to do this.

public string DisplayName
    {
        get
        {
            return string.Format("({0}) {1} {2}", this.UserID, this.FirstName, this.LastName);
        }
    }

The above code will return something like this (SU1234) Joe Bloggs

This format function contains ‘{0}’, ‘{1}’ and ‘{2}’. UserID will replace ‘{0}’, FirstName will replace ‘{1}’ and LastName will replace ‘{2}’

As you may of noticed I have changed the Column name to ‘DisplayName’, so once built you will need to add this back to your schema map.

Inserting Time & Date

In this example we will be adding the ‘Synchronised Date’ as this is one of the most simple things that you can do with DateTime.

public DateTime SyncDate
    {
        get
        {
            return DateTime.Now;
        }
    }

This will actually return the time and date when the data is loaded rather than when it gets synchronised, however it gives a reasonable value.

We could get slightly more complex and get a duration from two(2) dates.

    public TimeSpan Duration
    {
        get
        {
            DateTime dt1 = DateTime.Now;
            DateTime dt2 = dt1.AddMinutes(8).AddSeconds(45);
            return dt2.Subtract(dt1);
        }
    }

So in this code example we are adding eight(8) minutes and forty five(45) seconds to dt1. This is just to create a new date so that we can get a duration from the two of them. In a real situation you would have another date to compare against. For example, online exam started and online exam finished would give you the duration that the examination took to complete.

Pulling Data form other Data Sources

Importing data during a sync can be extremely useful if you want to merge two(2) data sources. In this example we will link ‘Contact’ with ‘ContactCreditCard’ not that you would ever want to do this, but for the purpose of this example this is what we’ll be doing :)

First off... Add the using Simego.DataSync.Helpers at the top of the code window (Click the plus icon to expand them)

// Create Lookup Helper Dictionary
    private OleDbDataLookupHelper<int, string> Lookup;
    
    // Create new Dynamic Column for Credit Card ID
    public string CreditCard
    {
        get
        {
            // Check to see if the Helper Dictionary contains a creadit card id for the current contact
            if (Lookup.Items.ContainsKey(this.ContactID))
            {
                // If contact has a credit card on file, return the credit card id
                return Lookup.Items[this.ContactID];
            }
            // Otherwise leave as blank
            return null;
        }
    }
    
    // This code happens before the data is loaded
    public override void Setup()
    {
        // Add items to the Lookup Helper Dictionary
        Lookup = new OleDbDataLookupHelper<int, string>(
            // This uses Data Source A's connection to the database.
            // This is fine if you are connecting to a table in the same database,
            // otherwise you would have to enter a connection string like this...
            // data source=MYSERVER;initial catalog=MYDATABASE;integrated security=false;persist security info=true;User ID=MYUSERNAME;Password=MYPASSWORD;
            this.DataSourceA.Settings["ConnectionString"],
            // This is the select statement. This selects all records from table Sales.ContactCreditCard
            "SELECT * FROM Sales.ContactCreditCard",
            // This is the column we want to use for the dictionary key
            "ContactID",
            // This is the column we want to use as the value
            "CreditCardID"
            );
    }

The above code creates a blank dictionary which we then populate during the setup by connecting to another table in the same database. We then create a new dynamic column which checks the dictionary for a value. If the dictionary contains an item for that contact, it then uses this a its value for the current row.

I hope this has helped you understand a bit more how dynamic columns work. If you need any more help just ask in the comments.

Using SQL Agent to Run Data Sync Projects

info@simego.com (Sean Cleaver) — Wed, 25 May 2011 16:42:38 +0100

If you prefer to use SQL Server Agent to schedule and run your Data Synchronisation Studio projects you can do so by calling our Run Tool from SQL Server.

Create a New Job

Add a new CmdExec Job Step

Set the Command to Simego.DataSync.Run.exe and path to your project file.

For Example

"C:\Program Files\Simego\Data Synchronisation Studio 3.0\Simego.DataSync.Run.exe" "C:\PathToProject\project.dsrun" -execute -window:minimize

Note: Data Sync will run under the user context of SQL Agent therefore your SQL Agent service account must have enough privileges to access the resources in your Data Sync projects.

CRM 2011 - Import Data and Create Relation to Account Entity

info@simego.com (Sean Cleaver) — Tue, 10 May 2011 14:31:48 +0100

This is an example to show you how to use Dynamic Columns to lookup values in a related CRM entity in order to create a Parent Child relationship as in Sales History to Accounts using Data Syncronisation Studio.

In this example I created a very basic sales history Excel spread sheet that contains our Internal Account Number and the value.

We need to lookup the Account Number against the Account entity to get the Guid of the Account to sync into the Sales History entity.

We use Dynamic Columns to Load the Account Entity, build a Dictionary of Account Numbers against Account ID's and use that to decorate the Source Data with a new Dynamic Column AccountID.

#region Usings
using System;
using System.Collections.Generic;
using System.Data;
using System.Globalization;
using System.Text;
using Simego.DataSync;
using Simego.DataSync.Providers.MSCrm2011;
#endregion

class DataSourceRowOverride : Simego.DataSync.DynamicColumns.DataSourceRowInternal //Do Not Change This Line
{
    //Category Index
    Dictionary<string, Guid> accountLookup = new Dictionary<string, Guid>();
    
    /// <summary>
    /// Create a Virtual ID column for our Key
    /// </summary>
    public Guid id 
    {
        get 
        {
            return Guid.NewGuid();
        }
    }
    
    /// <summary>
    /// Return the Account ID via the Lookup Dictionary
    /// </summary>
    public object accountid
    {
        get 
        { 
            if ( accountLookup.ContainsKey(this.accountnumber) )
                return accountLookup[this.accountnumber];
            
            return null;
        } 
    }

    public override void Setup()
    {
        //Get a DataTable object of the related List via a new CRM Reader
        MSCrmDataSourceReader reader = new MSCrmDataSourceReader();        
        reader.CrmServerUrl = ((MSCrmDataSourceReader)this.DataSourceB.Reader).CrmServerUrl;
        reader.Organisation = ((MSCrmDataSourceReader)this.DataSourceB.Reader).Organisation;
        reader.Entity = "account";
        //Create a Schema Map to return the ID and account number
        reader.SchemaMap = new DataSchema();
        reader.SchemaMap.Map.Add(new DataSchemaItem("accountid", typeof(Guid), true, false, false, -1));
        reader.SchemaMap.Map.Add(new DataSchemaItem("accountnumber", typeof(string), false, false, false, -1));
        //Load the Data Table
        DataTable accounts = reader.GetDataTable();        
        //Create a Lookup Index to help speed-up lookups
        foreach(DataRow r in accounts.Rows) 
            accountLookup.Add((string)r["accountnumber"], (Guid)r["accountid"]);
    }
}

So now we can map our source data to our Sales History Entity in CRM.

After we import the Data into CRM we now have Sales History items with the related Account defined correctly.

Note: In this example I created a virtual ID column as I had no row in my source Excel file, this causes all entries to be deleted and added on each run as the ID changes every time. In reality you would create a key based on multiple columns i.e. Account,Year,Month for example or store a source reference instead.

Integrate Applications with Data Sync

info@simego.com (Sean Cleaver) — Mon, 09 May 2011 15:22:01 +0100

We've been migrating some of the features of Ouvvi into a new Service for Data Synchronisation Studio. This makes it possible to integrate your applications with Data Sync to initiate the Sync when data actually changes to create almost real-time data synchronisation.

For example in SharePoint you might develop a List Event Handler that calls this Service to initiate a Sync when a List Item has been changed.

The service is really easy to setup and configure, this time we have No Database, No IIS etc. You simply install the service, set the user account, configure the settings in the ini file and start.

Supports

Time of Day Triggers
Time Interval Triggers
Remote Project Start
Remote Log View
Project Re-try
Queue based execution

We also have an interactive debug console so you can see what's happening in real-time inside the Sync service.

There's also a Web Status site where you can view the configuration, view the log, remotely start Data Sync projects. (does not use IIS)

All configuration is via a simple ini file like this, so it's really easy to configure. (ini file as it's easier than Xml for end users to work with)

To initiate a Sync project from a remote application you just have to send a HTTP request like this http://localhost:4000/Start/myproject to the Sync service, this then registers the job in the Job Queue and to start it immediately you send another HTTP request to http://localhost:4000/Wake/DataSync. If you do not send the Wake Request then the job will start the next time the Data Sync process runs i.e. 30 seconds.

This application is currently in private beta if you are a current Data Sync customer would like to know more please contact us at the usual address.

Dynamic Column Example Joining Related SharePoint Lists

info@simego.com (Sean Cleaver) — Wed, 27 Apr 2011 10:49:19 +0100

This is an example how you can use Dynamic Columns to pull in additional data to decorate your resultset with related Data. You could easily use this same process to decorate your data with additional data from other data sources. If you think about this makes Data Sync a very powerful Data Migration/Transformation tool.

In this example I created a SharePoint List with Product data from the Northwind Database I then created a Lookup List with Category names and linked the 2 lists together via a SharePoint Lookup Column.

The Goal is to pull out the "Description" column from the Category List for each Product in the Product List.

Here's the Data in SharePoint

Here is the code for the Dynamic Columns in here we load the "Categories" list into a Data Table from SharePoint and create an index to lookup the Description value for each row.

#region Usings
using System;
using System.Collections.Generic;
using System.Data;
using System.Globalization;
using System.Text;
using Simego.DataSync.Providers.MSSharePoint;
#endregion

class DataSourceRowOverride : Simego.DataSync.DynamicColumns.DataSourceRowInternal //Do Not Change This Line
{
    //Category Data
    DataTable categories;
    //Category Index
    Dictionary<int, DataRow> lookup = new Dictionary<int, DataRow>();
    
    /// <summary>
    /// Get the Category ID from the Category Lookup Column
    /// </summary>
    public int O_CategoryID
    {
        get { 
            int ix = this.Category.IndexOf(";#");
            if ( ix > 0 )
                return Convert.ToInt32(this.Category.Substring(0, ix));
            
            return 0; 
        }
    }

    /// <summary>
    /// Description
    /// </summary>
    public string O_CategoryDescription {         
        get 
        { 
            object val = lookup[O_CategoryID]["Description"];
            if ( val == DBNull.Value ) return null;
            return (string)val;
        } 
    }
    
    public override void Setup()
    {
        //Get a DataTable object of the related List via a new SharePoint Reader
        SharePointDataSourceReader reader = new SharePointDataSourceReader();
        reader.ListUrl = ((SharePointDataSourceReader)this.DataSourceA.Reader).ListUrl;
        reader.ListName = "Categories";
        //Get the Default Schema for this List
        reader.SchemaMap = reader.GetDefaultDataSchema();
        //Load the Data Table
        categories = reader.GetDataTable();        
        //Create a Lookup Index to help speed-up lookups
        foreach(DataRow r in categories.Rows) 
            lookup.Add((int)r["ID"], r);
    }
}

This is the result, here you can see we have pulled in the Description value from the related Categories List.

Data Sync Run Tool Automation Example

info@simego.com (Sean Cleaver) — Tue, 26 Apr 2011 12:53:52 +0100

Requires: Data Sync version 3.0.372 or higher.

This is an example of how to use the Data Sync Run Tool to automate several steps as part of an ETL type pipeline.

In this example we are going to execute the following steps.

Download an Excel Spreadsheet with data from a Http Source (SharePoint Document Library)
Import/Sync this Data with a SharePoint List
Delete the downloaded file once complete
Send an Email if there was an error

First we create an Excel file with the Data we want to Sync and Upload it to a SharePoint Document Library.

Next we create a Data Sync project that will Sync this Spreadsheet with a SharePoint List, save the project and exit Data Sync.

Start up the Data Sync Run Tool and add a new "HTTP Download File Step"

Edit the configuration for the Step i.e. enter the Destination Filename and Url to the Source file i.e. http://yoda/Shared%20Documents/products.xlsx

Next add the Data Sync project step from the File Import menu.

Next create a Delete File Step to Delete the Excel File after we have processed it.

Next add a Step to Send us an email if there was an error, the trick here is to set the Run Condition to "Failure" so this step only runs if there is an Error. You will also need to enter your SMTP server details under Tools Options for this to work.

Now run the project to check that it works as expected.

That's it now all you need do is save the Run project so you can open it any time to execute it or Schedule it with Windows Task Scheduler.

Avoiding SSL Errors

info@simego.com (Sean Cleaver) — Tue, 19 Apr 2011 16:52:45 +0100

We have added a new feature to Data Synchronisation Studio to allow you to override the SSL validation behaviour of any of the providers.

You might want to do this if your using a self signed development certificate in a non-production system. By setting the OverrideSSLCheck property in the project properties dialog to true forces any SSL Certificate validations to return a successful response even if the certificate is invalid.

Synchronising SharePoint Documents and Metadata

info@simego.com (Sean Cleaver) — Tue, 19 Apr 2011 16:40:49 +0100

We have been working on a new SharePoint provider for Data Synchronisation Studio that makes it now possible to Sync documents and metadata between SharePoint Document Libraries. You might use this provider if you want to migrate between SharePoint 2007 and SharePoint 2010 or just between an on Premise server farm and an Off Premise server farm.

This new provider is the same for Lists or Document Libraries and any custom columns you have on your library are now available.

With Dynamic Columns it makes it really easy to decorate your document Library columns with data, for example you can now pull files from a file share and decorate the metadata from a database and synchronise this to a SharePoint document library. Rather than a Bulk import on Day 1 you can incrementally synchronise changed files and metadata over time.

Create Website Thumbnail Images Based on Larger Originals Using Data Synchronisation Studio's Dynamic Columns

info@simego.com (Trem) — Thu, 14 Apr 2011 11:15:57 +0100

I would just like to demonstrate how to create smaller versions of your images on your website using the latest version of Data Synchronisation Studio and its dynamic columns feature.
The main reason you would want to do this is for a major performance boost and reduction in page loading time.

So if you can imagine, the image on the left is a large 700px x 1200px image that will be used for a zoom control so that customers can see the item in greater detail. This is fine for this purpose, however using this size for a product list will greatly increase page loading time, and also eat away at your bandwidth.

Resizing all site images by hand using an image editing suite will take you quite a while, and of course you’ll need the image editing suite in the first place. I’m sure there are a few solutions online to do this, however if your site is database driven and images have already been stored in the database, it makes it pretty difficult to do this.

Data Synchronisation Studio will pull the image data from the database and using dynamic columns we can manipulate the image using a few lines of code in C#

These thumbnail images can then be used for your item lists like so…

Reducing the image size with a bit of C#

So first of all we need to connect Data Sync to the site’s database (or a copy of)

So here I have opened the sites attachment table, which contains the images for the site.
It has many columns which include ‘Content Type’, ‘Name’, ‘Path’, ‘Length’, ‘Checksum’, ‘Data’ etc. The only column we will be using for the conversion is ‘Data’. However we will also use the ‘ID’ column so that we can synchronise back into another table.

At the moment my schema map looks like this, however this will change once we have created our dynamic column.

To create the dynamic column, select the dynamic columns tab and click enable. This will load the dynamic column generator with some demo code.

Where it has the demo column code, replace with this…

public byte[] SmallImage
    {
        get
        {
            try
            {
                // Create a new stream and read in the data. Where this.Data = the Data Column
                using (MemoryStream stream = new MemoryStream(this.Data))
                {
                    // Create the original image using the stream data
                    using (Image img = Image.FromStream(stream))
                    {
                        // Get the image dimensions
                        double width = img.Width;
                        double height = img.Height;
                        
                        // Only resize if the image is larger than the thumbnail size
                        if (height > 150)
                        {
                            // Find the aspect ratio so that we don't distort the image during reduction
                            double aspectRatio = width / height;
                            
                            // Set a fixed height for your thumbnails
                            height = 150;
                            
                            // Get the width based on the height and aspect ratio to get the correct scale
                            width = height * aspectRatio;
                        }
                        
                        // Create a new image based on the original and new size
                        using (Bitmap bmp = new Bitmap(img, (int)width, (int)height))
                        {
                            using (MemoryStream smallImage = new MemoryStream())
                            {
                                // Save the thumbnail image and return as a byte array
                                bmp.Save(smallImage, ImageFormat.Png);
                                return smallImage.ToArray();
                            }
                        }
                    }
                }
            }
            catch
            {
                return null;
            }
        }
    }

This code is only a basic way of reduction and only reduces the image based on its aspect ratio. You could modify the code to create fixed sized thumbnails, it really does depend on your requirements.

Click build to compile the code. If all is well, which it should be, go back to your schema map and add the new dynamic column by dragging it into the schema.

Click preview…
You can now see that we have an extra column which contains the thumbnail image data.

This can now be published to the existing table or a new table.

Just another thing that Data Synchronisation Studio can do.

Sources
www.amazon.co.uk

Filter Active Directory Records before Synchronisation

info@simego.com (Sean Cleaver) — Thu, 07 Apr 2011 11:25:56 +0100

You know Data Sync can synchronise user information data from your Active Directory into SharePoint and SQL etc. You can also filter what records to include by specifying a LDAP filter like this

(&(objectClass=user)(memberOf=CN=SimegoDevelopers,CN=Users,DC=office,DC=simego,DC=com))

However this can get really complicated real fast if you actually want to do something a bit more complex.

Therefore with Dynamic Column filtering it's much simpler, just override the BeginRow() method and apply your filtering with .NET Code instead. This will then be evaluated at runtime and only the rows that return true from this function will be included in the resulting data set.

If you have a large AD it's still a good idea to partially filter the results with an LDAP Filter to maybe limit the OU then apply the .NET filter for the trickier parts.

    public override bool BeginRow()
    {
        if ( this.MemberOf.Contains("Simego Developers") )
            return true;
        
        return false;
    }

UPDATE: IncludeRow() is now BeginRow()

CRM 2011 Import Custom Entities and Related Data

info@simego.com (Sean Cleaver) — Fri, 01 Apr 2011 11:02:56 +0100

In this example were showing how you can Import/Synchronise data from a SQL Server Database into an instance of CRM 2011 with custom entitles and Data Synchronisation Studio.

We're going to import Sales Order History from the Adventure Works database and relate each Order History line to an Account in the CRM system.

Using the Query below we can pull out the Sales Order Header Lines and in this example we are going to relate the Contact ID back to the Account in CRM (previously we imported the Contacts into the Account Entity using the rowguid as the accountid in the CRM system)

This is the typical result we get from the Query

Next we create a new Entity in CRM with 2 custom fields, Amount and Account

Amount Field

Account Field

We can then simply map the Data in Data Sync between the Source SQL Query and CRM entity like this.

Schema Map

Compare and Sync the Data

Once the Data is imported you can then create a view to view the list of Sales or records.

Sales order item showing Account Relationship connected.

Automate Export of Data from CRM 2011 Online

info@simego.com (Sean Cleaver) — Fri, 25 Mar 2011 15:58:12 +0100

With Data Sync it's really easy to setup an automation task to Export Data from your Cloud based CRM 2011 instance to an on-premise SQL Server for inclusion in what ever on premise applications require the data from your CRM system.

For CRM Dynamics online you will also need to install the Windows Identity Foundation (WIF).

Simply setup a Data Sync project to Connect to CRM and a Local Data Source (Data Sync can create a SQL Table automatically from the CRM Schema with the SQL Table Wizard).

To improve the performance only sync the specific columns you need, remove the unwanted columns from the Schema Map and only the columns in the Schema Map will be requested.

Compare the data to get a Change Set to apply to your On-Premise database

Sync the changes.

You can then set the project to run in the Run Tool and maybe run several other exports as well or transformations.

Once the project is running successfully in the run tool you can then schedule it to run with Windows Task Scheduler so it runs every day.

Sync/Import a Subset of a SQL Table

info@simego.com (Sean Cleaver) — Fri, 25 Mar 2011 13:39:43 +0100

This update to Data Sync allows you to filter the SQL Providers to return a subset of data without loosing the ability to write back to the data source.

The current filter option involves writing your own SQL Command however this prevents write back to the data source. Now you can supply your own WHERE clause as a separate parameter on the configuration instead and we just append it to the query we build. Therefore now you can sync a subset of data stored in a SQL Table.

This update is available in the beta release of Data Sync Studio version 3.0.356

Update/Clean/Repair Data

info@simego.com (Sean Cleaver) — Fri, 25 Mar 2011 12:15:46 +0100

This is an example of how you can use Data Sync to clean up existing data as part of a migration or just a clean up exercise.

If you set the source and destination objects in Data Sync to the same data source you can then use Data Transformation via Dynamic Columns to simply clean up the data, view the change results and synchronise the changes.

This is an example where we're Updating the Modified Timestamp and removing all the "-" from phone numbers and making sure there is no trailing or leading whitespace. This is a fairly trivial example but you could quite easily be more adventurous and virtually do what you like as your running full .NET Framework here.

Create the Transformation columns with the Dynamic Column code editor.

You can see here it's still fast as your running compiled code against your data.

And you can preview the changes before you apply them to your database (try that with SQL!)

And now it's all in a Data Sync project you can easily run the process again by just opening the Data Sync project and running compare and sync.

Data Sync with Azure Storage

info@simego.com (Sean Cleaver) — Wed, 23 Mar 2011 17:03:28 +0100

Ok so we built a version that works with Azure Blob Storage as too. This version doesn't have the same problem as the Amazon S3 version as we can store Created and Modified dates as additional meta data and return it in a single request.

We're going to use this provider on our Web Server to maintain a backup of all our Web Sites and Client Sites in Azure so if we loose the server we can restore very quickly. In fact we use Quvvi to backup all our databases daily as well and now we can use this provider with Ouvvi to post the backups to Azure automatically.

Download the provider here (requires Data Sync version 3.0.352)

Simego.DataSync.Providers.Azure

Data Sync with Amazon S3 Storage

info@simego.com (Sean Cleaver) — Tue, 22 Mar 2011 16:07:18 +0100

Out of interest I had a go at building a Data Sync provider that we could use to backup our Website content to Amazon S3.

It turned out to be pretty simple using the Amazon SDK, there's a couple of gotchas that cause a little trouble as we can't set the modified date of a file in S3 so instead we add some custom meta data for Created and Modified which in turn hurts our performance on the read. We have to issue a web request for each file's metadata to return the modified and created dates. Good news is that if you're not that worried about that you can remove them from the schema map and just rely on the Length of the file for change detection.

Download the provider from here

Simego.DataSync.Providers.AmazonS3.zip

Import/Export/Sync Data with Dynamics CRM 2011 Online

info@simego.com (Sean Cleaver) — Tue, 01 Mar 2011 15:48:14 +0100

With Data Sync 3.0 we have added a data provider for Microsoft Dynamics CRM 2011. With this provider you can read/write data to CRM 2011 including the hosted online version. This provider is written from the ground up with .NET 4.0 and uses the new WCF services to read and write the data.

For CRM Dynamics online you will also need to install the Windows Identity Foundation (WIF).

You can connect to any CRM entity and sync data from any of the Data Sync supported data sources, i.e. SQL Server, ODBC, OLEDB, SharePoint, Exchange, OData etc. If your planning on a migration project to CRM 2011 Data Sync is an ideal tool to manage data transfer as you can quickly migrate data in a repeatable and controllable manner.

As Data Sync 3.0 also includes Dynamic Columns you can also use Data Sync to decorate existing records with additional metadata by building these Dynamic Columns to reference the additional metadata.

Simply connect to your CRM instance by entering your credentials (Live ID for Online), Url and choose your organisation and entity.

You can then simply map your data in Data Sync and sync away …

Away you go…

Download Data Sync now here, connect to CRM Online and export and import whatever data you need.

Using Data Sync Dynamic Columns to build VCARD

info@simego.com (Sean Cleaver) — Tue, 22 Feb 2011 09:52:44 +0100

Infuse Consulting came up with an interesting use of Dynamic Columns to convert Contact Data stored in MS CRM into a VCard file which could then be loaded into Google Apps or anything that accepts a VCARD data file.

What you need is a Simple Data Provider which just outputs to a plain Text file

Here’s an example VCard Dynamic Column you just make up the VCARD data in a String and return it as a property.

As the simple data provider never returns any data you always get ADD Actions so it writes a new file each time, it’s pretty quick so not a problem really.

The final VCARD file that you can then import into Google Apps or any other program that accepts a VCARD file.

Data Sync: Generic Xml Provider

info@simego.com (Sean Cleaver) — Tue, 15 Feb 2011 12:41:11 +0100

Data Sync 3.0 is now shipping with a generic Xml Provider where you can turn any simple Xml Feed into a Data Source and Sync to SQL, Oracle, OleDB, SharePoint etc.

Suppose you have an Xml Feed like this something simple that represents rows and columns.

Simply connect it to data sync with the XML File provider

Specify the XPath expression to each node (column) you want to return

Hey presto one Xml Data Source ready for synching….

Create a safe Database for Development/Testing

info@simego.com (Sean Cleaver) — Mon, 14 Feb 2011 15:43:08 +0100

So we all know that developing with “REAL” data is nice however usually it’s not practical as the data is sensitive and cannot be accessed out side of a production environment. So how about Data Synchronisation Studio just obfuscate the data that’s sensitive and you can then use real data for development and testing.

In data sync create a mapping between a source database and your new test database. Then create some dynamic columns to obfuscate the sensitive data such as Client Name, Company Name etc.

Using the ObfuscateHelper class you can obfuscate a string into something less obvious but still printable.

i.e.

ObfuscateHelper.SimpleObfuscateValue(this.CompanyName);

The results then look a bit like this where the letters are repeated keeping case and any special characters.

Government Suppliers & Contracts Data Feed

info@simego.com (Sean Cleaver) — Fri, 11 Feb 2011 22:40:12 +0100

Today the government announced a new website http://www.contractsfinder.businesslink.gov.uk/ to give businesses access to all government contracts over £25,000 in value.

This site also has an Xml data feed available so I have put together this Data Provider for Data Synchronisation Studio so you can import the data on a daily basis into SharePoint, Excel, SQL Server etc. for analysis.

This adapter is only compatible with Data Sync 3.0 which you can get from here http://www.simego.com/Beta

The adapter is available from here http://www.simego.com/downloads/Simego.DataSync.ContractsFeed.zip you simply install it into Data Sync as an Add-In

Simply Connect to the ContractFeedReader data source

Here’s the Data Schema Loaded

Clicking Preview returns the data (you can then sync this to multiple data sources)

The Code

This is the Data Sync provider source code if you want to write your own provider or extend this one.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Xml;
using Simego.DataSync.Providers;
namespace Simego.DataSync.ContractsFeed

{

    public class ContractFeedReader : ReaderProviderBase

    {        

        [Category("Configuration")]

        public string FeedUrl { get; set;  }
        public ContractFeedReader()

        {

            FeedUrl = "http://contractsfinder.businesslink.gov.uk/service/GetNoticeXmlFile.ashx";

        }
        public override DataSchema GetDefaultDataSchema()

        {

            DataTable dt = new DataTable("ContractFeed");

            

            dt.Columns.Add("NoticeID", typeof(int));

            dt.Columns.Add("Reference", typeof(string));

            dt.Columns.Add("DateCreated", typeof(DateTime));

            dt.Columns.Add("DatePublished", typeof(DateTime));

            dt.Columns.Add("DeadlineDate", typeof(DateTime));

            dt.Columns.Add("ValueMin", typeof(double));

            dt.Columns.Add("ValueMax", typeof(double));

            dt.Columns.Add("Status", typeof(string));

            dt.Columns.Add("OrganisationName", typeof(string));

            dt.Columns.Add("OrganisationContact", typeof(string));

            dt.Columns.Add("Title", typeof(string));

            dt.Columns.Add("Description", typeof(string));

            dt.Columns.Add("Region", typeof(string));

            dt.Columns.Add("NoticeType", typeof(string));

            dt.Columns.Add("Url", typeof(string));
            dt.Columns["NoticeID"].AllowDBNull = false;

            dt.Columns["NoticeID"].Unique = true;

            

            return new DataSchema(dt); 

                        

        }
        public override DataTable GetDataTable()

        {

            //Create a Schema Mapping Helper

            DataSchemaMapping mapping = new DataSchemaMapping(SchemaMap, Side);

            

            DataTable dt = SchemaMap.GetDataTable();
            XmlDocument document = new XmlDocument();

            document.Load(FeedUrl);
            XmlNodeList list = document.SelectNodes("NOTICES/NOTICE");
            foreach (XmlNode node in list)

            {

                DataRow row = dt.NewRow();

                foreach (DataSchemaItem item in SchemaMap.GetIncludedColumns())

                {

                    string columnName = mapping.MapColumnToDestination(item);
                    switch (columnName)

                    {

                        case "NoticeID":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "NOTICEID"), item.DataType);

                                break;

                            }

                        case "Reference":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "REFERENCENUMBER"), item.DataType);

                                break;

                            }

                        case "DateCreated":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "DATECREATED"), item.DataType);

                                break;

                            }

                        case "DatePublished":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "DATEPUBLISHED"), item.DataType);

                                break;

                            }

                        case "DeadlineDate":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "DEADLINEDATE"), item.DataType);

                                break;

                            }

                        case "ValueMin":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "VALUEMIN"), item.DataType);

                                break;

                            }

                        case "ValueMax":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "VALUEMAX"), item.DataType);

                                break;

                            }

                        case "Status":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "STATUS"), item.DataType);

                                break;

                            }

                        case "OrganisationName":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "ORGANISATION/NAME"), item.DataType);

                                break;

                            }

                        case "OrganisationContact":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "ORGANISATION/CONTACTEMAIL"), item.DataType);

                                break;

                            }

                        case "Title":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "DETAILS/TITLE"), item.DataType);

                                break;

                            }

                        case "Description":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "DETAILS/DESCRIPTION"), item.DataType);

                                break;

                            }

                        case "Region":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "REGION/@NAME"), item.DataType);

                                break;

                            }

                        case "NoticeType":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "NOTICETYPE/@NAME"), item.DataType);

                                break;

                            }

                        case "Url":

                            {

                                row[columnName] = DataSchemaTypeConverter.ConvertTo(GetNodeVal(node, "URL"), item.DataType);

                                break;

                            }

                        default:

                            {

                                break;

                            }
                    }

                }
                dt.Rows.Add(row);

            }
            

            return dt;

        }
        private static object GetNodeVal(XmlNode node, string name)

        {

            XmlNode n = node.SelectSingleNode(name);

            if (n != null)

                return n.InnerText;
            return DBNull.Value;

        }
        public override List<ProviderParameter> GetInitializationParameters()

        {

            List<ProviderParameter> parameters = new List<ProviderParameter>();
            parameters.Add(new ProviderParameter("FeedUrl", this.FeedUrl));
            return parameters;

        }
        public override void Initialize(List<ProviderParameter> parameters)

        {

            foreach (ProviderParameter p in parameters)

            {

                switch (p.Name)

                {

                    case "FeedUrl":

                        {

                            this.FeedUrl = p.Value;

                            break;

                        }

                    default:

                        {

                            break;

                        }

                }

            }

        }

    }

}

Data Sync 3.0 - Run Tool

info@simego.com (Sean Cleaver) — Tue, 01 Feb 2011 20:31:40 +0100

Another new addition to Data Sync 3.0 is an upgrade to the run.exe program that you typically use to schedule Data Sync projects to run. Now this application is a full GUI application where you add steps and save as a *.dsrun solution file.

Each step is either a Data Sync project or any other step type that we ship such as Email and SQL Task. Each step is executed in turn subject to the “RunCondition” which is a status of the current batch so if the batch is failing you can add a step of type “Email” with a “RunCondition” of “Failure” to send an email if the process is failing.

Main UI Window

Sync SQL Database Wizard

Also we have a wizard in the program that can generate a batch of Data Sync projects and solution file to synchronise tables from one database to another without having to create each one in Data Sync.

Simply point the wizard at the source and destination databases, choose a save location and that’s it.

The Wizard then creates individual Data Sync project files and a solution file and opens it in the program ready to run.

Logging has also been improved to show you a nice list, the events are also recorded in the Windows Application log.

Data Sync 3.0 Lookup Helper Class

info@simego.com (Sean Cleaver) — Fri, 21 Jan 2011 16:08:51 +0100

We’ve created a lookup helper class that’s going to be part of Data Sync 3.0 to help when you need to lookup values that are in a different SQL Table in Dynamic Columns.

You need to import the Simego.DataSync library and then create an Instance of the OleDbDataLookupHelper object. This is a generic object so you can define the Key and Value type as any base type. Then you need to populate the list with values in the Setup method. Finally return a value by calling Lookup.Items[key].

The example below shows all the parts and as you can see it’s just a few lines of code.

Import SQL Data to SharePoint List

info@simego.com (Sean Cleaver) — Sat, 15 Jan 2011 14:07:20 +0100

Quick 1 minute video to demo how easy it is to import data from SQL Server into a SharePoint List with Data Synchronisation Studio, even creates the List on your SharePoint server for you.

Once the project is setup you can easily save it as a project and repeat the import applying only the changed data back to your SharePoint List.

The reverse is also possible by comparing the data in SharePoint against the SQL Table this way any changes in SharePoint are applied to your SQL Table.

Data Sync 3.0 – Dynamic Columns Demo Video

info@simego.com (Sean Cleaver) — Wed, 12 Jan 2011 16:20:38 +0100

Quick 1 minute demo to show how easy it is to create Data Transformations with Dynamic columns in Data Sync 3.0

Data Sync 3.0 – Row Filter

info@simego.com (Sean Cleaver) — Wed, 12 Jan 2011 13:39:54 +0100

With the Dynamic Column support in Data Sync 3.0 it is now possible to filter rows from Data Source A client side. You simply override the method IncludeRow() and return true if you want to keep it or false if you don’t want to pass it on to the comparison engine.

The example below only returns rows where the City equals “London”.

Data Sync 3.0 – Compile Time Error Improvements

info@simego.com (Sean Cleaver) — Wed, 12 Jan 2011 13:22:25 +0100

We’ve made some improvements with how compile time errors are displayed within the Dynamic Columns designer.

When we get a compile error the error line is now displayed in Yellow as below and double clicking on the error will take you to the line that has the error in the compiled output window.

The line with the error also displays the wavy red-line to guide you to the problem. To fix the error simply switch back to the Source Code editor and edit the code.

Data Sync Studio 3.0 - Intellisense

info@simego.com (Sean Cleaver) — Tue, 11 Jan 2011 09:22:51 +0100

We’ve made a few changes to the Dynamic Column feature in Data Sync 3.0 to include full intellisense, also now you can access any assembly that is loaded inside the Data Sync AppDomain.

With this new feature it makes it just too easy to transform your data from one format to another, I am not sure we should make it this easy!

Screenshot of the Designer

Validate Domain Names

info@simego.com (Sean Cleaver) — Tue, 04 Jan 2011 16:25:26 +0100

Going one step further we can validate the domain names by testing each row to see if there is a web site on the end with this simple Dynamic Column in Data Sync 3.0

using System.Net;
class DataSourceRowOverride : DataSourceRowInternal

{    

    public string EmailDomain { get { return GetDomainName(Email); } }

    public bool IsValidDomainName { get { return IsValidDomain(EmailDomain); } }
    public string GetDomainName(string value) 

    {

        var expression = new Regex(@"[A-Z0-9._%-]+@([A-Z0-9.-]+\.[A-Z]{2,4})", RegexOptions.IgnoreCase);

        

        var match = expression.Match(value);

        if ( match.Success )

            return match.Result("$1");

            

        return value;

    }
    public bool IsValidDomain(string domainName) 

    {

        try 

        {

            var request = HttpWebRequest.Create("http://www." + domainName);

            request.Timeout = 1000;

            using(var response = request.GetResponse()) 

            {

                

            }    

            return true;

        } 

        catch(WebException)

        {

            return false;    

        }

            

    }

}

Using Regular Expressions with Data Sync 3.0

info@simego.com (Sean Cleaver) — Tue, 04 Jan 2011 15:47:23 +0100

This is a real quick example of how to use a regular expression with Data Sync 3.0 Dynamic Columns to extract the Domain Name from a users Email address.

Dynamic Column Code

class DataSourceRowOverride : DataSourceRowInternal
{    
    public string EmailDomain { get { return GetDomainName(Email); } }
        
    public string GetDomainName(string value) 
    {
        var expression = new Regex(@"[A-Z0-9._%-]+@([A-Z0-9.-]+\.[A-Z]{2,4})", RegexOptions.IgnoreCase);
        
        var match = expression.Match(value);
        if ( match.Success )
            return match.Result("$1");
            
        return value;
    }
}

Referencing External Data with Data Sync 3.0

info@simego.com (Sean Cleaver) — Tue, 04 Jan 2011 15:29:29 +0100

This example shows how you can reference data which is held in an external database from within the new Dynamic Column support of Data Sync 3.0.

This is a rather trivial example however it shows the concept.

Override the “Setup” method and implement the data loading in this method, for this example were creating a Dictionary of Email Addresses keyed on the Row ID. Setup is only called once during the load process of the Data Source so it’s a good place to load this data.

The Settings object is a dictionary of settings configured for the Data Source so we can easily pull out the database connection string from this list.

The Property Email4 returns the Email address from the Dictionary rather than the actual Data source.

We can even add out own Using statements at the top of the class.

Our Code for the Project

using System.Data.OleDb;
class DataSourceRowOverride : DataSourceRowInternal

{

    public string Email4 { get { return Lookup[ID]; } }

    

    private Dictionary<int, string> Lookup = new Dictionary<int, string>();

    

    public override void Setup() 

    {

        string connectionString = Settings["ConnectionString"];    

        

        using(OleDbConnection c = new OleDbConnection(connectionString)) 

        {

            c.Open();

            

            using(OleDbCommand cmd = new OleDbCommand("SELECT * FROM Clients_D", c)) 

            {

                OleDbDataReader dbReader = cmd.ExecuteReader();

                while(dbReader.Read())

                    Lookup.Add((int)dbReader["ID"], (string)dbReader["Email"]);

                    

            }    

        }

    }

}

Complete Generated Code

#region Usings
using System;
using System.Collections.Generic;
using System.Data;
using System.Globalization;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
using System.Web;
using Simego.DataSync;
using Simego.DataSync.Interfaces;
#endregion
namespace Simego.DataSync.DynamicColumns_631D8F4B 

{

    using System.Data.OleDb;

    

    class DataSourceRowOverride_631D8F4B : DataSourceRowInternal_631D8F4B

    {

        public string Email4 { get { return Lookup[ID]; } }

        

        private Dictionary<int, string> Lookup = new Dictionary<int, string>();

        

        public override void Setup() 

        {

            string connectionString = Settings["ConnectionString"];    

            

            using(OleDbConnection c = new OleDbConnection(connectionString)) 

            {

                c.Open();

                

                using(OleDbCommand cmd = new OleDbCommand("SELECT * FROM Clients_D", c)) 

                {

                    OleDbDataReader dbReader = cmd.ExecuteReader();

                    while(dbReader.Read())

                        Lookup.Add((int)dbReader["ID"], (string)dbReader["Email"]);

                        

                }    

            }

        }

    }


    #region Generated Code

    class DataSourceRowInternal_631D8F4B : IDataSourceRow 

    {

        public Dictionary<string, string> Settings { get; set; }

        public IDataSourceReader Reader { get; set; }

        public DataRow Row { get; set; }

        public DataTable Table { get { return Row.Table; } }

        public TraceWriter Trace { get; set; }
        public int ID { get { return (int)Row["ID"]; } }

        public string Logo { get { return (string)(Row["Logo"] == DBNull.Value ? null : Row["Logo"]); } }

        public string Email { get { return (string)(Row["Email"] == DBNull.Value ? null : Row["Email"]); } }
        public virtual void Setup() { /* Override Setup to handle any Startup Initialization. */ }

        public virtual void BeginRow( /* Override BeginRow to handle any Initialization on each row. */) { }

        public virtual bool IncludeRow() { /* Override IncludeRow to filter out rows. */ return true; }

        public virtual void EndRow( /* Override EndRow to handle any Cleanup on each row. */ ) { } 

    }
    #endregion

}

Data Synchronisation Studio 3.0 : Dynamic Columns

info@simego.com (Sean Cleaver) — Fri, 31 Dec 2010 12:21:54 +0100

Data Sync 3.0 adds the ability to write custom Dynamic/Calculated columns to your Source Data Schema. You write these columns as Properties within a C# class and they are then exposed in the schema like this. Once they're designed you can simply drag them onto the Schema Map like any other column.

The design of the properties is built into Data Sync itself so you simply write the code and hit the Build button to verify that your code builds and update the schema.

First you need to enable Dynamic Columns in your project by clicking the “Enable Dynamic Columns” link in the Dynamic Columns window.

Now you can write your properties using usual C# syntax and with the full power of the .NET 4.0 framework.

Building the code generates an assembly at runtime which is used to return the values of the dynamic column.

With these new dynamic columns you can inject formatting, new columns, calculations etc into the sync process without having to resort to SQL as before.

You can also use this feature to clean up/fix your data by pointing the source and destination at the same database table and writing dynamic columns to transform the data.

Basecamp & Highrise Data Sync Provider

info@simego.com (Sean Cleaver) — Fri, 17 Dec 2010 11:53:54 +0100

Download Data Sync Add-in

I have just put together a Data Sync provider which connects to 37 Signals Highrise and Basecamp to pull Company and People data from those accounts so that you can integrate the data with your other systems.

Once you install the add-in into Data Sync just enter the URL to your Highrise or Basecamp account and your API key to get going.

Once you have connected to the Data Source it’s really easy to then import the data into other systems such as SharePoint Lists, SQL Databases etc. (i.e. Maintain your client list centrally in Highrise and sync it to your other LOB systems).

Connection Dialog

BaseCamp Data in Data Sync

Developing a Data Sync Custom Provider

info@simego.com (Sean Cleaver) — Fri, 10 Dec 2010 14:12:14 +0100

This is an example of a Data Provider for Data Sync that uses the base class ReadOnlyReaderProviderBase to reduce the effort required to build a provider.

Essentially you have to provide an implementation for the following methods.

GetDataTable – Return the Data.
GetDefaultDataSchema – Describe the Schema of the Source.
GetInitializationParameters – Describe what input parameters the provider requires.
Initialize – Populate the parameter values.

You will need to reference 2 Simego assemblies Simego.DataSync.dll and Simego.DataSync.Providers.dll

Once you have a provider built you just drop it in the Data Sync install directory to make it available.

Sample Code

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Xml;
using Simego.DataSync.Providers;
namespace Simego.DataSync.FXRateProvider

{

    [ProviderInfo(Name = "FX Rates")]

    public class FXRatesDataSourceReader : ReadOnlyReaderProviderBase

    {                        

        [Category("Connection")]

        [Description("Filename of the Excel file to read.")]

        public string BaseCurrency { get; set; }

        

        #region IDataSourceReader Members
        public FXRatesDataSourceReader()

            : base()

        {

            BaseCurrency = "USD";

        }
        public override DataTable GetDataTable()

        {

            //Get a Table to store the results.

            DataTable dt = SchemaMap.GetDataTable();

            dt.TableName = "FXRates";
            //Create a Schema Mapping Helper

            DataSchemaMapping mapping = new DataSchemaMapping(SchemaMap, Side);

            //Get the Rates from Yahoo

            Dictionary<string, double> rates = GetFXRates();

            

            //Caculate the Conversion Rate from other base currency

            double conversion_rate = 1 / rates[BaseCurrency];
            //Create a Sorted List of Rates

            var rateList = new List<string>();

            foreach (string k in rates.Keys)

                rateList.Add(k);

            rateList.Sort();
            //Populate the Rates Table

            foreach (string k in rateList)

            {

                DataRow newRow = dt.NewRow();

                foreach (DataSchemaItem item in SchemaMap.GetIncludedColumns())

                {

                    string columnName = mapping.MapColumnToDestination(item);

                    switch (columnName)

                    {

                        case "Currency":

                            {

                                newRow[item.ColumnName] = DataSchemaTypeConverter.ConvertTo(k, item.DataType); 

                                break;

                            }

                        case "Rate":

                            {

                                newRow[item.ColumnName] = DataSchemaTypeConverter.ConvertTo(Math.Round(rates[k] * conversion_rate, 4), item.DataType);

                                break;

                            }

                        default:

                            {

                                break;

                            }

                    }                    

                }
                dt.Rows.Add(newRow);
            }
            return dt;

        }

        

        public override DataSchema GetDefaultDataSchema()

        {

            DataTable dt = new DataTable("FXRates");

            dt.Columns.Add("Currency", typeof(string));

            dt.Columns.Add("Rate", typeof(double));
            dt.Columns["Currency"].AllowDBNull = false;

            dt.Columns["Currency"].Unique = true;

            dt.Columns["Rate"].AllowDBNull = false;
            return new DataSchema(dt);

        }

        

        public override List<ProviderParameter> GetInitializationParameters()

        {

            List<ProviderParameter> parameters = new List<ProviderParameter>();
            parameters.Add(new ProviderParameter("BaseCurrency", this.BaseCurrency));

            

            return parameters;

        }
        public override void Initialize(List<ProviderParameter> parameters)

        {

            foreach (ProviderParameter p in parameters)

            {                

                switch (p.Name)

                {

                    case "BaseCurrency":

                        {

                            this.BaseCurrency = p.Value;

                            break;

                        }

                    default:

                        {

                            break;

                        }

                }

            }

        }
        #endregion      

        

        private static Dictionary<string, double> GetFXRates()

        {

            Dictionary<string, double> rates = new Dictionary<string, double>();

            XmlReader reader = null;

            string name = null;

            double price = 0;

            try

            {

                reader = new XmlTextReader("http://finance.yahoo.com/webservice/v1/symbols/allcurrencies/quote;currency=true?view=basic&format=xml");
                //Add the Default USD Rate

                rates.Add("USD", 1);
                while (reader.Read())

                {

                    if (reader.NodeType == XmlNodeType.Element)

                    {
                        if (reader.Name.Equals("field", StringComparison.OrdinalIgnoreCase))

                        {

                            if (reader.GetAttribute("name") == "name")

                            {

                                string ccy = reader.ReadElementString();

                                if (ccy.StartsWith("USD/"))

                                    name = ccy.Substring(4);

                            }
                            if (reader.GetAttribute("name") == "price")

                            {

                                price = Convert.ToDouble(reader.ReadElementString());

                            }
                            if (name != null && price != 0)

                            {

                                rates.Add(name, price);
                                name = null;

                                price = 0;
                            }
                        }
                        if (reader.Name.Equals("resource", StringComparison.OrdinalIgnoreCase))

                        {

                            name = null;

                            price = 0;

                        }

                    }

                }

            }

            finally

            {

                if (reader != null)

                    reader.Close();

            }
            return rates;

        } 

    }

}

Importing FX Rates with Data Sync

info@simego.com (Sean Cleaver) — Tue, 07 Dec 2010 11:01:49 +0100

This is an example of how to build a custom data provider for Data Synchronsiation Studio to import FX (Currency Exchange) rates from the Yahoo rate service.

Once you have this provider you can then Sync/Import these rates daily into many other systems such as SharePoint, MS CRM, SQL Server etc.

Download Data Sync Add-In

FX Rate provider in the List of Providers

FX Rates in Data Sync Preview Window

The Code

The following code shows how to build a read-only data provider from scratch you simply compile the DLL and drop it into the Data Sync install directory. Data Sync will then use refection to load this assembly at runtime.

using System;

using System.Collections.Generic;

using System.ComponentModel;

using System.Data;

using System.Xml;

using Simego.DataSync.Interfaces;

using Simego.DataSync.Providers;

namespace Simego.DataSync.FXRateProvider

    [ProviderInfo(Name = "FX Rates")]

    public class FXRatesDataSourceReader : IDataSourceReader

        [Category("Synchronisation")]

        [Description("Enable ADD operations on Data source.")]

        public bool EnableAdd { get; set; }

        [Category("Synchronisation")]

        [Description("Enable UPDATE operations on Data source.")]

        public bool EnableUpdate { get; set; }

        [Category("Synchronisation")]

        [Description("Enable DELETE operations on Data source.")]

        public bool EnableDelete { get; set; }

        [Category("Connection")]

        [Description("Filename of the Excel file to read.")]

        public string BaseCurrency { get; set; }

        #region IDataSourceReader Members

        [Browsable(false)]

        public DataSchema SchemaMap { get; set; }

        [Browsable(false)]

        public DataSchemaSide Side { get; set; }

        [Browsable(false)]

        public bool ReadWrite { get; set; }

        [Browsable(false)]

        public IDataSourceLog Log { get; set; }

        public FXRatesDataSourceReader()

            ReadWrite = false;

            BaseCurrency = "USD";

        public DataTable GetDataTable()

            //Get a Table to store the results.

            DataTable dt = SchemaMap.GetDataTable();

            dt.TableName = "FXRates";

            //Create a Schema Mapping Helper

            DataSchemaMapping mapping = new DataSchemaMapping(SchemaMap, Side);

            //Get the Rates from Yahoo

            Dictionary<string, double> rates = GetFXRates();

            //Caculate the Conversion Rate from other base currency

            double conversion_rate = 1 / rates[BaseCurrency];

            //Create a Sorted List of Rates

            var rateList = new List<string>();

            foreach (string k in rates.Keys)

                rateList.Add(k);

            rateList.Sort();

            //Populate the Rates Table

            foreach (string k in rateList)

                DataRow newRow = dt.NewRow();

                foreach (DataSchemaItem item in SchemaMap.GetIncludedColumns())

                    string columnName = mapping.MapColumnToDestination(item);

                    switch (columnName)

                        case "Currency":

                                newRow[item.ColumnName] = DataSchemaTypeConverter.ConvertTo(k, item.DataType);

                                break;

                        case "Rate":

                                newRow[item.ColumnName] = DataSchemaTypeConverter.ConvertTo(Math.Round(rates[k] * conversion_rate, 4), item.DataType);

                                break;

                        default:

                                break;

                dt.Rows.Add(newRow);

            return dt;

        private static Dictionary<string, double> GetFXRates()

            Dictionary<string, double> rates = new Dictionary<string, double>();

            XmlReader reader = null;

            string name = null;

            double price = 0;

try

                reader = new XmlTextReader("http://finance.yahoo.com/webservice/v1/symbols/allcurrencies/quote;currency=true?view=basic&format=xml");

                //Add the Default USD Rate

                rates.Add("USD", 1);

                while (reader.Read())

                    if (reader.NodeType == XmlNodeType.Element)

                        if (reader.Name.Equals("field", StringComparison.OrdinalIgnoreCase))

                            if (reader.GetAttribute("name") == "name")

                                string ccy = reader.ReadElementString();

                                if (ccy.StartsWith("USD/"))

                                    name = ccy.Substring(4);

                            if (reader.GetAttribute("name") == "price")

                                price = Convert.ToDouble(reader.ReadElementString());

                            if (name != null && price != 0)

                                rates.Add(name, price);

                                name = null;

                                price = 0;

                        if (reader.Name.Equals("resource", StringComparison.OrdinalIgnoreCase))

                            name = null;

                            price = 0;

            finally

                if (reader != null)

                    reader.Close();

            return rates;

        public DataSchema GetDefaultDataSchema()

            DataTable dt = new DataTable("FXRates");

            dt.Columns.Add("Currency", typeof(string));

            dt.Columns.Add("Rate", typeof(double));

            dt.Columns["Currency"].AllowDBNull = false;

            dt.Columns["Currency"].Unique = true;

            dt.Columns["Rate"].AllowDBNull = false;

            return new DataSchema(dt);

        public string GetMappedSourceColumn(string name)

            return Utility.GetMappedSourceColumn(SchemaMap, Side, name);

        public List<ProviderParameter> GetInitializationParameters()

            List<ProviderParameter> parameters = new List<ProviderParameter>();

            parameters.Add(new ProviderParameter("BaseCurrency", this.BaseCurrency));

            return parameters;

        public void Initialize(DataSchema schema, DataSchemaSide side)

            this.SchemaMap = schema;

            this.Side = side;

        public void Initialize(List<ProviderParameter> parameters)

            foreach (ProviderParameter p in parameters)

                switch (p.Name)

                    case "BaseCurrency":

                            this.BaseCurrency = p.Value;

                            break;

                    default:

                            break;

        public IDataSourceWriter GetWriter()

            return null;

        public virtual byte[] GetBlobData(List<Simego.DataSync.Engine.DataCompareColumnItem> sourceRow)

            throw new NotCompatibleProviderException("This provider is not compatible with File based Synchronisation.");

        public virtual string GetFilePath(List<Simego.DataSync.Engine.DataCompareColumnItem> sourceRow)

            throw new NotCompatibleProviderException("This provider is not compatible with File based Synchronisation.");

        public virtual string GetFileName(List<Simego.DataSync.Engine.DataCompareColumnItem> sourceRow)

            throw new NotCompatibleProviderException("This provider is not compatible with File based Synchronisation.");

        #endregion

Quickly Create Master Detail View on SharePoint WSS

info@simego.com (Sean Cleaver) — Fri, 03 Dec 2010 08:42:11 +0100

This is a quick example of how to create a Master Detail view of your business data in WSS (not just MOSS). Many people are using the BDC to create these views on MOSS but you can do it quickly with Data Sync and out of the box web parts on WSS in about 10 minutes without any changes to your SharePoint server.

Step 1 – Get your data sources in order

This example uses 2 tables from the popular Northwind sample database, Suppliers and Products were going to create a simple Master Detail view where we select the Supplier and filter the Products based on the Supplier. i.e. Filter the SupplierID column in the Products table from the Supplier Table.

Step 2 – Import your Data to SharePoint

Connect Data Sync Studio to your first table or view i.e. Suppliers you will then see a schema map like this.

Now using the SharePoint List Wizard under the Tools Menu create the List on your SharePoint server (You could use existing lists or do this manually however this is much quicker). Select your server and a name for your list.

Now the Schema Map is linked inside Data Sync Studio, you can compare and Sync the Data to get the data into your SharePoint Lists.

Now follow the same procedure for the Products table.

Step 3 – Create your SharePoint Page

Create a new Web Part page and add the 2 lists to the page Suppliers & Products then create a new Web Part Connection between Products and Suppliers.

That’s it so now when you select a Supplier the Product list is filtered based on that selected Supplier.

All that’s left is to setup 2 Synchronisation jobs to sync the source data into your SharePoint lists.

Ouvvi – Service Log

info@simego.com (Sean Cleaver) — Mon, 04 Oct 2010 16:01:58 +0100

We have just rolled out an update to Ouvvi 1.2.32 to help those who use Ouvvi to integrate systems together and build custom step handlers.

You can now run the service component interactively in-process with your handlers which is easier than attaching to the Ouvvi service.

All log events (including Trace level regardless of the System Setting) are displayed in the log window and anything you write to the .NET Trace log are also captured. Additionally you can force the service out of sleep by waking up the “Project” or “Step” queues.

Data Synchronisation Studio Update

info@simego.com (Sean Cleaver) — Mon, 20 Sep 2010 10:42:51 +0100

We’ve rolled out an update to Data Synchronisation Studio 2.0 that now includes a Start Page which lists your most recent projects along with a Statistics page on the results to return some metrics about the compare process.

We have also added a multi-threaded project loader so you can see the project loading which helps with data providers that take a little while to connect to the data source.

Additionally we now save previous settings on some of the data providers to speed up connecting to your data source where you always enter the same server name.

This update is on the Beta Site now and is available for all customers that have an active subscription.

Service Desk – Updated Screenshots

info@simego.com (Sean Cleaver) — Tue, 06 Jul 2010 11:47:42 +0100

Here's some updated screenshots from our new Service Desk Solution.

The Dashboard displays basic Stats and allows you to filter the Ticket List by the Statistic. You will also see the Tag cloud where we Tag requests with meta data when they are created. This process is handled externally via a simple script which can be on any server so in our case we validate inbound messages against registered users in our Simego Database and Tag the message with the Product in question.

The Ticket List allows you to easily and quickly view summary information on each ticket before opening it.

When an Email message is received into the service desk the attachments are extracted, compressed and inserted into the database and then are available to view within the Service Request.

You simply reply inline to the customer from the web page.

Ouvvi handles pulling in the email messages from a mailbox and forwards the RAW Mime Message to the Service Desk. To speed things up Ouvvi creates a new Ouvvi Project Task for each message, that way we can import multiple messages at once across multiple threads and each message has it’s own re-try pattern to follow.

New Service Desk Application

info@simego.com (Sean Cleaver) — Mon, 21 Jun 2010 15:16:59 +0100

Here’s a little preview of a new Service Desk (aka Help Desk) solution that were working on.

ASP.NET MVC Service Desk Solution
SQL Server 2008 Database
User Authentication via OpenID, Windows, Custom Authentication
Integration with Ouvvi – to Pull Requests from Mailboxes and add to the Service Desk
Rules/Classification Engine - To tag service requests with MetaData and create Tag Clouds
Time based Escalation process
OData – R/W for API Access, PowerPivot, SQL Server Reporting Services

Dashboard

Instantly view the Status of the Service Desk and see what needs be be acted on quickly.

List of Open Tickets

Integrating multiple Systems with Ouvvi as a Message Broker

info@simego.com (Sean Cleaver) — Fri, 11 Jun 2010 11:27:16 +0100

Here’s a high level overview of how you can utilise the Ouvvi message processing pipeline to integrate multiple systems, transform messages, validate, decorate etc.

Using Ouvvi to handle this kind of operation has a number of benefits

Automatic Message re-try to handle those times when things don’t work first time.
Scale-out across servers or have stand-by servers ready to take over.
Separation of responsibility, Ouvvi projects run under their own security context.
Re-use coded steps across multiple projects to build new functionality dynamically.
Multi-threaded, run multiple processing threads per Ouvvi server.
Pause the processing pipe-line by stopping the Ouvvi service with no-loss of messages.
Simple Backup as all messages are stored in SQL Server Queue and can be easily restored or moved to a new server.
Simple programming interface, easily develop new steps, run service code in-process with Visual Studio to step through entire pipeline.
Unified logging, logs can be tracked back to individual message instance through the system.
User interface to configure, view and track operation.

Automated SVN Backup via Ouvvi

info@simego.com (Sean Cleaver) — Fri, 11 Jun 2010 09:56:43 +0100

Here’s an example of how to automate backup of an SVN repository on Windows with Ouvvi. We used to backup on each check-in but after a while that just got too much and we ended up with several GB of backup images on our server and had to periodically delete them. It was also slowing down the check-in so now we just to it every Sunday morning.

We have setup an end-of-week schedule to back up all our SVN repositories like this.

Each Task is configured like this to call the svnadmin.exe

svnadmin.exe hotcopy "C:\Repositories\xxxx" "C:\Backup\Svn\xxxx\{{FILEDATE}}"

Building a Web Hosting Solution with Ouvvi

info@simego.com (Sean Cleaver) — Thu, 10 Jun 2010 10:57:56 +0100

Ouvvi is far more than just a plain web based scheduling system. Recently we have been playing around creating a new solution to automatically provision web hosting space, domain mappings, SQL Databases etc.

This came about because I wanted to create an easier way to automatically configure new Hosting space, SQL Databases etc which is virtually instant. After try a few web hosts and finding these actions can take several hours to complete. This way I can easily manage multiple sites on our own dedicated servers.

This was also a Test to see if Ouvvi can successfully handle this kind of integration with 3rd Party applications.

This solution involves a ASP.NET MVC website for all the site configuration where you add web space accounts and delegate to Ouvvi on the back-end to handle asynchronous messages to create the actual space under IIS.

For Example we send this message to Ouvvi via the Ouvvi Web Service to Create a new Site

This message includes a return Url to notify the client application about the result.

The Response message is something like this

<ActionResponse Function="CreateSite">
<Result>
    <SiteCode>s101234</SiteCode>
    <TempUrl>http://s101234.simegohosting.com</TempUrl>
    <IPAddress>79.123.123.67</IPAddress>
    <SpaceAllocated>1024</SpaceAllocated>
    <SpaceUsed>0</SpaceUsed>
    <PhysicalPath>c:\sites\www\s101234</PhysicalPath>
    <Status>0</Status>
</Result>
</ActionResponse>

These messages are processed via the Ouvvi Pipeline and passed between steps via the Message Context. The message Context in Ouvvi allows you to pass metadata to your Steps to action.

Below shows the Ouvvi Project configuration we have 2 steps.

Step 1 – Handles all the real work and calls a custom handler to create sites etc from the Message stored in Context.
Step 2 – Handles the call-back to the client application. (We break it into 2 steps so that the automatic re-try a failure process is handled.)

Now we have a fully integrated solution with a visual feedback of status on the back-end and even if the server is busy Ouvvi will process the configuration steps later without impacting the UI to the client.

The best bit – This only took us a few days to get up and running.

My Experiments with OAuth

info@simego.com (Sean Cleaver) — Mon, 07 Jun 2010 14:25:18 +0100

I have been playing around with OAuth from C# and after researching available code/samples/libraries on the internet none seemed to be quite as neat and tidy as I would like. So I decided to write my own plus if I am going to put this into some new products were working on we needed to write our own implementation.

We now have a simple API where we setup a Configuration Object with details of the service were connecting to, Urls, Keys etc. Then just Call Authorize this handles all the authorization bouncing around (we call back on an interface IAuthControl to persist data between requests).

From then on we have a method GetServiceResponse which handles interaction with the service provider and signing the requests.

Here’s an example of Sending a Tweet to Twitter and Authorizing the application.

TwitterExample.cs

namespace oAuthWeb
{
    public class TwitterExample
    {
        const string REQUEST_TOKEN = "http://api.twitter.com/oauth/request_token";
        const string AUTHORIZE = "http://api.twitter.com/oauth/authorize";
        const string ACCESS_TOKEN = "http://api.twitter.com/oauth/access_token";

        public string Execute(HttpContext httpContext, IAuthControl control)
        {
            AuthConfiguration config = new AuthConfiguration
            {
                ConsumerKey = "xxxxxx",
                ConsumerSecret = "xxxxxxxxxx",

                RequestTokenUrl = REQUEST_TOKEN,
                RequestAuthorizeUrl = AUTHORIZE,
                RequestAccessTokenUrl = ACCESS_TOKEN,

CallbackUrl = http://test.oauth.com,

                TokenKey = httpContext.Session["OAuth_TokenKey"] as string,
                TokenSecret = httpContext.Session["OAuth_TokenSecret"] as string,
                State = (AuthState)Enum.Parse(typeof(AuthState), httpContext.Session["OAuth_State"] as string ?? AuthState.None.ToString())
            };

            if (Auth.Authorize(config, control))
            {
                AuthResponse response = Auth.GetServiceResponse(config,
                    new AuthRequest
                    {
                        Method = AuthRequestMethod.POST,
                        Url = "http://api.twitter.com/1/statuses/update.xml",
                        Body = "status=" + AuthHelper.UrlEncode("Test Tweet from my C# Code"),
                        ContentType = Auth.ContentType_ApplicationFormUrlEncoded
                    });

                if (response.Status == HttpStatusCode.OK)
                {
                    return response.Body;
                }
            }

            return null;
        }
    }
}

Default.aspx.cs

public partial class _Default : System.Web.UI.Page, IAuthControl
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            TwitterExample example = new TwitterExample();
            content.InnerHtml = Server.HtmlEncode(example.Execute(HttpContext.Current, this));
        }

#region IAuthControl Members

        public void RedirectTo(string url)
        {
            Response.Redirect(url);
        }

        public void SaveAuthState(AuthConfiguration config)
        {
            Session["OAuth_TokenKey"] = config.TokenKey;
            Session["OAuth_TokenSecret"] = config.TokenSecret;
            Session["OAuth_State"] = config.State.ToString();
        }

        public string GetRequestValue(string key)
        {
            return Request[key];
        }

#endregion
}

Data Synchronisation Studio – Obfuscate Options

info@simego.com (Sean Cleaver) — Tue, 25 May 2010 16:37:52 +0100

We’ve added a new Obfuscate Option to Data Sync so that it now has a simple mode where the text is replaced with the first character matching case and spaces etc. So you get results like this below.

Ouvvi – Uptime

info@simego.com (Sean Cleaver) — Tue, 25 May 2010 16:24:21 +0100

We have been running the latest build of Ouvvi 1.1.20 now for 22 straight days running jobs every 15 minutes and backup our databases every day.

Obfuscate Data with Data Synchronisation Studio

info@simego.com (Sean Cleaver) — Tue, 04 May 2010 20:29:41 +0100

Occasionally we need to obfuscate sensitive data whether it’s to create a test version of a production database for development or to send sensitive data outside of an organisation.

We have created a new function in Data Synchronisation Studio to Obfuscate data during the synchronisation from the client application.

To enable the Obfuscate function just select “Obfuscate” from the Data menu.

Now when you Compare A to B you will be presented with an Obfuscate Options dialog where you can then choose which String based columns to Obfuscate and whether to include a Salt value with the hash generation.

Below shows the results after the compare here you can see that the ProductName column is going to be updated with an Obfuscated value based on the hash of the original value and the Primary Key. The process is reasonably fast on one of our development machines it takes less than a second to process 4000 records.

We can simply synchronise now and our target database will be updated with the Obfuscated data.

Ouvvi Handler Changes

info@simego.com (Sean Cleaver) — Fri, 09 Apr 2010 10:45:58 +0100

Were making some changes to Ouvvi to make is far simpler to write new Handlers for the message pipeline. Before you had to write the Handler, Serialization Code and UI code. Now we have streamlined all this so you just write the handler and decorate the Data Model to describe how to render the configuration UI. The serialization is handled automatically via reflection. This handler then just needs to be registered in the database and it’s good to go.

So a very simple handler now looks like this

public class DirectoryCreateStepData
    {
        [Designer(ControlType = DesignerControlType.TextBox, Description = "Path", RequiredField = true)]
        public string Path { get; set; }
    }

    [Designer(Description = "Creates a Directory")]
    public class DirectoryCreateStepHandler : StepHandler<DirectoryCreateStepData>
    {
        public override void Execute(DirectoryCreateStepData data)
        {
            //Create Directory
            if (!Directory.Exists(data.Path))
                Directory.CreateDirectory(data.Path);

}
}

Now because the UI is dynamically rendered from the MetaData were writing an iPhone UI as well, which is great if your stuck down the Pub and need to quickly check that your end of day processes have all completed correctly.

Data Sync Consuming OData Feeds

info@simego.com (Sean Cleaver) — Thu, 25 Mar 2010 11:52:02 +0100

Were writing a new Data Provider for Data Sync that can consume an OData Feed for example below were pulling in the Speakers data from the recent MIX event using the published OData feed http://api.visitmix.com/OData.svc/

This makes importing and syncing changes from public OData feeds a real snap, point Data Sync at an OData Feed and then Synchronise it to your destination system. Even Sync it into a SharePoint List and setup some workflows to handle data changing events or let your users subscribe to the data.

SharePoint: Import CSV/ Excel Data File

info@simego.com (Sean Cleaver) — Wed, 10 Mar 2010 16:46:52 +0100

This example shows how you can use Data Synchronisation Studio to import data in a CSV file directly to a SharePoint list. Without making any configuration changes to your SharePoint environment, no web parts to install so no issues with your SharePoint Admins. You do not need to run this on your SharePoint Servers either and can even be used with Shared Hosting installs of SharePoint such as Microsoft Online Services.

First get a copy 15 day free trial of Data Sync from http://www.simego.com

Connect Data Sync to your CSV Data Source

You will now have some thing setup like this

What’s needed now is a destination list in SharePoint so you can either create one manually or use Data Sync to create one for you.

First adjust the schema map to match the Data Types were working with like this

Now go to the Tools menu and Choose Create SharePoint List and Step through the Wizard.

Select the Site to Create the List

Give the List a Name

Now Data Sync has this list mapped like this which needs a little adjustment, we need to change the “Key Column” back to “EntryID” and adjust the Data Types.

So the Schema Map should look like this

Now Compare the Data A->B to see what rows should be copied to SharePoint (i.e. all of them).

Next click the “Synchronise” toolbar button and click Start to copy the data across.

Here we have our data in SharePoint

You can then save this import as a Data Sync project and re-run it when you like and only the changes will be copied to SharePoint.

Data Synchronisation Studio 2.0 – Use your own SQL Queries

info@simego.com (Sean Cleaver) — Wed, 10 Mar 2010 15:21:48 +0100

In Data Synchronisation Studio 2.0 we have added the ability for you to connect SQL Data sources to your own custom queries rather than just Tables and Views.

If you use this feature the Data Source will be read-only but since this feature is probably only going to be used for pushing report type data up to SharePoint it’s one-way anyway.

When you connect the data source you can choose the “SQL” tab like this and enter any SQL SELECT Statement.

You can also tweak it if you like from the Data Source properties window like this

This feature has been added to the following data providers for Data Sync

Microsoft SQL Server
OLEDB Data sources
ODBC Data sources
Oracle
MySQL
Microsoft Access

Locally Run Ouvvi without IIS

info@simego.com (Sean Cleaver) — Tue, 09 Mar 2010 13:22:44 +0100

So if you just have a development machine (no access to a server) and want to run the Ouvvi Web Console without IIS you can use the Visual Studio Development Server instead.

Install Ouvvi as normal but skip the last configuration step at the end of the install.

Then you need to start the Development Web Server like this.

"C:\Program Files (x86)\Common Files\microsoft shared\DevServer\9.0\WebDev.WebServer.EXE" /port:4040 /path:"C:\Program Files (x86)\Simego\Ouvvi Website\Web"

Make sure you choose the right TCP port here I am using 4040.

Then you can finish the Ouvvi configuration however I had to re-start the server after provisioning the database as it didn’t pick up the web.config file changes but after that it was fine.

Remember Ouvvi only requires the web console for setup, configuration and status reporting the actual project/step processing occurs via the Ouvvi Service so you don’t need to run the Web console all the time.

Synchronising Data from a Simple Http Service

info@simego.com (Sean Cleaver) — Tue, 09 Mar 2010 10:32:28 +0100

We have created a new Data Provider for Data Synchronisation Studio 2.0 which accepts a CSV formatted response from a Http Stream.

This means you can easily integrate any system in which you can expose data as a Http response formatted as CSV with SharePoint or any of the other data providers which support writing back to the data store.

For example if you create a simple service that returns a response similar to this

You can then load this in Data Sync 2.0 by specifying the Uri to the Http response like this using the new CSV File/Http File Data Provider.

This can then use this as the source data in Data Sync to be synchronised to SharePoint.

You can download a simple example below that demonstrates how to create a Simple CSV Http response or Simple SOAP 1.1 Web Service to use with Data Sync.

Download Sample Code

Backup your SQL Express Databases for Free

info@simego.com (Sean Cleaver) — Fri, 26 Feb 2010 17:20:17 +0100

Did you know with Ouvvi Express edition you can easily schedule a SQL Backup of your departmental databases and compress them on a daily basis.

The setup is so simple and only takes a few minutes to get going, and as you create more databases all you need do is login to the web console and add more steps to the backup schedule.

You can see here every day were backing up 4 SQL databases in our Office.

To configure the step for the actual backup is quite easy you just create a SQL Backup Step and configure it like this.

This gives us a daily backup file like this on our server

You can then add notifications to the pipeline if the process fails and keep an eye on things via the RSS Feed.

Data Synchronisation Studio 2.0 and Ouvvi Beta out

info@simego.com (Sean Cleaver) — Thu, 25 Feb 2010 09:36:11 +0100

We have released to the Simego Website new Beta installs for the following :-

Data Synchronisation Studio 2.0

Supports Ouvvi
Side by Side install with Data Sync 1.0
UI Improvements
Select which rows to sync from comparison results
All fixes since 1.0.188 rolled up into 2.0
Experimental support for SharePoint 2010
Strong named assemblies
GAC installed main assemblies.

Available here www.simego.com/beta

DS 2.0 will be a free upgrade to all those customers who purchased data sync in the last 12 months. Outside of this there will be a small upgrade fee to extend your support and maintenance for a further 12 months.

Ouvvi Express 1.0

Free Web based Process Automation and scheduler system, supports backup of SQL Server Databases, Running SQL Commands, Starting SQL Jobs, Pinging Web Sites, Running Data Sync Projects, Email, SMS, RSS etc

See http://www.simego.com/Products/Ouvvi-Console

Install guide here Ouvvi Install & Setup

Data Synchronisation Studio 2.0

info@simego.com (Sean Cleaver) — Wed, 24 Feb 2010 09:11:33 +0100

We have a new feature for Data Synchronisation Studio 2.0 which we think you might like when you’re trying to control better which data to migrate between systems. You can see in the screenshot below we have added a new “Sync” column to the compare results this allows you to specify which rows to actually sync across to the destination system.

You can also now preview the data from each side as the schema map by clicking on the “Preview A” or “Preview B” mini-toolbar buttons. This allows you to see the data exactly as Data Sync will see it when it compares the data. It also prevents downloading all columns from the source system which can speed things up a little.

Other additions to the Schema Map include :-

Double Click Column in data source to add it to the Map.
Multi-select Schema Map Rows and delete with one click.

Also Data Synchronisation Studio 2.0 can be installed side-by-side with DS 1.0 and Data Sync 2.0 integrates with Ouvvi.

Using Ouvvi to Schedule SQL Database Backup

info@simego.com (Sean Cleaver) — Sun, 21 Feb 2010 13:31:14 +0100

We have just added a new Step Handler that runs a backup of a SQL Server Database and additionally you can ZIP up the backup image to save on disk space.

Below is a screenshot of the Step configuration and you can see here were using the Token expansion system for the backup filename which makes it easy to create a unique filename.

Tokens

This is a list of the static token values that we have added to Ouvvi

INSTANCEID
PROJECTID
DAY
MONTH
YEAR
FILEDATE
DATE
FILETIME
TIME
HOUR
MIN
SEC
MACHINE
USERNAME
DOMAIN
APPPATH
PERSONAL
SYSTEM
LOCALAPPLICATIONDATA
APPLICATIONDATA
COMMONAPPLICATIONDATA
PROGRAMFILES
TEMP
NEWGUID

Using the Ouvvi Status Feed

info@simego.com (Sean Cleaver) — Tue, 09 Feb 2010 11:46:47 +0100

So in Ouvvi we have created an RSS/Atom feed that returns the current status of all the projects in the console. This has a variety of uses including the ability to advertise to other business units that everything is all running correctly.

For example I am consuming the Status Feed here in Outlook and it allows me to see that the jobs are all running and I get an update each time it’s run.

Ouvvi – Running for another Week

info@simego.com (Sean Cleaver) — Mon, 08 Feb 2010 09:53:57 +0100

So we left the current build of Ouvvi running all last week (since 29th Jan) on our Server processing it’s jobs and it’s just running perfectly, no increase in Memory usage, low CPU usage and all services still running perfectly!

New Ouvvi Step Types

info@simego.com (Sean Cleaver) — Tue, 02 Feb 2010 10:28:25 +0100

We’ve added a few new Step Handlers to Ouvvi to extend the integration capabilities.

HttpWebRequest – Allows you to call an external web page and validate the result with a Regular Expression. This can simply be used to check that a remote service is running or used to notify an external service that an Ouvvi project has completed.

SQL Job – Allows you to Start a Job in SQL Server Agent typically this might be used to start an SSIS Package.

SMS Message – Like the Email Message handler you can also send SMS’s via the www.csoft.co.uk SMS Gateway Service.

Ouvvi Project – You can Start another Ouvvi Project.

Testing Ouvvi Uptime

info@simego.com (Sean Cleaver) — Tue, 02 Feb 2010 10:21:00 +0100

So were running Ouvvi internally here on our server to check the reliability of leaving the service completely unattended we have been running for several days now and all is working very well. We have two projects setup one that backs up our database from the Internet on Start of Day and End of Day, the other runs every 30 minutes and synchronises our Exchange Timesheet data with SQL and SharePoint.

Ouvvi Service(s) List here were running 3 instances of the Step Service on the same Server and you can see that the last service checkpoint was a few seconds ago and it’s been running since 29th Jan.

Task Manager is showing that Memory usage has not climbed up at all even when running several data sync projects every 30 minutes.

And CPU usage is pretty much flat which is really good.

Server Performance Running Ouvvi

info@simego.com (Sean Cleaver) — Tue, 26 Jan 2010 14:33:04 +0100

Were quite keen that Ouvvi can be deployed onto existing infrastructure and not impact normal server operations. In fact were running Ouvvi on our DC which is also running SQL Server, SharePoint and Team Foundation Server.

Below shows a Server CPU Graph running the Ouvvi Service. Here you can see that running all the Ouvvi Services in the background doesn’t impact server CPU load also the actual Memory usage is quite low at 16MB this value will increase depending on the size of the Data Sync projects your running but should drop down to around this figure when Ouvvi is idle.

Using Ouvvi to Schedule and Manage your Data Sync ETL

info@simego.com (Sean Cleaver) — Tue, 26 Jan 2010 12:45:38 +0100

Here’s a quick overview of how we are using Ouvvi internally to manage our Data Sync projects.

We have a series of Steps that basically download our Timesheet Data from Exchange Server which is then synchronised into a SQL Table. Then we create a View that returns statistics about the time spent on each project which is Synchronised into a SharePoint List with Data Sync.

So this project consists of the following Steps

Get Exchange Records
Update Job Codes (2 way sync of SharePoint List containing Job Codes)
Update Job Stats
Email support if there was an error.

In Ouvvi this is really easy to setup we just create a new project and add the steps like below.

Adding a Data Sync Step is easy too we just enter the path to the Data Sync Project like so.

We want this process to run at “Start of Day” and every 30 minutes so we just add the Triggers for “Start of Day” and 30 Minutes.

For the Failure report we just add a SMTP Step which only runs on the Run Condition “Failure” like so.

And that’s it all setup and running every day, logging the status to the Ouvvi logs and reporting any error to our HelpDesk. (Ouvvi has an automatic retry pattern so we need 3 failures in a row to initiate an actual error).

Anytime we need to kick this process off again Ad-Hoc we can just hit the Start button without having to log onto the server.

Coming Soon – Ouvvi for DataSync / ETL

info@simego.com (Sean Cleaver) — Wed, 13 Jan 2010 10:52:33 +0100

So we have been busy putting a new solution together to better manage Data Sync projects and ETL along with any solution that requires a series of steps to be followed. This could be an STP system, Order Processing, Data Import/Extract or anything really.

The solution involves a Web Front End application for Management and a Windows Service that handles the actual processing.

The system involves creating a project and adding steps to the project which are processed in turn, each step has a run condition which allows for example running a step when the batch fails such as a step to send an email/sms alert.

Projects and Steps are added to a Queue for processing to ensure that the system remains consistent between failures, re-starts etc and can be easily backed up via SQL Server backup.

Features

Scale-out across Processes/Servers/Machines for both Web Application and Service processing.
Resilient against failures via Retry Pattern and Queue System to maintain consistency.
Redundancy/Failover via running multiple services on separate machines.
Unified Logging where all messages from all projects/steps are captured into a single logging table
Logging clean up where expired logs are deleted (could also be archived)
Queue clean up where failed/unknown state messages are removed after retry count fails.
Start of Day/End of Day Triggers
Easily Develop new Step Handlers for integration of any system i.e. Execute SSIS Package, Run SP, BCP, Backup Database, FTP etc.
Easily Develop new Triggers such as File Drop, Web Service etc.
RSS Status Reports
Content Management to enable Project/Step documentation within the console.

Prototype Step Handler

Creating new Steps is as simple as creating a class that derives from StepHandler like so.

public class SqlStepHandler : StepHandler
    {
        public override void Execute()
        {
            LogService.LogEvent(InstanceID, ProjectID, StepIndex, LogSeverity.Information, "SqlStepHandler.Execute", "Start", this.Name);
        }
    }

Screenshot

System Requirements

We have chosen not to use the very latest software to build this solution so that the requirements are pretty minimal and therefore could be installed easily onto existing infrastructure or virtualized with Hyper-V or VMWare.

For the Web Application

Windows Server 2003+, Server 2008, Vista, 7
IIS 6.0 +
.NET Framework 3.5 SP1
SQL Server 2005+ (and Express)

For the Processing Service

Windows Server 2003+, Server 2008, Vista, 7
.NET Framework 2.0+
SQL Server 2005+ (and Express)

Accessing Word Document Properties via Office11 Interop Library

info@simego.com (Sean Cleaver) — Mon, 07 Dec 2009 20:26:48 +0100

To access Word Document Properties with the Office 2003 Interop Libraries for the .NET Framework you need to use reflection to access the collection and read the Name and Value properties of each item in the List.

For example to Return a List of Document Properties might do something like this.

Code Snippet
 public ICollection<DocumentProperty> GetDocumentProperties()
 {
     ICollection<DocumentProperty> returnValue = new List<DocumentProperty>();
 
     IEnumerable p = (IEnumerable)_Document.CustomDocumentProperties;
 
     foreach (object f in p)
     {
         Type item = f.GetType();
         try
         {
             returnValue.Add(new DocumentProperty
             {
                 Name = GetProperty<string>(item, f, "Name"),
                 Value = GetProperty<object>(item, f, "Value")
             });
         }
         catch (TargetInvocationException)
         {
             //Do nothing
         }
         catch (COMException)         
         {
             //Do nothing
         }
     }
 
     return returnValue;
 }
 
 private T GetProperty<T>(Type t, object o, string name)
 {
     return (T)t.InvokeMember(name,
                        BindingFlags.Default |
                        BindingFlags.GetProperty,
                        null, o,
                        null);
 }
 

Creating PDF Documents with PDF Creator

info@simego.com (Sean Cleaver) — Mon, 07 Dec 2009 17:41:59 +0100

This sample shows how you can create a command line utility program to automatically convert Word (.doc) documents into a PDF file with the PDF Creator Library.

Code Snippet
 using System;
using System.IO;
using System.Threading;
using Microsoft.Office.Interop.Word;
using PDFCreator;
 
namespace PdfPrinter
{
    class Program
    {
        static int Main(string[] args)
        {
            WordConstants Constants = new WordConstants();
 
            string PDF_Printer = "PDFCreator";
            
            string infile = null;
            string outfile = null;
 
            Console.WriteLine("Creating PDF Document");
 
            string currentArg = "";
            foreach (string arg in args)
            {
                if (arg.Equals("-infile:", StringComparison.OrdinalIgnoreCase) ||
                    arg.Equals("-outfile:", StringComparison.OrdinalIgnoreCase) ||
                    arg.Equals("-printer:", StringComparison.OrdinalIgnoreCase)
                    )
                {
                    currentArg = arg.ToLower();
                }
                else
                {
                    switch (currentArg)
                    {
                        case "-infile:":
                            {
                                infile = arg;
                                currentArg = null;
                                break;
                            }
                        case "-outfile:":
                            {
                                outfile = arg;
                                currentArg = null;
                                break;
                            }
                        case "-printer:":
                            {
                                PDF_Printer = arg;
                                currentArg = null;
                                break;
                            }
 
                        default:
                            {
                                break;
                            }
                    }
                }
            }
 
            //Verify the input file exists
            if (string.IsNullOrEmpty(infile) ||
                !File.Exists(infile))
            {
                Console.WriteLine("Error - Input file '{0}' does not exist.", infile);
                return -1;
            }
 
            if (string.IsNullOrEmpty(outfile))
            {
                Console.WriteLine("Error - No output file.");
                return -1;            
            }
 
            //Check that the output directory exists
            if (!Directory.Exists(Path.GetDirectoryName(outfile)))
            {
                Directory.CreateDirectory(Path.GetDirectoryName(outfile));
            }
 
            clsPDFCreator pdfCreator = null;
            try
            {
                pdfCreator = new clsPDFCreator();
 
                if (!pdfCreator.cStart("/NoProcessingAtStartup", false))
                {
                    Console.WriteLine("Unable to start PDFCreator.");
                    return -1;
                }
 
                pdfCreator.cWindowState = 1;
 
                // Save currently active printer.
                string defaultPrinter = pdfCreator.cDefaultPrinter;
 
                // Set parameters for saving the generating pdf automatically to a directory.
                clsPDFCreatorOptions pdfPrintOptions = pdfCreator.cOptions;
                // Use auto save functionality.
                pdfPrintOptions.UseAutosave = 1;
                // Use directory for saving the file.
                pdfPrintOptions.UseAutosaveDirectory = 1;
                // Name of the output directory.
                pdfPrintOptions.AutosaveDirectory = Path.GetDirectoryName(outfile);
                // Format in which file is to be saved. 0 if for pdf.
                pdfPrintOptions.AutosaveFormat = 0;
                // Name of the output file name.
                pdfPrintOptions.AutosaveFilename = Path.GetFileName(outfile);
                //Set the Options
                pdfCreator.cOptions = pdfPrintOptions;
                //Clear the Cache
                pdfCreator.cClearCache();
                //Stop the Printer
                pdfCreator.cPrinterStop = true;
 
                Console.WriteLine("Starting Word");
                // Create new instance of word application.
                ApplicationClass word = new ApplicationClass();
                // Set pdf creator as active printer. Name should be same as you gave while installation.
                try
                {
                    //Set the PDF Printer Current
                    word.ActivePrinter = PDF_Printer;
 
                    Object documentName = infile;
                    //Open the Document
                    Document document = word.Documents.Open(ref documentName, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue, ref Constants.MissingValue);
                    
                    Console.WriteLine("Printing PDF Document");
                    //Print the document to the PDF Printer
                    word.PrintOut(ref Constants.background, ref Constants.append, ref Constants.range, ref Constants.outputFileName, ref Constants.from, ref Constants.to, ref Constants.item, ref Constants.copies, ref Constants.pages, ref Constants.pageType, ref Constants.printToFile, ref Constants.collate, ref Constants.fileName, ref Constants.activePrinterMacGX, ref Constants.manualDuplexPrint, ref Constants.printZoomColumn, ref Constants.printZoomRow, ref Constants.printZoomPaperWidth, ref Constants.printZoomPaperHeight);
                    // Wait untill the document is queued.
                    while (pdfCreator.cCountOfPrintjobs != 1)
                    {
                        Thread.Sleep(100);
                    }
                    // Start the printing.
                    pdfCreator.cPrinterStop = false;
                    // Wait until the print queue is completed.
                    while (pdfCreator.cCountOfPrintjobs != 0)
                    {
                        Thread.Sleep(100);
                    }
                    // Close all the opened documents.
                    foreach (Document doc in word.Documents)
                    {
                        doc.Close(ref Constants.FalseValue, ref Constants.MissingValue, ref Constants.MissingValue);
                    }
 
                    // Stop the printer.
                    pdfCreator.cPrinterStop = true;
 
                }
                finally
                {
                    // Set back the default printer.
                    word.ActivePrinter = defaultPrinter;
                    word.Quit(ref Constants.FalseValue, ref Constants.MissingValue, ref Constants.MissingValue);
                }                
                
                Console.WriteLine("PDF Document Created {0}", pdfCreator.cOutputFilename);
 
            }
            finally
            {
                // Shutdown the PDF Printer
                if (pdfCreator != null)
                {
                    pdfCreator.cClose();
                    pdfCreator = null;
                }
            }
 
            return 0;
        }
    }
 
    class WordConstants
    {
        public Object MissingValue = Type.Missing;
        public Object FalseValue = false;
        public Object background = true;
        public Object append = true;
        public Object range = Microsoft.Office.Interop.Word.WdPrintOutRange.wdPrintAllDocument;
        public Object outputFileName = Type.Missing;
        public Object from = Type.Missing;
        public Object to = Type.Missing;
        public Object item = Microsoft.Office.Interop.Word.WdPrintOutItem.wdPrintDocumentContent;
        public Object copies = 1;
        public Object pages = Type.Missing;
        public Object pageType = Microsoft.Office.Interop.Word.WdPrintOutPages.wdPrintAllPages;
        public Object printToFile = false;
        public Object collate = Type.Missing;
        public Object fileName = Type.Missing;
        public Object activePrinterMacGX = Type.Missing;
        public Object manualDuplexPrint = Type.Missing;
        public Object printZoomColumn = Type.Missing;
        public Object printZoomRow = Type.Missing;
        public Object printZoomPaperWidth = Type.Missing;
        public Object printZoomPaperHeight = Type.Missing;
 
    }
}
 

Sample: Dynamically Creating Code at Runtime and Calling from your Program

info@simego.com (Sean Cleaver) — Mon, 07 Dec 2009 10:16:55 +0100

This is a small sample that shows how you can build code at runtime and generate an assembly to execute code which could be defined in a database or something. This example shows how to dynamically create a class that add the values of 2 numbers via the expression =Convert.ToInt32(a) + Convert.ToInt32(b).

using System;

using System.CodeDom.Compiler;

using System.Collections.Generic;

using System.Globalization;

using System.Reflection;

using System.Text;

using Microsoft.CSharp;

namespace ExpressionTest

  class Program

    static void Main(string[] args)

      List<KeyValuePair<string, string>> properties = new List<KeyValuePair<string, string>>();

      properties.Add(new KeyValuePair<string, string>("a", "1"));

      properties.Add(new KeyValuePair<string, string>("b", "200"));

      string expression = "=Convert.ToInt32(a) + Convert.ToInt32(b)";

      string className = "Expression1";

      string methodName = "GetSum";

      string namespaceName = "ExpressionTest";

      //Create a Class

      StringBuilder _class = new StringBuilder();

      //Start Usings

      _class.Append("using System;\r\n");

      //Start Namespace

      _class.AppendFormat("namespace {0} {{\r\n", namespaceName);

      //Start Class

      _class.AppendFormat("public class {0} {{\r\n", className);

      //Start Properties

      foreach (KeyValuePair<string, string> p in properties)

        _class.AppendFormat("public string {0};\r\n", p.Key);

      //Start Constructor

      _class.AppendFormat("public {0} () {{\r\n", className);

      foreach (KeyValuePair<string, string> p in properties)

        _class.AppendFormat("{0}=\"{1}\";\r\n", p.Key, p.Value);

      //End Constructor

      _class.Append("}\r\n");

      //Start Expression

      _class.AppendFormat("public string {0}() {{\r\n", methodName);

      if (expression.StartsWith("="))

        _class.AppendFormat("return Convert.ToString({0});\r\n", expression.Substring(1));

      //End Expression

      _class.Append("}\r\n");

      //End Class

      _class.Append("}\r\n");

      //End Namespace

      _class.Append("}\r\n");

      //Compile the Class

      CompilerParameters options = new CompilerParameters();

      options.GenerateInMemory = true;

      options.GenerateExecutable = false;

      foreach (AssemblyName assembly in typeof(Program).Assembly.GetReferencedAssemblies())

        options.ReferencedAssemblies.Add(string.Format("{0}.dll", assembly.Name));

      CSharpCodeProvider provider = new CSharpCodeProvider();

      CompilerResults results = provider.CompileAssemblyFromSource(options, _class.ToString());

      if (!results.Errors.HasErrors)

        //Invoke the Expression Method

        Type t = results.CompiledAssembly.GetType(string.Format("{0}.{1}", namespaceName, className));

        object instance = t.InvokeMember(t.Name,

          BindingFlags.CreateInstance |

          BindingFlags.Instance |

          BindingFlags.NonPublic |

          BindingFlags.Public, null, null, null, CultureInfo.CurrentCulture);

        if (instance != null)

          string result = (string)t.InvokeMember(methodName,

            BindingFlags.InvokeMethod |

            BindingFlags.Instance |

            BindingFlags.Public |

            BindingFlags.NonPublic, null, instance, null, CultureInfo.CurrentCulture);

          Console.WriteLine("{0}{1}", result, expression);

      Console.ReadLine();

Previewing SharePoint 2010 Foundation

info@simego.com (Sean Cleaver) — Sat, 05 Dec 2009 20:36:25 +0100

My setup which seems to work rather well for an Installation of SharePoint 2010 Foundation consists of the following

MacBook Pro 2.53Ghz 4GB Unibody Running Windows 7 x64 Ultimate
Intel X25-M 160 GB G2 SSD
VMWare 7 Workstation
Virtual Image of Windows Server 2008 Enterprise R2 x64 2GB RAM allocated, 1 CPU, 2 Cores
SharePoint 2010 Foundation Beta installed onto the Virtual in Standalone mode.

I was quite surprised how well this actually runs considering the system requirements for SharePoint 2010 but I have found it to be very snappy and it looks very good.

I think the trick here is the SSD without an SSD virtualization is pretty slow on a standard laptop but with an SSD it’s really quick well worth the upgrade costs. What's really impressive is if you switch between the Virtual and your base OS there’s no delay even if your installing software or something in the Virtual the main OS is still really responsive.

Changing the Maximum Content Length for IIS7

info@simego.com (Sean Cleaver) — Sat, 05 Dec 2009 20:25:00 +0100

Under IIS7 there’s a new setting that prevents large content being sent to the server. If your using NeatUpload then you will see the error as a 404. However you might want to allow for such large content for example posting a video file to the server. The trick is to update the maxAllowedContentLength attribute which can be easily done from your web.config file like so.

<system.webServer> 
    <security> 
      <requestFiltering> 
        <requestLimits maxAllowedContentLength="314572800"> 
          <headerLimits> 
          </headerLimits> 
        </requestLimits> 
      </requestFiltering> 
    </security> 
</system.webServer>

SQL Bits V - Celtic Manor - Newport Wales

info@simego.com (Sean Cleaver) — Wed, 18 Nov 2009 09:28:45 +0100

I will be attending the SQL Bits conference this week in Newport on Friday and Saturday so if anybody would like to hook up whilst I am there please drop me a line and we can arrange a time.

Data Synchronisation Studio – SQL Azure Support

info@simego.com (Sean Cleaver) — Tue, 10 Nov 2009 14:31:32 +0100

We have just updated Data Synchronisation Studio so that you can connect to a SQL Azure database. Previously you couldn’t browse the Databases and Schema when trying to connect, SQL Azure prevents calling the Database Schema tables directly from the OleDb provider for some reason this is all fixed now in Data Sync.

We have been testing some simple synchronisations between a local SQL Server and SQL Azure and so far found that writing data to SQL Azure to be pretty slow once we had some data in there the compare was pretty quick and any updates after that were also quick.

Download the latest version of Data Synchronisation Studio from http://www.simego.com/beta

Azure SQL Manager – Support Updated

info@simego.com (Sean Cleaver) — Mon, 09 Nov 2009 21:50:40 +0100

Today we have been updating our experimental support for Azure SQL. So far we have enough support for the application to be very useful for development/testing with Azure SQL.

Browse Objects

Databases
Tables
Views
Procedures
Functions
Synonyms
Indexes
Logins
Users

Script / Edit

Tables
Views
Procedures
Functions
Synonyms
Indexes
Logins
Users (assign roles, change passwords)

Also you can sync data between SQL Server and Azure SQL with the Data compare features.

Schema compare is partially working (compare works by sync has issues) except for Table changes which fails due to deprecated SQL functions.

Connecting to Azure SQL Server

Viewing Schema and Table Data

Download the latest build of SQL Tools from http://www.simego.com/beta

Experimental SQL Azure Support for SQL Admin Studio

info@simego.com (Sean Cleaver) — Mon, 09 Nov 2009 14:19:10 +0100

So I have been tinkering with SQL Azure this weekend and it looks pretty good but I needed a few tools to work with my databases. As SQL Admin Studio is my favorite tool for working with databases I have made a few tweaks so that it works with SQL Azure.

So in the latest beta build you can browse the database schema, edit table data, compare schema, compare and sync data etc.

There are a few problems with Logins, Users, Table Designer and the scripts that are generated from the schema compare but I hope to improve this over time.

Download: http://www.simego.com/beta

Virtualization on Windows 7

info@simego.com (Sean Cleaver) — Tue, 27 Oct 2009 20:01:23 +0100

Now that I have a shiny new Windows 7 install I was looking for ways to create a Virtual Image for SharePoint development on my Laptop as I really don't want to run Server and Hyper-V. After looking into the new XP Mode for Windows 7 I thought that surely the new Windows VIrtual PC must be able to run a server image. Sure enough Windows Virtual PC integrates really well with Windows 7 and best of all you can even create a Server Virtual image the only limitations I have found so far is that you can only use 32 bit OS's and a Single CPU so no SharePoint 2010.

Installing Visual Studio 2010 Beta 2 on Mac Book Pro

info@simego.com (Sean Cleaver) — Tue, 20 Oct 2009 22:31:14 +0100

So being one of those types who just has to go ahead and try the latest and greatest releases of Visual Studio. I went ahead and tried to install VS 2010 Beta 2 today on my Mac Book Pro Bootcamp Windows 7 x64 setup. But oh no it kept failing with some stupid Access Denied error. After a few reboots and an a cup of coffee I worked it out! The Mac partition is read-only and for some reason the .NET Framework 4 Installer tries to use this disk to expand the installation files and then just fails so the fix is to remove the Drive letter from the Mac partition and all is well.

Oh and VS2010 looks pretty good can't wait to to give it a proper spin....

Just worked out what I now need to run SharePoint 2010 -- Apple have just refreshed their iMac's so the new Quad i7 27' with 8GB RAM looks about right.

Simego gains Microsoft Certified Partner

info@simego.com (Sean Cleaver) — Tue, 13 Oct 2009 10:11:35 +0100

Great News! Today Simego became a Microsoft Certified Partner after much hard work over the past few months and Data Synchronisation Studio now passes both "Windows 7 Compatible" and "Works with Windows Server 2008 R2" Certification programs.

Sync List Data between SharePoint Farms

info@simego.com (Sean Cleaver) — Wed, 13 May 2009 12:21:46 +0100

We have been getting a few inquiries lately about how to sync data between an internal SharePoint environment and a Client SharePoint environment. The good news is that this is really easy with Data Synchronisation Studio. Just connect the Source A to your master internal SharePoint data and then connect the Destination to B which might be on the internet and choose which columns to Sync and go.

You can also sync documents this way too.

Or even publish an internal issues/tasks/calendar list to an external client portal so your clients can view how the project is running without needing access to your internal SharePoint environment.

Data Synchronisation Studio also connects with most hosted installations of SharePoint so if you don’t already have an external portal for clients this is a quick and cost effective solution.

Data Sync Beta Update

info@simego.com (Sean Cleaver) — Tue, 12 May 2009 11:20:20 +0100

We have updated the SharePoint data providers for Data Sync to allow you to choose which view to bind to whilst this was possible before by hacking the project file it’s now just a simple selection from the schema properties window.

See the screenshot below.

This is supported on both the SharePoint List provider and the SharePoint Document Library allowing you to filter the SharePoint Data via a Custom View.

Download the latest build from http://www.simego.com/beta

Create a Timesheet Application with SharePoint and Exchange

info@simego.com (Sean Cleaver) — Fri, 08 May 2009 10:04:10 +0100

Overview

This example shows how you can build a real-time (almost) timesheet reporting system by using Exchange Server, SQL Server and SharePoint Server along with Data Synchronisation Studio. Each employee will record their time within their Exchange calendar and define the entry as a Work Hours category. Data Sync will then aggregate all the entries from each user calendar and present a summary view within SharePoint.

Preview of Summary Page

Preview of Exchange Calendar Item

Recording time is as simple as creating a calendar item and setting a few properties.

Set the item Category to Work Hours
Set the value of Location to the Job identification Code
Set the body of the message to the description of the work completed

This data will then be synchronised via Data Synchronisation Studio to a SQL Server Table which can then be used for reporting.

Step 1 - Create Database

Create a SQL database with SQL Server called TimesheetDB to store the timesheet records from Exchange Server and job detail Records.

USE [TimesheetDB]
GO
SET QUOTED_IDENTIFIER OFF
GO
SET ANSI_NULLS OFF
GO
CREATE TABLE [dbo].[Job] (
    [DSID] uniqueidentifier DEFAULT(newid()) NOT NULL,
    [Code] nvarchar(255) NOT NULL,
    [Company] nvarchar(255) NOT NULL,
    [Name] nvarchar(255) NOT NULL,
    [Quote] int NOT NULL,
    [Active] int NOT NULL
)
GO
ALTER TABLE [dbo].[Job] ADD CONSTRAINT [PK_1AD0E075] PRIMARY KEY CLUSTERED ([DSID] ASC)
GO
SET QUOTED_IDENTIFIER OFF
GO
SET ANSI_NULLS OFF
GO
CREATE TABLE [dbo].[Billing] (
    [CalendarID] nvarchar(50) NOT NULL,
    [CreationDate] datetime NULL,
    [LastModified] datetime NULL,
    [StartDate] datetime NULL,
    [EndDate] datetime NULL,
    [Duration] int NULL,
    [Importance] ntext NULL,
    [Priority] int NULL,
    [Subject] nvarchar(255) NULL,
    [Body] ntext NULL,
    [Location] nvarchar(255) NULL,
    [Category] nvarchar(255) NULL,
    [MailBoxName] nvarchar(255) NULL
)
GO
ALTER TABLE [dbo].[Billing] ADD CONSTRAINT [PK_2FA475F7] PRIMARY KEY CLUSTERED ([CalendarID] ASC)
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_NULLS ON
GO
CREATE VIEW Timesheet_Stats

SELECT
    [Billing].Location AS Code,
    Job.Company,
    Job.Name,
    Job.Quote,
    ROUND(SUM(CAST(Billing.Duration as float))/60/60, 2) AS Total,
    ROUND(CAST(LAST_MONTH.Duration as float)/60/60, 2) AS [Last Month],
    ROUND(CAST(THIS_MONTH.Duration as float)/60/60, 2) AS [This Month]
FROM [Billing]
INNER JOIN Job ON Job.Code = [Billing].Location
LEFT JOIN ( SELECT [Location], SUM([Duration]) AS [Duration] FROM [dbo].[Billing] WHERE Month([StartDate]) = Month(DATEADD(m, -1, GETDATE())) GROUP BY [Location]) LAST_MONTH ON LAST_MONTH.Location = [Billing].[Location]
LEFT JOIN ( SELECT [Location], SUM([Duration]) AS [Duration] FROM [dbo].[Billing] WHERE Month([StartDate]) = Month(GETDATE()) GROUP BY [Location]) THIS_MONTH ON THIS_MONTH.Location = [Billing].[Location]
GROUP BY [Billing].Location, Job.Company, Job.Name, Job.Quote, Job.Active, LAST_MONTH.Duration, THIS_MONTH.Duration
HAVING Job.Active = 0
GO

Step 2 - Create Exchange Sync Project

To connect Data Sync to your exchange server you will need to define a control file that contains the connection details to your server and the mailboxes that should be included in the synchronisation.

Create an Xml file call Timesheet.xml that is similar to the example below.

<Exchange>   
  <Server>https://yourserver/exchange</Server>    
  <Username></Username>    
  <Password></Password>    
  <Domain></Domain>    
  <FormsAuthentication>False</FormsAuthentication>    
  <Calendar>#SERVER#/#MAILBOX#/Calendar</Calendar>    
  <DaysBackToReturn>90</DaysBackToReturn>    
  <Category>Work Hours</Category>    
  <MailBoxes>    
    <MailBox>user1@domain.com</MailBox>    
    <MailBox>user2@domain.com</MailBox>     
    <MailBox>user3@domain.com</MailBox>    
  </MailBoxes>    
</Exchange>

Now connect Data Source A of Data Sync to this Xml file via the Exchange Calendars data provider.

Then connect Data source B to the Billing Table in the Timesheet Database and your Data Sync project should now look like this.

Step 3 - Create Job List

Connect Data Sync to the Job Table and then create a SharePoint List called Job to Sync the table.

Screenshot of the Job project in Data Sync

The Job list should be configured for 2-way sync so that new Jobs added to the SharePoint List are synchronised with the Database. To do this go to the File menu and choose properties and set the sync option to SyncAandB.

Step 4 - Create Summary List

Connect Data Sync to the Timesheet_Stats View and then create a SharePoint List called Summary to Sync the view.

Step 5 - Setup Schedule

This project requires that the synchronisation is run in a particular order for example.

Sync Job List
Sync Exchange Data
Sync Summary List

To execute the synchronisation in this order you need to setup the Simego.DataSync.Run.exe command line application like this

Simego.DataSync.Run.exe Job.dsprj Exchange.dsprj Summary.dsprj

Now when the Synchronisation process is started each project will be processed in order.

Note - Mailbox Security

For data sync to read the entries from each user calendar the account that is used to run the sync requires Reviewer access at the Mailbox and Calendar level. Which can be configured via the Outlook Client.

You can download and use a fully featured version of Data Sync Studio for 15 Days to see how this Sharepoint solution works.

Click here to go to the Downloads

How to Synchronise a SharePoint Document Library with a File Share

info@simego.com (Sean Cleaver) — Fri, 08 May 2009 09:58:47 +0100

Overview

This article describes how to use Data Synchronisation Studio to Synchronise a File Share with a SharePoint Document Library. This could be useful during a migration from a traditional file server to SharePoint or to just publish internal documents to SharePoint.

Step 1 - Install Custom Web Services Solution

Download our custom SharePoint Web Services solution below and deploy to your SharePoint environment. To install the solution into your SharePoint environment use the STSADM application for example

STSADM.EXE -o addsolution -filename Simego.SharePoint.Services.wsp

Then deploy the solution from the SharePoint Central Administration console.

This custom web service allows us to update the Modified Timestamp on files that are synchronised with SharePoint.

Download SharePoint Solution Package

Step 2 - Connect to File Share

Enter the Path to your file share in the connection dialog for datasource A you can also define an ignore pattern list and optionally choose to recuirse through all sub directores.

Connect File Share Data Source

Step 3 - Connect to the Destination SharePoint Document Library

Enter the Url to your SharePoint Site in the connection dialog for datasource B.

1 - Connect to SharePoint Document Library Data Source
2 - Connect to SharePoint Document Library Data Source

Step 3 - Disable Compare on File Length

Set the compare option on the File Length Property to False to disable comparing the file length as the value reported by SharePoint is different to the file system length.

Data Sync Schema Map

Step 4 - Compare A->B and Synchronise

Run an initial compare to look for the new documents that need to be added to SharePoint in this case we should see all documents as the SharePoint list contains no documents.

Compare Results
Start a Synchronisation
Document Library in SharePoint Populated

Other Options

Synchronise Document Library to a File Share by reversing this process.
Synchronise Documents between different SharePoint Environments or Site Collections by connecting two SharePoint Document Libraries.
Synchronise two file shares by connecting two different file shares.

CSV File Provider for Data Sync

info@simego.com (Sean Cleaver) — Tue, 17 Mar 2009 10:09:33 +0100

Installation

Download the provider from http://www.simego.com/download then start Data Synchronisation Studio version 1.0.156 or higher and go to the Tools Menu and select “Add-in Manager”

Then click the Install button and step through the installation wizard when the install is complete you should see the provider in the installed add-ins list.

Now when you connect to a data source you will have a CSV File option

Data Synchronisation Studio - Update

info@simego.com (Sean Cleaver) — Mon, 16 Mar 2009 09:49:57 +0100

The latest build 1.0.156 of Data Synchronisation Studiois now avaliable for download from our web site.

This version includes fixes and improvements from customer feedback and support for the new MS CRM 4.0 add-on data provider.

We have also added a new Wizard that allows you to add new columns to an existing SharePoint List such as the DSID synchronisation column.

Also Create Scheduled Task is now working on Server 2008 x64

And finally you can now connect to SharePoint instances that are using Forms Based Authentication or are behind ISA server.

Use Custom AD Attributes with Data Sync AD Provider

info@simego.com (Sean Cleaver) — Mon, 16 Mar 2009 09:42:19 +0100

Hi,

We have updated the Data Sync AD provider to allow you to add additional properties without the need for us to build a specific providers.

When you re-open your AD project file you should notice a new file is created called projectname.properties.xml in the same directory this file contains all the properties that are available to you and is a simple xml document.

For example :-

<?xml version="1.0" encoding="utf-8"?>

</Properties>

All you need do is edit this file to add the additional properties where Name=the name of the property in Active Directory. Then re-open the project. You can then drag the new property from the Schema window onto the blank area of the Map to add the additional column.

For example add these properties to your file.

This new feature is in Data Sync Version 1.0.44 and higher the latest version can be downloaded from http://www.simego.com/download or http://www.simego.com/beta

CRM 4.0 Data Provider for Data Synchronisation Studio

info@simego.com (Sean Cleaver) — Mon, 16 Mar 2009 09:39:03 +0100

We have just released an add-on Data provider to Data Synchronisation Studio that allows you to connect to MS CRM 4.0 and synchronise your data between a variety of data sources such as MS SQL Server, Oracle, MySQL, SharePoint, OLEDB, ODBC etc.

We will post some examples on this blog later.

Automating Code Signing

info@simego.com (Sean Cleaver) — Mon, 19 Jan 2009 13:01:44 +0100

Were still using Nant to automate the build process for all Simego Products and recently we got a Code Signing Certificate and needed to add this into the build process. I looked at just adding the command line tasks for SignTool.exe to our build script however we have many assemblies and this process would be pretty boring. So I put together a quick Nant task that would reflect over the assembly and if it exposes a Company attribute of “Simego” then it sign’s the assembly.

Custom Nant Task

using System;
using System.Collections.Generic;
using System.Text;
using NAnt.Core.Attributes;
using NAnt.Core;
using System.IO;
using System.Reflection;
using System.Diagnostics;

namespace SimegoNantLib
{
    [TaskName("SignTool")]
    public class SignTool : Task
    {
        private string _path;
        private string _signToolPath;

        [TaskAttribute("signtoolpath", Required = false)]
        public string SignToolPath
        {
            get { return _signToolPath; }
            set { _signToolPath = value; }
        }

        [TaskAttribute("path", Required = false)]
        public string Path
        {
            get { return _path; }
            set { _path = value; }
        }

        protected override void ExecuteTask()
        {
            //Extract Certificate
            string certificatePath = System.IO.Path.Combine(System.IO.Path.GetTempPath(), string.Format("Cert{0}.pfx", Guid.NewGuid().GetHashCode().ToString("X8")));
            using (Stream s = this.GetType().Assembly.GetManifestResourceStream("SimegoNantLib.Certificate.pfx"))
            {
                byte[] data = new byte[s.Length];
                s.Read(data, 0, data.Length);

                using (FileStream fs = new FileStream(certificatePath, FileMode.CreateNew))
                {
                    fs.Write(data, 0, data.Length);
                }
            }

            try
            {
                //Process files in directory looking to see if the assembly is a Simego product.
                List<string> filesToSign = new List<string>();

                foreach (FileInfo fInfo in new DirectoryInfo(Path).GetFiles())
                {
                    if (
                        fInfo.Extension.Equals(".dll", StringComparison.OrdinalIgnoreCase) ||
                        fInfo.Extension.Equals(".exe", StringComparison.OrdinalIgnoreCase))
                    {
                        try
                        {
                            byte[] buffer = new byte[fInfo.Length];
                            using (FileStream fs = new FileStream(fInfo.FullName, FileMode.Open, FileAccess.Read))
                            {
                                fs.Read(buffer, 0, buffer.Length);
                            }

                            Assembly assembly = Assembly.Load(buffer);
                            AssemblyCompanyAttribute[] attrib = (AssemblyCompanyAttribute[])assembly.GetCustomAttributes(typeof(AssemblyCompanyAttribute), true);
                            if (attrib != null && attrib.Length > 0)
                            {
                                if (attrib[0].Company.Contains("Simego"))
                                {
                                    filesToSign.Add(fInfo.FullName);
                                }
                            }
                        }
                        catch { throw; }
                    }
                }

                //Sign Files
                string signToolCommand = "sign /f \"{0}\" /p {1} /t http://timestamp.comodoca.com/authenticode \"{2}\"";
                foreach (string file in filesToSign)
                {
                    ProcessStartInfo pStartInfo = new ProcessStartInfo(SignToolPath, string.Format(signToolCommand, certificatePath, "mypassword", file));
                    pStartInfo.WindowStyle = ProcessWindowStyle.Hidden;

                    Process p = Process.Start(pStartInfo);
                    p.WaitForExit();
                    if (p.ExitCode != 0)
                    {
                        throw new BuildException(string.Format("SignTool.exe returned a result of {0}", p.ExitCode));
                    }
                }

            }
            finally
            {
                if (File.Exists(certificatePath))
                {
                    File.Delete(certificatePath);
                }
            }
        }
    }
}

Then in my Nant build script I just add the custom task library.

<loadtasks assembly="D:\Projects\SQLTools\SimegoNantLib.dll" />

And add a target to process the files

Data Sync Studio - Beta Update

info@simego.com (Sean Cleaver) — Fri, 05 Dec 2008 12:29:38 +0100

A new release of Data Synchronisation Studio (v1.0.100.0) has been released to the beta site this new release provides support for the following :-

Microsoft Access Database Synchronisation
Generic ODBC Database Synchronisation
Generic OLEDB Database Synchronisation
File Synchronisation from the File System
SharePoint Document Library Synchronisation

The big new feature here is the ability to sync files and synchronise a SharePoint document library with a directory structure. This allows for a gradual migration from a file server onto SharePoint or to just automatically import documents from some other system.

To fully support updating of "Created" and "Modified" file dates a new custom feature must be installed on the SharePoint server contact us for more details about this.

Data Sync Beta Download Site

Data Sync Studio - Updated

info@simego.com (Sean Cleaver) — Mon, 10 Nov 2008 12:54:27 +0100

It's been a while since I posted an update about Data Synchronisation Studio. Well we have been really busy making improvements based on customer feedback.

Below are a few of the improvements we have made recently.

Ability to enable case in-sensitive comparisons.
Ability to ignore white space differences when comparing text.
You can now edit the Synchronisation Option from the UI under File->Properties.
New Connection dialog and the ability to provide your own connection interface on custom data providers.
New simple command line tool for 1-Way synchronisation's without the need to snapshot the source/destination data.
Updated sample code for the Custom Excel Data source.
Ability to refresh the schema of the data source without reloading the project.
Ability to re-connect a data source to a different connection without creating a new project.
Ability to see at a glance the details about the connection.
Ability to schedule a Windows Task from the UI.

The latest beta release is now 1.0.85 which is available from http://sharepoint.simego.com/KB/DataSyncStudio/default.aspx

SilverLight 2.0 ComboBox / DropDown

info@simego.com (Sean Cleaver) — Thu, 04 Sep 2008 11:19:09 +0100

I have had a few requests to provide a complete project with source code for the sample combo box solution for SilverLight.

I still seem to have an issue with the list changing the height of the control when the drop down is enabled. Changing the ZIndex of the Canvas makes no difference probably some combination of Canvas and StackPanel issue not too sure.

Download SilverlightComboBox.zip (10.5K)

Adding SilverLight Charts to SharePoint

info@simego.com (Sean Cleaver) — Tue, 02 Sep 2008 13:06:29 +0100

Would you believe it! After building the SharePoint Chart solution to work with Flash Charts I came across these SilverLight Charts from VISIFire which are just fantastic. So I have updated the code to work with these charts.

Remember to get SilverLight to work on your server you need to add the MIME Type "application/x-silverlight-app" for the XAP extension.

The Code adds a Web Part to SharePoint for displaying a Bar Chart from Data Stored within a SharePoint List.

Download Simego.SilverLight.Charts.zip (158.2K)

Create Dynamic Flash Charts with SharePoint

info@simego.com (Sean Cleaver) — Mon, 01 Sep 2008 20:58:41 +0100

Previously I had detailed how to create fancy dashboard pages in SharePoint by using free flash charts from Fusion Charts. Now I have created a Web Part that does all the hard work to dynamically display a Bar Chart from Data held within a SharePoint List which could then be easily modified to use BDC Data or SQL Server etc.

Below shows how the Chart might look on your SharePoint page you simply add the Web part to the page and setup the List to get the data from.

Chart Web Part

Chart Web Part Design View

This solution assumes that the Flash SWF file for the Chart in located in a Charts document library within the current Web.

You may download the source code for this solution to see how to build the solution yourself.

Download Simego.Charts.zip (8.8K)

How to: Write a CSV Provider for Data Sync Studio

info@simego.com (Sean Cleaver) — Thu, 28 Aug 2008 16:52:48 +0100

Recently I had been tasked with the requirement to import Weather Data from the Met Office Weather Data Feed. They basically dump a CSV file onto your server via FTP and it's up to you to import it into your own systems. There are a 101 ways of doing this but as Data Sync is my current favourite I thought I would just write a provider to do it.

The reason for this is to actually only apply the changes to the database rather than the brutal delete everything and import. Also as Data Sync supports multiple providers (Oracle, MySQL, MSSQL, SharePoint etc) I can easily import this data into a variety of systems.

Here's the source code

using System;
using System.Collections.Generic;
using System.Text;
using Simego.DataSync.Interfaces;
using Simego.DataSync;
using System.Data.OleDb;
using System.Data;
using Simego.DataSync.Providers;
using System.IO;

namespace Simego.Weather
{
    public class CurrentWeatherDataSourceReader : IDataSourceReader
    {
        private const string CONNECT_STR = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source={0};Extended Properties=\"text;FMT=Delimited\"";

        protected DataSchema _schemaMap;
        protected DataSchemaSide _side = DataSchemaSide.DataSourceA;
        private bool _readWrite = false;
        private string _weatherCsvFileName;

        public string WeatherCsvFileName
        {
            get { return _weatherCsvFileName; }
            set { _weatherCsvFileName = value; }
        }

#region IDataSourceReader Members

        public DataSchema SchemaMap
        {
            get { return _schemaMap; }
            set { _schemaMap = value; }
        }

        public DataSchemaSide Side
        {
            get { return _side; }
            set { _side = value; }
        }

        public bool ReadWrite
        {
            get { return _readWrite; }
            set { _readWrite = value; }
        }

        public DataTable GetDataTable()
        {
            #region Create Schema.ini
            string schemaIni = Path.Combine(Path.GetDirectoryName(WeatherCsvFileName), "Schema.ini");

if (File.Exists(schemaIni)) File.Delete(schemaIni);

            using (StreamWriter sw = File.CreateText(schemaIni))
            {
                sw.WriteLine(string.Format("[{0}]", Path.GetFileName(WeatherCsvFileName)));
                sw.WriteLine("Format=Delimited(,)");
                sw.WriteLine("MaxScanRows=100");
                sw.WriteLine("CharacterSet=ANSI");

                //Add the Columns....
                sw.WriteLine("Col1=\"Code\" Integer");
                sw.WriteLine("Col2=\"Name\" TEXT");
                sw.WriteLine("Col3=\"State\" TEXT");
                sw.WriteLine("Col4=\"Country\" TEXT");
                sw.WriteLine("Col5=\"Continent\" TEXT");
                sw.WriteLine("Col6=\"Type\" TEXT");
                sw.WriteLine("Col7=\"Hour\" Integer");
                sw.WriteLine("Col8=\"Day Name\" TEXT");
                sw.WriteLine("Col9=\"Day\" Integer");
                sw.WriteLine("Col10=\"Month\" Integer");
                sw.WriteLine("Col11=\"Year\" Integer");

                sw.WriteLine("Col12=\"WindDirection\" TEXT");
                sw.WriteLine("Col13=\"WindSpeed\" Double");
                sw.WriteLine("Col14=\"Temp\" Double");
                sw.WriteLine("Col15=\"SkyCode\" Integer");
                sw.WriteLine("Col16=\"Visibility\" TEXT");
                sw.WriteLine("Col17=\"Pressure\" Double");
                sw.WriteLine("Col18=\"Tendency\" TEXT");
                sw.WriteLine("Col19=\"Humidity\" Double");

}

            File.SetAttributes(schemaIni, File.GetAttributes(schemaIni) | FileAttributes.Hidden);
            #endregion
            DataTable dt = SchemaMap.GetDataTable();
            dt.TableName = "Forecast";

StringBuilder query = new StringBuilder();
query.AppendFormat("SELECT * FROM [{0}]", string.Format("{0}#csv", Path.GetFileNameWithoutExtension(this.WeatherCsvFileName).ToLower()));

            DataTable tmpDt = new DataTable();
            using (OleDbConnection connection = new OleDbConnection(string.Format(CONNECT_STR, Path.GetDirectoryName(WeatherCsvFileName))))
            {
                connection.Open();

                using (OleDbCommand cmd = new OleDbCommand(query.ToString(), connection))
                {
                    using (OleDbDataAdapter adapter = new OleDbDataAdapter(cmd))
                    {
                        //adapter.FillSchema(tmpDt, SchemaType.Source);
                        adapter.Fill(tmpDt);
                    }
                }
            }

ProcessData(tmpDt, dt);

return dt;

}

private void ProcessData(DataTable source, DataTable dt)
{

            foreach (DataRow row in source.Rows)
            {
                DataRow newRow = dt.NewRow();

                newRow["Code"] = row["Code"];
                newRow["SkyCode"] = row["SkyCode"];
                newRow["Temp"] = row["Temp"];
                dt.Rows.Add(newRow);
            }
        }

        public DataSchema GetDefaultDataSchema()
        {
            DataTable dt = new DataTable("WeatherData");

            DataColumn dc = new DataColumn("Code", typeof(int));
            dc.AllowDBNull = false;
            dc.Unique = true;

dt.Columns.Add(dc);

            dt.Columns.Add("SkyCode", typeof(int));
            dt.Columns.Add("Temp", typeof(double));
            return new DataSchema(dt);

}

        public string GetMappedSourceColumn(string name)
        {
            return Utility.GetMappedSourceColumn(SchemaMap, Side, name);
        }

        public List<ProviderParameter> GetInitializationParameters()
        {
            List<ProviderParameter> parameters = new List<ProviderParameter>();

parameters.Add(new ProviderParameter("WeatherCsvFileName", this.WeatherCsvFileName));

return parameters;
}

        public void Initialize(DataSchema schema, DataSchemaSide side)
        {
            this.SchemaMap = schema;
            this.Side = side;
        }

        public void Initialize(List<ProviderParameter> parameters)
        {
            foreach (ProviderParameter p in parameters)
            {
                switch (p.Name)
                {
                    case "WeatherCsvFileName":
                        {
                            this.WeatherCsvFileName = p.Value;
                            break;
                        }
                    default:
                        {
                            break;
                        }
                }
            }
        }

        public IDataSourceWriter GetWriter()
        {
            return null;
        }

#endregion

}
}

Billing Tracking System with Exchange and SharePoint

info@simego.com (Sean Cleaver) — Thu, 07 Aug 2008 22:29:32 +0100

As an owner of a small IT business I am always looking for ways to better manage my time & billing tracking. I also want to track almost real-time how much time my other staff have used against budget. Exchange provides an easy way for every one to record their time as Calendar entries by using the Location to store a Job reference code and set the Category to Work Hours. So no matter where everybody is working they can update their exchange calendar with their times and description and I can then synchronise the data into a SQL Database and summarise the Data in SharePoint.

Below is a Screenshot showing the Job Status in SharePoint and the Jobs that are currently active. These Jobs can be editied in SharePoint and synchronised back to SQL Server with Data Sync.

Requirements

Exchange Server 2007 User Accounts
SharePoint Server 2007 (WSS)
SQL Server Express 2005
Data Synchronisation Studio

1) Configure Data Synchronisation Studio to download Billing Entries from Exchange Calendars into SQL Server.

The ExchangeGroupCalendarDataSourceReader custom provider can read values from multiple exchange mailbox calendars and takes a single xml configuration file for it's settings.

Example Xml File

<Exchange>
<Server>https://myserver/exchange</Server>
<Username>user@simego.com</Username>
<Password></Password>
<Domain></Domain>
<FormsAuthentication>True</FormsAuthentication>
<Calendar>#SERVER#/#MAILBOX#/Calendar</Calendar>
<DaysBackToReturn>90</DaysBackToReturn>
<Category>Work Hours</Category>
<MailBoxes>
<MailBox>user1@simego.com</MailBox>
<MailBox>user2@simego.com</MailBox>
</MailBoxes>
</Exchange>

Crack open Data Sync and connect to the Exchange Store with the configuration xml file and create a SQL destination table called Billing.

2) Add a Table to SQL Server to list the JOB Codes we have in the System

CREATE TABLE [dbo].[Job] (
    [DSID] uniqueidentifier DEFAULT(newid()) NOT NULL,
    [Code] nvarchar(20) NOT NULL,
    [Company] nvarchar(255) NOT NULL,
    [Name] nvarchar(255) NOT NULL,
    [Quote] int DEFAULT(0) NOT NULL,
    [Active] int DEFAULT(0) NOT NULL
)

3) Create a View in SQL to show the calculated totals

CREATE VIEW Billing_Stats
AS
SELECT Location AS Code, Job.Company, Job.Name, Job.Quote, SUM(Duration)/60/60 AS Total
FROM [dbo].[Billing]
INNER JOIN Job ON Job.Code = Billing.Location
GROUP BY Location, Job.Company, Job.Name, Job.Quote, Job.Active
HAVING Job.Active = 0

4) Configure Data Sync to Synchronise the JOB Codes Table to a SharePoint List

5) Configure Data Sync to Synchronise the Calculated Totals View to a SharePoint List

This new Exchange Data Provider is currently in private beta with a few clients if you would like to know more please contact sales[at]simego.com.

Data Sync and MS Exchange Provider

info@simego.com (Sean Cleaver) — Wed, 06 Aug 2008 15:47:06 +0100

Were currently working on an Exchange Provider for Data Sync that will allow you Synchronize/Publish data from Exchange Calendars/Task Lists/Contact Lists to any other Data Source that supports writing such as SharePoint and SQL.

If you have a project that could use this functionality then please get in touch.

Sync AD Users to SharePoint - 2 Way

info@simego.com (Sean Cleaver) — Wed, 30 Jul 2008 15:04:33 +0100

So some of you have requested support to provide 2-Way Synchronization of AD Users between a SharePoint List and Active Directory. The good news is that the AD Provider for Data Synchronization Studio now supports this.

So you can effectively publish your AD Users to an Intranet Site to create a "Staff List" or "Telephone List" and then from this you can now allow your staff to update the personal information stored in the AD themselves by just simply editing the records in SharePoint. Then when the Synchronization occurs all changes are applied to the AD. There are a few limitations you can't create new AD Users this way and you can't delete users from the AD and certain properties are not updateable such as Member-Of etc.

This is part of release 1.0.48 and is available now from the download page.

Data Synchronisation Studio - Updated

info@simego.com (Sean Cleaver) — Fri, 06 Jun 2008 15:24:15 +0100

Just released an update to Data Synchronisation Studio (1.0.45) to allow you to create a SQL Table from an existing SharePoint list. This enables you to easily export your SharePoint data to a SQL Server database and synchronise any changes with the Sync Service. If you use SQL Server Reporting Services and would like to also use SharePoint data in your reports then this is a great solution.

Download from here

MOSS BDC Design Studio 2.0 - Updated

info@simego.com (Sean Cleaver) — Sat, 31 May 2008 15:47:15 +0100

Just released an update (version 2.0.5) to the web for MOSS BDC Design Studio 2.0 this release fixes an issue with database connection details being in lower case which MOSS SP1 did not like.

Download from Here

2-Way List Sync SQL Server <-> SharePoint

info@simego.com (Sean Cleaver) — Wed, 21 May 2008 20:35:57 +0100

Data Synchronization Studio supports 2 Way Synchronisation of data between SQL Server and SharePoint if you follow a couple of guidelines.

The main thing to consider is to use a GUID (uniqueidentifier) as the Primary Key (Unique Constraint) and if you call this column DSID then the Mapping will be easier.

Step 1 is to create a Table in your database that follows this schema like below.

CREATE TABLE [dbo].[ProductList] (
    [DSID] uniqueidentifier DEFAULT(newid()) NOT NULL,
    [CompanyName] nvarchar(40) NOT NULL,
    [ProductName] nvarchar(40) NOT NULL,
    [CategoryName] nvarchar(15) NOT NULL,
    [QuantityPerUnit] nvarchar(50) NULL,
    [UnitPrice] decimal(18, 0) DEFAULT(0) NULL,
    [UnitsInStock] smallint DEFAULT(0) NULL,
    [UnitsOnOrder] smallint DEFAULT(0) NULL
)
GO
ALTER TABLE [dbo].[ProductList] ADD CONSTRAINT [PK_ProductList_2FE911D7] PRIMARY KEY CLUSTERED ([DSID] ASC)
GO

Step 2 is to fill your table with the Data you would like to Synchronize.

Step 3 You will then need to fire up Data Synchronization Studio and create a List in SharePoint that will map to this table this should look something like the picture below.

Important: You should have a Column DSID mapped to a SharePoint column DSID of type GUID and it should be marked as the Unique Column.

Step 4 Modify the Project File (*.dsprj) to support 2-way sync by default the 2-way sync is disabled to prevent accidental deletion of data.

Open the *.dsprj file in your favourite text editor (notepad.exe) and look for the Xml Element Project you should see an Attribute called SyncOption change this value to be SyncAandB.

So the Project Element should look like this.

<Project Name="" Version="" SyncOption="SyncAandB">

You can now drop the project file into the Synchronization Service project folder (C:\Windows\DataSyncProjects) and start the service. Now you can make changes in both SQL Server and SharePoint and the change will be replicated in each system.

ComboBox / DropDownList for Silverlight 2.0

info@simego.com (Sean Cleaver) — Thu, 08 May 2008 20:36:22 +0100

After playing around with Silverlight 2.0 which is pretty cool BTW. I noticed that there is no ComboBox control yet. So I thought it can't be that hard to get something working so here's how I done it.

The finished ComboBox in Silverlight

The XAML

(Note: I am using a graphic for the DropDown icon you could draw it but I didn't have time to try that!)

The Code Behind

using System.Windows;
using System.Windows.Controls;
using System.Windows.Input;

namespace SilverlightApplication1
{
    public partial class ComboBox : UserControl
    {
        public ListBox TheList { get { return MyControl; } }

        public ComboBox()
        {
            InitializeComponent();
        }

private void StackPanel_MouseLeftButtonDown(object sender, MouseButtonEventArgs e)
{

            if (MyControl.Visibility == Visibility.Visible)
            {
                MyControl.Visibility = Visibility.Collapsed;
            }
            else
            {
                MyControl.Visibility = Visibility.Visible;
            }
        }

        private void MyControl_SelectionChanged(object sender, SelectionChangedEventArgs e)
        {
            MyTextBox.Text = MyControl.SelectedItem.ToString();
        }
    }
}

Then on your main Silverlight Page just add something like this... (After you add the namespace right!)

<MyCtrls:ComboBox x:Name="xyz" Canvas.Left="400" Canvas.Top="50"></MyCtrls:ComboBox>

Now there is certainly room for improvement but this shows how easy it is to create your own controls in Silverlight.

Synchronise Active Directory User Details to SharePoint (WSS)

info@simego.com (Sean Cleaver) — Sun, 30 Mar 2008 14:05:26 +0100

By using the Custom Active Directory Users Data Source for Data Sync Studio you can synchronise user details from Active Directory to a SharePoint List in either WSS or MOSS 2007.

First Start up Data Sync Studio

Then choose the Connect to Data Source Link on the Left Panel

Now pick 'Custom Source' and check that you are using the ActiveDirectoryUsersDataSourceReader then enter the details for your Active Directory in this example I am using the AD on the Microsoft Demo CRM VPC litwareinc.com.

Now the Schema is displayed on the left we now want to create a SharePoint list to import the Users details.

Go to the Tools Menu and Choose 'Create SharePoint List' this will start the List Wizard.

Choose the SharePoint Site to Create the List

Give the List a Name and pick the Column to display as the Title Field in SharePoint

Now both sides are Mapped in Data Sync Studio

Click the 'Compare A-B' button in the Toolbar to find the changes.

Then Click the Synchronise Button to push the User Details into the SharePoint List

Now your Details are available in SharePoint and you can use the SharePoint Search to look up Employees and find there phone numbers etc.

Once you have the saved the Project file you can use the Synchronisation Windows Service (part of Data Sync Studio) to keep the list in Sync with your Active Directory.

Download Data Sync Studio

Data Sync Studio (Beta) Update

info@simego.com (Sean Cleaver) — Thu, 27 Mar 2008 21:21:27 +0100

Hi,

I have just uploaded a new beta release of Data Sync Studio to the site to enable full reliable 2-way sync of data between a SharePoint list and another Data Source. There is one requirement of your source data to have a unique GUID column this is used to identify the row on both sides and therefore you can add rows to both the source and destination at the same time and they still sync.

The quickest way to get this setup working is to create a Table in your database with a column called DSID of type uniqueidentifier and set it to be the PrimaryKey. Then just connect to the source system with DS Studio and go to Tools -> Create SharePoint List to create the List and then the Map will be configured just right!

Download Data Sync Studio

MOSS BDC Design Studio 2.0 (Beta)

info@simego.com (Sean Cleaver) — Sun, 23 Mar 2008 13:01:10 +0100

A BETA release of MOSS BDC Design Studio version 2.0 is now available for download this new release adds support for different database systems.

SQL Server
Oracle
OLEDB Data Sources
ODBC Data Sources (including MySQL)
Web Services

There are also several bug fixes and refinements to the automatic BDC method generation (better handling of generated input parameters) along with improved connection defaults (auto-sense connection type based on user connection) created from the initial connection.

MOSS BDC Design Studio 2.0 Download

Data Sync Studio (Beta)

info@simego.com (Sean Cleaver) — Fri, 21 Mar 2008 15:02:19 +0100

Data Sync Studio for SharePoint has reached an initial milestone in the project where it is available for public download and trial.

This release adds support for the following data sources

SQL Server 2005+
SharePoint Server 2007 ( MOSS & WSS )
Oracle 9+
MySQL 4.0+
Custom Data Sources

Each Data Source can Synchronise between each other i.e. Oracle to MySQL, MySQL to SharePoint, Oracle to SharePoint, MySql to SQL Server you get the idea.

Also you can create your own Data Providers with Visual Studio 2005/2008 to plug in virtually any Data Source you like and there is a Sample Project for reading Excel files within the Install Directory.

Here's a few scenarios where you might think of using Data Sync Studio

Publish SharePoint Lists to SQL Server Tables for Reporting Services
Publish SQL Server Tables/Views to SharePoint for Excel Services or KPI's
Publish Application Configuration Data to SharePoint to allow for Access Control.
Publish SharePoint List Data to Oracle for Crystal Reports
With a Custom Provider Publish CRM data to SharePoint

Download Data Sync Studio

MOSS BDC Design Stuido 2.0

info@simego.com (Sean Cleaver) — Mon, 10 Mar 2008 14:42:36 +0100

MOSS BDC Design Studio 2.0 is nearly ready for release so check back here soon for news of the release. In this release we are supporting all database connectivity options i.e. SQL Server, Oracle, OLEDB, ODBC and Web Services. This update will be FREE to all existing customers and still only 199USD for all new customers.

Create Fancy Flash Charts on SharePoint

info@simego.com (Sean Cleaver) — Thu, 21 Feb 2008 16:38:34 +0100

To add fancy charts to SharePoint pages driven by data from either SharePoint, BDC or some other source is actually very easy.

You will need some good Flash Charts I recommend starting with FusionFreeCharts

Step 1 - Create a SharePoint Docment Library to hold the Flash SWF Files and upload them.

Step 2 - Create a Web Part to Generate the Xml Data For the Flash Chart

Using Visual Studio 2005 and the SharePoint Extensions Create a New WebPart and generate the Xml for the Chart in the Render Method.

Download ChartData.zip

Step 3 - Create a Web Page with SharePoint Designer to output the Xml Data to the Chart

This page should have next to no content i.e. no HTML Markup it would look something like this...

<%@ Page Language="C#" %>
<%@ Register Tagprefix="CD" Namespace="ChartDataProvider" Assembly="ChartData, Version=1.0.0.0, Culture=neutral, PublicKeyToken=9f4da00116c38ec5" %>
<CD:ChartDataProvider runat="server" id="ChartData1" __WebPartId="{BFE5A1BD-85F0-4E13-8756-6B223ECEBF33}" />

Step 4 - Add a Content Editor Web Part to a Page and Configure the Flash Chart.

Edit the Source of the WebPart to be something like this...

That's it real fancy charts in SharePoint you could do the same with Silverlight but I don't know of any free good Chart Controls.

Manage SSIS Configurations in SharePoint with DataSync Studio

info@simego.com (Sean Cleaver) — Wed, 13 Feb 2008 12:03:26 +0100

DataSync Studio allows you to publish SQL Tables to SharePoint and keep them in sync with each other. So while I was working on a SSIS Project I thought it would be really cool to manage the Configuration Settings in SharePoint.

So the first thing is to use Data Sync Studio to Connect the SQL Table to SharePoint and define the mapping between the List Columns.

Then configure the List Synchronisation Service and get it running, next just save the Data Sync Project file into the Sync Service Monitor Directory and your done.

Now it's really easy for me to create SharePoint views for each of the SSIS Configurations that I have i.e. 'DEV', 'UAT', 'LIVE' and I can also enable Version Control and Configure who has access to update the items. Now I don't need to create an application to edit the settings or give someone instructions on how to use SQL Management Studio.

To be part of the BETA Program for DataSync Studio please drop me a line at sales[at]simego.com

DataSync Studio for SharePoint List Sync

info@simego.com (Sean Cleaver) — Mon, 28 Jan 2008 12:16:52 +0100

Soon to be available is a solution that can easily import data from a SQL Source into your SharePoint Lists and allow you to keep both sources of data in Sync with each other.

All you need to is point the tool at your SQL Table and then either create a List from that or Map the fields to an existing List.

Screenshot

For more information please contact sales[at]simego.com

SharePoint QueryString(URL) Filter WebPart

info@simego.com (Sean Cleaver) — Sat, 19 Jan 2008 17:33:33 +0100

WSS only ships with a few basic Web Parts one that I miss from MOSS is the Query String URL Filter Web Part this allows you to easily filter lists of data based on a parameter supplied in the URL. i.e. Filter a List of Invoices based on Client ID. I want to use WSS for more than just holding documents so I thought lets build one.

To deploy the Web Part solution from the Server run the following command.

STSADM.EXE -o addsolution -filename FilterWebPart.wsp

You then need to Deploy the solution from Central Admin Console.

Now on your site you need to activate the Feature

Next edit a page and add the "Query String Filter" Web Part

To Configure the Web Part you need to Choose the name of the URL Parameter that will be passed to your Filtered List i.e. ID or ClientID etc.

Now you just need to connect your web parts from a List or Document Library Web Part choose Connections -> Get Filter Values From to connect the parts together.

Now you should have something working like this.

Download Web Part from here

Download FilterWebPart.zip (4.1K)

SharePoint List Data Synchronization

info@simego.com (Sean Cleaver) — Tue, 15 Jan 2008 22:08:18 +0100

Been looking at an alternative to the BDC which is both easy to set-up whilst compatible with WSS as well as MOSS.

So building on our SQL Database Differences Technology we have been building a SharePoint (or any other datasource) Synchronization application framework.

The idea is that you can now leverage all the features of SharePoint to manage your business applications static data or ease the migration from an old application to SharePoint.

For Example with this Sync Application you can expose your bespoke applications Leaf Tables in SharePoint and then receive Security, Content Approval, Workflow, Versioning, Auditing etc on your data. i.e. You could maintain a list of Currency Conversion Rates for your application in SharePoint and then ensure only valid users can update the data and see a full log of who updated the values and when.

The Synchronization is two-way and column level so that if you update a column value in SharePoint then only that Column is updated in the SQL Database and the opposite is also true.

Were also planning for an Active Directory Adapter for the Synchronization to publish User Accounts/Network Computers etc to WSS to manage a Staff Telephone/Contact list etc.

If you would like to know more and have a project which could benefit from this technology then please drop me a line at sales[at]simego.com

MOSS BDC Design Studio Version 1.1 Released

info@simego.com (Sean Cleaver) — Fri, 14 Dec 2007 12:36:26 +0100

We are pleased to announce the release of an updated version of our MOSS BDC Design Tool. This latest update adds the following features :-

Wizards for Creating Entities
Support for Creating User Defined Actions
Updated User Interface with Docking Windows
Improved Method Auto-generation
Create BDC Connection to virtually any data source (Progress, Navision, DB2, Oracle and more...) via SQL Server Linked Servers and Entity Creation from SQL Statements

This is a FREE update to all existing customers and new customers can download a FREE 14-Day Trial version from the Web Site.

Download MOSS BDC Design Studio

Data Compare - SQL Admin Studio

info@simego.com (Sean Cleaver) — Wed, 14 Nov 2007 22:09:10 +0100

Coming soon is Data Compare for SQL Admin Studio if you would like to try out this great new feature you can download the latest BETA of SQL Admin Studio from

http://www.simego.com/downloads/sql_tools_setup_beta.exe

If you notice any bugs or have any suggestions on how it works then please drop me a line at support[at]simego.com

SQL Tools Running on Windows 64 Bit

info@simego.com (Sean Cleaver) — Fri, 26 Oct 2007 19:25:03 +0100

Whilst most features have been working just fine under Windows 64 Bit some features mainly export and import between Access, Excel and CSV do not. This is because Microsoft do not ship a 64 Bit version of the Jet database driver.

The latest beta version 3.0.53 now forces 32 Bit mode to ensure full compatibility on 64 Bit Windows.

download from here

To force an .NET Application to run in the 32 Bit environment on 64 Bit Windows run the CorFlags.exe Application from the .NET SDK.

i.e. corflags.exe /32BIT+ myapp.exe

Automate Testing of Database Change Scripts

info@simego.com (Sean Cleaver) — Thu, 18 Oct 2007 20:46:47 +0100

As part of your build process you can now use the Nant Tasks of SQL Admin Studio to automate a process similar to this.

Backup the Test Database
Generate the Change Script from the Current Development Database and the Test database
Apply the Change Script
Run NUnit Tests if you like
Restore the Test Database back to it's original state

So in this example I have been working on changes in Northwind2 and now I want to automate the process of creating and testing my updates for the Northwind database.

<?xml version="1.0" ?>
<project name="Test Database Release" default="Test" xmlns="http://simego.com/nant.xsd">

<loadtasks assembly="C:\Program Files\Simego\SQL Tools\Simego.SQLTools.Tasks.dll" />

Enable Visual Studio 2005 Nant Build File Design Integration

info@simego.com (Sean Cleaver) — Wed, 17 Oct 2007 10:11:31 +0100

The first step is to generate an Xml XSD Schema file that will be used to enable intellisense in Visual Studio. Run the following Nant Build Script to generate the schema file and copy the file to the Visual Studio 2005 Schema Directory.

<?xml version="1.0" ?>
<project name="Build Schema" default="build">
<loadtasks assembly="C:\Program Files\Simego\SQL Tools\Simego.SQLTools.Tasks.dll" />

<target name="build">
<nantschema output="Nant.xsd" target-ns="http://simego.com/nant.xsd"/>
<copy file="Nant.xsd" todir="C:\Program Files\Microsoft Visual Studio 8\Xml\Schemas" />
</target>
</project>

Then to enable intellisense in Visual Studio open the build file as Xml in Visual Studio and add the namespace to the project element like this.

<?xml version="1.0" ?>
<project name="Build Schema" default="build" xmlns="http://simego.com/nant.xsd" >
<loadtasks assembly="C:\Program Files\Simego\SQL Tools\Simego.SQLTools.Tasks.dll" />

<target name="build">
<nantschema output="Nant.xsd" target-ns="http://simego.com/nant.xsd"/>
<copy file="Nant.xsd" todir="C:\Program Files\Microsoft Visual Studio 8\Xml\Schemas" />
</target>
</project>

More Nant Tasks Added to SQL Admin Studio

info@simego.com (Sean Cleaver) — Tue, 16 Oct 2007 19:45:40 +0100

Just added tasks for Backup & Restore to the Task Library in SQL Admin Studio

This makes it really easy to copy a database with backup & restore. The restore task will also automatically remap the database physical files so you can just backup the database and restore it as a new database by changing the name.

For Example

<?xml version="1.0" ?>
<project name="Copy Database" default="copy">
<loadtasks assembly="C:\Program Files\Simego\SQL Tools\Simego.SQLTools.Tasks.dll" />

</target>
</project>

Using SQL Admin Studio Nant Tasks to Copy Database Tables Data

info@simego.com (Sean Cleaver) — Tue, 16 Oct 2007 15:06:17 +0100

This Nant Script Sample shows how to use the Nant Tasks within SQL Admin Studio to :-

Create a Table based on a Table in a different database
Import Data into that New Table from a different database and server.

<?xml version="1.0" ?>
<project name="Copy Northwind Data" default="CopyData">
<property name="networkLib" value="DBMSSOCN" />
<property name="srcServerName" value=".\SQLEXPRESS" />
<property name="srcDatabaseName" value="Northwind" />

</target>
</project>

SQL Admin Studio - Now Shipping with Nant Tasks

info@simego.com (Sean Cleaver) — Tue, 16 Oct 2007 11:40:29 +0100

The latest Beta SQL Admin Studio now includes several Nant Build Tasks for Creating Tables, Copying Data between Servers and Scripting Database Differences. This allows you to create Nant Build Scripts that can download content from remote hosted SQL Databases i.e. DiscountAsp.Net and import into a local database for backup or development etc.

I will be posting a few guides on how to use these Nant Build Tasks over the next few days, however if you can't wait then please download the latest build from http://www.simego.com/sql_tools_setup_beta.exe and check out the documentation in the the Nant Tasks Guide.

SQL Server 2008 CTP and SQL Admin Studio

info@simego.com (Sean Cleaver) — Mon, 15 Oct 2007 09:51:04 +0100

Early Testing shows that SQL Admin Studio works just fine with the CTP of SQL Server 2008 so now you can use SQL Admin Studio for SQL2000, 2005 and 2008.

Manage Database Releases with SQL Admin Studio

info@simego.com (Sean Cleaver) — Fri, 12 Oct 2007 17:00:16 +0100

How many times have you come to the end of a milestone in your project and need to create a Database Schema Change Script? If you have been really good then you would have very carefully recorded each database change you made in an update script ready to go. However what do you do about all the other developers in your team are they just as reliable? What will happen if you miss something? How many hours will you waste hunting down some bug that returns the stock answer ‘Works on my machine’.

SQL Admin Studio to the rescue – No longer do you need to track each database change just generate a Database Schema Difference Script between your development and staging databases. The update scripts will even preserve data if you have re-ordered columns in your tables.

New BETA of SQL Admin Studio

info@simego.com (Sean Cleaver) — Fri, 12 Oct 2007 12:50:49 +0100

I have been busy adding new features to SQL Admin Studio lately and now have a new BETA available

Support for Synonyms
Support either WinMerge or TortoiseMerge for viewing Database Object Differences
Updated Column Text Editor (ALT+V or F3 to view/update Text columns in new Window)
Database Differences includes SYNONYMs
Script Database includes SYNONYMs
View Server Log Files
Bug Fix updating Columns of type XML

To download this update http://www.simego.com/downloads/sql_tools_setup_beta.exe

Compare SQL Database Schema Tool

info@simego.com (Sean Cleaver) — Tue, 02 Oct 2007 11:37:31 +0100

A FREE tool to quickly and easily compare 2 database schema's and identify missing or changed objects.

Supports

Tables
Table Columns & Data Types
Views
Stored Procedures
Functions

Download Compare Database Tool

To install just download and unzip the files then run then CompareDB.exe application.

Script Database Schema Tool

info@simego.com (Sean Cleaver) — Tue, 02 Oct 2007 11:33:01 +0100

A FREE Tool to quickly and easily script your SQL Server database Schema.

Supports

Tables
Views
Stored Procedures
Functions

Download Script Database Tool

To install just download and unzip the files then run then ScriptDB.exe application.

Bulk Uploading Files to SharePoint

info@simego.com (Sean Cleaver) — Mon, 10 Sep 2007 19:03:57 +0100

It always struck me that bulk uploading GB's of Documents to SharePoint should be easier that it appears to be I am sure there are many solutions out there but today I decided to write my own. Well in-fact it's not too tricky all you need is a windows service to monitor a File Drop directory and any files or folders get eaten by the service as it uploads them to SharePoint. To upload the files you should implement the FrontPage Extensions API with this you can also handle the Check In/Out and set any meta data at the same time. One gotcha is that I can't get the FrontPage extensions to create a folder in SharePoint so I resorted to using the SharePoint Web Services for that.

Synchronising Favourites

info@simego.com (Sean Cleaver) — Wed, 29 Aug 2007 21:01:16 +0100

If your anything like me and have so many machines all over the place it can be hard to keep everything in sync. Recently we have started using Groove here at Contract5 to synchronise documents across machines and into MOSS. You can also configure Groove to synchronise existing folders and this works really well for your favourites... Cool Eh?

Windows Messenger cannot sign-in

info@simego.com (Sean Cleaver) — Wed, 29 Aug 2007 20:51:36 +0100

This has been driving me crazy! I have 2 DELL C521 Machines that I use in my office one with Virtual Server running and the other Vista and for some time I have be unable to get Windows Live Messenger to sign-in after a Driver Update from Microsoft killed my Wireless adapter. I thought it was because I was using one of those Ethernet over your home power lines so I dug up the garden and put in a direct cable but oh no. So next I thought lets buy some new network cards I wanted 1Gpbs LAN anyway but guess what still no luck. Then I eventually found the solution on the web via Google...

netsh int tcp set global autotuninglevel=disabled

So why didn't I think of that ... it's so obvious now.

MOSS BDC Design Studio - Updated

info@simego.com (Sean Cleaver) — Sun, 05 Aug 2007 13:32:19 +0100

Today I have released an update to MOSS BDC Design Studio this release version 1.0.19 improves on the support for different authentication modes within database systems. You can now easily configure either PassThrough, RevertToSelf, Database Credentials or Single Sign On directly from the properties dialog. This release also adds Search/Replace to the Xml Output to allow for easier direct editing of the Xml by hand.

Screen shot showing Database Credentials

MOSS BDC Design Studio - Updated

info@simego.com (Sean Cleaver) — Sun, 05 Aug 2007 13:32:19 +0100

Minor Update to MOSS BDC Design Studio released today this version 1.0.20 fixes a few minor bugs and improves on the ability to auto-detect filters and columns from stored procedures.

This is a FREE upgrade for all customers and can be downloaded from the web site now Download

Remove the Time Element from DateTime in SQL Server

info@simego.com (Sean Cleaver) — Tue, 10 Jul 2007 09:30:31 +0100

Today a customer of SQL Admin Studio asked how can I remove the Time Element of a DateTime Column or GetDate() function call.

Simple: Just like this...

DECLARE @Today datetime
SET @Today = cast(Floor(cast(GETDATE() AS float)) AS datetime)

SELECT @Today

Sample - Creating a Single Sign On (SSO) Provider for MOSS BDC

info@simego.com (Sean Cleaver) — Sun, 08 Jul 2007 15:04:03 +0100

When working with the Business Data Catalog you will find that using PassThrough or RevertToSelf authentication does not work because of the Double Hop issue of passing credentials to other servers. So the recommendation is to use the Single Sign On Service to authenticate.

This sample is based on the sample in the MOSS SDK but is completed so that it works this will show how you can write your own provider which can be simpler than setting up the default SSO. You will still need to store the credentials in some secure location but you should get the idea.

To install the provider you will need to compile the assembly with Visual Studio and copy it into the GAC the easiest way to do this is to just drag in into the C:\Windows\Assembly folder.

You will also need to modify the connection details of your Application Definition file to look something like this.

<LobSystemInstance Name="Test">
<Properties>
<Property Name="AuthenticationMode" Type="System.String">WindowsCredentials</Property>
<Property Name="SsoApplicationId" Type="System.String">Test2</Property>
<Property Name="SsoProviderImplementation" Type="System.String">Contract5.SingleSignOn.SSOProvider, Contract5.SingleSignOn, Version=1.0.0.0, Culture=neutral, PublicKeyToken=f8a51aa7d82e214d</Property>
<Property Name="DatabaseAccessProvider" Type="Microsoft.Office.Server.ApplicationRegistry.SystemSpecific.Db.DbAccessProvider">SqlServer</Property>
<Property Name="RdbConnection Data Source" Type="System.String">SERVER1</Property>
<Property Name="RdbConnection Initial Catalog" Type="System.String">Test2</Property>
<Property Name="RdbConnection Integrated Security" Type="System.String">SSPI</Property>
</Properties>
</LobSystemInstance>

MOSS BDC Studio is driven by a template so you could update the template file to include the new SSO settings. Look at the Template files in the Installation directory for more detail.

Download Sample

MOSS BDC Design Studio - USB Drive Edition

info@simego.com (Sean Cleaver) — Fri, 06 Jul 2007 11:44:57 +0100

I have just released a version of MOSS BDC Design Studio that can be copied onto a standard USB Drive and run directly from the drive. This option is ideal for SharePoint Consultants that need to use this tool at different client sites without the need to install the software directly onto the client machine.

System.Transactions, SQLHelper and MSDTC

info@simego.com (Sean Cleaver) — Wed, 04 Jul 2007 21:49:09 +0100

The new TransactionScope class in .NET 2.0 is great but if you use it with the standard SQLHelper class then you will find that your SQL Transactions are escalated to MSDTC unless you take some precautions.

Generally you will use SQLHelper to Execute some SQL Command and this function will create it's own connection run the command and then close the database connection. However as soon as you do this twice within a TransactionScope you will now get a MSDTC transaction which really should be avoided if your only working with a single database.

What is needed is to run each database command on the same SQL Connection this way the transaction is not escalated to a MSDTC transaction. So you need to build a new 'ConnectionManager' class that will track subscribers and manage the connection. To do this you will need to maintain a list of subscribers as WeakReferences in a Collection (or Generic List<WeakReference> ) then on the first instance you will open the database connection and on subsequent instance calls you will return the same connection. Then as the objects are Disposed you will remove the WeakReference from the Subscribers collection until it reaches Zero then Close the database connection. Do not store the Subscriber Collection or Database Connection in a static variable or you will be asking for trouble when it comes to multiple threads instead use ThreadLocalStorage to manage the Connection Lifetime.

Example Usage of TransactionScope and ConnectionManager

//Starts the Transaction
using(TransactionScope = new TransactionScope()) {

//Ensures that all Database Calls in this Thread use the same connection until this
//object is Disposed
using(ConnectionManager cm = new ConnectionManager(this.ConnectionString)) {
//Now Run your Business Logic Methods.
}

}

MOSS BDC Design Studio - Stored Procedures Part II

info@simego.com (Sean Cleaver) — Sun, 24 Jun 2007 14:58:20 +0100

MOSS BDC Design Studio now has full drag & drop support for designing BDC Xml with Stored Procedures in-fact all you have to do is drag the Stored Procedure from the Database Schema Window onto the Design Surface and Give the Entity a Name and your done! You can even specify what procedures map to Finder, SpecificFinder and IDEnumerator and all the input parameters, return columns etc will be created automatically.

This is now in version 1.0.15 which is ready to download now.

SQL Tools - for USB Key

info@simego.com (Sean Cleaver) — Wed, 20 Jun 2007 19:46:06 +0100

Today I have released a new version of my SQL Tools Suite that includes SQL Admin Studio, SQL Import Studio and SQL Backup & Restore that will run from a USB Key and will not install any files onto the local machine. This is ideal for Consultants that may be required to work on many different sites and installing the application locally may not be possible.

To install copy the files in the Zip archive to your USB Key and then run each program and go through the activation process to install a license key for your USB Drive.

Simego.SQLTools.Explorer.exe
Simego.SQLImport.exe
Simego.Backup.Wizard.exe

Now when you plug your USB Key into another machine you will not need to re-activate the products.

All registered customers can download this version now.

Using Stored Procedures with MOSS BDC Design Studio

info@simego.com (Sean Cleaver) — Wed, 20 Jun 2007 13:53:11 +0100

So you want to use Stored Procedures to return your data then eh? Well no problem MOSS BDC Design Studio can do just that all you need do is use the Create Entity from SQL function which allows you to create an entity from any SQL Command i.e. Stored Procedures, SQL that join's multiple tables, Linked Servers (Ah! Navision, Oracle etc) etc.

View the step by step Walkthough on my Website.

MOSS BDC - Web Service Walkthrough

info@simego.com (Sean Cleaver) — Sat, 16 Jun 2007 16:36:23 +0100

I have posted a new walkthrough on my Web Site that shows how to connect the sample Northwind Web Service to MOSS via the BDC.

http://www.simego.com/walkthroughs.aspx

Automate Scripting your Database with SQL Admin Stuido

info@simego.com (Sean Cleaver) — Thu, 14 Jun 2007 11:01:00 +0100

So you want to easily script your database as part of your development life cycle? With the latest build 3.0.30 of SQL Admin Studio you can now Script the database from the command line.

Script Database

Simego.SQLTools.Explorer.exe -action scriptdatabase -server dev1 -database northwind -outfile "C:\Northwind.sql"

Script Database Differences

Simego.SQLTools.Explorer.exe -action comparedatabase -sourceserver dev1 -sourcedatabase northwind -server dev1 -database test1 -outfile "C:\Northwind_Compare.sql"

Run Template

Simego.SQLTools.Explorer.exe -action runtemplate -template "procedure for INSERT" -server dev1 -database Northwind -outfile "C:\Northwind_Template.sql"

With run template you can also pass

-table the table to pass to the template

-view the view to pass to the template

-procedure the procedure to pass to the template

Download Latest Build

Advanced Templates with SQL Admin Studio

info@simego.com (Sean Cleaver) — Tue, 12 Jun 2007 10:25:16 +0100

A Client was writing their own templates for SQL Admin Studio and asked if it was possible to build a Template that could ultimately generate the entire data access layer for their database project. Generally the templates execute against a single database object but with a little work it is possible for a template to call other templates you just need to compile each template before you call it. I had to move a few things around and create a few helper functions in SQL Admin Studio so for this to work you will need the latest beta release version 3.0.29 from http://www.simego.com/downloads/sql_tools_setup_beta.exe

To Compile a Template you need to call the TemplateCompiler and pass the Template filename

i.e. StreamWriterTemplateBase tableTemplate = TemplateCompiler.CompileTemplate(Path.Combine(TemplateCompiler.TemplateDirectory, @"MsSqlServer\SqlTable\C#\Entity Object C#.tpl"));

Then to Execute a Compiled Template you need to Call it’s RenderTemplate method.

i.e. tableTemplate.RenderTemplate(outputFileName, Database, T, null, null);

The sample Data Access Layer script will Create Entity C# Objects for each Table in your database and Create a Single Database.sql script file with all the SELECT/INSERT/UPDATE/DELETE Stored Procedures for each Table in the Database.

Download Sample Template (0.9K)

To install the Template into SQL Admin Studio copy the Template file from the Zip Archive to your
My Documents\My Templates\MsSqlServer\SqlDatabase\C# folder.

MOSS BDC Design Studio - Updated!

info@simego.com (Sean Cleaver) — Sat, 09 Jun 2007 18:44:34 +0100

Today I have released a minor update to my MOSS BDC Design Studio to improve the Web Services support.

You can now see more information in the Schema Explorer window Method Input Parameters, Return Types and Entity Property data types.
Fixed an Issue where the Web Service proxy was not created if you had a Web Method that referenced a DataType in System.Data.
Web Service Proxy is no longer created twice.

Create a MOSS BDC Application in 2 Hours

info@simego.com (Sean Cleaver) — Fri, 01 Jun 2007 16:14:13 +0100

Today I thought now lets just see how long it would take to build a Demo Online Banking Application using Microsoft Office SharePoint Server(MOSS) and the Business Data Catalog(BDC). So I created a sample Banking database and pulled in some data from the Northwind database so I had a few customers to work with. The aim was to provide the following features :-

Search for Client
Display Client Details
Display Client Bank Accounts with Balance Summary
Drill Down into the Bank Account
Display Bank Account Details
Display Transactions for the Account

This time I decided I was going to use the SharePoint Designer rather than just web parts so I went to work creating the BDC Entities with MOSS BDC Design Studio that took 15 minuites. Then I opened up SharePoint Designer and went to work.

This is the result.

Client/Customer Search (By Name or Client ID)

Client Desktop (Displaying Client Details as Bank Account Summaries)

Bank Account (Displaying Bank Account Details and Transactions for the Account)

Now how cool is this it literally took me no more than 2 hours start to finish to create a working demo using MOSS and the Business Data Catalog.

MOSS BDC Design Studio Updated

info@simego.com (Sean Cleaver) — Thu, 31 May 2007 09:51:56 +0100

Just updated MOSS BDC Design Studio to resolve an issue with creating entities from SQL or using Stored Procedures. This fix is in version 1.0.8 you can download this from my web site now.

To Use Stored Procedures

1) Create the Entity via the Tools menu 'Create Enitiy from SQL' enter the Stored Procedure with parameters so that it will return i.e GetProducts '%MyProduct%'.

2) Then edit the Finder Method and Add parameters that match the Stored Procedure input Parameter.

3) Edit the Finder method Properties and Change the Command Type = StoredProcedure and the CommandText so that it matches the Stored Procedure name.

Walk Through Using MOSS BDC Design Studio

info@simego.com (Sean Cleaver) — Wed, 30 May 2007 12:13:46 +0100

I have just posted a walk through guide on my website that shows how to use the MOSS BDC Design Studio to connect to the Northwind Product Table and create a Finder method to filter the results by Product Name.

View the article here

How to connect to Navision 3.7 with the BDC

info@simego.com (Sean Cleaver) — Wed, 30 May 2007 09:00:00 +0100

The Business Data Catalog allows us to connect to a variety data sources and expose this data in MOSS with the BDC Web Parts. By using Linked Servers in SQL Server we can easily extend the capabilities to other Database Systems such as DB2, Oracle and even the Navision database.

To connect to the Navision 3.7 Data Store with the Business Data Catalog you will need to setup a linked server in SQL Server 2005.

1. Install the C/ODBC Database Driver for Navision on the SQL Server that you plan to use with the BDC.

2. Define a System ODBC Source for the Connection to the Navision Database on the SQL Server.

3. Add a Linked Server to SQL Server

EXEC master.dbo.sp_addlinkedserver
@server = N'NAVISION',
@srvproduct=N'NAVISION',
@provider=N'MSDASQL',
@datasrc=N'NAVISION'
GO

4. Test the Connection with a suitable query tool such as SQL Admin Studio.

DBCC TRACEON(8765)
SELECT * FROM OPENQUERY(NAVISION, 'SELECT * FROM Customer')

5. Then by using MOSS BDC Design Studio you can create Entities for the BDC from SQL Commands i.e. From the Tools Menu ->Create Entity from SQL Command

Enter the following SQL Command and name it Customers you can then generate the BDC Schema.

DBCC TRACEON(8765)
SELECT * FROM OPENQUERY(NAVISION, 'SELECT * FROM Customer')

6. Generate the BDC Xml Application Definition and import into the BDC.

Export Data to Microsoft Excel

info@simego.com (Sean Cleaver) — Tue, 29 May 2007 14:39:53 +0100

I have had a few customers ask me how I export data to Excel in SQL Admin Studio and the answer is no secret. Under the hood I leverage the Microsoft ODBC Driver for Excel and write directly to that as if it was any SQL Database.

Download ExcelHelper2.zip (1.3K)

WebService Sample for MOSS BDC

info@simego.com (Sean Cleaver) — Tue, 29 May 2007 14:20:52 +0100

This code demonstrates how to simply create a web service that can be exposed via the Business Data Catalog (BDC) in Microsoft SharePoint Portal Server (MOSS).

This method allows you to integrate virtually any kind of system into MOSS with the BDC alternative methods involve using Linked Servers with SQL Server to connect to other Data sources such as Navision, 4Series AS400, DB2 etc.

Download NorthwindService.zip (48.8K)

SQL Admin Studio – Updated!

info@simego.com (Sean Cleaver) — Tue, 29 May 2007 14:01:00 +0100

Today I have uploaded the latest release of SQL Admin Studio Version 3.0.26

New Designer for creating Templates (Templates allow you to create custom Code, SQL etc from Database Schema)
Queries now run on a separate thread so they no longer block the UI.
Product Activation changes – You can now activate directly over the internet from the application itself.
SQL Import Studio is now in the same install image.
SQL Backup & Restore Wizard is now in the same install image.

The latest version is available here http://www.simego.com/download.aspx

Also Complementary licenses are available for good Blog/Web Site Review Articles.

Microsoft Office SharePoint Server Business Data Catalog Design Tool

info@simego.com (Sean Cleaver) — Tue, 29 May 2007 09:53:13 +0100

Today I have released a software tool to simplify the development of MOSS BDC Xml files to connect remote database and web service systems. This tool is currently in Beta and we invite you to try the application and in return for good Blog/Web Site Review articles we will provide complementary licenses for the Full Version.

MOSS Business Data Catalog Design Studio – Version 1.0 Beta

This is the first Beta Release of a tool to simplify creating BDC Schema for the MOSS Business Data Catalog. In the first beta version the following features are supported.

Connect to SQL Server Database
Connect to Web Services
Create multiple Entities from Table
Create multiple Entities from View
Create multiple Entities from SQL Command Text
Create multiple Entities from Object (Web Services)
Add/Edit/Remove Methods (Finder, SpecficFinder, IDEnumerator)
Add/Edit/Remove Method Parameters
Add/Edit/Remove Method Properties
Add/Edit/Remove Method Filters
Add/Edit/Remove Entity Associations
Save/Open Project Files
Edit Column Names
Change Column Order
Set Title/Key Columns
Generate BDC Schema

Supported OS: Windows XP SP2/Windows Server 2003 SP1/Vista

.NET Framework 2.0 Required

To download the latest beta please go to www.simego.com this version will expire on July 21^st 2007.