It’s possible to setup a near real-time bi-directional SharePoint List replicator between different SharePoint Sites/Farms etc with Ouvvi and Data Synchronisation Studio. Additionally there is nothing to install into your SharePoint environment so it can also work against hosted SharePoint or SharePoint which is locked down by your admins.

Ouvvi 2.0 has a new Trigger for SharePoint Lists which fires whenever items in the list change therefore if you configure a bi-directional Data Sync project and load it into Ouvvi and assign the triggers to the project you can build a replication system.

Here we define a new project in Ouvvi and create a single step which is the 2-Way Data Sync project. (You need to know how to build the 2-Way project with DSID replication columns).

We then define 2 triggers to query each list for changes, by default the triggers are executed every 30 seconds so you should expect the project to be kicked off within 30 seconds of a change (this is what I mean by near real-time).

You then assign the 2 triggers to the project so that whenever each one fires the project is started.

To prevent the project running sync jobs in parallel we define the project as Single Instance this way each start request is queued and only one instance is ever running.

You can also see in the logs when the sync runs how many items were changed.

And view real-time reports of the execution process.

Google provide several public holiday feeds such as this one here

http://www.google.com/calendar/ical/en.uk%23holiday%40group.v.calendar.google.com/public/basic.ics

Now with the Data Generator Data Provider in Data Synchronisation Studio 2012 and Dynamic Columns it’s easy to turn this into a Data Source.

The first step is to connect Data Source A to the Data Generator.

The Data Generator simply returns a number between 1 and the number you set in the properties.

All the magic happens in the Dynamic Column code where we override the Setup() method and load the Data from the Google service into a List of Events. Then for each Row Data Sync calls BeginRow() and we simply return whether there are any more events and return the calendar data as properties of the Row.

#region Usings
using System;
using System.Collections.Generic;
using System.Data;
using System.Globalization;
using System.Text;
using System.Linq;
using System.IO;
#endregion

class DataSourceRowOverride : Simego.DataSync.DynamicColumns.DataSourceRowInternal //Do Not Change This Line
{    
    private List<dynamic> Events = new List<dynamic>();
        
    public DateTime StartDate { get { return Events[ID-1].Date; } }
    public DateTime EndDate { get { return StartDate.Add(new TimeSpan(23, 59, 0)); } }
    public string Title { get { return Events[ID-1].Title; } }
    public bool AllDayEvent { get { return true; } }
    
    public override bool BeginRow()
    {        
        return ID <= Events.Count; // return false to remove row from results.
    }

    public override void Setup()
    {
        var request = System.Net.HttpWebRequest.Create("http://www.google.com/calendar/ical/en.uk%23holiday%40group.v.calendar.google.com/public/basic.ics");
                
        using(var reader = new StreamReader(request.GetResponse().GetResponseStream()))
        {            
            DateTime date = DateTime.MinValue;
            string title = null;
            string line = null;
            
            while((line = reader.ReadLine()) != null) 
            {                        
                if ( line.StartsWith("DTSTART") )
                    date = DateTime.ParseExact(line.Substring(19), "yyyyMMdd", CultureInfo.CurrentCulture);
                
                if ( line.StartsWith("SUMMARY:") )
                    title = line.Substring(8);
                
                if ( line.StartsWith("END:VEVENT") ) 
                    Events.Add(new { Date = date, Title = title });
            }
            
            //Order the Events by Date
            Events = Events.OrderBy(p => p.Date).ToList();
        }
    }
}

And there you have it, Calendar entries from a Public Calendar feed that you can import right into a SharePoint List.

Download Sample Project File

Need hundreds, thousands, millions, hundreds of millions?! of rows of test data? Well you probably didn’t think to use Data Synchronisation Studio. Why would you, it’s a synchronising tool right?

Well, yes, this is true. However, DS 2012 now includes a new provider that can generate as many rows as you need. Utilising this, we created a test data generator using dynamic columns. It can create you an entire user base in seconds!

If you have used Data Synchronisation Studio before, you will be familiar with data source sides A and B. You may also be familiar with the term Dynamic Columns.
This test generator works by creating a faux data source on side A using Dynamic Columns. You can then use this to sync to SQL, SharePoint etc. to give yourself an instant user base for testing.

Do you have Data Synchronisation Studio 2012 (v3.0.500) or above?

Yes I do, Download Test Data Generator Project
No, I need to Download the Latest Version of Data Synchronisation Studio

Features

• Generates random name combinations
• Generates dates of births
• Chooses from five different countries
• Chooses a region based on the country
• Chooses a city based on the region
• Generates random road names
• Creates phone numbers based on the country’s number format
• Creates user identification codes
• Generates email address based on the users names
• Creates a password hash
• Generates ‘Date Joined’ ‘Previous Login’ & ‘Last Login’
• Randomly selects a security question and answer
• Creates a security PIN

Columns

• ID (Int32)
• UID (Guid)
• UserCode (String)
• FirstName (String)
• MiddleInitial (String)
• LastName (String)
• DateOfBirth (DateTime)
• HouseNumberName (String)
• AddressLine1 (String)
• AddressLine2 (String)
• TownCity (String)
• Region (String)
• PostalCode (String)
• Country (String)
• PhoneNumber (String)
• MobileNumber (String)
• WorkNumber (String)
• Email (String)
• PasswordHash (String)
• Role (Int32)
• DateJoined (DateTime)
• PreviousLogin (DateTIme)
• LastLogin (DateTime)
• AccountStatus (Int32)
• SecurityQuestion (Int32)
• SecurityAnswer (String)
• SecurityPin (Int32)
• NewsletterSubscription (Boolean)
• AgreedToTerms (Boolean)
• Other (String)

How do I use this generator?

1. Make sure you have the correct version of Data Synchronisation Studio, as this project will only be supported by version 3.0.500 and above.
   
   
2. Download the project, using the link above
3. Open the project, making sure you are connected to the internet. This is so the project can download the names database (only happens once)
   
   
4. Set the row count to the amount of rows you would like to create and set the seed to zero for completely random data.
   
   
5. Click the ‘Preview A’ button to check the data (optional)
   
   
   This is optional, because it can take a few minutes to generate the rows. This depends on the machine spec and the row count chosen. Below is a basic guide to timings.
   
   10,000 = 1.5s
   100,000 = 12s
   1,000,000 = 2m
6. Now that you have setup the generator to your liking, we can go about inserting the data into a usable SQL table. You can choose other data sources, but for this post we’ll use SQL.
   Go to the tools menu and select create SQL Table.
7. Follow the instructions of the SQL Table Wizard
8. Once you’ve created your new SQL Table, Click ‘Compare A->B’
   This will compare the generated data with your new table. Obviously your new table is empty, so once the sync has completed you will see a list of items to add to SQL.
   
   
9. Click the ‘Synchronise’ button and then click ‘Start’
   This will fill your SQL Table with all of your new generated data. You now have a test database to work with!

We are pleased to announce today that we have shipped the first beta release of Data Synchronisation Studio 2012.

DS2012 has had a major overhaul to the underlying storage engine that was used to load/store and compare the data. We now have an over 90% reduction in memory usage and in some cases as much as 80% performance increases. It is now possible to use Data Sync with large data sets consisting of millions of rows.

Additionally the Dynamic Column feature has also been overhauled to ensure that the calculation occurs during the row load rather than after all rows are loaded. This allows for Partial load and Data Preview of large data sets.

We have a new exciting Data Provider for generating Test Data (Blog post to follow on this).

As always this update is free for our customers that have an existing maintenance subscription.

You can use Data Synchronisation Studio with Firebird via the ODBC connection provider and the Firebird ODBC Driver.

Ensure you have the Firebird/InterBase ODBC Driver installed and if you have Data Sync 64 bit edition you also have the 64 bit version of the ODBC Driver.

Select the ODBC Driver and enter the connection string to Firebird like this.

DRIVER=Firebird/InterBase(r) driver;UID=SYSDBA;PWD=masterkey;DBNAME=C:\Temp\EMPLOYEE.FDB

Then select the Object in the database or switch to the SQL Tab and enter your own SQL.

Once the Schema Loads you can start defining your Project.

Or just preview the Data to check the connection.

Data Synchronisation Studio has a new feature to make it unbelievably easy to link data by way of Lookup Columns. A column can now link to any other data via a special Lookup column, you could do this with Dynamic Columns before but now it’s just click…click…click.

You can connect SQL Data to SharePoint data for example or Join SQL Data across servers like SQL and Oracle or whatever you get the idea anything to anything.  

To get started connect your data source as before and select the column that represents the Link or Join

Then click the Icon in the Data source Toolbar to Add a Lookup

Once you configure the connection details you need to confirm the Lookup Join details which is essentially the Data Type and Column.

Now once the Lookup loads you will see all the columns from the related Data source listed and you can add any of them to the Schema Map as if they were already in the source.

That’s it really easy lookup columns in Data Synchronisation Studio, this feature is in release 3.0.444

For the Data Sync Exchange Providers to read Calendars,Contacts or Tasks from other mailboxes you will need to add Reviewer permission to the mailboxes for the user that is used in Data Sync. Fortunately you can do this with a PowerShell script like this.

Add-MailboxFolderPermission -Identity ayla@contoso.com:\Contacts -User Ed@contoso.com -AccessRights Reviewer

-Identity is mailbox and folder your targeting and –User is the user to grant access.

This blog article describes how you might synchronise Documents between SharePoint 2007 and SharePoint 2010 with Data Synchronisation Studio including custom Metadata columns whilst preserving the Created and Modified Timestamps and user accounts.

Here we have a Document Library in SharePoint 2007 with a few Documents and a Folder where going to Synchronise this to a new SharePoint 2010 document library.

Here’s the Empty Document Library in SharePoint 2010 which will be the Target

To define the project in Data Synchronisation Studio we need to use the SharePoint provider for SharePoint 2007 and the new SharePoint Client API provider for SharePoint 2010.

Connect Data Source A to the SharePoint Provider

Connect Data Source B to the SharePoint 2010 Client API Provider

Configure the Schema Map like this and add any other meta data columns you also wish to synchronise.

Ensure that the Document Library Update Behaviour is set for FileAndMetaData on each side to that the Documents are uploaded.

Run a Compare A->B and Synchronise the Documents

And now the documents have been synchronised including the Created/Modified Timestamps and User accounts.

As the number of Providers has increased that Data Synchronisation Studio now supports it was getting increasingly harder to locate the provider in the list. Therefore we have now re-worked the Connection Dialog into groups and allowed you to select your own Favourites which are easy to find under the Favourite Tab.

This update is in release 3.0.434

We have created a new Data Synchronisation Studio Provider that allows you to synchronise the Members in a SharePoint Security Group. You may want to use this to keep the Members in sync with Members in an Active Directory Group.

This new Provider is in release 3.0.432 of Data Synchronisation Studio.

Example

Synchronise Members from AD Group to SharePoint Group

Connect Data Source A to Active Directory

In this example we use a LDAP Filter to return only those users of a specific AD group, for example

(&(objectClass=User)(memberOf=CN=SharePoint Team Users,CN=Users,DC=corp,DC=litware,DC=inc))

Connect Data Source B to the Microsoft SharePoint Security Groups provider

Choose the SharePoint Group to synchronise from the dropdown.

Adjust the schema map like this so that the AD Account Name is mapped to Login name and Display Name is mapped to Name. Account Name should be the Key Column.

Your completed project will look like this.

Running a Compare A to B shows the differences and you can now synchronise the member lists.

And now after synchronisation in SharePoint the Group Members is populated.

Now that the users have been correctly added to SharePoint you can also setup a standard SharePoint List sync with the User Information List to populate other meta data from AD into SharePoint.