Dynamics CRM Online 2011 to 2013 API Import Performance

If you have upgraded your online instance to Dynamics CRM 2013 you may come across a reduction in performance when using the API to integrate external systems.

Historically, Dynamics CRM Online (2011) has had the ability to 'throttle' the amount of data being passed through the API but from our experience this has never been used. Dynamic CRM Online 2013 has throttling in place which can affect integration performance.

To get a better idea of how an much of a performance difference this can make we are going to compare insert/update and delete performance of Dynamics CRM Online 2011 vs 2013

Conservative Performance

We created a file containing 50000 (49996) cheque records with minimal fields:

• Recipient
• Amount
• RequestId

And we are going to insert it directly into a custom entity

50k Record Performance at 9:30 am EST with Simego DS3

*Run on DELL Desktop with SSD within a VM with 8Gb of memory

Deep Dive into Dynamics CRM Online 2013 Throttling

It has long been accepted that to achieve performance at the above 50/s level you will have to employ the following technologies:

• Parallel Loading
• Re-Use of Connection
• Re-Use of Authentication

These three techniques have worked well against Dynamics CRM Online and 2011... until now.

The Dynamics CRM 2013 SDK documentation states: Throttling of concurrent calls - for Microsoft Dynamics CRM Online there is a limit of 2 concurrent ExecuteMultipleRequest executions per organization. If that limit is exceeded, a 'Server Busy' fault is thrown before the first request is ever executed. For an on-premises deployment, throttling is not enabled by default.

Deeper down the message stack this throws an error related to the SQL Governor.

Essentially, if your integration solution attempts to create a third concurrent connection: GAME OVER. BUT this is configurable by Microsoft at an Organization level and you can request an amendment after discussions with the MS Online team:

https://community.dynamics.com/crm/f/117/t/121248.aspx

If you dig deep enough and experiment long enough there is an interesting behaviour.

Throttling will not engage if the batch size is 10 or below. This means you can have as many parallel connections as you like without getting the 'Server Busy' error.

Here is the accompanying video demonstrating the difference in performance of many threads with small batches.

You can investigate these performance numbers yourself by trying DS3 for free.