Food for thought: A scenario for handling massive amounts of data – part 2

In my last article I described a scenario for handling massive amounts of data. The scenario intrigued me. How could I implement a scenario like this?

On one side you find Dynamics CRM –  a massive online environment. On the other side you find Azure with an ever growing and maturing cloud platform. The challenge we are facing is to move data from one (cloud) platform to another.

In this article I’ll describe how I would implement this scenario.

Ats1

Under normal circumstances moving data is not a real challenge, however when we want to initiate this from Dynamics CRM we are bound to a very strict set of limitations.
The biggest limitation is the two minute execution limit before a sandbox process is terminated. Especially when moving large chunks data from one platform to another platform this limit will be a show stopper.

Fortunately, the CRM development team realized this and implemented Azure awareness inside the Dynamics CRM SDK to allow the execution context to be posted to the Microsoft Azure Service Bus. This allows us to register our plugin to be executed at an Azure end point.

Once a plugin is running in the Azure context, the two minute execution limit is gone.

On MSDN you can find a couple of great postings describing how to write Azure aware plugins. Although these articles have been written for CRM 2016, they can be applied on CRM 2015 (online and on-premise) as well.

Offloading data

The question that arises is: “With the knowledge of the articles above (and the blog post mentioned in the previous article), how would I offload the data from CRM?”

Ats3

We cannot offload the data from CRM directly, we need to have a process outside of CRM. This can be achieved by registering an azure aware plugin in the Azure context. The external plugin needs to be executed in an asynchronous context, as the execution will exceed the execution limit of two minutes.
The disadvantage of calling the plugin directly is that there is no feedback to CRM, e.g. progress, status etc.

A way to overcome this is by implementing a custom action on CRM. The custom action writes information to a custom enity about the job that is going to executed.
By creating the record in the custom entity, an asynchronous post create plugin is triggered that calls the external registered plugin; passing the execution context to Azure.

The Azure plugin starts the process, fetching the data from CRM, transforming the records and write them to the Azure Table Storage. And Last but not least it writes status information back to the custom entity from which the job was started.
Because the Azure hosted plugin has the execution context from CRM, we can use all of its services using standard API calls as if we were developing an ordinary plugin.

By reading the custom entity in CRM, we can see what the status of the job is, if there were any errors, how many records have been processed etc. (I’ll leave that part to your imagination). Giving us all the feedback we need.

One thought on “Food for thought: A scenario for handling massive amounts of data – part 2

  1. […] my last article I described a scenario in which large amounts of data were moved from CRM to Azure Table Storage. […]

Leave a Reply

Your email address will not be published. Required fields are marked *