In the previous article I described how I got sparked to investigate how I can create a bi-directional replication scenario in which I can synchronize data back and forth between entities.
For the sake of simplicity I’ll focus initially in synchronizing the data within a sinlge CRM organization. Later on I will describe how we can set up a mechanism that can replicate between CRM organizations either on-line, on-premise or in a hybrid mix. The only limitation is your imagination…
Before I come to that stage I’ll focus on some basics, and come with an solid answer to the question: “What is replication?”.
If we look at the definition in the Cambridge Dictionary, we can find the following definition:
[FORMAL] to make or do something again in exactly the same way:
[SPECIALIZED biology, computing] If organisms and genetic or other structures replicate, they make exact copies of themselves:
To put it in the context of this article: “Replication is a process in which an exact copy of data is made.”
This means that every change in the source entity should be reflected in the destination entity, this can be for all data fields or for a subset of the fields. In our case we will stick to a subset of relevant fields that we want to be replicated.
This answer raises a couple of questions:
- How do we guarantee that all changes are reflected?
- How can we design it for performance?
- How can we ensure that all changes will be replication in case of a failing data connection?
Back in the days when replication was introduced in SQL Server, I remember that a specialized database was created in which all changes were stored temporary. The changes were picked up a specific intervals and sent to a destination SQL Server. Once the change was sent safely, the change was removed from the specialized database.
In my mechanism I want to implement a mechanism like this, by storing the changes in a custom message entity we can queueu the messages and process it whenever is appropriate.
The messages are processed by using a FIFO mechanism (first in first out). When storing the changes in the custom message entity, we can specify what kind of change the message is (e.g. create, update, delete).
This would result in the image below. We have an entity in which we add, update or delete records. Every change results in a change message that is stored in a custom message entity.
This process can be straight forward. On an entity we can define a couple of asynchronous plug-in steps that react to create, update or delete messages. The message that is written can contain the following information:
Field containing the date and time of the action
Action of the message (create, update, delete)
- Regarding ObjectId
Entity reference to the source entity
Field containing the serialized data of the source entity (or from subset of fields)
Once we stored this data we can create a message pump. A message pump is a process that will read the message entity continuously in order to get the change messages.
The change messages are read in the same sequence they were added to the message entity. Once a change message is read by the message pump, based on the MessageAction the changes are reflected on the destination enity. Resulting in the picture below.
This covers the basics for setting up a simple one way replication pattern. What I didn’t describe is that I need to specify somewhere what the routing is going to be. This means that I need to specify where data from a certain entity is going to be transferred to.
Furthermore I want to be able to specify just the set of fields I’m interested in. I also need to define what to do in case the replication might fail…
I’m getting there but not quite yet. In the next article I’ll discuss the loose ends and from there on I can describe what it takes to create a mechanism that will enable bidirectional replication as the mechanism I described above will cause a replication storm in case of bidirectional replication.
Sound like fun to me!