#IoT2CRM: Time to get connected – part 2: Concepts of the Gateway

In the article #IoT2CRM: Time to get connected – part 1: Intro, I wrote about writing a demo scenario in which I would connect multiple Raspberry Pi’s to Dynamics CRM. In the scenario I described, I mentioned a gateway.

This article will describe the concepts behind the gateway, in my opinion the most important part within the integration.

For those who wonder why I keep hammering on having a gateway, I’ll highlight the most important benefits:

  • scalability
    CRM is not designed to handle an enormous amount of simultaneous service requests. Setting up a single connection can take forever. Furthermore we don’t know how many devices will setup a connection to our CRM system.
  • security
    By providing a gateway, we take away the direct connection between the device on the web and CRM. We don’t need to provide any account information. The Gateway will pick up the messages and passes is through to the CRM system (handling all security etc).
  • flexibility
    The gateway allows us, to change the CRM implementation without affecting the contract between the device and the gateway.

I can hear some of you thinking “why the hell do you want to change the CRM implementation? We want Dynamics CRM!”

The lifespan of devices is probably going to be a long one, at least for a large number of them. IoT devices are going to be implemented in e.g. kitchen appliances or cars. The technical lifespan of e.g. cars has an average of 20 years, kitchen equipment 12 to 15 years). It is more llike that the lifespan of our CRM system is shorter than the lifespan of the device out there.

We want to be able to serve the device thoughout it’s entire lifespan, therefor it is important to be really flexible. We must be able to change the implementation of the CRM system wihout affecting the connectivity to the devices out there. We don’t know if Dynamics CRM will be around in 20 years time.

Design considerations
In my design, I’ll look at concepts used in enterpise scale message brokers like BizTalk. BizTalk is a solution that easily can scale up, that is flexible and that guarantees that messages won’t get lost.
On the other hand I’m working on a tight budget (and I believe, many companies out there too and BizTalk is not one of the cheapest solutions out there), but I want a robust solution without messages getting lost.
Another consideration is that I always strive to deliver a maintenance free/low solution. I really love the appliance concept (fire and forget). My solution has to be as maintenance free as possible.

The concept
What came across my mind was the following high level concept:


We have this massive amount of messages sent by the IoT devices out there. These messages will be picked up by our (MVC based) Web API controller hosted on IIS (which can be load balanced if required).  As I want to use REST/JSON for the communications between the devices, the WEP API controller is in my eyes the way to go.
The received messages will be put in a temporary buffer. The buffer is a folder structure on disc (this can be a NAS, local server whatever). On the computer/network where the messages will be buffered there are going to be one or more Monitoring Services.

The task of the monitoring service is to pick a message from the buffer, transform it into a CRM call and send it out to CRM. The idea is to have one or more Monitoring services that will work with parallelism in order to send the data as fast as possible to CRM. The .Net framework allows us to use parallelism (MSDN). The advantage of using the parallel tasks of the .Net framework is that we don’t have to build our own multithreading.

Now suppose the CRM system is not available (it gets an overhaul, maintenance, whatever), the messages from the devices can still be received using the Web API controller. The only task of the controller is to receive and store the message.
This simple principle actually guarantees that we can always receive messages from devices out there (we just cannot guarantee how long it will take before a message gets processed). By breaking up the gateway into multiple components we get the ultimate flexibility.

In the next article I’ll implement the gateway…