My thoughts on the performance issue

At this moment we are having a Dynamics CRM online performance issue at one of our clients. In a joint effort with Microsoft we are trying to find the cause of the performance issue. As a backup scenario our client asked us to set up a full blown on-premise environment, which is –  as we speak –  up and running.

Speedometer

Our client is a world wide organization with offices in Europe and Asia and the head quarters based in the Netherlands. The asian office happened to have its own Dynamics CRM subscription.

This gave us the unique opportunity to test the solution we developed on a different tenant in a different CRM region…

The results were shocking:  One the left side we have the European tenant (CRM4) which is slow and unstable, on the right side we have the Asian tenant (CRM5) which is fast and stable despite the fact that we have to cross half the globe to access the data.

At this moment we have two environments (on-prem and crm5.dynamics.com) which are fast and show a consistent performance, and we have one environment (crm4.dynamics.com) which doesn’t have a consistent performance. To me this proves that the solution we built doesn’t have influence on the performance of Dynamics CRM.

My gut feeling tells me that we have an issue on the crm4.dynamics.com environment. I hope the Microsoft engineers can find the root cause.

In the investigation I did, I noticed that in order to call a page (e.g. Dashboard) from CRM an enormous amount of web requests are made to the Dynamics webserver. An average page has between 140 and 240 web requests. Furthermore the amount of data being transferred is big. A page can take a couple of megabytes of Html code, javascript, css files, images etc. This has a serious impact on the rendering experience in the browser.

Doing some investigation on an on-premise CRM environment (that I hosted in a virtual machine), I noticed that the w3wp.exe processes (these are the worker processes of Internet Information Server) that serve the Dynamics pages quite often have extreme high cpu loads for a longer period of time.
Can this be one of the causes of the dodgy performance we experience?

I suspect that due to the enormous amounts of webpages being served by the crm4.dynamics.com environment, the web frontend servers can’t keep up and literaly start to choke.

loads of organizations  x  loads of pages  x  100s requests per page x  couple of MB per page = webserver too busy!

I know you can’t compare the infrastructure of CRM on-premise with CRM online as a CRM online farm contains loads of load balanced servers. But a tiny voice in my heads keeps nagging…

This made me think of what could be changed within Dynamics CRM to improve the overall user experience, e.g.

  • The Turbo forms engine was already a big improvement in rendering performance, however the way the pages are built can be more intelligent. Everytime you change an entity page in CRM, you need to publish it. What if during publication an optimized html page was created in which all javascript and css webresources are bundled and compressed. This would reduce the number of requests to the webserver (reducing web server stress).
  • Use Html5 features like Html5 application caching or local storage. Pages can be called from the local machine instead of calling the webserver.
  • To go even more crazy: how about implementing a mechanism on the online environment in which organizations would get a fixed number of threads based on the size and the subscription? This way Microsoft could guarantee the QOS (Quality of Service), because heavily used organizations wouldn’t affect other organizations on the tenant.

In the meanwhile, I really hope that Microsoft will be able to turn the crm4.dynamics.com environment into a speedy one. * Fingers crossed! *