My thoughts on the performance issue

At this moment we are having a Dynamics CRM online performance issue at one of our clients. In a joint effort with Microsoft we are trying to find the cause of the performance issue. As a backup scenario our client asked us to set up a full blown on-premise environment, which is –  as we speak –  up and running.

Speedometer

Our client is a world wide organization with offices in Europe and Asia and the head quarters based in the Netherlands. The asian office happened to have its own Dynamics CRM subscription.

This gave us the unique opportunity to test the solution we developed on a different tenant in a different CRM region…

The results were shocking:  One the left side we have the European tenant (CRM4) which is slow and unstable, on the right side we have the Asian tenant (CRM5) which is fast and stable despite the fact that we have to cross half the globe to access the data.

At this moment we have two environments (on-prem and crm5.dynamics.com) which are fast and show a consistent performance, and we have one environment (crm4.dynamics.com) which doesn’t have a consistent performance. To me this proves that the solution we built doesn’t have influence on the performance of Dynamics CRM.

My gut feeling tells me that we have an issue on the crm4.dynamics.com environment. I hope the Microsoft engineers can find the root cause.

In the investigation I did, I noticed that in order to call a page (e.g. Dashboard) from CRM an enormous amount of web requests are made to the Dynamics webserver. An average page has between 140 and 240 web requests. Furthermore the amount of data being transferred is big. A page can take a couple of megabytes of Html code, javascript, css files, images etc. This has a serious impact on the rendering experience in the browser.

Doing some investigation on an on-premise CRM environment (that I hosted in a virtual machine), I noticed that the w3wp.exe processes (these are the worker processes of Internet Information Server) that serve the Dynamics pages quite often have extreme high cpu loads for a longer period of time.
Can this be one of the causes of the dodgy performance we experience?

I suspect that due to the enormous amounts of webpages being served by the crm4.dynamics.com environment, the web frontend servers can’t keep up and literaly start to choke.

loads of organizations  x  loads of pages  x  100s requests per page x  couple of MB per page = webserver too busy!

I know you can’t compare the infrastructure of CRM on-premise with CRM online as a CRM online farm contains loads of load balanced servers. But a tiny voice in my heads keeps nagging…

This made me think of what could be changed within Dynamics CRM to improve the overall user experience, e.g.

  • The Turbo forms engine was already a big improvement in rendering performance, however the way the pages are built can be more intelligent. Everytime you change an entity page in CRM, you need to publish it. What if during publication an optimized html page was created in which all javascript and css webresources are bundled and compressed. This would reduce the number of requests to the webserver (reducing web server stress).
  • Use Html5 features like Html5 application caching or local storage. Pages can be called from the local machine instead of calling the webserver.
  • To go even more crazy: how about implementing a mechanism on the online environment in which organizations would get a fixed number of threads based on the size and the subscription? This way Microsoft could guarantee the QOS (Quality of Service), because heavily used organizations wouldn’t affect other organizations on the tenant.

In the meanwhile, I really hope that Microsoft will be able to turn the crm4.dynamics.com environment into a speedy one. * Fingers crossed! *

4 thoughts on “My thoughts on the performance issue

  1. Helly says:

    Having battled performance and stability issues with CRM Online for the last few months I feel for you. We ended up going down the route of changing datacenters and then changing server groups once moved (we had ended up with other busy tenants). Both these actions improved the situation somewhat but still baseline performance below what we would have expected and often intermittent failures and issues, like you we’ve had Microsoft support help over a long period as well as a lot of investigation ourselves. I did find recently that we had duplicate plugin steps active (both custom plugins and also Activity Feeds – this has also been reported by other people). Early days yet but once the dupes were deactivated there appears to be far better performance (even doing things you would not think affected by this) and stability.
    Hope you get to the bottom of your issues!

    • Hi Helly,

      thank you for your support. From all over the world I’m receiving the same reactions. Looking at the solution we developed: we don’t use plugins, use a couple of simple workflows, a little bit javascript (to hide and show fields). Nothing spectacular. We keep digging…. we want to hit the rock bottom 🙂

  2. Jonas Rapp says:

    I hope you can bring your experiences and questions to the “CRM Online Performance Optimisations and Analysis” session with Roger Gilchrist at eXtreme365 🙂

    Really interested in what he has to say about it!

  3. Chad Rexin says:

    Regarding your comment on the HTML5 and using local storage, if you use Cases, that can use that if you use the Interactive Service dashboard. Unfortunately this isn’t available for all entities yet. If you get to the bottom of these differences, I’d be interested in what you or Microsoft finds as the root cause.

Leave a Reply

Your email address will not be published. Required fields are marked *