Battling performance issues

At the moment I’m working for one of our clients. The customer is migrating from Lotus Notes to Dynamics CRM online in combination with Office 365. Since the beginning of the project, the client has been plagued by an unpredictable performance.
One moment an account opens in just 2–3 seconds, while the next moment the same screen needs 25–30 seconds to open. A non workable situation for the client.

Performance issues can be caused by a large number of things. They can be caused by large datasets, customizations, client computers, the browser of choice, coorporate networks, problems at the ISP, problems within the Dynamics network, problems within Dynamics.

The biggest problem with fighting performance issues is availablity of time; you have to prepare yourself that finding the cause can and will consume a large amount of time and resources.

Where do you start? What strategy can you follow to isolate the performance problem? In this article I’ll describe the strategy we are following to find the cause of the unpredictable performance we are dealing with.

Step 1: Test on multiple browsers

Test at least on the majority of the following browsers:

  • Microsoft Internet Explorer 11
  • Microsoft Edge
  • Google Chrome
  • Mozilla FireFox
  • Safari on MacOS

This way you can eliminate the browser aspect. Some browsers will perform better than others (e.g. Chrome is faster than IE), but the difference in the initial load of a page should be pretty small.

Step 2: Test on multiple computers

Test on computers issued by the client, test on your own computer, test on other computers as well.

If the performance on the different computers differ significally, then you should take a look at the slow computer if this is the most used computer within the company. Find what is causing the slowness; this can be caused by a number of things:

  • Weak hardware (slow CPU, low in memory, conventional hard drive, slow network connection)
  • Installed OS (is the OS taking up too much resources)
  • Installed software (think anti virus, anti malware, windows services)

Make sure to test on a similar machine as well, to verify if the slowness also occurs on the other machine.

Step 3: Test on multiple network connections

Test on a number of network connections (wired, wireless, inside the corporate network, outside the corporate internet) to find out if there are differences in speed and behaviour.

If the speed from within the corporate network is significantly slower, then the corpote network needs further investigation.

Step 4: CRM itself

Once the environment has been checked, the CRM solutions present in the CRM environment need to be investigated as well. This is actually on of the first questions you will get from Microsoft support.

Check if you notice difference in performance if some customizations have been turned on or off. If the environment is much slower using the customizations then you need to take a serious look at it. The slowness can be caused by:

  • A large number of libraries that you include on an entity form
  • The javascript functions you call on the onload event of the entity form
  • Business rules applied on the entity
  • Workflows (realtime)
  • Plugins (what steps are registered)
  • The amount of fields on a form
  • The use of subgrids on a form (hide these in closed tab sections)
  • The fields shown in views (avoid related fields where possible)
  • The amount of rows shown in views (better keep it small)

If all customizations have been deactivated, you can start to reintroduce them one at the time. After the reintroduction, test if the performance starts to degrade. If so, then you might have found a cause of the problem.

Step 5: Fiddler can be your friend

In case you haven’t found anything so far, you can take a resort to tools like Fiddler. Using Fiddler you can see what calls are made from your browser to the CRM server (you can use it as well as a debugger tool) and how much time each call takes. This might give you a clue what is causing the slowness.

Step 6: External support

In case you didn’t manage to find the cause (which was our case), log a call at Microsoft Technical Support. They will contact you to try to solve the case. Be prepared for endless phone calls, a large number of emails and the number of actions you have to undertake to gather as much data as possible. Until now, my experience with the support engineers haven’t been that great; they keep repeating the same questions over and over.

Using our network, we were able to get a local former Microsoft Premier Field Support Engineer. We discussed what we have done until sofar to isolate the issue. He demoed an online environment he was using himself using a 4G dongle; a world of difference! His CRM was flying like a rocket (same region, same amount of customizations).

He came up with a number of good suggestions, one of them: ThousandEyes offers a network performance measurement tool that you can use to test your connections over a long period of time automatically. ThousandEyes can be used freely for 15 days. Exactly what we needed.

So, what’s next?

Our problem has not been solved yet. The customer is complaining about performance, we are trying to get a solid answer. In order to get a good answer I decided to do the following tests. These tests will run in the days to come:

  • Every two minutes we call the https://[org] page from the ThousandEyes hub in Amsterdam.
  • Using an end-point agent installed on a computer in the corporate network of the client we call the same page.



Time to pinpoint!

These test might look the same, but there is one big difference. The first test is run from the ThousandEyes hub and is intended to act as a benchmark.
Using this test we can prove that the performance of the CRM webserver is consistent over a long period of time.

The second test is setup to test the network performance of the client’s network. If calling the same page is taking much more time then the problem can be isolated to the client’s network; in that case we might need a network engineer!

In case both tests perform the same, then the problem might be in the CRM tenant itself.

For now, all we have to do is to gather data. Fingers crossed!