NCache Performance Counters: Analysis and Monitoring

NCache performance counters are very helpful to analyze application's performance. These counters can also help identify any problem within NCache or the network where NCache is operating. Performance counters are also used to distinguish between different products' throughput, so it is very important to understand them correctly.

In this document some of the main NCache performance counters are explained and how they should be used in order to analyze the performance or an issue.

Analyze Performance

NCache is mainly used to store application data as custom objects or sessions. So application's performance is highly dependent on NCache. NCache counters are divided in two parts; Client Side Counters and Server Side Counters.

Server Side Counters

Server side counters shows the overall performance of the clustered cache. They mainly provide with the information about how fast NCache server nodes reply to application requests. The counters that should be monitored for performance analysis are given below:

[NCache Cache Specific] Fetches/sec: This counter is the measure of number of fetch operations per second.
Note: Here fetch operations are Get or GetCacheItem operation calls.
[NCache Cache Specific] Additions/sec: This counter is the measure of number of add operations per second.
[NCache Cache Specific] Additions/sec: This counter is the measure of number of add operations per second.
[NCache Cache Specific] Updates/sec: This counter is the measure of number of insert operations per second.
[NCache Cache Specific] Deletes/sec: This counter is the measure of number of remove operations per second.
[NCache Cache Specific] Average ms/fetch: This counter is the measure of average time, in milliseconds, taken to complete one fetch operation.
Note: Here fetch operations are Get or GetCacheItem operation calls.
[NCache Cache Specific] Average ms/insert: This counter is the measure of average time, in milliseconds, taken to complete one insert operation.
[NCache Cache Specific] Average ms/remove:: This counter is the measure of average time, in milliseconds, taken to complete one remove operation.
[NCache Cache Specific] Average ms/cache operation: This counter is the measure of average time, in seconds, taken to complete one cache-operation.
[NCache Cache Specific] Evictions/sec: This counter is the measure of number of items evicted per second.
[NCache Server] Requests/sec:This counter is the measure of number of requests received (meaning cache commands like add, get, insert, remove etc.) from all clients by this cache server.

The first four counters in this list give the number of fetches, additions and updates per second. These are basically application requests on NCache.

The next four counters give the time spent on each request in milliseconds. If the numbers are high for these counters then it means that either the cache item is huge or the cluster is busy in other operations like state transfer, data balancing etc. For an object of size less than 10 KB it is normally less than a millisecond.

Requests/sec is the average of all the request on a cache server node whether they are additions, fetches or updates. A single NCache node can handle around 8000 requests/sec with 1KB object size.

If your cache becomes full and cache is evicting items then you will see a value for evictions/sec counter. Evictions can slow down the performance so it is important to keep log of this counter.

Client Side Counters

Client Side counters give the best view for NCache performance. From these counters you can tell the performance boost given by NCache to your application. Here are the counters that should be monitored:

[NCache Client] Average ms/add: This counter is the measures of average time, in milliseconds, spent to perform a single add operation.
[NCache Client] Average ms/fetch: This counter is the measure of average time spent to perform a single fetch operation.
Note: Here fetch operations are Get or GetCacheItem operation calls.
[NCache Client] Average ms/remove: This counter is the measure of average time spent to perform a single remove operation.
[NCache Client] Average ms/insert: : This counter is the measure of average time spent to perform a single insert operation.
[NCache Client] Average Item Size: This counter gives information regarding the average size of data added by a client in cache at a given time.
Note: If compression is enabled the average size is calculated using the compressed value.
[NCache Client] Compression/sec: This counter is the measure of rate of compression performed on the data per second.
Note: Compression per second occurs when compression is enabled. Also size is specified in bytes.
[NCache Client] Request Queue Size: This counter is the measure of the size of a request queue at a given instance on a cache client.
Note: Here requests are the operation calls for cache server. Also size is specified in bytes.

The first four counters here give the time spent by NCache to serve a given request, whether it is addition, fetch or an update. For a 1 KB object the time should be less than 2 ms, it can even be less depending on the network latency. Less than 10 KB object should be around 3 ms. If it's more than 3 ms then you should tune NCache settings so that it can perform better.

The first four counter values depend heavily on average item size. Large items will take more time to serve as compared to a small item. So it is important to log average items size if you are logging counters for performance analysis.

Compression/sec counter will only show value if you are using NCache compression feature.

Request queue size will have all the pending requests. If this value is constantly increasing then NCache cluster is unable to serve requests in a timely fashion. In this case you would either need add more nodes to the cluster or diagnose the problem on cluster side or network between the client server and cache server.

Identifying an Issue

Customers mostly face issues like slow performance, application not responding or the application is not fetching data from cache.

In case of a slow performance caused by NCache the value of above mentioned client side counters will constantly increase (especially request queue size). This will indicate that the problem is arising due to NCache. Involve Alachisoft Support if this is the case
If application is not fetching data or application is not responding then NCache client side can help to identify that whether the issue is with NCache or the application itself. If you are seeing that requests are being served by NCache in a timely manner then NCache is performing fine, the issue is with application itself

Here are some more counters from NCache that can be used for monitoring purposes:

NCache Client

[NCache Client] Fetches/sec: This counter is the measure of the number of fetch operations called by the cache client per second.
Note: Here fetch operations are Get or GetCacheItem operation calls. This counter is incremented irrespective of the success or failure of the operation.
[NCache Client] Additions/sec: This counter is the measure of the number of add operations called by the cache client per second.
Note: Addition/sec will also be incremented in case of a new key being added with insert operation call.
[NCache Client] Avg ms/event: This counter is the measure of average time spent to process a single event.
Note:Time is in milliseconds.
[NCache Client] Deletes/sec:This counter is the measure of delete/remove operation requests called by a cache client per second.
[NCache Client] Event Processed/sec: : This counter is the measure of the events executed by a cache client per second.
Note:This counter only exists in NCache 4.1 SP3
[NCache Client] Event Triggered/sec: This counter is the measure of the number of events received by a cache client per second.
Note:Events received by a client are thrown by the cache server. This counter only exists in NCache 4.1 SP3.
[NCache Client] Read Operations/sec: This counter is the measure of read operations called by a cache client per second.
Note:Here read operations are Get/Remove operations, also this counter is incremented irrespective of the success or failure of the operation.
[NCache Client] Updates/sec: This counter is the measure of the insert operations called by a cache client per second.
Note:Update/Sec will not increment in case of a new key being added with insert operation call instead addition/sec will be incremented.
[NCache Client] Write Operations/sec: This counter is the measure of the write operations called by a cache client per second.
Note:Here write operations are add/insert/remove/delete operation call.

NCache Server

[NCache Server] Responses/sec: This counter is the measure of number of responses sent (that is, cache response for commands like add, get, insert, remove etc.) to all clients by a specific cache server.
[NCache Server] Client Requests/sec: This counter is the measure of number of requests sent by all clients to the cache server.
[NCache Server] Client Responses/sec: This counter is the measure of number of responses received by all clients from the cache server.
[NCache Server] Client bytes sent/sec: This counter is the measure of this counter is the measure of bytes being sent from cache server to all its clients.
[NCache Server] Client bytes received/sec: This counter is the measure of bytes being received by cache server from all its clients.
[NCache Server] Socket send time (msec): This counter is the measure of time in milliseconds it took for the last message to be sent on over the socket.
[NCache Server] Socket send size (bytes): This counter is the measure of amount of data sent in the last message.
[NCache Server] NaglingMsgCount: This counter is the measure of time in milliseconds for which a sequenced message waits before it is processed.
[NCache Server] Socket recv time (msec): This counter is the measure of time in milliseconds it took to receive the last message.
[NCache Server] Socket recv size (bytes): This counter is the measure of how much data was received in the last message.
[NCache Server] Response Queue Count: This counter is the measure of number of items in response queue.
[NCache Server] Response Queue Size:This counter is the measure of size of response queue specified in bytes.

NCache Cluster

[NCache Cluster] State transfer/sec: This counter is the measure of number of items this node is either reading from other nodes or sending to other nodes during a state transfer mode.
[NCache Cluster] Mirror queue size: This counter is the measure of number of items in the 'Mirror Queue'.
Note:This value is only set in case of mirror topology.
[NCache Cluster] Sliding Index queue size: This counter is the measure of number of items in the Sliding-Index queue. Note: During replication in different topologies
Note: During replication in different topologies
[NCache Cluster] Data balance/sec: This counter is the measure of number of items a particular node is either reading from other nodes or sending to other nodes during a Data Load Balancing mode.
[NCache Cluster] Cluster ops/sec: This counter is the measure of number of clustered operations performed per second.
[NCache Cluster] Clustered opssent/sec: This counter is the measure of number of clustered operations sent to other nodes in cluster per second.
[NCache Cluster] Clustered opsrecv/sec: This counter is the measure of number of clustered operations received from other nodes in cluster per second.
[NCache Cluster] Response sent/sec: This counter is the measure of number of responses sent to other nodes in cluster per second.
[NCache Cluster] Bytes sent/sec: This counter is the measure of number of bytes sent per second to other nodes of the cluster.
[NCache Cluster] Bytes received/sec: This counter is the measure of number of bytes received per second from other nodes of the cluster.
[NCache Cluster] BcastQueueCount: This counter is the measure of number of items in Broadcast queue waiting to be processed on sequence.
[NCache Cluster] McastQueueCount: This counter is the measure of number of items in multicast queue waiting to be processed on sequence.

Cache Specific

[NCache Cache Specific] Event Queue Count: This counter is the measure of number of items in event queue.
Note: This counter only exists in NCache 4.1 SP3.
[NCache Cache Specific] Readthru/sec: This counter is the measure of number of readthru operations per second.
[NCache Cache Specific] Writethru/sec: This counter is the measure of number of writethru/writebehind operations per second.
[NCache Cache Specific] General Notifications Queue Size: This counter is the measure of number of General Notification Events in the queue.
[NCache Cache Specific] Count: This counter is the measure of number of items in the cache.
[NCache Cache Specific] CacheLastAccessCount: This counter is the measure of number of items which are older than the access interval specified in the config file.
[NCache Cache Specific] Average ms/cache operation: This counter is the measure of average time, in milliseconds, taken to complete one cache operation.
[NCache Cache Specific] Expirations/sec: This counter is the measure of number of items being expired currently per second.
[NCache Cache Specific] Hits/sec: This counter is the measure of number of successful fetch operations per second.
Note: Here fetch operations are Get or GetCacheItem operation calls
[NCache Cache Specific] Misses/sec: This counter is the measure of number of failed fetch operations per second.
Note: Here fetch operations are Get or GetCacheItem operation calls.
[NCache Cache Specific] Hits ratio/sec (%): This counter is the measure of ratio of 'number of successful Get operations per second' to 'total number of Get operations per second'.