Pub/Sub Design Pattern in .NET Distributed Cache

Pub Sub

Publisher Subscriber design patterns are an invaluable tool for building enterprise grade .NET/C# applications. Among them you will be very familiar with the Publish-Subscribe pattern also known as Pub/Sub. Just to refresh your memory, Pub/Sub is basically a messaging pattern where the senders of messages (publisher) do not have any knowledge about the intended recipients as in, which applications are the receivers or how many of them are there. Also, the publishing and listening application do not interact with each other directly but instead depend on a common medium, without knowledge of the producer or consumers presence.

This publish-subscribe design pattern is used when there are multiple applications of different roles deployed within an architecture and they need a mechanism to inform other applications about different events. These events could either be transient changes or data based. Transient changes are the data that the applications creates at runtime whereas the data based changes are the changes made in a data store e.g. databases. To implement this design pattern, developers traditionally turn towards using either of the 3 following methods.

1. Data Notifications provided by RDBMS

If the data store is limited to a relational database, using the database notification feature is the best available option. The database notification feature allows you to register your interest with the database server. Any change in the database result set, whether updated, added or delete, your applications is notified. This feature is mostly covered by all of the major relational databases like Microsoft SQL, Oracle Database e.t.c.

But these RDBMS are already at their limits because of their inability to scale and thus become a performance bottleneck. Since you do have to deal with it at present or future, therefore, you would not want to put unnecessary load on your databases. Moreover, this database notifications feature is inherently slow and it is advised to use only if you have no other options available. Another problem with it is that these notifications are only meant for data changes and do not cover other transient changes your application need to propagate. Therefore, using a database as a messaging medium is a not the best design choice.

2. Messaging Queues

The other option you have is to introduce a separate messaging queue in your architecture. While these messaging queues do a great job in helping you to transmit messages among applications, these queues are not data focused i.e. they do not monitor data changes in the databases or any other source. Also, these messaging queues cannot scale along side your applications tier.

3. Implement a Custom Solution

The last option left for you is to build your own Messaging platform/medium to suit your need. While this empty sandbox is very tempting at first, allowing you to build your own stuff no matter how you want to but there is the complexity, time and resources required for you to consider. While possible, building and managing a robust and scalable messaging platform is a very daunting task.

Now, the question remains that which solution is the easiest to use yet still covers all of the possible scenarios? A solution which is easy to incorporate, is scalable, highly available and yet also very reliable.

A Distributed Cache as a Messaging Platform

Fret not, there is an easy solution. A more modern way to incorporate a robust Message platform is to use an in-memory distributed cache. NCache is one such in-memory distributed Cache which is extremely fast and scalable and helps your applications to handle extreme transaction loads without your database becoming a bottleneck. NCache is usually deployed in the middle tier of your n-tier architecture. Take a look at the following diagram to better illustrate the deployment.

Figure1: NCache deployed in an N-Tier architecture

Here as you can see, NCache is basically a cluster of cache servers serving tens of thousands of requests per second for your .NET applications as well as Java apps by keeping frequently used data in-memory and helps avoid majority of the database trips.

You can use the same platform as a messaging medium for Pub/Sub design pattern. NCache is inherently made to be highly available in case of server crashes and also provides your applications the best possible performance. Being highly available and robust, none of your messages will be lost in worst case scenarios and your client applications will keep on working seamlessly. To understand how, see the following diagram which illustrates the above diagram but with a different perspective, in it data stores are hidden behind the cache cluster.

Figure2: NCache as a Messaging Platform

Here NCache acts as a messaging medium for your applications. To cater for different languages using the same medium, NCache uses fast compact serialization to convert your .NET or Java objects to binaries and transfer them over to the cache cluster. Using this feature, .NET applications can also interact with Java applications and vice versa. For more information take a look into portable data types. This allows your .NET and Java applications to use the same medium and coordinate through transferring native objects.

NCache undertakes PubSub design pattern under the name of Events and provides you with different methods to propagate your messages to other listening applications. Let’s take a look at both message types and see how a distributed cache can propagate them.

Let’s first consider the data changes your applications need to listen to. Since NCache is also a .NET key value store, therefore it also provides you with features to update your applications in case of any data changes within the cache. Because all of this is in-memory therefore there is no performance bottleneck. These updates could either be;

  • Cache Item level changes (Update or Remove)
  • Whole cache level data changes
  • Continuous Query
    • Where you register an SQL like query to be watch-full for if the result set changes within the cache. If it does your applications are notified.
  • Cluster changes, if any new node is added or removed or crashed. (For administration)

NCache also allows you to register SQL dependency with Microsoft SQL Database. This helps to keep your cache up to date with the database and thus your applications. For a full list of supported dependency types, check the database depndency page. These dependencies are actually data change notifications registered with different data stores but you let NCache handle it. Combine database notifications with NCache data notifications to enrich your specific use case.

On the other hand, if you want to just propagate simple messages to complex .NET or Java objects then you should try to use the custom messaging feature. Here one application can produce the data and fire an event, whereas the interested listeners will receive the event almost instantaneously. You can check the implementation under custom events documentation.

As you can see, that an in-memory distributed cache like NCache helps you to incorporate a messaging platform which allows you to build your applications on a robust platform without any worry of loosing your data. With NCache you have multiple ways to handle your data changes or your transient messages while scaling your infrastructure as needed and whenever needed.

Avatar for Basit Anwer

Leave a Reply

Your email address will not be published. Required fields are marked *