Keep your Cache in Sync with PostgreSQL in NCache

NCache, an in-memory distributed cache, keeps your data intact in a temporary storage for a faster experience of read and write operations. We have seen NCache do wonders with the speed when it comes to performing operations on the cache store. However, keeping NCache synchronized with the main data source is a major challenge when it comes to data consistency. The data source can be any database; however, this blog is about keeping your cache synchronized with PostgreSQL.

Two Separate Data Copies – A Challenge

Your application’s stack works with NCache such as the cache resides as a caching layer in between the database and the application. Your application is directly connected to the cache, thus sending all the operation calls directly to the cache such as read or write. Let’s assume that another user changes something in the database without letting the cache know. Now the data in the database is different from the data residing in the cache. The two separate copies of data results in data inconsistency which is a big challenge.

NCache Details NCache Docs Sync Cache with PostgreSQL

NCache Keeping Data in Sync

Given the data inconsistency challenge, NCache swoops in with a fix for it just like everything else. What you need to do is, synchronize your cache with the database using NCache database synchronization techniques. The database discussed in this blog, as mentioned earlier, is PostgreSQL – an open source object relational database with a well-developed feature set and high performance and reliability. We have a solution made and uploaded on GitHub on Synchronizing the Cache with PostgreSQL which shows how can your cache have improved scalability and data consistency.

The diagram below shows how does NCache help you in keeping the data synchronized with your PostgreSQL database.

Figure 1: Synchronize Cache with PostgreSQL

Synchronizing NCache with PostgreSQL Database

NCache lets you keep your data synchronized with PostgreSQL using NotifyExtensibleDependency, that is a synchronization technique provided by NCache to keep the cache and database synchronized. NotifyExtensibleDependency is used to implement the data invalidation technique using PostgreSQL dependency notification system. PostgreSQL dependency is built on the LISTEN/NOTIFY mechanism that is similar to the Pub/Sub model.

Along with database dependency, ReadThruProvider is used in order to communicate with data source directly. NCache calls the provider to load data from the data source directly using your custom logic. This keeps the data synchronized and fresh to avoid problems like stale data, along with providing improved scalability and faster read operations.

So, the basic components involved are:

- NotifyExtensibleDependency: NCache registers with a specific channel by calling LISTEN within NotifyExtensibleDependency Initialize method. On the database, a TRIGGER function is invoked that sends a Payload to the channel and upon listening, in case of data inconsistency, the data is removed from the cache.
- ReadThruProvider: This provider gets the updated data automatically, directly from the database upon data modification in the database.

NCache Details Read-Through Notification Based Dependency

Synchronize Cache using NotifyExtensibleDependency

NotifyExtensibleDependency is used to invalidate the data using the mechanism of PostgreSQL Dependency, that works on the model of Pub/Sub messaging and the model is called Listen/Notify. According to this mechanism, any update in the database is passed to the client listening on a specific channel through a NOTIFY command with an additional payload containing the information. The client in our specific case is NCache and on being notified about any change in the database the corresponding item is removed from the cache.

Let us look at the code below to see how an item is added with PostgreSQL dependency:

const string connectionString = "your-connection-string";

var customer = new Customer();

var cacheItem = new CacheItem(customer)

{

Dependency = new PostGreSQLDependency(connectionString, customer.customerid, "public", "customers", "customer_channel")

};

var key = customer.customerid;

var cache = CacheManager.GetCache("democache");

cache.Insert(key, cacheItem);

However, you need to understand the implementation of PostgreSQLDependency and the code below will help. The Initialize method contains a connection string to connect to the database and registers for notification upon any data modification in the specific data.

public PostGreSQLDependency(string connectionString, string dependencyKey, string schema, string table, string channel)

{

_connectionString = connectionString.Trim();

_dependencyKey = dependencyKey.Trim();

_schema = schema.Trim();

_table = table.Trim();

_channel = channel.Trim();

}

public override bool Initialize()

{

connection = new NpgsqlConnection(_connectionString);

connection.Open();

((NpgsqlConnection)connection).Notification += (o, e) =>

{

var entity = JsonConvert.DeserializeObject(e.AdditionalInformation);

if (entity.DependencyKey == _dependencyKey && entity.Table == _table && entity.Schema == _schema)

{

done = true;

}

};

using (var cmd = new NpgsqlCommand($"LISTEN {_channel};", (NpgsqlConnection)connection))

{

cmd.ExecuteNonQuery();

}

task = Task.Run(() =>

{

while (true)

{

((NpgsqlConnection)connection).Wait();

if (done)

{

break;

}

this.DependencyChanged.Invoke(this);

});

return true;

}

NCache Details NCache Docs Sync Cache with PostgreSQL

Auto-Reload Data using Read-Through

Using dependency, whenever the data is modified in the data source, it is cleared from cache. However, we still need to reload that data in the cache as well and for that NCache has a feature called Backing Source. We provide Read-Through provider with the freedom of adding your custom logic in the implementation. In the specified case, the Read-Through provider loads data from PostgreSQL database. You need to configure the Read-Through provider using the management tools before.

Using resync options in the read-through provider keeps the data in sync with the database. However, inside the ProviderCacheItem, the CacheItem has dependency added with it which means the new item added in the cache comes with dependency already added to it. It saves you the effort of adding dependency over and over again. Look at the LoadFromSource method inside the Read-Through provider to look at the implementation:

public ProviderCacheItem LoadFromSource(string key)

{

var query = $"SELECT customerid, address, country, city FROM customers WHERE customerid = '{key}'";

// Define a query returning a single row result set

NpgsqlCommand command = new NpgsqlCommand(query, _connection as NpgsqlConnection);

var reader = command.ExecuteReader();

ProviderCacheItem providerCacheItem = null;

while (reader.Read())

{

if (providerCacheItem == null)

{

var customer = new Customer()

{

customerid = reader[0] as string,

address = reader[1] as string,

country = reader[2] as string,

city = reader[3] as string,

};

providerCacheItem = new ProviderCacheItem(customer)

{

Dependency = new PostGreSQLDependency(_connectionString, customer.customerid, "public", "customers", "customer_channel"),

ResyncOptions = new ResyncOptions(true)

};

}

Adding the item with read-through now, simply loads modified items directly into the cache using database dependency. Hence your cache stays synchronized with PostgreSQL database and all your stale data problems go away.

Conclusion

To put everything in the blog in simple terms, your cache and database need to remain synchronized with each other. If the cache is not aware of any modifications being made in the database, your application can keep on working and working with stale cache data. This problem is solved very efficiently with NCache PostgreSQL Dependency feature. There are tons of other cool features which you can see by simply going to NCache website.

NCache Details Edition Comparison NCache Download