Latency and cost of Internet-based services are driving the proliferation of application-level caching to improve the user perceived latency, reduce the Internet traffic and improve the scalability and availability of origin servers. However, there is no transparent and automatic way to design and implement this type of caching, since that this usually depends on specific details of the application being cached. As a result, it requires application developers to reason about an orthogonal aspect, which is unrelated to the business logic. Moreover, this concern is typically addressed in an ad-hoc way and, consequently, is a time-consuming and error-prone task, becoming a common source of bugs. In this paper, we present the results of a qualitative study of how developers handle caching logic in their web applications, which involved the investigation of 10 software projects. The analysis of our results allowed us to extract cache-related concerns addressed by developers to improve their application's performance and scalability. Based on our analysis, we derived guidelines and patterns for application-level caching, which provide guidance to developers while designing, implementing and managing application-level caching, thus supporting developers in this challenging task that is crucial in enterprise web applications.
Click here if you would like to download the patterns in pdf format.In order to investigate different aspects of caching, it was important to select representative systems that make extensive use of application-level caching. To obtain applications that employ application-level caching, we searched through open-source repositories, which information can be easily retrieved for our study. Based on text search mechanisms, we assessed how many occurrences of cache implementation and issues were present in applications to ensure they would contribute to the study. The more caching-related implementation and issues, the better, so that the rationale behind choices regarding application-level caching could be extracted. Moreover, we managed to obtain commercial applications with partner companies interested in the results of this study.
Project name | Domain | Language | KLOC | Analyzed commit |
---|---|---|---|---|
S1 | Market trend analysis | Ruby | 21 | |
Pencilblue | CMS and blogging platform | Javascript | 33 | a3b378bb195a1641427a023f30728adb8fd85a94 |
Spree | E-commerce | Ruby | 50 | 8e74d76c41ce3b9dbb5e8f66f015793176a7451d |
Shopizer | E-commerce | Java | 57 | d3bb68ec908c440267e7084ed95392afe2aa1a1f |
Discourse | Platform for community discussion | Ruby | 88 | 16f509afb94553ee16fa50258de6937a8360878b |
OpenCart | E-commerce | PHP | 123 | 76a772f9a11f37d3ceb37866b45cd2beee42fc7a |
OpenMRS API and web application | Patient-based medical record system | Java | 127 | 971ada09e1763cb9df57dc290958f0de34029788 |
ownCloud core | File sharing solution for online collaboration and storage | PHP | 193 | b877b0508bb08868909b2b270a7753a9ba868c7a |
PrestaShop | E-commerce | PHP | 245 | a36c389729429fb7395f903d2cc6ebd57ab958e9 |
Open edX | Online courses platform | Python | 250 | 22e01a8c9527add15e8bcc9589c02900fbb2c63b |
Based on the analysis, we have identified application-level caching decisions and behaviors adopted by developers. Our findings allowed us to propose a set of guidelines for the development of a caching approach.
Evaluate different abstraction levels to cache. It is important to cache data where it reduces the most processing power and round trips, choosing locations that support the lifetime needed for the cached items, despite where it is located in the application. Different levels of caching provide different behavior, and possibilities must be analyzed. For instance, caching in the model or database level offers higher hit ratios, while caching in presentation layer can reduce the application processing overhead significantly in the application in the case of a hit. However, in the latter case, hit ratios are in general lower. It is possible to cache data at various layers of an application, according to the following layer-by-layer considerations.
Stack caching layers. It is reasonable to say that the more data cached, the lower the chance of being hit without any content already loaded. Caching might be at the client, proxy server, inside the application in presentation, business, and model logics, or database. Despite the same data may be cached in multiple locations, when the cache expires in one of them, the application will not be hit with an entirely uncached content, avoiding processing and network round trips. However, it is important to keep in mind that caching layers imply a range of responsibilities, such as consistency conditions and constraints, and extra code and configuration. Due to this, it is important to consider many caching layers but, at the same time, achieve a good trade-off between caching benefits and implementation effort.
Separate dynamic from static data. Content can be distinguished in static, dynamic, and user-specific. By partitioning the content, it is easier to select portions of the data to cache.
Evaluate application boundaries. Communication between application and external components is a common bottleneck and, consequently, an opportunity for caching. Consider caching for database queries, remote server calls and requests to web services, which are made across a network.
Specify selection criteria. Selecting the right data to cache involves a great reasoning effort given that data manipulated by web applications range in dynamicity, from being completely static to changing constantly. To optimize this selection process, there are four primary selection criteria used by developers while detecting cacheable content, which should be used in decisions regarding whether to cache. These criteria are described below, ordered according to their importance; i.e.\ the higher the influence level, the earlier it is presented.
Evaluate staleness and lifetime of cached data. Every piece of cached data is already potentially stale, it is important to rethink the degree of integrity and potential staleness that the application can compromise for increased performance and scalability. Many cache implementations adopted an expiration policy to invalidate cached data based on a timeout since weak consistency is easier than defining a hard-to-maintain, but more robust, invalidation process. In short, developers must ensure that the expiration policy matches the pattern of access to applications that use the data, which is based on determining how often the cached information is allowed to be outdated, and relaxing freshness when possible.
Avoid caching per-user data. It is recommended to avoid caching per-user data unless the user base is small and the total size of the cached data does not require an excessive amount of memory; otherwise, it can cause a memory bottleneck. However, if users tend to be active for a while and then go away again, caching per-user data for short-time periods may be an appropriate approach. For instance, a search engine that caches query results by each user, so that it can page through results efficiently.
Avoid caching volatile data. Data should be caches when it is frequently used and is not continually changing. Developers should remember that caching is most effective for relatively stable data, or data that is frequently read. Caching volatile data, which is required to be accurate or updated in real time, should be avoided.
Do not discard small improvements. The user perceived latency is reduced by any caching solutions employed. This means that even not obvious scenarios should be target of caching, i.e.\ it is not true that solely data that is frequently used and expensive to retrieve or create should considered for caching. Furthermore, data that is expensive to retrieve and is modified on a periodic basis can still improve performance and scalability when properly managed. Caching data even for a few seconds can make a large difference in high volume sites. If the data is handled more often than it is updated, it is also a candidate for caching.
Keep the cache API simple. Caching logic tends to be spread all over the application, and a good solution should be employed to avoid writing messy code at the cost of high maintenance efforts.
Define naming conventions. To define appropriate names for cached data, it is important to assign a name that is related to its context, the data itself, and the caching location. It can provide two direct benefits: (a) prevention of key conflicts, and (b) guidance of cache actions such as updates and deletes of stale data in case of changes in the source of information.
Perform cache actions asynchronously. For large caches, it is adequate to load the cache asynchronously with a separate thread or batch process. Moreover, when an expired cache is requested, it needs to be repopulated and doing so synchronously affects response time and blocks the request processing thread.
Do not use cache as data storage. An application can modify data held in a cache, but the cache should be considered as a transient data store that can disappear at any time. Therefore, developers should not save valuable data only in the cache, but keep the information where it should be as well, minimizing the chances of losing data if the cache unexpectedly becomes unavailable.
Perform measurements. Caching is an optimization technique, and as any optimization, it is important perform measurements before making substantial changes, given that not all application performance and scalability problems can be solved with caching. Furthermore, if unnecessarily employed, caching can eventually decrease performance rather than improve it.
Document and report measurements. To compare and reproduce performance tests employed, it is important to document the setup used to perform the application analysis. It includes modules enabled, particular configurations and hardware settings.
Consider using supporting libraries and frameworks. Supporting libraries and frameworks can raise the level of abstraction of cache implementation and provide useful features. In addition, they can scale in a much easier and faster way than application-wide solutions.
Tune default configurations. Default configurations provided by external components serve as a start point. However, because they are generic and may not fit the application specificities, other configurations must be evaluated and possibly adopted.
Use of transparent caching components. The use of transparent caching solutions to address bottlenecks outside the application boundaries such as databases, final HTML pages or fragments, and static assets can provide fast results. These solutions do not explore application specificities, but can still provide performance benefits for typical usage scenarios.
Based on our study, we derived patterns of explanations, which can be used for supporting a decision made by a software system. Moreover, we identified the components these patterns must have, which comprise a template for a caching pattern catalog. These components are (i) a classification, (ii) the pattern intent, (iii) the solution proposed, and (v) an example.
Implementation
Under certain circumstances the cost of populating the cache is very expensive (data provided by third party systems via webservices or the result of a heavy calculation process, and others). In such a scenario whenever the cache is invalidated (by automatic expiration or invalidation) the request processing is blocked while getting fresh data, which affects client-response time.
The method in the following code example shows an implementation of the Cache-aside pattern based on asynchronous processing. An object is identified by using an ID as the key. The asyncGet method uses this key and attempts to retrieve an item from the cache. If a matching item is found, it is returned. If there is no match in the cache, it should retrieve the object from a data store, adds it to the cache, and then returns it (the code that actually retrieves the data from the data store has been omitted because it is data store dependent). Note that the load from cache has a timeout, in order to ensure that the cache interaction do not block the processing.
// Get a cache client
// it can be a trird-party library or an implemented module
// this facade should provide at least get, set and remove methods
Cache cache = Cache.getInstance();
public List<Product> getProducts() {
List<Product> products = null;
Future<Object> f = (List<Product>) cache.asyncGet("products");
try {
// Try to get a value, for up to 5 seconds, and cancel if it
// doesn't return
products = f.get(5, TimeUnit.SECONDS);
// throws expecting InterruptedException, ExecutionException
// or TimeoutException
} catch (Exception e) {
// Since we don't need this, go ahead and cancel the operation.
// This is not strictly necessary, but it'll save some work on
// the server. It is okay to cancel it if running.
f.cancel(true);
// Do other timeout related stuff
}
if (products == null) {
products = getProductsFromDB();
// updates into cache should not block the request
// return the user request as soon as possible
cache.asyncSet("products", products);
}
return products;
}
public Product getProduct(String id) {
Product product = null;
Future<Object> f = (Product) cache.asyncGet("product" + id);
try {
product = f.get(5, TimeUnit.SECONDS);
} catch (Exception e) {
f.cancel(true);
}
if (products == null) {
product = getProductFromDB(id);
// updates into cache should not block the request
// return the user request as soon as possible
cache.asyncSet("product" + id, products);
}
return product;
}
The code below demonstrates how to invalidate an object in the cache when the value is changed by the application. The code updates the original data store and then removes the cached item from the cache by calling the Remove method, specifying the key.
The order of the steps in this sequence is important. If the item is removed before the cache is updated, there is a small window of opportunity for a client application to fetch the data (because it is not found in the cache) before the item in the data store has been changed, resulting in the cache containing stale data.
public void updateProduct(Product product) {
updateProductIntoDB(product);
cache.asyncDelete("products");
//optionally, it is possible to update the data into cache
cache.asyncSet("product" + id, product);
}
public void deleteProduct(String id) {
deleteProductFromDB(id);
cache.asyncDelete("products");
cache.asyncDelete("product" + id);
}
Design
Cache has limited size, so it is important to use the available space to cache data that maximizes the benefits provided to the application. Otherwise, it can end up reducing application performance instead of improving it, consuming more cache memory and at the same time suffering from cache misses, where the data is not getting served from cache but is fetched from the source.
Even though there are many criteria that contribute for identifying the level of data cacheability, there is a subset that would confirm this decision regardless of the values of the other criteria. Figure below expresses a flowchart of the reasoning process to decide whether to cache or not data, based on the observation of data and cache properties. Changeability is the first criterion that should be analyzed while selecting cacheable data, then usage frequency, shareability, computation complexity, and cache properties should be considered. All criteria are tightly related to the application specificities and should be specified by the developer.
In addition, we list content properties that should be avoided, which do not convey the influence factors in a good way and lead to problems such as cache trashing.
We list some typical scenarios where data should be cached and also give explanations based on the criteria presented.
Design and Maintenance
It is usually impractical to expect that cached data will always be completely consistent with the data in the data store. Applications should implement a strategy that helps to ensure that the data in the cache is up to date as far as possible, but can also detect and handle situations that arise when the data in the cache has become stale. An inappropriate expiration policy may result in frequent invalidation of the cached data, which negates the benefits of caching.
Every piece of cached data is already potentially stale, and a good trade-off between performance benefits and cost of invalidation approaches should be achieved. Its necessary to determine the appropriate time interval to refresh data, and design a notification process to indicate that the cache needs refreshing. If the data is held too long, it runs the risk of using stale data, and if it was expired too frequently, it could affect performance.
Deciding on the expiration algorithm that is right for the scenario includes the following possibilities:
Figure above expresses a flowchart with the reasoning process to decide the appropriate consistency approach, based on observation of data properties. Changeability is the first property that should be analyzed while deciding, then staleness level and the amount of operations and dependencies related to the data should be considered. All properties are tightly related to the application specificities and should be defined by developer.
Consider a stock ticker, which shows the stock quotes. Although the stock rates are continuously updated, the stock ticker can safely be removed or even updated after a fixed time interval of some minutes.
Implementation
Keys are important to keep track of the content cached while debugging or when it is necessary to invalidate and delete stale data from cache, in the case of changes in the source of information.
When choosing a cache key, you should ensure that it is unique to the object being cached, and that it appropriately varies by any contextual values.
The key can be scoped to its particular function area, and be formatted with varying parameters. Its make easy to debug and track caching values.
var key = string.Format("MyClass.MyMethod:{0}:{1}", myParam1, myParam2);