Carrot2

Processing results cache takes up too much memory

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: 3.1.0
  • Fix Version/s: 3.1.1
  • Component/s: Framework Core
  • Labels:
    None

Description

It looks like after the upgrade to the newest version of Ehcache, the caches started taking up all available memory. I haven't yet looked at the code to see if this can lead to out of memory errors, but it may be an unpleasant surprise for those who embed Carrot2.

Issue Links

This issue is related to:
CARROT-584 Consider wrapping Ehcache entries with SoftReferences Major Open

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Dawid Weiss added a comment -

Maybe we should embed an example ehcache configuration to prevent this?

Show
Dawid Weiss added a comment - Maybe we should embed an example ehcache configuration to prevent this?
Hide
Permalink
Stanisław Osiński added a comment -

Yep. Right now I can't remember if we configure Ehcache entirely in the code or through some config, but my guess is that the defaults might have changed between Ehcache releases, hence the different memory usage.

Show
Stanisław Osiński added a comment - Yep. Right now I can't remember if we configure Ehcache entirely in the code or through some config, but my guess is that the defaults might have changed between Ehcache releases, hence the different memory usage.
Hide
Permalink
Stanisław Osiński added a comment -

After some brief investigation, it looks like the entries are not evicted from cache when their time to live is exceeded (the heap dump contains old entries, accessed only once).

Show
Stanisław Osiński added a comment - After some brief investigation, it looks like the entries are not evicted from cache when their time to live is exceeded (the heap dump contains old entries, accessed only once).
Hide
Permalink
Stanisław Osiński added a comment -

After some more investigation into this, it seems there's nothing wrong with Ehcache, it simply doesn't expire elements from memory cache:

Note that at any point in time there will usually be some expired elements in the cache. Memory sizing of an application must always take into account the maximum size of each cache. There is a convenience method which can provide an estimate of the size in bytes of the MemoryStore. See calculateInMemorySize(). It returns the serialized size of the cache. Do not use this method in production. It is very slow. It is only meant to provide a rough estimate.

The alternative would have been to have an expiry thread. This is a trade-off between lower memory use and short locking periods and cpu utilisation. The design is in favour of the latter. For those concerned with memory use, simply reduce the maxElementsInMemory.

(http://ehcache.org/documentation/storage_options.html)

I think we can safely decrease the number of in-memory elements from 1000 to 100. Still, it's interesting why the size of the cache increased 2 times or so on the demo server...

Show
Stanisław Osiński added a comment - After some more investigation into this, it seems there's nothing wrong with Ehcache, it simply doesn't expire elements from memory cache:
Note that at any point in time there will usually be some expired elements in the cache. Memory sizing of an application must always take into account the maximum size of each cache. There is a convenience method which can provide an estimate of the size in bytes of the MemoryStore. See calculateInMemorySize(). It returns the serialized size of the cache. Do not use this method in production. It is very slow. It is only meant to provide a rough estimate. The alternative would have been to have an expiry thread. This is a trade-off between lower memory use and short locking periods and cpu utilisation. The design is in favour of the latter. For those concerned with memory use, simply reduce the maxElementsInMemory.
(http://ehcache.org/documentation/storage_options.html) I think we can safely decrease the number of in-memory elements from 1000 to 100. Still, it's interesting why the size of the cache increased 2 times or so on the demo server...
Hide
Permalink
Stanisław Osiński added a comment -

I've compared heap dumps of the 3.0.1 and 3.1.0 webapps after executing 60 queries, and they look almost the same. I have no idea what caused the memory usage increase. For the time being, I've decreased the maximum cache size.

Show
Stanisław Osiński added a comment - I've compared heap dumps of the 3.0.1 and 3.1.0 webapps after executing 60 queries, and they look almost the same. I have no idea what caused the memory usage increase. For the time being, I've decreased the maximum cache size.
Hide
Permalink
Stanisław Osiński added a comment -

The issue is a result of the Ehcache design choice not to remove the obsolete element from the memory cache. Two short-term fixes have been done:

1. Lowering the number of in-memory cache elements from 1000 to 100. If you need further customizations of the Ehcache configuration, modify the default controller-ehcache.xml to suit your needs and put it on the class path to shadow the original configuration.

2. Disabling caching in the Document Clustering Server. If you need to re-enable caching, please modify the WEB-INF/classes/config.xml file included in the DCS WAR.

Show
Stanisław Osiński added a comment - The issue is a result of the Ehcache design choice not to remove the obsolete element from the memory cache. Two short-term fixes have been done: 1. Lowering the number of in-memory cache elements from 1000 to 100. If you need further customizations of the Ehcache configuration, modify the default controller-ehcache.xml to suit your needs and put it on the class path to shadow the original configuration. 2. Disabling caching in the Document Clustering Server. If you need to re-enable caching, please modify the WEB-INF/classes/config.xml file included in the DCS WAR.

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Carrot2. Try JIRA - bug tracking software for your team.